Method and system for training a Foundation model

Abstract

The present application relates to a method for training a Foundation model, comprising the steps of: providing (S1) at least one image and / or video datum containing image information from at least one domain and at least one image identifier; providing (S2) at least one general knowledge graph (GKG) containing information about the at least one domain; extracting (S3) a sub-graph from the general knowledge graph depending on the at least one image identifier; generating (S4) at least one textual sentence depending on the extracted sub-graph and depending on the at least one image identifier; converting (S5) the at least one textual sentence into at least one sentence feature vector using a large language model (LLM); and comparing (S6) the at least one sentence feature vector with an image feature vector of the at least one image and / or video datum.Training (S7) the Foundation model based on the comparison; and deploying (S8) the trained Foundation model.

Description

The invention relates to a method and a system for training a foundation model. The invention further relates to a method for classifying and / or categorizing and / or segmenting image and / or video data, and to a computer program having program code and a computer-readable data carrier.Prior ArtAdvances in machine learning have led to extensive developments in artificial intelligence, particularly in the field of visual perception. One aspect of this development is the ability of neural networks to create visual embeds. Visual embedding is a compact representation of visual input that allows a computer to understand and act based on visual information.In this context, the foundation model or base model has proven to be an effective method to generate visual embeddings based on linguistic embedding. The foundation model uses a deep neural network (DNN) trained to capture the semantic meaning of a text description and generate a corresponding visual representation.The use of DNNs allows the base model to learn complex relationships between linguistic and visual information and to transition high-dimensional visual data to a low-dimensional representation. This makes it possible to process visual information more efficiently and to use it in other applications such as image recognition, video analysis or even in the development of autonomously acting robots.The training of a foundation model usually requires large amounts of annotated data in which visual inputs are linked to corresponding text descriptions. By learning from this data, the model can develop a general ability to generate visual embeds that can be applied to a wide range of tasks. The known training methods also have optimization potential.From the scientific publication "Learning Visual Models using a Knowledge Graph as a Trainer."; https: / / arxiv.org / pdf / 2103.00020.pdf, a method for training deep neural networks with semantic knowledge graphs / ontology is known.From the scientific publication "Radford, A., Kim, J.W., Hallacy, C., Ramsh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Kruger, G., & Sutskever, I. (n.d.). Learning Transferable Visual Models From Natural Language Supervision" is a general method that learns a foundation model (german: basic model) from text and images. Here, a base model is trained with a deep neural network (DNN) to generate visual embedding based on a speech embedding given by the text description. However, particularly in the case of small amounts of data with respect to the image and / or video data, it is challenging to create balanced and / or illustrative image descriptions.From the scientific publication "Santurkar, S., Dubois, Y., Taori, R., Liang, P., & Hashimoto, T. (2022). Is a Capture Wordh a Thick Images? A Controlled Study for Representation Learning. http: / / arxiv.org / abs / 2207.07635" discloses the meaning of a degree of description of inscriptions of image and / or video data, in particular in the case of low data volume.Approaches such as Radford et al. learn robust basic models by using descriptive captions. These labels are then converted into latent vectors, i.e., word embeds, using language models. Then, the embeddings are used to control the learning process of the basic models. Both the text corpus and the images are in this case cached from the web.In low data scenarios, such as road sign recognition, there are no detailed labels for each sign in the publicly available text corpora. Further, the conventional finding method with the standard cross entropy loses a large part of the generalization strength of the basic model.In the case of finalizing with manually created inscriptions, moreover, much manual work is performed without a relevant underlying schema. This can be difficult and prone to errors, especially in the case of very specific domains.The object of the invention is thus to specify an improved method for training a foundation model. It is also an object to specify an improved trained foundation model.The object is achieved by a method for training a foundation model according to the features of claim 1. the object is achieved by a system for training a foundation model according to the features of claim 12.Disclosure of the InventionIn the present case, according to a first aspect, a method for training a foundation model is proposed. The method comprising the steps of:providing at least one picture and / or video data item comprising picture information of at least one domain and at least one picture identification;providing at least one general knowedge graph, GKG, having information about the at least one domain;extracting a partial graph from the general know-wedge graph as a function of the at least one image label;generating at least one textual sentence depending on the extracted sub-graph and depending on the at least one image label;converting the at least one textual sentence into at least one sentence feature vector by a Large Language Model, LLM;comparing the at least one set feature vector with an image feature vector of the at least one image and / or video datum;training the foundation model based on the comparison; andproviding the trained foundation model.It goes without saying that the steps according to the invention and further optional steps do not necessarily have to be carried out in the sequence shown, but can also be carried out in a different sequence. Further intermediate steps can also be provided. The individual steps can also comprise one or more substeps without thereby departing from the scope of the method according to the invention.By the present application of a general-knowledge graph, in particular a domain-specific graph, which preferably follows an underlying ontology, it is possible to generate highly descriptive image and / or video descriptions for domain-specific image and / or video data. With the aid of a meaningful and / or comprehensive and / or comprehensive description of the domain by a general know-edge graph, it is possible in the present case to train the foundation model in a more efficient manner. The foundation model can be more specifically matched to new and / or changing regions and / or situations of a domain, wherein already existing training data in the form of image and / or video data can furthermore be used. The model performance of the foundation model trained in this way is thereby improved compared to the prior art.In the present case, a method for training a foundation model, which can be described in particular by a deep neural network, by means of, in particular semantic, knowledge graphs / ontologys is described. In this context, in particular contrasting methods between embedding knowledge graphs and embedding neural networks are described. In this case, in particular, the fine tuning of basic models with domain-specific contents is examined, wherein the class-specific information is extracted from the knowledge graph, in particular in the form of RDF molecules. More preferably, a set is generated for each molecule. This is then used as input for a large speech model, large language model (LLM for short).Herein, a visual base model is refined to fit new domains and / or new domain parameters and / or domain information. This can be used, for example, for the correct recognition of traffic signs. Especially if one has limited access to (training) data, it is important to use very clear and meaningful labels to train more efficiently, i.e., with less domain specific data, and yet develop a more powerful foundation model. This is achieved in the present case. In the present case, relevant knowledge about a domain in the form of at least one sub-graph, in particular a so-called RDF molecule, can be extracted by the framework of the method from an, in particular generic, general-knowledge graph (GKG). These "molecules" are preferably transformed into natural language phrases as well as general latent vector representations (language embeds) using a large language model (LLM). Subsequently, the foundation model, which is preferably represented by a deep neural network, is trained to adapt a visual embedding from the at least one image and / or video data to the speech embeddings generated by the general know-wedge graph in the form of the natural-speech phrases.The present method also relates to the adaptation of an in particular polynomial reduction algorithm for utilizing word size displacements, which are in particular adapted and / or adaptable to a word size of a predetermined computer hardware. Based thereon, the technical effect of an efficient hardware implementation of the algorithm or foundation model can be achieved."Image labeling", also known as labeling, refers to a process of assigning at least one category and / or tag to visual data, such as images and / or video. The label or label is preferably a description or tag that identifies and identifies specific features, objects or classes in an image. The objective of the image label is preferably to understand and categorize the visual content to train machine systems, interpret visual information, and make automated decisions. Labeling images enables computer models to learn to recognize and distinguish objects, scenes, activities, or certain visual features. The image label may take various forms depending on what information is needed. Individual objects may be provided with specific labels, for example, in order to identify them in a scene. In addition, attributes such as colors, shapes or texture features can preferably also be annotated. In some cases, spatial information such as the position or extent of the objects in the image is also acquired. Labeling of images is often performed by human annotators who have visual understanding and domain knowledge. They analyze the image material and assign the corresponding labels. These annotations then serve as a basis for training machine learning models that learn, recognize visual patterns, and are capable of analyzing and classifying new, non-annotated images.A "general knowledge graph" (German: general knowledge graph or generic knowledge graph) is preferably a structured database that has knowledge and / or information about a broad range of domains or topics. The general know-wedge graph preferably has domain-specific information. It is preferably a semantic network that represents information in the form of graphs by the association of entities (e.g. persons, locations, events) with their attributes and relationships. By organizing knowledge in the form of a general-know-edge graph, it is preferably possible to model and understand complex relationships and / or dependencies between different entities. The graph allows questions to be asked, relationships to be analyzed, relationships to be analyzed and / or new knowledge to be derived. The general know-edge graph is preferably used by AI systems and search engines to provide a more comprehensive and contextual response to user requests. By associating and evaluating information from the knowledge graph, AI systems can develop a deeper understanding of texts, questions, and queries and generate more precise and relevant answers.A "partial graph" or even molecule of a general knowledge graph preferably relates to a section or subset of the entire knowledge graph. It is preferably a delimited area, which has specific entities, attributes and / or relationships, which are preferably connected to one another and / or represent a specific knowledge area and / or a specific domain.A "large language model", LLM, (German: large language model) refers to a powerful and / or extensive AI model trained to generate human-like texts, answer questions, conduct dialogs, and / or communicate in natural language. It is based on techniques of machine learning, in particular on the field of deep learning. A large language model preferably uses a deep neural network with a large number of parameters to achieve high complexity and / or flexibility in speech processing. These models are preferably trained on comprehensive text corpora derived from books, articles, web pages and other written sources. Training of a large language model is preferably performed by the application of self-supervised learning, in which the model learns from the text data, preferably without requiring specific annotations or human interventions. By exposure to a large amount of text, the model can learn complex language patterns, grammar rules, semantic meanings, and contextual dependencies. In the present case, all phrases generated on the basis of the sub-graph, together with their names, serve as input text corpus for learning word or phrase embeddings using such a large language model. As the large language model, for example, BERT or GPT may be used. A standalone LLM is also conceivable, in particular in the case of sensitive domain information.A "foundation model" (German base model) refers to a base model or set of techniques that serve as a starting point for the development of continuing models and / or applications. In terms of machine learning and artificial intelligence, a foundation model typically refers to a basic neural network and / or architecture suitable for a wide range of tasks or applications. It is preferably used as a starting point for training specialized models that are tailored to specific domains or tasks. A foundation model may preferably be trained with comprehensive training data and deep learning methods to develop a basic understanding of patterns, features, and relationships in the data. It preferably serves as a basis for learning general properties and / or features and can serve as a basis for extraction of features, classification of data, generation of texts or other tasks. In the present case, a fine tuning of the basic model is particularly preferably carried out with the previously calculated word / sentence embeddings which are independent of the image data. The base model is trained by the input of the image and / or video data and based on the monitoring of the word / sentence embeds.In a preferred embodiment, the foundation model or the base model converts the image information of the at least one image and / or video datum into the at least one image feature vector. The foundation model preferably comprises a deep neural network. An image feature vector, also known as a sentence feature vector or feature vector, preferably refers to a numerical representation of visual features or information extracted from an image. In the field of machine learning, the image feature vector is preferably used to transform visual information into a computer processable form.In a preferred embodiment, the at least one general know-wedge graph is provided from domain knowledge, which is described in particular by a domain expert, and / or from information extracted from the at least one image and / or video date.In a preferred embodiment, the at least one textual sentence comprises at least one natural language sentence. The textual sentence thus preferably comprises at least one statement and / or a semantic context / context which is / is comprised in a text formulated in a language comprehensible to humans.In a preferred embodiment, the large language model comprises a GPT-LLM and / or a BERT-LLM. In principle, other LLMs are also conceivable, so that the enumeration should not be understood as restrictive.In a preferred embodiment, the at least one image and / or video datum is captured by at least one optical sensor, in particular a camera and / or a lidar sensor and / or a radar sensor and / or an ultrasonic sensor. In principle, other sensors and / or sensor data are also conceivable, as long as they can be processed by the foundation model and / or converted into a data feature vector and / or written text by natural language.In a preferred embodiment, the at least one image and / or video data item is generated by data augmentation from existing image and / or video data. Data augmentation refers to a machine learning technique in which artificially new data points are generated by transforming and / or modifying existing data. The goal of data augmentation is to increase the amount and variety of available training data to improve the performance and robustness of machine learning models. In data augmentation, various transformations may be used depending on the type of data and the requirements of the model. For example, in the field of image processing, operations such as cropping, scaling, rotating, mirroring, or adding noise may be applied to generate new images that are slightly different from the original data.In a preferred embodiment, the extraction of the partial graph from the general know-edge graph as a function of the at least one image label comprises extraction by means of RDF molecule extraction. RDF molecule extraction preferably refers to the process of identifying and / or extracting RDF molecules from unstructured or semi-structured data sources, such as a general know-length graph. RDF preferably stands for resource description framework, a flexible data model for representing information in the form of triples (subject predicate object), which can be interpreted as statements or facts. In RDF molecule extraction, attempts are made to identify specific statements and / or facts from a text corpus and / or a data source and to convert them into RDF structures. This process can take place at different levels, from the extraction of individual triples to the identification of complex molecules consisting of a series of contiguous triples. In this case, RDF molecules are preferably extracted from the general know-length graph in the form of sub-graphs, which have information, for example road sign-specific information. A corresponding set is preferably generated for each RDF moleculeIn a preferred embodiment, training the foundation model on the basis of the comparison comprises minimizing a loss value resulting from the comparison of the set feature vector and the image feature vector as a function of a loss function assigned to the foundation model. Particularly preferably, the minimization of the loss value preferably takes place successively and / or iteratively. The foundation model is preferably trained to establish a relationship between sentence feature vectors and image feature vectors. The sentence feature vector here preferably represents the speech properties and / or information, while the image feature vector preferably describes the visual properties and / or information. The foundation model preferably learns how well the set feature vector and / or the image feature vector match and / or how well they are correlated. The training is preferably based on a loss function that quantitates the difference and / or the deviation between the set feature vector and the image feature vector. The objective is preferably to minimize this loss value to optimize the match or correlation between the two feature vectors. The preferred embodiment of the training process involves stepwise or iteratively minimizing the loss value. This means that the foundation model is trained in in particular successive iterations and / or steps, wherein the loss value should be further reduced in each step. This iterative approach allows the foundation model to continually improve and more effectively grasp the feature relationships.In a preferred embodiment, a production line comprising the device compilation for generating predefinable products is also provided. A production line is preferably a sequence of production stations and / or work areas arranged to cooperate to produce at least one product. This production line may include various equipment, machines and / or equipment configured to produce the predetermined products. In this embodiment, it is emphasized that in the preferred embodiment, a specific device assemblage is present in the production line. This device compilation could in particular comprise machines, robots, automated assembly lines, tools and / or other devices which are required for the production of the specified products.In a preferred embodiment, after the production line is provided, the method further comprises the step of: generating at least one predeterminable product using the device compilation. After the production line has been provided, according to this embodiment a method is carried out which aims at producing at least one predeterminable product. This step is carried out using the existing device compilation in the production line. It is not specified in any more detail here what type of product is involved or what the precise sequence of the production process looks like. The focus is that the method in the preferred embodiment aims at producing at least one predeterminable product in the production line with the aid of the provided device compilation.In the present case, according to a second aspect, a system for training a foundation model is specified. The system comprising: a providing device which is designed to provide at least one image and / or video date which has image information of at least one domain and at least one image identifier; and at least one general know-wedge graph, GKG, which has information about the at least one domain; and an evaluation and calculation device which is designed to extract a sub-graph from the general know-wedge graph as a function of the at least one image identifier, to generate at least one textual sentence as a function of the extracted sub-graph and as a function of the at least one image identifier; to convert the at least one textual sentence into at least one sentence feature vector by a large language model, LLM; comparing the at least one set feature vector with an image feature vector of the at least one image and / or video datum; and training the foundation model on the basis of the comparison; wherein the provision device is further configured to provide the trained foundation model.In the present case, according to a third aspect, a method for classifying and / or categorizing and / or segmenting image and / or video data is specified. The method comprising the steps of: providing a foundation model trained by the present method; providing at least one, in particular characterized, image and / or video datum; converting the at least one image and / or video datum into at least one image feature vector by the trained foundation model; and classifying and / or categorizing and / or segmenting the at least one image and / or video datum depending on the image feature vector by a trained classification and / or categorization and / or segmentation algorithm of machine learning, in particular by a trained Gaussian model and / or linear model.In the inference of the present trained foundation model, a classification and / or categorization and / or segmentation of domain-specific image and / or video data is preferably carried out. This can be done with the aid of a Gaussian process or a linear model. In this case, so-called Gaussians, in particular comprising at least one mean value and / or a covariance matrix, are preferably adapted for each class. The adaptation is preferably carried out on the basis of the random samples for the training data, in particular in the visual embedding space. Alternatively, a linear layer or multilayer perceptron (MLP for short) can also be trained on the trained embedding space of the basic model. An MLP consists of three main types of layers: input layer, hidden layers and output layer. The input layer receives the input data that is passed to the network. The hidden layers are intermediate layers that lie between the input layer and the output layer. The output layer provides the predictions or classification results of the network.The present method or the resulting trained foundation model can be used for the analysis of (image and / or video) data obtained from a sensor. In the present case, the term "image and / or video data" can also be replaced by "sensor data". The sensor may determine measurements of the environment in the form of sensor signals, which may be given by the following elements, for example, digital images, e.g., video, radar, LiDAR, ultrasound, motion, thermal images, audio signals, and / or specific data, such as, for example, 1D data (e.g., in production). In principle, it is also possible to obtain information about elements encoded by the sensor signal on the basis of a sensor signal. In other words, an indirect measurement may be performed based on a sensor signal used as a direct measurement. This is also understood as a virtual sensor system. Furthermore, the present method or the trained foundation model resulting therefrom can be used to classify and / or categorize and / or segment the sensor data in order in particular to detect the presence or absence of objects in the sensor data and / or to perform semantic segmentation of the sensor data, e.g. with respect to traffic signs and / or road surfaces and / or pedestrians and / or vehicles and / or others. The present method or the trained foundation model resulting therefrom can also be used for determining one or more continuous values, i.e. for carrying out a regression analysis, e.g. with respect to a distance and / or a speed and / or an acceleration and / or a tracking of an element, e.g. an object, in the data. The present method or the resulting trained foundation model can be used to detect anomalies in a technical system. For example, Gaussian deviations and / or other uncertainty values can be used to detect anomalies.For example, it is necessary to ensure that an automated vehicle does not collide with pedestrians. Specifically, based on the semantic segmentation, a computer calculates depth information of all pedestrians present in an image space, further calculates a trajectory around these pedestrians, and controls the autonomous driving vehicle to follow this trajectory so close as not to hit pedestrians. This applies in principle also to any mobile robot in order to avoid people who could be in their travel path and / or out of their movement path. For this purpose, the foundation model trained according to the invention can be effectively used.Furthermore, the foundation model trained according to the invention can also be used in combination with a regression algorithm in order to determine an exact spatial orientation of the vehicle, in particular using data from yaw rate and / or linear acceleration sensors of a vehicle.In the present case, a control device is particularly preferably also claimed, which is comprised in an autonomous vehicle and / or a robotic system and / or an industrial machine, and on which the present method according to the first aspect or the third aspect can be at least partially executed.The present active learning training method may also provide a foundation model that may learn in which operating point of an engine to check exhaust emission of the engine. For this purpose, the engine is preferably operated at this operating point, the exhaust emissions are measured and input into the actively learning foundation model as input data until the model is deemed good enough.In an automated vehicle, the algorithm described here, in particular the active learning algorithm or the foundation model, preferably defines predetermined scenarios for which image and / or video data and / or data from alternative sensors are to be collected. The image and / or video data captured by the corresponding at least one sensor of the vehicle are preferably analyzed by the trained foundation model and the scenario represented in the image and / or video data is classified (e.g. by recognition and / or classification of objects in the image and / or video data. If the scenario shown corresponds to a predetermined scenario, the corresponding image and / or video datum is preferably transmitted to a back-end computer, which in particular collects such image and / or video data from a plurality of vehicles and uses this image and / or video data for active (post) training of the foundation model or of a machine learning system, e.g. an image classifier, which is updated in the automated vehicle preferably continuously and / or cyclically and / or in intervals.In a networked physical system, e.g., a networked automated vehicle, an anomaly detector may also be used to detect whether a selected frame of predefined length (e.g., 5 s) from an accelerometer time series has an anomaly. If this is the case, this frame is transmitted to a back-end computer, where it can be used, for example, to define corner cases for checking the ML system, on the basis of which the connected physical system is operated.In the present case, a computer program with program code is also claimed in order to execute at least parts of the method according to the first aspect or the third aspect, in each case in one of its embodiments, when the computer program is executed on a computer. In other words, according to the invention, a computer program (product) comprising instructions which, when the program is executed by a computer, cause the computer to execute the method / steps of the method according to the invention in one of its embodiments.In the present case, a computer-readable data carrier with program code of a computer program is also proposed, in order to use at least parts of the method according to the first aspect or the third aspect, in each case in one of its embodiments, when the computer program is executed on a computer. In other words, the invention relates to a computer-readable (storage) medium comprising instructions which, when executed by a computer, cause the computer to carry out the method / steps of the method according to the invention in one of its embodiments.The described embodiments and developments can be combined with one another as desired.Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described above or below with respect to the exemplary embodiments, which combinations are not explicitly mentioned.Brief Description of the DrawingsThe accompanying drawings are intended to provide a further understanding of the embodiments of the invention. They illustrate embodiments and serve, in the context of the description, to explain principles and concepts of the invention.Other embodiments and many of the advantages mentioned are evident with reference to the drawings. The elements of the drawings shown are not necessarily shown true to scale with respect to one another.The following are shown: FIG. 1 shows a schematic flow diagram of an exemplary embodiment of the present method for training a foundation model; FIG. 2 shows a schematic block diagram of an exemplary embodiment of the present method for training a foundation model; and FIG. 3 shows a schematic block diagram of an exemplary embodiment of a method for classifying and / or categorizing and / or segmenting image and / or video data.In the figures of the drawings, identical reference numerals designate identical or functionally identical elements, components or components, unless indicated to the contrary.FIG. 1 shows a schematic flow diagram of a method for training a foundation model.In any given embodiment, the method can be carried out at least partially by a system 1, which can include a plurality of components, not shown in detail, for example one or a plurality of provision devices and / or at least one evaluation and computing device. It is understood that the provision device can be formed jointly with the evaluation and computing device or can be different therefrom. Furthermore, the system can comprise a storage device and / or an output device and / or a display device and / or an input device.According to the invention, the computer-implemented method comprises at least the following steps:In a step S 1, at least one image and / or video datum is provided, which has image information of at least one domain and at least one image identifier. The at least one image and / or video datum is detected, for example, by at least one optical sensor. Alternatively or additionally, the at least one image and / or video datum is generated by data augmentation from existing image and / or video data.In a step S 2, at least one general know-edge graph, GKG, is provided which has information about the at least one domain. The at least one general know-wedge graph is preferably provided from domain knowledge, which is described in particular by a domain expert, and / or from information extracted from the at least one image and / or video date.In a step S 3, a partial graph is extracted from the general know-wedge graph as a function of the at least one image identifier. The extraction is preferably carried out by extraction by means of RDF molecule extraction.In a step S 4, at least one textual sentence is generated as a function of the extracted sub-graph and as a function of the at least one image identifier.In a step S 5, the at least one textual sentence is converted into at least one sentence feature vector by a Large Language Model, LLM. The large language model has, for example, a GPT-LLM and / or a BERT-LLM.In a step S 6, the at least one set feature vector is compared with an image feature vector of the at least one image and / or video datumIn a step S 7, the foundation model is trained on the basis of the comparison. Training S 7 is preferably carried out by minimizing a loss value resulting from the comparison of the set feature vector and the image feature vector as a function of a loss function assigned to the foundation model. The loss value is preferably minimized successively and / or iteratively.In a step S 8, the trained foundation model is provided. Providing the trained foundation model preferably refers to a process in which the fully trained model is provided for use in a productive environment and / or application.FIG. 2 shows a schematic block diagram of an exemplary embodiment of the present method for training a foundation model. For training a foundation model 200, which is provided in particular in its raw architecture, at least one image and / or video datum 202 is provided in the present case, which has image information of at least one domain and at least one image identifier 204. Preferably, a plurality of image and / or video data are provided. Furthermore, for a specific domain in which the image and / or video evaluation is to take place, for example when recognizing road traffic signs, a general knowledge graph 206 is provided, which has detailed and / or professional information about the at least one domain. A partial graph 210 is extracted from the general know-wedge graph 206, in particular by means of an extraction model 208 as a function of the at least one image identifier 204. The extraction model 208 may in particular comprise an RDF molecule extraction model. By means of a sentence generation model 212, at least one textual sentence is preferably generated as a function of the extracted sub-graph 210 and as a function of the at least one image identifier 204. The sentence generation model 212 may match a provided large language model 214. An exemplary set is indicated by reference numeral 213. A CONVERSION of the at least one textual sentence into at least one sentence feature vector 216 is performed by the large language model 214. The foundation model 200 preferably converts the image information of the at least one image and / or video datum 202 into the at least one image feature vector 218. The at least one set feature vector 216 is compared to the image feature vector 218, in particular in an n-dimensional vector space 220. This takes place in the course of training the foundation model 200 on the basis of the comparison.The foundation model 200 is trained by minimizing a loss value 222 resulting from the comparison of the sentence feature vector 216 and the image feature vector 218 as a function of a loss function 224 assigned to the foundation model 200. The minimization of the loss value preferably takes place successively and / or iteratively, which is indicated schematically by the arrows in FIG. 2.After the loss value is minimized to a predetermined level and / or a threshold value is reached, the trained foundation model can be provided. The trained foundation model is identified by the reference numeral 300.FIG. 3 shows a method for classifying and / or categorizing and / or segmenting image and / or video data 302. FIG. 3 thus shows the trained foundation model 300 in the inference. The trained foundation model 300 converts the at least one image and / or video data 302 into at least one image feature vector (not shown in FIG. 3 ). Based thereon, the at least one image and / or video datum 302 is classified and / or categorized and / or segmented as a function of the image feature vector by a trained classification and / or categorization and / or segmentation algorithm 304 of machine learning. This can be, for example, a pre-trained Gaussian model that is trained to assign a sense content of the image feature vector to a specific class and / or category and / or a specific segment.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Cited Non-Patent LiteratureLearning Visual Models using a Knowledge Graph as a Trainer."; https: / / arxiv.org / pdf / 2103.00020.pdf

[0006] Radford, A., Kim, J.W., Hallacy, C., Ramsh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Kruger, G., & Sutskever, I. (n.d.). Learning Transferable Visual Models From Natural Language Supervision

[0007] Santurkar, S., Dubois, Y., Taori, R., Liang, P., & Hashimoto, T. (2022). Is a Capture Wordh a Thick Images? A Controlled Study for Representation Learning. http: / / arxiv.org / abs / 2207.07635

[0008]

Claims

Method for training a foundation model (200), the method comprising the steps of: - providing (S1) at least one image and / or video date (202) comprising image information of at least one domain and at least one image identifier (204); - providing (S2) at least one general know-wedge graph (206), GKG, comprising information about the at least one domain; - extracting (S3) a partial graph (210) from the general know-wedge graph (206) as a function of the at least one image identifier (204); - generating (S4) at least one textual set (213) as a function of the extracted partial graph (210) and as a function of the at least one image identifier (204); converting (S5) the at least one textual sentence (213) into at least one sentence feature vector (216) by a Large Language Model, LLM (214); comparing (S6) the at least one sentence feature vector (216) with an image feature vector (218) of the at least one image and / or video date (202); training (S7) the foundation model (200) on the basis of the comparison; and providing (S8) the trained foundation model (200, 300).The method of claim 1, wherein the foundation model (200) converts the image information of the at least one image and / or video data (202) to the at least one image feature vector (218).The method according to claim 1 or 2, wherein the at least one general know-wedge graph (206) is provided from domain knowledge, which is described in particular by a domain expert, and / or from information extracted from the at least one image and / or video date (202).The method of any preceding claim, wherein the at least one textual sentence (213) comprises at least one natural language sentence.The method of any preceding claim, wherein the large language model (214) comprises a GPT LLM and / or a BERT LLM.Method according to one of the preceding claims, wherein the at least one image and / or video datum (202) is detected by at least one optical sensor.Method according to one of the preceding claims, wherein the at least one image and / or video data item (202) is generated by data augmentation from existing image and / or video data.Method according to one of the preceding claims, wherein the extraction (S3) of the sub-graph (210) from the general knowledge graph (206) as a function of the at least one image label (204) comprises extraction by means of RDF molecule extraction (208).Method according to one of the preceding claims, wherein the training (S7) of the foundation model (200) on the basis of the comparison comprises minimizing a loss value (222) resulting from the comparison of the sentence feature vector (216) and the image feature vector (218) as a function of a loss function (224) assigned to the foundation model (200), and wherein the minimizing of the loss value (222) is preferably carried out successively and / or iteratively.Method according to one of the preceding claims, wherein a production line comprising the device compilation for producing predeterminable products is also provided.Method according to Claim 10, wherein the method, after the provision of the production line, further comprises the step: generating at least one predeterminable product using the device compilation.A system (1) for training a foundation model (200), the system (1) comprising: - a providing device configured to provide at least one image and / or video date (202) comprising image information of at least one domain and at least one image identifier (204); and at least one general know-edge graph, GKG, (206) of the information about the at least one domain; and an evaluation and computing device which is designed to extract a partial graph (210) from the general know-wedge graph (206) as a function of the at least one image label (204), to generate at least one textual sentence (213) as a function of the extracted partial graph (210) and as a function of the at least one image label (204); to convert the at least one textual sentence (213) into at least one sentence feature vector (216) by a large language model, LLM, (214); to compare the at least one sentence feature vector (216) with an image feature vector (218) of the at least one image and / or video date (204); and train the foundation model (200) on the basis of the comparison; wherein the provision device is further configured to provide the trained foundation model (200, 300).Method for classifying and / or categorizing and / or segmenting image and / or video data, the method comprising the steps of: - providing a foundation model (300) trained by the method according to one of Claims 1 to 11; - providing at least one, in particular characterized, image and / or video datum (302); - converting the at least one image and / or video datum (302) into at least one image feature vector by the trained foundation model (300); and classifying and / or categorizing and / or segmenting the at least one image and / or video datum (302) as a function of the image feature vector by a trained classification and / or categorization and / or segmentation algorithm of machine learning (304).A computer program comprising program code for executing at least parts of a method according to any one of claims 1 to 11 or claim 13 when said computer program is executed on a computer.Computer-readable data medium with program code of a computer program for executing at least parts of a method according to one of Claims 1 to 11 or Claim 13, when the computer program is executed on a computer.

Images