Method and system for training a base model

Abstract

The invention relates to a method for training a basic model and / or a graph-based neural network, comprising the following steps: providing at least one image data and / or video data having image information from at least one domain and at least one image identification; providing at least one common sense graph GKG having information about at least one domain; providing at least one textual description of image information of the at least one image data and / or video data; embedding the at least one text description into a graph-based neural network GNN by means of a large language model LLM; embedding the common sense graph into a graph-based neural network; generating an image-text feature vector according to the at least one text description and the common sense graph through a graph-based neural network; generating an image feature vector through the basic model; training a basic model based on the image-text feature vector; alternatively, a graph-based neural network is trained based on the embedded text description, the embedded common sense graph, and according to the image feature vector.

Description

Technical Field

[0001] The invention relates to a method and a system for training a basic model. The invention also relates to a method for grading and / or classifying and / or segmenting image data and / or video data, as well as a computer program and a computer-readable data carrier with program code. Background Art

[0002] Advances in the field of machine learning have led to diverse developments in artificial intelligence, particularly in the field of visual perception. One aspect of this development is the ability of neural networks to generate visual embeddings. Visual embeddings are compact representations of visual inputs that enable computers to understand and act upon visual information.

[0003] In this context, the base model or basic model has been shown to be an effective method for generating visual embeddings based on language embeddings. The base model uses a deep neural network (DNN) that is trained to detect the semantics of text descriptions and generate corresponding visual representations.

[0004] The use of DNNs enables the underlying model to learn complex relationships between language and visual information and to convert high-dimensional visual data into low-dimensional representations. This makes it possible to process visual information more efficiently and use it in other applications, such as image recognition, video analysis, or even in the development of autonomously reactive robots.

[0005] Training the base model typically requires a large amount of annotated data, where visual inputs are associated with corresponding textual descriptions. By learning from this data, the model can develop a general ability to generate visual embeddings that can be applied to a wide variety of tasks. Known training methods still have potential for improvement.

[0006] From the scientific publication “Learning Visual Models using a Knowledge Graph as a Trainer”; https: / / arxiv.org / pdf / 2103.00020.pdf, a method for training deep neural networks with the help of semantic knowledge graphs / ontologies is known.

[0007] A general method for learning a base model (German: Basismodell) from text and images is known from the scientific publication "Learning Transferable Visual Models From Natural Language Supervision" by Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askel, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (nd). Here, the base model is trained with the aid of a deep neural network (DNN) to generate a visual embedding based on the language embedding given by the text description. However, in particular in the case of a small amount of data with respect to image data and / or video data, it is challenging to create a balanced and / or intuitive image description.

[0008] From the scientific publication “Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning” by Santurkar, S., Dubois, Y., Taori, R., Liang, P., and Hashimoto, T. (2022), http: / / arxiv.org / abs / 2207.07635 The importance of the inscription level of image data and / or video data, especially in the case of small data volumes, is known from .

[0009] Schemes such as Radford et al. learn a robust base model by using image signatures for description. The captions are then converted into latent vectors, i.e. word embeddings, with the help of a language model. The embeddings are then used to control the learning process of the base model. Here, both the text corpus and the images are crawled from the web.

[0010] In low-data scenarios, such as identifying street nameplates, detailed inscriptions for each nameplate do not exist in publicly available text corpora. Furthermore, traditional fine-tuning methods using standard cross-entropy lose most of the generalization power of the base model.

[0011] A lot of manual work is also done when fine-tuning with the help of manually created inscriptions, without the relevant underlying solution. This can be difficult and error-prone, especially in the case of very specific domains.

[0012] Furthermore, deep neural networks (DNNs) or base models usually form visual embeddings based only on the data distribution. It is known that when using DNNs in the real world, the data distribution always changes. In the base model, the context is provided by an unstructured description of each image. It is seen that the context provided by the description helps the base model generalize to multiple domains. Of course, this context in the description is only partially available on the Internet.

[0013] The invention is therefore based on the object of proposing an improved method for training a base model.Another object is to propose an improved trained base model.

[0014] This object is achieved by a method for training a base model according to the features of patent claim 1. This object is achieved by a system for training a base model according to the features of patent claim 12. This object is achieved by a method for grading and / or classifying and / or segmenting image data and / or video data according to the features of patent claim 13. Summary of the invention

[0015] At present, according to a first aspect, a method for training a basic model, in particular a deep neural network DNN and / or a graph-based neural network is proposed. The method has the following steps:

[0016] - providing at least one image data and / or video data, said image data and / or video data having image information from at least one domain and at least one image identifier;

[0017] - providing at least one common sense graph GKG, said common sense graph having information about at least one domain;

[0018] - providing at least one textual description of image information of at least one image data and / or video data;

[0019] - embedding at least one text description into a graph-based neural network GNN with the help of a large language model LLM;

[0020] -Embed common sense graphs into graph-based neural networks;

[0021] - generating a graph-text feature vector based on at least one text description and information content, in particular from a common sense graph, by means of a graph-based neural network;

[0022] - generating an image feature vector by means of a base model, in particular a base model in the form of a visual embedding in an embedding space;

[0023] - Train the basic model based on the image-text feature vector; or,

[0024] - training a graph-based neural network based on the information content of the embedded text description, in particular the embedded common sense graph, and according to the image feature vector; and training a base model based on the image feature vector and in particular the image information content of at least one image data and / or video data; and

[0025] -Provide trained base models and / or trained graph-based neural networks.

[0026] It is easy to understand that the steps according to the present invention and other optional steps do not necessarily have to be performed in the order shown, but can also be performed in a different order. In addition, other intermediate steps can be provided. Each step can also include one or more sub-steps without thereby departing from the scope of the method according to the present invention.

[0027] Currently, it is possible to generate highly descriptive image descriptions and / or video descriptions for domain-specific image data and / or video data by using common sense graphs that are particularly domain-specific and preferably follow the underlying ontology. Currently, it is possible to train the basic model in a more efficient manner by describing the domain convincingly and / or extensively and / or comprehensively by means of common sense graphs. The basic model can be better adapted to new and / or changing domains and / or domain situations, wherein already existing training data in the form of image data and / or video data can also be used. The model performance of the basic model trained in this way can thus be improved relative to the prior art.

[0028] Currently, a method for training a basic model is described with the aid of a knowledge graph / ontology, in particular semantic, which can be described in particular by a deep neural network. In particular, a comparison method between embedding in a knowledge graph and embedding in a neural network is described here. In particular, a fine-tuning of the basic model with domain-specific content is investigated here, wherein level-specific information is extracted from the knowledge graph, in particular in the form of RDF molecules. Particularly preferably, a sentence is generated for each molecule. The sentence is then used as input for a large language model (Large Language Model, LLM for short).

[0029] Currently, the visual base model is refined to adapt it to the new domain and / or new domain parameters and / or domain information.

[0030] The present method also involves adapting, in particular, the polynomial reduction algorithm to fully utilize the word length shift, which is particularly adapted and / or adaptable to the word length of the predetermined computer hardware. Based on this, the technical effect of efficient hardware implementation of the algorithm or basic model can be achieved.

[0031] The base model preferably learns image embeddings by adapting them to the embeddings of unstructured text descriptions. Instead of collecting images from the Internet, graph-based metadata is currently preferably used as a supplement to the image data and / or video data. For example, the metadata can be obtained from the fields of autonomous driving and / or production and / or the Internet of Things. Currently, the domain-specific metadata is converted into a structured format of a common sense graph. In principle, it is possible to supplement the missing information by suitable reasoning methods. Such structured metadata is then preferably combined with a text embedding or at least one text description so that it can be used as context for the image data and / or video data. With the help of the scheme, it is feasible to train a more robust and / or controlled underlying model or base model, especially for domains with graph-based context.

[0032] Here, graph-based metadata is typically collected on a large scale, such as in automobiles via CAN buses, at assembly lines in production facilities, and / or in IoT devices. Since the metadata is typically graph-based, additional graph-based information usage can improve the training of the base model relative to the prior art.

[0033] In particular, the inclusion of prior knowledge in the form of GKGs helps to train base models that can be better generalized to real-world applications. Here, GKGs add valuable domain knowledge about heterogeneous metadata combinations and the possibility to interact with the data. Based on the rich context, a base model or a base model can thus be trained in order to be used for image comparison.

[0034] "Image identification", also known as labeling, refers to the process of assigning at least one category and / or label to visual data (such as images and / or videos). Labels or tags are preferably descriptions or labels that identify and characterize specific features, objects or levels in an image. The goals of image identification are preferably to: understand and classify visual content in order to train machine systems; interpret visual information and make automated decisions. By labeling images, computer models can learn to: recognize and distinguish objects, scenes, activities or specific visual features. Image identification can take different forms depending on what information is needed. For example, specific labels can be provided for individual objects to identify the objects in the scene. In addition, attributes such as color, shape or tissue characteristics can also be preferably annotated. In some cases, spatial information, such as the location or range of objects in the image, is also detected. Image labeling is usually performed by human annotators with visual understanding and domain knowledge. They analyze the image material and assign corresponding labels. The annotations are then used as a basis for training machine learning models, which learn to recognize visual patterns and are able to analyze new, unannotated images and perform classification.

[0035] A "common sense graph" (German: allgemeiner Wissensgraph oder generischer Wissensgraph common sense graph or general knowledge graph) is preferably a structured database with knowledge and / or information about a broad category of domains or topics. The common sense graph preferably has domain-specific information. It is preferably a semantic network that represents information in the form of a graph by linking entities (such as people, places, events) with their attributes and relationships. By organizing knowledge in the form of a common sense graph, it is preferably possible to: model and understand complex associations and / or correlations between different entities. The graph enables: asking questions, exploring relationships, analyzing associations and / or deriving new knowledge. The common sense graph is preferably used by artificial intelligence systems and search engines to provide more comprehensive and contextually relevant responses to user queries. By linking and evaluating information in the knowledge graph, artificial intelligence systems can develop a deeper understanding of text, questions and queries, and generate more precise and relevant responses.

[0036] "Large Language Model", LLM (German: groβes Sprachmodell) refers to a powerful and / or large-scale artificial intelligence model that is trained to generate human-like text, answer questions, guide conversations and / or communicate in natural language. The artificial intelligence model is based on machine learning techniques, in particular the field of deep learning. Large language models preferably use deep neural networks with a large number of parameters to achieve high complexity and / or flexibility in language processing. The model is preferably trained on a large text corpus derived from books, articles, web pages and other written sources. Training a large language model is preferably carried out using self-supervised learning, where the model learns from text data, preferably without specific annotations or manual intervention. By being exposed to a large amount of text, the model can learn complex language patterns, grammatical rules, semantics and contextual dependencies. Currently, all sentences generated based on partial graphs together with their tags are used as input text corpora for learning word or sentence embedding using such a large language model. For example, BERT or GPT can be used as a large language model. Independent LLMs can also be envisioned, especially in cases where domain information is sensitive.

[0037] "Base model" (German Basismodell) refers to a basic model or basic set of techniques used as a starting point for developing further guided models and / or applications. With regard to machine learning and artificial intelligence, a base model generally refers to an underlying neural network and / or architecture that is suitable for a wide variety of tasks or applications. The model is preferably used as a starting point for training more specific models, which are customized for specific domains or tasks. Preferably, the base model can be trained with the aid of a wide range of training data and deep learning methods to develop a basic understanding of patterns, features, and associations in the data. The base model is preferably used as a basis for learning general characteristics and / or features, and can be used as a basis for feature extraction, data classification, text generation or other tasks. Currently, it is particularly preferred to fine-tune the base model with the aid of previously calculated word / sentence embeddings that are independent of image data. The base model is trained by inputting image data and / or video data and based on monitoring of word / sentence embeddings. The base model preferably includes a deep neural network with a transformer model structure.

[0038] A "graph neural network GNN" is a machine learning model that works on a graph structure. The graph neural network is used to analyze patterns and relationships in data organized in graph form. GNN has applications in fields such as social network analysis, recommendation systems, and computer graphics.

[0039] At present, therefore, a general knowledge graph (GKG) is preferably provided and / or created and / or used based on metadata (automatically) collected for each image data and / or video data. In addition, text embedding of the text description is preferably performed via a large language model (LLM). The embedding is preferably performed together with the information content of the GKG embedded via a GNN. The base model or deep neural network (DNN) is preferably trained by adapting the visual embedding of the base model or deep neural network to the image-text feature vector by means of a contrastive loss value. Preferably, the DNN is trained by means of the calculated image-text feature vector, which is in particular independent of the image data and / or video data. Alternatively, the DNN and the GNN are trained together. Here, the image-text feature vector is preferably trained by means of the input / information content of the GKG and the text description and monitoring of the image embedding (visual embedding) of the DNN. Preferably, the DNN is trained by means of the input of image data and / or video data and by monitoring of the image-text feature vector.

[0040] In a preferred embodiment, at least one common sense graph is provided based on metadata about the domain, wherein additional information is preferably supplemented by means of reasoning. Additional information can be supplemented by means of reasoning (conclusions). This means that logical reasoning and deduction methods are used to derive or generate additional information from existing data in the common sense graph. Thus, the common sense graph should be made more comprehensive and more effective.

[0041] In a preferred embodiment, at least one general knowledge graph is provided from domain knowledge, in particular described by a domain expert, and / or from information extracted from at least one image data and / or video data.

[0042] In a preferred embodiment, at least one text description has at least one natural language sentence generated based on the information content of the common sense graph. Therefore, the text description preferably includes at least one statement and / or semantic context, and the statement / semantic context is included in the text expressed in a language understandable to humans.

[0043] In a preferred embodiment, the large language model has a GPT-LLM and / or a BERT-LLM. In principle, other LLs are also conceivable, so that this list should not be understood as limiting.

[0044] In a preferred embodiment, at least one image data and / or video data is detected by at least one optical sensor, in particular a camera and / or a lidar sensor and / or a radar sensor and / or an ultrasonic sensor. In principle, other sensors and / or sensor data are also conceivable, as long as they can be processed by the basic model and / or converted into data feature vectors and / or can be described in text by natural language.

[0045] In a preferred embodiment, at least one image data and / or video data is generated from existing image data and / or video data by data augmentation. Data augmentation refers to a technique in machine learning in which new data points are artificially generated by transforming and / or modifying existing data. The goal of data augmentation is to increase the range and variety of available training data to improve the performance and robustness of machine learning models. Depending on the data type and model requirements, different transformations can be applied during data augmentation. In the scope of data processing, operations such as cropping, scaling, rotating, mirroring or adding noise can be used in particular to generate new images that are slightly different from the original data.

[0046] In a preferred embodiment, at least one text description of image information of at least one image data and / or video data is provided with the extraction of a partial graph from a common sense graph according to at least one image identifier by means of RDF molecule extraction. RDF molecule extraction preferably refers to the process of identifying and / or extracting RDF molecules from an unstructured or semi-structured data source (e.g., a common sense graph). RDF preferably stands for Resource Description Framework, i.e., a flexible data model for representing information in the form of triples (subject-predicate-object), which can be interpreted as statements or facts. During RDF molecule extraction, an attempt is made to: identify specific statements and / or facts from a text corpus and / or data source and convert them into an RDF structure. The process can occur at different levels, from extracting a single triple to identifying a complex molecule consisting of a series of connected triples. Here, RDF molecules are preferably extracted from the common sense graph in the form of a partial graph, which has, for example, street nameplate-specific information. Preferably, a corresponding statement is generated for each RDF molecule. A "partial graph" or molecule of a common sense graph preferably refers to a part or subset of the entire knowledge graph. Preferably, it is a delimited area with specific entities, attributes and / or relationships, which are preferably interconnected and / or represent specific knowledge areas and / or specific domains. At present, relevant knowledge about a domain in the form of at least one partial graph, in particular so-called RDF molecules, can be extracted from a particularly general common sense graph (English: General-Knowledge-Graphs, GKG) by the framework of this method. Preferably, a large language model (English: Large-Language-Model, LLM) is used to convert the "molecules" into natural language sentences and universal latent vector representations (language embeddings). Subsequently, a basic model preferably represented by a deep neural network is trained to adapt the visual embedding from at least one image data and / or video data to the language embedding in the form of natural language sentences generated from the common sense graph.

[0047] In a preferred embodiment, the basic model and / or the graph-based neural network is trained based on the loss value that can be calculated from the text feature vector and the image feature vector, in particular by continuously and / or iteratively minimizing the loss value. The basic model is preferably trained to establish the relationship between the sentence feature vector and the image feature vector. Here, the sentence feature vector preferably represents language characteristics and / or information, and the image feature vector preferably describes visual characteristics and / or information. The basic model preferably learns: how consistent the sentence feature vector and / or image feature vector are and / or how related they are. The training is preferably based on a loss function that quantifies the difference and / or deviation between the sentence feature vector and the image feature vector. The goal is preferably to minimize the loss value in order to optimize the consistency or correlation between the two feature vectors. The preferred embodiment of the training process includes minimizing the loss value step by step or iteratively. This means: the basic model is trained in particularly successive iterations and / or steps, wherein the loss value should be further reduced in each step. This iterative scheme is implemented for the basic model: continuously improving and more effectively detecting feature relationships. This also applies to graph-based neural networks.

[0048] In a preferred embodiment, a production line comprising an equipment grouping for producing preset products is also provided. The production line is preferably a sequence of production stations and / or work areas, which are arranged so that they work together to manufacture at least one product. The production line may include various equipment, machines and / or facilities configured to produce preset products. In the embodiment, it is emphasized that in a preferred embodiment, there is a specific equipment grouping in the production line. The equipment grouping may include in particular machines, robots, automated assembly lines, tools and / or other equipment required to produce the preset products.

[0049] In a preferred embodiment, after providing the production line, the method further comprises the step of producing at least one presettable product using the equipment grouping. After providing the production line, according to the embodiment, the method for manufacturing at least one presettable product is performed. The steps are performed using the existing equipment grouping in the production line. It is not currently specified what type of product is or what the exact flow of the manufacturing process is. The focus is on the following: the method in the preferred embodiment is intended to produce at least one presettable product with the aid of the equipment grouping provided in the production line.

[0050] At present, according to the second aspect, a system for training a basic model is proposed. The system has: a providing device, the providing device is configured to: provide at least one image data and / or video data, the image data and / or video data having image information from at least one domain and at least one image identifier; provide at least one common sense graph GKG having information about at least one domain, and provide at least one text description of the image information of at least one image data and / or video data; and an evaluation and calculation device, the evaluation and calculation device is configured to: embed at least one text description into a graph-based neural network GNN with the help of a large language model LLM; embed the common sense graph into a graph-based neural network; generate a graph-text feature vector according to at least one text description and the common sense graph by a graph-based neural network; generate an image feature vector by a basic model, which is particularly in the form of a visual embedding in an embedding space; train a basic model based on the graph-text feature vector; or train a graph-based neural network based on the embedded text description, the embedded common sense graph and according to the image feature vector; and train the basic model based on the graph-text feature vector and at least one image data and / or video data; wherein the providing device is also configured to: provide a trained basic model and / or a trained graph-based neural network.

[0051] According to a third aspect, a method for grading and / or classifying and / or segmenting image data and / or video data is currently proposed. The method comprises the following steps: providing a basic model trained by the current method; providing at least one, in particular identified, image data and / or video data; converting at least one image data and / or video data into at least one image feature vector by means of the trained basic model; and grading and / or classifying and / or segmenting at least one image data and / or video data according to the image feature vector by means of a trained grading algorithm and / or classification algorithm and / or segmentation algorithm of machine learning, in particular by means of a trained Gaussian model and / or linear model.

[0052] In the inference of the currently trained basic model, a classification and / or classification and / or segmentation of domain-specific image data and / or video data is preferably performed. This can be done with the aid of a Gaussian process or a linear model. Here, preferably for each level, a so-called Gaussian program (Gaussianer) is adapted, which in particular includes at least one mean value and / or covariance matrix. Preferably, the adaptation is performed based on a sample of the training data, in particular in the visual embedding space. Alternatively, linear layers or multilayer perceptrons (MLP for short) can also be trained on the trained embedding space of the basic model. MLP consists of three main types of layers: input layer, hidden layer and output layer. The input layer receives the input data passed to the network. The hidden layer is an intermediate layer located between the input layer and the output layer. The output layer provides the prediction or classification result of the network.

[0053] The method or the trained basic model derived therefrom can be used to analyze (image and / or video) data obtained by the sensor. Currently, the term "image data and / or video data" can also be replaced by "sensor data". The sensor can obtain environmental measurements in the form of sensor signals, which can be given, for example, by the following elements: digital images, such as video, radar, lidar, ultrasound, motion, thermal images, audio signals and / or specific data, such as one-dimensional data (for example in production). In principle, it is also feasible to obtain information about elements encoded by the sensor signal based on the sensor signal. In other words, indirect measurements can be performed based on sensor signals used as direct measurements. This is also understood as a virtual sensor device. In addition, the method or the trained basic model generated therefrom can be used to classify and / or classify and / or segment the sensor data in order to specifically identify the presence or absence of objects in the sensor data, and / or perform semantic segmentation of sensor data, for example in terms of traffic signs and / or road surfaces and / or pedestrians and / or vehicles and / or other aspects. The method or the trained base model generated thereby can also be used to determine a continuous value or multiple continuous values, i.e., to perform regression analysis in data, for example, regarding distance and / or speed and / or acceleration and / or tracking elements, such as objects. The method or the trained base model generated thereby can be used to: identify anomalies in engineering systems. For example, Gaussian deviations and / or other uncertainty values ​​can be used to identify anomalies.

[0054] For example, it must be ensured that the automated vehicle does not collide with pedestrians. Based on the semantic segmentation, the computer calculates in particular the depth information of all pedestrians present in the image space, also calculates the trajectories around these pedestrians, and controls the automated vehicle so that it follows the trajectories closely, so as not to collide with any pedestrians. In principle, this also applies to every mobile robot, so as to avoid people who may be in its path and / or enter its path of movement. For this purpose, the basic model trained according to the invention can be effectively used.

[0055] Furthermore, the basic model trained according to the invention can also be used in conjunction with a regression algorithm in order to determine the exact spatial orientation of the vehicle, in particular using data from a yaw rate sensor and / or a linear acceleration sensor of the vehicle.

[0056] At present, particularly preferably, protection is also claimed for a control device which is included in an autonomous vehicle and / or a robotic system and / or an industrial machine and on which the method according to the first aspect or the third aspect can be at least partially performed.

[0057] The current active learning training method can also provide a basic model that can learn to determine at which operating point of the engine the exhaust emissions of the engine are to be checked. To this end, the engine is preferably operated at the operating point, the exhaust emissions are measured and entered as input data into the active learning basic model until the model is deemed to be sufficiently good.

[0058] In an automated vehicle, the algorithm or base model described here, in particular for active learning, preferably determines a predetermined scene for which image data and / or video data and / or data from a substitute sensor should be collected. Preferably, the image data and / or video data detected by at least one corresponding sensor of the vehicle are analyzed by the trained base model and the scene represented in the image data and / or video data is classified (e.g. by identifying and / or classifying objects in the image data and / or video data). If the represented scene corresponds to the predetermined scene, the corresponding image data and / or video data are preferably transmitted to a backend computer, which in particular collects such image data and / or video data from a plurality of vehicles and uses them for actively (re)training a base model or a machine learning system, such as an image classifier, which is preferably updated continuously and / or cyclically and / or regularly in the automated vehicle.

[0059] In networked physical systems, such as networked automated vehicles, anomaly detectors can also be used to identify whether selected frames of a predefined length (e.g., 5 seconds) in the acceleration sensor time series have anomalies. If this is the case, the frames are transmitted to a backend computer, where they can be used, for example, to define extreme cases for testing a machine learning system, and the connected physical system is run based on the results.

[0060] At present, protection is also claimed for a computer program with a program code so that when the computer program is executed on a computer, at least part of the method according to the first aspect or the third aspect, respectively, in its implementation mode thereof, is performed. In other words, according to the present invention, a computer program (product) is provided, which includes instructions, and when the program is executed by a computer, the instructions cause the computer to perform the method / steps according to the method according to the present invention and its implementation mode thereof.

[0061] At present, a computer-readable data carrier with a program code of a computer program is also proposed, so that when the computer program is executed on a computer, at least part of the method according to the first aspect or the third aspect and its implementation method are executed. In other words, the present invention relates to a computer-readable (storage) medium, which includes instructions, which, when executed by a computer, cause the computer to perform the method / steps according to the method according to the present invention and its implementation method.

[0062] The described embodiments and developments can be combined with one another as desired.

[0063] Further possible embodiments, refinements and embodiments of the invention also include combinations of inventive features described above or below with reference to the exemplary embodiments that are not explicitly mentioned. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are intended to facilitate a further understanding of the embodiments of the present invention. The drawings illustrate the embodiments and, together with the description, serve to explain the principles and concepts of the present invention.

[0065] Further embodiments and many of the advantages mentioned are apparent from the drawings. The elements shown in the drawings are not necessarily shown to scale with respect to one another.

[0066] in:

[0067] Figure 1 A schematic flow chart showing one embodiment of the present method for training a base model; and

[0068] Figure 2 A schematic block diagram showing an embodiment of the method for training a base model.

[0069] In the drawings, unless otherwise indicated, the same reference signs denote identical or functionally identical elements, components or parts. DETAILED DESCRIPTION

[0070] Figure 1 A schematic flow chart of a method for training a base model is shown.

[0071] In any embodiment, the method can be at least partially performed by a system 1, which can include a plurality of components not shown in detail, such as one or more providing devices and / or at least one evaluating and calculating device. It goes without saying that the providing device can be constructed together with the evaluating and calculating device or can be constructed differently therefrom. In addition, the system can include a storage device and / or an output device and / or a display device and / or an input device.

[0072] According to the present invention, the computer-implemented method comprises at least the following steps:

[0073] In step S1, at least one image data and / or video data is provided, wherein the image data and / or video data has image information of at least one field and at least one image identifier. For example, the at least one image data and / or video data is detected by at least one optical sensor. Alternatively or additionally, the at least one image data and / or video data is generated from existing image data and / or video data by data enhancement.

[0074] In step S2, at least one common sense graph GKG is provided, which has information about at least one domain. Preferably, at least one common sense graph is provided from domain knowledge (the domain knowledge is particularly described by domain experts) and / or from information extracted from at least one image data and / or video data. Preferably, at least one common sense graph is provided based on metadata about the domain, wherein further information is preferably supplemented by means of reasoning.

[0075] In step S3, at least one text description of image information of at least one image data and / or video data is provided.

[0076] In step S4, at least one text description is embedded into a graph-based neural network GNN with the aid of a large language model LLM.

[0077] In step S5, the common sense graph is embedded into the graph-based neural network.

[0078] In step S6, a graph-text feature vector is generated according to at least one text description and a common sense graph through a graph-based neural network.

[0079] In step S7 , an image feature vector is generated by the base model, in particular, the image feature vector is in the form of a visual embedding in an embedding space.

[0080] In step S8A, a basic model is trained based on the image-text feature vector.

[0081] Alternatively, a graph-based neural network may be trained based on the embedded text description, the embedded common sense graph and according to the image feature vector; and a basic model may be trained based on the image-text feature vector and at least one image data and / or video data.

[0082] Figure 2A schematic block diagram of an embodiment of the method for training a basic model 200 and / or a graph-based neural network 202 is shown. In this case, at least one image data and / or video data 204 is provided, which has image information of at least one domain and at least one image identifier 206. Preferably, the at least one image data and / or video data 204 can be supplemented with further image data and / or video data by means of data augmentation 208. At least one common sense graph 210 is provided, which has information about at least one domain. The common sense graph 210 is particularly preferably based on meta-information collected automatically about the domain. Based on the common sense graph 210, at least one text description 212 of the image information of at least one image data and / or video data 204 is preferably provided. It is also possible to generate a further description from the at least one text description 212 via data or text augmentation. The at least one text description 212 is embedded in the graph-based neural network 202 by means of a large language model 214. The large language model 214 can be, for example, GPT-3. The information content of the common sense graph 210 is embedded in the graph-based neural network 202. The graph-based neural network 202 generates a graph-text feature vector 216 based on at least one text description 212 and the information content of the common sense graph 210. The base model 200 generates an image feature vector 218, in particular in the form of a visual embedding in an embedding space 220. For example, the base model 200 is trained based on the graph-text feature vector 216. Alternatively, the graph-based neural network 202 can also be trained based on the embedded text description 212, the embedded information content from the common sense graph 210 and based on the image feature vector 218 together with the base model 200, wherein the base model 200 is trained based on the graph-text feature vector 216 and the image content of at least one image data and / or video data 204.

[0083] The training S8A, S8B of the basic model 200 and / or the graph-based neural network 202 is performed based on a loss value 222 that can be calculated from the text feature vector 216 and the image feature vector 218 , in particular by continuously and / or iteratively minimizing the loss value 222 .

Claims

1. A method for training a basic model (200), in particular a deep neural network DNN and / or a graph-based neural network (202), comprising the following steps: - providing (S1) at least one image data and / or video data (204), said image data and / or video data having image information from at least one domain and at least one image identification (206); - providing (S2) at least one common sense graph GKG (210), the common sense graph having information about the at least one domain; - providing (S3) at least one text description of image information of the at least one image data and / or video data (204); - embedding (S4) the at least one text description (212) in the graph-based neural network GNN (202) by means of a large language model LLM (214); - embedding (S5) the common sense graph (210) into the graph-based neural network (202); - generating (S6) a graph-text feature vector (216) according to the at least one text description (212) and the common sense graph (210) by means of the graph-based neural network (202); - generating (S7) an image feature vector (218) by means of the base model (200); - training (S8A) the basic model (200) based on the image-text feature vector (216); or, - training (S8B) the graph-based neural network based on the embedded text description (212), the embedded common sense graph (210) and according to the image feature vector (218); and training the basic model (200) based on the graphic feature vector (216) and the at least one image data and / or video data (204); and - Providing (S9) a trained base model (200) and / or a trained graph-based neural network (202).

2. The method according to claim 1, wherein the at least one common sense graph (210) is provided based on metadata about the domain, wherein further information is preferably supplemented by means of reasoning.

3. The method according to claim 1 or 2, wherein the at least one common sense graph (210) is provided from domain knowledge described in particular by domain experts and / or from information extracted from the at least one image data and / or video data (204).

4. The method according to any one of the preceding claims, wherein the at least one text description (212) has at least one natural language sentence generated based on the information content of the common sense graph (210).

5. The method according to any of the preceding claims, wherein the large language model (214) comprises a GPT-LLM and / or a BERT-LLM.

6. The method according to any of the preceding claims, wherein the at least one image data and / or video data (204) is detected by at least one optical sensor.

7. The method according to any of the preceding claims, wherein the at least one image data and / or video data (204) is generated from existing image data and / or video data by data enhancement.

8. A method according to any one of the preceding claims, wherein providing (S3) at least one text description of image information of the at least one image data and / or video data (204) comprises extracting a partial graph from the common sense graph (210) according to the at least one image identifier (206) by means of RDF molecule extraction.

9. A method according to any of the preceding claims, wherein the base model (200) and / or the graph-based neural network (202) is trained (S8A, S8B) based on a loss value (222) that can be calculated from the graph-text feature vector (216) and the image feature vector (218), in particular by continuously and / or iteratively minimizing the loss value (222).

10. Method according to any of the preceding claims, wherein a production line comprising an arrangement of devices for producing a pre-settable product is also provided.

11. The method according to claim 10, wherein after providing the production line, the method further comprises the following steps: At least one predefinable product is produced using the device grouping.

12. A system (1) for training a base model (200), in particular a deep neural network DNN and / or a graph-based neural network (202), the system (1) comprising: - a providing device, the providing device being configured to: provide at least one image data and / or video data (204), the image data and / or video data having image information from at least one domain and at least one image identifier (206); providing at least one common sense graph GKG (210) having information about the at least one domain, and providing at least one text description of image information of the at least one image data and / or video data (204); and - an evaluation and calculation device, the evaluation and calculation device being configured to: embed the at least one text description (212) into the graph-based neural network (GNN) (202) by means of a large language model (LLM) (214); embed the common sense graph (210) into the graph-based neural network (202); generate a graph-text feature vector (216) by means of the graph-based neural network (202) based on the at least one text description (212) and the common sense graph (210); generate an image feature vector (218) by means of the basic model (200); train the basic model (200) based on the graph-text feature vector (216); or train the graph-based neural network (202) based on the embedded text description (212), the embedded common sense graph (210) and based on the image feature vector (218); and train the basic model (200) based on the graph-text feature vector (216) and the at least one image data and / or video data (204); The providing device is further configured to provide a trained basic model (200) and / or a trained graph-based neural network (202).

13. A method for classifying and / or categorizing and / or segmenting image data and / or video data, the method comprising the following steps: - providing a base model (200) trained by a method according to any one of claims 1 to 11; - providing at least one, in particular identified, image data and / or video data (204); - converting the at least one image data and / or video data (204) into at least one image feature vector by means of a trained base model (200); and - Classifying and / or categorizing and / or segmenting the at least one image data and / or video data (204) according to the image feature vector by means of a trained classification algorithm and / or classification algorithm and / or segmentation algorithm of machine learning.

14. A computer program having a program code for performing at least part of the method according to any one of claims 1 to 11 or claim 13 when the computer program is executed on a computer.

15. A computer-readable data carrier having a program code of a computer program for executing at least part of the method according to any one of claims 1 to 11 or claim 13 when the computer program is executed on a computer.

Images