Image interpretation method and apparatus based on visual language model

Abstract

The application relates to an image interpretation method and device based on a visual language model. The method comprises the following steps: determining sample reasoning instructions according to visual annotations of sample images and a target knowledge graph; generating image interpretation information of the sample images based on the sample images and the sample reasoning instructions through a question and answer engine; training a to-be-trained visual language model based on the sample images, the sample reasoning instructions and the image interpretation information of the sample images, to determine a target visual language model; and determining image interpretation information of a target image according to the target image and target reasoning instructions through the target visual language model. The above scheme unifies the data format of a model training data set, improves the data set construction efficiency, saves the labor cost, and enables the trained visual language model to perform deep knowledge reasoning.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to an image interpretation method and apparatus based on a visual language model. Background Technology

[0002] Visual language models are multimodal models in the field of artificial intelligence that combine computer vision and natural language processing techniques. They aim to establish semantic connections between vision and language through joint training, achieving cross-modal understanding and generation. Currently, large-scale multimodal models combining visual and language understanding capabilities have achieved significant application results in general domains. However, because current datasets used for training visual language models are typically constructed for single tasks, the data formats are inconsistent, making them unsuitable for directly training visual language models that need to understand complex instructions. Furthermore, since the label information in these datasets is often isolated text strings, visual language models trained on these datasets cannot learn deep knowledge and therefore cannot perform deep reasoning. Solving these problems often requires manually standardizing the data format and manually expanding the dataset, increasing labor costs and reducing the efficiency of dataset construction. Therefore, how to construct datasets with a unified data format and simultaneously train visual language models capable of deep knowledge reasoning using these datasets is a problem that needs to be solved. Summary of the Invention

[0003] Therefore, it is necessary to provide a visual language model-based image interpretation method and apparatus that can train a visual language model capable of deep knowledge reasoning to address the aforementioned technical problems.

[0004] In a first aspect, this application provides an image interpretation method based on a visual language model, the method comprising:

[0005] Based on the visual annotations of the sample images and the target knowledge graph, sample reasoning instructions are determined; wherein, the sample reasoning instructions include at least one of the following: causal reasoning instructions generated based on the causal relationships between entities in the knowledge graph, and counterfactual reasoning instructions generated based on exclusionary judgments of the knowledge graph;

[0006] Based on the sample image and the sample reasoning instructions, the question-answering engine generates image explanation information for the sample image.

[0007] Based on the sample images, the sample inference instructions, and the image interpretation information of the sample images, train the visual language model to be trained and determine the target visual language model;

[0008] The target visual language model is used to determine the image interpretation information of the target image based on the target image and the target inference instructions.

[0009] In one embodiment, based on the visual annotations of the sample images and the target knowledge graph, sample inference instructions are determined, including:

[0010] The category label of the sample image is determined based on the visual annotation of the sample image;

[0011] Based on the association results between the category tags and the target knowledge graph, the entity information of the category tags at the classification level is determined;

[0012] Based on the visual annotations and the entity information, a sample reasoning instruction is generated.

[0013] In one embodiment, the visual annotation includes at least two of the following: bounding boxes for object detection, masks for instance segmentation, category labels for image classification, or point coordinates for dense counting.

[0014] In one embodiment, based on the sample image, the sample inference instruction, and the image interpretation information of the sample image, a visual language model to be trained is trained, and a target visual language model is determined, including:

[0015] The sample image is encoded by the visual encoder in the visual language model to be trained, and the image features of the sample image are determined.

[0016] The image features of the sample image are format-converted by the projector in the visual language model to be trained, and the image tags of the sample image are determined.

[0017] Based on the image labels of the sample images, the sample inference instructions, and the image interpretation information of the sample images, the natural language processing model in the visual language model to be trained is trained, and the target visual language model is determined based on the training results.

[0018] In one embodiment, the natural language processing model in the visual language model to be trained is trained based on the image tags of the sample images, the sample inference instructions, and the image interpretation information of the sample images. The target visual language model is then determined based on the training results, including:

[0019] The model training dataset and the model testing dataset are determined based on the image labels of the sample images, the sample inference instructions, and the image interpretation information of the sample images;

[0020] Using the model training dataset, the natural language processing model in the visual language model to be trained is trained to determine the model to be detected;

[0021] Using the model test dataset, determine whether the model to be detected has been trained. If so, determine the target visual language model based on the model to be detected, the visual encoder, and the projector.

[0022] In one embodiment, using the model test dataset to determine whether the model to be tested has been trained successfully includes:

[0023] Using the model test dataset, the cross-entropy loss function of the model to be tested is determined;

[0024] If the cross-entropy loss function is less than a preset loss function threshold, then the training of the model to be detected is determined to be complete.

[0025] In one embodiment, the above-described image interpretation method based on a visual language model further includes:

[0026] The target visual language model is evaluated based on the model test dataset to determine the knowledge missing information of the target visual language model.

[0027] Based on the knowledge gap information, update inference instructions are generated, and enhanced training data is generated based on the update inference instructions, so as to retrain the target visual language model using the enhanced training data.

[0028] In one embodiment, the image interpretation information of the target image is determined based on the target image and the target inference instructions using the target visual language model, including:

[0029] The target image is encoded using the visual encoder in the target visual language model to determine the image features of the target image;

[0030] The image features of the target image are converted into a format by the projector in the target visual language model to determine the image tag of the target image;

[0031] The image interpretation information of the target image is determined by the natural language processing model in the target visual language model, based on the image tag of the target image and the target inference instruction.

[0032] In one embodiment, the target knowledge graph is a multimodal knowledge graph, and the node information of the target knowledge graph includes text and / or typical images.

[0033] Secondly, this application also provides an image interpretation device based on a visual language model, the device comprising:

[0034] The sample reasoning instruction acquisition module is used to determine sample reasoning instructions based on the visual annotation of the sample image and the target knowledge graph; wherein, the sample reasoning instructions include at least one of the following instructions: causal reasoning instructions generated based on the causal relationship between entities in the knowledge graph, and counterfactual reasoning instructions generated based on the exclusionary judgment of the knowledge graph.

[0035] The sample information acquisition module is used to generate image explanation information of the sample image based on the sample image and the sample reasoning instruction through the question answering engine;

[0036] The model training module is used to train the visual language model to be trained and determine the target visual language model based on the sample image, the sample inference instruction and the image interpretation information of the sample image.

[0037] The target information acquisition module is used to determine the image interpretation information of the target image based on the target image and the target inference instructions through the target visual language model.

[0038] Thirdly, this application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0039] Based on the visual annotations of the sample images and the target knowledge graph, sample reasoning instructions are determined; wherein, the sample reasoning instructions include at least one of the following: causal reasoning instructions generated based on the causal relationships between entities in the knowledge graph, and counterfactual reasoning instructions generated based on exclusionary judgments of the knowledge graph;

[0040] Based on the sample image and the sample reasoning instructions, the question-answering engine generates image explanation information for the sample image.

[0041] Based on the sample images, the sample inference instructions, and the image interpretation information of the sample images, train the visual language model to be trained and determine the target visual language model;

[0042] The target visual language model is used to determine the image interpretation information of the target image based on the target image and the target inference instructions.

[0043] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0044] Based on the visual annotations of the sample images and the target knowledge graph, sample reasoning instructions are determined; wherein, the sample reasoning instructions include at least one of the following: causal reasoning instructions generated based on the causal relationships between entities in the knowledge graph, and counterfactual reasoning instructions generated based on exclusionary judgments of the knowledge graph;

[0045] Based on the sample image and the sample reasoning instructions, the question-answering engine generates image explanation information for the sample image.

[0046] Based on the sample images, the sample inference instructions, and the image interpretation information of the sample images, train the visual language model to be trained and determine the target visual language model;

[0047] The target visual language model is used to determine the image interpretation information of the target image based on the target image and the target inference instructions.

[0048] The aforementioned image interpretation method and apparatus based on a visual language model determines sample inference instructions based on visual annotations of sample images and a target knowledge graph; generates image interpretation information for sample images based on the sample images and sample inference instructions using a question-answering engine; trains a visual language model to be trained based on the sample images, sample inference instructions, and image interpretation information of the sample images to determine a target visual language model; and determines image interpretation information for the target image based on the target visual language model and target inference instructions. This solves the problem that visual language models trained on datasets often contain isolated text strings, preventing them from learning deep knowledge and thus hindering deep knowledge inference, and also addresses the issue of inconsistent data formats in the datasets used to train visual language models. The above scheme can construct multi-level sample reasoning instructions based on visual annotations of sample images through target knowledge graphs, thereby obtaining image explanation information of sample images corresponding to multi-level sample reasoning instructions. Based on sample images, sample reasoning instructions, and image explanation information of sample images, the visual language model can be trained, which can enrich the training corpus of the visual language model, unify the data format of the model training dataset, improve the efficiency of dataset construction, save human resources costs, and enable the trained visual language model to perform deep knowledge reasoning. Attached Figure Description

[0049] Figure 1 This is an application environment diagram of an image interpretation method based on a visual language model in one embodiment;

[0050] Figure 2 This is a flowchart illustrating an image interpretation method based on a visual language model in one embodiment;

[0051] Figure 3Here is an example diagram illustrating the generation of image explanation information in one embodiment;

[0052] Figure 4 Example diagram of image interpretation information generation in another embodiment;

[0053] Figure 5 This is a structural block diagram of an image interpretation device based on a visual language model in one embodiment;

[0054] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] The image interpretation method based on a visual language model provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 determines sample inference instructions based on the visual annotations of the sample images and the target knowledge graph; wherein the sample inference instructions include at least one of the following: causal inference instructions generated based on the causal relationships between entities in the knowledge graph, and counterfactual inference instructions generated based on exclusionary judgments of the knowledge graph; using a question-answering engine, it generates image explanation information for the sample images based on the sample images and the sample inference instructions; it trains a visual language model to be trained based on the sample images, the sample inference instructions, and the image explanation information of the sample images to determine the target visual language model; using the target visual language model, it determines the image explanation information of the target image based on the target image and the target inference instructions, and sends the image explanation information of the target image to terminal 102 via a communication network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0057] In one embodiment, such as Figure 2As shown, an image interpretation method based on a visual language model is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0058] S210. Determine the sample inference instructions based on the visual annotations of the sample images and the target knowledge graph.

[0059] The sample reasoning instructions include at least one of the following: causal reasoning instructions generated based on causal relationships between entities in the knowledge graph, and counterfactual reasoning instructions generated based on exclusionary judgments in the knowledge graph.

[0060] The sample images can be images from the desired domain, and the target knowledge graph is the knowledge graph corresponding to the desired domain. For example, if the desired domain is the ocean, then the sample images are images from the ocean domain, and the target knowledge graph is the knowledge graph corresponding to marine life. The causal reasoning instructions are generated by expanding the target knowledge graph to include not only classification relationships but also causal / functional relationships. For example, a classification relationship could be: sea urchins belong to echinoderms; a causal / functional relationship could be: sea urchins feed on algae on rocks, therefore sea urchins will appear in rocky areas; a certain type of fish lives in coral reefs, therefore this type of fish will appear in coral reef environments.

[0061] For example, the sample image is a picture of a sea urchin on a reef. The sample inference instruction could be: "Please explain why the sea urchin in the picture is in this location?" The ideal image explanation could be: "The sea urchin in the picture is probably feeding on algae on the reef, which is its typical behavior." Based on this, the target visual language model trained subsequently no longer simply identifies objects, but learns the deep logic between objects and their living environment and behavior.

[0062] Counterfactual reasoning instructions refer to the system's intelligent generation of inference instructions regarding "not existing in the image" or "why not." For example, a sample image containing only starfish and sea urchins might be used, with the inference instruction: "Does the image contain any organism belonging to the 'Chordata' phylum? Please explain why." An ideal image explanation would be: "No. The organisms in the image are starfish and sea urchins, which, according to the knowledge graph, belong to the 'Echinodermata' phylum, not the 'Chordata' phylum." Based on this, the subsequently trained target visual language model can handle negation, exclusion, and counterfactual thinking, improving its reasoning capabilities.

[0063] For example, the category label of the sample image is determined based on the visual annotation of the sample image; the entity information of the category label at the classification level is determined based on the association result between the category label and the target knowledge graph; and the sample reasoning instruction is generated based on the annotation type of the visual annotation and the entity information.

[0064] It's important to note that a knowledge graph is a semantic network that uses visualization technology to display knowledge resources and their interrelationships. It consists of nodes, edges, and attributes, and is used to reveal the dynamic development patterns of a knowledge domain. Nodes represent entities, and edges represent relationships between entities. Entities represent things or concepts in the real world, possessing unique identifiers and attributes. Relationships describe the connections between entities, typically existing as "entity-relationship-entity" triples. A knowledge graph mainly comprises two core parts: a data layer and a schema layer, which together constitute the logical framework of the knowledge graph. The data layer is responsible for storing specific knowledge units, using a triple structure to store factual information. The triple structure can be an entity-relationship-entity structure or an entity-attribute-attribute value structure. For example, an entity can be a specific person's name, location, etc., a relationship represents the semantic connection between entities, such as teacher-student relationships and geographical location associations, and attribute values ​​describe entity characteristics, such as age and education level. The data layer can be dynamically expanded, supporting the addition of new nodes and edges without affecting the original data structure. The schema layer defines the logical framework and constraints of the knowledge graph, establishing hierarchical relationships between knowledge units through ontological concepts such as course chapters and knowledge point classifications. For example, in the organic chemistry course knowledge graph, the schema layer defines a nested structure of "compound classification → physical properties → chemical properties," while the data layer stores specific data such as the boiling point and reaction mechanism of specific compounds. Entity information refers to all attribute values ​​corresponding to entities identified from the target knowledge graph based on category labels.

[0065] Common reasoning instructions include: logical reasoning, mathematical operations, knowledge application, and multimodal reasoning. Logical reasoning involves conditional reasoning or analogical reasoning based on known conditions, such as: "If all dogs like bones, and Xiaoming's family has a dog, does Xiaoming's dog like to eat bones?" Mathematical operations involve basic arithmetic and solving equations, such as: "Calculate the result of 3 multiplied by 7." Knowledge application requires answering questions using domain knowledge, such as: "Explain why water freezes below zero degrees Celsius." Multimodal reasoning involves cross-modal tasks such as image description and text-to-image generation, such as: "Generate a picture from this description."

[0066] Specifically, the source dataset is obtained, containing sample images and their corresponding visual annotations. The visual annotations include image annotation information such as category labels, bounding boxes, and keypoints. Image category labels categorize images into different classes or combinations of classes based on their content, primarily used for image classification tasks. Sample image category labels include, but are not limited to, binary labels, multi-class labels, single-label multi-class classification, and multi-label classification. The source dataset can be composed of data extracted from the COCO (Common Objects in Context) dataset, the YOLO (You Only Look Once) dataset, and / or the CSV (Comma-Separated Values) dataset. The COCO dataset is a large image dataset containing rich scenes and object categories, widely used for computer vision tasks such as object detection, segmentation, and image captioning. The YOLO dataset is a set of labeled data specifically designed for training the YOLO object detection algorithm; its core function is to help the model learn how to identify target objects and their locations in images. A CSV dataset is a tabular data collection stored in comma-separated value format, typically containing multiple columns, each corresponding to different attributes, with multiple rows forming a complete data record. The process involves determining the category labels of sample images from their visual annotations and associating these labels with a target knowledge graph. Based on the target knowledge graph, entity information for each category label at one or more classification levels is obtained. Entity information includes the standardized name of the category label, its classification level in the target knowledge graph, and other metadata about the category label. A knowledge graph is a semantic network that describes entities and their relationships in a structured way, representing knowledge relationships in the real world in a graphical structure. The target knowledge graph is the knowledge graph of the desired domain, which could be the marine domain. A task instruction generation engine is used to generate sample inference instructions based on the visual annotations and entity information. These sample inference instructions are knowledge inference instructions that require the visual language model to perform inferences using the hierarchy in the knowledge base, determining the image interpretation of the sample image obtained by the visual language model based on the sample inference instructions. For example, a sample image might be labeled "sea urchin." Visual annotations for the sample image are obtained, including bounding boxes for sea urchins. Based on these annotations, a target knowledge graph for the marine domain is queried. According to the target knowledge graph, sea urchins are classified as echinoderms, which is the entity information determined by the target knowledge graph. Meanwhile, a task instruction generation engine is a software tool or framework used to process, schedule, and execute specific tasks. It can transform high-level task instructions into executable low-level instructions and coordinate different modules or components to execute according to predetermined logic.

[0067] For example, visual annotations include at least two of the following: bounding boxes for object detection, masks for instance segmentation, category labels for image classification, or point coordinates for dense counting.

[0068] Visual annotation always includes category labels for image classification. The bounding box in object detection is a rectangular box used to mark the location and extent of a target object in an image, typically defined by four coordinate values. The mask in instance segmentation is a pixel-level segmentation result representing the boundary of each individual object in an image, usually existing in binary form, used to distinguish the boundaries of different instances. Image-wide labeling in image classification refers to assigning a single or multiple category labels to the entire image, primarily used to distinguish the category to which the image belongs. For example, a picture of a cat would be labeled "cat," while an image containing both cats and dogs might be labeled "cat and dog" or labeled separately. Point coordinates in an image are ordered pairs used to determine the pixel positions in an image, usually in pixels, described by both horizontal and vertical dimensions. Point coordinates used for dense counting refer to the point coordinates of multiple targets in a sample image.

[0069] It should be noted that when the visual annotation type is point coordinates for dense counting, the image interpretation information of the subsequently generated sample images is a structured data object containing the total count of point coordinates and each point coordinate. That is, it is a collection of data elements that are manipulated and processed in the program for each point coordinate. The data exchange format of the above structured data object is JSON (JavaScript Object Notation, a lightweight data exchange format).

[0070] The above scheme can determine entity information from the target knowledge graph based on the category labels of sample images, generate sample reasoning instructions according to the label type of visual annotation and entity information, and by fusing the knowledge graph, enable the model training data of the visual language model obtained later to go beyond simple visual labels, and endow the visual language model with the ability to understand domain concepts and perform logical reasoning.

[0071] S220. Using a question-answering engine, image explanation information for the sample images is generated based on the sample images and sample reasoning instructions.

[0072] The question-answering engine is an intelligent system built on natural language processing (NLP) technology, primarily used to automatically answer user questions. It responds quickly to user needs by using a predefined database of frequently asked questions combined with semantic matching algorithms. The image explanations provided for the sample images can be structured or unstructured. The NLP technology used can be models such as BERT or GPT, which can improve language understanding capabilities through training on large-scale corpora.

[0073] For example, if the visual annotations of a sample image include "sea urchin" and "starfish," and the target knowledge graph determines that all the visual annotations belong to the phylum "Echinodermata," the engine will generate a completely new knowledge reasoning instruction that does not exist in the original dataset. This is because the original dataset contains relatively few knowledge reasoning instructions. The reasoning instruction generated using the above method can compensate for the deficiencies in the original dataset. Based on the deep information in the knowledge graph, the knowledge reasoning instruction can more comprehensively encompass the relevant knowledge information of the sample image, thus generating a knowledge reasoning instruction that does not exist in the original dataset. Therefore, the sample reasoning instruction could be "Please draw all organisms belonging to the phylum 'Echinodermata' in the image." The image explanation information of the sample image needs to include the bounding boxes of the sea urchin and starfish, as well as the coordinates of the sea urchin and starfish bounding boxes.

[0074] S230. Based on sample images, sample inference instructions, and image interpretation information of sample images, train the visual language model to be trained and determine the target visual language model.

[0075] It should be noted that the visual language model includes a visual encoder, a projector, and a natural language processing model.

[0076] The visual encoder typically employs a CLIP (Contrastive Language–Image Pre-training) model based on the Transformer architecture, trained with millions of image-text pairs, enabling it to associate images with text. The projector is used to convert the output of the visual encoder into a format understandable by the natural language processing model, usually presented as image tags.

[0077] S240. Using the target visual language model, determine the image interpretation information of the target image based on the target image and the target reasoning instructions.

[0078] Specifically, by inputting the target image and target inference instructions into the target visual language model, the image interpretation information obtained by the target visual language model based on the target inference instructions can be obtained.

[0079] For example, if the target image is an image containing birds, and the target inference instruction is "Please draw the birds in the image", then the image interpretation information output by the target visual language model includes the bird bounding box diagram and the coordinates of the bird bounding box in the target image.

[0080] In the aforementioned image interpretation method based on a visual language model, the following steps are taken: First, sample inference instructions are determined based on the visual annotations of the sample images and the target knowledge graph. Second, image interpretation information for the sample images is generated using a question-answering engine, based on the sample images and the sample inference instructions. Third, a visual language model is trained based on the sample images, the sample inference instructions, and the image interpretation information of the sample images to determine the target visual language model. Finally, the image interpretation information for the target image is determined based on the target image and the target inference instructions using the target visual language model. This method addresses the problem that the label information in the dataset used to train the visual language model is often isolated text strings, which prevents the visual language model trained on the dataset from learning deep knowledge and thus hinders deep knowledge inference. It also solves the problem of inconsistent data formats in the dataset used to train the visual language model. The above scheme can construct multi-level sample reasoning instructions based on visual annotations of sample images through target knowledge graphs, thereby obtaining image explanation information of sample images corresponding to multi-level sample reasoning instructions. Based on sample images, sample reasoning instructions, and image explanation information of sample images, the visual language model can be trained, which can enrich the training corpus of the visual language model, unify the data format of the model training dataset, improve the efficiency of dataset construction, save human resources costs, and enable the trained visual language model to perform deep knowledge reasoning.

[0081] For example, such as Figure 3As shown, sample images labeled "sea urchin" and "starfish" are obtained from dataset A. Visual annotations for these images are acquired, including bounding boxes for sea urchins and starfish. Dataset A is the FathomNet dataset, which contains data in both COCO and JSON formats. Based on the visual annotations, a target knowledge graph for the marine domain is queried. According to the target knowledge graph, both sea urchins and starfish belong to the phylum Echinodermata, which is the entity information determined by the target knowledge graph. The visual annotations and the phylum Echinodermata can be input into a task instruction generation engine. The engine generates sample inference instructions based on these entity information, such as, but not limited to, "Please draw bounding boxes for all organisms belonging to the phylum Echinodermata in the image." The sample images and the generated inference instructions are then input into a question-answering engine, which generates image explanations for the sample images based on these instructions. For example, the image interpretation information of the sample image includes: sea urchin bounding box, starfish bounding box, sea urchin coordinates, and starfish coordinates. After determining the sample inference instructions and the image interpretation information of the sample image, the visual language model to be trained can be trained using the sample image, sample inference instructions, and the image interpretation information of the sample image to determine the target visual language model for the marine domain. Using the target visual language model, the image interpretation information of the target image is determined based on the target image and target inference instructions in the marine domain.

[0082] In one embodiment, based on sample images, sample inference instructions, and image interpretation information of the sample images, a visual language model to be trained is trained, and a target visual language model is determined, including:

[0083] The visual encoder in the visual language model to be trained encodes the sample image to determine its image features. The projector in the visual language model to be trained performs format conversion on the image features of the sample image to determine the image label of the sample image. Based on the image label of the sample image, the sample inference instructions and the image interpretation information of the sample image, the natural language processing model in the visual language model to be trained is trained, and the target visual language model is determined based on the training results.

[0084] Specifically, the sample images are encoded by the visual encoder in the visual language model to be trained, decomposing them into semantic feature vectors, i.e., image features. The projector maps the image features extracted by the visual encoder to a dimension consistent with the text embedding space through linear transformation or a multilayer perceptron, determining the image tokens, i.e., ensuring that the sample image information can be understood by the visual language model. Regular expressions can be used to match the lexical units in the sample inference instructions level by level to determine the text tokens of the sample inference instructions. Based on the image tokens of the sample images, the text tokens of the sample inference instructions, and the image interpretation information of the sample images, the natural language processing model in the visual language model to be trained is trained, and the target visual language model is determined based on the training results.

[0085] The above method proposes a way to train a visual language model, and the trained visual language model can perform deep knowledge reasoning.

[0086] In one embodiment, the natural language processing model in the visual language model to be trained is trained based on the image tags of the sample images, the sample inference instructions, and the image interpretation information of the sample images. The target visual language model is then determined based on the training results, including:

[0087] Based on the image labels, inference instructions, and interpretation information of the sample images, the model training dataset and the model testing dataset are determined. Using the model training dataset, the natural language processing model in the visual language model to be trained is trained to determine the model to be detected. Using the model testing dataset, it is determined whether the model to be detected has been trained. If so, the target visual language model is determined based on the model to be detected, the visual encoder, and the projector.

[0088] Specifically, image labels, sample inference instructions, and image interpretation information of sample images are encapsulated accordingly to determine the sample dataset. 80% of the data in the sample dataset is used as the model training dataset, and the remaining 20% ​​is used as the model testing dataset. The natural language processing model within the visual language model to be trained is used to train the model, and the trained visual language model is used as the detection model. The model prediction accuracy of the detection model is verified using the model testing dataset. If the model prediction accuracy meets the preset accuracy requirements, the detection model is considered to have completed training and is used as the target visual language model.

[0089] The process involves using a model test dataset to determine whether the model to be tested has been successfully trained, including:

[0090] The model test dataset is used to determine the cross-entropy loss function of the model to be tested; if the cross-entropy loss function is less than the preset loss function threshold, the training of the model to be tested is considered complete.

[0091] The above scheme, after training the visual language model to be trained, verifies the trained visual language model. If the trained visual language model meets the requirements, it is used as the target visual language model, which can improve the reliability of the target visual language model.

[0092] In one embodiment, the image interpretation information of the target image is determined based on the target image and the target inference instructions using the target visual language model, including:

[0093] The target image is encoded by a visual encoder in the target visual language model to determine its image features; the image features are then converted into a format by a projector in the target visual language model to determine its image tags; and finally, the image interpretation information of the target image is determined by a natural language processing model in the target visual language model based on its image tags and target inference instructions.

[0094] The above scheme can determine the image interpretation information of the target image based on the target image and the target inference instructions through the target visual language model, thereby improving the efficiency of image interpretation information acquisition.

[0095] In one embodiment, the above-described image interpretation method based on a visual language model further includes:

[0096] The performance of the target visual language model is evaluated based on the model test dataset to determine the knowledge gaps in the target visual language model. Update inference instructions are generated based on the knowledge gaps, and enhanced training data is generated based on the update inference instructions to retrain the target visual language model.

[0097] Specifically, the performance of the target visual language model is evaluated based on the model test dataset, and knowledge gaps are determined according to the evaluation results. The performance evaluation data and model defect types can be pre-correlated to automatically determine the knowledge gaps in the target visual language model based on the performance evaluation data and the correlation between the evaluation data and model defect types. For example, if the target visual language model consistently confuses two similar-looking starfish, the model defect, i.e., the knowledge gap, is a deficiency in its understanding of these two similar-looking starfish. Based on the knowledge gaps in the target visual language model, update inference instructions are generated by adding knowledge gap information to the target knowledge graph, and enhanced training data is generated based on these update inference instructions to retrain the target visual language model.

[0098] Understandably, the above solution can identify the shortcomings of the target visual language model and add new model training data accordingly. By retraining the target visual language model based on the new model training data and adjusting the model parameters, the performance of the target visual language model can be improved.

[0099] In one embodiment, the target knowledge graph is a multimodal knowledge graph, and the node information of the target knowledge graph includes text and / or typical images.

[0100] Among them, multimodal knowledge graphs are semantic network systems that integrate multi-dimensional information such as text, images, audio, and video, and enhance machine cognitive abilities by fusing heterogeneous data.

[0101] For example, nodes in a knowledge graph are not only text entities, but can also include typical images, sounds, and 3D models. For instance, the node "Atlantic cod" can be associated with one or more standard-format images in addition to its text description.

[0102] For example, a sample image is an image containing various types of fish, and the sample inference instruction is: "Based on this standard image, please draw a box around all fish of the same species in the image." The aforementioned standard image is the Atlantic cod standard image retrieved from the knowledge graph. Another sample image contains an image containing various types of birds, and the sample inference instruction is an audio clip and the instruction: "Please identify the bird in the image that makes this call."

[0103] Understandably, the above scheme introduces multimodal knowledge graphs and cross-modal reasoning instructions, thereby extending reasoning from the two-dimensional space of "text-image" to the three-dimensional space of "image-text-image" and even "audio-text-image", generating multimodal and cross-modal reasoning training data, which can improve the reasoning ability of the target visual language model.

[0104] For example, based on the above embodiments, the image interpretation method based on a visual language model includes:

[0105] Obtain a source dataset containing sample images and their corresponding visual annotations. Determine the category labels of the sample images from their visual annotations and associate them with the target knowledge graph. Based on the target knowledge graph, obtain entity information for each category label at one or more classification levels. Using a task instruction generation engine, generate sample inference instructions based on the visual annotations and entity information. These instructions must include at least one of the following: causal inference instructions generated based on causal relationships between entities in the knowledge graph, and counterfactual inference instructions generated through exclusionary judgments based on the knowledge graph. The target knowledge graph is a multimodal knowledge graph, and its node information includes text and / or typical images. Visual annotations include at least two of the following: bounding boxes for object detection, masks for instance segmentation, category labels for image classification, or point coordinates for dense counting. When the visual annotation type is point coordinates for dense counting, the image interpretation information of the subsequently generated sample images is a structured data object containing the total count of point coordinates and the data of each point coordinate—that is, a collection of data elements manipulated and processed by the program for each point coordinate. The question-answering engine generates image explanation information for the sample images based on the sample images and sample reasoning instructions.

[0106] The visual encoder in the visual language model to be trained encodes the sample image, decomposing it into semantic feature vectors, i.e., image features. The projector maps the image features extracted by the visual encoder to a dimension consistent with the text embedding space through linear transformation or multilayer perceptron, determining the image token. The text token of the sample inference instruction is determined by matching the lexical units in the sample inference instruction step by step through regular expressions.

[0107] Image labels, sample inference instructions, and image interpretation information of sample images are encapsulated accordingly to determine the sample dataset. 80% of the data in the sample dataset is used as the model training dataset, and the remaining 20% ​​is used as the model testing dataset. The natural language processing model within the visual language model to be trained is used to train the model. The trained visual language model is then used as the detection model. The cross-entropy loss function of the detection model is determined using the model testing dataset. If the cross-entropy loss function is less than a preset loss function threshold, the detection model is considered to have completed training, and the trained detection model is used as the target visual language model.

[0108] The performance of the target visual language model is evaluated using a model testing dataset, and knowledge gaps are identified based on the evaluation results. Performance evaluation data and model defect types can be pre-associated to automatically determine the target visual language model's knowledge gaps based on the performance evaluation data and the correlation between the evaluation data and model defect types. Based on the knowledge gaps in the target visual language model, update inference instructions are generated by adding the knowledge gap information to the target knowledge graph. Enhanced training data is then generated based on these update inference instructions to retrain the target visual language model.

[0109] The target image is encoded by a visual encoder in the target visual language model to determine its image features; the image features are then converted into a format by a projector in the target visual language model to determine its image tags; and finally, the image interpretation information of the target image is determined by a natural language processing model in the target visual language model based on its image tags and target inference instructions.

[0110] For example, based on the above embodiments, taking the conversion of deep-sea biometric datasets as an example, the image interpretation method based on a visual language model includes:

[0111] Sample data was extracted from datasets A, B, and C to form the source dataset. The sample data includes sample images and their visual annotations. Dataset A is the FathomNet dataset, containing data in both COCO and JSON formats. Dataset B is a Kaggle dataset, with data in CSV format. Dataset C is a private marine fish classification dataset, with its data structure consisting of folders named by category. The FathomNet dataset is an AI-powered image database for the ocean, containing extensive object detection annotations for marine life, aiming to accelerate the development of underwater image analysis technologies by standardizing and integrating expert-annotated data. The Kaggle datasets are structured or unstructured data collections provided by the Kaggle platform, covering multiple fields such as finance, healthcare, and image and text processing.

[0112] The data in different formats in the source dataset is parsed to extract the path and visual annotations for each sample image. The visual annotations include the bounding box, point coordinates, and category label of the sample image. The category labels include "seaurchin", "Holothuroidea", "starfish", and "Gadus morhua" (cod).

[0113] The system iterates through category labels, queries the target knowledge graph corresponding to the World Register of Marine Species, and recursively retrieves the complete taxonomic tree information of each category label from phylum to species. AphiaIDs are unique identifiers assigned to marine organisms by the International Standard for Marine Organisms (ISMO) to ensure the accuracy and global uniformity of classification information. For example, both "sea urchin" and "starfish" belong to "Echinodermata" (phylum Echinodermata). The classification information is cached locally in an easily queryable JSON or graph database file as a localized domain knowledge base.

[0114] For an image containing the labels “sea urchin” and “starfish” extracted from dataset A, the generated sample inference instruction can be: “Please mark the sea cucumber in the image with bounding boxes”, and the corresponding image explanation information of the sample image is a string or JSON object containing the coordinates of the bounding boxes in standard format.

[0115] like Figure 4 As shown, sample images with 15 sea urchin dot annotations are extracted from dataset B. Using a task instruction generation engine, a sample inference instruction is generated based on the sample images. This instruction could be something like, "Please return the number of sea urchins in the image and the location of each sea urchin in JSON format." The corresponding image explanation information is a precise JSON string: {"name": "Echinoidea", "count": 15, "locations": [[x1, y1],[x2, y2], ...]}. Here, 'name' is the name; 'Echinoidea' is the scientific name of a species of sea urchin within the phylum Echinodermata, mainly including sea urchins and heart-shaped sea urchins; 'count' refers to the number of sea urchins; and 'locations' refers to the coordinates of the locations.

[0116] For a sample image containing Atlantic cod extracted from dataset C, the generated sample inference instruction can be: "What kind of fish is in this picture?" and the corresponding image explanation information for the sample image can be: "This is Atlantic cod (Gadus morhua)".

[0117] The sample images, sample inference instructions, and image interpretation information of the sample images are encapsulated and packaged into a JSON file. This JSON file contains a unique identifier for each sample image, its image path, and a list of dialogues. For example, a single line of data in the JSON file could be: {"id": "image_001", "image": "path / to / image_001.jpg","conversations": [{"from": "human", "value": "Please return the number of sea urchins in the image and the location of each sea urchin in JSON format."}, {"from": "gpt", "value": "{\"name\": \"Echinoidea\", \"count\": 15, \"locations\": [[...]]}"}]}. The data in the JSON file can be used to train the visual language model to be trained and to determine the target visual language model. `id` refers to the identity identifier; `image_001` refers to the image identifier of the sample image; `path / to / image_001.jpg` refers to the loading path of the sample image; `conversations` refers to the chat list; `[{"from": "human", "value": "Please return the number of sea urchins in the image and the location of each sea urchin in JSON format."}` represents the content of the chat list.

[0118] Through the above implementation methods, this invention successfully transforms three originally isolated, heterogeneous, and task-specific datasets into a unified, domain-knowledge-rich, high-quality training corpus that supports multi-task learning automatically and intelligently. It achieves end-to-end automated processing, reducing manual data organization and annotation work from weeks or even months to hours, significantly improving data processing efficiency. By integrating knowledge graphs, the training data transcends simple visual labels, endowing the model with the ability to understand domain concepts and perform logical reasoning. It automatically creates multi-task training samples including classification, detection, dense counting, and knowledge reasoning, comprehensively stimulating the integrated capabilities of the visual language model.

[0119] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0120] Based on the same inventive concept, this application also provides a visual language model-based image interpretation apparatus for implementing the above-described visual language model-based image interpretation method. The solution provided by this apparatus is similar to the implementation described in the above-described method; therefore, the specific limitations in one or more embodiments of the visual language model-based image interpretation apparatus provided below can be found in the limitations of the visual language model-based image interpretation method described above, and will not be repeated here.

[0121] In one embodiment, such as Figure 5 As shown, an image interpretation device based on a visual language model is provided, including: a sample inference instruction acquisition module 501, a sample information acquisition module 502, a model training module 503, and a target information acquisition module 504, wherein:

[0122] The sample reasoning instruction acquisition module 501 is used to determine the sample reasoning instruction based on the visual annotation of the sample image and the target knowledge graph; wherein, the sample reasoning instruction includes at least one of the following instructions: causal reasoning instruction generated based on the causal relationship between entities in the knowledge graph, and counterfactual reasoning instruction generated based on the exclusionary judgment of the knowledge graph.

[0123] The sample information acquisition module 502 is used to generate image explanation information of the sample image based on the sample image and sample reasoning instructions through the question answering engine;

[0124] The model training module 503 is used to train the visual language model to be trained and determine the target visual language model based on sample images, sample inference instructions and image interpretation information of sample images.

[0125] The target information acquisition module 504 is used to determine the image interpretation information of the target image based on the target image and the target inference instructions through the target visual language model.

[0126] For example, the sample inference instruction acquisition module 501 is specifically used for:

[0127] Determine the category label of the sample image based on the visual annotations of the sample image;

[0128] Based on the association results between category labels and the target knowledge graph, determine the entity information of the category labels at the classification level;

[0129] Generate sample reasoning instructions based on visual annotations and entity information.

[0130] Visual annotations include at least two of the following: bounding boxes for object detection, masks for instance segmentation, category labels for image classification, or point coordinates for dense counting.

[0131] Furthermore, the model training module 503 is specifically used for:

[0132] The sample images are encoded by the visual encoder in the visual language model to be trained, and the image features of the sample images are determined.

[0133] The image features of the sample images are format-converted by the projector in the visual language model to be trained, and the image labels of the sample images are determined.

[0134] Based on the image labels, inference instructions, and interpretation information of the sample images, the natural language processing model in the visual language model to be trained is trained, and the target visual language model is determined based on the training results.

[0135] Furthermore, the model training module 503 is also specifically used for:

[0136] Based on the image labels of the sample images, the sample inference instructions, and the image interpretation information of the sample images, the model training dataset and the model testing dataset are determined;

[0137] Using the model training dataset, the natural language processing model in the visual language model to be trained is used to train the model and determine the model to be detected.

[0138] The model test dataset is used to determine whether the model to be detected has been trained. If so, the target visual language model is determined based on the model to be detected, the visual encoder, and the projector.

[0139] For example, the model training module 503 is also specifically used for:

[0140] The cross-entropy loss function of the model to be tested is determined using the model test dataset;

[0141] If the cross-entropy loss function is less than the preset loss function threshold, then the training of the model to be detected is considered complete.

[0142] For example, the model training module 503 is also specifically used for:

[0143] The performance of the target visual language model is evaluated based on the model test dataset to determine the knowledge gaps in the target visual language model.

[0144] Based on the missing knowledge information, update inference instructions are generated, and enhanced training data is generated based on the update inference instructions, so as to retrain the target visual language model through the enhanced training data.

[0145] For example, the target information acquisition module 504 is specifically used for:

[0146] The target image is encoded using a visual encoder in the target visual language model to determine its image features;

[0147] The image features of the target image are converted into a format by the projector in the target visual language model to determine the image tag of the target image.

[0148] By using the natural language processing model in the target visual language model, the image interpretation information of the target image is determined based on the image tagging and target inference instructions of the target image.

[0149] For example, the target knowledge graph mentioned above is a multimodal knowledge graph, and the node information of the target knowledge graph includes text and / or typical images.

[0150] The modules in the aforementioned image interpretation device based on visual language models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0151] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an image interpretation method based on a visual language model. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0152] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0153] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0154] Step 1: Determine the sample reasoning instructions based on the visual annotations of the sample images and the target knowledge graph; wherein the sample reasoning instructions include at least one of the following instructions: causal reasoning instructions generated based on the causal relationships between entities in the knowledge graph, and counterfactual reasoning instructions generated based on exclusionary judgments of the knowledge graph;

[0155] Step 2: Using a question-answering engine, generate image explanation information for the sample images based on the sample images and sample reasoning instructions;

[0156] Step 3: Based on the sample images, sample inference instructions, and image interpretation information of the sample images, train the visual language model to be trained and determine the target visual language model;

[0157] Step 4: Using the target visual language model, determine the image interpretation information of the target image based on the target image and the target inference instructions.

[0158] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0159] Step 1: Determine the sample reasoning instructions based on the visual annotations of the sample images and the target knowledge graph; wherein the sample reasoning instructions include at least one of the following instructions: causal reasoning instructions generated based on the causal relationships between entities in the knowledge graph, and counterfactual reasoning instructions generated based on exclusionary judgments of the knowledge graph;

[0160] Step 2: Using a question-answering engine, generate image explanation information for the sample images based on the sample images and sample reasoning instructions;

[0161] Step 3: Based on the sample images, sample inference instructions, and image interpretation information of the sample images, train the visual language model to be trained and determine the target visual language model;

[0162] Step 4: Using the target visual language model, determine the image interpretation information of the target image based on the target image and the target inference instructions.

[0163] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0164] Step 1: Determine the sample reasoning instructions based on the visual annotations of the sample images and the target knowledge graph; wherein the sample reasoning instructions include at least one of the following instructions: causal reasoning instructions generated based on the causal relationships between entities in the knowledge graph, and counterfactual reasoning instructions generated based on exclusionary judgments of the knowledge graph;

[0165] Step 2: Using a question-answering engine, generate image explanation information for the sample images based on the sample images and sample reasoning instructions;

[0166] Step 3: Based on the sample images, sample inference instructions, and image interpretation information of the sample images, train the visual language model to be trained and determine the target visual language model;

[0167] Step 4: Using the target visual language model, determine the image interpretation information of the target image based on the target image and the target inference instructions.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0169] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0170] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image interpretation method based on a visual language model, characterized in that, include: Based on the visual annotations of the sample images and the target knowledge graph, sample reasoning instructions are determined; wherein, the sample reasoning instructions include at least one of the following: causal reasoning instructions generated based on the causal relationships between entities in the knowledge graph, and counterfactual reasoning instructions generated based on exclusionary judgments of the knowledge graph; Based on the sample image and the sample reasoning instructions, the question-answering engine generates image explanation information for the sample image. Based on the sample images, the sample inference instructions, and the image interpretation information of the sample images, train the visual language model to be trained and determine the target visual language model; The target visual language model is used to determine the image interpretation information of the target image based on the target image and the target reasoning instructions. Based on the sample images, the sample inference instructions, and the image interpretation information of the sample images, a visual language model to be trained is trained, and a target visual language model is determined, including: The sample image is encoded by the visual encoder in the visual language model to be trained, and the image features of the sample image are determined. The image features of the sample image are format-converted by the projector in the visual language model to be trained, and the image tags of the sample image are determined. Based on the image labels of the sample images, the sample inference instructions, and the image interpretation information of the sample images, the natural language processing model in the visual language model to be trained is trained, and the target visual language model is determined based on the training results; Based on the image tags of the sample images, the sample inference instructions, and the image interpretation information of the sample images, the natural language processing model in the visual language model to be trained is trained, and the target visual language model is determined based on the training results, including: The model training dataset and the model testing dataset are determined based on the image labels of the sample images, the sample inference instructions, and the image interpretation information of the sample images; Using the model training dataset, the natural language processing model in the visual language model to be trained is trained to determine the model to be detected; Using the model test dataset, determine whether the model to be detected has been trained. If so, determine the target visual language model based on the model to be detected, the visual encoder, and the projector. The performance evaluation data and model defect types are correlated in advance. The target visual language model is evaluated based on the model test dataset. The knowledge missing information of the target visual language model is determined based on the performance evaluation data and the correlation between the evaluation data and the model defect types. An update inference instruction is generated based on the target knowledge graph and the knowledge missing information, and enhanced training data is generated based on the update inference instruction, so as to retrain the target visual language model using the enhanced training data.

2. The method according to claim 1, characterized in that, Based on the visual annotations of the sample images and the target knowledge graph, determine the sample inference instructions, including: The category label of the sample image is determined based on the visual annotation of the sample image; Based on the association results between the category tags and the target knowledge graph, the entity information of the category tags at the classification level is determined; Based on the visual annotations and the entity information, a sample reasoning instruction is generated.

3. The method according to claim 2, characterized in that, The visual annotations include at least two of the following: bounding boxes for object detection, masks for instance segmentation, category labels for image classification, or point coordinates for dense counting.

4. The method according to claim 1, characterized in that, Using the model test dataset, determine whether the model to be tested has been trained successfully, including: Using the model test dataset, the cross-entropy loss function of the model to be tested is determined; If the cross-entropy loss function is less than a preset loss function threshold, then the training of the model to be detected is determined to be complete.

5. The method according to claim 1, characterized in that, The target visual language model determines the image interpretation information of the target image based on the target image and the target inference instructions, including: The target image is encoded using the visual encoder in the target visual language model to determine the image features of the target image; The image features of the target image are converted into a format by the projector in the target visual language model to determine the image tag of the target image; The image interpretation information of the target image is determined by the natural language processing model in the target visual language model, based on the image tag of the target image and the target inference instruction.

6. The method according to claim 1, characterized in that, The target knowledge graph is a multimodal knowledge graph, and the node information of the target knowledge graph includes text and / or typical images.

7. An image interpretation device based on a visual language model, characterized in that, The image interpretation device based on the visual language model includes: The sample reasoning instruction acquisition module is used to determine sample reasoning instructions based on the visual annotation of the sample image and the target knowledge graph; wherein, the sample reasoning instructions include at least one of the following instructions: causal reasoning instructions generated based on the causal relationship between entities in the knowledge graph, and counterfactual reasoning instructions generated based on the exclusionary judgment of the knowledge graph. The sample information acquisition module is used to generate image explanation information of the sample image based on the sample image and the sample reasoning instruction through the question answering engine; The model training module is used to train the visual language model to be trained and determine the target visual language model based on the sample image, the sample inference instruction and the image interpretation information of the sample image. The target information acquisition module is used to determine the image interpretation information of the target image based on the target image and the target reasoning instructions through the target visual language model; Based on the sample images, the sample inference instructions, and the image interpretation information of the sample images, a visual language model to be trained is trained, and a target visual language model is determined, including: The sample image is encoded by the visual encoder in the visual language model to be trained, and the image features of the sample image are determined. The image features of the sample image are format-converted by the projector in the visual language model to be trained, and the image tags of the sample image are determined. Based on the image labels of the sample images, the sample inference instructions, and the image interpretation information of the sample images, the natural language processing model in the visual language model to be trained is trained, and the target visual language model is determined based on the training results; Based on the image tags of the sample images, the sample inference instructions, and the image interpretation information of the sample images, the natural language processing model in the visual language model to be trained is trained, and the target visual language model is determined based on the training results, including: The model training dataset and the model testing dataset are determined based on the image labels of the sample images, the sample inference instructions, and the image interpretation information of the sample images; Using the model training dataset, the natural language processing model in the visual language model to be trained is trained to determine the model to be detected; Using the model test dataset, determine whether the model to be detected has been trained. If so, determine the target visual language model based on the model to be detected, the visual encoder, and the projector. The performance evaluation data and model defect types are correlated in advance. The target visual language model is evaluated based on the model test dataset. The knowledge missing information of the target visual language model is determined based on the performance evaluation data and the correlation between the evaluation data and the model defect types. An update inference instruction is generated based on the target knowledge graph and the knowledge missing information, and enhanced training data is generated based on the update inference instruction, so as to retrain the target visual language model using the enhanced training data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images