Image tag understanding method and device, electronic equipment and storage medium

Abstract

The invention discloses an image label understanding method and device, electronic equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: marking a target label for a target image; the target label comprises a scene label and an in-scene entity category label; identifying text information in the target image; constructing a knowledge graph of the scene and entities in the scene, and converting the knowledge graph into a knowledge base; inputting the target image, the target label and the text information as question information into a knowledge base for retrieval to obtain related knowledge information; and fusing the related knowledge information and the question information into cue words, and inputting the cue words into the large language model to obtain label understanding information of the target image. The knowledge graph of the scene and the entity in the scene is constructed, the knowledge graph is converted into the knowledge base, and the knowledge base is mounted by using the large language model, so that the large language model can understand the mutual relationship between the scene tag and the text in the target image, and the understanding of the target image tag is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image label understanding method, device, electronic device and storage medium. Background Art

[0002] The image labeling function mainly identifies the scene content of the image and the entities contained in the image, and outputs the corresponding label results. Early image labeling methods used a small multi-classification model to extract features from the image and then regressed to multiple category labels, but the above methods have many problems. The small model has a weak ability to extract features. For complex scene images, it cannot directly extract the core content of the image, resulting in insufficient understanding of the image and ultimately misclassification; the small model is highly dependent on training data, resulting in poor generalization, and it is easy to cause misrecognition when the same category of scene images are very different. Therefore, the relevant technology proposes to use a multimodal model RAM based on the visual language and training (VLP) framework to recognize images. The model uses a large number of image-text pairs for training without manual annotation, and uses fine-grained text information to achieve good image labeling capabilities. Subsequently, the relevant technology introduced LLM (using the LLM model to describe the target in detail) into the image tagging stage, enhanced the model's open set label recognition capability (i.e., the recognition of category labels not provided during training), and constructed a new scene label recognition algorithm RAM++.

[0003] Although a series of image labeling methods proposed by related technologies have good effects on open domain datasets and can output images to understand the general scene, the main targets in the image, and related actions or states, the output results are always a discrete bunch of labels. The correlation between entities in the scene is not summarized and organized, and the detailed meaning or sub-categories of the scene are not recognized and output, and the image labels are not fully understood. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide an image tag understanding method, device, electronic device and storage medium to improve the understanding of image tags.

[0005] To achieve the above objective, an embodiment of the present application provides an image tag understanding method in one aspect, the method comprising the following steps:

[0006] Annotating a target image with a target label; wherein the target label includes a scene label and an entity category label within the scene;

[0007] Recognizing text information in the target image;

[0008] Construct a knowledge graph of scenarios and entities within the scenarios, and convert the knowledge graph into a knowledge base;

[0009] Inputting the target image, the target label and the text information as question information into the knowledge base for retrieval to obtain relevant knowledge information;

[0010] The relevant knowledge information and the question information are fused into prompt words, and the prompt words are input into a large language model to obtain label understanding information of the target image.

[0011] In some embodiments, the step of labeling the target image with a target label comprises the following steps:

[0012] Labeling the target image as a pre-label;

[0013] Training a multimodal model using the target image annotated with the pre-label;

[0014] The trained multimodal model is used to identify the target image to obtain the target label.

[0015] In some embodiments, identifying text information in the target image comprises the following steps:

[0016] The Chinese text of the target image is detected and recognized as the text information.

[0017] In some embodiments, before inputting the target image, the target label, and the text information as question information into the knowledge base for retrieval to obtain relevant knowledge information, the method further includes the following steps:

[0018] Using an open vocabulary detection model to detect whether the target label is accurate;

[0019] Remove inaccurate target labels.

[0020] In some embodiments, the step of constructing a knowledge graph of scenes and entities within the scenes comprises the following steps:

[0021] Acquire multiple images of different scenes at different times as scene images;

[0022] Extracting scene entities in the scene image;

[0023] extracting a target object entity from the scene image according to the scene entity;

[0024] Determining the relationship between the scene entity and the target object entity;

[0025] The knowledge graph of the scene and the entities within the scene is constructed based on the mutual relationships.

[0026] In some embodiments, converting the knowledge graph into a knowledge base comprises the following steps:

[0027] Extracting the scene entities, the target object entities and the mutual relationships from the knowledge graph using a natural language extraction method, and generating descriptive statements for each of the scene entities in combination with the scene definition;

[0028] Using the LLM embedding model to segment the descriptive sentence according to segmentation symbols and word length restrictions to obtain segmented text;

[0029] Using the LLM embedding model to extract features from the segmented text to obtain a feature vector matrix;

[0030] An index is constructed for the feature vector matrix and written into a database as the knowledge base.

[0031] In some embodiments, the step of inputting the target image, the target label, and the text information as question information into the knowledge base for retrieval to obtain relevant knowledge information includes the following steps:

[0032] The question information is retrieved from the knowledge base and the related knowledge information is generated using a retrieval enhancement generation model.

[0033] To achieve the above objective, another aspect of the present application provides an image tag understanding device, the device comprising:

[0034] A labeling unit, used to label the target image with a target label; wherein the target label includes a scene label and an entity category label within the scene;

[0035] A text recognition unit, used to recognize text information in the target image;

[0036] A knowledge acquisition unit, used to construct a knowledge graph of scenes and entities within the scenes, and convert the knowledge graph into a knowledge base;

[0037] A question retrieval unit, used for inputting the target image, the target label and the text information as question information into the knowledge base for retrieval to obtain relevant knowledge information;

[0038] The label understanding unit is used to fuse the relevant knowledge information and the question information into prompt words, and input the prompt words into a large language model to obtain label understanding information of the target image.

[0039] In some embodiments, the label marking unit includes:

[0040] A pre-labeling unit, used to label the target image as a pre-label;

[0041] A model training unit, configured to train a multimodal model using the target image annotated with the pre-label;

[0042] The target label marking unit is used to identify the target image using the trained multimodal model to obtain the target label.

[0043] In some embodiments, the text recognition unit includes:

[0044] The text recognition subunit is used to detect and recognize the Chinese text in the target image as the text information.

[0045] In some embodiments, the apparatus further comprises a label screening unit, wherein the label screening unit comprises:

[0046] A label detection unit, used to detect whether the target label is accurate using an open vocabulary detection model;

[0047] The label removal unit is used to remove the inaccurate target label.

[0048] In some embodiments, the knowledge acquisition unit includes a knowledge graph construction unit, and the knowledge graph construction unit includes:

[0049] An image acquisition unit, used to acquire multiple images of different scenes at different times as scene images;

[0050] A scene entity extraction unit, used to extract scene entities in the scene image;

[0051] a target object entity extraction unit, configured to extract a target object entity from the scene image according to the scene entity;

[0052] A mutual relationship determination unit, used to determine the mutual relationship between the scene entity and the target object entity;

[0053] The knowledge graph construction subunit is used to construct the knowledge graph of the scene and the entities in the scene based on the mutual relationships.

[0054] In some embodiments, the knowledge acquisition unit further includes a knowledge base conversion unit, and the knowledge base conversion unit includes:

[0055] A sentence generation unit, configured to extract the scene entities, the target object entities and the mutual relationships from the knowledge graph using a natural language extraction method, and generate descriptive sentences for each of the scene entities in combination with a scene definition;

[0056] A text segmentation unit, used for segmenting the descriptive sentence according to segmentation symbols and word length restrictions by using an LLM embedding model to obtain segmented text;

[0057] A feature extraction unit, used for extracting features from the segmented text using the LLM embedding model to obtain a feature vector matrix;

[0058] The knowledge base construction unit is used to construct an index for the feature vector matrix and write it into a database as the knowledge base.

[0059] In some embodiments, the question retrieval unit includes:

[0060] The question retrieval subunit is used to retrieve the question information in the knowledge base and generate the relevant knowledge information using the retrieval enhancement generation model.

[0061] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned image label understanding method when executing the computer program.

[0062] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned image label understanding method is implemented.

[0063] The embodiments of the present application include at least the following beneficial effects:

[0064] The present application can annotate target images with target labels; wherein the target labels include scene labels and entity category labels within the scene; identify text information in the target image; construct a knowledge graph of scenes and entities within the scene, and convert the knowledge graph into a knowledge base; input the target image, target label, and text information into the knowledge base as question information for retrieval to obtain relevant knowledge information; fuse the relevant knowledge information and question information into prompt words, and input the prompt words into a large language model to obtain label understanding information of the target image. By constructing a knowledge graph of scenes and entities within the scene, converting the knowledge graph into a knowledge base, and using a large language model to mount the knowledge base, the large language model can understand the relationship between scene labels and text in the target image, thereby improving the understanding of the target image label. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 A flowchart of an image label understanding method provided in an embodiment of the present application;

[0067] Figure 2 An example flowchart for constructing a knowledge graph provided in an embodiment of the present application;

[0068] Figure 3 An example flow chart of converting a knowledge graph into a knowledge base provided in an embodiment of the present application;

[0069] Figure 4 An example flow chart of an image tag understanding method provided in an embodiment of the present application;

[0070] Figure 5 An example structural diagram of a labeling model provided in an embodiment of the present application;

[0071] Figure 6 An example diagram of a knowledge graph provided in an embodiment of the present application;

[0072] Figure 7 A schematic diagram of the structure of an image label understanding device provided in an embodiment of the present application;

[0073] Figure 8 A schematic diagram of the structure of a knowledge graph construction unit provided in an embodiment of the present application;

[0074] Fig. 9 A schematic diagram of the structure of a knowledge base conversion unit provided in an embodiment of the present application;

[0075] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0077] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0078] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0079] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0080] Before describing the embodiments of the present application in detail, some terms and related technologies involved in the embodiments of the present application are first described as follows:

[0081] Knowledge Graph: A semantic network that reveals the relationships between entities.

[0082] Multimodal model: An AI model that can process multiple types of data, such as text, images, audio, and video.

[0083] Image Tag: Understand the image scene content and typical object targets in the scene, and output one or more labels corresponding to the image.

[0084] Large Language Model (LLM): A language model trained with large-scale expected data is one of the methods of natural language processing.

[0085] Open-set: The label set may or may not belong to the training data label set, and the labels are open.

[0086] Retrieval-augmented Generation (RAG) is a model that combines retrieval and generation techniques. It generates answers or content by referring to information from external knowledge bases. It has strong interpretability and customization capabilities and is suitable for multiple natural language processing tasks such as question-answering systems, document generation, and intelligent assistants.

[0087] Image recognition and understanding has long been an important direction in the field of images. From the initial classification and recognition of single targets in images to the current recognition and understanding of complex scene images, the corresponding recognition methods have also developed from small classification models to current multimodal large models. Similarly, image labeling is an important task in scene recognition and understanding. How to label images in detail and accurately is also a difficult task. For example, stores can be divided into different stores according to the different items sold, such as clothing stores, fruit stores, grain and oil stores, supermarkets, etc. Usually, the labeling models in the industry can only recognize coarse category scenes and objects in the scene, such as stores, oranges, apples, people, etc., but the existing technology cannot well associate this knowledge to obtain a detailed description of the scene. In addition, there are plaques with corresponding names at the entrances of many restaurants and schools. Usually, the model can only recognize whether the scene is a restaurant or a school, and cannot accurately recognize the specific name of the restaurant or school.

[0088] In view of the above problems, this application proposes a multimodal image scene label understanding method combined with a knowledge graph. This application can include a multimodal image labeling model to output multiple category labels, an open domain detection model to verify the reliability of the multimodal model labels, and a text detection and recognition method to extract the main text in the image. After that, the reliable category multi-labels are combined with the Chinese detection and recognition results to construct a joint text information as the prompt information. Due to the hallucination problem of large language models, when some unrelated labels or texts are input, some seemingly reasonable but actually incorrect results will be output. Therefore, this application proposes to build a large language model question-answering system mounted with a locally built knowledge graph. By loading the existing knowledge of the local knowledge base, the input prompt information is integrated and understood, and finally a detailed description of the image scene understanding is output.

[0089] The embodiments of the present application provide an image label understanding method, device, electronic device and storage medium. The technical solution of the present application includes: labeling a target image with a target label; wherein the target label includes a scene label and an entity category label within the scene; identifying text information in the target image; constructing a knowledge graph of scenes and entities within the scene, and converting the knowledge graph into a knowledge base; inputting the target image, target label and text information into the knowledge base as question information for retrieval to obtain relevant knowledge information; fusing the relevant knowledge information and question information into prompt words, and inputting the prompt words into a large language model to obtain label understanding information of the target image. By constructing a knowledge graph of scenes and entities within the scene, converting the knowledge graph into a knowledge base, and using a large language model to mount the knowledge base, the large language model can understand the relationship between the scene label and the text in the target image, thereby improving the understanding of the target image label.

[0090] The embodiment of the present application provides an image label understanding method, which relates to the field of image processing technology. The image label understanding method provided by the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements an image label understanding method, etc., but is not limited to the above forms.

[0091] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0092] Reference Figure 1 The present application embodiment provides an image tag understanding method, which may include but is not limited to S100 to S140, as follows:

[0093] S100: labeling a target image with a target label; wherein the target label includes a scene label and an entity category label within the scene.

[0094] Specifically, the target image may be an image in different scenes. This embodiment may identify the scene in the target image and label the corresponding scene label. It may also perform target detection on the target image, identify the targets and categories contained therein, and use the detected targets as entities in the scene and label them with corresponding entity category labels. The scene label and the entity category label may be used as target labels for the target image.

[0095] Furthermore, S100 may include S101 to S103:

[0096] S101: labeling the target image as a pre-label.

[0097] Exemplarily, this embodiment can use an open source scene labeling model and a preset scene labeling small model to infer and label the target image with pre-labels; then, the pre-labeled target images are batch detected through a general detection model trained by business data to verify the reliability of the pre-labels; a small number of unreliable pre-labels are manually verified and re-labeled, and finally a large number of target images labeled with multiple pre-labels are obtained.

[0098] S102: Training a multimodal model using the target image annotated with the pre-label.

[0099] Exemplarily, this embodiment can use the target image annotated with pre-labels to train a multimodal model based on the clip double-tower structure.

[0100] S103: Using the trained multimodal model to identify the target image, and obtain the target label.

[0101] Specifically, the multimodal model is used to infer the target image to obtain the scene label and the entity category label in the scene, and the inference result is stored in the document for subsequent operations. The storage result format is as follows: ImageName, Tag1, Tag2, Tag3,…

[0102] To further improve the reliability of the target tag, the embodiment of the present application may further include the following steps:

[0103] Using an open vocabulary detection model to detect whether the target label is accurate;

[0104] Remove inaccurate target labels.

[0105] Specifically, this embodiment can perform target detection on an open set, that is, for the target label output by the multimodal model, the open vocabulary detection model GroundingDINO is used to detect the target image, and the reliability of the target label is fed back according to the detection result (if the target corresponding to the category is detected, the target label is considered to be accurate, and if it is not detected, the target label is considered to be inaccurate and removed) to improve the reliability of the target label.

[0106] S110: Identify text information in the target image.

[0107] It is understandable that the target image may contain text information, and this embodiment can recognize the text therein to obtain the text information.

[0108] Furthermore, S110 may include:

[0109] The Chinese text of the target image is detected and recognized as the text information.

[0110] Exemplarily, this embodiment can use the lightweight ChineseOCR-lite method to detect and recognize Chinese text in the target image, and merge the recognized Chinese text with the above-mentioned target tag to obtain the following storage text format: ImageName, Tag1, Tag2, Tag3, Text1, Text2, Text3,…

[0111] S120: Construct a knowledge graph of scenes and entities within the scenes, and convert the knowledge graph into a knowledge base.

[0112] It can be understood that there is a relationship between scenes and entities within the scenes, and a corresponding knowledge graph can be constructed based on the relationship. In order to improve the ability of the large language model to understand the knowledge graph, this embodiment can convert the knowledge graph into a knowledge base.

[0113] Further, refer to Figure 2 , the step of constructing the knowledge graph of the scene and the entities in the scene in S120 includes S201 to S205:

[0114] S201: Acquire multiple images of different scenes at different times as scene images.

[0115] Exemplarily, images of indoor and outdoor scenes in the morning, noon, evening and seasons of various regions are acquired, and close-up images of objects, damaged images, duplicate images, etc. are cleaned and removed.

[0116] S202: Extracting scene entities from the scene image.

[0117] Exemplarily, the location hierarchy definition is determined based on the collected scene images, and a list of collected locations is compiled, for example, indoor locations include living rooms, dining rooms, kitchens, offices, meeting rooms, and the like.

[0118] S203: Extracting a target object entity from the scene image according to the scene entity.

[0119] Specifically, the target object entity may be a typical object that is representative in the scene. Representative objects in each scene, such as a sofa, a coffee table, etc., are extracted as the target object entity based on scene image analysis.

[0120] S204: Determine the relationship between the scene entity and the target object entity.

[0121] Specifically, the relationship between the scene entity and the target item entity is determined, such as the inclusion relationship, position relationship, attribute relationship, etc.; the relationship between typical item entities in the scene can also be determined, such as AND, OR, mutual exclusion, etc.

[0122] S205: Construct the knowledge graph of the scene and the entities in the scene according to the mutual relationship.

[0123] As a further embodiment, refer to Figure 3 The step of converting the knowledge graph into a knowledge base in S120 includes S301 to S304:

[0124] S301: extracting the scene entities, the target object entities and the mutual relationships from the knowledge graph using a natural language extraction method, and generating descriptive statements for each of the scene entities in combination with the scene definition.

[0125] Specifically, the knowledge graph is converted into a local knowledge document: the natural language extraction method is used to extract the scene entities, target object entities and their relationships in the knowledge graph, and the corresponding descriptive sentences are formed by combining the scene definitions, descriptions and other related information on the wiki. For example, "the living room is an area in a home used for leisure and social activities. The living room contains typical items such as sofas, coffee tables, and televisions." Finally, the descriptive sentences of each scene entity generated are stored in the local knowledge document.

[0126] S302: Using the LLM embedding model to segment the descriptive sentence according to segmentation symbols and word length restrictions to obtain segmented text.

[0127] For example, the Chinese embedding model bge-large-zh-v1.5 is selected as the LLM embedding model to segment descriptive sentences.

[0128] Specifically, segmentation is performed based on segmentation symbols (period, exclamation mark, question mark, line break, etc.) and the token (word unit) length limit of the Chinese embedding model.

[0129] S303: Using the LLM embedding model to perform feature extraction on the segmented text to obtain a feature vector matrix.

[0130] S304: construct an index for the feature vector matrix and write it into a database as the knowledge base.

[0131] Specifically, the FAISS database is selected to construct an index based on the obtained eigenvector matrix and write it into the database.

[0132] S130: Input the target image, the target label and the text information as question information into the knowledge base for retrieval to obtain relevant knowledge information.

[0133] Specifically, this embodiment can retrieve relevant knowledge information corresponding to the question information in the knowledge base.

[0134] Further, S130 may include:

[0135] The question information is retrieved from the knowledge base and the related knowledge information is generated using a retrieval enhancement generation model.

[0136] Specifically, the question information is sent to RAG to retrieve the knowledge base that is most relevant to the question information.

[0137] S140: The relevant knowledge information and the question information are integrated into prompt words, and the prompt words are input into a large language model to obtain label understanding information of the target image.

[0138] Specifically, the large language model refers to the question information and the relevant knowledge information retrieved to generate the final answer, that is, a true, accurate, and detailed description of the image scene understanding. The local knowledge base guides the large language model to generate detailed, true, and reliable answers that conform to human subjective understanding when integrating label information and outputting scene label descriptions.

[0139] Next, the solution of the embodiment of the present application will be described in detail and explained in conjunction with specific application examples:

[0140] Reference Figure 4 The solution of this embodiment includes labeling model, open set target detection, Chinese detection and recognition, building a large language model mounted on the local knowledge graph, and reasoning application.

[0141] 1. Labeling model ( Figure 5 A structural diagram of an optional labeling model):

[0142] 1.1 Build scene multi-label data (i.e. pre-label):

[0143] Use the open source scene labeling model and the preset scene labeling model to infer the target image and mark the pre-labels. Then, use the general detection model trained by business data to perform batch detection on the pre-labeled target images to verify the reliability of the pre-labels. Manually verify and re-label a small number of unreliable pre-labels, and finally obtain a large number of target images marked with multiple pre-labels.

[0144] 1.2 Training a multimodal model based on the clip dual-tower structure:

[0145] Since the multimodal model based on the clip twin-tower structure contains an image and text alignment module, the multimodal model directly using this architecture can better understand the image semantic information. However, since the clip model itself uses a large amount of network image and text data for training, including a large number of English scenes and corresponding English text descriptions, there are large differences in the scenes itself and the actual surveillance camera shooting scenes in the Chinese environment, and there are also large differences in the Chinese-English translation description process. Therefore, this fourteenth example uses the scene pre-label constructed in 1.1 to fully fine-tune the multimodal model of the clip twin-tower structure to improve the image understanding accuracy of the multimodal model in the Chinese environment.

[0146] 1.3 Use the multimodal model to infer the target image to obtain the target label, that is, the scene label and the entity category label in the scene. The inference result is stored in the document for subsequent operations. The storage result format is as follows: ImageName, Tag1, Tag2, Tag3, …

[0147] 2. Open Set Object Detection:

[0148] The open vocabulary detection model GroundingDINO is used to detect the target image, and the reliability of the target label is fed back based on the detection result (if the target corresponding to the category is detected, the label is considered accurate, and if it is not detected, the label is considered inaccurate) to improve the reliability of the target label.

[0149] 3. Chinese detection and recognition:

[0150] The lightweight ChineseOCR-lite method is used to detect and recognize Chinese text in the target image, and the Chinese text is fused with the target label to obtain the stored text format as follows: ImageName, Tag1, Tag2, Tag3, Text1, Text2, Text3,…

[0151] 4. Build a large language model mounted on the local knowledge graph:

[0152] 4.1 Build a knowledge graph for local scene understanding:

[0153] In the process of understanding and analyzing the scenes, it was found that the definitions of some specific scenes are strongly correlated with human activities and typical objects in the scenes. For example, an office is an indoor place where people work, usually containing office equipment such as desks, chairs, computers, monitors, etc.; the living room is mainly used to provide a place for people to rest and entertain, so typical objects in the scene are sofas, coffee tables, TVs, etc. In order to better understand each scene, these fourteen examples extract and organize human behaviors, target objects in typical scenes, and the correlation between scenes to construct a scene understanding knowledge graph. Figure 6 This is an optional knowledge graph example. The detailed steps for building a knowledge graph are as follows:

[0154] 1) Data collection: Obtain indoor and outdoor scene images in various regions in the morning, noon, evening and seasons, and clean and remove close-up images of objects, damaged images, duplicate images, etc.

[0155] 2) Scene entity extraction: Determine the place hierarchy definition based on the collected scene images and organize the collected place list, for example, indoor places include living rooms, dining rooms, kitchens, offices, meeting rooms, etc.

[0156] 3) Target object entity extraction: Based on scene image analysis, representative objects in each scene, such as sofas, coffee tables, etc., are extracted as target object entities.

[0157] 4) Relationship determination: Determine the relationship between scene entities and target item entities, such as inclusion relationship, position relationship, attribute relationship, etc.; you can also determine the relationship between typical item entities in the scene, such as AND, OR, mutual exclusion, etc.

[0158] 4.2 Knowledge graph is converted into local knowledge base and mounted:

[0159] Since the current multimodal model and large language model cannot directly have an in-depth understanding of the knowledge graph, it is necessary to convert the knowledge graph into a local knowledge base and mount it using a large language model. The detailed operations are as follows:

[0160] 1) Knowledge graph is converted into local knowledge document: The scene entities, target object entities and their relationships in the knowledge graph are extracted using natural language extraction methods, and the corresponding descriptive sentences are formed by combining the scene definition, description and other related information on wiki. For example, "The living room is an area in a family used for leisure and social activities. The living room contains typical items such as sofas, coffee tables, and televisions." Finally, the descriptive sentences of each scene entity are stored in the local knowledge document.

[0161] 2) Select a suitable large language feature extraction model: In this embodiment, the Chinese embedding model bge-large-zh-v1.5 can be selected.

[0162] 3) Text segmentation: Descriptive sentences are segmented according to segmentation symbols (period, exclamation mark, question mark, line break, etc.) and the token length limit of the embedding model.

[0163] 4) Text vectorization: The segmented text is sent to the Chinese embedding model for feature extraction to obtain a feature vector matrix.

[0164] 5) Data storage: In this embodiment, the FAISS database may be selected to construct an index for the feature vector matrix and write it into the database.

[0165] 6) Use the RAG (retrieval-augmented generation) function of the LangChain framework to mount the local knowledge base and guide the large language model to output real and reliable scene descriptions.

[0166] 5. Reasoning Application:

[0167] In the actual application stage, the target label and text information of the target image to be inferred obtained through the above 1 and 3 operations are used as question information and sent to RAG to retrieve and recall the knowledge most relevant to the question information. The retrieved relevant knowledge information is fused with the question information as a prompt and sent to the large language model. The large language model generates the final answer with reference to the question information and the retrieved relevant knowledge information, that is, the real, accurate and detailed understanding description of the image scene understanding.

[0168] In response to the problem that the labels output by the scene labeling model in the industry are not accurate enough, this embodiment proposes an open set detection method to verify the labeling results output by the labeling model, and to help improve the reliability of the labels. In view of the weak Chinese text extraction ability of the scene labeling model for images, a lightweight Chinese detection and recognition method is introduced to identify and extract the text information in the image, and combine the obtained text information with the label information. In view of the problem that general large language models are prone to hallucinations, it is proposed to construct a local knowledge graph based on the correlation between scenes and the correlation between scenes and target items in the scene, and convert the knowledge graph into a knowledge base. The method of using a large language model to mount the knowledge base helps the large language model understand the scene labels and the relationship between the text content in the image, reducing the generation of hallucinations.

[0169] This embodiment provides a multimodal image label understanding solution that combines image recognition, target detection, OCR detection and recognition, large language model question and answer, and knowledge graph. Image information is acquired through detection and recognition methods such as graphic models, and the large language model is used to associate the objects recognized in the image with the scene labels. The local knowledge base is mounted to guide the large language model to generate detailed, true, and reliable answers that conform to human subjective understanding when integrating label information and outputting scene label descriptions. This embodiment enhances the in-depth understanding of image scenes in surveillance camera scenarios, and plays an important role in the subsequent calling of different industry algorithms for different scenarios. Due to the mounting of the local knowledge base, the image can be more accurately understood locally. And this embodiment can realize intelligent image understanding question and answer, and can be used as part of the intelligent system later.

[0170] Reference Figure 7 The embodiment of the present application further provides an image label understanding device, which can implement the above-mentioned image label understanding method, and the device includes:

[0171] A labeling unit, used to label the target image with a target label; wherein the target label includes a scene label and an entity category label within the scene;

[0172] A text recognition unit, used to recognize text information in the target image;

[0173] A knowledge acquisition unit, used to construct a knowledge graph of scenes and entities within the scenes, and convert the knowledge graph into a knowledge base;

[0174] A question retrieval unit, used for inputting the target image, the target label and the text information as question information into the knowledge base for retrieval to obtain relevant knowledge information;

[0175] The label understanding unit is used to fuse the relevant knowledge information and the question information into prompt words, and input the prompt words into a large language model to obtain label understanding information of the target image.

[0176] In some embodiments, the label marking unit includes:

[0177] A pre-labeling unit, used to label the target image as a pre-label;

[0178] A model training unit, configured to train a multimodal model using the target image annotated with the pre-label;

[0179] The target label marking unit is used to identify the target image using the trained multimodal model to obtain the target label.

[0180] In some embodiments, the text recognition unit includes:

[0181] The text recognition subunit is used to detect and recognize the Chinese text in the target image as the text information.

[0182] In some embodiments, the apparatus further comprises a label screening unit, wherein the label screening unit comprises:

[0183] A label detection unit, used to detect whether the target label is accurate using an open vocabulary detection model;

[0184] The label removal unit is used to remove the inaccurate target label.

[0185] Reference Figure 8 In some embodiments, the knowledge acquisition unit includes a knowledge graph construction unit, and the knowledge graph construction unit includes:

[0186] An image acquisition unit, used to acquire multiple images of different scenes at different times as scene images;

[0187] A scene entity extraction unit, used to extract scene entities in the scene image;

[0188] a target object entity extraction unit, configured to extract a target object entity from the scene image according to the scene entity;

[0189] A mutual relationship determination unit, used to determine the mutual relationship between the scene entity and the target object entity;

[0190] The knowledge graph construction subunit is used to construct the knowledge graph of the scene and the entities in the scene based on the mutual relationships.

[0191] Reference Fig. 9 In some embodiments, the knowledge acquisition unit further includes a knowledge base conversion unit, and the knowledge base conversion unit includes:

[0192] A sentence generation unit, configured to extract the scene entities, the target object entities and the mutual relationships from the knowledge graph using a natural language extraction method, and generate descriptive sentences for each of the scene entities in combination with a scene definition;

[0193] A text segmentation unit, used for segmenting the descriptive sentence according to segmentation symbols and word length restrictions by using an LLM embedding model to obtain segmented text;

[0194] A feature extraction unit, used for extracting features from the segmented text using the LLM embedding model to obtain a feature vector matrix;

[0195] The knowledge base construction unit is used to construct an index for the feature vector matrix and write it into a database as the knowledge base.

[0196] In some embodiments, the question retrieval unit includes:

[0197] The question retrieval subunit is used to retrieve the question information in the knowledge base and generate the relevant knowledge information using the retrieval enhancement generation model.

[0198] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0199] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image tag understanding method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0200] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0201] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0202] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0203] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes an image label understanding method of the embodiment of this application;

[0204] Input / output interface 1003, used to implement information input and output;

[0205] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0206] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0207] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0208] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image label understanding method is implemented.

[0209] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0210] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0211] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0212] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0213] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0215] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0216] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0217] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0218] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0220] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0221] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for understanding image labels, characterized in that: The method comprises the following steps: Annotating a target image with a target label; wherein the target label includes a scene label and an entity category label within the scene; Recognizing text information in the target image; Construct a knowledge graph of scenarios and entities within the scenarios, and convert the knowledge graph into a knowledge base; Inputting the target image, the target label and the text information as question information into the knowledge base for retrieval to obtain relevant knowledge information; The relevant knowledge information and the question information are fused into prompt words, and the prompt words are input into a large language model to obtain label understanding information of the target image.

2. The image label understanding method according to claim 1, characterized in that: The step of labeling the target image with a target label comprises the following steps: Labeling the target image as a pre-label; Training a multimodal model using the target image annotated with the pre-label; The trained multimodal model is used to identify the target image to obtain the target label.

3. The image label understanding method according to claim 1, characterized in that: The identifying of text information in the target image comprises the following steps: The Chinese text of the target image is detected and recognized as the text information.

4. The image label understanding method according to claim 1, characterized in that: Before inputting the target image, the target label and the text information as question information into the knowledge base for retrieval to obtain relevant knowledge information, the method further includes the following steps: Using an open vocabulary detection model to detect whether the target label is accurate; Remove inaccurate target labels.

5. The image label understanding method according to claim 1, characterized in that: The construction of the knowledge graph of scenes and entities in the scenes includes the following steps: Acquire multiple images of different scenes at different times as scene images; Extracting scene entities in the scene image; extracting a target object entity from the scene image according to the scene entity; Determining the relationship between the scene entity and the target object entity; The knowledge graph of the scene and the entities within the scene is constructed based on the mutual relationships.

6. The image label understanding method according to claim 5, characterized in that: The step of converting the knowledge graph into a knowledge base comprises the following steps: Extracting the scene entities, the target object entities and the mutual relationships from the knowledge graph using a natural language extraction method, and generating descriptive statements for each of the scene entities in combination with the scene definition; Using the LLM embedding model to segment the descriptive sentence according to segmentation symbols and word length restrictions to obtain segmented text; Using the LLM embedding model to extract features from the segmented text to obtain a feature vector matrix; An index is constructed for the feature vector matrix and written into a database as the knowledge base.

7. The image label understanding method according to any one of claims 1 to 6, characterized in that: The step of inputting the target image, the target label and the text information as question information into the knowledge base for retrieval to obtain relevant knowledge information comprises the following steps: The question information is retrieved from the knowledge base and the related knowledge information is generated using a retrieval enhancement generation model.

8. An image label understanding device, characterized in that: The device comprises: A labeling unit, used to label the target image with a target label; wherein the target label includes a scene label and an entity category label within the scene; A text recognition unit, used to recognize text information in the target image; A knowledge acquisition unit, used to construct a knowledge graph of scenes and entities within the scenes, and convert the knowledge graph into a knowledge base; A question retrieval unit, used for inputting the target image, the target label and the text information as question information into the knowledge base for retrieval to obtain relevant knowledge information; The label understanding unit is used to fuse the relevant knowledge information and the question information into prompt words, and input the prompt words into a large language model to obtain label understanding information of the target image.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements an image label understanding method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, an image tag understanding method according to any one of claims 1 to 7 is implemented.

Images