Image recognition method, device, equipment, storage medium and program product

Abstract

The embodiments of the present application provide an image recognition method, apparatus, device, storage medium and program product, which relate to the field of artificial intelligence technology. The method includes: visually encoding an image to be recognized, obtaining target image features, and obtaining N candidate categories from a set of fine-grained categories based on the target image features. The category description texts of the N candidate categories are understood to retrieve the target attributes used to distinguish the N candidate categories, and the attribute information of the N candidate categories under the target attributes is extracted. The category description texts of the N candidate categories and the attribute information of the N candidate categories under the target attributes are added as enhanced information to the fine-grained image recognition process for the image to be recognized, so as to combine information of multiple modalities for fine-grained image recognition, effectively improve the understanding of the distinguishing features between fine-grained categories, and help to obtain more accurate fine-grained image recognition results, thereby improving the accuracy of fine-grained image recognition.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to an image recognition method, apparatus, device, storage medium, and program product. Background Art

[0002] With the development of artificial intelligence (AI) technology, the accuracy of computer vision tasks has gradually improved. Fine-grained visual recognition (FGVR) is a computer vision task that requires more detailed sub-classification of coarse-grained large-category images. Since the feature differences between subcategories are small and difficult to distinguish, fine-grained image recognition is more difficult than general / generic images recognition tasks.

[0003] In the related technology, the Contrastive Language-Image Pre-Training (CLIP) model is used to extract the visual modal features of the target image, and then the fine-grained category of the target image is predicted by calculating the similarity between the visual modal features of the target image and the textual modal features of each fine-grained category name.

[0004] However, the CLIP model focuses on identifying the association between text and images, and has a poor understanding of the distinguishing features between fine-grained categories. Therefore, the predicted fine-grained category of the target image often does not match the actual category of the target image. Therefore, when the CLIP model is used for fine-grained image recognition, its accuracy is low. Summary of the invention

[0005] Embodiments of the present application provide an image recognition method, apparatus, device, storage medium, and program product for improving the accuracy of fine-grained image recognition.

[0006] On the one hand, an embodiment of the present application provides an image recognition method, including:

[0007] Visually encode the image to be identified to obtain the target image features;

[0008] Based on the target image feature, obtaining N candidate categories from a fine-grained category set, where the fine-grained category set is a set of multiple subcategories of the same category, where N is greater than 1;

[0009] Based on the category description texts of the N candidate categories, extract the attribute information of the N candidate categories under the target attribute; wherein the target attribute is: an attribute used to distinguish the N candidate categories;

[0010] Based on the image to be identified, the category description texts of the N candidate categories and the attribute information of the N candidate categories under the target attribute, the target category of the image to be identified is obtained from the N candidate categories.

[0011] On the one hand, an embodiment of the present application provides an image recognition device, including:

[0012] A visual retrieval module is used to perform visual encoding on the image to be identified to obtain target image features; based on the target image features, N candidate categories are obtained from a fine-grained category set, where the fine-grained category set is a set of multiple subcategories of the same category, where N is greater than 1;

[0013] A text retrieval module, used to extract attribute information of each of the N candidate categories under a target attribute based on the category description text of each of the N candidate categories; wherein the target attribute is: an attribute used to distinguish the N candidate categories;

[0014] The recognition module is used to obtain the target category of the image to be recognized from the N candidate categories based on the image to be recognized, the category description text of each of the N candidate categories and the attribute information of each of the N candidate categories under the target attribute.

[0015] Optionally, the text retrieval module is specifically used for:

[0016] Generate a first prompt text based on the category description texts of the N candidate categories and the target attribute;

[0017] The first prompt text is used as a prompt instruction, and a text modality model in a trained multimodal large language model is input for reasoning to obtain attribute information of each of the N candidate categories under the target attribute.

[0018] Optionally, the text retrieval module is specifically used for:

[0019] Through the text modality model, the following steps are performed:

[0020] Based on the context information contained in the first prompt text, the first prompt text is subjected to text intent recognition to obtain a first recognition result; the first recognition result indicates: a first intention tendency operation for the target attribute of each of the N candidate categories;

[0021] By performing the first intention tendency operation, attribute information of each of the N candidate categories under the target attribute is obtained from the corpus.

[0022] Optionally, the text retrieval module is specifically used for:

[0023] Selecting the target attribute by accessing an external knowledge base;

[0024] or,

[0025] The second prompt text is used as a prompt instruction, and the text modal model is input for reasoning to obtain a basic attribute set for distinguishing different fine-grained categories, wherein the second prompt text includes at least one of the following category description texts: category description text of the same category to which the fine-grained category set belongs, and category description texts of multiple fine-grained categories in the fine-grained category set;

[0026] The target attribute is selected from the basic attribute set.

[0027] Optionally, the text retrieval module is specifically used for:

[0028] Through the text modality model, the following steps are performed:

[0029] Based on the context information contained in the second prompt text, the second prompt text is subjected to text intent recognition to obtain a second recognition result; the second recognition result indicates: a second intention tendency operation for distinguishing different fine-grained categories;

[0030] By performing the second intention tendency operation, a basic attribute set for distinguishing different fine-grained categories is obtained.

[0031] Optionally, the visual retrieval module is specifically used for:

[0032] Obtaining a category image feature of each fine-grained category in the fine-grained category set, and determining a first similarity between the target image feature and the category image feature of each fine-grained category;

[0033] The obtained multiple first similarities are sorted in descending order, and the fine-grained categories corresponding to the first N similarities are taken as candidate categories.

[0034] Optionally, the visual retrieval module is specifically used for:

[0035] Acquire a set of example images corresponding to a plurality of fine-grained categories in the set of fine-grained categories, wherein each fine-grained category corresponds to at least one example image in the set of example images;

[0036] Performing visual encoding on each example image in the example image set to obtain corresponding example image features;

[0037] Clustering the obtained multiple example image features to obtain image feature clusters corresponding to the multiple fine-grained categories, wherein one image feature cluster includes: at least one example image feature, and the at least one example image feature includes: a feature clustering center of the one image feature cluster;

[0038] The feature clustering center of the image feature cluster corresponding to each fine-grained category is used as the category image feature of each fine-grained category.

[0039] Optionally, the visual retrieval module is specifically used for:

[0040] Performing text encoding on the category description text of each fine-grained category in the fine-grained category set to obtain corresponding category text features;

[0041] Determining a second similarity between the target image feature and the category text feature of each fine-grained category;

[0042] The obtained multiple second similarities are sorted in descending order, and the fine-grained categories corresponding to the second similarities ranked in the top N positions are taken as candidate categories.

[0043] Optionally, the identification module is specifically used for:

[0044] Generate target prompt content according to a preset prompt content template based on the image to be identified, the category description text of each of the N candidate categories, and the attribute information of each of the N candidate categories under the target attribute;

[0045] The target prompt content is used as a prompt instruction and input into the multimodal large language model for reasoning to obtain the target category of the image to be recognized.

[0046] Optionally, the identification module is specifically used for:

[0047] The following steps are performed using the multimodal large language model:

[0048] Performing text encoding on the text content in the target prompt content to obtain text modality features, wherein the text content includes: category description texts of the N candidate categories and attribute information of the N candidate categories under the target attribute;

[0049] Visually encoding the image to be identified in the target prompt content to obtain target image features;

[0050] Fusing the text modality feature and the target image feature to obtain a fused coding feature;

[0051] Multiple rounds of iterative text prediction are performed based on the fused coding features to obtain the target category of the image to be recognized, wherein, in a round of text prediction, the output result of this round of text prediction process is obtained based on the fused coding features and the output result of the previous round.

[0052] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned image recognition method when executing the program.

[0053] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned image recognition method.

[0054] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned image recognition method.

[0055] In an embodiment of the present application, single-modality self-enhanced retrieval is performed in the visual modality and the textual modality respectively, and the obtained retrieval results of the visual modality (i.e., the category description texts of the N candidate categories) and the retrieval results of the textual modality (i.e., the attribute information of the N candidate categories in the target attributes) are used as enhanced information and added to the reasoning process of fine-grained image recognition to obtain the final fine-grained category from the N candidate categories. This can effectively improve the understanding of the distinguishing features between fine-grained categories, thereby improving the accuracy of fine-grained image recognition.

[0056] Secondly, the image to be identified is visual modal information, and the retrieval results of the visual modality and the retrieval results of the text modality are both text modal information. This application uses different modalities to express information in different forms, which makes the expression of information richer. Therefore, when combining information from multiple modalities for fine-grained image recognition, it helps to improve the accuracy of understanding fine-grained images, thereby obtaining more accurate fine-grained image recognition results and improving the accuracy of fine-grained image recognition.

[0057] In addition, the visual modality model in the multimodal large language model is called for visual modality retrieval; the text modality model in the multimodal large language model is called for text modality retrieval. Finally, the image to be recognized, the retrieval results obtained by the visual modality retrieval, and the retrieval results obtained by the text modality retrieval are integrated, and the multimodal large language model is used for reasoning to obtain the final fine-grained image recognition result. The entire fine-grained image recognition process only uses the multimodal large language model, and no additional external models are used, avoiding dependence on external models and eliminating the need for information exchange with external models, thereby effectively improving the efficiency of fine-grained image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1 A schematic diagram of a system architecture provided in an embodiment of the present application;

[0060] Figure 2 An example of an application scenario provided by the present application Figure 1 ;

[0061] Figure 3 An example of an application scenario provided by the present application Figure 2 ;

[0062] Figure 4 A schematic diagram of an image recognition method provided in an embodiment of the present application Figure 1 ;

[0063] Figure 5 A schematic diagram of the structure of a multimodal large language model provided in an embodiment of the present application;

[0064] Figure 6 A flowchart of a multimodal large language model splitting method provided in an embodiment of the present application;

[0065] Figure 7 A flowchart of a visual encoding process provided in an embodiment of the present application;

[0066] Figure 8 A flowchart of a visual modality retrieval process provided in an embodiment of the present application;

[0067] Fig. 9 A flow chart of a text modality model reasoning process provided in an embodiment of the present application Figure 1;

[0068] Fig.10 A flow chart of a text modality model reasoning process provided in an embodiment of the present application Figure 2 ;

[0069] Fig.11 A flow chart of a text modality model reasoning process provided in an embodiment of the present application Figure 3 ;

[0070] Fig.12 A flow chart of a text modality model reasoning process provided in an embodiment of the present application Figure 4 ;

[0071] Fig.13 A flowchart of a multimodal large language model reasoning process provided in an embodiment of the present application;

[0072] Fig.14 An example diagram of iterative text prediction provided in an embodiment of the present application;

[0073] Fig.15 A schematic diagram of an image recognition method provided in an embodiment of the present application Figure 2 ;

[0074] Fig.16 A schematic diagram of an image recognition method provided in an embodiment of the present application Figure 3 ;

[0075] Fig.17 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0076] Fig.18 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0078] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0079] The embodiments of the present application relate to artificial intelligence (AI) technology, and are mainly designed based on computer vision (CV) technology in artificial intelligence technology.

[0080] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0081] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0082] Computer vision technology is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in various visual fields can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0083] Transformer: It is a common deep learning model architecture that has been widely used in various fields such as natural language processing, computer vision (CV), and speech processing. When Transformer was first proposed, it was a sequence-to-sequence model architecture for machine translation. It consists of an encoder and a decoder. Both the encoder and the decoder are composed of a series of Transformer blocks with the same structure. Each Transformer block consists of at least a multi-head self-attention layer and a feedforward neural network layer. At present, Transformer has become a common architecture in natural language processing and is often used as a pre-training model. In addition to language-related applications, Transformer is also used in computer vision, audio processing and other fields.

[0084] Large Language Model (LLM), also known as large model, natural language model or large language model, refers to a natural language processing model with large-scale parameters and training data. The training process of large language models usually adopts unsupervised learning, that is, the model is trained through a large-scale text corpus to learn the probability distribution and language rules of the language. During the training process, large language models usually use a language model (LanguageModel) as the objective function, that is, the model parameters are optimized by maximizing the prediction probability of the next word. It is trained on a large-scale corpus and can generate high-quality natural language texts, such as articles, conversations, etc.

[0085] Modality: In the embodiments of this application, each media form or carrier mode of information can be called a modality, for example, audio, video, image and text can be called a modality. In the field of AI, multimodality refers to the ability to combine and use multiple different media forms (also called input or output modes), that is, the AI ​​system can understand and process multiple types of data.

[0086] Multimodal Large Language Model (MLLM) is a model extended from LLM that has the ability to receive and reason about multimodal information. In other words, MLLM is usually based on a large language model and incorporates other non-textual modal information to complete various multimodal tasks.

[0087] Pretrained Model: refers to a model trained on a large-scale corpus, usually using unsupervised learning methods, such as autoencoders, language models, etc. The basic idea of ​​a pretrained model is to use a large-scale corpus and unsupervised learning methods to allow the model to learn a large amount of general knowledge and rules, thereby serving as the basic model for various natural language processing tasks.

[0088] Embedding: It usually refers to the process of mapping data (such as text, pictures, audio, etc.) to a low-dimensional vector space. Specifically, Embeddings can be regarded as a feature representation learning technology that can convert raw data (such as words in text, sentences or pixels in images, etc.) into low-dimensional, continuous vector representations that can capture the semantic or structural information of the original data.

[0089] Prompt: Prompt text is used to provide input text or instructions to the model to guide the model to generate a specific type of response. The prompt can be a question, a description, a task instruction, or even a part of the conversation history. By designing and optimizing the prompt, the model can be guided to generate expected responses or complete specific tasks.

[0090] Prompt Learning: It is a natural language generation technology based on human prompts. It uses the knowledge of pre-trained language models to better use the knowledge of pre-trained language models. It adds additional text to the input segment to improve the generation quality and efficiency of the final generated model. The basic idea is to give a human prompt or partial generation result during the generation process, called prompt text, which represents a command or instruction to instruct the large language model to perform an action or generate output, so that the model can continue to generate text on this basis. Prompt learning can be applied to various natural language generation tasks, such as text summarization, machine translation, dialogue generation, etc.

[0091] The Contrastive Language-Image Pre-Training (CLIP) model is a multimodal pre-trained neural network and an effective and scalable method for learning from natural language supervision. The core idea of ​​the model is to use a large amount of paired data of images and texts for pre-training to learn the alignment relationship between images and texts. The CLIP model has two modalities, one is the text modality and the other is the visual modality, and includes two main parts: 1. Text encoding: used to convert text into a low-dimensional vector representation (Embeding). 2. Visual encoding: used to convert images into vector representations (Embedding). In the prediction stage, the CLIP model generates prediction results by calculating the cosine similarity between text and image vectors.

[0092] Fine-grained Visual Recognition (FGVR), also known as fine-grained image classification, is a computer vision task that refers to the more detailed sub-classification of coarse-grained large-category images.

[0093] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless cars, autonomous driving, drones, digital twins, virtual humans, robots, Artificial Intelligence Generated Content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0094] The solution provided in the embodiments of the present application mainly involves the application of artificial intelligence technology in fine-grained image recognition. Fine-grained image recognition has a wide range of research needs and application scenarios both in industry and academia, mainly including the recognition of different types of birds, dogs, flowers, cars, airplanes, etc. In real life, there is a great application demand for identifying different subcategories. For example, in ecological protection, effective identification of different types of organisms is an important prerequisite for ecological research. For example, in the medical image classification scenario, by extracting tiny features in medical images, the accuracy of medical image classification is improved to better assist doctors in disease diagnosis, lesion localization and treatment planning.

[0095] In the process of fine-grained image recognition, the solution provided by the embodiment of the present application realizes fine-grained image classification with the help of the expert knowledge of the multimodal large language model itself. Specifically, according to the inherent capabilities of the multimodal large language model itself, the multimodal large language model is split into a visual modality model and a text modality model. The N candidate categories corresponding to the image to be identified are retrieved from the fine-grained category set through the visual modality model. The attribute information of the target attributes used to distinguish the N candidate categories is retrieved through the text modality model. The retrieval results of the visual modality (i.e., the category description texts of the N candidate categories), the retrieval results of the text modality (i.e., the attribute information of the target attributes of the N candidate categories), and the image to be identified are input into the multimodal large language model for reasoning to obtain the final fine-grained image classification result.

[0096] The design concept of the embodiments of the present application is introduced below.

[0097] In practical applications, fine-grained image recognition, as a computer vision task, requires more detailed sub-classification of coarse-grained basic category (i.e., large category) images. Since the feature differences between subcategories are small and difficult to distinguish, fine-grained image recognition is more difficult than ordinary general image recognition tasks.

[0098] In the related technology, the pre-trained CLIP model is used to extract the visual modal features of the target image, and then the fine-grained category of the target image is predicted by calculating the similarity between the visual modal features of the target image and the textual modal features of each fine-grained category name.

[0099] However, the CLIP model focuses on identifying the association between text and images, and has a poor understanding of the distinguishing features between fine-grained categories. Therefore, the predicted fine-grained category of the target image often does not match the actual category of the target image. Therefore, when the CLIP model is used for fine-grained image recognition, its accuracy is low.

[0100] Based on this, an embodiment of the present application provides an image recognition method, in which the image to be recognized is first visually encoded to obtain target image features. Then, based on the target image features, N candidate categories are obtained from a fine-grained category set to understand the image to be recognized in a visual modality, so as to retrieve N candidate categories matching the image to be recognized from the fine-grained category set.

[0101] Considering that different fine-grained categories differ in some attributes, after obtaining N candidate categories, the category description texts of the N candidate categories are understood in the text modality to retrieve the target attributes used to distinguish the N candidate categories, and extract the attribute information of each of the N candidate categories under the target attributes.

[0102] Finally, the retrieval results of the visual modality (i.e., the category description texts of the N candidate categories), the retrieval results of the textual modality (i.e., the attribute information of the N candidate categories in the target attributes) and the image to be identified are combined for reasoning to obtain the target category of the image to be identified from the N candidate categories.

[0103] In an embodiment of the present application, single-modality self-enhanced retrieval is performed in the visual modality and the textual modality respectively, and the obtained retrieval results of the visual modality (i.e., the category description texts of the N candidate categories) and the retrieval results of the textual modality (i.e., the attribute information of the N candidate categories in the target attributes) are used as enhanced information and added to the reasoning process of fine-grained image recognition to obtain the final fine-grained category from the N candidate categories. This not only effectively improves the understanding of the distinguishing features between fine-grained categories, but also improves the accuracy of fine-grained image recognition.

[0104] Secondly, the image to be identified is visual modal information, and the retrieval results of the visual modality and the retrieval results of the text modality are both text modal information. This application uses different modalities to express information in different forms, which makes the expression of information richer. Therefore, when combining information from multiple modalities for fine-grained image recognition, it helps to improve the accuracy of understanding fine-grained images, thereby obtaining more accurate fine-grained image recognition results and improving the accuracy of fine-grained image recognition.

[0105] In addition, the visual modality model in the multimodal large language model is called for visual modality retrieval, and the text modality model in the multimodal large language model is called for text modality retrieval, thus realizing self-enhanced retrieval within the multimodal large language model. Then, the retrieval results of the visual modality retrieval and the text modality retrieval are used to enhance the fine-grained recognition reasoning of the multimodal large language model for the image to be recognized, and the final recognition result is obtained. The entire fine-grained image recognition process only uses the multimodal large language model, and no additional external models are used, thus avoiding dependence on external models and not requiring information exchange with external models, thereby effectively improving the efficiency of fine-grained image recognition.

[0106] The following is a brief introduction to the system architecture diagram applicable to the technical solution of the embodiment of the present application. It should be noted that the system architecture diagram introduced below is only used to illustrate the embodiment of the present application and is not a limitation.

[0107] refer to Figure 1 , which is a system architecture diagram applicable to an embodiment of the present application. The system architecture includes at least a terminal device 101 and a server 102. The number of terminal devices 101 can be one or more, and the number of servers 102 can also be one or more. The present application does not specifically limit the number of terminal devices 101 and servers 102.

[0108] The terminal device 101 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart home appliance, an intelligent voice interaction device, an intelligent vehicle-mounted device, etc., but is not limited thereto.

[0109] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, but is not limited to these.

[0110] It should be noted that the method in the embodiment of the present application can be executed separately by the terminal device 101 or the server 102, or can be executed jointly by the terminal device 101 and the server 102. When executed separately by the terminal device 101 or the server 102, the multimodal large language model is deployed on the terminal device 101 or the server 102. The multimodal large language model includes at least: a visual modality model and a text modality model. The terminal device 101 or the server 102 calls the visual modality model to retrieve N candidate categories corresponding to the image to be identified from the fine-grained category set. The text modality model is called to retrieve the attribute information of the N candidate categories on the target attribute (i.e., the distinguishing attributes of the N candidate categories). Finally, through the multimodal large language model, based on the retrieval results of the visual modality (i.e., the category description text of the N candidate categories), the retrieval results of the text modality (i.e., the attribute information of the N candidate categories on the target attribute) and the image to be identified, reasoning is performed to obtain the final fine-grained image classification result.

[0111] When executed jointly by the server 102 and the terminal device 101, the visual modality model and the text modality model can be disassembled from the multimodal large language model, and the visual modality model and the text modality model are deployed on the terminal device 101, and the server 102 locally deploys the multimodal large language model before the split. The terminal device 101 calls the visual modality model to retrieve N candidate categories corresponding to the image to be identified from the fine-grained category set. The text modality model is called to retrieve the attribute information of the N candidate categories on the target attribute (i.e., the distinguishing attribute of the N candidate categories). Then the retrieval results of the visual modality model, the retrieval results of the text modality, and the image to be identified are sent to the server 102. The server 102 inputs the retrieval results of the visual modality model, the retrieval results of the text modality, and the image to be identified into the multimodal large language model for reasoning to obtain the final fine-grained image classification result, and the two cooperate to realize the fine-grained image recognition process. In actual application, specific configuration can be made according to the situation, and this application is not specifically limited here.

[0112] Among them, the server 102 and the terminal device 101 may include one or more processors, memories, and interactive I / O interfaces, etc. In addition, the server 102 may also be configured with a database, which may be used to store model parameters of a multimodal large language model, etc. Among them, the memories of the server 102 and the terminal device 101 may also store program instructions required to be executed in the image recognition method provided in the embodiment of the present application, and these program instructions can be used to implement the fine-grained image recognition process provided in the embodiment of the present application when executed by the processor.

[0113] It should be noted that when the image recognition method provided in the embodiment of the present application is executed by the server 102 or the terminal device 101 alone, the system architecture of the present application may also include only a single device of the server 102 or the terminal device 101, or the server 102 and the terminal device 101 may be considered to be the same device. Of course, in actual application, when the image recognition method provided in the embodiment of the present application is jointly executed by the server 102 and the terminal device 101, the server 102 and the terminal device 101 may also be the same device, that is, the server 102 and the terminal device 101 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0114] In some embodiments, the user can provide an image to be identified through the terminal device 101 to initiate a fine-grained image recognition process, so that the server 102 can receive the image to be identified provided by the user, and then use the image recognition method of the embodiment of the present application to determine the fine-grained image classification result of the image to be identified, and return it to the terminal device 101 for presentation.

[0115] In the embodiment of the present application, the terminal device 101 and the server 102 may be directly or indirectly connected to each other through one or more networks. The network may be a wired network or a wireless network, for example, the wireless network may be a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, and of course, other possible networks, which are not limited in the embodiment of the present application.

[0116] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0117] The solution provided in the embodiments of this application can be applied to fine-grained image recognition in various application scenarios, such as medical image classification scenarios, ecological protection scenarios, commodity identification scenarios, security scenarios, etc. The solution can also be used as an adaptation method for multimodal large models in any downstream visual tasks. The solution can also be applied as a basic technology in various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, audio and video, etc.

[0118] The following is an illustrative description of the application scenarios to which the solution provided in this application is applicable.

[0119] Application scenario 1: Fine-grained image recognition of flowers in ecological protection scenarios.

[0120] See also Figure 2 , calling the visual modality model in the multimodal large language model to perform visual modality retrieval on the flower image, and outputting the category names of two candidate categories, specifically: "1. Cherry blossoms; 2. Plum blossoms".

[0121] The text modality model in the multimodal large language model is called to perform text modality retrieval based on the category names of the above two candidate categories, obtain the target attributes (flower color and petal shape) used to distinguish the above two candidate categories, and extract the attribute information of the above two candidate categories under the target attributes. Then the following retrieval results are output: "1. Attributes of cherry blossoms: The flower color of cherry blossoms is pink, and the petal shape of cherry blossoms is oval. 2. Attributes of plum blossoms: The flower color of plum blossoms is pink, and the petal shape of plum blossoms is oval."

[0122] The flower image, the retrieval results output by the visual modality model, and the retrieval results output by the text modality model are input into the multimodal large language model for reasoning, and the fine-grained image recognition result of the flower image is output, that is, the category name of the target category corresponding to the flower image: "cherry blossom".

[0123] Application scenario 2: Fine-grained image recognition of medical images in medical image classification scenario.

[0124] See also Figure 3 , calling the visual modality model in the multimodal large language model to perform visual modality retrieval on the lung nodule image, and outputting the category names of 2 candidate categories, specifically: "1. Class II lung nodules; 2. Class III lung nodules".

[0125] The text modality model in the multimodal large language model is called to perform text modality retrieval based on the category names of the above two candidate categories, obtain the target attributes (size and growth rate) used to distinguish the above two candidate categories, and extract the attribute information of each of the above two candidate categories under the target attribute. Then the following retrieval results are output:

[0126] “1. Attributes of Class II pulmonary nodules: less than 5mm in diameter, and basically no growth within a year.

[0127] 2. Attributes of Category III pulmonary nodules: diameter greater than 5 mm, and growth within one year.

[0128] The lung nodule image, the retrieval results output by the visual modality model, and the retrieval results output by the text modality model are input into the multimodal large language model for reasoning, and the fine-grained image recognition result of the lung nodule image is output, that is, the category name of the target category of the lung nodule image: "Class II lung nodule".

[0129] It should be noted that the embodiments of the present application are not limited to the several application scenarios exemplified above, but may also be other application scenarios, and the present application does not make any specific limitation on this.

[0130] based on Figure 1 The system architecture diagram shown in the figure, the embodiment of the present application provides a process of an image recognition method, such as Figure 4 As shown, the process of the method is executed by a computer device, which may be Figure 1 The terminal device 101 and / or server 102 shown includes the following steps:

[0131] Step 401, visually encode the image to be recognized to obtain target image features.

[0132] In an embodiment of the present application, the multimodal large language model has the ability to receive and infer multimodal information. Specifically, based on the large language model, it infers the input content of multiple modalities and outputs the inference results, wherein the input content of multiple modalities includes: non-text modal information and text modal information. Non-text modal information can be images, audio or video, etc. Text modal information can be: prompt text (prompt) that guides the output of the multimodal large language model. For example, "What kind of flower is in the picture".

[0133] In some embodiments, see Figure 5 ,The multimodal large language model includes: visual encoder, text encoder, connector and large language model.

[0134] The visual encoder performs visual encoding on the input image to obtain visual modality features. The visual modality features are processed through the connector and the processing results are input into the large language model, which can bridge the gap between non-text modality and text modality so that the large language model can better understand the visual modality features (i.e. non-text modality features). The text modality encoder performs text encoding on the input text to obtain text modality features, which are then input into the large language model. The large language model performs reasoning based on the visual modality features and text modality features to obtain the reasoning results.

[0135] In this application, see Figure 6 , based on the multimodal large language model Its inherent ability to transform multimodal large language models Split into visual modality models and text modality models These two single-mode expert models, the visual modality model At least includes: visual encoder; text modality model is also called language modality model, text modality model At least includes: text encoder and large language model. Visual modality model Used for visual modality retrieval, text modality model Perform text modality retrieval and multimodal large language model It is used to integrate the retrieval results of the visual modality, the retrieval results of the text modality and the image to be recognized for reasoning, and output the final fine-grained image recognition results.

[0136] In a possible implementation, the multimodal large language model can be split into a visual modal model, a text modal model, and other model structures. In fine-grained image recognition, the visual modal model or the text modal model can be called separately for reasoning, or the multimodal large language model composed of the visual modal model, the text modal model, and other model structures can be called for reasoning. In other words, essentially only one multimodal large language model is deployed on the computer device. Calls to the visual modal model or the text modal model are all internal calls to the multimodal large language model. Therefore, the reasoning process does not need to rely on an external model, and there is no information exchange with the external model, thereby effectively improving the efficiency of model reasoning.

[0137] Of course, you can also split the multimodal large language model into a visual modal model, a text modal model, and other model structures, and then copy the visual modal model and the text modal model and deploy them independently of the multimodal large language model. In other words, a multimodal large language model is deployed on the computer device, and the visual modal model and the text modal model copied from the multimodal large language model are also deployed. When performing fine-grained image recognition, you can call a separately deployed visual modal model or text modal model for reasoning, or you can call the deployed multimodal large language model for reasoning.

[0138] In the embodiment of the present application, the visual encoder in the visual modality model visually encodes the image to be recognized to obtain the target image features. In practical applications, the visual encoder can be implemented by using Vision Transformer (VIT), or the encoding part in other possible models can be used, which is not limited by the embodiment of the present application.

[0139] See also Figure 7 The figure is a flow chart of a visual encoding process provided by an embodiment of the present application. First, the image to be recognized is segmented to obtain multiple image blocks, and the image blocks are input into a vectorization layer, which is used to perform image encoding on the multiple image blocks respectively to obtain corresponding image block encoding features.

[0140] According to the positions of the multiple image blocks in the image to be identified, the corresponding position coding features are obtained. Then, according to the obtained position coding features, the obtained image block coding features are serialized and coded to obtain the corresponding target image features. Specifically, Figure 7 1 to 9 are the position coding features corresponding to the 9 image blocks. For each image block, the position coding feature can be added to the corresponding image block coding feature according to its position in the image to be identified, and finally the image block coding feature with the added position coding feature is input to the coding layer for serialization coding to obtain the corresponding target image feature.

[0141] The encoding layer may be an encoder in a Transformer model, or other possible encoder structures may be used, which is not limited in the embodiments of the present application.

[0142] Step 402: Obtain N candidate categories from a set of fine-grained categories based on the target image features.

[0143] Specifically, the fine-grained category set is a set of multiple subcategories of the same category. That is, the fine-grained category set includes multiple fine-grained categories, and the multiple fine-grained categories are subcategories of the same category (ie, the basic category), wherein the basic category is also called a large category or a coarse-grained category.

[0144] For example, the fine-grained category set includes three fine-grained categories: rose, peony, and peony. The three fine-grained categories are subcategories of the basic category (flower).

[0145] In addition to the visual encoder, the visual modality model also includes a pre-built feature library for visual modality retrieval, which includes: the general category features of each fine-grained category in the fine-grained category set. After extracting the target image features of the image to be identified through the visual modality model, the target image features are compared with the general category features of each fine-grained category in the feature library to obtain N candidate categories from the fine-grained category set, where N is greater than 1.

[0146] In some embodiments, the general category features of the fine-grained categories contained in the feature library are: category image features of the fine-grained categories. The embodiment of the present application uses at least the following method to obtain the category image features of each fine-grained category:

[0147] See also Figure 8 , obtain a set of example images corresponding to multiple fine-grained categories, each fine-grained category corresponds to at least one example image in the example image set. When a fine-grained category corresponds to multiple example images, these multiple example images can be different in dimensions such as posture, lighting, and perspective. Then, each example image in the example image set is visually encoded to obtain the corresponding example image features.

[0148] Specifically, each example image in the example image set is visually encoded by the visual encoder in the visual modality model to obtain the example image feature set. The process of visually encoding the example image is the same as the process of visually encoding the image to be identified, which will not be repeated here.

[0149] Next, cluster multiple example image features in the obtained example image feature set to obtain image feature clusters corresponding to multiple fine-grained categories. In the embodiment of the present application, clustering algorithms (Clustering) for clustering example image features include but are not limited to: Kmeans clustering algorithm, aggregation clustering algorithm, fuzzy logic-based clustering algorithm, and density-based clustering algorithm.

[0150] Each image feature cluster includes: at least one example image feature, and these example image features include: the feature cluster center of the image feature cluster. When an image feature cluster contains only one example image feature, then the example image feature is also the feature cluster center of the image feature cluster.

[0151] The feature clustering center of the image feature cluster represents the common image features of the fine-grained categories associated with the image feature cluster. Therefore, this application uses the feature clustering center of the image feature cluster corresponding to each fine-grained category as the category image feature of each fine-grained category, and saves the category image feature of each fine-grained category in the feature library.

[0152] In the embodiment of the present application, the visual modality model in the multimodal large language model is called to perform visual encoding on multiple fine-grained category example images to obtain corresponding example image features. Then, the obtained multiple example image features are clustered to construct a feature library for visual modality retrieval. This process uses the single mode expert model of the multimodal large language model for reasoning, does not need to rely on external models, and does not need to exchange information with external models, thereby effectively improving the efficiency of constructing the feature library.

[0153] In some embodiments, when the general category features of the fine-grained category included in the feature library are: category image features of the fine-grained category, the embodiment of the present application uses at least the following methods to perform visual modality retrieval:

[0154] See also Figure 8 , first obtain the category image features of each fine-grained category in the fine-grained category set. The specific acquisition method has been introduced in the previous article and will not be repeated here. The image to be identified is visually encoded by the visual encoder in the visual modality model to obtain the target image features. Visual model retrieval is performed based on the target image features and the feature library to obtain the category description texts of N candidate categories. In the specific implementation, the first similarities between the target image features and the category image features of each fine-grained category in the feature library are determined. Then, the multiple first similarities obtained are sorted in order from large to small, and the fine-grained categories corresponding to the first similarities ranked in the top N are used as candidate categories, and the category description texts of the N candidate categories are output.

[0155] Specifically, the first similarity between the target image feature and the category image feature of the fine-grained category can be: cosine similarity, Euclidean distance, Manhattan distance, etc. The larger the first similarity, the smaller the difference between the target image feature and the category image feature; the greater the probability that the image to be identified corresponding to the target image feature belongs to the fine-grained category corresponding to the category image feature; accordingly, the smaller the first similarity, the larger the difference between the target image feature and the category image feature; the smaller the probability that the image to be identified corresponding to the target image feature belongs to the fine-grained category corresponding to the category image feature.

[0156] Therefore, after calculating the first similarities between the target image features and the category image features of each fine-grained category in the feature library, the multiple first similarities obtained are sorted in order from large to small, and the fine-grained categories corresponding to the first similarities ranked in the top N are taken as candidate categories, and the category description text of each candidate category is output, where N can be set according to actual conditions.

[0157] It should be noted that in addition to the implementation method of selecting the N fine-grained categories with the highest first similarities as candidate categories, you can also set a similarity threshold first, and after calculating the first similarity between the target image features and the category image features of each fine-grained category in the feature library, select the fine-grained category corresponding to the first similarity greater than the similarity threshold as the candidate category.

[0158] In the embodiment of the present application, the visual modality model in the multimodal large language model is called to visually encode the example images of multiple fine-grained categories to build a feature library; at the same time, the visual modality model is used to visually encode the image to be identified to obtain the target image features. By comparing the target image features with the category image features in the feature library, self-enhanced retrieval in the visual modality is achieved to obtain N candidate categories that guide fine-grained image recognition reasoning, giving full play to the expert knowledge of the multimodal large language model in the visual modality, while improving the accuracy of screening candidate categories from multiple fine-grained categories.

[0159] Considering that it is difficult to obtain example images of fine-grained categories in some scenarios, there are very few or even no example images of fine-grained categories. In this case, it is difficult to construct a feature library by extracting category image features of example images of fine-grained categories.

[0160] In view of this, in some embodiments, the general category features of the fine-grained categories contained in the feature library are: category text features of the fine-grained categories, that is, the category description text of each fine-grained category in the fine-grained category set is text-encoded to obtain the corresponding category text features.

[0161] Specifically, the category description text of the fine-grained category can be the category name of the fine-grained category. For example, the category description texts of multiple fine-grained categories related to flowers are: peony, rose, etc. In addition, the category description text of the fine-grained category can also include other auxiliary description information. For example, the category description texts of multiple fine-grained categories related to flowers are: peony (Paeonia genus of Ranunculaceae), rose (Rosa genus of rose), etc.

[0162] The category description text of the fine-grained category is converted into a low-dimensional vector representation (Embeding) through the text encoder in the multimodal large language model to obtain the corresponding category text features. In the actual encoding process, each text content in the category description text can be encoded separately to obtain the text encoding features of each text content, and the obtained text encoding features can be spliced ​​together to obtain the final category text features. It is also possible to splice the various text contents of the category description text according to a certain format and then perform overall text encoding to obtain the category text features. Among them, the text encoding feature can also be called text representation, text representation or text representation vector, etc.

[0163] At this time, the embodiment of the present application adopts at least the following methods to perform visual modality retrieval:

[0164] Determine the second similarities between the target image feature and the category text feature of each fine-grained category; sort the obtained multiple second similarities in order from large to small, and use the fine-grained categories corresponding to the second similarities ranked in the top N as candidate categories.

[0165] Specifically, the second similarity between the target image features and the category text features of each fine-grained category is calculated through the pre-trained visual language model. The visual language model refers to a multimodal model that can learn from images and texts at the same time, such as the CLIP model.

[0166] After calculating the second similarities between the target image features and the category text features of each fine-grained category in the feature library, the multiple second similarities obtained are sorted in order from large to small, and the fine-grained categories corresponding to the top N second similarities are taken as candidate categories, where N can be set according to actual conditions.

[0167] It should be noted that in addition to the implementation method of selecting the N fine-grained categories with the highest second similarities as candidate categories, a similarity threshold can also be set first. After calculating the second similarity between the target image features and the category text features of each fine-grained category in the feature library, the fine-grained category corresponding to the second similarity greater than the similarity threshold is selected as the candidate category.

[0168] In an embodiment of the present application, in a scenario where it is difficult to obtain example images of fine-grained categories, it is also possible to construct a feature library for visual modal retrieval by extracting category text features of category description text of fine-grained categories, and then compare the target image features of the image to be identified with the category text features in the feature library to retrieve N candidate categories from the fine-grained category set, thereby expanding the usage scenarios of visual modal retrieval.

[0169] Step 403: extracting attribute information of each of the N candidate categories under the target attribute based on the category description text of each of the N candidate categories.

[0170] Specifically, the category description text of the candidate category may be the category name of the candidate category. In addition, the category description text of the candidate category may also include other auxiliary description information.

[0171] The target attribute is: an attribute used to distinguish N candidate categories. In this application, the target attribute can be one or more. The attribute information of the candidate category under the target attribute refers to: the specific content of the candidate category under the target attribute. For example, when the candidate category is "peony" and the target attribute is "color", the attribute information of the candidate category under the target attribute is: "red".

[0172] When obtaining target attributes, the embodiments of the present application provide at least the following methods:

[0173] In the first implementation mode, target attributes are selected by accessing an external knowledge base.

[0174] Specifically, the target attribute for distinguishing the N candidate categories is directly obtained by accessing the external knowledge base. When the external knowledge base does not directly store the target attribute for distinguishing the N candidate categories, but only stores the expert knowledge associated with the target attribute for distinguishing the N candidate categories, the external knowledge base is accessed to obtain the expert knowledge, and then the expert knowledge is summarized and summarized by the text modality model in the multimodal large language model to obtain the target attribute for distinguishing the N candidate categories.

[0175] In the second implementation mode, the second prompt text is used as the prompt instruction, and the text modal model is input for reasoning to obtain a basic attribute set for distinguishing different fine-grained categories; then the target attribute is selected from the basic attribute set, wherein the second prompt text includes at least one of the following category description texts: category description text of the same category to which the fine-grained category set belongs, and category description text of multiple fine-grained categories in the fine-grained category set.

[0176] Specifically, the same category to which the fine-grained category set belongs is the basic category described above. The category description text of the basic category may be the category name of the basic category. In addition, the category description text of the basic category may also include other auxiliary description information.

[0177] In addition, the multiple fine-grained categories in the above-mentioned fine-grained category set can be: all fine-grained categories in the fine-grained category set or selected parts of the fine-grained categories. Similarly, the category description text of the fine-grained category can be the category name of the fine-grained category. In addition, the category description text of the fine-grained category can also include other auxiliary description information.

[0178] The second prompt text may be generated according to a preset prompt content template and based on the category description text of the basic category or the category description texts of multiple fine-grained categories in the fine-grained category set; the second prompt text may also be generated according to a preset prompt content template and based on the category description text of the basic category and the category description texts of multiple fine-grained categories in the fine-grained category set.

[0179] The prompt content template is a template type that can be recognized by the text modal model, and is used to help the text modal model understand the semantic information of each part of the content. The prompt content template presets the position of each content and the order between them, and the corresponding second prompt text can be generated according to the prompt content template. In actual applications, in addition to the category description text described above, the second prompt text can also contain other auxiliary content.

[0180] The second prompt text is used as a prompt instruction and input into the text modal model in the multimodal large language model for reasoning. The text modal model is based on the prompt learning method and predicts the subsequent content of the prompt instruction according to the input prompt instruction, that is, a basic attribute set used to distinguish different fine-grained categories under the basic category, wherein the different fine-grained categories may be all or part of the fine-grained categories in the fine-grained category set; in addition, the different fine-grained categories may also include other fine-grained categories under the basic category outside the fine-grained category set. Of course, different fine-grained categories may also be other fine-grained categories under the basic category outside the fine-grained category set, and this application does not make specific limitations on this.

[0181] The basic attribute set can be output as an attribute list or other data structure. All or part of the basic attributes are selected from the basic attribute set as target attributes.

[0182] For example, see Fig. 9 ,In the fine-grained image recognition scenario of realizing the basic category (flower), the category description texts of multiple fine-grained categories in the ,fine-grained category set are : peony, rose, peony, and crabapple.

[0183] According to the preset prompt content template, the following prompt text can be constructed: "In a picture of a flower, which attributes can distinguish the types of flowers (for example, peony, rose, peony, crabapple)? Please list the 3 most distinguishing attributes."

[0184] Using the above prompt text as the prompt instruction, the text modality model in the trained multimodal large language model is input for reasoning. The text modality model outputs the following results in the form of an attribute list:

[0185] 1. Petal shape

[0186] 2. Flower Color

[0187] 3. The shape and arrangement of stamens and pistils”

[0188] Subsequently, the three basic attributes in the attribute list can be used as target attributes for distinguishing N candidate categories.

[0189] In some embodiments, when the second prompt text is used as a prompt instruction and input into a text modality model in a trained multimodal large language model for reasoning, the text modality model performs the following steps:

[0190] Based on the context information contained in the second prompt text, the second prompt text is subjected to text intent recognition to obtain a second recognition result, and the second recognition result indicates: a second intent tendency operation for distinguishing different fine-grained categories. By performing the second intent tendency operation, a basic attribute set for distinguishing different fine-grained categories is obtained.

[0191] Specifically, the second intention tendency operation refers to: obtaining a basic attribute set for distinguishing different fine-grained categories under the basic category. After the text modality model determines the above intention recognition result of the second prompt text through text intent recognition, it can directly predict the basic attribute set for distinguishing different fine-grained categories under the basic category based on the expert knowledge obtained by pre-training; it can also access an external knowledge base to obtain a basic attribute set for distinguishing different fine-grained categories under the basic category. The basic attribute set can be output as an attribute list or other data structure.

[0192] In the third implementation mode, a third prompt text is generated based on the category description texts of the N candidate categories. The third prompt text is used as a prompt instruction and input into a text modality model for reasoning to obtain at least one target attribute for distinguishing the N candidate categories.

[0193] That is to say, the category description texts of the N candidate categories retrieved by visual modality are directly used to construct the third prompt text. The third prompt text is used as a prompt instruction and input into the text modality model in the trained multimodal large language model for reasoning. The text modality model performs text intent recognition on the third prompt text based on the context information contained in the third prompt text, and directly obtains at least one target attribute for distinguishing the N candidate categories based on the intent recognition result.

[0194] For example, see Fig.10 In the fine-grained image recognition scenario of the basic category (flower), the category description texts of multiple fine-grained categories in the fine-grained category set are: peony, rose, peony, and crabapple. Two candidate categories are selected from the multiple fine-grained categories, namely peony and rose.

[0195] According to the preset prompt content template, the following prompt text can be constructed: "In a picture of a flower, which attributes can distinguish the type of flower (for example, peony, rose)? Please list the 3 most distinguishing attributes."

[0196] The above prompt text is used as the prompt instruction and input into the text modality model in the trained multimodal large language model for inference. The text modality model outputs the following three basic attributes in the form of an attribute list:

[0197] 1. Petal shape

[0198] 2. Flower Color

[0199] 3. The shape and arrangement of stamens and pistils”

[0200] In the embodiment of the present application, by calling the text modality model in the multimodal large language model and adopting a dialogue method, the target attributes for distinguishing N candidate categories are obtained, which fully utilizes the expert knowledge of the multimodal large language model in the text modality and improves the accuracy of the distinguishing attributes of the candidate categories.

[0201] In some embodiments, after obtaining the target attribute for distinguishing the N candidate categories, a first prompt text is generated based on the category description text and the target attribute of each of the N candidate categories. The first prompt text is used as a prompt instruction, and the text modality model in the trained multimodal large language model is input for reasoning to obtain the attribute information of each of the N candidate categories under the target attribute.

[0202] Specifically, the first prompt text is generated according to the preset prompt content template based on the category description texts and target attributes of the N candidate categories. In practical applications, the first prompt text may include other auxiliary content in addition to the category description texts and target attributes of the N candidate categories.

[0203] The first prompt text is used as the prompt instruction and input into the text modality model in the multimodal large language model for reasoning. The text modality model is based on the prompt learning method and predicts the subsequent content of the prompt instruction according to the input prompt instruction, that is, the attribute information of each of the N candidate categories under the target attribute.

[0204] In an embodiment of the present application, by calling the text modal model in the multimodal large language model and adopting a multi-round dialogue method, the target attribute for distinguishing N candidate categories and the attribute information under the target attribute are obtained. In this process, only the text modal model inside the multimodal large language model is called, and no external model is used, thereby avoiding dependence on the external model. This not only gives full play to the expert knowledge of the multimodal large language model in the text modality, but also avoids the time consumption caused by calling the external model, thereby improving the efficiency of model reasoning.

[0205] In some embodiments, when the first prompt text is used as a prompt instruction and input into a text modality model in a trained multimodal large language model for reasoning, the text modality model performs the following steps:

[0206] Based on the context information contained in the first prompt text, the first prompt text is subjected to text intent recognition to obtain a first recognition result; the first recognition result indicates: a first intention tendency operation for the target attribute of each of the N candidate categories. By performing the first intention tendency operation, attribute information of each of the N candidate categories under the target attribute is obtained from the corpus.

[0207] Specifically, the first intention tendency operation for the target attribute of each of the N candidate categories refers to: extracting the attribute information of each of the N candidate categories under the target attribute. After the text modality model determines the above intention recognition result of the second prompt text through text intent recognition, it can query and obtain matching attribute information from the corpus based on the category description text and target attribute of the N candidate categories.

[0208] For example, see Fig.11 In the fine-grained image recognition scenario of the basic category (flower), the category description texts of multiple fine-grained categories in the fine-grained category set are: peony, rose, peony, and crabapple. The candidate categories are: peony and rose. The target attributes that can distinguish the two candidate categories include: petal shape, flower color, and the shape and arrangement of stamens and pistils.

[0209] According to the preset prompt content template, the following prompt text can be constructed: "Please output the attribute information that can distinguish the attributes of peony and rose in the image according to the following attribute list.

[0210] Attribute list:

[0211] 1. Petal shape: XXX

[0212] 2. Flower color: XXX

[0213] 3. The shape and arrangement of stamens and pistils: XXX"

[0214] Using the above prompt text as the prompt instruction, the text modality model in the trained multimodal large language model is input for reasoning. The text modality model outputs the following results:

[0215] "Attributes of peony: 1. Petal shape: obovate. 2. Flower color: reddish purple. 3. Shape and arrangement of stamens and pistils: The pistil is located in the center of the flower, shaped like an old garlic, with five to seven petals; the stamens surround the pistil and are arranged neatly when it first blooms.

[0216] Attributes of roses: 1. Petal shape: obovate with sharp serrations on the edges. 2. Flower color: red. 3. Shape and arrangement of stamens and pistils: stamens usually appear as small particles on the filaments, while pistils have stigmas and ovaries. "

[0217] In an embodiment of the present application, the text intent of the input prompt instruction is recognized through the text modal model in the multimodal large language model; then, according to the intention recognition result, combined with the expert knowledge of the multimodal large language model in the text modality, the corresponding intention tendency operation is performed to obtain the target attributes for distinguishing N candidate categories and the attribute information of the target attributes, without calling the external knowledge base to obtain the attribute information, thereby improving the efficiency of model reasoning.

[0218] Step 404 , based on the image to be identified, the category description texts of the N candidate categories, and the attribute information of the N candidate categories under the target attribute, the target category of the image to be identified is obtained from the N candidate categories.

[0219] Specifically, the image to be identified is visual modal information, the category description text of each of the N candidate categories and the attribute information of each of the N candidate categories under the target attribute are all textual modal information, wherein the category description text of the N candidate categories is the retrieval result obtained by calling the visual modal model in the multimodal large language model for visual modal retrieval, and the attribute information of each of the N candidate categories under the target attribute is the retrieval result obtained by calling the text modal model in the multimodal large language model for textual modal retrieval.

[0220] By integrating the image to be recognized, the retrieval results obtained by visual modal retrieval, and the retrieval results obtained by textual modal retrieval, and reasoning with a multimodal large language model that can perform model inference on multimodal information, the target category of the image to be recognized is obtained from N candidate categories as the final fine-grained image recognition result.

[0221] In an embodiment of the present application, single-modality self-enhanced retrieval is performed in the visual modality and the textual modality respectively, and the obtained retrieval results of the visual modality (i.e., the category description texts of the N candidate categories) and the retrieval results of the textual modality (i.e., the attribute information of the N candidate categories in the target attributes) are used as enhanced information and added to the reasoning process of fine-grained image recognition, which can effectively improve the understanding of the distinguishing features between fine-grained categories, thereby improving the accuracy of fine-grained image recognition.

[0222] Secondly, the image to be identified is visual modal information, and the retrieval results of the visual modality and the retrieval results of the text modality are both text modal information. This application uses different modalities to express information in different forms, which makes the expression of information richer. Therefore, when combining multiple modal information (visual modal information and text modal information) for fine-grained image recognition, it helps to improve the accuracy of understanding fine-grained images, thereby obtaining more accurate fine-grained image recognition results, thereby improving the accuracy of fine-grained image recognition.

[0223] In addition, the visual modality model in the multimodal large language model is called for visual modality retrieval; the text modality model in the multimodal large language model is called for text modality retrieval. Finally, the image to be recognized, the retrieval results obtained by the visual modality retrieval, and the retrieval results obtained by the text modality retrieval are integrated, and the multimodal large language model is used for reasoning to obtain the final fine-grained image recognition result. The entire fine-grained image recognition process only uses the multimodal large language model, and no additional external models are used, avoiding dependence on external models and eliminating the need for information exchange with external models, thereby effectively improving the efficiency of fine-grained image recognition.

[0224] In some embodiments, according to a preset prompt content template, based on the image to be identified, the category description text of each of the N candidate categories, and the attribute information of each of the N candidate categories under the target attribute, the target prompt content is generated. The target prompt content is used as a prompt instruction, and the multimodal large language model is input for reasoning to obtain the target category of the image to be identified.

[0225] Specifically, according to the preset prompt content template, based on the image to be identified, the category description text of each of the N candidate categories, and the attribute information of each of the N candidate categories under the target attribute, the target prompt content is generated. The target prompt content includes both visual modal information and text modal information. In practical applications, the text modal information in the target prompt content may include other auxiliary content in addition to the category description text of each of the N candidate categories and the attribute information of each of the N candidate categories under the target attribute.

[0226] The target prompt content is used as the prompt instruction and input into the trained multimodal large language model for inference. The multimodal large language model is based on the prompt learning method and predicts the subsequent content of the prompt instruction according to the input prompt instruction, that is, the category description text of the target category of the image to be recognized.

[0227] For example, see Fig.12 ,In the fine-grained image recognition scenario of realizing the basic category (flower), the candidate categories are: peony and rose; Fig.11 Based on the obtained attribute information of the target attributes that can distinguish the two candidate categories, combined with the category names of the two candidate categories and the image to be identified, the target prompt content can be constructed according to the preset prompt content template.

[0228] The above target prompt content is used as the prompt instruction and input into the multimodal large language model for inference. The multimodal large language model outputs the target category corresponding to the following image to be identified: "Peony"

[0229] In the embodiments of the present application, different modalities are used to express information in different forms, which enriches the way of expressing information. Therefore, when combining information of multiple modalities (image to be identified and text modal information) for fine-grained image recognition, it helps to improve the accuracy of understanding fine-grained images, thereby obtaining more accurate fine-grained image recognition results and improving the accuracy of fine-grained image recognition.

[0230] In some embodiments, see Fig.13 , when the target prompt content is used as the prompt instruction and input into the multimodal large language model for reasoning, the multimodal large language model performs the following steps:

[0231] Step 1301, text encoding is performed on the text content in the target prompt content to obtain text modality features.

[0232] Specifically, the text content in the target prompt content includes: category description texts of each of the N candidate categories and attribute information of each of the N candidate categories under the target attribute.

[0233] Call the text encoder in the multimodal large language model to perform text encoding on the text content in the target prompt content. In the actual encoding process, the category description texts of the N candidate categories and the attribute information of each of the N candidate categories under the target attribute can be encoded separately to obtain the corresponding text modal features. Then the obtained text modal features are concatenated to obtain the final text modal features. Alternatively, the category description texts of the N candidate categories and the attribute information of each of the N candidate categories under the target attribute can be concatenated in a certain format and then the overall text encoding can be performed to obtain the text modal features.

[0234] Step 1302: Visually encode the image to be identified in the target prompt content to obtain target image features.

[0235] The visual encoder in the multimodal large language model is called to perform visual encoding on the image to be recognized to obtain the target image features. The specific encoding process has been introduced in the previous article and will not be repeated here.

[0236] Step 1303, fuse the text modality features and the target image features to obtain fused coding features.

[0237] Step 1304, performing multiple rounds of iterative text prediction based on the fused coding features to obtain the target category of the image to be recognized, wherein, in a round of text prediction, the output result of the current round of text prediction process is obtained based on the fused coding features and the output result of the previous round.

[0238] In the embodiments of the present application, the fused encoding feature can be understood as a feature sequence composed of a token sequence. The fused encoding feature can be used as prompt information to generate a subsequent text sequence, which is the category description text for characterizing the target category of the image to be recognized. The process of iterative text prediction is a process of decoding the fused encoding feature, so a decoder can be used to implement it, such as the decoder of Transformer.

[0239] In a possible implementation, multiple text predictions can be performed in an autoregressive manner. That is, in one text prediction, the output result of the current text prediction process is obtained based on the fused encoding feature and the output result of the previous text prediction process. In the first text prediction, the prediction is directly made based on the fused encoding feature. After one text prediction is completed, the prediction result is concatenated after the input feature used in this time for the next text prediction until the iterative stop condition is met.

[0240] See Fig.14 As shown in . Here is an example diagram of iterative text prediction, where the fused encoding feature can be represented as a token sequence . Through the multimodal large language model, the first iterative prediction is performed, that is, the next token is generated according to the token sequence . Adding it to the generated token sequence, a new token sequence can be obtained. Then, the multimodal large language model will perform the second iterative prediction according to the generated token sequence , generate the next token , and add it to the generated token sequence, then a new token sequence

[0241] The iterative prediction termination conditions in the embodiments of the present application include at least one of the following conditions:

[0242] (1) When the length of the predicted token sequence reaches its upper limit value, the iterative prediction stops.

[0243] (2) When the terminator is decoded or predicted, that is, after the character representing the end of the sequence above, the iterative prediction ends. The terminator represents the end of a sequence.

[0244] For example, in the fine-grained image recognition scenario of the flower shown in Fig.12 , the next token obtained in the first iterative prediction is "mu", and the next token obtained in the second iterative prediction The terminator is obtained during the third iterative prediction, and the iterative prediction ends, outputting the category description text "Peony" of the target category of the image to be identified.

[0245] In the embodiment of the present application, during the fine-grained image recognition process, the visual modality model in the multimodal large language model is called to perform visual modality retrieval; the text modality model in the multimodal large language model is called to perform text modality retrieval. Then, the retrieval results obtained by the self-enhanced retrieval of the single expert model inside the multimodal large language model are integrated with the image to be recognized, and input into the multimodal large language model for reasoning to obtain the final fine-grained image recognition result, which can effectively enhance the multimodal large language model's understanding of the differences between different fine-grained categories, thereby improving the accuracy of the fine-grained image recognition results obtained by reasoning.

[0246] In order to better explain the embodiment of the present application, the following introduces a fine-grained image recognition method provided by the embodiment of the present application in combination with the fine-grained image recognition scenario of the basic category (flower). The process of the method can be as follows: Figure 1 The terminal device 101 shown in the figure may also be executed by the server 102, or the terminal device 101 and the server 102 may interact to execute the following steps: Fig.15 As shown, it includes the model splitting stage, visual modality retrieval stage, text modality retrieval stage and fine-grained reasoning stage.

[0247] 1. In the model splitting stage: according to the multimodal large language model Its inherent ability to transform multimodal large language models Split into visual modality models and text modality models These two single-modality expert models. Visual modality model Includes visual encoder and feature library. Text modality model Includes text encoders and large language models.

[0248] 2. In the visual modality retrieval stage: through the visual modality model The visual encoder in the example image set for each example image Perform visual encoding to obtain the corresponding sample image feature f, as shown in the following formula (1):

[0249]

[0250] A clustering algorithm is used to cluster the obtained sample image feature set (i.e., multiple sample image features) to obtain image feature clusters corresponding to multiple fine-grained categories. The feature clustering center of the image feature cluster corresponding to each fine-grained category is used as the category image feature of each fine-grained category. Multiple category image features constitute a feature library. , the above process can be expressed as the following formula (2):

[0251]

[0252] Clustering is a clustering algorithm. .

[0253] After obtaining the feature library, the image to be recognized is extracted through the visual encoder in the visual modality model. Corresponding target image features , and then calculate the target image features The cosine similarity between each category image feature in the feature library is calculated, and the obtained cosine similarities are sorted in descending order. The fine-grained categories of the category image features corresponding to the cosine similarities ranked in the top N are taken as candidate categories, and the category names of each candidate category are taken as the retrieval results R of the visual modality. V Output, the above process can be expressed as the following formula (3):

[0254]

[0255] Retrievalg is a visual retrieval algorithm.

[0256] For example, in Fig.16 The visual modality retrieval stage 1601 shown in the figure uses the feature library to retrieve the image to be identified. The three most similar categories are: 1. Artichoke; 2. Watercress; 3. Passion fruit.

[0257] 3. In the text modal retrieval stage:

[0258] Generate prompt text based on the preset prompt content template and the category names L of multiple fine-grained categories in the fine-grained category set . With prompt text As a prompt, enter the text modal model Perform reasoning to obtain the attribute list A used to distinguish multiple fine-grained categories. The above process can be expressed as the following formula (4):

[0259]

[0260] Next, according to the preset prompt content template, the search results R based on the visual modality Vand attribute list A (that is, all basic attributes in attribute list A are used as target attributes) to generate prompt text . With prompt text As a prompt, enter the text modal model Perform reasoning to obtain the attribute information of each of the N candidate categories under each basic attribute in the attribute list A, and use the obtained attribute information as the retrieval result R of the text modality L Output, the above process can be expressed as the following formula (5):

[0261]

[0262] For example, in Fig.16 The text modality retrieval stage 1602 shown, the prompt text constructed by the first round of dialogue The question is: "In a picture of a flower, what attributes can distinguish the types of flowers (e.g., gladiolus, artichoke, watercress, watercress, passionflower, etc.)? Please list the 3 most discriminative attributes."

[0263] With prompt text As a prompt, enter the text modal model The output result (i.e., attribute list A) obtained by reasoning is:

[0264] 1. Petal shape

[0265] 2. Flower Color

[0266] 3. The shape and arrangement of stamens and pistils”.

[0267] Prompt text for the second round of dialogue construction For example: "Please output the attribute information that can distinguish artichoke, watercress, and passion fruit in the image according to the following attribute list.

[0268] Attribute list:

[0269] 1. Petal shape: XXX

[0270] 2. Flower color: XXX

[0271] 3. The shape and arrangement of stamens and pistils: XXX"

[0272] With prompt text As a prompt, enter the text modal model Perform reasoning and obtain the output result (i.e., the retrieval result R of the text modality) L )for:

[0273] “Artichoke properties: 1. Petal shape: oval, leaf-like. 2. Flower color: from white to pink. 3. Stamen and pistil shape and arrangement: Most stamens have long filaments, and pistils are solitary.

[0274] Attributes of Watercress: 1. Petal shape: elongated and pointed. 2. Flower color: from white to pink or purple. 3. Shape and arrangement of stamens and pistils: stamens are numerous and have long filaments.

[0275] Passionflower attributes: 1. Petal shape: round. 2. Flower color: bright and colorful. 3. Shape and arrangement of stamens and pistils: not obvious".

[0276] 4. In the fine-grained reasoning stage:

[0277] According to the preset prompt content template, the search results R of the visual modality are V , retrieval results of text mode R L 、Image to be recognized Integrate and get the target prompt content . Use target to prompt content As a prompt, input the multimodal large language model Perform inference and output the image to be recognized The target category name C, the above process can be expressed as the following formula (6):

[0278]

[0279] For example, in Fig.16 The text modality retrieval stage 1603 shown, the target prompt content constructed by the third round of dialogue Includes: reference information and images to be identified , where the reference information includes: the retrieval result R of the visual modality V , retrieval results of text mode R L In addition, the target prompt content It also includes text content to assist in the question and answer process: "Based on the picture and reference information, answer which specific category the flower in this picture belongs to? Choose one from artichoke, watercress, and passion fruit."

[0280] Prompt content with goals As a prompt, input the multimodal large language model After reasoning, the output result is: Watercress.

[0281] In an embodiment of the present application, single-modality self-enhanced retrieval is performed in the visual modality and the textual modality respectively, and the obtained retrieval results of the visual modality (i.e., the category description texts of the N candidate categories) and the retrieval results of the textual modality (i.e., the attribute information of the N candidate categories in the target attributes) are used as enhanced information and added to the reasoning process of fine-grained image recognition, which can effectively improve the understanding of the distinguishing features between fine-grained categories, thereby improving the accuracy of fine-grained image recognition.

[0282] Secondly, the image to be identified is visual modal information, and the retrieval results of the visual modality and the retrieval results of the text modality are both text modal information. This application uses different modalities to express information in different forms, which makes the expression of information richer. Therefore, when combining information from multiple modalities for fine-grained image recognition, it helps to improve the accuracy of understanding fine-grained images, thereby obtaining more accurate fine-grained image recognition results and improving the accuracy of fine-grained image recognition.

[0283] In addition, the visual modality model in the multimodal large language model is called for visual modality retrieval; the text modality model in the multimodal large language model is called for text modality retrieval. Finally, the image to be recognized, the retrieval results obtained by the visual modality retrieval, and the retrieval results obtained by the text modality retrieval are integrated, and the multimodal large language model is used for reasoning to obtain the final fine-grained image recognition result. The entire fine-grained image recognition process only uses the multimodal large language model, and no additional external models are used, avoiding dependence on external models and eliminating the need for information exchange with external models, thereby effectively improving the efficiency of fine-grained image recognition.

[0284] In order to verify the performance of the image recognition method of the present application in fine-grained image recognition compared with other related technologies, the accuracy (ACC) of the image recognition method of the present application is compared with the accuracy (ACC) of other technical solutions, as shown in the following Table 1:

[0285]

[0286] Table 1 shows the improvement of the technical solution of this application in fine-grained image recognition test compared with other previous technical solutions (CLIP-KMeans, BLIP-2, CaSED, FineR), and the test sets involved include Bird-200, Car-196, Dog-120, and FlowerF-102. As can be seen from Table 1, the image recognition method of this application has surpassed other previous technical solutions on most test sets and has achieved a significant lead in average performance, reflecting the better performance and stronger generalization of this application in fine-grained image recognition.

[0287] In addition, the image recognition method in this application can be migrated to a variety of multimodal large language models for fine-grained image recognition. The following compares the accuracy (ACC) of the method of this application and the previous method when they are migrated to a variety of multimodal large language models (LLaVA-7B, InstructBLIP, mPLUG-OWL, Qwen-VL). The test sets involved include FlowerF-102, Oxfordpets, and CUB-200, as shown in Table 2 below:

[0288]

[0289] It can be seen from Table 2 that the image recognition method in the present application is accurate in fine-grained image recognition, and the image recognition method in the present application can be migrated to a variety of multimodal large language models, which brings certain performance improvements, reflecting the versatility of the method of the present application.

[0290] Based on the same technical concept, the present application embodiment provides a structural schematic diagram of an image recognition device, such as Fig.17 As shown, the image recognition device 1700 includes:

[0291] The visual retrieval module 1701 is used to perform visual encoding on the image to be identified to obtain target image features; based on the target image features, obtain N candidate categories from a fine-grained category set, where the fine-grained category set is a set of multiple subcategories of the same category, where N is greater than 1;

[0292] The text retrieval module 1702 is used to extract attribute information of each of the N candidate categories under a target attribute based on the category description text of each of the N candidate categories; wherein the target attribute is: an attribute used to distinguish the N candidate categories;

[0293] The recognition module 1703 is used to obtain the target category of the image to be recognized from the N candidate categories based on the image to be recognized, the category description texts of the respective N candidate categories and the attribute information of the respective N candidate categories under the target attribute.

[0294] Optionally, the text retrieval module 1702 is specifically used for:

[0295] Generate a first prompt text based on the category description texts of the N candidate categories and the target attribute;

[0296] The first prompt text is used as a prompt instruction, and a text modality model in a trained multimodal large language model is input for reasoning to obtain attribute information of each of the N candidate categories under the target attribute.

[0297] Optionally, the text retrieval module 1702 is specifically used for:

[0298] Through the text modality model, the following steps are performed:

[0299] Based on the context information contained in the first prompt text, the first prompt text is subjected to text intent recognition to obtain a first recognition result; the first recognition result indicates: a first intention tendency operation for the target attribute of each of the N candidate categories;

[0300] By performing the first intention tendency operation, attribute information of each of the N candidate categories under the target attribute is obtained from the corpus.

[0301] Optionally, the text retrieval module 1702 is specifically used for:

[0302] Selecting the target attribute by accessing an external knowledge base;

[0303] or,

[0304] The second prompt text is used as a prompt instruction, and the text modal model is input for reasoning to obtain a basic attribute set for distinguishing different fine-grained categories, wherein the second prompt text includes at least one of the following category description texts: category description text of the same category to which the fine-grained category set belongs, and category description texts of multiple fine-grained categories in the fine-grained category set;

[0305] The target attribute is selected from the basic attribute set.

[0306] Optionally, the text retrieval module 1702 is specifically used for:

[0307] Through the text modality model, the following steps are performed:

[0308] Based on the context information contained in the second prompt text, the second prompt text is subjected to text intent recognition to obtain a second recognition result; the second recognition result indicates: a second intention tendency operation for distinguishing different fine-grained categories;

[0309] By performing the second intention tendency operation, a basic attribute set for distinguishing different fine-grained categories is obtained.

[0310] Optionally, the visual retrieval module 1701 is specifically used for:

[0311] Obtaining a category image feature of each fine-grained category in the fine-grained category set, and determining a first similarity between the target image feature and the category image feature of each fine-grained category;

[0312] The obtained multiple first similarities are sorted in descending order, and the fine-grained categories corresponding to the first N similarities are taken as candidate categories.

[0313] Optionally, the visual retrieval module 1701 is specifically used for:

[0314] Acquire a set of example images corresponding to the multiple fine-grained categories, each fine-grained category corresponding to at least one example image in the set of example images;

[0315] Performing visual encoding on each example image in the example image set to obtain corresponding example image features;

[0316] Clustering the obtained multiple example image features to obtain image feature clusters corresponding to the multiple fine-grained categories, wherein one image feature cluster includes: at least one example image feature, and the at least one example image feature includes: a feature clustering center of the one image feature cluster;

[0317] The feature clustering center of the image feature cluster corresponding to each fine-grained category is used as the category image feature of each fine-grained category.

[0318] Optionally, the visual retrieval module 1701 is specifically used for:

[0319] Performing text encoding on the category description text of each fine-grained category in the fine-grained category set to obtain corresponding category text features;

[0320] Determining a second similarity between the target image feature and the category text feature of each fine-grained category;

[0321] The obtained multiple second similarities are sorted in descending order, and the fine-grained categories corresponding to the second similarities ranked in the top N positions are taken as candidate categories.

[0322] Optionally, the identification module 1703 is specifically used for:

[0323] Generate target prompt content according to a preset prompt content template based on the image to be identified, the category description text of each of the N candidate categories, and the attribute information of each of the N candidate categories under the target attribute;

[0324] The target prompt content is used as a prompt instruction and input into the multimodal large language model for reasoning to obtain the target category of the image to be recognized.

[0325] Optionally, the identification module 1703 is specifically used for:

[0326] The following steps are performed using the multimodal large language model:

[0327] Performing text encoding on the text content in the target prompt content to obtain text modality features, wherein the text content includes: category description texts of the N candidate categories and attribute information of the N candidate categories under the target attribute;

[0328] Visually encoding the image to be identified in the target prompt content to obtain target image features;

[0329] Fusing the text modality feature and the target image feature to obtain a fused coding feature;

[0330] Multiple rounds of iterative text prediction are performed based on the fused coding features to obtain the target category of the image to be recognized, wherein, in a round of text prediction, the output result of this round of text prediction process is obtained based on the fused coding features and the output result of the previous round.

[0331] In an embodiment of the present application, single-modality self-enhanced retrieval is performed in the visual modality and the textual modality respectively, and the obtained retrieval results of the visual modality (i.e., the category description texts of the N candidate categories) and the retrieval results of the textual modality (i.e., the attribute information of the N candidate categories in the target attributes) are used as enhanced information and added to the reasoning process of fine-grained image recognition, which can effectively improve the understanding of the distinguishing features between fine-grained categories, thereby improving the accuracy of fine-grained image recognition.

[0332] Secondly, the image to be identified is visual modal information, and the retrieval results of the visual modality and the retrieval results of the text modality are both text modal information, that is, this application uses different modalities to express information in different forms, which makes the expression of information richer. Therefore, when combining multiple modal information (visual modal information and text modal information) for fine-grained image recognition, it helps to improve the accuracy of understanding fine-grained images, thereby obtaining more accurate fine-grained image recognition results, thereby improving the accuracy of fine-grained image recognition.

[0333] In addition, the visual modality model in the multimodal large language model is called for visual modality retrieval; the text modality model in the multimodal large language model is called for text modality retrieval. Finally, the image to be recognized, the retrieval results obtained by the visual modality retrieval, and the retrieval results obtained by the text modality retrieval are integrated, and the multimodal large language model is used for reasoning to obtain the final fine-grained image recognition result. The entire fine-grained image recognition process only uses the multimodal large language model, and no additional external models are used, avoiding dependence on external models and eliminating the need for information exchange with external models, thereby effectively improving the efficiency of fine-grained image recognition.

[0334] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 1 The terminal device and / or server shown, such as Fig.18 As shown, it includes at least one processor 1801 and a memory 1802 connected to the at least one processor. The specific connection medium between the processor 1801 and the memory 1802 is not limited in the embodiment of the present application. Fig.18 For example, the processor 1801 and the memory 1802 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0335] In the embodiment of the present application, the memory 1802 stores instructions that can be executed by at least one processor 1801, and the at least one processor 1801 can perform the steps of the above-mentioned image recognition method by executing the instructions stored in the memory 1802.

[0336] Among them, the processor 1801 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and fine-grained image recognition can be achieved by running or executing instructions stored in the memory 1802 and calling data stored in the memory 1802. Optionally, the processor 1801 may include one or more processing units, and the processor 1801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1801. In some embodiments, the processor 1801 and the memory 1802 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0337] Processor 1801 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0338] The memory 1802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1802 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. The memory 1802 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer device, but is not limited thereto. The memory 1802 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, used to store program instructions and / or data.

[0339] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned image recognition method.

[0340] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned image recognition method.

[0341] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0342] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0343] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0344] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0345] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0346] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An image recognition method, characterized in that: include: Visually encode the image to be identified to obtain the target image features; Based on the target image feature, obtaining N candidate categories from a fine-grained category set, where the fine-grained category set is a set of multiple subcategories of the same category, where N is greater than 1; By using a text modality model in a trained multimodal large language model, based on the category description texts of the N candidate categories, attribute information of each of the N candidate categories under a target attribute is extracted; wherein the target attribute is: an attribute used to distinguish the N candidate categories; Through the multimodal large language model, based on the image to be identified, the category description texts of the N candidate categories and the attribute information of the N candidate categories under the target attribute, the target category of the image to be identified is obtained from the N candidate categories.

2. The method according to claim 1, characterized in that The extracting attribute information of each of the N candidate categories under the target attribute based on the category description texts of each of the N candidate categories by using the text modality model in the trained multimodal large language model includes: Generate a first prompt text based on the category description texts of the N candidate categories and the target attribute; The first prompt text is used as a prompt instruction, and a text modality model in a trained multimodal large language model is input for reasoning to obtain attribute information of each of the N candidate categories under the target attribute.

3. The method according to claim 2, characterized in that The method of using the first prompt text as a prompt instruction, inputting a text modality model in a trained multimodal large language model for reasoning, and obtaining attribute information of each of the N candidate categories under the target attribute includes: Through the text modality model, the following steps are performed: Based on the context information contained in the first prompt text, the first prompt text is subjected to text intent recognition to obtain a first recognition result; the first recognition result indicates: a first intention tendency operation for the target attribute of each of the N candidate categories; By performing the first intention tendency operation, attribute information of each of the N candidate categories under the target attribute is obtained from the corpus.

4. The method according to claim 2, characterized in that The target attributes are obtained in the following way: Selecting the target attribute by accessing an external knowledge base; or, The second prompt text is used as a prompt instruction, and the text modal model is input for reasoning to obtain a basic attribute set for distinguishing different fine-grained categories, wherein the second prompt text includes at least one of the following category description texts: category description text of the same category to which the fine-grained category set belongs, and category description texts of multiple fine-grained categories in the fine-grained category set; The target attribute is selected from the basic attribute set.

5. The method according to claim 4, characterized in that The second prompt text is used as a prompt instruction, and the text modality model is input for reasoning to obtain a basic attribute set for distinguishing different fine-grained categories, including: Through the text modality model, the following steps are performed: Based on the context information contained in the second prompt text, the second prompt text is subjected to text intent recognition to obtain a second recognition result; the second recognition result indicates: a second intention tendency operation for distinguishing different fine-grained categories; By performing the second intention tendency operation, a basic attribute set for distinguishing different fine-grained categories is obtained.

6. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining N candidate categories from a fine-grained category set based on the target image features includes: Obtaining a category image feature of each fine-grained category in the fine-grained category set, and determining a first similarity between the target image feature and the category image feature of each fine-grained category; The obtained multiple first similarities are sorted in descending order, and the fine-grained categories corresponding to the first N similarities are taken as candidate categories.

7. The method according to claim 6, characterized in that The obtaining of the category image feature of each fine-grained category in the fine-grained category set includes: Acquire a set of example images corresponding to a plurality of fine-grained categories in the set of fine-grained categories, wherein each fine-grained category corresponds to at least one example image in the set of example images; Performing visual encoding on each example image in the example image set to obtain corresponding example image features; Clustering the obtained multiple example image features to obtain image feature clusters corresponding to the multiple fine-grained categories, wherein one image feature cluster includes: at least one example image feature, and the at least one example image feature includes: a feature clustering center of the one image feature cluster; The feature clustering center of the image feature cluster corresponding to each fine-grained category is used as the category image feature of each fine-grained category.

8. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining N candidate categories from a fine-grained category set based on the target image features includes: Performing text encoding on the category description text of each fine-grained category in the fine-grained category set to obtain corresponding category text features; Determining a second similarity between the target image feature and the category text feature of each fine-grained category; The obtained multiple second similarities are sorted in descending order, and the fine-grained categories corresponding to the second similarities ranked in the top N positions are taken as candidate categories.

9. The method according to claim 2, characterized in that The step of obtaining the target category of the image to be identified from the N candidate categories based on the image to be identified, the category description texts of the N candidate categories, and the attribute information of the N candidate categories under the target attribute by using the multimodal large language model includes: Generate target prompt content according to a preset prompt content template based on the image to be identified, the category description text of each of the N candidate categories, and the attribute information of each of the N candidate categories under the target attribute; The target prompt content is used as a prompt instruction and input into the multimodal large language model for reasoning to obtain the target category of the image to be recognized.

10. The method according to claim 9, characterized in that The step of using the target prompt content as a prompt instruction, inputting the multimodal large language model for reasoning, and obtaining the target category of the image to be recognized includes: The following steps are performed using the multimodal large language model: Performing text encoding on the text content in the target prompt content to obtain text modality features, wherein the text content includes: category description texts of the N candidate categories and attribute information of the N candidate categories under the target attribute; Visually encoding the image to be identified in the target prompt content to obtain target image features; Fusing the text modality feature and the target image feature to obtain a fused coding feature; Multiple rounds of iterative text prediction are performed based on the fused coding features to obtain the target category of the image to be recognized, wherein, in a round of text prediction, the output result of this round of text prediction process is obtained based on the fused coding features and the output result of the previous round.

11. An image recognition device, characterized in that: include: The visual retrieval module is used to visually encode the image to be identified and obtain the target image features; Based on the target image feature, obtaining N candidate categories from a fine-grained category set, where the fine-grained category set is a set of multiple subcategories of the same category, where N is greater than 1; A text retrieval module is used to extract attribute information of each of the N candidate categories under a target attribute based on the category description text of each of the N candidate categories through a text modality model in a trained multimodal large language model; wherein the target attribute is an attribute used to distinguish the N candidate categories; A recognition module is used to obtain a target category of the image to be recognized from the N candidate categories through the multimodal large language model based on the image to be recognized, the category description text of each of the N candidate categories, and the attribute information of each of the N candidate categories under the target attribute.

12. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device, and when the computer program is run on the computer device, the computer device executes the steps of the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 10.

Images