Fine-grained image recognition method and device, computer equipment and storage medium

Abstract

The invention discloses a fine-grained image identification method and device, computer equipment and a storage medium, relates to the technical field of artificial intelligence, is applied to an identification scene of fine-grained images related to financial or medical services, and comprises the following steps: dividing a target fine-grained image into a plurality of image blocks, and inputting the plurality of image blocks into a visual Transform encoder, generating a first feature representation of the target fine-grained image; based on a target knowledge graph corresponding to the target fine-grained image, extracting an entity and a relationship related to the target fine-grained image, and inputting the entity and the relationship into a graph neural network encoder to generate a second feature representation of the target fine-grained image; on the basis of a cross attention mechanism, executing cross attention operation on the first feature representation and the second feature representation, and generating a fusion feature representation of the target fine-grained image; and based on the fusion feature representation, generating an image recognition result of the target fine-grained image by using a multi-layer perceptron. According to the invention, the accuracy of fine-grained image recognition can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and is applied to the recognition scenario of fine-grained images related to finance or medical services. In particular, it relates to a fine-grained image recognition method, device, computer equipment and non-volatile computer-readable storage medium. Background Art

[0002] At present, with the rapid development of computer vision technology, fine-grained image recognition (FGIR) has become a key research direction in this field. FGIR focuses on accurately classifying images of different subcategories under the same category. These subcategories may be very similar in appearance, but there are subtle differences in details. For example, different species of birds can be distinguished in bird recognition, and different models of cars can be distinguished in car recognition.

[0003] FGIR has a wide range of applications, covering many important areas such as healthcare, biodiversity monitoring, climate change assessment, smart retail and transportation systems. In the healthcare field, FGIR can help automatically identify and classify different cell types, thereby assisting in the diagnosis and treatment of diseases. In biodiversity monitoring, FGIR can assist scientists in classifying and studying different species. In addition, in smart retail and transportation systems, the application of FGIR technology can improve the accuracy of commodity identification and the level of intelligence in traffic management.

[0004] Although deep learning technology, especially convolutional neural networks (CNNs), has made significant progress in the field of FGIR, the technology still faces a series of challenges. One of the most important challenges is the problem of small differences between classes and large differences within classes. In many cases, the differences between images of different subcategories at the pixel level are very small, which requires the FGIR method to have extremely high sensitivity and discrimination ability; at the same time, images within the same subcategory may show large changes due to factors such as lighting, angle, occlusion, etc., which further increases the difficulty of recognition. That is, in the existing technology, the current fine-grained image recognition method still needs to be improved in accuracy.

[0005] In summary, how to provide a fine-grained image recognition method, apparatus, computer device and non-volatile computer-readable storage medium that can effectively improve the accuracy of fine-grained image recognition is a problem that needs to be urgently solved by those skilled in the art. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a method, apparatus, computer equipment and non-volatile computer-readable storage medium for recognizing fine-grained images that can be used in financial technology or other related fields, aiming to solve the problem of how to effectively improve the accuracy of fine-grained image recognition.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for recognizing fine-grained images, comprising:

[0009] Dividing a target fine-grained image into a plurality of image blocks, and inputting the plurality of image blocks into a visual Transformer encoder to generate a first feature representation of the target fine-grained image;

[0010] Based on the target knowledge graph corresponding to the target fine-grained image, extract entities and relationships related to the target fine-grained image, and input the entities and relationships into a graph neural network encoder to generate a second feature representation of the target fine-grained image;

[0011] Based on a cross-attention mechanism, performing a cross-attention operation on the first feature representation and the second feature representation to generate a fused feature representation of the target fine-grained image;

[0012] Based on the fused feature representation, a multi-layer perceptron is used to generate an image recognition result of the target fine-grained image.

[0013] In a further technical solution, the method for recognizing fine-grained images, wherein the target fine-grained image is divided into a plurality of image blocks, and the plurality of image blocks are input into a visual Transformer encoder to generate a first feature representation of the target fine-grained image, comprises:

[0014] Obtaining a target fine-grained image input by a target user;

[0015] Evenly dividing the target fine-grained image into a plurality of image blocks of the same size and without overlap;

[0016] After extracting and serializing the features of the multiple image blocks, a corresponding feature sequence is generated;

[0017] The feature sequence is input into a visual Transformer encoder to generate a first feature representation of the target fine-grained image.

[0018] In a further technical solution, the method for recognizing fine-grained images, wherein the target knowledge graph corresponding to the target fine-grained image is used to extract entities and relationships related to the target fine-grained image, and the entities and relationships are input into a graph neural network encoder to generate a second feature representation of the target fine-grained image, including:

[0019] Determine a corresponding target knowledge graph according to the image content of the target fine-grained image;

[0020] Analyzing the target knowledge graph, identifying and extracting entities and relationships in the target knowledge graph that are related to the target fine-grained image;

[0021] The entities and relations are input into a graph neural network encoder to generate a second feature representation of the target fine-grained image.

[0022] In a further technical solution, the fine-grained image recognition method, wherein the first feature representation includes a visual feature;

[0023] The step of determining the target knowledge graph corresponding to the target fine-grained image according to the image content of the target fine-grained image includes:

[0024] Searching in a knowledge graph library according to the visual features of the target fine-grained image;

[0025] According to the retrieval results, a target knowledge graph corresponding to the target fine-grained image is obtained.

[0026] In a further technical solution, the method for recognizing fine-grained images, wherein the cross-attention mechanism is based on which the first feature representation and the second feature representation are subjected to a cross-attention operation to generate a fused feature representation of the target fine-grained image, comprising:

[0027] Performing an adjustment operation on the first feature representation and the second feature representation to ensure that the two are consistent in format and dimension;

[0028] Using the first feature representation as a query and the second feature representation as a key and a value, calculating the similarity between the keys in the first feature representation and the second feature representation using a cross-attention mechanism, and generating an attention weight;

[0029] Based on the attention weights, weighted summation is performed on the values ​​in the second feature representation to generate a fused feature representation of the target fine-grained image.

[0030] In a further technical solution, the method for recognizing fine-grained images, wherein the adjusting operation is performed on the first feature representation and the second feature representation to ensure that the two are consistent in format and dimension, comprises:

[0031] Analyzing the formats of the first feature representation and the second feature representation, and adaptively adjusting the formats of the two according to the format analysis result to ensure that the first feature representation and the second feature representation are consistent in format;

[0032] Analyze the dimensions of the first feature representation and the second feature representation, and adaptively adjust the dimensions of the two according to the dimension analysis result to ensure that the first feature representation and the second feature representation are consistent in dimension.

[0033] In a further technical solution, the method for recognizing fine-grained images, wherein the image recognition result of the target fine-grained image is generated by using a multilayer perceptron based on the fused feature representation, comprises:

[0034] Inputting the fused feature representation into a multi-layer perceptron to generate an image recognition result of the target fine-grained image;

[0035] The image recognition result is output or displayed.

[0036] A fine-grained image recognition device, comprising:

[0037] A first feature representation generation module, used for dividing a target fine-grained image into a plurality of image blocks, and inputting the plurality of image blocks into a visual Transformer encoder to generate a first feature representation of the target fine-grained image;

[0038] A second feature representation generation module is used to extract entities and relationships related to the target fine-grained image based on a target knowledge graph corresponding to the target fine-grained image, and input the entities and relationships into a graph neural network encoder to generate a second feature representation of the target fine-grained image;

[0039] a fused feature representation generating module, configured to perform a cross-attention operation on the first feature representation and the second feature representation based on a cross-attention mechanism to generate a fused feature representation of the target fine-grained image;

[0040] An image recognition result generation module is used to generate an image recognition result of the target fine-grained image based on the fused feature representation using a multi-layer perceptron.

[0041] A computer device, wherein the computer device comprises at least one processor; and

[0042] a memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, the fine-grained image recognition method as described in any one of the above items can be implemented.

[0044] A non-volatile computer-readable storage medium stores a computer program, wherein the computer program, when executed by at least one processor, can implement any of the above-described methods for recognizing fine-grained images.

[0045] Compared with the prior art, the present invention provides a method, device, computer equipment and non-volatile computer-readable storage medium for recognizing fine-grained images, wherein the method includes: dividing a target fine-grained image into multiple image blocks, and inputting the multiple image blocks into a visual Transformer encoder to generate a first feature representation of the target fine-grained image; based on a target knowledge graph corresponding to the target fine-grained image, extracting entities and relationships related to the target fine-grained image, and inputting the entities and relationships into a graph neural network encoder to generate a second feature representation of the target fine-grained image; based on a cross-attention mechanism, performing a cross-attention operation on the first feature representation and the second feature representation to generate a fused feature representation of the target fine-grained image; based on the fused feature representation, using a multi-layer perceptron to generate an image recognition result of the target fine-grained image. In this way, the method of the present invention can effectively improve the accuracy of fine-grained image recognition by obtaining an image recognition result by fusing the visual information and knowledge information of the fine-grained image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic flow chart of a fine-grained image recognition method provided by an embodiment of the present invention.

[0048] Figure 2 A schematic diagram of program modules of a fine-grained image recognition device provided by an embodiment of the present invention.

[0049] Figure 3 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] In the description of the present invention, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps therein is not limited and can be appropriately adjusted as needed.

[0052] Various non-limiting embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0053] See also Figure 1 The embodiment of the present invention provides a method for recognizing fine-grained images, wherein the method specifically comprises the following steps:

[0054] S100, dividing a target fine-grained image into a plurality of image blocks, and inputting the plurality of image blocks into a visual Transformer encoder to generate a first feature representation of the target fine-grained image;

[0055] S200, based on the target knowledge graph corresponding to the target fine-grained image, extracting entities and relationships related to the target fine-grained image, and inputting the entities and relationships into a graph neural network encoder to generate a second feature representation of the target fine-grained image;

[0056] S300, based on a cross attention mechanism, performing a cross attention operation on the first feature representation and the second feature representation to generate a fused feature representation of the target fine-grained image;

[0057] S400: Based on the fused feature representation, generate an image recognition result of the target fine-grained image using a multi-layer perceptron.

[0058] Furthermore, the method for recognizing fine-grained images, wherein the step S100, dividing the target fine-grained image into a plurality of image blocks, and inputting the plurality of image blocks into a visual Transformer encoder to generate a first feature representation of the target fine-grained image, specifically comprises the steps of:

[0059] Obtaining a target fine-grained image input by a target user;

[0060] Evenly dividing the target fine-grained image into a plurality of image blocks of the same size and without overlap;

[0061] After extracting and serializing the features of the multiple image blocks, a corresponding feature sequence is generated;

[0062] The feature sequence is input into a visual Transformer encoder to generate a first feature representation of the target fine-grained image.

[0063] In the specific implementation, in this embodiment, a target fine-grained image input by a target user is obtained; the target fine-grained image is evenly divided into multiple image blocks of the same size and non-overlapping; after feature extraction and serialization of the multiple image blocks, a corresponding feature sequence is generated; the feature sequence is input into a visual Transformer encoder to generate a first feature representation of the target fine-grained image; wherein the specific implementation process of each of the above steps is roughly as follows:

[0064] 1. Image input and preprocessing:

[0065] Provide a user interface that allows the target user to upload a target fine-grained image; this can be achieved through a web interface or a desktop application;

[0066] Once the target fine-grained image is uploaded, the image is preprocessed, including scaling to a uniform size, converting the color space (e.g., from RGB to grayscale), and normalizing pixel values;

[0067] 2. Image block division:

[0068] The preprocessed target fine-grained image is evenly divided into multiple non-overlapping image blocks of the same size; this usually involves calculating the width and height of the image and then slicing it according to a predefined block size (e.g., 16x16 or 32x32 pixels);

[0069] 3. Feature extraction and serialization:

[0070] Apply a feature extraction model to each image patch; this typically involves loading a pre-trained deep learning model such as a convolutional neural network (CNN) or a specific layer of a visual transformer.

[0071] Serialize the extracted feature vectors, that is, arrange them into a long sequence in a certain order; if position encoding is used, add the position encoding to each feature vector to preserve the spatial position information of the image block;

[0072] 4. Input and processing of feature sequences:

[0073] Pass the serialized feature sequence as input to the visual Transformer encoder; process the feature sequence through the Transformer encoder, use its self-attention mechanism to capture the mutual relationship between image patches, and generate the first feature representation of the target fine-grained image;

[0074] 5. Generation and output of feature representation:

[0075] The visual Transformer encoder outputs the first feature representation, which will include the visual features of the target fine-grained image (such as color, texture, shape, etc.);

[0076] The generated first feature representation is used for subsequent image recognition tasks, such as classification, retrieval or other analysis;

[0077] Through the above process, a first feature representation of the target fine-grained image can be generated.

[0078] Further, the method for recognizing fine-grained images, wherein the step S200, based on the target knowledge graph corresponding to the target fine-grained image, extracts entities and relationships related to the target fine-grained image, and inputs the entities and relationships into a graph neural network encoder to generate a second feature representation of the target fine-grained image, specifically comprises the steps of:

[0079] Determine a corresponding target knowledge graph according to the image content of the target fine-grained image;

[0080] Analyzing the target knowledge graph, identifying and extracting entities and relationships in the target knowledge graph that are related to the target fine-grained image;

[0081] The entities and relations are input into a graph neural network encoder to generate a second feature representation of the target fine-grained image.

[0082] Further, in the method for recognizing fine-grained images, the first feature representation includes visual features;

[0083] The step of determining the corresponding target knowledge graph according to the image content of the target fine-grained image specifically includes the following steps:

[0084] Searching in a knowledge graph library according to the visual features of the target fine-grained image;

[0085] According to the retrieval results, a target knowledge graph corresponding to the target fine-grained image is obtained.

[0086] In specific implementation, in this embodiment, according to the visual features of the target fine-grained image, a search is performed in the knowledge graph library; according to the search results, a target knowledge graph corresponding to the target fine-grained image is obtained; the target knowledge graph is analyzed to identify and extract entities and relationships related to the target fine-grained image in the target knowledge graph; the entities and relationships are input into a graph neural network encoder to generate a second feature representation of the target fine-grained image; wherein the specific implementation process of each of the above steps is roughly as follows:

[0087] 1. Knowledge graph retrieval:

[0088] Retrieve from the knowledge graph based on the visual features of the generated target fine-grained image; perform complex queries using retrieval algorithms, such as feature similarity-based searches or graph database query languages ​​(such as Cypher for Neo4j);

[0089] 2. Knowledge graph analysis:

[0090] Once the relevant target knowledge graph is retrieved, an analysis process is performed to identify and extract entities and relations related to the target fine-grained image; this can be done through natural language processing (NLP) techniques to understand and process the semantic information in the target knowledge graph;

[0091] 3. Graph Neural Network Encoding:

[0092] Build a graph neural network (GNN) encoder (such as a Graph Transformer encoder) to process the extracted entities and relations; this includes defining the architecture of the GNN, such as choosing graph convolutional layers or graph attention layers, and setting training parameters;

[0093] Implement the training process of the graph neural network encoder to ensure that it can learn how to extract useful feature representations from entities and relations;

[0094] 4. Feature representation generation:

[0095] Based on the entities and relations related to the target fine-grained image extracted from the target knowledge graph, a second feature representation of the target fine-grained image is generated through a graph neural network encoder;

[0096] Through the above process, a second feature representation of the target fine-grained image can be generated.

[0097] Furthermore, the method for recognizing fine-grained images, wherein the step S300, based on the cross-attention mechanism, performs a cross-attention operation on the first feature representation and the second feature representation to generate a fused feature representation of the target fine-grained image, specifically comprises the steps of:

[0098] Performing an adjustment operation on the first feature representation and the second feature representation to ensure that the two are consistent in format and dimension;

[0099] Using the first feature representation as a query and the second feature representation as a key and a value, calculating the similarity between the keys in the first feature representation and the second feature representation using a cross-attention mechanism, and generating an attention weight;

[0100] Based on the attention weights, weighted summation is performed on the values ​​in the second feature representation to generate a fused feature representation of the target fine-grained image.

[0101] Furthermore, the method for recognizing fine-grained images, wherein the adjusting operation is performed on the first feature representation and the second feature representation to ensure that the two are consistent in format and dimension, specifically comprises the steps of:

[0102] Analyzing the formats of the first feature representation and the second feature representation, and adaptively adjusting the formats of the two according to the format analysis result to ensure that the first feature representation and the second feature representation are consistent in format;

[0103] Analyze the dimensions of the first feature representation and the second feature representation, and adaptively adjust the dimensions of the two according to the dimension analysis result to ensure that the first feature representation and the second feature representation are consistent in dimension.

[0104] In specific implementation, in this embodiment, the formats of the first feature representation and the second feature representation are analyzed, and the formats of the two are adaptively adjusted according to the format analysis results to ensure that the first feature representation and the second feature representation are consistent in format; the dimensions of the first feature representation and the second feature representation are analyzed, and the dimensions of the two are adaptively adjusted according to the dimension analysis results to ensure that the first feature representation and the second feature representation are consistent in dimension; the first feature representation is used as a query, and the second feature representation is used as a key and a value, and the similarity between the keys in the first feature representation and the second feature representation is calculated using a cross attention mechanism to generate an attention weight; based on the attention weight, the values ​​in the second feature representation are weighted summed to generate a fused feature representation of the target fine-grained image; wherein the specific implementation process of each of the above steps is roughly as follows:

[0105] 1. Format and dimensionality analysis of feature representation:

[0106] An analysis tool may be used to examine the format and dimensionality of the first feature representation and the second feature representation;

[0107] 2. Format and dimension adjustment:

[0108] According to the analysis results, adjust the format and dimension of the first feature representation and the second feature representation to make them consistent; this may involve reshaping the feature vector, changing the data type, or rearranging the feature vector to ensure that the two can be compatible for subsequent processing;

[0109] 3. Cross-attention mechanism implementation:

[0110] Represent the first feature as query (Q) and the second feature as key (K) and value (V);

[0111] Use dot product or other similarity metric to calculate the similarity between query (Q) and key (K) and generate attention weights;

[0112] 4. Application of attention weight:

[0113] Using the generated attention weights, perform a weighted sum of the values ​​in the second feature representation (V); this can be achieved through matrix multiplication or weighted average operation;

[0114] 5. Fusion feature representation generation:

[0115] The weighted summation result is used as the fusion feature representation, which combines visual information and knowledge information;

[0116] A post-processing step may be performed, which may include normalization or further feature selection on the fused feature representation to improve its quality and usability;

[0117] Through the above process, a fused feature representation of the target fine-grained image can be generated.

[0118] Furthermore, the method for recognizing fine-grained images, wherein the step S400, based on the fusion feature representation, using a multi-layer perceptron to generate an image recognition result of the target fine-grained image, specifically comprises the steps of:

[0119] Inputting the fused feature representation into a multi-layer perceptron to generate an image recognition result of the target fine-grained image;

[0120] The image recognition result is output or displayed.

[0121] In the specific implementation, in this embodiment, the fusion feature representation is input into a multilayer perceptron to generate an image recognition result of the target fine-grained image; the image recognition result is output or displayed; wherein the specific implementation process of each of the above steps is roughly as follows:

[0122] 1. Multilayer Perceptron (MLP) Construction:

[0123] Construct a multilayer perceptron that receives the fused feature representation as input; a multilayer perceptron typically contains one or more hidden layers and an output layer; the number of nodes and activation function (such as ReLU) of the hidden layer are determined according to the task requirements;

[0124] 2. Feature representation input:

[0125] Input the fused feature representation of the target fine-grained image into a pre-trained multi-layer perceptron; this usually involves data preprocessing, such as normalizing the feature vectors to match the data processing method when the multi-layer perceptron was trained;

[0126] 3. Forward propagation:

[0127] Perform the forward propagation process of the multi-layer perceptron, calculate the output of the hidden layer, and finally generate the prediction results of the output layer; the output layer usually uses the softmax activation function for multi-category classification;

[0128] 4. Result analysis:

[0129] Parsing the output of a multi-layer perceptron, usually selecting the class with the highest probability as the predicted class for the target fine-grained image; this involves converting the raw scores or probabilities of the output layer into understandable class labels;

[0130] 5. Result output or display:

[0131] Output or display the image recognition results of the target fine-grained image to the target user in an appropriate manner; this can be achieved by console printing, graphical user interface (GUI) display or writing the results to a file;

[0132] Through the above process, an image recognition result of the target fine-grained image can be generated, and the image recognition result can be output or displayed.

[0133] It can be seen from the above method embodiments that the fine-grained image recognition method provided by the present invention includes: dividing the target fine-grained image into multiple image blocks, and inputting the multiple image blocks into the visual Transformer encoder to generate a first feature representation of the target fine-grained image; based on the target knowledge graph corresponding to the target fine-grained image, extracting entities and relationships related to the target fine-grained image, and inputting the entities and relationships into the graph neural network encoder to generate a second feature representation of the target fine-grained image; based on the cross-attention mechanism, performing a cross-attention operation on the first feature representation and the second feature representation to generate a fused feature representation of the target fine-grained image; based on the fused feature representation, using a multi-layer perceptron to generate an image recognition result of the target fine-grained image. In this way, the method of the present invention can effectively improve the accuracy of fine-grained image recognition by fusing the visual information and knowledge information of the fine-grained image to obtain the image recognition result.

[0134] Specifically, the fine-grained image recognition method provided by the embodiment of the present invention can at least achieve the following beneficial technical effects:

[0135] 1. Improved classification accuracy. By effectively fusing visual information and knowledge information, the proposed method can better distinguish fine-grained categories that exhibit small inter-class variations and large intra-class variations. This improved classification capability can improve the performance of FGIR tasks in a wide range of applications, such as biodiversity monitoring, climate change assessment, smart retail, and smart transportation.

[0136] 2. It can improve the robustness to complex visual scenes. The method of the present invention can better handle complex visual scenes, making it more robust and adaptable to various real-world FGIR scenes;

[0137] 3. It can improve generalization and transferability. By combining knowledge graph information, the method of the present invention can enhance visual information with rich semantic information and high-level knowledge, which can achieve better generalization and transferability between different FGIR tasks and domains;

[0138] 4. Improved scalability and adaptability. The modular architecture of the method of the present invention consists of independent visual Transformer encoders, graph neural network encoders, and fusion stages, which can achieve greater scalability and adaptability. This approach allows the content of the modular architecture to be easily adjusted and fine-tuned to adapt to different FGIR tasks, data sets, and computing resources;

[0139] 5. It can promote cross-research on computer vision and knowledge graphs. The method of the present invention not only improves the performance of FGIR, but also promotes the research on the combination of computer vision and knowledge graphs. By demonstrating the potential of integrating knowledge graph information into deep learning models, the method of the present invention can inspire further exploration of this interdisciplinary field, thereby bringing new breakthroughs and progress in these two fields.

[0140] Of course, the method of the present invention can be specifically applied at least in the recognition scenario of fine-grained images related to financial or medical services. For example, in the medical field, it can solve many tasks that require a high degree of sophistication, such as medical image diagnosis, pathological section analysis, skin disease recognition, and long-term patient monitoring;

[0141] The specific application scenarios are:

[0142] 1. Medical image diagnosis: The method of the present invention can perform in-depth analysis on medical images such as X-rays, CT scans, and MRIs, extract fine features, and combine relevant medical knowledge to make more accurate diagnosis and judgment;

[0143] 2. Pathological section analysis: The method of the present invention can embed relevant pathological knowledge in the knowledge graph into the deep learning model to identify subtle changes in pathological section images, thereby improving the accuracy of diagnosis;

[0144] 3. Skin disease recognition: By extracting fine-grained features from skin images and combining them with skin disease knowledge in the knowledge graph, the method of the present invention can improve the recognition accuracy of skin diseases;

[0145] 4. Long-term patient monitoring: The method of the present invention can conduct in-depth analysis of the patient's medical images, monitor the health status of patients with chronic diseases (such as diabetes, hypertension, etc.), and adjust the treatment plan when necessary.

[0146] It should be understood that, although the present application provides method operation steps as described in the embodiments or flowcharts, more or less operation steps may be included based on conventional or non-creative labor, and these operation steps are not necessarily performed in sequence according to the embodiment or flowchart. The order of steps listed in the embodiment or flowchart is only one way of executing the order of many steps, and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. A person of ordinary skill in the art can understand from the description of the embodiment of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in exchange, etc. Moreover, at least a part of the steps in the embodiment or flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0147] Based on the above method embodiment, please refer to Figure 2 Another embodiment of the present invention further provides a fine-grained image recognition device, wherein the device specifically includes:

[0148] A first feature representation generating module 11 is used to divide the target fine-grained image into a plurality of image blocks, and input the plurality of image blocks into a visual Transformer encoder to generate a first feature representation of the target fine-grained image;

[0149] A second feature representation generating module 12 is used to extract entities and relationships related to the target fine-grained image based on the target knowledge graph corresponding to the target fine-grained image, and input the entities and relationships into a graph neural network encoder to generate a second feature representation of the target fine-grained image;

[0150] A fused feature representation generating module 13, configured to perform a cross-attention operation on the first feature representation and the second feature representation based on a cross-attention mechanism to generate a fused feature representation of the target fine-grained image;

[0151] The image recognition result generating module 14 is used to generate the image recognition result of the target fine-grained image by using a multi-layer perceptron based on the fused feature representation.

[0152] Furthermore, in the fine-grained image recognition device, the first feature representation generation module 11 specifically includes:

[0153] An image acquisition unit, used to acquire a target fine-grained image input by a target user;

[0154] An image division unit, used for evenly dividing the target fine-grained image into a plurality of image blocks of the same size and without overlap;

[0155] A feature extraction and serialization unit, configured to extract and serialize features of the plurality of image blocks to generate corresponding feature sequences;

[0156] The feature sequence input unit is used to input the feature sequence into the visual Transformer encoder to generate a first feature representation of the target fine-grained image.

[0157] Furthermore, in the fine-grained image recognition device, the second feature representation generation module 12 specifically includes:

[0158] A knowledge graph determination unit, used to determine a corresponding target knowledge graph according to the image content of the target fine-grained image;

[0159] A knowledge graph analysis unit, configured to analyze the target knowledge graph, identify and extract entities and relationships in the target knowledge graph that are related to the target fine-grained image;

[0160] The entity and relationship input unit is used to input the entities and relationships into a graph neural network encoder to generate a second feature representation of the target fine-grained image.

[0161] Further, in the fine-grained image recognition device, the first feature representation includes a visual feature;

[0162] Determining the target knowledge graph corresponding to the target fine-grained image according to the image content of the target fine-grained image specifically includes:

[0163] Searching in a knowledge graph library according to the visual features of the target fine-grained image;

[0164] According to the retrieval results, a target knowledge graph corresponding to the target fine-grained image is obtained.

[0165] Furthermore, in the fine-grained image recognition device, the fusion feature representation generation module 13 specifically includes:

[0166] an adjustment operation execution unit, configured to execute an adjustment operation on the first feature representation and the second feature representation to ensure that the two are consistent in format and dimension;

[0167] a similarity calculation unit, configured to use the first feature representation as a query and the second feature representation as a key and a value, calculate the similarity between the keys in the first feature representation and the second feature representation by using a cross attention mechanism, and generate an attention weight;

[0168] A weighted summation unit is used to perform weighted summation on the values ​​in the second feature representation based on the attention weight to generate a fused feature representation of the target fine-grained image.

[0169] Furthermore, in the fine-grained image recognition device, the step of performing an adjustment operation on the first feature representation and the second feature representation to ensure that the two are consistent in format and dimension specifically includes:

[0170] Analyzing the formats of the first feature representation and the second feature representation, and adaptively adjusting the formats of the two according to the format analysis result to ensure that the first feature representation and the second feature representation are consistent in format;

[0171] Analyze the dimensions of the first feature representation and the second feature representation, and adaptively adjust the dimensions of the two according to the dimension analysis result to ensure that the first feature representation and the second feature representation are consistent in dimension.

[0172] Furthermore, in the fine-grained image recognition device, the image recognition result generation module 14 specifically includes:

[0173] A fused feature representation input unit, used for inputting the fused feature representation into a multi-layer perceptron to generate an image recognition result of the target fine-grained image;

[0174] The result output or display unit is used to output or display the image recognition result.

[0175] The specific implementation process and effects of the device embodiment of the present invention can be found in the above method embodiment, which will not be described in detail here.

[0176] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides a computer device, wherein the computer device 10 includes:

[0177] memory 120 and one or more processors 110, Figure 3 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a communication bus or other means. Figure 3 The example of connecting via a communication bus is taken as an example.

[0178] The processor 110 is used to complete various control logics of the computer device 10. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP, or any other such configuration.

[0179] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the computer program corresponding to the fine-grained image recognition method in any of the above-mentioned method embodiments of the present invention. The processor 110 executes various functional applications and data processing of the computer device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the fine-grained image recognition method in any of the above-mentioned method embodiments.

[0180] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device or at least one function; the data storage area may store data created according to the use of the computer device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the computer device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] One or more units are stored in the memory 120, and when executed by one or more processors 110, the fine-grained image recognition method in any of the above method embodiments can be implemented, for example, the above described method can be implemented. Figure 1 The method comprises steps S100 to S400.

[0182] Those skilled in the art will understand that Figure 3The hardware structure diagram shown in the figure is only a schematic diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0183] Based on the above method embodiments, another embodiment of the present invention further provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the fine-grained image recognition method in any of the above method embodiments can be implemented, for example, the above-described Figure 1 The method comprises steps S100 to S400.

[0184] As an example, the non-volatile storage medium can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable ROM (EEPROM), or a flash memory. Volatile memory can include a random access memory (RAM) as an external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synch link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0185] Based on the above method embodiments, another embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by a processor, the fine-grained image recognition method in any of the above method embodiments can be implemented, for example, the above described Figure 1 The method comprises steps S100 to S400.

[0186] The embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can exist in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0188] Conditional language such as "can," "can," "might," or "may," unless specifically stated otherwise or otherwise understood within the context as used, is generally intended to convey, among other things, that a particular embodiment can include (while other embodiments do not) a particular feature, element, and / or operation. Thus, such conditional language is also generally intended to imply that a feature, element, and / or operation is required for one or more embodiments anyway or that one or more embodiments must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular embodiment.

[0189] What has been described herein, in this specification and in the drawings includes examples of a recognition method, device, computer equipment, and non-volatile computer-readable storage medium that can provide a fine-grained image. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it can be recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is obvious that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure, but all these various modifications should belong to the scope of protection of the claims attached to the present invention. In addition, or in an alternative, other embodiments of the present disclosure may be obvious from consideration of the present specification and the drawings and from the practice of the present disclosure as presented herein. It is intended that the examples proposed in the present specification and the drawings are considered to be illustrative and not restrictive in all respects. Although specific terms are used herein, they are used in a general and descriptive sense and are not used for limiting purposes.

Claims

1. A fine-grained image recognition method, characterized in that: include: Dividing a target fine-grained image into a plurality of image blocks, and inputting the plurality of image blocks into a visual Transformer encoder to generate a first feature representation of the target fine-grained image; Based on the target knowledge graph corresponding to the target fine-grained image, extract entities and relationships related to the target fine-grained image, and input the entities and relationships into a graph neural network encoder to generate a second feature representation of the target fine-grained image; Based on a cross-attention mechanism, performing a cross-attention operation on the first feature representation and the second feature representation to generate a fused feature representation of the target fine-grained image; Based on the fused feature representation, a multi-layer perceptron is used to generate an image recognition result of the target fine-grained image.

2. The fine-grained image recognition method according to claim 1, characterized in that: The step of dividing the target fine-grained image into a plurality of image blocks and inputting the plurality of image blocks into a visual Transformer encoder to generate a first feature representation of the target fine-grained image includes: Obtaining a target fine-grained image input by a target user; Evenly dividing the target fine-grained image into a plurality of image blocks of the same size and without overlap; After extracting and serializing the features of the multiple image blocks, a corresponding feature sequence is generated; The feature sequence is input into a visual Transformer encoder to generate a first feature representation of the target fine-grained image.

3. The fine-grained image recognition method according to claim 2, characterized in that: The step of extracting entities and relationships related to the target fine-grained image based on the target knowledge graph corresponding to the target fine-grained image, and inputting the entities and relationships into a graph neural network encoder to generate a second feature representation of the target fine-grained image includes: Determine a corresponding target knowledge graph according to the image content of the target fine-grained image; Analyzing the target knowledge graph, identifying and extracting entities and relationships in the target knowledge graph that are related to the target fine-grained image; The entities and relations are input into a graph neural network encoder to generate a second feature representation of the target fine-grained image.

4. The fine-grained image recognition method according to claim 3, characterized in that: The first feature representation includes a visual feature; The step of determining the target knowledge graph corresponding to the target fine-grained image according to the image content of the target fine-grained image includes: Searching in a knowledge graph library according to the visual features of the target fine-grained image; According to the retrieval results, a target knowledge graph corresponding to the target fine-grained image is obtained.

5. The fine-grained image recognition method according to claim 3 or 4, characterized in that: The method of performing a cross attention operation on the first feature representation and the second feature representation based on the cross attention mechanism to generate a fused feature representation of the target fine-grained image includes: Performing an adjustment operation on the first feature representation and the second feature representation to ensure that the two are consistent in format and dimension; Using the first feature representation as a query and the second feature representation as a key and a value, calculating the similarity between the keys in the first feature representation and the second feature representation using a cross-attention mechanism, and generating an attention weight; Based on the attention weights, weighted summation is performed on the values ​​in the second feature representation to generate a fused feature representation of the target fine-grained image.

6. The fine-grained image recognition method according to claim 5, characterized in that: The performing of an adjustment operation on the first feature representation and the second feature representation to ensure that the two are consistent in format and dimension includes: Analyzing the formats of the first feature representation and the second feature representation, and adaptively adjusting the formats of the two according to the format analysis result to ensure that the first feature representation and the second feature representation are consistent in format; Analyze the dimensions of the first feature representation and the second feature representation, and adaptively adjust the dimensions of the two according to the dimension analysis result to ensure that the first feature representation and the second feature representation are consistent in dimension.

7. The method for recognizing fine-grained images according to claim 6, characterized in that: The method of generating an image recognition result of the target fine-grained image by using a multi-layer perceptron based on the fused feature representation includes: Inputting the fused feature representation into a multi-layer perceptron to generate an image recognition result of the target fine-grained image; The image recognition result is output or displayed.

8. A fine-grained image recognition device, characterized in that: include: A first feature representation generation module, used for dividing a target fine-grained image into a plurality of image blocks, and inputting the plurality of image blocks into a visual Transformer encoder to generate a first feature representation of the target fine-grained image; A second feature representation generation module is used to extract entities and relationships related to the target fine-grained image based on a target knowledge graph corresponding to the target fine-grained image, and input the entities and relationships into a graph neural network encoder to generate a second feature representation of the target fine-grained image; a fused feature representation generating module, configured to perform a cross-attention operation on the first feature representation and the second feature representation based on a cross-attention mechanism to generate a fused feature representation of the target fine-grained image; An image recognition result generation module is used to generate an image recognition result of the target fine-grained image based on the fused feature representation using a multi-layer perceptron.

9. A computer device, characterized in that: The computer device comprises at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, the method for recognizing fine-grained images according to any one of claims 1 to 7 can be implemented.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method for recognizing a fine-grained image as described in any one of claims 1 to 7 can be implemented.

Images