Image fine-grained recognition method and device, storage medium and computer equipment

Abstract

The application discloses an image fine-grained recognition method and device, a storage medium and computer equipment, relates to the technical field of image recognition and the field of medical image recognition, and mainly aims to solve the problems of low recognition efficiency and accuracy of fine-grained images. Mainly including obtaining an image to be recognized and a target description text matched with a basic category of the image to be recognized, the target description text including description texts under different fine categories; using a trained feature extraction model to extract features of the image to be recognized and the target description text respectively to obtain an image feature vector and a text feature vector, the trained image recognition model being obtained based on text samples crawled by a network and a joint loss function; determining an image-text similarity matrix based on the image feature vector and the text feature vector, and determining a recognition result of the image to be recognized based on the image-text similarity matrix. The method is mainly used for fine-grained image recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, the field of medical image recognition, and in particular, to an image fine-grained recognition method and device, a storage medium, and a computer device. BACKGROUND

[0002] Fine-grained image recognition, that is, fine classification of images. For example, traditional image recognition can only recognize whether an image is a dog or a bird, while the research direction of fine-grained image recognition is to further recognize the breed of the dog on the basis of recognizing that the image is a dog. This method has a wide range of applications in fine recognition of animal species, lesion recognition of medical images, product recognition in unmanned retail scenarios, and vehicle recognition in smart city traffic.

[0003] Existing fine-grained image recognition mainly uses a strong supervision method, which requires human-labeled information such as target bounding boxes and part labeling points in images to train the model. The labeling process requires expert knowledge to label each image to be classified individually, which requires a long time and a lot of manpower, resulting in low efficiency of image recognition. At the same time, training the model based on human-labeled samples has weak robustness in actual application, resulting in low accuracy of image recognition. In particular, in the classification and recognition of medical images, due to the large differences in tissue morphology and lesion morphology of each patient, the model trained based on human labeling has low accuracy in the classification and recognition of medical images in actual application. SUMMARY

[0004] Therefore, the present application provides an image fine-grained recognition method and device, a storage medium, and a computer device, which mainly aims to solve the problem of low efficiency and accuracy of existing fine-grained image recognition.

[0005] According to one aspect of the present application, an image fine-grained recognition method is provided, comprising:

[0006] obtaining an image to be recognized and a target description text matching a basic category of the image to be recognized, the target description text including description texts under different fine categories;

[0007] using a trained feature extraction model to extract features from the image to be recognized and the target description text, respectively, to obtain an image feature vector and a text feature vector, the trained image recognition model being obtained based on text samples crawled from a network and a joint loss function;

[0008] determine an image-text similarity matrix based on the image feature vector and the text feature vector, and determine the recognition result of the image to be recognized based on the image-text similarity matrix.

[0009] Further, before the image feature vector and the text feature vector are obtained by using the trained feature extraction model to extract features from the image to be recognized and the target description text respectively, the method further comprises:

[0010] obtaining a training sample set, the training sample set comprising image-text sample pairs under different fine categories, the image-text sample pair comprising an image sample and a description text sample corresponding to the image sample crawled from a target website;

[0011] constructing an initial feature extraction model, the initial feature extraction model comprising an image encoder and a text encoder;

[0012] constructing a joint loss function based on the training sample set, the image encoder and the text encoder, and training the initial feature extraction model using the joint loss function to obtain a trained feature extraction model.

[0013] Further, the constructing of the joint loss function based on the training sample set, the image encoder and the text encoder comprises:

[0014] constructing a first loss function based on two encoding results of the same image sample by the image encoder;

[0015] constructing a second loss function based on a similarity matrix of the image sample feature vector and the text sample feature vector and a label mask matrix;

[0016] weighting and fusing the first loss function and the second loss function to obtain the joint loss function.

[0017] Further, before the training sample set is obtained, the method further comprises:

[0018] obtaining an expected fine category, and obtaining at least one image sample corresponding to each fine category according to the expected fine category;

[0019] respectively crawling at least one description text from a target website according to each of the expected fine categories;

[0020] extracting any one of the description texts as a description text sample from the description texts corresponding to the image samples, and constructing an image-text sample pair comprising the description text sample and the image sample to obtain the training sample set.

[0021] Further, the trained feature extraction model comprises an image encoder and a text encoder, and the trained feature extraction model is used to extract features of the to-be-identified image and the description text of the target basic category, to obtain an image feature vector and a text feature vector.

[0022] The image encoder is used to extract features of the to-be-identified image, to obtain an initial image feature vector, and the initial image feature vector is subjected to high-dimensional feature mapping, to obtain an image feature vector.

[0023] The text encoder is used to tokenize the target description text, and the tokenized target description text is subjected to feature extraction, to obtain an initial text feature vector.

[0024] The initial text feature vector is subjected to high-dimensional feature mapping, to obtain a text feature vector.

[0025] Further, the obtaining of the to-be-identified image and the target description text matched with the basic category of the to-be-identified image comprises:

[0026] An image recognition request is obtained, and the image recognition request carries a to-be-identified image and image collection terminal information.

[0027] The image collection terminal information is analyzed, a business attribute corresponding to the to-be-identified image is determined, and a target description text is matched from description texts of different basic categories according to the business attribute.

[0028] Further, before the obtaining of the to-be-identified image and the target description text matched with the basic category of the to-be-identified image, the method further comprises:

[0029] An expected fine category of an expected to-be-identified image is obtained.

[0030] At least one piece of description text corresponding to each of the expected fine categories is crawled from a target website according to the expected fine category.

[0031] The description texts of different expected fine categories are divided into basic categories according to a preset basic category, to obtain description texts of different basic categories.

[0032] According to another aspect of the present application, an image fine-grained recognition device is provided, comprising:

[0033] An obtaining module is configured to obtain a to-be-identified image and a target description text matched with a basic category of the to-be-identified image, and the target description text comprises description texts under different fine categories.

[0034] An extraction module is configured to perform feature extraction on the to-be-identified image and the target description text respectively by using a trained feature extraction model to obtain an image feature vector and a text feature vector, and the trained image recognition model is obtained based on a text sample crawled by a network and a joint loss function.

[0035] A determination module is configured to determine an image-text similarity matrix based on the image feature vector and the text feature vector, and determine the identification result of the to-be-identified image based on the image-text similarity matrix.

[0036] Further, the apparatus further comprises:

[0037] The acquisition module is further configured to acquire a training sample set, and the training sample set comprises image-text sample pairs under different fine categories, wherein each image-text sample pair comprises an image sample and a description text sample corresponding to the image sample and crawled from a target website.

[0038] A first construction module is configured to construct an initial feature extraction model, and the initial feature extraction model comprises an image encoder and a text encoder.

[0039] A training module is configured to construct a joint loss function based on the training sample set, the image encoder and the text encoder, and train the initial feature extraction model by using the joint loss function to obtain a trained feature extraction model.

[0040] Further, the training module comprises:

[0041] A second construction module is configured to construct a first loss function based on two encoding results of the same image sample by using the image encoder.

[0042] A third construction module is configured to construct a second loss function based on a similarity matrix of an image sample feature vector and a text sample feature vector and a label mask matrix.

[0043] A fusion module is configured to perform weighted fusion based on the first loss function and the second loss function to obtain a joint loss function.

[0044] Further, the apparatus further comprises:

[0045] The acquisition module is further configured to acquire an expected fine category, and acquire at least one image sample corresponding to each fine category according to the expected fine category.

[0046] A crawling module is configured to crawl at least one description text from a target website according to each of the expected fine categories.

[0047] A fourth constructing module is configured to extract any one segment from the description text corresponding to the image sample as a description text sample, and construct an image-text sample pair by combining the description text sample and the image sample, so as to obtain a training sample set.

[0048] Further, the extracting module comprises:

[0049] A first extracting unit is configured to perform feature extraction on the image to be recognized by using the image encoder, so as to obtain an initial image feature vector, and perform high-dimensional feature mapping on the initial image feature vector, so as to obtain an image feature vector;

[0050] A processing unit is configured to perform tokenization processing on the target description text by using the text encoder, and perform feature extraction on the target description text after the tokenization processing, so as to obtain an initial text feature vector;

[0051] A mapping unit is configured to perform high-dimensional feature mapping on the initial text feature vector, so as to obtain a text feature vector.

[0052] Further, the obtaining module comprises:

[0053] An obtaining unit is configured to obtain an image recognition request, wherein the image recognition request carries an image to be recognized and image collection terminal information;

[0054] A matching unit is configured to analyze the image collection terminal information, determine a business attribute corresponding to the image to be recognized, and match a target description text from description texts of different basic categories according to the business attribute.

[0055] Further, the device further comprises:

[0056] The obtaining module is further configured to obtain expected fine categories of expected recognition images.

[0057] A second crawling module is configured to respectively crawl at least one segment of description text corresponding to each of the expected fine categories from a target website according to the expected fine categories.

[0058] A dividing module is configured to divide the description texts of different expected fine categories into description texts of different basic categories according to preset basic categories.

[0059] According to another aspect of the present application, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction causes a processor to perform operations corresponding to the image fine-grained recognition method.

[0060] According to still another aspect of the present application, a computer device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface being capable of communicating with each other through the communication bus;

[0061] The memory is configured to store at least one executable instruction, which causes the processor to perform operations corresponding to the image fine-grained recognition method.

[0062] By means of the technical solution described above, the technical solution provided by the embodiments of the present application has at least the following advantages:

[0063] The present application provides an image fine-grained recognition method and device, a storage medium and a computer device. Firstly, a to-be-recognized image and a target description text matched with a basic category of the to-be-recognized image are obtained, and the target description text includes description texts under different fine categories. Then, a feature extraction model that has been trained is used to extract features of the to-be-recognized image and the target description text respectively to obtain an image feature vector and a text feature vector. The image recognition model that has been trained is obtained based on text samples crawled from a network and a joint loss function. An image-text similarity matrix is determined based on the image feature vector and the text feature vector, and a recognition result of the to-be-recognized image is determined based on the image-text similarity matrix. Compared with the prior art, the embodiments of the present application train the model by means of text samples crawled from a network, thereby avoiding manual text labeling of image samples, greatly reducing the model training time, effectively improving the learning ability of the model based on the joint loss function, and improving the accuracy and efficiency of image recognition based on the contrast learning manner. In the classification and recognition process of medical images, the model is trained based on the joint loss function containing image coding loss, which can greatly reduce the error of organizing image coding. Meanwhile, the contrast learning is performed based on each tissue image in the image and the image description text, which can effectively improve the recognition degree of tissues and lesions, thereby improving the classification and recognition accuracy of medical images.

[0064] The above description is only a summary of the technical solutions of the present application. In order to enable one of ordinary skill in the art to better understand the technical means of the present application and implement it according to the content of the description, and in order to enable the above and other purposes, features and advantages of the present application to be more apparent and understandable, the following specific embodiments of the present application are described in detail. BRIEF DESCRIPTION OF DRAWINGS

[0065] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to depict only preferred embodiments of the application, and therefore should not be considered to limit the scope of the application in any way. Similarly, like reference numerals have been used in the drawings to depict like parts of the application. In the drawings:

[0066] Figure 1 A flow chart of an image fine-grained recognition method provided by an embodiment of the present application is shown;

[0067] Figure 2 A flow chart of another image fine-grained recognition method provided by an embodiment of the present application is shown;

[0068] Figure 3 A feature recognition model training schematic diagram provided by an embodiment of the present application is shown;

[0069] Figure 4 A block diagram of an image fine-grained recognition device provided by an embodiment of the present application is shown;

[0070] Figure 5 A structural schematic diagram of a computer device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0071] Exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0072] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0073] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0074] The existing image fine-grained recognition mainly adopts a strong supervision method, which needs to rely on manual annotation information such as target bounding box and part annotation point in the image to train the model, wherein, the marking process needs to rely on expert knowledge to individually annotate each image to be classified, which needs to consume a long time and more manpower, resulting in low efficiency of image recognition, at the same time, the model is trained based on the manually annotated samples, the robustness of the model in actual application is weak, resulting in the technical problem of low accuracy of image recognition. The embodiment of the present application provides an image fine-grained recognition method, as shown in the method applied to a computer device such as a server, wherein the server can be a stand-alone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN: Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, such as intelligent medical systems, digital medical platforms, etc. The method comprises: Figure 1

[0075] 101、Obtain an image to be identified, and a target description text matched with a basic category of the image to be identified.

[0076] In the embodiment of the present application, the number of images to be identified can be one or more, and the content of the image can be an animal image captured in an animal research scene, a to-be-settled commodity captured in a self-service vending scene, a vehicle image captured in an intelligent city traffic scene, a drug and medical device image captured in a medical service scene, etc. The present application embodiment is not limited. The basic category can be a basic category corresponding to different scenes, for example, a wide category such as retail goods, drugs, animals, and vehicles, or a coarse category in a certain scene, for example, fruits, vegetables, packaged products, and bulk products in a self-service vending scene. The target description text includes description texts in different fine categories, wherein the description text in each fine category can be one or more. The fine category can be the smallest classification category, for example, a warbler and a Samoyed dog, or a specific commodity or drug name, for example, a certain brand of pasta and a cold medicine produced by a certain pharmaceutical factory, etc. The present application embodiment is not limited.

[0077] ​It should be noted that the target description text matched with the basic category of the to-be-identified image can be based on different collection terminals of the to-be-identified image, for example, the image collected by the terminal equipment in the supermarket aquatic product area is of the aquatic product basic category, the image collected by the terminal equipment in the snack area is of the snack basic category, or can be based on different uploading terminals of the to-be-identified image, for example, the image uploaded by the terminal equipment in the hospital western medicine pharmacy is of the western medicine category, and the image uploaded by the terminal equipment in the hospital traditional Chinese medicine pharmacy is of the traditional Chinese medicine category. The to-be-identified image is identified based on the target description text matched with the basic category of the to-be-identified image, rather than based on the global description text, which can greatly reduce the number of texts, reduce the calculation amount of feature extraction and subsequent similarity calculation, and thus improve the efficiency of image identification. In addition, for the application scenario with many basic categories, the categories are pre-divided from the text angle, which is more conducive to improving the accuracy of image identification.

[0078] 102. Feature extraction is performed on the to-be-identified image and the target description text respectively by using the trained feature extraction model to obtain an image feature vector and a text feature vector.

[0079] In the embodiment of the application, the trained feature extraction model includes an image encoder for performing feature extraction on an image and a text encoder for performing feature extraction on a text, which can respectively perform feature extraction on the to-be-identified image and the target description text to obtain an image feature vector of the to-be-identified image and a text feature vector of the target description text. The trained image recognition model is trained based on network crawled text samples, that is, the text part of the model training sample is not based on expert knowledge and artificial marking sample, but based on description text of different fine category image samples crawled from the network to construct the text sample. The network crawled text sample is used to construct the text sample, which can greatly reduce the workload and time consumption based on artificial marking, thereby improving the efficiency of model training, ensuring the real-time and diversity of the description text, and effectively improving the robustness of the model.

[0080] 103. An image-text similarity matrix is determined based on the image feature vector and the text feature vector, and an identification result of the to-be-identified image is determined based on the image-text similarity matrix.

[0081] In the embodiment of the present application, after obtaining the image feature vector and the text feature vector, the image feature vector and the text feature vector are multiplied in matrix to calculate the similarity of the to-be-identified image and the description text under different fine categories, and an image-text similarity matrix is obtained. For each to-be-identified image, the vector with the maximum similarity value is found from the image-text similarity matrix, and the fine category of the description text corresponding to the vector is taken as the category of the current to-be-identified image. For example, in the medical image classification and identification process, the image description texts of different lesions under different tissues are extracted to obtain text feature vectors, the image feature vector corresponding to the medical image is multiplied with the text feature vector in matrix, the similarity of the image feature vector and the text feature vector in each multiplication relationship in the matrix is calculated, and the description text corresponding to the text feature vector with the maximum similarity of the image feature vector is taken as the category of the medical image. For example, the description text with the maximum similarity is "liver malignant tumor three stages", and "liver malignant tumor three stages" is the fine-grained classification result of the category of the medical image. Since the fine category can be a fine category, a minimum classification category, or a specific commodity or drug name, the identification result of the to-be-identified image can be a small category of the object in the to-be-identified image, such as an over-the-counter drug or a prescription drug, or the name of the object in the to-be-identified image, such as a liver tissue benign tumor or a lung tissue malignant tumor.

[0082] In one embodiment of the present application, in order to further illustrate and limit, as shown in Figure 2 Before the step of extracting features of the to-be-identified image and the target description text by using the trained feature extraction model to obtain the image feature vector and the text feature vector, the method further comprises:

[0083] 201, obtaining a training sample set.

[0084] 202, constructing an initial feature extraction model.

[0085] 203, constructing a joint loss function based on the training sample set, the image encoder and the text encoder, and training the initial feature extraction model by using the joint loss function to obtain a trained feature extraction model.

[0086] In the embodiment of the present application, the training sample set includes image-text sample pairs under different fine-grained categories. An image-text sample pair includes an image sample and a description text sample corresponding to the image sample crawled from a target website. The image-text sample pair is constructed based on the image sample and the description text sample corresponding to the image sample. In the training process, the image sample and the description text sample are input into the model in pairs to train the model. The description text sample is obtained by crawling corresponding description text from the target website according to each image sample. Multiple description texts corresponding to the image sample can be crawled from the target website. The description text sample paired with the image sample is a randomly extracted description text from the multiple description texts. The target website can be a Baidu, Wikipedia, or other encyclopedia website, or a product introduction website, which is not limited in the embodiment of the present application. The randomly extracted description text and the corresponding image sample form the image-text sample pair, which can make the training samples more diverse, thereby effectively improving the robustness of the model.

[0087] In the embodiment of the present application, the loss function is a joint loss function. The joint loss function is obtained by weighted fusion of the individual loss function based on the image encoder and / or the text encoder and the overall loss function of the image encoder and the text encoder. The weight of each loss function can be customized according to specific application needs, which is not limited in the embodiment of the present application.

[0088] In one embodiment of the present application, in order to further illustrate and limit, the step of training the initial feature extraction model using the image sample and the description text sample to obtain the trained feature extraction model includes:

[0089] constructing a first loss function based on the two encoding results of the same image sample by the image encoder;

[0090] constructing a second loss function based on the similarity matrix of the image sample feature vector and the text sample feature vector and the label mask matrix;

[0091] performing weighted fusion based on the first loss function and the second loss function to obtain a joint loss function.

[0092] In the embodiment of the present application, in the training process, the image sample and the description text sample are input into the model in batches, and the cls token position of the last layer encoder output of each input image is taken as the global feature description of the image. In a batch of inputs, the image sample feature vector is the text sample feature vector is Wherein, N is the number of images, D is the dimension of the feature vector, I is the Ith image, and T is the Tth text. The image sample feature vector and the text sample feature vector are calculated by using the contrast learning method to obtain the similarity S, and the formula is: The feature vector of each image sample should have the maximum similarity with the feature vector of the text description of the corresponding fine category, so the optimization goal is to make the similarity in the same fine category position in the similarity matrix S as large as possible and the similarity in other positions as small as possible. For example, input a batch of 5 images, wherein the 2nd and 4th images belong to the same category, and the similarity values in the 4th column of the 2nd row and the 2nd column of the 4th row in the generated 5*5 similarity matrix should be the largest, and the similarity values in other positions should be smaller than the positions. Here, the corresponding mask matrix can be generated from the real labels of the images to optimize the goal, and the positions corresponding to the above in the mask are 1 and the other positions are 0. Then, the two-dimensional cross-entropy loss function of the similarity matrix and the mask, i.e. the second loss function, is calculated to optimize the network model.

[0093] In the embodiment of the application, since there is dropout, i.e. a neural network with random dropout, in the training process of the deep learning network, the two feature extraction results of the same image by the encoder are different. In order to improve the feature learning ability of the image encoder, the image sample is input into the image encoder twice, and the image feature vectors obtained by the two times are represented as and Wherein, the image and the text are input into the model in batches, N is the number of images, D is the dimension of the vector, and I is the Ith image. Since they have the same size, the similarity matrix S1 of the two can be directly calculated, and the formula is: S1=f I ·f I '(2). By calculating the contrast learning loss function (infoNCE), i.e. the first loss function, the values on the diagonal of the similarity matrix are maximized to obtain a better image feature representation.

[0094] It should be noted that the image encoder and the text encoder are trained by using the contrast learning method, and the joint loss function is constructed based on the first loss function corresponding to the image encoder and the second loss function composed of the image encoder and the text encoder, which can effectively improve the feature learning ability of the image encoder and the text encoder, thereby improving the accuracy of image recognition of the model.

[0095] In one embodiment of the application, in order to further illustrate and limit, before the step of obtaining the training sample set, the method further comprises:

[0096] obtaining at least one image sample corresponding to each fine category respectively according to the expected fine category;

[0097]

[0098] extracting any one of the description texts corresponding to the image sample as a description text sample, and constructing an image-text sample pair with the description text sample and the image sample to obtain a training sample set.

[0099] In the embodiment of the application, the expected fine category is the fine category expected to be divided under each basic category. At least one image sample is obtained for each expected fine category, and according to the expected fine category, an automatic script is used to perform data crawling and data cleaning on the target website to obtain a group of description texts for describing the image samples under the corresponding expected fine category. Since the description text sample is a plurality of paragraphs, and in the training process, each image sample is matched with a description text sample, in order to ensure the robustness of the model, a random extraction method of the description text sample is used to randomly select a description text from the plurality of description texts under the fine category corresponding to the image sample as the description text sample of the image sample, and the image sample and the description text sample form an image-text sample pair, thereby obtaining a training sample set including image-text sample pairs of different fine categories.

[0100] It should be noted that the fine-grained category often has a detailed encyclopedia description. By using an automatic crawler program to collect the description text, the image samples of the same fine category are randomly matched based on the crawled description text, which avoids manual labeling of the image samples and greatly reduces the time for model training, thereby improving the efficiency of model training. In addition, the image samples are randomly matched based on the crawled description text, which can make the matching description text content of the image more flexible, diverse and more timely. Training the model with such a sample pair can effectively ensure the robustness of the model.

[0101] In one embodiment of the application, in order to further illustrate and limit, the step of using the trained feature extraction model to respectively extract features from the to-be-identified image and the description text of the target basic category to obtain an image feature vector and a text feature vector includes:

[0102] extracting features from the to-be-identified image using the image encoder to obtain an initial image feature vector, and performing high-dimensional feature mapping on the initial image feature vector to obtain an image feature vector;

[0103] ​Tokenize the target description text by using the text encoder, and perform feature extraction on the tokenized target description text to obtain an initial text feature vector;

[0104] Perform high-dimensional feature mapping on the initial text feature vector to obtain a text feature vector.

[0105] In the embodiment of the application, the trained feature extraction model includes an image encoder and a text encoder. The image encoder can be constructed based on a pre-trained image feature extraction model, such as a ViT (Vision Transformer) network model. The text encoder can be constructed based on a pre-trained tokenizer and a natural language processing model, such as a Tokenizer network model and a BERT (Bidirectional Encoder Representation from Transformers) network model. Since the matrix calculation process requires uniform dimensions, the image encoder and the text encoder each further include a gating unit for mapping the results of the last layer of the image feature extraction model and the results of the natural language processing model to a uniform high-dimensional image-text feature space, thereby obtaining an image feature vector and a text feature vector in the same high-dimensional space. Figure 3 As shown in the figure, the image to be recognized is input into the image branch, and the Vision Transformer network model containing multiple layers of neural networks is used for feature extraction to obtain an initial image feature vector, which is then mapped to a high dimension by the gating unit to obtain an image feature vector. The target description text is input into the image branch, tokenized by the Tokenizer network model, and then the BERT model is used for text feature extraction to obtain an initial text feature vector, which is then mapped to a high dimension by the gating unit to obtain a text feature vector.

[0106] In one embodiment of the application, to further illustrate and limit, the step of obtaining the image to be recognized and the target description text matching the basic category of the image to be recognized includes:

[0107] An image recognition request is obtained, which carries an image to be recognized and image capture terminal information.

[0108] The image capture terminal information is parsed to determine the business attribute corresponding to the image to be recognized, and the target description text is matched from the description texts of different basic categories according to the business attribute.

[0109] In the embodiment of the present application, when image recognition is needed, the terminal device or the image collection device sends an image recognition request to the server. After the current execution subject server receives the image recognition request, it determines the business attribute corresponding to the image content collected by the current image collection terminal from the pre-established mapping relationship between the image collection terminal information and the business attribute according to the image collection terminal information carried by the image recognition request. For example, the image collection terminal information of the Chinese medicine shop is No. 01, and the corresponding business attribute is Chinese medicine sales. The image collection terminal information of the western medicine shop is No. 02, and the corresponding business attribute is western medicine sales. The image collection terminal information of the automatic vending supermarket is No. 03, and the corresponding business attribute is daily commodity sales. According to the business attribute, the corresponding basic category is matched from the pre-set description text basic category. For example, the business attribute is Chinese medicine sales, and the Chinese medicine description text is determined as the target description text. The business attribute is daily commodity sales, and the daily commodity description text is determined as the target description text. The image collection terminal information can be a terminal device number, or network ID information, and the embodiment of the present application is not limited specifically. By pre-determining the basic category of the description text according to the image collection terminal information, the amount of text participating in the calculation in the image comparison process can be reduced, and the matching degree of the description text content and the image is higher, so as to improve the accuracy and efficiency of image recognition.

[0110] In one embodiment of the present application, in order to further illustrate and limit, before the step of obtaining the target description text information matched with the basic category of the to-be-recognized image, the method further comprises:

[0111] Obtaining an expected fine category of an expected recognition image;

[0112] According to the expected fine category, at least one piece of description text corresponding to each of the expected fine categories is crawled from a target website respectively;

[0113] According to the preset basic category, the description texts of different expected fine categories are divided into basic categories to obtain description texts of different basic categories.

[0114] In the embodiment of the present application, the expected recognition image is an image expected to be recognized according to the business demand, and the expected fine category is a fine category expected to be recognized. In some cross applications of multiple scene categories, the business demand of the image can have multiple directions, and the image base categories corresponding to different business demands are different, that is, the expected fine category includes the expected fine category corresponding to different base categories. For example, traditional Chinese medicine, western medicine and daily commodities belong to different base categories, but in the hospital scene, the three categories correspond to business demands. Since there are a large number of fine categories under each base category, if the fine categories under the global base category corresponding to the business demand are all calculated with the to-be-recognized image, the calculation amount will be very large, therefore, after obtaining the expected fine category, the expected fine category is divided according to the base category, so that when facing the to-be-recognized image in a specific business scene, only the description text of the base category meeting the current business scene is extracted for similarity calculation with the to-be-recognized image, thereby reducing the calculation amount and difficulty.

[0115] The present application provides an image fine-grained recognition method, first obtaining a to-be-recognized image and a target description text matched with a base category of the to-be-recognized image, the target description text including description texts under different fine categories; using a trained feature extraction model to extract features of the to-be-recognized image and the target description text respectively to obtain an image feature vector and a text feature vector, the trained image recognition model being obtained based on a network crawled text sample and a joint loss function; determining an image-text similarity matrix based on the image feature vector and the text feature vector, and determining a recognition result of the to-be-recognized image based on the image-text similarity matrix. Compared with the prior art, the embodiment of the present application avoids manual text labeling of image samples by training from a network crawled text sample, greatly reduces the model training time, effectively improves the learning ability of the model based on the joint loss function, and improves the accuracy and efficiency of image recognition based on the contrast learning method.

[0116] Further, as an implementation of the method shown in the above Figure 1 The embodiment of the present application provides an image fine-grained recognition device, as shown in the above Figure 4 The device comprises:

[0117] The acquisition module 31 is configured to acquire a to-be-recognized image and a target description text matched with a base category of the to-be-recognized image, the target description text including description texts under different fine categories.

[0118] extracting module 32, configured to perform feature extraction on the to-be-identified image and the target description text respectively by using the trained feature extraction model to obtain an image feature vector and a text feature vector, the trained image recognition model being trained based on the text samples crawled from the network and the joint loss function;

[0119] determining module 33, configured to determine an image-text similarity matrix based on the image feature vector and the text feature vector, and determine the recognition result of the to-be-identified image based on the image-text similarity matrix.

[0120] Further, the apparatus further comprises:

[0121] The obtaining module 31 is further configured to obtain a training sample set, the training sample set comprising image-text sample pairs under different fine categories, the image-text sample pair comprising an image sample and a piece of description text sample corresponding to the image sample crawled from a target website;

[0122] The first constructing module is configured to construct an initial feature extraction model, the initial feature extraction model comprising an image encoder and a text encoder;

[0123] The training module is configured to construct a joint loss function based on the training sample set, the image encoder and the text encoder, and train the initial feature extraction model by using the joint loss function to obtain the trained feature extraction model.

[0124] Further, the training module comprises:

[0125] The second constructing module is configured to construct a first loss function based on two encoding results of the same image sample by using the image encoder;

[0126] The third constructing module is configured to construct a second loss function based on a similarity matrix of the image sample feature vector and the text sample feature vector and a label mask matrix;

[0127] The fusion module is configured to perform weighted fusion based on the first loss function and the second loss function to obtain the joint loss function.

[0128] Further, the apparatus further comprises:

[0129] The obtaining module 31 is further configured to obtain an expected fine category, and obtain at least one image sample corresponding to each fine category according to the expected fine category;

[0130] The crawling module is configured to crawl at least one piece of description text from a target website according to each of the expected fine categories;

[0131] A fourth construction module is configured to extract any one segment from the description text corresponding to the image sample as a description text sample, and construct an image-text sample pair by combining the description text sample and the image sample, to obtain a training sample set.

[0132] Further, the extraction module 32 comprises:

[0133] A first extraction unit is configured to perform feature extraction on the to-be-identified image by using the image encoder to obtain an initial image feature vector, and perform high-dimensional feature mapping on the initial image feature vector to obtain an image feature vector.

[0134] A processing unit is configured to perform tokenization processing on the target description text by using the text encoder, and perform feature extraction on the tokenized target description text to obtain an initial text feature vector.

[0135] A mapping unit is configured to perform high-dimensional feature mapping on the initial text feature vector to obtain a text feature vector.

[0136] Further, the acquisition module 31 comprises:

[0137] An acquisition unit is configured to acquire an image identification request, wherein the image identification request carries a to-be-identified image and image acquisition terminal information.

[0138] A matching unit is configured to analyze the image acquisition terminal information, determine a business attribute corresponding to the to-be-identified image, and match a target description text from description texts of different basic categories according to the business attribute.

[0139] Further, the apparatus further comprises:

[0140] The acquisition module 31 is further configured to acquire expected fine categories of expected identified images.

[0141] A second crawling module is configured to respectively crawl at least one segment of description text corresponding to each of the expected fine categories from a target website according to the expected fine categories.

[0142] A division module is configured to perform basic category division on the description texts of different expected fine categories according to preset basic categories, to obtain description texts of different basic categories.

[0143] The application provides an image fine-grained recognition device. Firstly, an image to be recognized and a target description text matched with a basic category of the image to be recognized are acquired, and the target description text comprises description texts under different fine categories. A feature extraction model that has been trained is used to perform feature extraction on the image to be recognized and the target description text respectively to obtain an image feature vector and a text feature vector. The image recognition model that has been trained is obtained based on a text sample crawled from a network and a joint loss function. An image-text similarity matrix is determined based on the image feature vector and the text feature vector, and a recognition result of the image to be recognized is determined based on the image-text similarity matrix. Compared with the prior art, the embodiment of the application avoids manual text labeling of image samples by training from a text sample crawled from a network, greatly reduces model training time, effectively improves the learning ability of the model based on training of the model based on a joint loss function, and improves the accuracy and efficiency of image recognition based on a comparison learning method.

[0144] According to an embodiment of the application, a storage medium is provided, and the storage medium stores at least one executable instruction. The computer executable instruction can execute the image fine-grained recognition method in any method embodiment described above.

[0145] Figure 5 A structural schematic diagram of a computer device according to an embodiment of the application is shown, and embodiments of the application do not limit the specific implementation of the computer device.

[0146] As shown in Figure 5 the computer device can include a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0147] The processor 402, the communications interface 404, and the memory 406 can communicate with each other through the communications bus 408.

[0148] The communications interface 404 is configured to communicate with network elements of other devices, such as clients or other servers.

[0149] The processor 402 is configured to execute the program 410, and specifically can execute related steps in the image fine-grained recognition method embodiments described above.

[0150] Specifically, the program 410 can include program code, and the program code includes computer operation instructions.

[0151] The processor 402 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to perform the embodiments of the present application. The computer device can include one or more processors of the same type or different types, such as one or more CPUs and one or more ASICs.

[0152] The memory 406 is configured to store a program 410. The memory 406 can include a high-speed RAM memory, and can further include a non-volatile memory, such as at least one disk memory.

[0153] The program 410 can be specifically configured to enable the processor 402 to perform the following operations:

[0154] Obtaining a to-be-recognized image and a target description text matched with a basic category of the to-be-recognized image, the target description text including description texts under different fine categories;

[0155] Performing feature extraction on the to-be-recognized image and the target description text respectively by using a trained feature extraction model to obtain an image feature vector and a text feature vector, the trained image recognition model being trained based on a network crawled text sample and a joint loss function;

[0156] Determining an image-text similarity matrix based on the image feature vector and the text feature vector, and determining a recognition result of the to-be-recognized image based on the image-text similarity matrix.

[0157] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be realized by using general computing devices, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and can be realized by program codes executable by the computing devices, so that they can be stored in storage devices and executed by the computing devices, and in some cases, the steps shown or described can be executed in different orders, or they can be manufactured into individual integrated circuit modules or a single integrated circuit module. Therefore, the present application is not limited to any specific combination of hardware and software.

[0158] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.

Claims

1. A fine-grained image recognition method, characterized in that, include: Obtain the image to be identified and the target description text that matches the basic category of the image to be identified, wherein the target description text includes description text under different fine categories; The trained feature extraction model is used to extract features from the image to be identified and the target description text respectively, to obtain image feature vector and text feature vector. The trained feature extraction model is obtained by training based on text samples crawled from the Internet and a joint loss function. Based on the image feature vector and the text feature vector, an image-text similarity matrix is ​​determined, and based on the image-text similarity matrix, the recognition result of the image to be recognized is determined; Before using the trained feature extraction model to extract features from the image to be identified and the target description text to obtain image feature vectors and text feature vectors, the method further includes: A training sample set is obtained, comprising image-text sample pairs under different fine categories. Each image-text sample pair includes an image sample and a descriptive text sample corresponding to the image sample, crawled from a target website. The construction process of the training sample set includes: obtaining an expected fine category, and obtaining at least one image sample corresponding to each fine category; crawling at least one descriptive text from the target website for each expected fine category; extracting any segment from the descriptive text corresponding to the image sample as a descriptive text sample, and constructing an image-text sample pair with the image sample to obtain the training sample set. Construct an initial feature extraction model, which includes an image encoder and a text encoder; The joint loss function is constructed based on the training sample set, the image encoder, and the text encoder, specifically including: constructing a first loss function based on the two encoding results of the same image sample by the image encoder; constructing a second loss function based on the similarity matrix and label mask matrix of the image sample feature vector and the text sample feature vector; performing weighted fusion of the first loss function and the second loss function to obtain the joint loss function; and using the joint loss function to train the initial feature extraction model to obtain the trained feature extraction model.

2. The method according to claim 1, characterized in that, The trained feature extraction model includes an image encoder and a text encoder. The trained feature extraction model is used to extract features from the image to be identified and the target description text, respectively, to obtain image feature vectors and text feature vectors, including: The image encoder is used to extract features from the image to be identified to obtain an initial image feature vector, and the initial image feature vector is then subjected to high-dimensional feature mapping to obtain an image feature vector. The target description text is tokenized using the text encoder, and features are extracted from the tokenized target description text to obtain an initial text feature vector. The initial text feature vector is subjected to high-dimensional feature mapping to obtain the text feature vector.

3. The method according to claim 1 or 2, characterized in that, The acquisition of the image to be identified and the target description text that matches the basic category of the image to be identified include: Obtain an image recognition request, wherein the image recognition request carries the image to be recognized and image acquisition terminal information; The image acquisition terminal information is parsed to determine the business attributes corresponding to the image to be identified, and the target description text is matched from the description text of different basic categories based on the business attributes.

4. The method according to claim 3, characterized in that, Before acquiring the image to be identified and the target description text information that matches the basic category of the image to be identified, the method further includes: Obtain the expected fine category of the image to be recognized; Based on the expected detailed categories, crawl at least one descriptive text segment corresponding to each expected detailed category from the target website; The description texts of different expected fine categories are divided into basic categories according to the preset basic categories to obtain description texts of different basic categories.

5. An image fine-grained recognition device, characterized in that, include: The acquisition module is used to acquire an image to be identified and target description text that matches the basic category of the image to be identified, wherein the target description text includes description text under different fine categories; The extraction module is used to extract features from the image to be identified and the target description text using a pre-trained feature extraction model, respectively, to obtain image feature vectors and text feature vectors. The pre-trained feature extraction model is trained based on text samples crawled from the network and a joint loss function. The determining module is used to determine an image-text similarity matrix based on the image feature vector and the text feature vector, and to determine the recognition result of the image to be recognized based on the image-text similarity matrix; The acquisition module is further configured to acquire a training sample set, which includes image-text sample pairs under different fine categories. Each image-text sample pair includes an image sample and a descriptive text sample corresponding to the image sample, crawled from a target website. The process of constructing the training sample set includes: acquiring an expected fine category, and acquiring at least one image sample corresponding to each fine category; crawling at least one descriptive text from the target website for each expected fine category; extracting any segment from the descriptive text corresponding to the image sample as a descriptive text sample, and constructing an image-text sample pair with the image sample to obtain the training sample set. The first construction module is used to construct an initial feature extraction model, which includes an image encoder and a text encoder. The training module is used to construct a joint loss function based on the training sample set, the image encoder, and the text encoder. Specifically, it includes: constructing a first loss function based on the two encoding results of the same image sample by the image encoder; constructing a second loss function based on the similarity matrix and label mask matrix of the image sample feature vector and the text sample feature vector; performing a weighted fusion of the first loss function and the second loss function to obtain the joint loss function; and training the initial feature extraction model using the joint loss function to obtain the trained feature extraction model.

6. A storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the image fine-grained recognition method as described in any one of claims 1-4.

7. A computer device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the image fine-grained recognition method as described in any one of claims 1-4.

Images