Image recognition method, method for training image recognition model and corresponding device

Abstract

The embodiments of this specification provide an image recognition method, a method for training an image recognition model, and a corresponding device. The main technical solutions include: obtaining an image to be recognized; using an image recognition model to determine the image category corresponding to the image to be recognized; wherein the image recognition model extracts the visual feature representation of the image to be recognized; using the visual feature representation of the image to be recognized to match the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the image to be recognized, the fused feature representation including the confidence distribution of the image to be recognized on each attribute word; using the fused feature representation of the image to be recognized to predict the image category corresponding to the image to be recognized. This application can improve the recognition accuracy of image recognition on fine-grained image categories.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of image processing technology, and in particular, to an image recognition method, a method for training an image recognition model, and corresponding devices. Background Art

[0002] The rapid development of image processing technology has driven the emergence of image recognition technology, which has gradually become a key component of the field of artificial intelligence. Image recognition refers to the use of computers to process and analyze images to identify the types of objects contained in the images. Currently, image recognition has been widely applied in various fields, such as face recognition in security, risk person identification in risk control, traffic sign recognition in transportation, and obstacle recognition in autonomous driving.

[0003] Currently, more and more people in the industry are proposing to use deep learning methods for image recognition. However, image recognition based on deep learning methods currently has difficulty distinguishing fine-grained image categories, resulting in poor recognition accuracy. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification disclose an image recognition method, a method and an apparatus for training an image recognition model, so as to improve the recognition accuracy of image recognition on fine-grained image categories.

[0005] According to a first aspect, the present disclosure provides an image recognition method, the method comprising:

[0006] Obtain the image to be recognized;

[0007] An image recognition model is used to determine the image category corresponding to the image to be recognized; wherein the image recognition model extracts a visual feature representation of the image to be recognized; the visual feature representation of the image to be recognized is matched with the feature representation of each attribute word in an attribute vocabulary to obtain a fused feature representation of the image to be recognized, the fused feature representation including the confidence distribution of the image to be recognized on each attribute word; and the image category corresponding to the image to be recognized is predicted using the fused feature representation of the image to be recognized.

[0008] According to an achievable method in an embodiment of the present application, predicting the image category corresponding to the image to be identified by using the fused feature representation of the image to be identified includes:

[0009] Obtaining the image category corresponding to the image to be identified by using the distance between the fused feature representation of the image to be identified and the attribute prototype of each image category;

[0010] The attribute prototype of the image category is obtained by utilizing the fusion feature representation of the support images of the image category.

[0011] According to an achievable manner in an embodiment of the present application, the method further includes: performing, for each image category, the following steps:

[0012] Get the supporting images for this image category;

[0013] For each supporting image, extract the visual feature representation of the supporting image, and use the visual feature representation of the supporting image to match the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the supporting image, which includes the confidence distribution of the supporting image on each attribute word;

[0014] By utilizing the fused feature representations corresponding to the supporting images of the image category, the fused feature representation of the image category is obtained as the attribute prototype of the image category.

[0015] According to an achievable manner in an embodiment of the present application, the visual feature representation of the image to be identified includes a visual feature representation of each image block in the image to be identified;

[0016] Matching the visual feature representation of the image to be identified with the feature representation of each attribute word in the attribute vocabulary includes:

[0017] Calculate the similarity between the visual feature representation of each block and the feature representation of each attribute word in the attribute vocabulary to obtain the similarity vector corresponding to each attribute word;

[0018] For each attribute word, the following steps are performed: a similarity vector corresponding to the attribute word is subjected to cross-attention processing with the visual feature representation of the image to be identified to obtain a visual representation of the attribute word; and a confidence score of the image to be identified on the attribute word is obtained using the feature representation and visual representation of the attribute word.

[0019] The confidence of the image to be identified on each attribute word is used to obtain a fusion feature representation of the image to be identified.

[0020] According to an achievable method in an embodiment of the present application, the image categories include different obstacle categories, or different aircraft categories, or different commodity categories, or different person categories.

[0021] According to a second aspect, a method for training an image recognition model is provided, the method comprising:

[0022] Acquire training data comprising a plurality of training samples, wherein the training samples include image samples and their corresponding image category labels;

[0023] An image recognition model is trained using the training data, wherein the image recognition model extracts a visual feature representation of an input image sample; the visual feature representation of the image sample is matched with the feature representation of each attribute word in an attribute vocabulary to obtain a fused feature representation of the image sample, the fused feature representation including the confidence distribution of the image sample on each attribute word; the fused feature representation of the image sample is used to predict the image category corresponding to the image sample; the training objectives include: minimizing the difference between the image category corresponding to the image sample predicted by the image recognition model and the image category label.

[0024] According to an achievable method in an embodiment of the present application, using the fusion feature representation of the image sample to predict the image category corresponding to the image sample includes:

[0025] The image category corresponding to the image sample is obtained by using the distance between the fusion feature representation of the image sample and the attribute prototype of each image category;

[0026] The attribute prototype of the image category is obtained by utilizing the fusion feature representation of the support images of the image category.

[0027] According to an achievable manner in an embodiment of the present application, the method further includes: performing, for each image category, the following steps:

[0028] Get the supporting images for this image category;

[0029] For each supporting image, extract the visual feature representation of the supporting image, and use the visual feature representation of the supporting image to match the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the supporting image, which includes the confidence distribution of the supporting image on each attribute word;

[0030] By utilizing the fused feature representations corresponding to the supporting images of the image category, the fused feature representation of the image category is obtained as the attribute prototype of the image category.

[0031] According to an achievable manner in an embodiment of the present application, the visual feature representation of the image sample includes a visual feature representation of each image block in the image sample;

[0032] The matching of the visual feature representation of the image sample with the feature representation of each attribute word in the attribute vocabulary includes:

[0033] Calculate the similarity between the visual feature representation of each block and the feature representation of each attribute word in the attribute vocabulary to obtain the similarity vector corresponding to each attribute word;

[0034] For each attribute word, the following steps are performed: cross-attention processing is performed on the similarity vector corresponding to the attribute word and the visual feature representation of the image sample to obtain the visual representation of the attribute word; the confidence of the image sample on the attribute word is obtained using the feature representation and visual representation of the attribute word;

[0035] The confidence of the image sample on each attribute word is used to obtain the fusion feature representation of the image sample.

[0036] According to an achievable method in an embodiment of the present application, the training sample further includes an attribute word label corresponding to the image sample;

[0037] The training goal also includes minimizing the difference between a target attribute word predicted by using the fusion feature representation of the image sample and the attribute word label corresponding to the image sample.

[0038] According to a third aspect, an image recognition device is provided, the device comprising:

[0039] An image acquisition unit, configured to acquire an image to be recognized;

[0040] A category prediction unit is configured to use an image recognition model to determine the image category corresponding to the image to be identified; wherein the image recognition model extracts a visual feature representation of the image to be identified; uses the visual feature representation of the image to be identified to match the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the image to be identified, and the fused feature representation includes the confidence distribution of the image to be identified on each attribute word; and uses the fused feature representation of the image to be identified to predict the image category corresponding to the image to be identified.

[0041] According to a fourth aspect, a device for training an image recognition model is provided, the device comprising:

[0042] A sample acquisition unit is configured to acquire training data comprising a plurality of training samples, wherein the training samples include image samples and their corresponding image category labels;

[0043] A model training unit is configured to train an image recognition model using the training data, wherein the image recognition model extracts a visual feature representation of an input image sample; matches the visual feature representation of the image sample with the feature representation of each attribute word in an attribute vocabulary to obtain a fused feature representation of the image sample, the fused feature representation including the confidence distribution of the image sample on each attribute word; and predicts the image category corresponding to the image sample using the fused feature representation of the image sample; the training objectives include minimizing the difference between the image category corresponding to the image sample predicted by the image recognition model and the image category label.

[0044] According to a fifth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described above.

[0045] According to a sixth aspect, the present disclosure provides a computing device, comprising a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the method described above is implemented.

[0046] It can be seen from the above technical solutions that the present disclosure can have the following advantages:

[0047] 1) The present invention assists image recognition by fusing attribute words in an attribute vocabulary with the visual features of an image, thereby enhancing the ability to understand image semantics and the ability to distinguish fine-grained image categories, improving recognition accuracy, and having more prominent advantages in recognizing fine-grained image categories.

[0048] 2) The feature extraction method provided by the present disclosure, which integrates attribute words as auxiliary modalities into visual feature representation, can be well applied to the knowledge transfer process of small sample image recognition implemented by prototype networks, thereby improving the recognition accuracy of small sample images and the generalization of the model.

[0049] 3) The feature extraction method provided by the present disclosure, which integrates attribute words as auxiliary modalities into the visual feature representation, can extract the fusion features of attribute and visual modalities for both the support image and the query image (i.e., the image to be identified), thereby ensuring a full semantic understanding of the query image and the support image during the meta-learning process and improving the recognition accuracy.

[0050] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 An exemplary system architecture diagram to which the embodiments of the present disclosure may be applied is shown;

[0053] Figure 2 A flowchart of an image recognition method provided in an embodiment of the present disclosure;

[0054] Figure 3A flow chart of a feature extraction method provided in an embodiment of the present disclosure;

[0055] Figure 4 A schematic diagram of the principle of an image recognition model provided in an embodiment of the present disclosure;

[0056] Figure 5 A flowchart of a method for training an image recognition model provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of the principle of a training image recognition model provided in an embodiment of the present disclosure;

[0058] Figure 7 A structural diagram of an image recognition device provided in an embodiment of the present disclosure;

[0059] Figure 8 A structural diagram of the apparatus for training an image recognition model provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0061] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0062] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0063] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0064] In order to facilitate the understanding of this application, the system architecture to which this application is applicable is first briefly described. Figure 1 An exemplary system architecture to which the embodiments of the present application can be applied is shown. Figure 1 As shown in , the system architecture includes a model training device and an image recognition device.

[0065] After the model training device obtains the training data of multiple training samples, it uses the method provided in the embodiment of the present application to perform model training in the offline stage to obtain an image recognition model.

[0066] In the online stage, the image recognition device uses the established image recognition model to recognize the image to be recognized and obtain the image category.

[0067] The model training device and image recognition device can be installed on separate servers, on the same server or server group, or on separate or shared cloud servers. A cloud server, also known as a cloud computing server or cloud host, is a host product within the cloud computing service ecosystem. It addresses the management difficulties and poor scalability of traditional physical hosts and virtual private servers (VPs). The model training device and image recognition device can also be installed on a computer terminal with strong computing power.

[0068] It should be noted that, in addition to performing image recognition online, the above-mentioned image recognition device may also perform image recognition offline, for example, performing image recognition on batches of images to be recognized.

[0069] It should be understood that Figure 1 The number of model training devices, image recognition devices, and image recognition models in the embodiment is merely illustrative. Any number of model training devices, image recognition devices, and image recognition models may be provided as required.

[0070] Figure 2 This is a flow chart of the image recognition method provided by the embodiment of the present disclosure. It can be understood that the method can be Figure 1 The image recognition device in the system shown is used to perform the operation. Figure 2 , the method may include the following steps:

[0071] Step 201: Obtain an image to be recognized.

[0072] Step 203: Determine the image category corresponding to the image to be identified using the image recognition model; wherein the image recognition model extracts a visual feature representation of the image to be identified; matches the visual feature representation of the image to be identified with the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the image to be identified, the fused feature representation including the confidence distribution of the image to be identified on each attribute word; and predict the image category corresponding to the image to be identified using the fused feature representation of the image to be identified.

[0073] It can be seen from the technical content provided by the above embodiments that the present disclosure assists image recognition by fusing the attribute words in the attribute vocabulary with the visual features of the image, thereby enhancing the ability to understand the semantics of the image and the ability to distinguish between fine-grained image categories, thereby improving recognition accuracy.

[0074] The following describes each step in the above process in detail. The image to be identified obtained in step 201 can be an image captured from the Internet, an image obtained from a server or database, or an image obtained from a user terminal. The purpose of this application is to identify a specific image category from the image to be identified.

[0075] Depending on the application scenario, the target contained in the image to be identified is different. For example, in the field of bird identification, the image to be identified may contain birds, and the task of the image recognition device is to identify the specific bird type from the image to be identified. For another example, in the field of transportation, the image to be identified may contain obstacles (such as vehicles), and the task of the image recognition device is to identify the specific type of obstacle from the image to be identified. For another example, in the field of commodity identification, the image to be identified contains commodities, and the task of the image recognition device is to identify the specific type of commodity from the image to be identified.

[0076] The following describes in detail the above step 203 , ie, “determining the image category corresponding to the image to be recognized by using the image recognition model”, in conjunction with an embodiment.

[0077] In traditional image recognition methods, visual feature representations are mainly extracted from the image to be recognized, and the image category is predicted based on the visual feature representations. However, visual feature representations are difficult to achieve high accuracy in fine-grained image features and have poor transferability. The image recognition model involved in the embodiments of the present disclosure provides a new feature extraction method, such as Figure 3 As shown in , it mainly includes the following steps:

[0078] Step 301: extracting visual feature representations of the image to be recognized, including visual feature representations of each image block.

[0079] When extracting the visual feature representation of the image to be recognized, any image encoding module can be used to encode visual features such as color, texture, and SIFT (Scale-invariant feature transform) to obtain the visual feature representation. The image encoding module can use a bag-of-visual-words model or ViT (Vision Transformer).

[0080] More specifically, the image to be identified can be segmented into tiles of a preset size. Visual feature representations are then extracted for each tile to obtain a visual feature representation for each tile. When encoding each tile, the image coding network can further encode the positional information of each tile in addition to the aforementioned visual features to obtain a visual feature representation for each tile. Since extracting visual feature representations is an existing technology, it will not be discussed in detail here.

[0081] For the image to be identified X, encode it to obtain the visual feature representation F of the image to be identified V , the visual features of each block are expressed as: V = {v1,v2,…,v R}, where v r represents the visual feature representation of the rth block of the image to be recognized, and R is the total number of blocks in the image to be recognized.

[0082] The following steps are the specific process of matching the visual feature representation of the image to be identified with the feature representation of each attribute word in the attribute vocabulary.

[0083] Step 303: Calculate the similarity between the visual feature representation of each image block and the feature representation of each attribute word in the attribute vocabulary to obtain a similarity vector corresponding to each attribute word.

[0084] In the disclosed embodiment, an attribute word library can be first constructed for the image category space corresponding to the image recognition model (including the image categories that the image recognition model needs to recognize). The attribute word library contains attribute words for all image categories in the image category space. These attribute words actually describe the attribute characteristics of different image categories in the form of text.

[0085] For example, consider an image recognition model used for fine-grained bird identification. The image category space contains 200 bird species. While the differences between these species are subtle, their attributes can be expressed through descriptions using attribute words. For example, the attribute word library might include terms like true feathers, down feathers, large size, small size, red beak, short beak, short and rounded wings, sharp claws, black wing feathers, red wing spots, and crimson eye circles. Different bird types can be described using different sets of attribute words. Matching the visual feature representation with the feature representations of each attribute word in the attribute word library is essentially the process of determining the attribute word set that corresponds to the image to be identified.

[0086] Each attribute word in the attribute word library can be pre-encoded to obtain the feature representation of each attribute word. Assume that the feature representation of each attribute word in the attribute word library is D = {a1, a2, ..., a M}, where a mThe similarity between the visual feature representation of the rth image block and the feature representation of the mth attribute word can be calculated using the following formula:

[0087]

[0088] Wherein, the superscript T represents transposition, and W1 is the model parameter of the image recognition model.

[0089] Step 305: Execute for each attribute word separately: perform cross-attention processing on the similarity vector corresponding to the attribute word and the visual feature representation of the image to be identified to obtain the visual representation of the attribute word; use the feature representation and visual representation of the attribute word to obtain the confidence of the image to be identified on the attribute word.

[0090] Cross-attention processing of the similarity vector corresponding to the attribute word and the visual feature representation of the image to be identified is essentially a process of associating the visual features of the image to be identified with the attribute words that the image to be identified may have, thereby obtaining the possibility that the image to be identified corresponds to each attribute word.

[0091] For example, the visual representation F of the mth attribute word m It can be:

[0092]

[0093] The confidence Ω of the image to be identified X on the mth attribute word m It can be:

[0094]

[0095] W2 is the model parameter of the image recognition model.

[0096] The final Ω m In fact, it is the possibility that the image to be identified corresponds to the mth attribute word, and it also reflects the possibility that the target in the image to be identified has the mth attribute word.

[0097] Step 307: Using the confidence of the image to be identified on each attribute word, a fusion feature representation of the image to be identified is obtained.

[0098] For the image X to be identified, the confidence distribution Ω(X) on each attribute word in the attribute dictionary can be obtained:

[0099] Ω(X)={Ω1,Ω2,...,Ω M}

[0100] The above confidence distribution Ω(X) can be used as the fusion feature representation of the image to be identified.

[0101] Continue to see Figure 2 In step 203, when using the fused feature representation of the image to be identified to predict the image category corresponding to the image to be identified, a classification module can be used to map the fused feature representation of the image to be identified to a specific image category, that is, to obtain the probability values ​​of the image to be identified corresponding to each image category, and the image category with the highest probability value is used as the predicted image category corresponding to the image to be identified.

[0102] Ordinary deep learning methods are extremely dependent on large-scale annotated data, and this defect greatly limits the application of deep learning methods in actual image recognition tasks. In response to this problem, more and more researchers have begun to study how to solve image recognition problems based on a small number of image annotated samples. Most of the current image recognition methods achieve image recognition by learning the visual features contained in the image, but when visual features are used as transferable knowledge for recognition on small sample categories, the accuracy is poor. However, the feature extraction method of incorporating attribute words into the visual feature representation provided in the above embodiments of the present disclosure can be well applied to the knowledge transfer process of small sample image recognition.

[0103] Taking the common Prototypical Network (ProtoNet) as an example, support images can be pre-obtained for each image category. For each image category, all support images have the target corresponding to that image category. For example, for the image category "sand finches," all support images include sand finches; for the image category "grouse," all support images include grouse.

[0104] like Figure 4 As shown in , for each support image, a fusion feature representation of each support image is extracted. That is, for each support image, the visual feature representation of the support image is extracted, and the visual feature representation of the support image is matched with the feature representation of each attribute word in the attribute word library to obtain the fusion feature representation of the support image. The fusion feature representation includes the confidence distribution of the support image on each attribute word. Figure 3 The feature extraction process shown is the same as that of the identified image, so it will not be repeated here.

[0105] Then, using the fused feature representations corresponding to the supporting images of the image category, the fused feature representation of the image category is obtained as the attribute prototype of the image category. The so-called prototype (Protype) is a concept involved in the prototype network, which refers to the use of a vector to represent each category.

[0106] The method used is to cluster the fused feature representations of the supporting images of an image category and take the fused feature representation of the cluster centroid (i.e., cluster center) as the attribute prototype of the image category. Alternatively, the fused feature representations of the supporting images of an image category can be averaged and the obtained fused feature representation can be used as the attribute prototype of the image category. For example, the attribute prototype p of image category n is n It can be:

[0107]

[0108] is the support image set of image category n, s i for The i-th support image in , K is the number of support images corresponding to image category n.

[0109] The image category corresponding to the image to be identified is obtained by using the distance between the fused feature representation Ω(X) of the image to be identified and the attribute prototype of each image category. For example, the probability distribution of the image to be identified in each image category can be predicted based on the softmax of the distance between the fused feature representation Ω(X) of the image to be identified and the attribute prototype:

[0110]

[0111] in, is the predicted image category, d() is the distance function, and N is the total number of image categories.

[0112] Figure 5 This is a flow chart of a method for training an image recognition model provided in an embodiment of the present application. The method can be performed by Figure 1 The model training device in the system shown is executed. Figure 5 As shown in , the method may include the following steps:

[0113] Step 501: Acquire training data containing a plurality of training samples, where the training samples include image samples and their corresponding image category labels.

[0114] Step 503: Using the training data to train the image recognition model, wherein the image recognition model extracts a visual feature representation of the input image sample; using the visual feature representation of the image sample to match the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the image sample, the fused feature representation includes the confidence distribution of the image sample on each attribute word; using the fused feature representation of the image sample to predict the image category corresponding to the image sample; the training objectives include: minimizing the difference between the image category corresponding to the image sample predicted by the image recognition model and the image category label.

[0115] The image recognition model uses the visual feature representation of the image sample to match the feature representation of each attribute word in the attribute word library to obtain the fusion feature representation. Figure 3 The relevant records are not repeated here.

[0116] As one of the feasible ways, a classification module can be used in the image recognition model, and the classification module uses the fused feature representation of the image sample to predict the image category corresponding to the image sample. That is, the classification module maps the fused feature representation of the image sample to the specific image category, and obtains the probability value of the image sample corresponding to each image category, and takes the image category with the highest probability value as the image category corresponding to the predicted image sample.

[0117] The training goal is to make the image classification predicted by the image recognition model as close as possible to the image classification label of the image sample. A loss function, such as the cross-entropy loss function, can be constructed based on this training goal. In each iteration, the value of the loss function is used to update the model parameters using methods such as gradient descent until the preset training termination conditions are met. These termination conditions may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0118] If a prototype network is used to solve the problem of small sample category recognition and knowledge transfer, a set of support images can be obtained in advance for each image category, and a meta-learning approach can be adopted. In this case, a fused feature representation of each support image can be extracted for each support image. That is, for each support image, the visual feature representation of the support image is extracted, and the visual feature representation of the support image is matched with the feature representation of each attribute word in the attribute word library to obtain a fused feature representation of the support image. The fused feature representation includes the confidence distribution of the support image on each attribute word. This part is related to Figure 3 The feature extraction process shown is the same as that of the identified image, so it will not be repeated here.

[0119] During the training process, when using the fused feature representation of the image sample to predict the image category corresponding to the image sample, the distance between the fused feature representation of the image sample and the attribute prototype of each image category can be used to obtain the image category corresponding to the image sample.

[0120] The attribute prototype of an image category is obtained by using the fused feature representations of the supporting images of that image category. For example, after clustering the fused feature representations of the supporting images of an image category, the fused feature representation of the cluster centroid (i.e., cluster center) is taken as the attribute prototype of the image category. Alternatively, the fused feature representations of the supporting images of an image category are averaged, and the resulting fused feature representation is used as the attribute prototype of the image category.

[0121] When training a model based on this, attribute prediction tasks can be used to assist in model training. In this case, in addition to image samples and their corresponding image category labels, training samples can also include attribute word labels corresponding to the image samples. That is, the image samples can be labeled with the attribute words of the objects they contain. These attribute words come from the attribute dictionary.

[0122] like Figure 6 As shown in , during the training process, target attribute words can be predicted based on the fused feature representation of the image sample, that is, the image sample is predicted to contain the target attribute word. This prediction can be implemented by a decoding module. Based on this, a further training objective can be included: minimizing the difference between the predicted target attribute word and the attribute word label corresponding to the image sample.

[0123] The loss function L can be pre-constructed based on the above two training objectives CLS and L AG In each iteration, the total loss function is used to update the model parameters using methods such as gradient descent until the preset training termination conditions are met. The training termination conditions may include the total loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0124] For example, the total loss function L can be:

[0125] L=L CLS +λL AG (6)

[0126] Among them, λ is the preset weight coefficient.

[0127]

[0128] Among them, q j is the jth image sample, and Q is the number of image samples. j For q j Annotated image category labels, For q j Predicted image category.

[0129]

[0130] Among them, z i For s i The confidence distribution formed by the annotated attribute words, in which the confidence corresponding to the annotated attribute words is 1, and the confidence corresponding to other attribute words is 0. j For q j The confidence distribution formed by the labeled attribute words, in which the confidence corresponding to the labeled attribute words is 1, and the confidence corresponding to other attribute words is 0.

[0131] The above method provided in this application can be applied to a variety of application scenarios. For example, in the field of autonomous driving, obstacle recognition is performed on images collected by autonomous driving vehicles to identify specific obstacle categories, thereby providing a reference for the control strategy of autonomous driving vehicles. For another example, in the aviation field, specific aircraft categories are identified for collected images to provide a reference for aviation control. For another example, on an e-commerce platform, specific product categories are identified for product images uploaded by users or product images uploaded by merchants, thereby providing a reference for product recommendations, product information feedback, product listing, etc. For another example, on a picture platform, images containing people uploaded by users or images containing people in a database are identified to identify specific types of people, thereby providing a reference for infringement review, sensitive person review, etc. And so on.

[0132] The above is a detailed description of the method provided by the present disclosure. The following is a detailed description of the device provided by the present disclosure.

[0133] Figure 7 A structural diagram of an image recognition device according to an embodiment of the present disclosure is shown as follows: Figure 7 As shown, the apparatus 700 may include: an image acquisition unit 701 and a category prediction unit 702. The main functions of each component unit are as follows:

[0134] The image acquisition unit 701 is configured to acquire an image to be recognized.

[0135] The category prediction unit 702 is configured to use an image recognition model to determine the image category corresponding to the image to be identified; wherein the image recognition model extracts a visual feature representation of the image to be identified; uses the visual feature representation of the image to be identified to match the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the image to be identified, and the fused feature representation includes the confidence distribution of the image to be identified on each attribute word; and uses the fused feature representation of the image to be identified to predict the image category corresponding to the image to be identified.

[0136] As one possible implementation method, the visual feature representation of the image to be identified includes the visual feature representation of each image block in the image to be identified.

[0137] When the image recognition model uses the visual feature representation of the image to be recognized to match the feature representation of each attribute word in the attribute vocabulary, it can calculate the similarity between the visual feature representation of each block and the feature representation of each attribute word in the attribute vocabulary respectively, and obtain the similarity vector corresponding to each attribute word; execute for each attribute word respectively: cross-attention processing is performed on the similarity vector corresponding to the attribute word and the visual feature representation of the image to be recognized to obtain the visual representation of the attribute word; use the feature representation and visual representation of the attribute word to obtain the confidence of the image to be recognized on the attribute word; use the confidence of the image to be recognized on each attribute word to obtain the fused feature representation of the image to be recognized.

[0138] As one of the feasible ways, when an image recognition model uses the fused feature representation of an image to be recognized to predict the image category corresponding to the image to be recognized, it can use the distance between the fused feature representation of the image to be recognized and the attribute prototype of each image category to obtain the image category corresponding to the image to be recognized; wherein the attribute prototype of the image category is obtained by using the fused feature representation of the supporting images of the image category.

[0139] In this scenario, the image recognition model further performs the following steps for each image category: obtaining supporting images for the image category; extracting the visual feature representation of each supporting image, and using the visual feature representation of the supporting image to match the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the supporting image, which includes the confidence distribution of the supporting image on each attribute word; using the fused feature representation corresponding to each supporting image of the image category, the fused feature representation of the image category is obtained as the attribute prototype of the image category.

[0140] Figure 8 A diagram showing the structure of a device for training an image recognition model according to an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the apparatus 800 may include: a sample acquisition unit 801 and a model training unit 802. The main functions of each component unit are as follows:

[0141] The sample acquisition unit 801 is configured to acquire training data including a plurality of training samples, where the training samples include image samples and their corresponding image category labels.

[0142] The model training unit 802 is configured to train an image recognition model using training data, wherein the image recognition model extracts a visual feature representation of an input image sample; matches the visual feature representation of the image sample with the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the image sample, the fused feature representation including the confidence distribution of the image sample on each attribute word; and predicts the image category corresponding to the image sample using the fused feature representation of the image sample; the training objectives include: minimizing the difference between the image category corresponding to the image sample predicted by the image recognition model and the image category label.

[0143] As one of the possible implementation methods, the visual feature representation of the above-mentioned image sample includes the visual feature representation of each block in the image sample. When the image recognition model uses the visual feature representation of the image sample to match the feature representation of each attribute word in the attribute vocabulary, it can respectively calculate the similarity between the visual feature representation of each block and the feature representation of each attribute word in the attribute vocabulary to obtain the similarity vector corresponding to each attribute word; and perform the following for each attribute word: cross-attention processing of the similarity vector corresponding to the attribute word and the visual feature representation of the image sample to obtain the visual representation of the attribute word; using the feature representation and visual representation of the attribute word, obtain the confidence of the image sample on the attribute word; and using the confidence of the image sample on each attribute word, obtain the fused feature representation of the image sample.

[0144] As one of the feasible ways, the image recognition model can be implemented based on the prototype network. That is, when using the fused feature representation of the image sample to predict the image category corresponding to the image sample, the distance between the fused feature representation of the image sample and the attribute prototype of each image category can be used to obtain the image category corresponding to the image sample; wherein the attribute prototype of the image category is obtained by using the fused feature representation of the supporting image of the image category.

[0145] In this way, the image recognition model can be executed separately for each image category: obtain the supporting images of the image category; for each supporting image, extract the visual feature representation of the supporting image, and use the visual feature representation of the supporting image to match the feature representation of each attribute word in the attribute vocabulary to obtain the fused feature representation of the supporting image, which includes the confidence distribution of the supporting image on each attribute word; use the fused feature representation corresponding to each supporting image of the image category to obtain the fused feature representation of the image category as the attribute prototype of the image category.

[0146] Furthermore, when training models based on this foundation, attribute prediction tasks can be used to assist in model training. The aforementioned training samples can also include attribute word labels corresponding to the image samples. In this case, the training objective also includes minimizing the difference between the target attribute word predicted using the fused feature representation of the image sample and the attribute word label corresponding to the image sample.

[0147] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0148] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0149] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the aforementioned method embodiments.

[0150] And an electronic device comprising:

[0151] one or more processors; and

[0152] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0153] The present application also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods described in the aforementioned method embodiments when executed by a processor.

[0154] The memory may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc.

[0155] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0156] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image recognition method, characterized in that: The method comprises: An attribute word library is pre-built for the image category space corresponding to the image recognition model; wherein the image category space includes: different fine-grained image categories within the same category that the image recognition model corresponding to the image category space needs to recognize; wherein the attribute word library includes attribute words for all image categories in the image category space, and the attribute words describe the attribute characteristics of different fine-grained image categories within the same category in the form of text; wherein different types of image categories are described by different sets of attribute words; Obtain the image to be recognized; Determining the image category corresponding to the image to be identified using an image recognition model; wherein the image recognition model extracts a visual feature representation of the image to be identified; matching the visual feature representation of the image to be identified with the feature representations of each attribute word in an attribute vocabulary to obtain a fused feature representation of the image to be identified, the fused feature representation including the confidence distribution of the image to be identified on each attribute word, and predicting the image category corresponding to the image to be identified using the fused feature representation of the image to be identified; The visual feature representation of the image to be identified includes the visual feature representation of each image block in the image to be identified; The step of matching the visual feature representation of the image to be identified with the feature representations of each attribute word in the attribute vocabulary to obtain a fused feature representation of the image to be identified includes: Calculate the similarity between the visual feature representation of each block and the feature representation of each attribute word in the attribute vocabulary to obtain the similarity vector corresponding to each attribute word; For each attribute word, the following steps are performed: a similarity vector corresponding to the attribute word is subjected to cross-attention processing with the visual feature representation of the image to be identified to obtain a visual representation of the attribute word; and a confidence score of the image to be identified on the attribute word is obtained using the feature representation and visual representation of the attribute word. The confidence of the image to be identified on each attribute word is used to obtain a fusion feature representation of the image to be identified.

2. The method according to claim 1, characterized in that Predicting the image category corresponding to the image to be identified by using the fused feature representation of the image to be identified includes: Obtaining the image category corresponding to the image to be identified by using the distance between the fused feature representation of the image to be identified and the attribute prototype of each image category; The attribute prototype of the image category is obtained by utilizing the fusion feature representation of the support images of the image category.

3. The method according to claim 2, characterized in that The method further comprises: performing, for each image category, respectively: Get the supporting images for this image category; For each supporting image, extract the visual feature representation of the supporting image, and use the visual feature representation of the supporting image to match the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the supporting image, which includes the confidence distribution of the supporting image on each attribute word; By utilizing the fused feature representations corresponding to the supporting images of the image category, the fused feature representation of the image category is obtained as the attribute prototype of the image category.

4. The method according to any one of claims 1 to 3, characterized in that The image categories include different obstacle categories, or different aircraft categories, or different commodity categories, or different person categories.

5. A method for training an image recognition model, characterized in that: The method comprises: An attribute word library is pre-built for the image category space corresponding to the image recognition model; wherein the image category space includes: different fine-grained image categories within the same category that the image recognition model corresponding to the image category space needs to recognize; wherein the attribute word library includes attribute words for all image categories in the image category space, and the attribute words describe the attribute characteristics of different fine-grained image categories within the same category in the form of text; wherein different types of image categories are described by different sets of attribute words; Acquire training data comprising a plurality of training samples, wherein the training samples include image samples and their corresponding image category labels; An image recognition model is trained using the training data, wherein the image recognition model extracts a visual feature representation of an input image sample; the visual feature representation of the image sample is matched with the feature representation of each attribute word in an attribute vocabulary to obtain a fused feature representation of the image sample, the fused feature representation including the confidence distribution of the image sample on each attribute word; the fused feature representation of the image sample is used to predict the image category corresponding to the image sample; the training objective includes minimizing the difference between the image category corresponding to the image sample predicted by the image recognition model and the image category label; The visual feature representation of the image sample includes the visual feature representation of each block in the image sample; The method of obtaining a fused feature representation of the image sample by matching the visual feature representation of the image sample with the feature representation of each attribute word in the attribute vocabulary includes: Calculate the similarity between the visual feature representation of each block and the feature representation of each attribute word in the attribute vocabulary to obtain the similarity vector corresponding to each attribute word; For each attribute word, the following steps are performed: cross-attention processing is performed on the similarity vector corresponding to the attribute word and the visual feature representation of the image sample to obtain the visual representation of the attribute word; the confidence of the image sample on the attribute word is obtained using the feature representation and visual representation of the attribute word; The confidence of the image sample on each attribute word is used to obtain the fusion feature representation of the image sample.

6. The method according to claim 5, characterized in that The image categories corresponding to the image samples predicted by using the fusion feature representation of the image samples include: The image category corresponding to the image sample is obtained by using the distance between the fusion feature representation of the image sample and the attribute prototype of each image category; The attribute prototype of the image category is obtained by utilizing the fusion feature representation of the support images of the image category.

7. The method according to claim 6, characterized in that The method further comprises: performing, for each image category, respectively: Get the supporting images for this image category; For each supporting image, extract the visual feature representation of the supporting image, and use the visual feature representation of the supporting image to match the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the supporting image, which includes the confidence distribution of the supporting image on each attribute word; By utilizing the fused feature representations corresponding to the supporting images of the image category, the fused feature representation of the image category is obtained as the attribute prototype of the image category.

8. The method according to claim 5, characterized in that The visual feature representation of the image sample includes a visual feature representation of each image block in the image sample; The matching of the visual feature representation of the image sample with the feature representation of each attribute word in the attribute vocabulary includes: Calculate the similarity between the visual feature representation of each block and the feature representation of each attribute word in the attribute vocabulary to obtain the similarity vector corresponding to each attribute word; For each attribute word, perform the following steps: cross-attention processing on the similarity vector corresponding to the attribute word and the visual feature representation of the image sample to obtain the visual representation of the attribute word; use the feature representation and visual representation of the attribute word to obtain the confidence of the image sample on the attribute word.

9. The method according to claim 6, characterized in that The training samples also include attribute word labels corresponding to the image samples; The training goal also includes minimizing the difference between a target attribute word predicted by using the fusion feature representation of the image sample and the attribute word label corresponding to the image sample.

10. An image recognition device, characterized in that: The device comprises: An image acquisition unit, configured to acquire an image to be recognized; a category prediction unit configured to determine the image category corresponding to the image to be identified using an image recognition model; wherein the image recognition model extracts a visual feature representation of the image to be identified; matches the visual feature representation of the image to be identified with the feature representations of each attribute word in an attribute vocabulary to obtain a fused feature representation of the image to be identified, the fused feature representation including the confidence distribution of the image to be identified on each attribute word, and thereby predict the image category corresponding to the image to be identified using the fused feature representation of the image to be identified; The attribute word library is pre-constructed for the image category space corresponding to the image recognition model; the image category space includes: different fine-grained image categories within the same category that the image recognition model corresponding to the image category space needs to recognize; the attribute word library includes attribute words for all image categories in the image category space, and the attribute words describe the attribute characteristics of different fine-grained image categories within the same category in the form of text; different types of image categories are described by different sets of attribute words; The visual feature representation of the image to be identified includes the visual feature representation of each block in the image to be identified; When the image recognition model uses the visual feature representation of the image to be recognized to match the feature representation of each attribute word in the attribute vocabulary to obtain the fused feature representation of the image to be recognized, the model is configured to execute: respectively calculate the similarity between the visual feature representation of each block and the feature representation of each attribute word in the attribute vocabulary to obtain the similarity vector corresponding to each attribute word; respectively execute for each attribute word: cross-attention processing of the similarity vector corresponding to the attribute word and the visual feature representation of the image to be recognized to obtain the visual representation of the attribute word; use the feature representation and visual representation of the attribute word to obtain the confidence of the image to be recognized on the attribute word; use the confidence of the image to be recognized on each attribute word to obtain the fused feature representation of the image to be recognized.

11. A device for training an image recognition model, characterized in that: The device comprises: A sample acquisition unit is configured to acquire training data comprising a plurality of training samples, wherein the training samples include image samples and their corresponding image category labels; A model training unit is configured to train an image recognition model using the training data, wherein the image recognition model extracts a visual feature representation of an input image sample; matches the visual feature representation of the image sample with the feature representation of each attribute word in an attribute vocabulary to obtain a fused feature representation of the image sample, the fused feature representation including a confidence distribution of the image sample on each attribute word; and predict an image category corresponding to the image sample using the fused feature representation of the image sample; wherein the training objective includes minimizing the difference between the image category corresponding to the image sample predicted by the image recognition model and the image category label; The attribute word library is pre-constructed for the image category space corresponding to the image recognition model; the image category space includes: different fine-grained image categories within the same category that the image recognition model corresponding to the image category space needs to recognize; the attribute word library includes attribute words for all image categories in the image category space, and the attribute words describe the attribute characteristics of different fine-grained image categories within the same category in the form of text; different types of image categories are described by different sets of attribute words; The visual feature representation of the image sample includes a visual feature representation of each image block in the image sample; When the image recognition model uses the visual feature representation of the image sample to match the feature representation of each attribute word in the attribute vocabulary to obtain a fused feature representation of the image sample, it is configured to execute: respectively calculate the similarity between the visual feature representation of each block and the feature representation of each attribute word in the attribute vocabulary to obtain a similarity vector corresponding to each attribute word; respectively execute for each attribute word: cross-attentionally process the similarity vector corresponding to the attribute word and the visual feature representation of the image sample to obtain a visual representation of the attribute word; use the feature representation and visual representation of the attribute word to obtain the confidence of the image sample on the attribute word; use the confidence of the image sample on each attribute word to obtain a fused feature representation of the image sample.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 9.

13. A computing device comprising a memory and a processor, characterized in that: The memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 9 is implemented.

Images