Small sample fine-grained image recognition method, device and equipment and storage medium

Abstract

The invention relates to a small sample fine-grained image recognition method and device, equipment and a storage medium. The method comprises the following steps: acquiring a training sample, wherein the training sample comprises a plurality of images; according to the training sample, training the initial recognition model to obtain a trained recognition model; the trained recognition model is adopted to recognize a small sample fine-grained image to be recognized, and an image category is obtained; wherein the step of training the initial recognition model comprises the steps of inputting a training sample into the initial recognition model, and generating a global attention graph based on a double-flow attention mechanism; and constructing a multi-path joint optimization module, and optimizing parameters of the initial recognition model according to the global attention map until a preset condition is met, thereby obtaining a trained recognition model. The identification model is used to identify the small sample fine-grained image to be identified, fine inter-class differences in the fine-grained image can be accurately identified, and a good identification result is obtained.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a small sample fine-grained image recognition method, device, equipment and storage medium. Background Art

[0002] With the rise of deep learning, the performance of visual recognition tasks has made rapid progress. Fine-grained recognition, as a basic task in visual recognition, plays a vital role in practical applications. The purpose of fine-grained recognition is to distinguish different subcategories under the same category (such as CUB, Stanford Cars and Stanford Dogs datasets).

[0003] However, existing fine-grained image recognition models often rely on a large amount of manually labeled data, which greatly limits the ability of these models to identify new objects with only a small number of labeled samples. Small-sample learning can effectively alleviate the burden of manual annotation brought by a large amount of data. Its purpose is to identify new categories with only a limited number of samples (such as 1 sample or 5 samples) by transferring knowledge learned from past experience. Unlike general images with significant category differences, the subtle inter-class differences and large intra-class differences in fine-grained images make the small-sample fine-grained image recognition task more difficult. Summary of the invention

[0004] Based on this, it is necessary to provide a small sample fine-grained image recognition method, device, equipment and storage medium to address the above technical problems.

[0005] In a first aspect, the present application provides a small sample fine-grained image recognition method. The method comprises:

[0006] Acquire a training sample, wherein the training sample includes a plurality of images;

[0007] According to the training samples, an initial recognition model is trained to obtain a trained recognition model;

[0008] Use the trained recognition model to identify the small sample fine-grained image to obtain the image category;

[0009] The step of training the initial recognition model includes:

[0010] Input the training sample into the initial recognition model, and generate a global attention map based on a dual-stream attention mechanism;

[0011] A multi-path joint optimization module is constructed, and the parameters of the initial recognition model are optimized according to the global attention map until the preset conditions are met to obtain a trained recognition model.

[0012] In one embodiment, inputting the training sample into the initial recognition model and generating a global attention map based on a dual-stream attention mechanism includes:

[0013] Based on a weight-sharing network, encoding each of the images in the training sample to obtain a three-dimensional tensor feature of each image;

[0014] Based on the dual-stream attention mechanism, constructing feature relationship pairs of the three-dimensional tensor features, and generating a global attention map based on the feature relationship pairs;

[0015] Among them, the dual-stream attention mechanism includes spatial attention and channel attention.

[0016] In one embodiment, constructing feature relationship pairs of the three-dimensional tensor features based on the dual-stream attention mechanism, and generating a global attention map based on the feature relationship pairs, includes:

[0017] Based on the spatial attention, constructing a spatial pair relationship of each of the three-dimensional tensor features, reshaping and splicing the spatial pair relationship, and obtaining a spatial attention map based on the spatial flow;

[0018] Based on the channel attention, construct a channel pair relationship of each of the three-dimensional tensor features, reshape and concatenate the channel pair relationship, and obtain a channel attention map based on the channel flow;

[0019] The three-dimensional tensor features are weighted by the spatial attention map and the channel attention map to obtain the global attention map.

[0020] In one embodiment, based on the spatial attention, constructing the spatial pair relationship of each of the three-dimensional tensor features, reshaping and splicing the spatial pair relationship, and obtaining a spatial attention map based on the spatial flow, includes:

[0021] The dot product of two spatial vector features is calculated to obtain the spatial pairwise relationship between the two. The spatial pairwise relationship is reshaped into a three-dimensional tensor and then concatenated with the three-dimensional tensor feature and processed into a spatial attention map, which is expressed as:

[0022]

[0023] in, Represents the spatial pairwise relationship from the i-th spatial vector feature to the j-th spatial vector feature; represents the spatial pairwise relationship from the jth spatial vector feature to the ith spatial vector feature; x i 、x jRespectively represent the spatial vector features of the three-dimensional tensor features, and i = 1, 2, ..., N; j = 1, 2, ..., N; and θ s denote the 1×1 spatial convolution layer and the corresponding network parameters respectively; ⊙ denotes the dot product operator; GAP and [·,·] denote the global average pooling and feature concatenation operations respectively; X denotes the three-dimensional tensor feature; S denotes the set of images in the training sample; R s express and The reshaped 3D tensor is A s Represents the spatial attention map.

[0024] In one of the embodiments, the multi-path joint optimization module includes a global loss unit, a small sample loss unit and a center loss unit;

[0025] The constructing of a multi-path joint optimization module and optimizing the parameters of the initial recognition model according to the global attention map includes:

[0026] A global loss unit, a small sample loss unit and a center loss unit are used to perform complementary loss coupling learning on the global attention map, and the parameters of the initial recognition model are optimized according to the learning results.

[0027] In one embodiment, the step of constructing a multi-path joint optimization module and optimizing the parameters of the initial recognition model according to the global attention map further includes:

[0028] Performing a global average pooling operation on the global attention map to obtain a semantic enhancement feature;

[0029] A global loss unit, a small sample loss unit and a center loss unit are used to perform complementary loss coupling learning on the semantic enhancement features.

[0030] In one embodiment, the method of determining whether the preset condition is met includes:

[0031] If the number of optimizations of the initial recognition model reaches a preset number, or the output accuracy of the initial recognition model meets the preset accuracy, it is considered that the preset condition is met.

[0032] In a second aspect, the present application also provides a small sample fine-grained image recognition device. The device comprises:

[0033] An acquisition module, used for acquiring training samples, wherein the training samples include a plurality of images;

[0034] A training module is used to train an initial recognition model according to the training samples to obtain a trained recognition model; wherein the step of training the initial recognition model includes:

[0035] Input the training sample into the initial recognition model, and generate a global attention map based on a dual-stream attention mechanism;

[0036] Constructing a multi-way joint optimization module, and optimizing the parameters of the initial recognition model according to the global attention map until the preset conditions are met to obtain a trained recognition model;

[0037] The recognition module is used to use the trained recognition model to recognize the small sample fine-grained images to be recognized and obtain the image category.

[0038] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method steps described in any one of the first aspects are implemented.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0040] The above-mentioned small sample fine-grained image recognition method, device, equipment and storage medium have at least the following advantages:

[0041] This application inputs the acquired training samples into the initial recognition model, generates a global attention map based on the dual-stream attention mechanism; constructs a multi-way joint optimization module, and optimizes the parameters of the initial recognition model according to the global attention map until the preset conditions are met to obtain a trained recognition model. For the above-mentioned trained recognition model, the dual-stream attention mechanism can construct a global attention map of spatial stream and channel stream features, which enhances the expression of the semantic features of the image; the multi-way joint optimization module establishes the key characteristics of multiple metric losses, learns the independence and correlation of the global features of the image, and thus obtains better discriminative features. The recognition model of this application is used to identify small sample fine-grained images to be identified, which can accurately identify subtle inter-class differences in fine-grained images and obtain better recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 This is an application environment diagram of a small sample fine-grained image recognition method in an embodiment;

[0045] Figure 2 Schematic diagram of a process of a small sample fine-grained image recognition method in one embodiment;

[0046] Figure 3 A schematic diagram of a process for generating a global attention map in one embodiment;

[0047] Figure 4 A schematic diagram of a process for generating a global attention map in another embodiment;

[0048] Figure 5 is a structural block diagram of a small sample fine-grained image recognition device in an embodiment;

[0049] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0051] While some exemplary embodiments of the present invention have been described for the purpose of illustration, it should be understood that the present invention may be implemented in other ways not specifically shown in the drawings.

[0052] The small sample fine-grained image recognition method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0053] Among them, the terminal 102 can send training samples to the server 104 so that the server 104 processes the training samples. For example, the server 104 inputs the training samples into the initial recognition model, generates a global attention map based on the dual-stream attention mechanism, builds a multi-channel joint optimization module, and optimizes the parameters of the initial recognition model according to the global attention map until the preset conditions are met to obtain a trained recognition model. The terminal 102 also sends a small sample fine-grained image to be recognized to the server 104, so that the server 104 uses the trained recognition model to recognize it and obtain the image category. In this way, the server 104 feeds back the image category to the terminal 102.

[0054] The above-mentioned small sample fine-grained image recognition method inputs the acquired training samples into the initial recognition model, generates a global attention map based on the dual-stream attention mechanism; constructs a multi-way joint optimization module, and optimizes the parameters of the initial recognition model according to the global attention map until the preset conditions are met to obtain a trained recognition model. For the above-mentioned trained recognition model, the dual-stream attention mechanism can construct a global attention map of spatial stream and channel stream features, which enhances the expression of the semantic features of the image; the multi-way joint optimization module establishes the key characteristics of multiple metric losses, learns the independence and correlation of the global features of the image, and thus obtains better discriminative features. The recognition model of the present application is used to identify the small sample fine-grained images to be identified, and the subtle inter-class differences in the fine-grained images can be accurately identified to obtain better recognition results.

[0055] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0056] In an exemplary embodiment, the present application provides a small sample fine-grained image recognition method. Figure 1 The server 104 in FIG. 1 is used as an example for explanation.

[0057] See also Figure 2 , Figure 2 The following is a flow chart of a small sample fine-grained image recognition method according to the present embodiment, which specifically includes the following steps:

[0058] Step S202: obtaining a training sample, where the training sample includes a plurality of images.

[0059] Specifically, the training samples include a query set and a support set. Generally speaking, the query set is a set of images that need to be classified, and the support set is a set of images containing a certain category, which serves as a reference to assist classification. Represents the training sample, which includes C train categories, randomly construct a query set S1 and support set S2 from the training samples, where the support set S2 consists of K categories and each category contains C labeled images, such as 1 or 5 labeled samples. The above query set S1 and support set S2 are called a small sample learning task K-way C-shot. In addition, the query set S1 and support set S2 have the same labels but no overlapping samples.

[0060] Furthermore, this application adopts a round mechanism and a fully connected supervision method to train the base class data, where the round mechanism means that from the training sample A large number of K-way C-shot tasks are sampled for training, and the fully connected supervision represents the use of training samples The category space of all labels is used to train the model parameters Θ.

[0061] Step S204: train an initial recognition model according to the training samples to obtain a trained recognition model.

[0062] The step of training the initial recognition model includes:

[0063] Input the training samples into the initial recognition model and generate a global attention map based on the two-stream attention mechanism;

[0064] A multi-way joint optimization module is constructed, and the parameters of the initial recognition model are optimized according to the global attention map until the preset conditions are met to obtain a trained recognition model.

[0065] Specifically, when processing complex fine-grained images, a single attention model is difficult to fully capture the deep correlation and spatial semantic information between features. For this reason, this application adopts a dual-stream attention mechanism to improve the ability to express subtle features in small sample fine-grained image recognition. By constructing a global attention map, the correlation between features can be accurately captured, thereby significantly enhancing the semantic feature expression of small sample fine-grained images and improving the recognition performance and generalization ability of the model.

[0066] Furthermore, in this embodiment, on the constructed global attention map, different loss metrics are collaboratively learned for features through a multi-way joint optimization module to guide the realization of independence and correlation of global features and improve the discriminability of features.

[0067] Optionally, the determination method of satisfying the preset condition includes:

[0068] If the number of optimizations of the initial recognition model reaches a preset number, or the output accuracy of the initial recognition model meets the preset accuracy, it is considered that the preset conditions are met.

[0069] Step S206: Use the trained recognition model to recognize the small sample fine-grained image to obtain the image category.

[0070] Specifically, the small sample fine-grained image to be identified is input into the trained recognition model after preprocessing, and the above-mentioned two-stream attention mechanism is used to extract image features, and then classification is performed based on the extracted image features, and finally the category to which the image belongs is obtained.

[0071] The above-mentioned small sample fine-grained image recognition method inputs the acquired training samples into the initial recognition model, generates a global attention map based on the dual-stream attention mechanism; constructs a multi-way joint optimization module, and optimizes the parameters of the initial recognition model according to the global attention map until the preset conditions are met to obtain a trained recognition model. For the above-mentioned trained recognition model, the dual-stream attention mechanism can construct a global attention map of spatial stream and channel stream features, which enhances the expression of the semantic features of the image; the multi-way joint optimization module establishes the key characteristics of multiple metric losses, learns the independence and correlation of the global features of the image, and thus obtains better discriminative features. The recognition model of the present application is used to identify the small sample fine-grained images to be identified, and the subtle inter-class differences in the fine-grained images can be accurately identified to obtain better recognition results.

[0072] See also Figure 3 Optionally, the training samples are input into the initial recognition model, and a global attention map is generated based on the two-stream attention mechanism, including:

[0073] Step S302: encoding each image in the training sample based on the weight-sharing network to obtain a three-dimensional tensor feature of each image.

[0074] Step S304, based on the dual-stream attention mechanism, construct feature relationship pairs of each three-dimensional tensor feature, and generate a global attention map based on each feature relationship pair; wherein the dual-stream attention mechanism includes spatial attention and channel attention.

[0075] Specifically, the weight-sharing network refers to using a feature extraction network whose weights are shared between the query set and the support set images, thereby ensuring that the images of the two sets are mapped to the same feature space. Θ , encode the images in the query set S1 and the support set S2 to obtain the feature map X∈R M ×W×H, where X is a three-dimensional tensor feature with M channels, width W and height H. It should be understood that the three-dimensional tensor feature generated for each image is usually a highly compressed high-level feature representation, including global and local features of the image. In addition, the feature extraction network is usually based on CNN for feature extraction, and different network architectures can be built according to the recognition requirements. This application does not limit its specific architecture.

[0076] See also Figure 4 Optionally, based on the two-stream attention mechanism, feature relationship pairs of each three-dimensional tensor feature are constructed, and based on each feature relationship pair, a global attention map is generated, including:

[0077] Step S402: Based on spatial attention, construct the spatial pair relationship of each three-dimensional tensor feature, reshape and splice the spatial pair relationship, and obtain a spatial attention map based on the spatial flow.

[0078] Step S404: Based on the channel attention, the channel pair relationship of each three-dimensional tensor feature is constructed, and the channel pair relationship is reshaped and concatenated to obtain a channel attention map based on the channel flow.

[0079] Step S406, weighted processing is performed on the three-dimensional tensor features through the spatial attention map and the channel attention map to obtain a global attention map.

[0080] Specifically, for the feature map X∈R extracted in the above step M×W×H , each spatial position of X can be represented as an M-dimensional vector, that is, all spatial positions of X contain N M-dimensional vectors (N = W × H). Let the spatial vector feature of X be represented as x i ∈R M , where i=1,2,…,N.

[0081] Optionally, based on spatial attention, a spatial pairwise relationship of each three-dimensional tensor feature is constructed, and the spatial pairwise relationship is reshaped and concatenated to obtain a spatial attention map based on a spatial flow, including:

[0082] Calculate the dot product of two spatial vector features to obtain the spatial pairwise relationship between the two, which is expressed as:

[0083]

[0084] in, Represents the spatial pairwise relationship from the i-th spatial vector feature to the j-th spatial vector feature;

[0085] Represents the spatial pairwise relationship from the jth spatial vector feature to the ith spatial vector feature;

[0086] S represents the set of images in the training sample.

[0087] Furthermore, the spatial pairwise relationship is reshaped into a three-dimensional tensor and then concatenated with the three-dimensional tensor feature. The expression is:

[0088]

[0089] Among them, x i 、x j Respectively represent the spatial vector features of the three-dimensional tensor features, and i = 1, 2, ..., N; j = 1, 2, ..., N;

[0090] and θ s Respectively represent the 1×1 spatial convolution layer and the corresponding network parameters;

[0091] ⊙ represents the dot product operator;

[0092] GAP and [·,·] represent global average pooling and feature concatenation operations, respectively;

[0093] X represents a three-dimensional tensor feature;

[0094] R s express and The reshaped 3D tensor is

[0095] Furthermore, the concatenated three-dimensional tensor Y s Processed into a spatial attention map, its expression is:

[0096]

[0097] Among them, A s ∈R H×W .

[0098] Using a similar construction method, the channel attention map is constructed Specifically, first construct a channel pair relationship and Then reshape the channel pair relationship into and The concatenated feature Y c ∈R M×(2M+1) , replace Y s , get the channel attention map A c .

[0099] Furthermore, through the spatial attention map A s And channel attention map A c, weighted processing is performed on the three-dimensional tensor features to obtain the global attention map, which is expressed as:

[0100]

[0101] Through the above spatial attention map A s And channel attention map A c The dual-stream attention mechanism performs weighted attention on the extracted feature map X, which can enhance the semantic information representation of fine-grained images and reduce the interference of background appearance.

[0102] Optionally, the multi-way joint optimization module includes a global loss unit, a small sample loss unit and a center loss unit. Then, a multi-way joint optimization module is constructed, and the parameters of the initial recognition model are optimized according to the global attention map, including:

[0103] The global loss unit, small sample loss unit and center loss unit are used to perform complementary loss coupling learning on the global attention map, and the parameters of the initial recognition model are optimized according to the learning results.

[0104] Specifically, the small sample loss is based on a round mechanism. By calculating the feature center of the query set feature to the corresponding category of the support set, local clusters are formed to learn the correlation of global features. The global loss learns the independence of global features and plays an effective auxiliary role in the small sample loss. It supervises each feature and projects the feature into the label space of the base class. In order to make the distribution of fine-grained features more clustered. The center loss is further introduced to learn the correlation of global features, so that each feature is close to the feature center of different categories in the base class label space. Promote global cluster generation.

[0105] Optionally, construct a multi-way joint optimization module and optimize the parameters of the initial recognition model according to the global attention map, and also include:

[0106] Perform global average pooling on the global attention map to obtain semantically enhanced features.

[0107] The global loss unit, small sample loss unit and center loss unit are used to perform complementary loss coupling learning on the semantically enhanced features.

[0108] Specifically, the enhanced features obtained by the above two-stream attention mechanism First, use the global average pooling operation (GlobalAverage Pooling, GAP) to generate The global vector feature By adopting this solution, the spatial dimension of the entire feature map can be reduced to one dimension, reducing the complexity of calculation and avoiding overfitting; at the same time, global average pooling considers the entire feature map and can retain more global information.

[0109] The global loss unit, small sample loss unit and center loss unit are used to learn the semantic enhancement features through complementary loss coupling, including:

[0110] The small sample loss metric is used to calculate the distance between the query image feature and each prototype of the support image. Each prototype represents the average of the global features of the same class in the support set. The small sample loss can be defined as:

[0111]

[0112] Among them, S1 is the number of samples in the query set, p k and p k' Represent the query set features respectively The prototypes of the same category in the corresponding support set and the prototypes of category k'. F represents the metric function.

[0113] The global loss fully supervises the global vector features of the image during training, learns the independent representation of the features, and also plays a certain auxiliary role in the small sample loss. The global loss can be expressed as:

[0114]

[0115] Among them, y s1 Representation characteristics of y s1 label categories. T and W represent the transposition symbol and weight parameter respectively. and b c Indicates bias. C train Indicates the number of categories.

[0116] In view of the fact that the inter-class differences in fine-grained images are small and the intra-class differences are large, this will limit the distribution differences of images of different categories in the feature space. In order to solve this problem, the center loss is introduced to effectively reduce the intra-class dispersion of fine-grained images and increase the inter-class dispersion. The definition of the center loss is:

[0117]

[0118] in, Indicates the yth s1 Therefore, the multi-way joint loss optimization can be expressed as:

[0119] L total =L globel +α Lfew +β Lcenter

[0120] Among them, α and β represent hyperparameters, which are used to balance different losses and control the performance of the model.

[0121] In order to enable those skilled in the art to fully understand the present application, the steps of training the initial recognition model in the small sample fine-grained image recognition method of the present application are described in detail below:

[0122] Step 1: Obtain training samples, wherein the training samples include a training set and a test set, the training set includes a query set and a support set, and the test set also includes a query set and a support set.

[0123] Step 2: Use the feature extraction network to extract features from the query set and support set in the training set to obtain the three-dimensional tensor features of each image, where the query set feature is X s1 ∈R M×W×H , the support set feature is X s2 ∈R M×W×H .

[0124] Step 3: Use the two-stream attention mechanism to enhance the semantic information representation of fine-grained images and reduce the interference of noise information. The enhanced query set features can be expressed as:

[0125]

[0126] Similarly, the enhanced support set feature can be expressed as:

[0127]

[0128] Step 4: Generated by global average pooling operation and The global vector features of :

[0129]

[0130] The similarity score between the query image feature and the support image feature is calculated by the metric function. The higher the score, the closer the query image feature is to the prototype. Finally, the category with the highest score is used to determine the category to which the query image belongs. Specifically, the distance metric is used. Calculate the distance from the query image feature to each type of prototype in the support image. Each prototype represents the average of the same type of global features in the support set, which can be expressed as:

[0131]

[0132] Among them, k' and p k' They represent the number of categories and prototype features in the support set respectively.

[0133] Step 5: Use the global loss unit, small sample loss unit and center loss unit to calculate the error between the category predicted by the model and the true category, and optimize the parameters of the initial recognition model according to the error size.

[0134] Step 6: Repeat steps 2 to 5 above until the number of optimizations reaches a preset number or the output accuracy of the model meets the preset accuracy, and a trained recognition model is obtained. It should be understood that the training sample also includes a validation set. After each round of training, the validation set is used to monitor the calculation accuracy and loss value of the model to adjust hyperparameters and prevent overfitting.

[0135] Step 7: Use the test set to evaluate the trained recognition model. The evaluation steps include: using the feature extraction network to extract the three-dimensional tensor features of each image in the test set, then enhancing the three-dimensional tensor features based on the two-stream attention mechanism, and then generating global vector features of the enhanced features through global average pooling operations. Finally, the similarity score is calculated through the metric function, and the category with the highest score is determined as the category of the query image.

[0136] The above-mentioned small sample fine-grained image recognition method uses a dual-stream attention mechanism to perform pairwise aggregation of the features of the spatial stream and the channel stream on a global scale at the same time to generate high-dimensional correlation features. This mechanism can accurately capture the correlation between features, thereby significantly enhancing the semantic feature expression of small sample fine-grained images and improving the recognition performance and generalization ability of the model. At the same time, on the constructed global attention feature map, the multi-way joint optimization module performs collaborative learning of different loss metrics on the features, which can guide the realization of the independence and correlation of global features and improve the discriminability of the features. The recognition model of the present application is used to identify the small sample fine-grained images to be identified, and subtle inter-class differences in fine-grained images can be accurately identified to obtain better recognition results.

[0137] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0138] Based on the same inventive concept, the embodiment of the present application also provides a small sample fine-grained image recognition device for implementing the small sample fine-grained image recognition method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more small sample fine-grained image recognition device embodiments provided below can refer to the limitations of the small sample fine-grained image recognition method above, and will not be repeated here.

[0139] See also Figure 5 In an exemplary embodiment, the present application provides a small sample fine-grained image recognition device, including: an acquisition module, a training module and a recognition module, wherein:

[0140] The acquisition module is used to acquire training samples, which include multiple images.

[0141] The training module is used to train the initial recognition model according to the training samples to obtain the trained recognition model; wherein the step of training the initial recognition model includes: inputting the training samples into the initial recognition model, generating a global attention map based on the dual-stream attention mechanism; constructing a multi-way joint optimization module, and optimizing the parameters of the initial recognition model according to the attention map until the preset conditions are met to obtain the trained recognition model.

[0142] The recognition module is used to use the trained recognition model to recognize the small sample fine-grained images to be recognized and obtain the image category.

[0143] Optionally, the training module inputs the training samples into the initial recognition model, and generates a global attention map based on the dual-stream attention mechanism, including: encoding each image in the training sample based on a weight-sharing network to obtain three-dimensional tensor features of each image; based on the dual-stream attention mechanism, constructing feature relationship pairs of each three-dimensional tensor feature, and generating a global attention map based on each feature relationship pair; wherein the dual-stream attention mechanism includes spatial attention and channel attention.

[0144] Optionally, the training module constructs feature relationship pairs of each three-dimensional tensor feature based on the dual-stream attention mechanism, and generates a global attention map based on each feature relationship pair, including: based on spatial attention, constructing spatial paired relationships of each three-dimensional tensor feature, reshaping and splicing the spatial paired relationships, and obtaining a spatial attention map based on the spatial stream. Based on channel attention, constructing channel paired relationships of each three-dimensional tensor feature, reshaping and splicing the channel paired relationships, and obtaining a channel attention map based on the channel stream. Through the spatial attention map and the channel attention map, the three-dimensional tensor features are weighted to obtain a global attention map.

[0145] Optionally, the multi-way joint optimization module includes a global loss unit, a small sample loss unit and a center loss unit. Then the training module constructs the multi-way joint optimization module and optimizes the parameters of the initial recognition model according to the global attention map, including: using the global loss unit, the small sample loss unit and the center loss unit to perform complementary loss coupling learning on the global attention map, and optimizing the parameters of the initial recognition model according to the learning results.

[0146] Optionally, the training module constructs a multi-way joint optimization module and optimizes the parameters of the initial recognition model according to the global attention map, and also includes: performing a global average pooling operation on the global attention map to obtain semantic enhancement features. The semantic enhancement features are learned by complementary loss coupling using a global loss unit, a small sample loss unit, and a center loss unit.

[0147] The above-mentioned small sample fine-grained image recognition device uses a dual-stream attention mechanism to perform pairwise aggregation of the features of the spatial stream and the channel stream on a global scale at the same time to generate high-dimensional correlation features. This mechanism can accurately capture the correlation between features, thereby significantly enhancing the semantic feature expression of small sample fine-grained images and improving the recognition performance and generalization ability of the model. At the same time, on the constructed global attention feature map, the multi-way joint optimization module performs collaborative learning of different loss metrics on the features, which can guide the realization of the independence and correlation of global features and improve the discriminability of the features. The recognition model of the present application is used to identify the small sample fine-grained images to be identified, and subtle inter-class differences in fine-grained images can be accurately identified to obtain better recognition results.

[0148] Each module in the above-mentioned small sample fine-grained image recognition device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0149] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a small sample fine-grained image recognition method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0150] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0151] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned small sample fine-grained image recognition method when executing the computer program.

[0152] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned small sample fine-grained image recognition method are implemented.

[0153] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processors involved in each embodiment provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., but are not limited thereto.

[0154] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A small sample fine-grained image recognition method, characterized in that: The method comprises: Acquire a training sample, wherein the training sample includes a plurality of images; According to the training samples, an initial recognition model is trained to obtain a trained recognition model; Use the trained recognition model to identify the small sample fine-grained image to obtain the image category; The step of training the initial recognition model includes: Input the training sample into the initial recognition model, and generate a global attention map based on a dual-stream attention mechanism; A multi-path joint optimization module is constructed, and the parameters of the initial recognition model are optimized according to the global attention map until the preset conditions are met to obtain a trained recognition model.

2. The method according to claim 1, characterized in that The step of inputting the training sample into the initial recognition model and generating a global attention map based on a dual-stream attention mechanism includes: Based on a weight-sharing network, encoding each of the images in the training sample to obtain a three-dimensional tensor feature of each image; Based on the dual-stream attention mechanism, constructing feature relationship pairs of the three-dimensional tensor features, and generating a global attention map based on the feature relationship pairs; Among them, the dual-stream attention mechanism includes spatial attention and channel attention.

3. The method according to claim 2, characterized in that The method of constructing feature relationship pairs of the three-dimensional tensor features based on the dual-stream attention mechanism and generating a global attention map based on the feature relationship pairs includes: Based on the spatial attention, constructing a spatial pair relationship of each of the three-dimensional tensor features, reshaping and splicing the spatial pair relationship, and obtaining a spatial attention map based on the spatial flow; Based on the channel attention, construct a channel pair relationship of each of the three-dimensional tensor features, reshape and concatenate the channel pair relationship, and obtain a channel attention map based on the channel flow; The three-dimensional tensor features are weighted by the spatial attention map and the channel attention map to obtain the global attention map.

4. The method according to claim 3, characterized in that Based on the spatial attention, constructing the spatial pair relationship of each of the three-dimensional tensor features, reshaping and splicing the spatial pair relationship, and obtaining a spatial attention map based on the spatial flow, including: The dot product of two spatial vector features is calculated to obtain the spatial pairwise relationship between the two. The spatial pairwise relationship is reshaped into a three-dimensional tensor and then concatenated with the three-dimensional tensor feature and processed into a spatial attention map, which is expressed as: in, Represents the spatial pairwise relationship from the i-th spatial vector feature to the j-th spatial vector feature; represents the spatial pairwise relationship from the jth spatial vector feature to the ith spatial vector feature; x i 、x j Respectively represent the spatial vector features of the three-dimensional tensor features, and i = 1, 2, ..., N; j = 1, 2, ..., N; and θ s denote the 1×1 spatial convolution layer and the corresponding network parameters respectively; ⊙ denotes the dot product operator; GAP and [·,·] denote the global average pooling and feature concatenation operations respectively; X denotes the three-dimensional tensor feature; S denotes the set of images in the training sample; R s express and The reshaped 3D tensor is A s Represents the spatial attention map.

5. The method according to claim 1, characterized in that: The multi-path joint optimization module includes a global loss unit, a small sample loss unit and a center loss unit; The constructing of a multi-path joint optimization module and optimizing the parameters of the initial recognition model according to the global attention map includes: A global loss unit, a small sample loss unit and a center loss unit are used to perform complementary loss coupling learning on the global attention map, and the parameters of the initial recognition model are optimized according to the learning results.

6. The method according to claim 1, characterized in that The constructing of a multi-path joint optimization module and optimizing the parameters of the initial recognition model according to the global attention map further includes: Performing a global average pooling operation on the global attention map to obtain a semantic enhancement feature; A global loss unit, a small sample loss unit and a center loss unit are used to perform complementary loss coupling learning on the semantic enhancement features.

7. The method according to claim 1, characterized in that The determination method of satisfying the preset condition includes: If the number of optimizations of the initial recognition model reaches a preset number, or the output accuracy of the initial recognition model meets the preset accuracy, it is considered that the preset condition is met.

8. A small sample fine-grained image recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring training samples, wherein the training samples include a plurality of images; A training module is used to train an initial recognition model according to the training samples to obtain a trained recognition model; wherein the step of training the initial recognition model includes: Input the training sample into the initial recognition model, and generate a global attention map based on a dual-stream attention mechanism; Constructing a multi-way joint optimization module, and optimizing the parameters of the initial recognition model according to the global attention map until the preset conditions are met to obtain a trained recognition model; The recognition module is used to use the trained recognition model to recognize the small sample fine-grained images to be recognized and obtain the image category.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images