A method, apparatus, device, and storage medium for small-sample fine-grained image recognition.

Abstract

This application relates to a method, apparatus, device, and storage medium for small-sample fine-grained image recognition. The method includes: acquiring training samples, which include multiple images; training an initial recognition model based on the training samples to obtain a trained recognition model; and using the trained recognition model to recognize the small-sample fine-grained image to be recognized, thereby obtaining the image category. The step of training the initial recognition model includes: inputting the training samples into the initial recognition model, generating a global attention map based on a two-stream attention mechanism; constructing a multi-path joint optimization module, and optimizing the parameters of the initial recognition model based on the global attention map until preset conditions are met, thus obtaining a trained recognition model. Using the recognition model of this application to recognize the small-sample fine-grained image to be recognized can accurately identify subtle inter-class differences in the fine-grained image, resulting in good recognition results.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method, apparatus, device and storage medium for small-sample fine-grained image recognition. Background Technology

[0002] With the rise of deep learning, the performance of visual recognition tasks has made remarkable progress. Fine-grained recognition, as a fundamental task in visual recognition, plays a crucial role in practical applications. The purpose of fine-grained recognition is to distinguish different subcategories within the same category (e.g., the CUB, Stanford Cars, and Stanford Dogs datasets).

[0003] However, existing fine-grained image recognition models often rely on large amounts of manually labeled data, which severely limits their ability to recognize novel objects with only a few labeled samples. Few-shot learning effectively alleviates the burden of manual annotation from large datasets by transferring knowledge learned from past experience to identify new categories with only a limited number of samples (e.g., 1 or 5 samples). Unlike general images, which exhibit significant class differences, fine-grained images contain subtle inter-class differences and large intra-class differences, making few-shot fine-grained image recognition tasks more challenging. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, device, and storage medium for small-sample fine-grained image recognition to address the aforementioned technical problems.

[0005] Firstly, this application provides a method for small-sample, fine-grained image recognition. The method includes:

[0006] Obtain training samples, which include multiple images;

[0007] Based on the training samples, an initial recognition model is trained to obtain a trained recognition model;

[0008] The trained recognition model is used to identify small, fine-grained images to be recognized, and the image category is obtained.

[0009] The steps for training the initial recognition model include:

[0010] The training samples are input into the initial recognition model, and a global attention map is generated based on the two-stream attention mechanism;

[0011] A multi-path joint optimization module is constructed, and the parameters of the initial recognition model are optimized according to the global attention map until the preset conditions are met, thus obtaining a trained recognition model.

[0012] In one embodiment, the step of inputting the training samples into the initial recognition model and generating a global attention map based on a two-stream attention mechanism includes:

[0013] Based on a weight-sharing network, each image in the training samples is encoded to obtain the three-dimensional tensor features of each image;

[0014] Based on the two-stream attention mechanism, feature relationship pairs of each of the three-dimensional tensor features are constructed, and a global attention graph is generated based on each of the feature relationship pairs;

[0015] The dual-stream attention mechanism includes spatial attention and channel attention.

[0016] In one embodiment, the step of constructing feature relationship pairs for each of the three-dimensional tensor features based on the two-stream attention mechanism, and generating a global attention map based on each of the feature relationship pairs, includes:

[0017] Based on the spatial attention, a spatial pairwise relationship of each of the three-dimensional tensor features is constructed, and the spatial pairwise relationship is reshaped and spliced ​​to obtain a spatial attention graph based on spatial flow.

[0018] Based on the channel attention, a channel pairing relationship is constructed for each of the three-dimensional tensor features. The channel pairing relationship is then reshaped and spliced ​​to obtain a channel attention map based on channel flow.

[0019] The three-dimensional tensor features are weighted using the spatial attention map and the channel attention map to obtain the global attention map.

[0020] In one embodiment, the step of constructing spatial pairwise relationships for each of the three-dimensional tensor features based on the spatial attention, and performing reshaping and splicing operations on the spatial pairwise relationships to obtain a spatial attention map based on spatial flow includes:

[0021] Calculate the dot product of the features of two spatial vectors to obtain the spatial pairwise relationship between them; reshape the spatial pairwise relationship into a three-dimensional tensor and concatenate it with the three-dimensional tensor features, then process it into a spatial attention map, the expression of which is:

[0022]

[0023] in, This represents the spatial pairwise relationship between the i-th spatial vector feature and the j-th spatial vector feature; This represents the spatial pairwise relationship between the j-th spatial vector feature and the i-th spatial vector feature; x i x jLet i and j represent the spatial vector features of the three-dimensional tensor features, where i = 1, 2, ..., N; j = 1, 2, ..., N. and θ s represents a 1×1 spatial convolutional layer and its corresponding network parameters; ⊙ represents the dot product operator; GAP and [·,·] represent global average pooling and feature concatenation operations, respectively; X represents the 3D tensor feature; S represents the set of images in the training samples; R s express and The reconstructed three-dimensional tensor, and A s Represents a spatial attention map.

[0024] In one embodiment, the multi-path joint optimization module includes a global loss unit, a small sample loss unit, and a central loss unit;

[0025] The construction of the multi-path joint optimization module, and the optimization of the parameters of the initial recognition model based on the global attention map, includes:

[0026] The global attention map is subjected to complementary loss coupling learning using a global loss unit, a few-sample loss unit, and a center loss unit, and the parameters of the initial recognition model are optimized based on the learning results.

[0027] In one embodiment, the construction of the multi-path joint optimization module and the optimization of the parameters of the initial recognition model based on the global attention map further includes:

[0028] Global average pooling is performed on the global attention map to obtain semantically enhanced features;

[0029] The semantic enhancement features are learned by complementary loss coupling using a global loss unit, a few-sample loss unit, and a central loss unit.

[0030] In one embodiment, the method for determining whether the preset conditions are met includes:

[0031] If the initial recognition model is optimized a preset number of times, or the output accuracy of the initial recognition model meets a preset accuracy, then the preset condition is considered to be met.

[0032] Secondly, this application also provides a small-sample fine-grained image recognition device. The device includes:

[0033] The acquisition module is used to acquire training samples, which include multiple images;

[0034] The training module is used to train an initial recognition model based on the training samples to obtain a trained recognition model; wherein the steps of training the initial recognition model include:

[0035] The training samples are input into the initial recognition model, and a global attention map is generated based on the two-stream attention mechanism;

[0036] A multi-path joint optimization module is constructed, and the parameters of the initial recognition model are optimized according to the global attention map until the preset conditions are met, so as to obtain a trained recognition model.

[0037] The recognition module is used to identify small, fine-grained images of the target image using a trained recognition model, and to obtain the image category.

[0038] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method steps described in any one of the first aspects.

[0039] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the method steps described in any one of the first aspects.

[0040] The above-mentioned small-sample fine-grained image recognition methods, apparatuses, devices, and storage media have at least the following advantages:

[0041] This application inputs the acquired training samples into an initial recognition model, generates a global attention map based on a two-stream attention mechanism, constructs a multi-path joint optimization module, and optimizes the parameters of the initial recognition model according to the global attention map until preset conditions are met, resulting in a trained recognition model. In the trained recognition model, the two-stream attention mechanism constructs a global attention map of spatial and channel flow features, enhancing the expression of semantic features of the image; the multi-path joint optimization module establishes multiple key features for measuring loss, learns the independence and correlation of global features of the image, thereby obtaining better discriminative features. Using the recognition model of this application to recognize small-sample, fine-grained images, it can accurately identify subtle inter-class differences in fine-grained images, obtaining good recognition results. Attached Figure Description

[0042] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a diagram illustrating the application environment of a small-sample fine-grained image recognition method in one embodiment.

[0045] Figure 2 This is a flowchart illustrating a small-sample fine-grained image recognition method in one embodiment;

[0046] Figure 3 This is a flowchart illustrating the steps for generating a global attention map in one embodiment;

[0047] Figure 4 This is a flowchart illustrating the steps for generating a global attention map in another embodiment;

[0048] Figure 5 This is a structural block diagram of a small-sample fine-grained image recognition device in one embodiment;

[0049] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0051] Some exemplary embodiments of the invention have been described for illustrative purposes. It should be understood that the invention may be implemented in other ways not specifically shown in the accompanying drawings.

[0052] The small-sample, fine-grained image recognition method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0053] Terminal 102 can send training samples to server 104, enabling server 104 to process the training samples. For example, server 104 inputs the training samples into an initial recognition model, generates a global attention map based on a two-stream attention mechanism, constructs a multi-path joint optimization module, and optimizes the parameters of the initial recognition model according to the global attention map until preset conditions are met, resulting in a trained recognition model. Terminal 102 also sends small-sample, fine-grained images to be recognized to server 104, enabling server 104 to use the trained recognition model to recognize them and obtain the image category. Server 104 then feeds back the image category to terminal 102.

[0054] The aforementioned small-sample fine-grained image recognition method inputs the acquired training samples into an initial recognition model, generates a global attention map based on a two-stream attention mechanism, constructs a multi-path joint optimization module, and optimizes the parameters of the initial recognition model according to the global attention map until preset conditions are met, resulting in a trained recognition model. In the trained recognition model, the two-stream attention mechanism constructs a global attention map of spatial and channel flow features, enhancing the expression of semantic features of the image; the multi-path joint optimization module establishes multiple key features for measuring loss, learns the independence and correlation of global features of the image, thereby obtaining better discriminative features. Using the recognition model of this application to recognize small-sample fine-grained images, it can accurately identify subtle inter-class differences in fine-grained images, obtaining good recognition results.

[0055] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0056] In one exemplary embodiment, this application provides a small-sample fine-grained image recognition method, which will be applied to the following embodiments. Figure 1 We will use server 104 as an example to illustrate this.

[0057] Please see Figure 2 , Figure 2 This is a flowchart illustrating a small-sample fine-grained image recognition method according to this embodiment, specifically including the following steps:

[0058] Step S202: Obtain training samples, which include multiple images.

[0059] Specifically, the training samples include a query set and a support set. Typically, the query set is the set of images to be classified, and the support set is a set of images containing one class, used as a reference to assist in classification. For example, using... The training samples are represented by C. train For each category, a query set S1 and a support set S2 are randomly constructed from the training samples. The support set S2 consists of K categories, and each category contains C labeled images, such as 1 or 5 labeled samples. The query set S1 and support set S2 are referred to as a K-way C-shot learning task. Furthermore, the query set S1 and support set S2 have samples with the same labels but no overlap.

[0060] Furthermore, this application employs a round-robin mechanism and fully connected supervision to train the base class data, where the round-robin mechanism represents the process of training samples from... The training was performed using a large number of K-way C-shot tasks, and the fully connected supervised representation utilized the training samples. The model parameters Θ are trained using the category space of all labels.

[0061] Step S204: Train the initial recognition model based on the training samples to obtain the trained recognition model.

[0062] The steps for training the initial recognition model include:

[0063] The training samples are input into the initial recognition model, and a global attention map is generated based on the two-stream attention mechanism;

[0064] A multi-path joint optimization module is constructed, and the parameters of the initial recognition model are optimized based on the global attention map until the preset conditions are met, thus obtaining a trained recognition model.

[0065] Specifically, when processing complex, fine-grained images, a single attention pattern is insufficient to fully capture the deep-seated relationships and spatial semantic information between features. Therefore, this application employs a dual-stream attention mechanism to enhance the representation of subtle features in small-sample, fine-grained image recognition. By constructing a global attention map, the correlation between features can be accurately captured, thereby significantly enhancing the semantic feature representation of small-sample, fine-grained images and improving the model's recognition performance and generalization ability.

[0066] Furthermore, in this embodiment, on the constructed global attention map, a multi-path joint optimization module is used to perform collaborative learning on the features by measuring different losses, which guides the realization of the independence and relevance of global features and improves the discriminativeness of features.

[0067] Optionally, the determination method for satisfying preset conditions includes:

[0068] If the initial recognition model is optimized a preset number of times, or if the output accuracy of the initial recognition model meets the preset accuracy, then the preset conditions are considered to be met.

[0069] Step S206: Using the trained recognition model, the small sample fine-grained images to be recognized are identified to obtain the image category.

[0070] Specifically, after preprocessing, the small sample fine-grained images to be identified are input into the trained recognition model. The dual-stream attention mechanism described above is used to extract image features, and then classification is performed based on the extracted image features to finally obtain the category to which the image belongs.

[0071] The aforementioned small-sample fine-grained image recognition method inputs the acquired training samples into an initial recognition model, generates a global attention map based on a two-stream attention mechanism, constructs a multi-path joint optimization module, and optimizes the parameters of the initial recognition model according to the global attention map until preset conditions are met, resulting in a trained recognition model. In the trained recognition model, the two-stream attention mechanism constructs a global attention map of spatial and channel flow features, enhancing the expression of semantic features of the image; the multi-path joint optimization module establishes multiple key features for measuring loss, learns the independence and correlation of global features of the image, thereby obtaining better discriminative features. Using the recognition model of this application to recognize small-sample fine-grained images, it can accurately identify subtle inter-class differences in fine-grained images, obtaining good recognition results.

[0072] Please see Figure 3 Optionally, the training samples are input into the initial recognition model, and a global attention map is generated based on the two-stream attention mechanism, including:

[0073] Step S302: Based on the weight-sharing network, each image in the training samples is encoded to obtain the three-dimensional tensor features of each image.

[0074] Step S304: Based on the two-stream attention mechanism, construct feature relationship pairs for each three-dimensional tensor feature, and generate a global attention map based on each feature relationship pair; wherein, the two-stream attention mechanism includes spatial attention and channel attention.

[0075] Specifically, a weight-sharing network refers to a feature extraction network whose weights are shared across the images in both the query and support sets, ensuring that the images in both sets are mapped to the same feature space. For example, through a feature extraction network F... Θ The images in the query set S1 and the support set S2 are encoded to obtain the feature map X∈R. M ×W×HHere, X is a three-dimensional tensor feature with M channels, a width of W, and a height of H. It should be understood that the three-dimensional tensor feature generated for each image is typically a highly compressed high-level feature representation, containing both global and local features of the image. Furthermore, feature extraction networks are typically based on CNNs for feature extraction, and different network architectures can be built according to the recognition requirements; this application does not limit the specific architecture.

[0076] Please see Figure 4 Optionally, based on the two-stream attention mechanism, feature relationship pairs for each 3D tensor feature are constructed, and a global attention map is generated based on each feature relationship pair, including:

[0077] Step S402: Based on spatial attention, construct spatial pairwise relationships for each three-dimensional tensor feature, and perform reshaping and splicing operations on the spatial pairwise relationships to obtain a spatial attention graph based on spatial flow.

[0078] Step S404: Based on channel attention, construct the channel pairing relationship of each three-dimensional tensor feature, and perform reshaping and splicing operations on the channel pairing relationship to obtain a channel attention map based on channel flow.

[0079] Step S406: The three-dimensional tensor features are weighted using the spatial attention map and the channel attention map to obtain the global attention map.

[0080] Specifically, for the feature map X∈R extracted in the above steps M×W×H Each spatial location of X can be represented as an M-dimensional vector, meaning that all spatial locations of X contain N M-dimensional vectors (N = W × H). Let the spatial vector characteristics of X be represented as x. i ∈R M , where i = 1, 2, ..., N.

[0081] Optionally, based on spatial attention, spatial pairwise relationships of each 3D tensor feature are constructed, and the spatial pairwise relationships are reshaped and spliced ​​to obtain a spatial attention map based on spatial flow, including:

[0082] Calculating the dot product of the features of two spatial vectors yields the spatial pairwise relation between them, expressed as:

[0083]

[0084] in, This represents the spatial pairwise relationship between the i-th spatial vector feature and the j-th spatial vector feature;

[0085] This represents the spatial pairwise relationship between the j-th spatial vector feature and the i-th spatial vector feature;

[0086] S represents the set of images in the training samples.

[0087] Furthermore, after reshaping the spatial pairwise relationships into a three-dimensional tensor, it is concatenated with the three-dimensional tensor features, and its expression is:

[0088]

[0089] Where, x i x j Let i and j represent the spatial vector features of the three-dimensional tensor features, where i = 1, 2, ..., N; j = 1, 2, ..., N.

[0090] and θ s These represent a 1×1 spatial convolutional layer and its corresponding network parameters, respectively.

[0091] ⊙ represents the dot product operator;

[0092] GAP and [·,·] represent global average pooling and feature concatenation operations, respectively;

[0093] X represents a three-dimensional tensor feature;

[0094] R s express and The reconstructed three-dimensional tensor, and

[0095] Furthermore, the spliced ​​three-dimensional tensor Y s The processing is done as a spatial attention map, the expression of which is:

[0096]

[0097] Among them, A s ∈R H×W .

[0098] Using a similar construction method, a channel attention map is obtained. Specifically, the first step is to establish paired channel relationships. and Then reshape the channel pair relationship into and The feature Y obtained by splicing c ∈R M×(2M+1) Replace the above formula with Y s The channel attention map A is obtained. c .

[0099] Furthermore, through spatial attention map A s Channel attention map A cWe perform weighted processing on the three-dimensional tensor features to obtain the global attention map, whose expression is:

[0100]

[0101] Based on the above spatial attention diagram A s Channel attention map A c The dual-stream attention mechanism, which applies attention weights to the extracted feature map X, can enhance the semantic information representation of fine-grained images and reduce interference from background appearance.

[0102] Optionally, the multi-path joint optimization module includes a global loss unit, a few-sample loss unit, and a center loss unit. The multi-path joint optimization module is then constructed, and the parameters of the initial recognition model are optimized based on the global attention map, including:

[0103] A complementary loss coupling learning method is used to learn the global attention map by employing a global loss unit, a few-sample loss unit, and a center loss unit, and the parameters of the initial recognition model are optimized based on the learning results.

[0104] Specifically, the few-shot loss, based on a round-robin mechanism, calculates feature centers from query set features to corresponding categories in the support set, forming local clusters and learning the correlation of global features. The global loss learns the independence of global features, effectively assisting the few-shot loss by supervising each feature and projecting it into the base class label space. To further cluster fine-grained features, a center loss is introduced to learn the correlation of global features, bringing each feature closer to the feature centers of different categories in the base class label space, thus promoting global cluster generation.

[0105] Optionally, constructing a multi-path joint optimization module and optimizing the parameters of the initial recognition model based on the global attention map also includes:

[0106] Global average pooling is performed on the global attention map to obtain semantically enhanced features.

[0107] We employ a global loss unit, a few-sample loss unit, and a central loss unit to perform complementary loss coupling learning on semantic enhancement features.

[0108] Specifically, the enhanced features obtained from the aforementioned two-stream attention mechanism First, global average pooling (GAP) is used to generate... global vector features This approach reduces the spatial dimension of the entire feature map to one dimension, lowers computational complexity, and avoids overfitting. Meanwhile, global average pooling considers the entire feature map, preserving more global information.

[0109] The semantic enhancement features are coupled with complementary losses using global loss units, few-sample loss units, and central loss units, including:

[0110] The few-shot loss metric calculates the distance from the query image features to each class prototype of the supporting image, where each prototype represents the average of global features of the same class in the support set. The few-shot loss can be defined as:

[0111]

[0112] Where S1 is the number of samples in the query set, p k and p k' These represent the query set features respectively. This corresponds to prototypes of the same category and prototypes of category k' in the support set. F represents the metric function.

[0113] The global loss method provides fully supervised training on the global vector features of the image, learning independent representations of the features, and also plays a supporting role in few-sample loss. The global loss can be expressed as:

[0114]

[0115] Among them, y s1 Representation of features of y s1 There are 1 label category. T and W represent the transpose sign and weight parameter, respectively. and b c Indicates bias. C train Indicates the number of categories.

[0116] For fine-grained images, which exhibit small inter-class differences and large intra-class differences, this limits the distributional variation of different image classes in the feature space. To address this issue, a center loss is introduced to effectively reduce intra-class dispersion and increase inter-class dispersion in fine-grained images. The center loss is defined as follows:

[0117]

[0118] in, Indicates the yth s1 The center of each category. Therefore, the multi-path joint loss optimization can be expressed as:

[0119] L total =L globel +α Lfew +β Lcenter

[0120] Here, α and β represent hyperparameters used to balance different losses and control the performance of the model.

[0121] In order to enable those skilled in the art to fully understand this application, the following describes in detail the steps of training the initial recognition model in the small-sample fine-grained image recognition method of this application:

[0122] Step 1: Obtain training samples, which include a training set and a test set. The training set includes a query set and a support set, and the test set also includes a query set and a support set.

[0123] Step 2: Employ a feature extraction network to extract features from the query set and support set in the training set, obtaining the 3D tensor features of each image. The query set features are denoted as X. s1 ∈R M×W×H The support set features are X s2 ∈R M×W×H .

[0124] Step 3: Enhance the semantic information representation of fine-grained images through a dual-stream attention mechanism to reduce the interference of noise. The enhanced query set features can be represented as:

[0125]

[0126] Similarly, the enhanced support set features can be represented as:

[0127]

[0128] Step 4: Generated through global average pooling operation and Global vector features:

[0129]

[0130] The similarity score between the query image features and the supporting image features is calculated using a metric function. A higher score indicates that the query image features are closer to the original image. The category of the query image is ultimately determined by the category with the highest score. Specifically, this is achieved through a distance metric. Calculate the distance from the query image features to each class prototype of the supporting image. Each prototype represents the average of the global features of the same class in the support set, which can be expressed as:

[0131]

[0132] Where, k' and p k' These represent the number of supporting categories and prototype features, respectively.

[0133] Step 5: Using global loss units, small sample loss units, and center loss units, calculate the error between the predicted category and the true category. Based on the magnitude of the error, optimize the parameters of the initial recognition model.

[0134] Step Six: Repeat steps Two through Five above until the preset number of optimization attempts is reached, or the model's output accuracy meets the preset accuracy, to obtain the trained recognition model. It should be understood that the training samples also include a validation set. After each round of training, the validation set is used to monitor the model's computational accuracy and loss value, in order to adjust hyperparameters and prevent overfitting.

[0135] Step 7: Evaluate the trained recognition model using a test set. The evaluation steps include: extracting the 3D tensor features of each image in the test set using a feature extraction network, enhancing the 3D tensor features based on a two-stream attention mechanism, generating global vector features of the enhanced features through global average pooling, and finally calculating the similarity score through a metric function, determining the category with the highest score as the category of the query image.

[0136] The aforementioned small-sample fine-grained image recognition method employs a dual-stream attention mechanism to simultaneously aggregate features from both spatial and channel flows globally, generating high-dimensional correlated features. This mechanism accurately captures the correlations between features, significantly enhancing the semantic feature representation of small-sample fine-grained images and improving the model's recognition performance and generalization ability. Simultaneously, on the constructed global attention feature map, a multi-path joint optimization module performs collaborative learning with different loss measures on the features, guiding the realization of the independence and correlation of global features and improving feature discriminativity. Using the recognition model of this application to recognize small-sample fine-grained images, it can accurately identify subtle inter-class differences in fine-grained images, yielding good recognition results.

[0137] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0138] Based on the same inventive concept, this application also provides a small-sample fine-grained image recognition device for implementing the small-sample fine-grained image recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more small-sample fine-grained image recognition device embodiments provided below can be found in the limitations of the small-sample fine-grained image recognition method described above, and will not be repeated here.

[0139] Please see Figure 5 In one exemplary embodiment, this application provides a small-sample fine-grained image recognition device, including: an acquisition module, a training module, and a recognition module, wherein:

[0140] The acquisition module is used to acquire training samples, which include multiple images.

[0141] The training module is used to train an initial recognition model based on training samples to obtain a trained recognition model. The steps of training the initial recognition model include: inputting training samples into the initial recognition model, generating a global attention map based on a two-stream attention mechanism, constructing a multi-path joint optimization module, and optimizing the parameters of the initial recognition model according to the attention map until a preset condition is met to obtain a trained recognition model.

[0142] The recognition module is used to identify small, fine-grained images of the target image using a trained recognition model, and to obtain the image category.

[0143] Optionally, the training module inputs the training samples into the initial recognition model and generates a global attention map based on the two-stream attention mechanism, including: encoding each image in the training samples based on a weight-sharing network to obtain the three-dimensional tensor features of each image; constructing feature pairs of each three-dimensional tensor feature based on the two-stream attention mechanism, and generating a global attention map based on each feature pair; wherein, the two-stream attention mechanism includes spatial attention and channel attention.

[0144] Optionally, the training module, based on a two-stream attention mechanism, constructs feature pairs for each 3D tensor feature and generates a global attention map based on these pairs. This includes: constructing spatial pairwise relationships for each 3D tensor feature based on spatial attention, reshaping and concatenating these relationships to obtain a spatial attention map based on spatial flow; constructing channel pairwise relationships for each 3D tensor feature based on channel attention, reshaping and concatenating these relationships to obtain a channel attention map based on channel flow; and weighting the 3D tensor features using the spatial and channel attention maps to obtain the global attention map.

[0145] Optionally, the multi-path joint optimization module includes a global loss unit, a few-shot loss unit, and a center loss unit. The training module then constructs the multi-path joint optimization module and optimizes the parameters of the initial recognition model based on the global attention map. This includes: using the global loss unit, few-shot loss unit, and center loss unit to perform complementary loss coupling learning on the global attention map, and optimizing the parameters of the initial recognition model based on the learning results.

[0146] Optionally, the training module constructs a multi-path joint optimization module and optimizes the parameters of the initial recognition model based on the global attention map. It also includes performing global average pooling on the global attention map to obtain semantically enhanced features. Complementary loss coupling learning is then performed on the semantically enhanced features using a global loss unit, a few-shot loss unit, and a central loss unit.

[0147] The aforementioned small-sample fine-grained image recognition device employs a dual-stream attention mechanism to simultaneously aggregate features from both spatial and channel flows globally, generating high-dimensional correlated features. This mechanism accurately captures the correlations between features, significantly enhancing the semantic feature representation of small-sample fine-grained images and improving the model's recognition performance and generalization ability. Simultaneously, on the constructed global attention feature map, a multi-path joint optimization module performs collaborative learning with different loss measures on the features, guiding the realization of the independence and correlation of global features and improving feature discriminability. Using the recognition model of this application to recognize small-sample fine-grained images, it can accurately identify subtle inter-class differences in fine-grained images, obtaining good recognition results.

[0148] Each module in the aforementioned small-sample fine-grained image recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0149] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a small-sample, fine-grained image recognition method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0150] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0151] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described small-sample fine-grained image recognition method.

[0152] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the small-sample fine-grained image recognition method described above.

[0153] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0154] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for fine-grained image recognition with small sample sizes, characterized in that, The method includes: Obtain training samples, which include multiple images; Based on the training samples, an initial recognition model is trained to obtain a trained recognition model; The trained recognition model is used to identify small, fine-grained images to be recognized, and the image category is obtained. The steps for training the initial recognition model include: The training samples are input into the initial recognition model, and a global attention map is generated based on the two-stream attention mechanism; A multi-path joint optimization module is constructed, and the parameters of the initial recognition model are optimized according to the global attention map until the preset conditions are met, so as to obtain a trained recognition model. The step of inputting the training samples into the initial recognition model and generating a global attention map based on a two-stream attention mechanism includes: Based on a weight-sharing network, each image in the training samples is encoded to obtain the three-dimensional tensor features of each image; Based on the two-stream attention mechanism, feature relationship pairs of each of the three-dimensional tensor features are constructed, and a global attention graph is generated based on each of the feature relationship pairs; The dual-stream attention mechanism includes spatial attention and channel attention; The method of constructing feature relationship pairs for each of the three-dimensional tensor features based on the two-stream attention mechanism, and generating a global attention map based on each of the feature relationship pairs, includes: Based on the spatial attention, a spatial pairwise relationship of each of the three-dimensional tensor features is constructed, and the spatial pairwise relationship is reshaped and spliced ​​to obtain a spatial attention graph based on spatial flow. Based on the channel attention, a channel pairing relationship is constructed for each of the three-dimensional tensor features. The channel pairing relationship is then reshaped and spliced ​​to obtain a channel attention map based on channel flow. The three-dimensional tensor features are weighted using the spatial attention map and the channel attention map to obtain the global attention map; The process of constructing spatial pairwise relationships for each of the three-dimensional tensor features based on the spatial attention, and then reshaping and splicing these spatial pairwise relationships to obtain a spatial attention graph based on spatial flow includes: Calculate the dot product of the features of two spatial vectors to obtain the spatial pairwise relationship between them; reshape the spatial pairwise relationship into a three-dimensional tensor and then concatenate it with the three-dimensional tensor features, and process it into a spatial attention map; The multi-path joint optimization module includes a global loss unit, a small sample loss unit, and a central loss unit; The construction of the multi-path joint optimization module, and the optimization of the parameters of the initial recognition model based on the global attention map, includes: The global attention map is subjected to complementary loss coupling learning using a global loss unit, a few-sample loss unit, and a center loss unit, and the parameters of the initial recognition model are optimized based on the learning results. The construction of the multi-path joint optimization module, and the optimization of the parameters of the initial recognition model based on the global attention map, further includes: Global average pooling is performed on the global attention map to obtain semantically enhanced features; The semantic enhancement features are learned by complementary loss coupling using a global loss unit, a few-sample loss unit, and a central loss unit.

2. The method according to claim 1, characterized in that, The method for determining whether the preset conditions are met includes: If the initial recognition model is optimized a preset number of times, or the output accuracy of the initial recognition model meets a preset accuracy, then the preset condition is considered to be met.

3. A small-sample fine-grained image recognition device, used to implement the method of claim 1 or 2, characterized in that, The device includes: The acquisition module is used to acquire training samples, which include multiple images; The training module is used to train an initial recognition model based on the training samples to obtain a trained recognition model; wherein the steps of training the initial recognition model include: The training samples are input into the initial recognition model, and a global attention map is generated based on the two-stream attention mechanism; A multi-path joint optimization module is constructed, and the parameters of the initial recognition model are optimized according to the global attention map until the preset conditions are met, so as to obtain a trained recognition model. The recognition module is used to identify small, fine-grained images of the target image using a trained recognition model, and to obtain the image category.

4. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 1 or 2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 1 or 2.

Images