Reinforced complementary fine-grained image classification network system

Abstract

The reinforced complementary fine-grained image classification network system comprises a reinforced complementary learning network structure for extracting features of a backbone network, driving two sub-networks to perform reinforcement learning and complementary learning respectively, thereby jointly extracting target features and achieving detailed and comprehensive identification of a recognized target object; a DM driving module for cropping and enlarging a region having the greatest impact on a result, deleting the region in an original image, and sending the region to the reinforced complementary learning network structure, which can help the model to perform end-to-end training; a DM loss function for enabling the DM driving module to locate a key region and constantly optimize position information of the reinforced complementary learning network structure while providing an accurate mask position; and a verification data set for verifying performance of the reinforced complementary learning network structure.

Description

Technical fields:

[0001] This invention relates to a fine-grained image classification network system with reinforcement complementarity. Background technology:

[0002] Image analysis uses mathematical models combined with image processing techniques to analyze underlying features and upper-level structures, extracting information data with a certain degree of intelligence. Image analysis focuses on constructing descriptive methods for images, using symbols to represent various images rather than performing calculations on the images themselves, and using various related knowledge for reasoning. Image analysis is also closely related to research on human vision; research on certain recognizable modules in the human visual mechanism can promote the improvement of computer vision capabilities.

[0003] Fine-grained images are an important part of image analysis. Because the objects they identify are various subcategories of different types, and the differences between subcategories are very subtle and concentrated in multiple local regions, fine-grained recognition tasks are very challenging. Some fine-grained networks tend to focus on a certain region when judging the target category, resulting in a lack of other auxiliary discrimination region features, thus failing to perform detailed and comprehensive recognition of the target. Summary of the Invention:

[0004] This invention provides a reinforced complementary fine-grained image classification network system with a reasonable structural design. It incorporates a reinforced complementary fine-grained image classification network, utilizing a backbone network for feature extraction and driving two sub-networks to perform reinforcement learning and complementary learning respectively. While employing reinforcement learning to acquire more detailed fine-grained image features, attention erasure is used to enable the complementary network to acquire supplementary discriminative regions of the target, thereby increasing the network's overall target perception capability. Validation experiments conducted on multiple public datasets evaluate the system model's performance, achieving detailed and comprehensive target recognition and improving the effectiveness of fine-grained image recognition. This system can be widely applied in classification tasks and solves the problems existing in the prior art.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0006] A fine-grained image classification network system with enhanced complementarity, the classification network system comprising:

[0007] The reinforcement complementary learning network structure is used to extract data from the backbone network and drive the other two sub-networks to perform reinforcement learning and complementary learning respectively, so as to jointly extract target features and achieve detailed and comprehensive recognition of the target object.

[0008] The DM-driven module is used to crop and enlarge the region that has the greatest impact on the result, delete the region from the original image, and feed it into the reinforcement complementary learning network structure, which can help the model to perform end-to-end training.

[0009] The DM loss function is used to enable the DM driving module to locate key areas, continuously optimize and reinforce the location information of the complementary learning network structure, and provide accurate mask locations.

[0010] Validation datasets, which are used to validate the performance of reinforcement complementarity learning network architectures, include CUB-200-2011, Stanford Cars, and FGVC-Aircraft;

[0011] The reinforcement-complementary learning network structure includes a base network, a reinforcement network, and a complementary network to construct a three-way classification network to aggregate the overall and local features of the target object. This allows us to obtain both the overall semantic information and the local semantic information of the object. Then, global average pooling is performed on the features output by each network, and the pooled features are concatenated to form a 6144 vector. A 200-dimensional classification layer is added to this vector for end-to-end training, and finally, the classification result is obtained through Softmax.

[0012] The DM driving module will crop and enlarge the region with the greatest impact on the result and send it into the enhancement network; the DM driving module will also delete the region with the greatest impact on the result from the original image and send it into the complementary network.

[0013] The DM driving module receives the feature map obtained after training the basic network. Then it generates a square region centered at (x,y) with half the length of l as the side length. It then crops and enlarges the region and sends it into the reinforcement network. At the same time, it generates an image mask based on the region and inputs it into the complementary network for complementary learning.

[0014] The DM driving module consists of two fully connected layers. The input is a feature map, and the output is the most important local region of the neural network. The most important local region can be automatically located through the fully connected layer.

[0015] Given an image X, it is input into a pre-trained convolutional layer for feature extraction. Tn represents the overall parameters. The entire process can be described as convolving, pooling, and activating X, finally generating a probability distribution p. The probability distribution formula is:

[0016] p(X)=f(Tn*X)

[0017] Where f(·) represents a fully connected layer, which transforms the features extracted by the convolutional neural network into a feature vector, and uses softmax to transform the vector into a probability value;

[0018] The initialization parameters of the DM driver module are calculated using the following formula:

[0019]

[0020] Where f represents the feature map output by the last layer in the convolutional neural network, n represents the number of feature maps, d represents the total number of feature maps, and F is the total feature map after adding up all the feature maps.

[0021] The formula for comparing the mean of the DM driver module is as follows:

[0022]

[0023] Where h and w represent the width and height of the feature map, respectively. The mean of the representative feature map;

[0024] pass and F i,j The size comparison generates the initial coordinates of the bounding box center. After obtaining the initial coordinates, the model can automatically optimize these coordinates based on the training process. Then, the region needs to be cropped and enlarged to obtain a more detailed local region before being fed into the reinforcement network for learning. The coordinates of the top-left and bottom-right corners of the local region are obtained based on the center coordinates and the side length. The top-left corner coordinates are denoted as (t...). lx ,t rx The coordinates of the lower right corner are (t) ly ,t ry The calculation formula is:

[0025] t lx =xl,t rx =yl

[0026] t ly =x+l,t ry =y+l;

[0027] The cropping operation can be viewed as a multiplication between the original image and the module, expressed as:

[0028]

[0029] Among them, X crop This is the cropped area. This represents the cropping operation between the original image and the template. M(·) is the attention mask, and its expression is:

[0030] M(·)=[μ(itlx )-μ(it ly )]×[μ(jt rx )-μ(jt ry )]

[0031] Where i and j are arbitrary points in the feature map. If i and j are inside the feature map, then the value of M(·) is 1; otherwise, the value is 0. Meanwhile, μ(·) is a continuously differentiable function, and its expression is:

[0032]

[0033] The bilinear interpolation algorithm is used to enlarge the size of the extracted local region. The defensive local region is obtained based on the ratio between the original image and the local region. The algorithm formula is as follows:

[0034]

[0035] in, and X a These represent the areas of the local region and the overall region, respectively. It is the area ratio, X local It is a magnified local area.

[0036] The DM loss function is:

[0037]

[0038] Where m represents the size of a batch, W represents the output of the fully connected layer, and y i Let x represent the category of the i-th image. i represents the feature vector of the i-th image before the fully connected layer, b represents the network bias, and s is the number of target categories. This is to continuously optimize and strengthen the network's positional information, while providing more accurate mask positions so that the complementary network can learn secondary features.

[0039] The verification method for the verification dataset includes the following steps:

[0040] S1, the feature extraction network is trained using the training weights of the backbone network Inception-V3 on Image Net. The parameters of the pooling layer, input layer and convolutional layer are retained, and the existing fully connected layer and softmax layer are removed to fine-tune the network and train the data used in this paper.

[0041] S2, by strengthening the computation of key regions in the complementary learning network structure, finds the coordinate information of the most critical region, cropps and amplifies it, and produces more refined training results.

[0042] This invention employs the aforementioned structure, using a reinforcement complementary learning network to extract features from the backbone network, driving the other two sub-networks to perform reinforcement learning and complementary learning respectively, thereby jointly extracting target features and achieving detailed and comprehensive recognition of the target object. The DM-driven module crops and enlarges the region with the greatest impact on the results, deleting this region from the original image before feeding it into the reinforcement complementary learning network structure, which helps the model perform end-to-end training. The DM loss function enables the DM-driven module to locate key regions, continuously optimizing the positional information of the reinforcement complementary learning network structure while providing accurate mask positions. The performance of the reinforcement complementary learning network structure is verified using a validation dataset, demonstrating its advantages of accuracy, practicality, and superior performance. Attached image description:

[0043] Figure 1 This is a schematic diagram of the structure of the present invention.

[0044] Figure 2 This is a network structure feature diagram of the present invention. Detailed implementation method:

[0045] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0046] like Figure 1-2 As shown, a complementary fine-grained image classification network system is provided, the classification network system comprising:

[0047] The reinforcement complementary learning network structure is used to extract data from the backbone network and drive the other two sub-networks to perform reinforcement learning and complementary learning respectively, so as to jointly extract target features and achieve detailed and comprehensive recognition of the target object.

[0048] The DM-driven module is used to crop and enlarge the region that has the greatest impact on the result, delete the region from the original image, and feed it into the reinforcement complementary learning network structure, which can help the model to perform end-to-end training.

[0049] The DM loss function is used to enable the DM driving module to locate key areas, continuously optimize and reinforce the location information of the complementary learning network structure, and provide accurate mask locations.

[0050] Validation datasets, which are used to validate the performance of reinforcement complementarity learning network architectures, include CUB-200-2011, Stanford Cars, and FGVC-Aircraft;

[0051] The reinforcement-complementary learning network structure includes a base network, a reinforcement network, and a complementary network to construct a three-way classification network to aggregate the overall and local features of the target object. This allows us to obtain both the overall semantic information and the local semantic information of the object. Then, global average pooling is performed on the features output by each network, and the pooled features are concatenated to form a 6144 vector. A 200-dimensional classification layer is added to this vector for end-to-end training, and finally, the classification result is obtained through Softmax.

[0052] The DM driving module will crop and enlarge the region with the greatest impact on the result and send it into the enhancement network; the DM driving module will also delete the region with the greatest impact on the result from the original image and send it into the complementary network.

[0053] The DM driving module receives the feature map obtained after training the basic network. Then it generates a square region centered at (x,y) with half the length of l as the side length. It then crops and enlarges the region and sends it into the reinforcement network. At the same time, it generates an image mask based on the region and inputs it into the complementary network for complementary learning.

[0054] The DM driving module consists of two fully connected layers. The input is a feature map, and the output is the most important local region of the neural network. The most important local region can be automatically located through the fully connected layer.

[0055] Given an image X, it is input into a pre-trained convolutional layer for feature extraction. Tn represents the overall parameters. The entire process can be described as convolving, pooling, and activating X, finally generating a probability distribution p. The probability distribution formula is:

[0056] p(X)=f(Tn*X)

[0057] Where f(·) represents a fully connected layer, which transforms the features extracted by the convolutional neural network into a feature vector, and uses softmax to transform the vector into a probability value;

[0058] The initialization parameters of the DM driver module are calculated using the following formula:

[0059]

[0060] Where f represents the feature map output by the last layer in the convolutional neural network, n represents the number of feature maps, d represents the total number of feature maps, and F is the total feature map after adding up all the feature maps.

[0061] The formula for comparing the mean of the DM driver module is as follows:

[0062]

[0063] Where h and w represent the width and height of the feature map, respectively. The mean of the representative feature map;

[0064] pass and F i,j The size comparison generates the initial coordinates of the bounding box center. After obtaining the initial coordinates, the model can automatically optimize these coordinates based on the training process. Then, the region needs to be cropped and enlarged to obtain a more detailed local region before being fed into the reinforcement network for learning. The coordinates of the top-left and bottom-right corners of the local region are obtained based on the center coordinates and the side length. The top-left corner coordinates are denoted as (t...). lx ,t rx The coordinates of the lower right corner are (t) ly ,t ry The calculation formula is:

[0065] t lx =xl,t rx =yl

[0066] t ly =x+l,t ry =y+l;

[0067] The cropping operation can be viewed as a multiplication between the original image and the module, expressed as:

[0068]

[0069] Among them, X crop This is the cropped area. This represents the cropping operation between the original image and the template. M(·) is the attention mask, and its expression is:

[0070] M(·)=[μ(it lx )-μ(it ly )]×[μ(jt rx )-μ(jt ry )]

[0071] Where i and j are arbitrary points in the feature map. If i and j are inside the feature map, then the value of M(·) is 1; otherwise, the value is 0. Meanwhile, μ(·) is a continuously differentiable function, and its expression is:

[0072]

[0073] The bilinear interpolation algorithm is used to enlarge the size of the extracted local region. The defensive local region is obtained based on the ratio between the original image and the local region. The algorithm formula is as follows:

[0074]

[0075] in, and X a These represent the areas of the local region and the overall region, respectively. It is the area ratio, X local It is a magnified local area.

[0076] The DM loss function is:

[0077]

[0078] Where m represents the size of a batch, W represents the output of the fully connected layer, and y i Let x represent the category of the i-th image. i represents the feature vector of the i-th image before the fully connected layer, b represents the network bias, and s is the number of target categories. This is to continuously optimize and strengthen the network's positional information, while providing more accurate mask positions so that the complementary network can learn secondary features.

[0079] The verification method for the verification dataset includes the following steps:

[0080] S1, the feature extraction network is trained using the training weights of the backbone network Inception-V3 on Image Net. The parameters of the pooling layer, input layer and convolutional layer are retained, and the existing fully connected layer and softmax layer are removed to fine-tune the network and train the data used in this paper.

[0081] S2, by strengthening the computation of key regions in the complementary learning network structure, finds the coordinate information of the most critical region, cropps and amplifies it, and produces more refined training results.

[0082] The working principle of the reinforcement-complementary fine-grained image classification network system in this embodiment of the invention is as follows: A reinforcement-complementary fine-grained image classification network is added. The backbone network is used for feature extraction, driving two sub-networks to perform reinforcement learning and complementary learning respectively. While adopting the learning method of the reinforcement model to obtain more detailed fine-grained image features, attention erasure is used to enable the complementary network to obtain the supplementary discriminative region of the target, thereby increasing the network's overall perception ability of the target. Combined with the verification experiments carried out on multiple public datasets, the performance of the system model is evaluated, achieving detailed and comprehensive recognition of the target, improving the effect of fine-grained image recognition, and can be widely applied in classification tasks.

[0083] For a recognition network, the features of interest are often concentrated on a certain region of the target, which becomes the most important feature for recognizing the target. However, the model designed in this application can recognize targets in a wider range. That is, it no longer depends on a certain salient feature, but can also achieve detailed and comprehensive recognition of the target by relying on secondary features.

[0084] The reinforcement-complementary learning network structure includes a base network, a reinforcement network, and a complementary network to construct a three-way classification network that aggregates the overall and local features of the target object. This allows for the acquisition of both the overall and local semantic information of the object. Then, global average pooling is performed on the features output by each network, and the pooled features are concatenated to form a 6144 vector. A 200-dimensional classification layer is added to this vector for end-to-end training, and finally, the classification result is obtained through Softmax.

[0085] Since traditional neural networks do not leverage the advantages of deep neural networks for localization and recognition learning, this application proposes a DM-driven module to help the backbone network find the rectangular region that has the greatest impact on the results during training. Specifically, it has two functions: firstly, it crops and enlarges the region that has the greatest impact on the results and feeds it into the reinforcement network; secondly, it erases the rectangular region in the original image and feeds it into the complementary network. At the same time, the DM-driven module has a very low computational cost and can help the model perform end-to-end training.

[0086] The DM driver module receives the feature map obtained after training the base network. Then it generates a square region centered at (x,y) with a side length of half l. It then crops and enlarges this region before feeding it into the reinforcement network. At the same time, it can also generate an image mask based on this region and input it into the complementary network for complementary learning.

[0087] In this process, the high-response region of the feature map is the key to obtaining coordinates (x,y). The DM driving module consists of two fully connected layers. The input is the feature map, and the output is the most important local region of the neural network. The most important local region can be automatically located through the fully connected layer. Therefore, the size of the bounding box is limited. It cannot exceed 2 / 3 of the longest side of the overall image and cannot be less than 1 / 3 of the shortest side of the image.

[0088] Specifically, given an image X, it is input into a pre-trained convolutional layer for feature extraction. Tn represents the overall parameters. The entire process can be described as convolving, pooling, and activating X, finally generating a probability distribution p. The probability distribution formula is:

[0089] p(X)=f(Tn*X)

[0090] Here, f(·) represents a fully connected layer, which converts the features extracted by the convolutional neural network into a feature vector, and uses softmax to convert the vector into a probability value.

[0091] The next step is to generate the position and length parameters of the square bounding box, for

[0092] [x,y,l]=g(Tn*X)

[0093] Where x, y, l are the horizontal and vertical coordinates of the bounding box in X and half the side length, respectively. g(·) represents the DM driving module, which consists of two fully connected layers. Since the weight parameters of the network initialization have a great influence on the model, the feature maps output by the last layer of the basic network are added together to make the semantic information of the feature maps richer and the generated bounding box more accurate.

[0094] The formula for calculating the initialization parameters of the DM driver module is as follows:

[0095]

[0096] Where f represents the feature map output by the last layer in the convolutional neural network, n represents the number of feature maps, d represents the total number of feature maps, and F is the total feature map after adding up all the feature maps.

[0097] Furthermore, the formula for comparing the mean values ​​of the DM driver modules is as follows:

[0098]

[0099] Where h and w represent the width and height of the feature map, respectively. The mean of the representative feature map;

[0100] pass and F i,j The size comparison generates the initial coordinates of the bounding box center. After obtaining the initial coordinates, the model can automatically optimize these coordinates based on the training process. Then, the region needs to be cropped and enlarged to obtain a more detailed local region before being fed into the reinforcement network for learning. The coordinates of the top-left and bottom-right corners of the local region are obtained based on the center coordinates and the side length. The top-left corner coordinates are denoted as (t...). lx ,t rx The coordinates of the lower right corner are (t) ly ,t ry The calculation formula is:

[0101] t lx =xl,t rx =yl

[0102] t ly =x+l,t ry =y+l.

[0103] Once the corresponding coordinate information is obtained, the cropping operation can be viewed as a multiplication between the original image and the module, expressed as:

[0104]

[0105] Among them, X crop This is the cropped area. This represents the cropping operation between the original image and the template. M(·) is the attention mask, and its expression is:

[0106] M(·)=[μ(it lx )-μ(it ly )]×[μ(jt rx )-μ(jt ry )]

[0107] Where i and j are arbitrary points in the feature map. If i and j are inside the feature map, then the value of M(·) is 1; otherwise, the value is 0. Meanwhile, μ(·) is a continuously differentiable function, and its expression is:

[0108]

[0109] To crop and enlarge the image, bilinear interpolation is used to expand the size of the extracted local region. Based on the ratio of the original image to the local region, the enlarged local region can be obtained. The algorithm formula is as follows:

[0110]

[0111] in, and X a These represent the areas of the local region and the overall region, respectively. It is the area ratio, X local It is a magnified local area.

[0112] Similarly, to train the complementary network, the generated local regions are transformed into mask images. The pixels of the mask are uniformly set to the average pixel value of the original image, and the remaining parts are replaced with white pixels, as shown in the following formula:

[0113]

[0114] Next, the mask is erased from the original image based on the previously obtained position information. The resulting masked image is then fed into the complementary model for training. The specific process is as follows:

[0115]

[0116] In the above formula, the values ​​at each position in the pixel matrix formed by the original image represent different pixels. In the mask image, 1 represents a black pixel, and the pixel value in the RGB channel is (0, 0, 0). By calculating the position using the original image and the mask, the black pixel part is directly filled with the original image pixels, while the mask pixels replace the original image pixels. This is how the image after erasing the key area is obtained.

[0117] For the DM loss function, since a suitable loss function has a positive impact on model training, the commonly used loss function for fine-grained image recognition is the softmax loss function, the specific formula of which is:

[0118]

[0119] Where m represents the size of a batch, W represents the output of the fully connected layer, yi represents the category of the i-th image, xi represents the feature vector of the i-th image before the fully connected layer, b represents the network bias, and s is the number of target categories. This is to continuously optimize and strengthen the location information of the network, while providing more accurate mask positions so that the complementary network can learn secondary features.

[0120] To help reinforce complementary learning network structures to find more accurate features, the probability values ​​p of the samples output by the backbone model can be increased. k The probability value p generated by the reinforcement model k+1 In comparison, referring to the difference generated by the two models, when p k >p k +1 No loss occurs when p k <p k+1 When this occurs, a loss is incurred; therefore, this loss function can help strengthen the network to find more accurate features, and after extracting accurate features, it can help the backbone network to locate more accurately, and the two reinforce each other.

[0121] Meanwhile, in the complementary model, since the features extracted by the backbone feature extraction will be deleted, the features extracted by the backbone model have no relation to those of the complementary model. However, the precise local regions provided by the backbone model are helpful for the complementary model to learn secondary features. Therefore, it is only necessary to ensure that the backbone model and the reinforcement model can locate the key local regions. From the above example, we can see that the total loss of the model is:

[0122]

[0123] in These are the modulation coefficients, used to balance the two loss functions.

[0124] For the validation datasets in this application, which mainly include CUB-200-2011, Stanford Cars and FGVC-Aircraft, in order to improve the model's ability to focus on secondary features, a small portion of the data may need to be erased multiple times in key regions to obtain all features. Therefore, it is necessary to conduct erasure experiments on the datasets used to find the appropriate number of erasures.

[0125] The validation method for the validation dataset includes the following steps: The feature extraction network is trained using the training weights of the Inception-V3 backbone network on ImageNet. Pooling layers, input layers, and convolutional layer parameters are retained, while existing fully connected layers and softmax layers are removed to fine-tune the network and train it on the data used in this paper. The network structure is reinforced by enhancing the computation of key regions, finding the coordinate information of the most critical regions, cropping and amplifying them to produce more refined training results.

[0126] It should be noted that the experimental environment was conducted using PyTorch version 1.71, with an Nvidia Genforce 3060 Ti GPU and an i7-10700K CPU. The optimizer selected was SGD, with an initial learning rate of 0.0001, a momentum hyperparameter of 0.9, a batch size of 32, and 200 epochs for each epoch.

[0127] In summary, the reinforcement-complementary fine-grained image classification network system in this embodiment adds a reinforcement-complementary fine-grained image classification network. It utilizes a backbone network for feature extraction and drives two sub-networks to perform reinforcement learning and complementary learning respectively. While employing reinforcement model learning methods to acquire more detailed fine-grained image features, it uses attention erasure to enable the complementary network to acquire supplementary discriminative regions of the target, thereby increasing the network's overall target perception capability. Combined with validation experiments conducted on multiple public datasets, the performance of the system model is evaluated, achieving detailed and comprehensive target recognition and improving the effect of fine-grained image recognition. This system can be widely applied in classification tasks.

[0128] The above specific embodiments should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, any alternative improvements or modifications made to the embodiments of the present invention shall fall within the scope of protection of the present invention.

[0129] Any aspects of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A fine-grained image classification network system with enhanced complementarity, characterized in that, The classification network system includes: The reinforcement complementary learning network structure is used to extract data from the backbone network and drive the other two sub-networks to perform reinforcement learning and complementary learning respectively, so as to jointly extract target features and achieve detailed and comprehensive recognition of the target object. The DM-driven module is used to crop and enlarge the region that has the greatest impact on the result, delete the region from the original image, and feed it into the reinforcement complementary learning network structure, which can help the model to perform end-to-end training. The DM loss function is used to enable the DM driving module to locate key areas, continuously optimize and reinforce the location information of the complementary learning network structure, and provide accurate mask locations. Validation datasets, which are used to validate the performance of reinforcement complementarity learning network architectures, include CUB-200-2011, Stanford Cars, and FGVC-Aircraft; The reinforcement-complementary learning network structure includes a base network, a reinforcement network, and a complementary network to construct a three-way classification network to aggregate the overall and local features of the target object. This allows us to obtain both the overall semantic information and the local semantic information of the object. Then, global average pooling is performed on the features output by each network, and the pooled features are concatenated to form a 6144 vector. A 200-dimensional classification layer is added to this vector for end-to-end training, and finally, the classification result is obtained through Softmax. The DM driving module will crop and enlarge the region with the greatest impact on the result and send it into the enhancement network; the DM driving module will also delete the region with the greatest impact on the result from the original image and send it into the complementary network. The DM driver module receives the feature map obtained after training the base network, and then it generates a feature map with... Centered on a square region with a side length of half that of l, the region is cropped, enlarged, and fed into the reinforcement network. Simultaneously, an image mask is generated based on this region and input into the complementarity network for complementary learning. The DM driving module consists of two fully connected layers. The input is a feature map, and the output is the most important local region of the neural network. The most important local region can be automatically located through the fully connected layer. Given an image X, input it into a pre-trained convolutional layer for feature extraction. Representing the overall parameters, the entire process can be described as performing convolution, pooling, and activation on X, finally generating a probability distribution p. The probability distribution formula is: in, This represents a fully connected layer, which transforms the features extracted by the convolutional neural network into feature vectors and uses softmax to convert these vectors into probability values. The initialization parameters of the DM driver module are calculated using the following formula: Where f represents the feature map output by the last layer in the convolutional neural network, n represents the number of feature maps, d represents the total number of feature maps, and F is the total feature map after adding up all the feature maps. The formula for comparing the mean of the DM driver module is as follows: in, h, w These represent the width and height of the feature map, respectively. The mean of the representative feature map; pass and The size comparison generates the initial coordinates of the bounding box center. After obtaining the initial coordinates, the model can automatically optimize these coordinates based on the training process. Then, the region needs to be cropped and enlarged to obtain a more detailed local region before being fed into the reinforcement network for learning. The coordinates of the top-left and bottom-right corners of the local region are obtained based on the center coordinates and the side length. The top-left corner coordinates are denoted as... The coordinates in the lower right corner are The calculation formula is: ； The cropping operation can be viewed as a multiplication between the original image and the module, expressed as: in, This is the cropped area. This indicates a cropping operation between the original image and the template. The attention mask is expressed as follows: in, i , j At any point in the feature map, if i , j Located inside the feature map, then The value is 1 if it is true, otherwise the value is 0; at the same time It is a continuously differentiable function, and its expression is: ； The bilinear interpolation algorithm is used to enlarge the size of the extracted local region. The defensive local region is obtained based on the ratio between the original image and the local region. The algorithm formula is as follows: in, and These represent the areas of the local region and the overall region, respectively. It's the area ratio. It is a magnified local area.

2. The fine-grained image classification network system with reinforcement complementarity according to claim 1, characterized in that, The DM loss function is: Where m represents the size of a batch, W represents the output of the fully connected layer, yi represents the category of the i-th image, xi represents the feature vector of the i-th image before the fully connected layer, b represents the network bias, and s is the number of target categories. This is to continuously optimize and strengthen the location information of the network, while providing more accurate mask positions so that the complementary network can learn secondary features.

3. The fine-grained image classification network system with reinforcement complementarity according to claim 1, characterized in that, The verification method for the verification dataset includes the following steps: S1, the feature extraction network is trained using the training weights of the backbone network Inception-V3 on Image Net. The parameters of the pooling layer, input layer and convolutional layer are retained, and the existing fully connected layer and softmax layer are removed to fine-tune the network and train the data used in this paper. S2, by strengthening the computation of key regions in the complementary learning network structure, finds the coordinate information of the most critical region, cropps and amplifies it, and produces more refined training results.

Images