Image recognition transfer reinforcement learning method and device based on grad-cam

Abstract

The application relates to the technical field of migration reinforcement learning, and provides a Grad-CAM-based image recognition migration reinforcement learning method and device.The application replaces the feature extractor of a second intelligent agent model with the feature extractor of a trained first intelligent agent model in a training environment, so that the migration of the feature extractor is realized; the parameters of the unfreezed features of the replaced feature extractor are obtained through Grad-CAM; and the parameters of the unfreezed features of the replaced feature extractor are trained and adjusted in a target domain, so that the influence caused by the feature offset between the target domain and the source domain is eliminated, the learning efficiency and performance of the intelligent agent model after the migration of the feature extractor are improved, and the number of parameters that need to be adjusted and trained is reduced, and the model complexity is reduced.

Description

Technical Field

[0001] This application relates to the field of transfer reinforcement learning technology, and in particular to an image recognition transfer reinforcement learning method and apparatus based on Grad-CAM. Background Technology

[0002] In real-world applications, due to the rapid and complex changes in situation, the agent model used for image recognition on aircraft can only undergo minimal training for the current environment before being deployed. This is because the original agent model used for transfer learning cannot be trained extensively and sufficiently, as this would lead to feature shift issues in the target domain with limited training. The model becomes dependent on features from the source domain, which is difficult to correct, making it difficult to adapt to the target domain distribution. The main reason why transfer learning with insufficient training yields less improvement than direct training in the target domain is that fine-tuning irrelevant features introduces noise. Especially with insufficient training, the model tends to remember irrelevant details instead of learning important features for image recognition. Furthermore, too many adjustable feature parameters require more data, and with limited training, the model struggles to overcome its dependence on the source domain feature distribution, making it difficult to adapt to the target domain feature distribution.

[0003] Therefore, a technical solution is urgently needed to solve the above problems. Summary of the Invention

[0004] Therefore, it is necessary to provide an image recognition transfer reinforcement learning method and apparatus based on Grad-CAM to address the aforementioned technical problems.

[0005] A Grad-CAM-based image recognition transfer reinforcement learning method includes the following steps: Construct a first intelligent agent model, in which ResNet18 is used as the feature extractor of an online Q-network, and the fully connected layer of ResNet18 is the fully connected layer that outputs the Q value of the action. Create a training environment and perform deep learning training on the first agent model in the source domain to obtain the feature extractor of the trained first agent model and the images that the first agent model misidentified during the training process in the source domain. Using the feature maps of incorrectly identified images output from the terminal convolutional layer of ResNet18 as input samples and the true category of the incorrectly identified images as the target category, Grad-CAM is used to generate heatmaps corresponding to the feature maps of the incorrectly identified images. Based on the heatmap and a preset positive contribution threshold, obtain the set of two-dimensional coordinates corresponding to the important regions of image recognition on the image with incorrect recognition; Two-dimensional coordinates are mapped onto the feature map of layer 4 of the feature extractor, and the activation values ​​of all channel features at a specified position on the feature map are obtained based on the mapping result. Furthermore, the channel feature with the strongest response at each two-dimensional coordinate after mapping is obtained and recorded as the important feature set corresponding to the image that was misidentified. Obtain the set of important features corresponding to all images that were misidentified, and take the union of all important feature sets to obtain the important feature index array; Create a second agent model that is identical to the first agent model, replace the feature extractor of the second agent model with the feature extractor of the trained first agent model, and unfreeze the parameters of the features in the replacement feature extractor that are the same as the important feature index array. In the target domain, the parameters of the features thawed after replacement by gradient descent training are adjusted to obtain the second agent model with fine-tuned parameters, thus completing the transfer reinforcement learning.

[0006] In one embodiment, the source domain is a visible light image of a real aircraft under normal field of view, and the target domain is a grayscale image of a real aircraft under infrared detection.

[0007] In one embodiment, using the feature map of the incorrectly identified image output from the ResNet18 terminal convolutional layer as the input sample, and the true category of the incorrectly identified image as the target category, Grad-CAM is used to generate a heatmap corresponding to the feature map of the incorrectly identified image, including: Using the feature maps of incorrectly identified images output from the final convolutional layer of ResNet18 as input samples, and the true class of the incorrectly identified images as the target class, Grad-CAM is used to obtain the weights of each channel feature of the input sample and the corresponding target class: ; in, For target category c No. k Weights of each channel feature; For target category c Score; Z represents the spatial dimension of the feature map; For the first k Feature maps of each channel feature; i This represents the coordinate position of the feature map in the vertical direction; j This represents the coordinate position of the feature map in the horizontal direction; For the first k The feature map of each channel feature is in ( i , j The eigenvalue at () location; The obtained weights are weighted and summed with the original heatmap of the input sample to generate a heatmap corresponding to the feature map of the misidentified image: ; in, For target category c The corresponding heatmap.

[0008] In one embodiment, a set of two-dimensional coordinates corresponding to important regions of image recognition on an image with incorrect recognition is obtained based on a heatmap and a preset positive contribution threshold, including: The values ​​on the heatmap are normalized, and the set of two-dimensional coordinates corresponding to the important regions of image recognition in the image with incorrect recognition is found based on the preset positive contribution threshold.

[0009] In one embodiment, a training environment is created to perform deep learning training on a first agent model in the source domain, resulting in a feature extractor for the trained first agent model and images that the first agent model misidentified during the source domain training process, including: The training environment is created and the values ​​of the network parameters are determined. The image input to the neural network is defined as the state, the image category as the action, and the total number of categories as the action space size. The experience tuple in the experience replay buffer is defined as including the current state, the next state, the action, the reward, and the termination flag. The reward function is defined as +1 for correctly identifying a single image and -1 for an error. Only one image is identified in each training session. The network parameters include experience replay capacity, discount factor, exploration rate, exploration rate decay rate, learning rate, number of training iterations, and experience batch size. The exploration rate includes the initial exploration rate and the final exploration rate. The first agent model is trained in the current round using the image in the source domain as input, and the training environment is reset. The action is selected based on the exploration rate. If the random value is lower than the exploration rate, the action is selected randomly. If the random value is higher than the exploration rate, the action with the largest Q value is selected. Execute the selected action, obtain and package the next state, action, reward and termination flag into an experience tuple and store it in the experience replay buffer; When the number of experience tuples in the experience replay buffer reaches the experience batch size, sample from the experience replay buffer and calculate the current Q value, the next Q value, and the target Q value. The loss is calculated and backpropagation and gradient pruning are performed. The loss function is calculated according to the following formula: ; in, For network parameters; Network parameters for the target Q value; For state Take action below The instant reward obtained afterward; This serves as a buffer for experience replay. To ensure uniform sampling from the experience playback buffer; This is the expected value; The loss function; For network parameters The calculated state Select action The optimal value; Network parameters determined by the target Q value The calculated state Select action The optimal value; Discount factor; The network parameters are updated based on the loss, and the current exploration rate is decayed based on a preset exploration rate decay coefficient after the network parameters are updated. End the current training round, determine whether the preset number of training iterations has been reached. If it has, output the trained first agent model, the feature extractor of the trained first agent model, and the images that the first agent model misidentified during the current training round in the source domain. If it has not reached the preset number of training iterations, proceed to the next training round.

[0010] In one embodiment, the Q value is calculated according to the following formula: ; in, Q value, indicating the state Select action The optimal value; State; For action; The next state; The action to be chosen for the next state; Discount factor; For state Select action The optimal value; For the reward function; Let be the state transition probability.

[0011] In one embodiment, the ResNet18 consists of 16 convolutional layers and 2 fully connected layers.

[0012] In one embodiment, the ResNet18 employs a layered stacked residual block design, with each residual block containing two sets of 3×3 convolutional layers, batch normalization, and ReLU activation functions.

[0013] In one embodiment, the parameters of the features thawed from the replaced feature extractor are adjusted through gradient descent training in the target domain to obtain a second agent model with fine-tuned parameters, thus completing transfer reinforcement learning, including: In the target domain, the weights and bias parameters of the unfrozen features of the replaced feature extractor are adjusted through gradient descent training. The feature extractor with adjusted parameters is used as the feature extractor of the second agent model to obtain the second agent model with fine-tuned parameters, thus completing the transfer reinforcement learning.

[0014] A Grad-CAM-based image recognition transfer reinforcement learning device includes: The agent construction module is used to construct the first agent model, which uses ResNet18 as the feature extractor of the online Q network, and the fully connected layer of ResNet18 is the fully connected layer that outputs the Q value of the action. The source domain training module is used to create a training environment to perform deep learning training on the first agent model in the source domain, and obtain the feature extractor of the trained first agent model and the images that the first agent model misidentified during the source domain training process. The heatmap generation module is used to generate heatmaps corresponding to the feature maps of the incorrectly identified images as input samples and the true category of the incorrectly identified images as the target category through Grad-CAM. The two-dimensional coordinate acquisition module is used to obtain the set of two-dimensional coordinates corresponding to the important regions of image recognition on the image with recognition errors based on the heat map and the preset positive contribution threshold. The important feature set acquisition module is used to map two-dimensional coordinates onto the feature map of layer 4 of the feature extractor, and obtain the activation values ​​of all channel features at a specified position on the feature map based on the mapping result. It further obtains the channel feature with the strongest response at each two-dimensional coordinate after mapping and records it as the important feature set corresponding to the image that was misidentified. The important feature index array acquisition module is used to obtain the set of important features corresponding to all images that are misidentified, and take the union of all important feature sets to obtain the important feature index array. The migration module is used to create a second agent model that is consistent with the first agent model. It replaces the feature extractor of the second agent model with the feature extractor of the trained first agent model and unfreezes the parameters of the features in the replacement feature extractor that are the same as the important feature index array. The parameter fine-tuning module is used to adjust the parameters of the features that have been thawed after replacement by gradient descent training in the target domain, thereby obtaining the second agent model with fine-tuned parameters and completing the transfer reinforcement learning.

[0015] The aforementioned Grad-CAM-based image recognition transfer reinforcement learning method and apparatus utilizes ResNet18 as the feature extractor of an online Q-network, enabling the agent model to effectively extract multi-level features with generalizable characteristics. Simultaneously, the fully connected layers used for classification in ResNet18 are removed and replaced with fully connected layers that output action Q-values, thus making the ResNet18 output the action Q-values. A training environment is created where the first agent model is trained using the source domain as input, yielding the trained feature extractor and images incorrectly identified during source domain training. The trained feature extractor then replaces the feature extractor of the second agent model, achieving feature transfer. Grad-CAM generates heatmaps corresponding to the feature maps of the incorrectly identified images and further obtains the important feature sets corresponding to all incorrectly identified images. The union of all important feature sets yields an important feature index array, thus obtaining... The parameters of the thawed features of the replaced feature extractor are obtained. By training and adjusting the parameters of the thawed features of the replaced feature extractor in the target domain, the influence of feature offset between the target domain and the source domain is eliminated. This improves the learning efficiency and performance of the agent model in target domain image recognition after feature extractor transfer. Since only the parameters of the thawed features need to be trained and adjusted, the number of parameters that need to be adjusted is reduced, thus reducing the model complexity. Furthermore, the parameters of the thawed features correspond to important features, allowing the model to ignore the interference of secondary features. This allows computing resources to be concentrated on optimizing features with strong discriminative power in the target domain, thereby accelerating convergence. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an image recognition transfer reinforcement learning method based on Grad-CAM, as provided in one embodiment. Figure 2 A selection diagram of an important feature index array provided for one embodiment; Figure 3 This is an accuracy analysis graph of an image recognition transfer reinforcement learning method based on parameter fine-tuning under different training amounts, provided in one embodiment. Figure 4An accuracy analysis diagram of an image recognition transfer reinforcement learning method based on Grad-CAM provided in one embodiment under different training amounts; Figure 5 This is an analysis of the accuracy of a parameter-fine-tuned image recognition transfer reinforcement learning method for different categories under different training amounts, as provided in one embodiment. Figure 5 (a) is a graph showing the accuracy analysis of each category under normal training volume. Figure 5 (b) is a plot showing the accuracy analysis for each class with one-quarter of the training data. Figure 5 (c) is a graph showing the accuracy of each class under one-eighth of the training volume; Figure 6 This is an analysis of the accuracy of a Grad-CAM-based image recognition transfer reinforcement learning method under different training levels, provided in one embodiment. Figure 6 (a) is a graph showing the accuracy analysis of each category under normal training volume. Figure 6 (b) is a plot showing the accuracy analysis for each class with one-quarter of the training data. Figure 6 (c) is a graph showing the accuracy of each class under one-eighth of the training volume; Figure 7 This is a recall analysis chart for different categories of an image recognition transfer reinforcement learning method based on parameter fine-tuning under different training amounts, provided in one embodiment. Figure 7 (a) is a graph showing the recall rate analysis for each category under normal training conditions. Figure 7 (b) is a graph showing the recall rate analysis for each class with one-quarter of the training time. Figure 7 (c) is a graph showing the recall rate of each category under one-eighth of the training volume; Figure 8 This is a graph showing the recall rate analysis of different categories under different training levels for an image recognition transfer reinforcement learning method based on Grad-CAM, provided in one embodiment. Figure 8 (a) is a graph showing the recall rate analysis for each category under normal training conditions. Figure 8 (b) is a graph showing the recall rate analysis for each class with one-quarter of the training time. Figure 8 (c) is a graph showing the recall rate of each category under one-eighth of the training volume; Figure 9 Here is an analysis of the F1 scores of different categories for an image recognition transfer reinforcement learning method based on parameter fine-tuning under different training amounts, as provided in one embodiment. Figure 9 (a) is a graph showing the F1 scores for each category under normal training conditions. Figure 9 (b) is a graph showing the F1 scores for each category under one-quarter training time. Figure 9 (c) is a graph showing the F1 scores for each category under one-eighth of the training volume; Figure 10Here is an analysis of the F1 scores of different categories for an image recognition transfer reinforcement learning method based on Grad-CAM under different training amounts, as provided in one embodiment. Figure 10 (a) is a graph showing the F1 scores for each category under normal training conditions. Figure 10 (b) is a graph showing the F1 scores for each category under one-quarter training time. Figure 10 (c) is a graph showing the F1 scores for each category under one-eighth of the training volume; Figure 11 A comparison chart of the relative accuracy of different methods provided in one embodiment at various training levels; Figure 12 This is a comparison chart of the relative accuracy of different methods for each category under various training amounts, provided in one embodiment. Figure 12 (a) is a comparison of the relative accuracy of each category under normal training volume. Figure 12 (b) is a comparison chart of the relative accuracy of each class under one-quarter training volume. Figure 12 (c) is a comparison chart of the relative accuracy of each category under one-eighth of the training volume; Figure 13 This is a comparison chart of the relative recall rates of different methods for each category at various training levels, provided in one embodiment. Figure 13 (a) is a comparison chart of the relative recall rates of each category under normal training conditions. Figure 13 (b) is a comparison chart of relative recall rates for each class at one-quarter training volume. Figure 13 (c) is a comparison chart of relative recall rates for each category under one-eighth of the training volume; Figure 14 This is a comparison chart of relative F1 scores for each class under different training levels for one embodiment. Figure 14 (a) is a comparison of the relative F1 scores for each category under normal training conditions. Figure 14 (b) is a comparison chart of relative F1 scores for each category under one-quarter training time. Figure 14 (c) is a comparison chart of the relative F1 scores of each category under one-eighth of the training volume; Figure 15 This is a structural block diagram of an image recognition transfer reinforcement learning device based on Grad-CAM, provided in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] In one embodiment, such as Figure 1 As shown, an image recognition transfer reinforcement learning method based on Grad-CAM is provided, including the following steps: Step 201: Construct the first intelligent agent model. The first intelligent agent model uses ResNet18 as the feature extractor of the online Q network. The fully connected layer of ResNet18 is the fully connected layer that outputs the Q value of the action.

[0020] Step 202: Create a training environment and perform deep learning training on the first agent model in the source domain to obtain the feature extractor of the trained first agent model and the images that the first agent model misidentified during the training process in the source domain.

[0021] Step 203: Using the feature map of the incorrectly identified image output from the ResNet18 terminal convolutional layer as the input sample and the true category of the incorrectly identified image as the target category, a heatmap corresponding to the feature map of the incorrectly identified image is generated through Grad-CAM.

[0022] Step 204: Based on the heatmap and the preset positive contribution threshold, obtain the set of two-dimensional coordinates corresponding to the important regions of image recognition on the image with recognition errors.

[0023] Step 205: Map the two-dimensional coordinates onto the feature map of layer 4 of the feature extractor, and obtain the activation values ​​of all channel features at the specified position on the feature map based on the mapping result. Further obtain the channel feature with the strongest response at each two-dimensional coordinate after mapping, and record it as the important feature set corresponding to the image that was misidentified.

[0024] It's important to note that the 512-channel features of layer 4 of the feature extractor are abstract vectors. We only know that these 512 dimensions are used to represent high-level semantics in the image, but we cannot know their specific meanings. By obtaining the two-dimensional coordinates of important regions for image recognition from the original image (i.e., the image that was misidentified), and then mapping these coordinates onto the feature map of layer 4, we detect the activation values ​​of all dimensions of the mapped locations on the feature map to find the feature with the strongest response. This feature is the most important feature in the original image corresponding to the two-dimensional coordinates.

[0025] Step 206: Obtain the set of important features corresponding to all images that were misidentified, and take the union of all the sets of important features to obtain the important feature index array.

[0026] Step 207: Create a second agent model that is identical to the first agent model, replace the feature extractor of the second agent model with the feature extractor of the trained first agent model, and unfreeze the parameters of the features in the replacement feature extractor that are the same as the important feature index array.

[0027] Step 208: In the target domain, the parameters of the features thawed after replacement by gradient descent training are adjusted to obtain the second agent model with fine-tuned parameters, thus completing the transfer reinforcement learning.

[0028] In the aforementioned Grad-CAM-based image recognition transfer reinforcement learning method, ResNet18 is used as the feature extractor of an online Q-network, enabling the agent model to effectively extract multi-level features with generalizable characteristics. Simultaneously, the fully connected layer used for classification in ResNet18 is removed and replaced with a fully connected layer that outputs action Q-values, thus making the ResNet18 output the action Q-values. A training environment is created where the first agent model is trained using the source domain as input, yielding the feature extractor of the trained first agent model and the images incorrectly identified during source domain training. The feature extractor of the trained first agent model then replaces the feature extractor of the second agent model, achieving feature transfer. Grad-CAM generates heatmaps corresponding to the feature maps of the incorrectly identified images, and further obtains the important feature sets corresponding to all incorrectly identified images. The union of all important feature sets yields an important feature index array, thus obtaining... The parameters of the thawed features of the replaced feature extractor are obtained. By training and adjusting the parameters of the thawed features of the replaced feature extractor in the target domain, the influence of feature offset between the target domain and the source domain is eliminated. This improves the learning efficiency and performance of the agent model in target domain image recognition after feature extractor transfer. Since only the parameters of the thawed features need to be trained and adjusted, the number of parameters that need to be adjusted is reduced, thus reducing the model complexity. Furthermore, the parameters of the thawed features correspond to important features, allowing the model to ignore the interference of secondary features. This allows computing resources to be concentrated on optimizing features with strong discriminative power in the target domain, thereby accelerating convergence.

[0029] In one embodiment, the source domain is the recognition of a real aircraft visible light image under normal field of view, and the target domain is the recognition of a real aircraft grayscale image under infrared detection.

[0030] In one embodiment, step 202 involves creating a training environment and performing deep learning training on the first agent model in the source domain to obtain a feature extractor for the trained first agent model and images that the first agent model misidentified during the source domain training process, including: The training environment is created and the values ​​of the network parameters are determined. The image input to the neural network is defined as the state, the image category as the action, and the total number of categories as the action space size. The experience tuple in the experience replay buffer is defined as including the current state, the next state, the action, the reward, and the termination flag. The reward function is defined as +1 for correctly identifying a single image and -1 for an error. Only one image is identified in each training session. The network parameters include experience replay capacity, discount factor, exploration rate, exploration rate decay rate, learning rate, number of training iterations, and experience batch size. The exploration rate includes the initial exploration rate and the final exploration rate.

[0031] The first agent model is trained in the current round using the image in the source domain as input, and the training environment is reset.

[0032] The action is selected based on the exploration rate. If the random value is lower than the exploration rate, the action is selected randomly. If the random value is higher than the exploration rate, the action with the highest Q value is selected.

[0033] Execute the selected action, retrieve and package the next state, action, reward, and termination flag into an experience tuple and store it in the experience replay buffer.

[0034] When the number of experience tuples in the experience replay buffer reaches the experience batch size, sample from the experience replay buffer and calculate the current Q value, the next Q value, and the target Q value.

[0035] The loss is calculated and backpropagation and gradient pruning are performed. The loss function is calculated according to the following formula: ; in, For network parameters; Network parameters for the target Q value; For state Take action below The instant reward obtained afterward; This serves as a buffer for experience replay. To ensure uniform sampling from the experience playback buffer; This is the expected value; The loss function; For network parameters The calculated state Select action The optimal value; Network parameters determined by the target Q value The calculated state Select action The optimal value; This is the discount factor.

[0036] The network parameters are updated based on the loss, and the current exploration rate is decayed based on a preset exploration rate decay coefficient after the network parameters are updated.

[0037] End the current training round, determine whether the preset number of training iterations has been reached. If it has, output the trained first agent model, the feature extractor of the trained first agent model, and the images that the first agent model misidentified during the current training round in the source domain. If it has not reached the preset number of training iterations, proceed to the next training round.

[0038] In this embodiment, an experience replay mechanism is introduced. An experience replay buffer stores experience tuples composed of transfer sample data generated by the exchange between the agent model and the environment. During the model network parameter update phase, samples are taken from the experience replay buffer to calculate the current Q-value, the next Q-value, and the target Q-value. The loss is then calculated to update the network parameters, thereby effectively improving learning efficiency. Simultaneously, gradient pruning is performed to prevent gradient explosion.

[0039] In one embodiment, the Q value is calculated according to the following formula: ; in, Q value, indicating the state Select action The optimal value; State; For action; The next state; The action to be chosen for the next state; Discount factor; For state Select action The optimal value; For the reward function; Let be the state transition probability.

[0040] In one embodiment, ResNet18 consists of 16 convolutional layers and 2 fully connected layers.

[0041] In one embodiment, ResNet18 employs a layered stacked residual block design, with each residual block containing two sets of 3×3 convolutional layers, batch normalization, and ReLU activation functions.

[0042] In this embodiment, a residual connection mechanism is introduced to achieve cross-layer identity mapping of the original feature information. The layered stacked residual block design uses skip connections to add the input features to the convolutional layer output element-wise, thereby effectively mitigating gradient vanishing and gradient exploding in deep networks. Furthermore, this design improves the stability of model training and effectively enhances the accuracy of image recognition.

[0043] In one embodiment, step 203, using the feature map of the incorrectly identified image output from the ResNet18 terminal convolutional layer as the input sample and the true category of the incorrectly identified image as the target category, generates a heatmap corresponding to the feature map of the incorrectly identified image through Grad-CAM, including: Using the feature maps of incorrectly identified images output from the final convolutional layer of ResNet18 as input samples, and the true class of the incorrectly identified images as the target class, Grad-CAM is used to obtain the weights of each channel feature of the input sample and the corresponding target class: ; in, For target category c No. k Weights of each channel feature; For target category c Score; Z represents the spatial dimension of the feature map; For the first k Feature maps of each channel feature; i This represents the coordinate position of the feature map in the vertical direction; j This represents the coordinate position of the feature map in the horizontal direction; For the first k The feature map of each channel feature is in ( i , j The eigenvalue at ().

[0044] The obtained weights are weighted and summed with the original heatmap of the input sample to generate a heatmap corresponding to the feature map of the misidentified image: ; in, For target category c The corresponding heatmap.

[0045] In this embodiment, Grad-CAM calculates the gradient of the feature map of the terminal convolutional layer of ResNet18 through the backpropagation mechanism of the target category prediction score, and aggregates the gradient using global average pooling to obtain the weight of each channel feature of the input sample and the corresponding target category.

[0046] In one embodiment, step 204, obtaining a set of two-dimensional coordinates corresponding to important regions of image recognition on images with erroneous recognition based on a heatmap and a preset positive contribution threshold, includes: The values ​​on the heatmap are normalized, and the set of two-dimensional coordinates corresponding to the important regions of image recognition in the image with incorrect recognition is found based on the preset positive contribution threshold.

[0047] In one embodiment, step 208 involves training and adjusting the parameters of the thawed features of the replaced feature extractor in the target domain using gradient descent to obtain a second agent model with fine-tuned parameters, thus completing transfer reinforcement learning. This includes: In the target domain, the weights and bias parameters of the unfrozen features of the replaced feature extractor are adjusted through gradient descent training. The feature extractor with adjusted parameters is used as the feature extractor of the second agent model to obtain the second agent model with fine-tuned parameters, thus completing the transfer reinforcement learning.

[0048] In this embodiment, only the parameters of the features thawed after replacement are trained and adjusted, so as to ensure the learning efficiency and performance of the agent model in target domain image recognition as much as possible in a short time after the feature extractor is transferred.

[0049] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0050] To verify the effectiveness of the method described in this invention, the specific effects of the method described in this invention and the image recognition transfer reinforcement learning method based on parameter fine-tuning were compared. The image recognition transfer reinforcement learning method based on parameter fine-tuning includes the following steps: A first agent model is constructed, using ResNet18 as the feature extractor of an online Q-network, with the fully connected layers of ResNet18 serving as fully connected layers that output the Q-values ​​of actions. A training environment is created, and the first agent model is trained using deep learning in the source domain to obtain the feature extractor of the trained first agent model. A second agent model identical to the first agent model is created, and the feature extractor of the second agent model is replaced with the feature extractor of the trained first agent model. The parameters of all features of the replaced feature extractor are then unfrozen. The parameters of all features of the replaced feature extractor are adjusted through gradient descent training in the target domain to obtain the second agent model with fine-tuned parameters, thus completing the transfer reinforcement learning.

[0051] It is understood that the difference between the method described in this invention and the image recognition transfer reinforcement learning method based on parameter fine-tuning is that the image recognition transfer reinforcement learning method based on parameter fine-tuning unfreezes the parameters of all features in the replaced feature extractor, while the method described in this invention only unfreezes the parameters of features in the replaced feature extractor that are the same as the important feature index array.

[0052] The experiment only changed the total training amount, but did not change the ratio of the source domain to the target domain training amount, that is, it ensured that the ratio of the source domain to the target domain was 5:2 for each training amount. The design is shown in Table 1.

[0053] Table 1 Experimental Training Quantity Design Table

[0054] Based on Grad-CAM and combining the results of multiple experiments, the important feature index array in the 512-channel dimensional features of layer 4 of the feature extractor was selected, such as... Figure 2 As shown in the figure, the blue squares represent the selected important feature index array. Precision, recall, and F1 score were used for comparative evaluation in the experiment, and all experimental results are the average of five identical experiments.

[0055] The accuracy of the image recognition transfer reinforcement learning method based on parameter fine-tuning under different test conditions (normal training volume, 1 / 4 training volume, and 1 / 8 training volume) is as follows: Figure 3 As shown in the figure. The accuracy of the Grad-CAM-based image recognition transfer reinforcement learning method under different test conditions with normal training volume, one-quarter training volume, and one-eighth training volume is as follows. Figure 4 As shown. From Figure 3 As can be seen, under normal training conditions, the accuracy in the target domain after transfer is 71.14%, even surpassing the 69.99% accuracy in the source domain. However, when the training amount is insufficient, its accuracy in the target domain drops significantly. Even with one-eighth of the training amount, the accuracy after transfer is only 20.61%, less than 1% higher than the 19.88% accuracy without transfer, indicating a significant risk of negative transfer. Figure 4As can be seen, although the accuracy in the target domain after transfer is 68.91% and the accuracy in the source domain is 70.3% under normal training conditions, the accuracy performance in the target domain after transfer does not exceed the performance in the source domain as the method based on parameter fine-tuning. However, the method of the present invention can still maintain a good transfer effect when the training amount is insufficient. According to the accuracy performance of the three test scenarios, it is not difficult to see that the accuracy of the Grad-CAM-based method in the target domain after transfer is 68.91%, 42.35%, and 26.4% respectively, with the training amount decreasing. The accuracy of the target domain test without transfer is 54.8%, 28.72%, and 17.24% respectively. It can be seen that even when the training amount is reduced to one-eighth, the accuracy performance of transfer is still nearly 10% higher than that without transfer. This shows that the method of the present invention can still have a good accuracy performance in the target domain even when the training amount is insufficient.

[0056] The accuracy of the image recognition transfer reinforcement learning method based on parameter fine-tuning in various categories under different conditions, such as normal training volume, 1 / 4 training volume, and 1 / 8 training volume, is as follows: Figure 5 As shown in the figure. The image recognition transfer reinforcement learning method based on Grad-CAM achieves the accuracy of each class under different conditions: normal training amount, one-quarter training amount, and one-eighth training amount. Figure 6 As shown. By Figure 5 It's easy to see that under normal training levels, there was no negative transfer across aircraft categories. However, when the training volume decreased to one-quarter and one-eighth, negative transfer occurred in the accuracy performance of the target domain after the transfer. It's evident that the curves for the source domain accuracy and the target domain accuracy after transfer were highly similar under normal training levels. However, as the training volume decreased, the target domain accuracy curve showed a significantly larger decrease than the corresponding curve for the source domain. Specifically, in the C17 category, under normal training levels, the target domain accuracy was about 1% higher than the source domain. With one-quarter of the training volume, this became about 5% higher, and with one-eighth of the training volume, the difference exceeded 30%. This indicates that insufficient training volume resulted in poor transfer performance in accuracy. Conversely, [the text abruptly ends here]. Figure 6As shown, the method of this invention exhibits negative transfer in class F4 under normal training conditions. However, it maintains good transfer performance even with reduced training volume. Whether at one-quarter or one-eighth of the training volume, the accuracy of the target domain test is higher than the source domain test accuracy for some aircraft classes after transfer. Specifically, the accuracy difference on C2 is around 20% at one-quarter of the training volume, and the difference on A10 is also around 20% at one-eighth of the training volume, indicating that the transfer performance remains good even with reduced training volume. Furthermore, it is noteworthy that at one-eighth of the training volume, the transfer and non-transfer curves of the parameter fine-tuning method overlap significantly, while the transfer and non-transfer curves of the method of this invention show a clear boundary. This indicates that the method of this invention performs better in terms of accuracy performance even with insufficient training volume.

[0057] The recall rates of image recognition transfer reinforcement learning methods based on parameter fine-tuning under different conditions (normal training volume, 1 / 4 training volume, and 1 / 8 training volume) for each class are shown below. Figure 7 As shown in the figure. The recall rates of each class of the Grad-CAM-based image recognition transfer reinforcement learning method under different test conditions, including normal training amount, one-quarter training amount, and one-eighth training amount, are as follows. Figure 8 As shown. According to Figure 7 and Figure 8 The recall performance under normal training levels can be considered similar between the two methods. The curves for the target domain test recall after transfer largely overlap with the curves for the source domain, and neither method exhibits negative transfer. However, at a quarter training level, the parameter fine-tuning-based method shows negative transfer within 10% on B2, B52, F15, F16, and F35, while the method of this invention shows negative transfer nearly 20% on EF2000. Nevertheless, the curves for the target domain and source domain test recall after transfer for both methods largely overlap and intersect, and both methods show clear boundaries from the non-transfer curve. However, when the training volume drops to one-eighth, the target domain recall curve after transfer based on parameter fine-tuning not only shows a clear boundary with the source domain recall curve, but also overlaps and intersects with the non-transferred target domain recall curve in many places, indicating that its transfer effect has dropped significantly. On the other hand, the target domain recall curve after transfer of the method of this invention can still largely maintain multiple overlaps and intersections with the source domain recall curve, and has a relatively clear boundary with the non-transferred target domain curve as a whole.

[0058] The image recognition transfer reinforcement learning method based on parameter fine-tuning achieves F1 scores for each class under different conditions: normal training volume, one-quarter training volume, and one-eighth training volume. Figure 9As shown, the F1 scores for each class of the Grad-CAM-based image recognition transfer reinforcement learning method are tested under different conditions: normal training amount, one-quarter training amount, and one-eighth training amount. Figure 10 As shown above. Figure 9 and Figure 10 The F1 score curves shown are highly similar to the recall curves at normal and quarter-training levels. However, it is noteworthy that at one-eighth of the training level, the F1 score curves of the transfer and non-transfer methods based on parameter fine-tuning do not show a significant intersection. On the contrary, the F1 scores of the transfer method are more than 10% higher than those of the non-transfer method on B52, C130, EF2000, and Rafale, indicating a slight boundary. Compared to the previous precision and recall curves, the F1 score curves of the present invention show a more obvious boundary with the non-transfer method, with the difference exceeding 20% ​​in categories A10, B1, and B2. Considering that the F1 score is largely affected by precision and recall, this indicates that both methods perform well in handling the balance between false positives and false negatives at various training levels.

[0059] Based on the above analysis of the basic index data of the two methods, it is easy to conclude that both methods can maintain a good transfer effect under normal training volume. However, when the training volume is insufficient, although both methods show a decline in index performance, the decline of the method based on parameter fine-tuning is significantly greater than that of the method of the present invention.

[0060] Meanwhile, experiments were conducted to compare and analyze the transfer effects of the two methods under different training amounts, and to evaluate the superiority and inferiority of the different methods and their transfer effects under different training amounts.

[0061] The relative accuracy of the method of this invention compared with the parameter fine-tuning method under normal training volume, one-quarter training volume, and one-eighth training volume is as follows: Figure 11 As shown. By Figure 11 As shown, under normal training conditions, the relative accuracy of the parameter fine-tuning method is higher than that of the method of this invention, with a difference of about 5%. However, when the training volume is reduced to one-quarter and one-eighth, the relative accuracy of the method of this invention is higher than that of the parameter fine-tuning method. Furthermore, the difference is about 5% at one-quarter training volume, but exceeds 10% at one-eighth training volume. It is clear that the relative accuracy performance of the parameter fine-tuning method decreases more significantly than that of the method of this invention when the training volume decreases. It is easy to infer that the relative accuracy gap between the two methods will further increase when the training volume is reduced even further. Therefore, it can be concluded that the parameter fine-tuning method has better relative accuracy performance when training is sufficient, while the method of this invention has better relative accuracy performance when training is insufficient.

[0062] The relative accuracy of the method of this invention and the parameter-based fine-tuning method for each category under normal training volume, one-quarter training volume, and one-eighth training volume is compared as follows: Figure 12 As shown. According to Figure 12 As shown, by analyzing the vertical axis positions of the curves for each aircraft category on the graph, it is easy to see that, with normal training volume, the method of this invention only has a higher relative accuracy rate than the parameter-based fine-tuning method for six aircraft categories: B2, F14, F22, J10, JAS39, and V22. The largest difference is in the F22 category, where the relative accuracy difference does not reach 20%. However, for A10, the relative accuracy difference between the parameter-based fine-tuning method and the method of this invention is as high as 40%. But when the training volume is reduced to one-quarter, the relative accuracy rate of the method of this invention is higher than that of the method of this invention for more than half of the aircraft categories. When the training volume is further reduced to one-eighth, the relative accuracy rate of only four categories—C130, C17, F14, and US2—does not exceed that of the parameter-based fine-tuning method, but the difference between these categories does not exceed 15%. In contrast, the method of this invention, with the same one-eighth training volume, has a relative accuracy rate of more than 30% higher than that of the parameter-based fine-tuning method for the F35 aircraft category. It is easy to see that as the amount of training decreases, the relative accuracy performance of the method of the present invention gradually outperforms the method based on parameter fine-tuning, and the gap between the two gradually increases.

[0063] The relative recall rates of the method of this invention and the parameter fine-tuning-based method for each class are compared under normal training volume, one-quarter training volume, and one-eighth training volume, for example. Figure 13 As shown. By Figure 13 As shown, under normal training conditions, the parameter fine-tuning-based method still outperforms the method of this invention overall. However, at one-quarter training volume, the parameter fine-tuning-based method only outperforms the method of this invention in relative recall for seven aircraft categories: A10, B1, C2, EF2000, F4, Rafale, and V22. The difference between A10 and B1 categories is less than 5%. At one-eighth training volume, although the relative recall of the method described in this invention is lower than that of the parameter fine-tuning-based image recognition transfer reinforcement learning method for US2 category, with a difference of up to 40%, the relative recall of the method described in this invention is not only higher than that of the parameter fine-tuning-based method for A10, B2, C2, and F22 categories, but the difference is also over 40%. Furthermore, the relative recall of the method described in this invention is better than that of the parameter fine-tuning-based image recognition transfer reinforcement learning method for more than two-thirds of the aircraft categories. Therefore, it can be shown that as the training volume decreases, the relative recall of the method described in this invention outperforms the relative recall of the parameter fine-tuning-based image recognition transfer reinforcement learning method overall.

[0064] The method of this invention and the parameter-based fine-tuning method are compared in terms of relative F1 scores for each class under normal training volume, one-quarter training volume, and one-eighth training volume. Figure 14 As shown. According to Figure 14 As shown, under normal training conditions, only the Grad-CAM method outperforms the Grad-CAM method in relative F1 scores for five categories: B2, F14, F22, J10, and JAS39. Even the largest difference in relative F1 score is only around 15% for F22, while the differences for B2 and JAS39 don't even exceed 5%. In contrast, the parameter-fine-tuning method outperforms the method of this invention by more than 10% in categories A10, C130, F16, F35, J20, and Rafale. Overall, the parameter-fine-tuning method is still better. However, when the training amount decreases to one-quarter and one-eighth, the relative positions of the curves clearly show that only a few aircraft categories show better relative F1 scores for the parameter-fine-tuning method. Therefore, it is easy to see that as the training amount decreases, the performance advantage of the proposed method over the parameter-fine-tuning-based image recognition transfer reinforcement learning method in relative F1 scores gradually increases.

[0065] Based on the analysis of the above evaluation metrics, it is clear that the method based on parameter fine-tuning is superior under normal training conditions for all evaluation metrics, while the method of this invention is superior under one-quarter and one-eighth training conditions. Therefore, it can be concluded that the method of this invention achieves better transfer performance when the training volume is insufficient.

[0066] In one embodiment, such as Figure 15 A Grad-CAM-based image recognition transfer reinforcement learning device is provided, comprising: The agent construction module 901 is used to construct the first agent model. The first agent model uses ResNet18 as the feature extractor of the online Q network, and the fully connected layer of ResNet18 is the fully connected layer that outputs the Q value of the action. The source domain training module 902 is used to create a training environment to perform deep learning training on the first agent model in the source domain, thereby obtaining the feature extractor of the trained first agent model and the images that the first agent model misidentifies during the source domain training process. The heatmap generation module 903 is used to take the feature map of the incorrectly identified image output by the terminal convolutional layer of ResNet18 as the input sample and the true category of the incorrectly identified image as the target category, and generate the heatmap corresponding to the feature map of the incorrectly identified image through Grad-CAM. The two-dimensional coordinate acquisition module 904 is used to acquire the set of two-dimensional coordinates corresponding to the important regions of image recognition on the image with recognition errors based on the heat map and the preset positive contribution threshold. The important feature set acquisition module 905 is used to map two-dimensional coordinates onto the feature map of layer 4 of the feature extractor, and obtain the activation values ​​of all channel features at a specified position on the feature map based on the mapping result. It further obtains the channel feature with the strongest response at each two-dimensional coordinate after mapping and records it as the important feature set corresponding to the image that was misidentified. The important feature index array acquisition module 906 is used to acquire the important feature set corresponding to all images that are incorrectly identified, and to take the union of all important feature sets to obtain the important feature index array. The migration module 907 is used to create a second agent model that is consistent with the first agent model, replace the feature extractor of the second agent model with the feature extractor of the trained first agent model, and unfreeze the parameters of the features in the replacement feature extractor that are the same as the important feature index array. The parameter fine-tuning module 908 is used to adjust the parameters of the features thawed from the feature extractor after replacement in the target domain through gradient descent training, so as to obtain the second agent model after parameter fine-tuning and complete the transfer reinforcement learning.

[0067] Specific limitations regarding the Grad-CAM-based image recognition transfer reinforcement learning device can be found in the limitations of the Grad-CAM-based image recognition transfer reinforcement learning method described above, and will not be repeated here. Each module in the aforementioned Grad-CAM-based image recognition transfer reinforcement learning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0068] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A Grad-CAM-based image recognition transfer reinforcement learning method, characterized in that, Includes the following steps: Construct a first intelligent agent model, in which ResNet18 is used as the feature extractor of an online Q-network, and the fully connected layer of ResNet18 is the fully connected layer that outputs the Q value of the action. Create a training environment and perform deep learning training on the first agent model in the source domain to obtain the feature extractor of the trained first agent model and the images that the first agent model misidentified during the training process in the source domain. Using the feature maps of incorrectly identified images output from the terminal convolutional layer of ResNet18 as input samples and the true category of the incorrectly identified images as the target category, Grad-CAM is used to generate heatmaps corresponding to the feature maps of the incorrectly identified images. Based on the heatmap and a preset positive contribution threshold, obtain the set of two-dimensional coordinates corresponding to the important regions of image recognition on the image with incorrect recognition; Two-dimensional coordinates are mapped onto the feature map of layer 4 of the feature extractor, and the activation values ​​of all channel features at a specified position on the feature map are obtained based on the mapping result. The channel feature with the strongest response at each two-dimensional coordinate after mapping is further obtained and recorded as the important feature set corresponding to the image that was misidentified. Obtain the set of important features corresponding to all images that were misidentified, and take the union of all important feature sets to obtain the important feature index array; Create a second agent model that is identical to the first agent model, replace the feature extractor of the second agent model with the feature extractor of the trained first agent model, and unfreeze the parameters of the features in the replacement feature extractor that are the same as the important feature index array. In the target domain, the parameters of the features thawed after replacement are adjusted by training with gradient descent to obtain the second agent model with fine-tuned parameters, thus completing the transfer reinforcement learning.

2. The Grad-CAM-based image recognition transfer reinforcement learning method according to claim 1, characterized in that, The source domain is the visible light image recognition of a real aircraft under normal field of view, and the target domain is the grayscale image recognition of a real aircraft under infrared detection.

3. The Grad-CAM-based image recognition transfer reinforcement learning method according to claim 2, characterized in that, Using the feature maps of incorrectly identified images output from the final convolutional layer of ResNet18 as input samples, and the true class of the incorrectly identified images as the target class, Grad-CAM generates heatmaps corresponding to the feature maps of the incorrectly identified images, including: Using the feature maps of incorrectly identified images output from the final convolutional layer of ResNet18 as input samples, and the true class of the incorrectly identified images as the target class, Grad-CAM is used to obtain the weights of each channel feature of the input sample and the corresponding target class. in, For target category c No. k Weights of each channel feature; For target category c Score; Z represents the spatial dimension of the feature map; For the first k Feature maps of each channel feature; i This represents the coordinate position of the feature map in the vertical direction; j This represents the coordinate position of the feature map in the horizontal direction; For the first k The feature map of each channel feature is in ( i , j The eigenvalue at () location; The obtained weights are weighted and summed with the original heatmap of the input sample to generate a heatmap corresponding to the feature map of the misidentified image: in, For target category c The corresponding heatmap.

4. The Grad-CAM-based image recognition transfer reinforcement learning method according to claim 3, characterized in that, Based on the heatmap and a preset positive contribution threshold, a set of two-dimensional coordinates corresponding to important regions for image recognition in images with incorrect recognition is obtained, including: The values ​​on the heatmap are normalized, and the set of two-dimensional coordinates corresponding to the important regions of image recognition in the image with incorrect recognition is found based on the preset positive contribution threshold.

5. The Grad-CAM-based image recognition transfer reinforcement learning method according to claim 1, characterized in that, A training environment is created to perform deep learning training on the first agent model in the source domain, resulting in a feature extractor for the trained first agent model and images that the first agent model misidentified during the source domain training process, including: The training environment is created and the values ​​of the network parameters are determined. The image input to the neural network is defined as the state, the image category as the action, and the total number of categories as the action space size. The experience tuple in the experience replay buffer is defined as including the current state, the next state, the action, the reward, and the termination flag. The reward function is defined as +1 for correctly identifying a single image and -1 for an error. Only one image is identified in each training session. The network parameters include experience replay capacity, discount factor, exploration rate, exploration rate decay rate, learning rate, number of training iterations, and experience batch size. The exploration rate includes the initial exploration rate and the final exploration rate. The first agent model is trained in the current round using the image in the source domain as input, and the training environment is reset. The action is selected based on the exploration rate. If the random value is lower than the exploration rate, the action is selected randomly. If the random value is higher than the exploration rate, the action with the largest Q value is selected. Execute the selected action, obtain and package the next state, action, reward and termination flag into an experience tuple and store it in the experience replay buffer; When the number of experience tuples in the experience replay buffer reaches the experience batch size, sample from the experience replay buffer and calculate the current Q value, the next Q value, and the target Q value. The loss is calculated and backpropagation and gradient pruning are performed. The loss function is calculated according to the following formula: in, For network parameters; Network parameters for the target Q value; For state Take action below The instant reward obtained afterward; This serves as a buffer for experience replay. To sample uniformly from within the experience playback buffer; This is the expected value; The loss function; For network parameters The calculated state Select action The optimal value; Network parameters determined by the target Q value The calculated state Select action The optimal value; Discount factor; The network parameters are updated based on the loss, and the current exploration rate is decayed based on a preset exploration rate decay coefficient after the network parameters are updated. End the current training round, determine whether the preset number of training iterations has been reached, and if so, output the trained first agent model, the feature extractor of the trained first agent model, and the images that the first agent model misidentified during the current training round in the source domain. If not, proceed to the next training round.

6. The Grad-CAM-based image recognition transfer reinforcement learning method according to claim 5, characterized in that, The Q value is calculated according to the following formula: in, Q value, indicating the state Select action The optimal value; State; For action; The next state; The action to be chosen for the next state; Discount factor; For state Select action The optimal value; For the reward function; Let be the state transition probability.

7. The Grad-CAM-based image recognition transfer reinforcement learning method according to claim 1, characterized in that, The ResNet18 consists of 16 convolutional layers and 2 fully connected layers.

8. The Grad-CAM-based image recognition transfer reinforcement learning method according to claim 7, characterized in that, The ResNet18 employs a layered stacked residual block design, with each residual block containing two sets of 3×3 convolutional layers, batch normalization, and ReLU activation functions.

9. The Grad-CAM-based image recognition transfer reinforcement learning method according to claim 6, characterized in that, In the target domain, the parameters of the thawed features of the replaced feature extractor are adjusted through gradient descent training to obtain the second agent model with fine-tuned parameters, thus completing transfer reinforcement learning, including: In the target domain, the weights and bias parameters of the unfrozen features of the replaced feature extractor are adjusted through gradient descent training. The feature extractor with adjusted parameters is used as the feature extractor of the second agent model to obtain the second agent model with fine-tuned parameters, thus completing the transfer reinforcement learning.

10. A Grad-CAM-based image recognition transfer reinforcement learning device, characterized in that, include: The intelligent agent construction module is used to construct the first intelligent agent model. The first intelligent agent model uses ResNet18 as the feature extractor of the online Q network, and the fully connected layer of ResNet18 is the fully connected layer that outputs the Q value of the action. The source domain training module is used to create a training environment to perform deep learning training on the first agent model in the source domain, and obtain the feature extractor of the trained first agent model and the images that the first agent model misidentified during the source domain training process. The heatmap generation module is used to generate heatmaps corresponding to the feature maps of the incorrectly identified images as input samples and the true category of the incorrectly identified images as the target category through Grad-CAM. The two-dimensional coordinate acquisition module is used to obtain the set of two-dimensional coordinates corresponding to important regions of image recognition on images with recognition errors based on heatmaps and preset positive contribution thresholds. The important feature set acquisition module is used to map two-dimensional coordinates onto the feature map of layer 4 of the feature extractor, and obtain the activation values ​​of all channel features at a specified position on the feature map based on the mapping result. It further obtains the channel feature with the strongest response at each two-dimensional coordinate after mapping and records it as the important feature set corresponding to the image that was misidentified. The important feature index array acquisition module is used to obtain the set of important features corresponding to all images that are misidentified, and to obtain the important feature index array by taking the union of all important feature sets. The migration module is used to create a second agent model that is consistent with the first agent model. It replaces the feature extractor of the second agent model with the feature extractor of the trained first agent model and unfreezes the parameters of the features in the replacement feature extractor that are the same as the important feature index array. The parameter fine-tuning module is used to adjust the parameters of the features thawed from the feature extractor after replacement in the target domain through gradient descent training, so as to obtain the second agent model with fine-tuned parameters and complete the transfer reinforcement learning.

Images