Model training method, image recognition method, equipment and storage medium

Abstract

The invention provides a model training method, an image recognition method, equipment and a storage medium, and the method comprises the steps: inputting a first sample image into a first image recognition model, and outputting a first prediction classification result and a first prediction mask; inputting the target feature image into a degradation prediction network, and outputting a degradation parameter prediction result; calculating a first loss function value based on the first prediction classification result and the first prediction mask; and calculating a second loss function value based on the degradation parameter and the degradation parameter prediction result; and adjusting model parameters of the first image recognition model based on the two loss functions, and training the first image recognition model. According to the embodiment of the invention, the method optimizes the network parameters through predicting the classification result, predicting the loss function of the mask and predicting the degradation parameter loss function, so as to pay attention to the image features of the classification and the mask at the same time in the model training process, and can extract the image features which can be restored to the image features before degradation under different degradation degrees. Therefore, the recognition accuracy of the model is improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a model training method, an image recognition method, a device and a storage medium. Background Art

[0002] Object detection achieves accurate identification and location of objects in images by combining deep learning, machine learning, and computer vision technologies. It is one of the indispensable core technologies in many modern intelligent systems.

[0003] However, in some cases where there are relatively few sample images, the target detection task cannot extract feature images during the training phase or the extracted feature images are relatively scarce, resulting in inaccurate recognition of targets. Or, in the case where the quality of the input image is relatively low during the target detection process, it may lead to problems such as failure to recognize targets in the image or inaccurate recognition of targets. Summary of the invention

[0004] The present application proposes a model training method, an image recognition method, a device and a storage medium, which can solve the technical problem that the target finally recognized in the target detection task is not accurate enough.

[0005] The first embodiment of the present application proposes a model training method, including:

[0006] Inputting a first sample image into a first image recognition model, and outputting a first prediction classification result and a first prediction mask, wherein the first sample image is generated by adding a degradation parameter to the second sample image;

[0007] Inputting a first target feature image output by the first image recognition model and a second target feature image output by the second image recognition model into a degradation prediction network, and outputting a degradation parameter prediction result;

[0008] Calculating a first loss function value of the first image recognition model based on the first predicted classification result and the first predicted mask;

[0009] Calculating a second loss function value based on the degradation parameter and the degradation parameter prediction result;

[0010] The model parameters of the first image recognition model are adjusted based on the first loss function value and the second loss function value, and the training is continued until a preset training completion condition is met to obtain a trained first image recognition model.

[0011] The embodiment of the second aspect of the present application provides a model training device, including:

[0012] An input module, used for inputting a first sample image into a first image recognition model, and outputting a first prediction classification result and a first prediction mask, wherein the first sample image is generated by adding a degradation parameter to the second sample image;

[0013] The input module is further used to input the first target feature image output by the first image recognition model and the second target feature image output by the second image recognition model into the degradation prediction network, and output the degradation parameter prediction result;

[0014] a calculation module, configured to calculate a first loss function value of the first image recognition model based on the first predicted classification result and the first predicted mask;

[0015] The calculation module is further used to calculate a second loss function value based on the degradation parameter and the degradation parameter prediction result;

[0016] An adjustment module is used to adjust the model parameters of the first image recognition model based on the first loss function value and the second loss function value, and continue training until a preset training completion condition is met to obtain a trained first image recognition model.

[0017] The third aspect of the present application provides an image recognition method, including:

[0018] Inputting the image to be recognized into a first image recognition model, where the first image recognition model is trained using the model training method described in the first aspect;

[0019] Determine, by using the first image recognition model, an equivariant representation feature image of the feature image corresponding to the image to be recognized before degradation;

[0020] Using the first image recognition model to recognize a target object in the image to be recognized in the equivariant representation feature image;

[0021] Output the classification result of the target object in the image to be identified.

[0022] The fourth aspect of the present application provides an object detection device, including:

[0023] An input module, used to input the image to be recognized into a first image recognition model, wherein the first image recognition model is trained using the model training method described in the first aspect;

[0024] A determination module, configured to determine, by using the first image recognition model, an equivariant representation feature image of the feature image corresponding to the image to be recognized before degradation;

[0025] A recognition module, configured to recognize a target object in the image to be recognized in the equivariant representation feature image using the first image recognition model;

[0026] The output module is used to output the classification result of the target object in the image to be identified.

[0027] An embodiment of the fifth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect or the third aspect above.

[0028] The embodiment of the sixth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect or the third aspect above.

[0029] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0030] The present application proposes a model training method, image recognition method, device and storage medium, including: inputting a first sample image into a first image recognition model, outputting a first predicted classification result and a first predicted mask; inputting a target feature image into a degradation prediction network, outputting a degradation parameter prediction result; calculating a first loss function value based on the first predicted classification result and the first predicted mask; and calculating a second loss function value based on the degradation parameter and the degradation parameter prediction result; adjusting the model parameters of the first image recognition model based on the above two loss functions, and training the first image recognition model. The embodiment of the present application optimizes network parameters by predicting the loss function of the classification result and the predicted mask and the degradation parameter loss function, so as to pay attention to the image features of the classification and the mask at the same time during the model training process, and to be able to extract image features that can be restored to the pre-degradation state under different degradation degrees, thereby improving the recognition accuracy of the model.

[0031] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0033] Figure 1 A flow chart of a model training method provided by an embodiment of the present application is shown;

[0034] Figure 2 A schematic diagram of the structure of a preset downsampling layer provided in an embodiment of the present application is shown;

[0035] Figure 3 A schematic diagram of the structure of an attention mechanism module provided by an embodiment of the present application is shown;

[0036] Figure 4 A schematic diagram of the structure of a network model provided by an embodiment of the present application is shown;

[0037] Figure 5 A schematic diagram of the structure of a CRB module provided in an embodiment of the present application is shown;

[0038] Figure 6 A schematic diagram of the structure of a target network model provided by an embodiment of the present application is shown;

[0039] Figure 7-Figure 9 An example diagram of image degradation provided by an embodiment of the present application is shown;

[0040] Fig.10 A schematic diagram of the structure of a degradation prediction model provided by an embodiment of the present application is shown;

[0041] Fig.11 A flowchart of an image recognition method provided by an embodiment of the present application is shown;

[0042] Fig.12 A schematic diagram of the structure of a model training device provided in one embodiment of the present application is shown;

[0043] Fig.13 A schematic diagram showing the structure of an image recognition device provided by an embodiment of the present application is shown.

[0044] Fig.14 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0045] Fig.15 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0047] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0048] The model training method of the present application can be executed by a computing device, which can be a server, such as one server, multiple servers, a server cluster, a cloud computing platform, etc. Optionally, the computing device can also be a terminal device, such as a mobile phone, a tablet computer, a game console, a portable computer, a desktop computer, an advertising machine, an all-in-one machine, etc. The present application does not limit the device type and number of computing devices.

[0049] The present application proposes a model training method, image recognition method, device and storage medium, including: inputting a first sample image into a first image recognition model, outputting a first predicted classification result and a first predicted mask; inputting a target feature image into a degradation prediction network, outputting a degradation parameter prediction result; calculating a first loss function value based on the first predicted classification result and the first predicted mask; and calculating a second loss function value based on the degradation parameter and the degradation parameter prediction result; adjusting the model parameters of the first image recognition model based on the above two loss functions, and training the first image recognition model. The embodiment of the present application optimizes network parameters by predicting the loss function of the classification result and the predicted mask and the degradation parameter loss function, so as to pay attention to the image features of the classification and the mask at the same time during the model training process, and to be able to extract image features that can be restored to the pre-degradation state under different degradation degrees, thereby improving the recognition accuracy of the model.

[0050] A model training method proposed according to an embodiment of the present application is described below in conjunction with the accompanying drawings.

[0051] See also Figure 1 , the method specifically comprises the following steps:

[0052] S101: Input a first sample image into a first image recognition model, and output a first prediction classification result and a first prediction mask.

[0053] The first sample image is generated by adding degradation parameters to the second sample image. In the embodiment of the present application, the degradation parameters may include motion blur parameters and noise information.

[0054] Among them, the first sample image and the second sample image can be images including the target object. The first sample image and the second sample image are sample images in a small sample field. The small sample field is a field where the sample image acquisition process is relatively complicated, such as the medical field, military field, aviation field, etc.

[0055] The first predicted classification result may be the probability distribution of the target object in each category or the specific direction of the target object, and the first predicted mask is the outline of the target object.

[0056] It should be noted that in small sample recognition tasks, it is difficult for image recognition models to learn highly discriminative features between targets. There is a high degree of similarity between targets of the same type and different models, such as the difference between aircraft of different sizes. Therefore, it is difficult to extract features for accurately distinguishing targets of different models, and it is difficult for the network to learn highly discriminative features from limited samples, resulting in low classification accuracy. Therefore, the first image recognition model can be used to output the predicted mask of the target object, so that the first image recognition model can learn to identify the contour features of the target object and improve the accuracy of the final output classification result.

[0057] S102, inputting the first target feature image output by the first image recognition model and the second target feature image output by the second image recognition model into a degradation prediction network, and outputting a degradation parameter prediction result.

[0058] The input of the second image recognition model is the second sample image.

[0059] The first target feature image is generated by the first sample image, and therefore includes degradation parameters, while the second target feature image is generated based on the second sample image, and therefore does not include degradation parameters. The degradation prediction network can determine the degradation parameter prediction result based on the difference information between the first target feature image and the second target feature image. The degradation parameter prediction result can enable the first image recognition model to learn to extract image features that can be restored to the image before degradation under different degradation degrees, thereby improving the accuracy of the classification result outputted at the end.

[0060] S103. Calculate a first loss function value of the first image recognition model based on the first predicted classification result and the first predicted mask.

[0061] The loss function value of the first image recognition model for the target object recognition function includes two parts, namely, the classification loss function value corresponding to the first predicted classification result and the mask loss function value corresponding to the first predicted mask. The two parts of the loss function value can be assigned corresponding weights to obtain the first loss function value of the first image recognition model.

[0062] In some embodiments, the process of calculating the first loss function value of the first image recognition model based on the first predicted classification result and the first predicted mask can be implemented as follows: obtaining the label classification result and the label mask of the target object; determining the classification loss function value of the first image recognition model based on the label classification result and the first predicted classification result; determining the mask loss function value of the first image recognition model based on the label mask and the first predicted mask; and assigning corresponding weights to the loss function values ​​of the two parts respectively to obtain the first loss function value of the first image recognition model.

[0063] S104. Calculate a second loss function value based on the degradation parameter and the degradation parameter prediction result.

[0064] The degradation parameter is the true value of the degradation parameter prediction result. Therefore, the second loss function value can be calculated based on the difference information between the degradation parameter and the degradation parameter prediction result.

[0065] S105. Adjust the model parameters of the first image recognition model based on the first loss function value and the second loss function value, and continue training until a preset training completion condition is met to obtain a trained first image recognition model.

[0066] By adjusting the first model parameters of the first image recognition model through the loss function of predicting the classification result and predicting the mask, that is, the first loss function value, the image features of the classification and the mask can be taken into consideration at the same time during the model training process so as to improve the recognition accuracy of the model. Further, by adjusting the second model parameters of the first image recognition model through the degradation parameter loss function, that is, the second loss function value, the image features that can be restored to the image features before degradation under different degrees of degradation can be extracted, thereby further improving the recognition accuracy of the model.

[0067] The first model parameter and the second model parameter may be the same or different, which will not be described in detail here.

[0068] The present application proposes a model training method, including: inputting a first sample image into a first image recognition model, outputting a first predicted classification result and a first predicted mask; inputting a target feature image into a degradation prediction network, outputting a degradation parameter prediction result; calculating a first loss function value based on the first predicted classification result and the first predicted mask; and calculating a second loss function value based on the degradation parameter and the degradation parameter prediction result; adjusting the model parameters of the first image recognition model based on the above two loss functions, and training the first image recognition model. The embodiment of the present application adjusts the first model parameters of the first image recognition model by predicting the loss function of the classification result and the predicted mask, i.e., the first loss function value, so that the image features of the classification and the mask can be paid attention to at the same time during the model training process, so as to improve the recognition accuracy of the model. Further, the degradation parameter loss function, i.e., the second loss function value adjusts the second model parameters of the first image recognition model, so as to extract image features that can be restored to the pre-degradation state under different degradation degrees, and further improve the recognition accuracy of the model.

[0069] In some embodiments, the above method also includes: inputting the second sample image into the second image recognition model, outputting a second predicted classification result and a second predicted mask; and calculating a third loss function value of the second image recognition model based on the second predicted classification result and the second predicted mask.

[0070] Among them, the second sample image is a sample image that does not include degradation parameters, that is, a high-quality sample image. In order to improve the ability of the first image recognition model to extract image features that can be restored to the image features before degradation under different degrees of degradation, it is also necessary to use the second prediction classification result output by the second image recognition model and the third loss function value corresponding to the second prediction mask to train the first image recognition model, wherein the calculation method of the third loss function value is the same as that of the first loss function value, which will not be repeated here.

[0071] In some embodiments, the model parameters of the first image recognition model are adjusted based on the first loss function value and the second loss function value, and training is continued until a preset training completion condition is met to obtain a trained first image recognition model, including: determining the weight coefficients corresponding to the first loss function value, the second loss function value, and the third loss function value; determining the total loss function value of the first image recognition model based on the first loss function value, the second loss function value, and the third loss function value and their corresponding weight coefficients; adjusting the model parameters of the first image recognition model based on the total loss function value, and continuing training until a preset training completion condition is met to obtain a trained first image recognition model.

[0072] It can be understood that the total loss function corresponding to the first image recognition model includes two parts, namely, the target detection loss function and the degradation parameter prediction loss, wherein the target detection loss function is obtained based on the first loss function value and the third loss function value, and the degradation parameter prediction loss is the second loss function value.

[0073] In some embodiments, the total loss function is as shown in formula (1):

[0074] L total =L obj +λ dpp ·L dpp (1)

[0075] Among them, L total is the total loss function, L obj is the target detection loss function, L dpp is the degradation parameter prediction loss function, λ dpp Predict the weights of the loss function for the degradation parameters.

[0076] The target detection loss function is shown in formula (2):

[0077] L obj =λ obj-H ·L obj-H +λ obj-L ·L obj-L (2)

[0078] L obj-H and L obj-Lare the first loss function and the third loss function, respectively, obj-H and λ obj-L are the weights of the first loss function and the third loss function respectively.

[0079] Among them, the first loss function and the third loss function adopt the same calculation method, including confidence loss L conf , classification loss L cls and the localization loss L box Three parts, among which the first loss function or the third loss function is shown in formula (3):

[0080] L=λ conf ·L conf +λ cls ·L cls +λ box ·L box (3)

[0081] Among them, L is the first loss function or the third loss function, λ conf is the weight of the confidence loss, λ cls is the weight of classification loss, λ box is the weight of the positioning loss.

[0082] In addition, the first loss function value or the third loss function value can also be calculated by formula (4):

[0083] L=L cls +λL mask (4)

[0084] Among them, L is the first loss function or the third loss function, L cls To predict the loss function corresponding to the classification result, cross entropy loss is used for calculation, L Mask To predict the loss function corresponding to the mask, the cross entropy loss is used for calculation, and λ is the weight of the loss function corresponding to the predicted mask.

[0085] Among them, the loss function corresponding to the predicted classification result is shown in formula (5):

[0086]

[0087] Among them, m is the number of samples, n is the number of categories, The i-th sample is the label of the k-th category, The probability of predicting the kth category for the i-th sample.

[0088] The loss function corresponding to the predicted mask is shown in formula (6):

[0089]

[0090] Where m is the number of samples, H and W are the height and width of the image, is the label of the pixel at position (u,v) in the i-th sample as background or target, is the probability that the pixel at position (u, v) in the i-th sample is the background or target.

[0091] In some embodiments, a first sample image is input into a first image recognition model, and a first predicted classification result and a first predicted mask are output, including: inputting the sample image into a backbone network of the image recognition model, and introducing an attention mechanism in a preset downsampling layer during a downsampling operation on the sample image using multiple downsampling layers of the backbone network to obtain first feature images of different scales; using the backbone network to input the first feature image with the smallest scale into a fusion network of a first target detection network, and outputting second feature images of different scales; using the fusion network to input second feature images of different scales into a detection network of the first image recognition model, and outputting a target area map of the target object; using the detection network to input the target area map into the classification output network and the mask output network of the first image recognition model, respectively, and outputting a first predicted classification result and a first predicted mask.

[0092] In some embodiments, the first image recognition model includes: a backbone network, a fusion network, a detection network, a feature extraction network, a classification output network, and a mask output network.

[0093] In some embodiments, the backbone network includes multiple downsampling layers to extract first image features of different scales, and the fusion network further performs upsampling operations on the first image features with the smallest scale to obtain second image features of different scales. The fusion network is used to input the second image features of different scales into the detection network of the first image recognition model to output a target area map of the target object.

[0094] The target area map includes the target object, and in order to reduce the amount of calculation, the target area Figure 1 It is generally the smallest area map that includes the target object.

[0095] Furthermore, the detection network inputs the target area map into the classification output network and the mask output network of the first image recognition model respectively, and the classification output network outputs the predicted classification result and the predicted mask.

[0096] Specifically, the process of extracting first image features of different scales by the backbone network can be implemented as follows: in the process of downsampling the sample image using multiple downsampling layers of the backbone network, an attention mechanism is introduced in the preset downsampling layer to obtain first image features of different scales.

[0097] Each downsampling layer is used to downsample the corresponding output according to the corresponding downsampling multiple to obtain first image features of different scales.

[0098] However, in small sample tasks, it is difficult for the backbone network to extract the features of the target area. Therefore, the EMA attention mechanism can be introduced into the backbone network to make the backbone network pay more attention to the features of the area where the target object is located in the image.

[0099] In order to simplify the structure of the backbone network and make the backbone network pay more attention to the features of the area where the target object is located in the image, it is not necessary to introduce the attention mechanism in each downsampling layer. The attention mechanism can be introduced in a certain downsampling layer, that is, the attention mechanism is introduced in the preset downsampling layer.

[0100] Among them, the downsampling layer generally includes two branches, namely the convolution branch and the Bottleneck branch. Both branches use 1×1 convolution for channel dimension reduction. The Bottleneck branch introduces a residual structure, which can avoid the problem of gradient disappearance while increasing the network depth. Finally, the output results of the two branches are spliced ​​in the channel dimension, the original features and the extracted deep features are fused, and the final output is obtained through a convolution.

[0101] In order to improve the network's ability to extract features of smaller target areas, the EMA attention mechanism can be introduced in the preset downsampling layer containing rich local information. In order to avoid increasing the amount of calculation, the EMA module is only introduced in the convolution branch. The structure of the preset downsampling layer is as follows: Figure 2 The EMA attention mechanism is used to increase the feature weight of the target object region in the original features, and the characteristics of the preset downsampling layer cross-stage connection are used to improve the accuracy of the final output features of the module.

[0102] In some embodiments, the attention mechanism module structure is as follows Figure 3As shown in the figure, the EMA attention mechanism module is improved based on the Coordinate Attention (CA) module. In order to preserve the information on each channel of the feature map and reduce the computational cost, the EMA module groups the input feature map channels evenly on the basis of the CA module to obtain G sub-features. In order to aggregate multi-scale spatial structural information, the module sets three branches to learn attention weights. Among them, the first two branches decompose the global information into one-dimensional features in the X and Y directions for encoding, and use the shared 1×1 convolution to obtain new features. The features in the two directions are respectively subjected to the Sigmoid activation function to obtain the final weights, and are fused with the grouped features of the original input. The third branch uses 3×3 convolution to further extract multi-scale features. Finally, the outputs of the two parallel branches of pooling and convolution are fused using the cross-space learning structure. By transferring information between features of different scales, the module can use the correlation between features of different scales to enhance the consistency and richness of feature representation. While automatically learning the weights of each channel, the three branches accurately embed spatial information into the channel, which is conducive to improving the detection accuracy of the network.

[0103] The detection network inputs the target area map into the classification output network and the mask output network of the first image recognition model respectively, and the classification output network outputs the predicted classification result and the predicted mask for calculating the classification loss and the mask loss corresponding to the first image recognition model.

[0104] During the training process, mask loss is introduced to supervise the first image recognition model, so that the feature extraction network can extract features that meet both the classification task and the mask extraction task, improve the feature extraction network's attention to the target area features, and learn the features related to the target and the contour, thereby improving the classification accuracy.

[0105] In some embodiments, the detection network may first output the target area map to the feature extraction network, and further input the feature image extracted by the feature extraction network to the classification output network and the mask output network respectively.

[0106] Among them, the feature extraction network is used to extract the image features of the target area map.

[0107] In some embodiments, the structures of the feature extraction network, the classification output network, and the mask output network are as follows: Figure 4 shown.

[0108] Among them, the feature extraction network includes multiple downsampling layers for extracting image features of different scales.

[0109] The classification output network includes a global average pooling (GAP) layer and a fully connected (FC) layer. The GAP layer is used to average all elements of each feature map to obtain a scalar output for each feature map. The result of this operation is to convert the entire feature map into a vector of fixed length, which represents the overall information of the feature map. The FC layer is used to integrate and classify the features extracted by the previous layers and finally output the prediction results.

[0110] The mask output network includes multiple upsampling layers, which are used to restore the input image features, extract feature images of multiple scales, and output the corresponding mask results.

[0111] like Figure 3 As shown, the feature extraction network includes 4 downsampling layers, and the mask output network includes 4 upsampling layers.

[0112] In some embodiments, the structure of the mask output network is as follows: Figure 4 As shown, the mask output network includes multiple upsampling layers, transposed convolution layers and convolution layers, wherein the upsampling layer includes a transposed convolution layer and a channel bidirectional convolution cycle (Channel-wise Bi-directional Convolutional Recurrent, CRB for short) module, wherein the upsampling multiples of all transposed convolution layers in the mask output network are the same as the downsampling multiples of the input feature image.

[0113] The input of the mask output network is a feature map of size (C×(H / r)×(W / r)), where C is the number of channels of the feature map of this layer, H and W are the height and width of the image when input to the network, and r is the downsampling multiple of the feature map relative to the input image. The output of the mask guidance branch is the predicted mask image. The predicted mask height and width are the same as the input image height and width, the number of channels is 2, and the fraction of pixels in the image that are background pixels and target pixels is output respectively.

[0114] The input feature map is upsampled using transposed convolution. The height and width of the upsampled feature map are twice as large as the original ones, and the number of channels is reduced to half. A total of r transposed convolutions are used in the mask output network, so that the output mask has the same width and height as the input image.

[0115] In some embodiments, the structure of the CRB module is as follows: Figure 5As shown in the figure, the CBR module consists of a 3×3 convolutional layer, a BN layer, and a ReLU activation function. The upsampled feature maps are fused to enhance the network's expressiveness. It should be noted that the output of this module is the same size as the input feature map, and no additional downsampling is introduced. The feature map obtained by the last upsampling is subjected to a 1×1 convolution to obtain the output mask.

[0116] In some embodiments, a backbone network is used to input a first feature image with the smallest scale into a fusion network, and a second feature image of a different scale is output, including: in a process of performing an upsampling operation on the first feature image with the smallest scale using a first fusion network of the fusion network, a third feature image obtained by each upsampling operation is fused with the first feature image of the same scale to obtain a fourth feature image of a different scale; in a process of performing a downsampling operation on the fourth feature image with the smallest scale using a second fusion network of the fusion network, a fifth feature image obtained by each downsampling operation is fused with the first feature image and the fourth feature image of the same scale to obtain second feature images of different scales.

[0117] The fusion network includes a first fusion network and a second fusion network, and the network structures of the first fusion network and the second fusion network can be a top-down feature pyramid structure (Feature Pyramid Networks, FPN for short) and a bottom-up (Path Aggregation Network, PAN for short) structure, respectively, wherein a jump connection is added between the backbone network and the second fusion network, wherein the jump connection can be added between at least one network layer in the backbone network and the second fusion network, the first fusion network includes multiple upsampling network layers, and the second fusion network includes multiple downsampling layers. When performing feature fusion, a learnable weight is set to fuse feature maps from different paths, so that the network can adaptively learn more important features from small samples and improve detection accuracy.

[0118] Specifically, in the process of upsampling the first image feature with the smallest scale by the first fusion network, the third image feature obtained by each upsampling operation is fused with the first image feature with the same scale to obtain a fourth image feature of a different scale. In the process of downsampling the fourth image feature with the smallest scale by the second fusion network, the fifth image feature obtained by each downsampling operation is fused with the first image feature and the fourth image feature with the same scale to obtain a second image feature of a different scale.

[0119] It should be noted that only when there is a jump connection between the backbone network and the second fusion network, the second fusion network can fuse the corresponding first image feature with the fourth image feature and the fifth image feature of the same scale.

[0120] In some embodiments, a first target feature image output by a first image recognition model and a second target feature image output by a second image recognition model are input into a degradation prediction network, and a degradation parameter prediction result is output, including: inputting the first target feature image output by a first preset downsampling layer in the first image recognition model and the second target feature image output by a second preset downsampling layer in the second image recognition model into the degradation prediction network, and using the degradation prediction network to obtain a first feature vector corresponding to the first target feature image and a second feature vector corresponding to the second target feature image; using the degradation prediction network to splice the first feature vector and the second feature vector to obtain difference information between the first feature vector and the second feature vector, the difference information including the size of the predicted motion blur parameter, the predicted direction of the applied blur parameter, and noise information; passing the spliced ​​feature vector through multiple fully connected layers in sequence to obtain the number of predicted degradation parameters; and obtaining the degradation parameter prediction result based on the difference information and the number of degradation parameters.

[0121] Among them, the model structures of the first image recognition model and the second image recognition model both include a backbone network, wherein the backbone network includes multiple downsampling layers. In order to obtain accurate degradation parameter prediction results, the first target feature image and the second target feature image input into the degradation prediction model are consistent in size. It can be understood that the larger the size of the feature image, the richer the information of image degradation is retained, which is conducive to degradation parameter prediction; on the other hand, if the feature image participates in the multi-scale feature fusion of the subsequent fusion network, it is beneficial to improve the overall performance of the network. Therefore, the feature image output by the preset downsampling layer is generally larger in size, and the retained image degradation information is richer, and the feature map output by the preset downsampling layer will participate in the multi-scale feature fusion of the subsequent fusion network.

[0122] In some embodiments, the model structure of the target network model is as follows: Figure 6 As shown, the target network model includes a first image recognition model, a second image recognition model and a degradation prediction model. The first image recognition model and the second image recognition model both include a backbone network, a fusion network and a detection network, and the backbone network includes a backbone network including 5 downsampling layers C1, C2, C3, C4 and C5, where C i The downsampling factor is 2 i Downsampling layer. Since the feature image output by the downsampling layer C3 is relatively large, the retained image degradation information is relatively rich, and the feature image output by C3 participates in the multi-scale feature fusion of the subsequent fusion network, the feature image output by the backbone network C3 is used to predict the degradation parameters.

[0123] In some embodiments, obtaining a degradation parameter prediction result output by a degradation prediction model includes: using the degradation prediction model to obtain a first feature vector corresponding to a first feature image and a second feature vector corresponding to a second feature image; using the degradation prediction model to splice the first feature vector and the second feature vector to obtain a spliced ​​feature vector, the spliced ​​feature vector including difference information between the first feature vector and the second feature vector, the difference information including the size of a predicted motion blur parameter, the direction of the predicted motion blur parameter, and noise information; passing the spliced ​​feature vector through multiple fully connected layers in sequence to obtain the number of predicted degradation parameters; and obtaining a degradation parameter prediction result based on the difference information and the number of predicted degradation parameters.

[0124] The degradation parameters added in the first sample image mainly include two degradation factors: motion blur parameter and noise information. The degradation process can be expressed by equation (7):

[0125] g(x,y)=h(x,y)*f(x,y)+η(x,y) (7)

[0126] Among them, f(x, y) is the clear image, that is, the first sample image; g(x, y) is the degraded image, that is, the second sample image; h(x, y) is the motion blur parameter; η(x, y) is the noise information.

[0127] Motion blur is caused by the relative motion between the camera and the background and target during exposure. In a short exposure time, the motion blur caused by the drone can be simplified to uniform linear motion blur, and its point spread function can be expressed by equation (8):

[0128]

[0129] Among them, d is the number of pixels in the image reflected by the relative motion between the drone and the target, which is also called the size of the blur kernel in the convolution operation; θ is the angle between the drone's motion direction and the x-axis direction when it is reflected in the image.

[0130] Noise information can be transmission channel interference, electronic system noise, and noise caused by digital image processing, which can all be regarded as additive noise. A zero-mean additive white Gaussian noise model can be used to simulate camera sensor noise, and the standard deviation σ is used to describe the noise information.

[0131] In some embodiments, Figure 7-Figure 9 is an example of image degradation, where Figure 7 For high-quality images and target objects identified based on high-quality images, Figure 8 For Figure 7 Based on the above, we introduce an image with linear motion blur of scale d=11 and θ=45° and the target object identified on the image. Fig. 9 For Figure 7 Based on the above, an image with a noise standard deviation of σ = 10 / 255 is introduced and the target object is identified on the image. It can be seen that under the influence of motion blur and noise, the grayscale, texture and other features of the image are seriously distorted, making it difficult to extract discriminative features for target detection and recognition.

[0132] Since the degradation parameters include motion blur parameters and noise information, the degradation prediction model can be used to predict the size d and direction θ of the motion blur. For noise, the degradation prediction model is used to predict the standard deviation σ of the noise.

[0133] In some embodiments, the structure of the degradation prediction model is as follows: Fig.10 As shown, the first sample image and the second sample image are respectively extracted through the weight-sharing backbone network to obtain the first target feature image F1 and the second target feature image F2, both of which are of size C×H×W. In order to retain the directional information in the feature map, the global maximum pooling operations in the X direction and the Y direction are used to extract the directional information in the feature maps F1 and F2, and then two feature vectors of size (C×(H+W))×1×1 are obtained by flattening and connecting them respectively, and then the two are connected to obtain a feature vector of size (2×C×(H+W))×1×1. In this way, the difference information in the two F1 and F2 is compared, and then three fully connected layers are sequentially passed to obtain a feature vector of size N p The output of ×1×1 is the predicted degradation parameter and the predicted degradation parameter, where N p is the number of degenerate parameters.

[0134] In some embodiments, the second loss function value is calculated based on the degradation parameter and the degradation parameter prediction result, as shown in formula (9):

[0135]

[0136] Among them, L dpp is the second loss function value, N p To predict the number of degradation parameters, p i is the true value corresponding to the i-th degradation parameter, is the predicted value of the i-th degradation parameter.

[0137] In some embodiments, before inputting the first sample image into the first image recognition model and outputting the first predicted classification result and the first predicted mask, the method further includes:

[0138] Get the training set corresponding to the target object;

[0139] The multiple original images in the training set are scaled according to preset ratios to obtain second sample images corresponding to the multiple original images.

[0140] In the embodiment of the present application, the data set is expanded by scaling the images in the training set, increasing the number of training images while alleviating the problem of single target scale. The images in the training set are scaled proportionally, pixels are added around them to keep them consistent with the original image size, and added to the training set to expand the number of samples in the training set.

[0141] An image recognition method proposed according to an embodiment of the present application is described below in conjunction with the accompanying drawings.

[0142] See also Fig.11 , the method specifically comprises the following steps:

[0143] S1101. Input an image to be recognized into a first image recognition model.

[0144] The first image recognition model is trained using the above-mentioned model training method.

[0145] S1102: Determine, using the first image recognition model, an equivariant representation feature image of a feature image corresponding to the image to be recognized before degradation.

[0146] S1103: using the first image recognition model to recognize a target object in the image to be recognized in the equivariant representation feature image.

[0147] S1104: Output the classification result of the target object in the image to be identified.

[0148] Among them, degradation equivariance means that the degradation of the entire image is equivalent to the degradation of the image's internal visual structure. Degradation equivariance representation uses the encoded features of the original image and the transformed image to predict the conversion relationship between the two, so that the network can learn this feature, thereby reducing the impact of image degradation on other visual tasks such as image classification. Typical algorithms based on this method include MAET and RestoreDet.

[0149] In some embodiments, after extracting the feature image of the image to be recognized, the first image recognition model determines an equivariant representation feature image of the feature image before degradation based on the learned equivariant representation, and further recognizes the target object in the image to be recognized from the equivariant representation feature image.

[0150] The present application also provides a model training device, which is used to execute the model training method provided in any of the above embodiments. Fig.12 As shown, the device includes: an input module 1201, a calculation module 1202 and an adjustment module 1203.

[0151] An input module 1201 is used to input a first sample image into a first image recognition model, and output a first prediction classification result and a first prediction mask, wherein the first sample image is generated by adding a degradation parameter to the second sample image;

[0152] The input module 1201 is further used to input the first target feature image output by the first image recognition model and the second target feature image output by the second image recognition model into the degradation prediction network, and output the degradation parameter prediction result;

[0153] A calculation module 1202, configured to calculate a first loss function value of the first image recognition model based on the first predicted classification result and the first predicted mask;

[0154] The calculation module 1202 is further used to calculate a second loss function value based on the degradation parameter and the degradation parameter prediction result;

[0155] The adjustment module 1203 is used to adjust the model parameters of the first image recognition model based on the first loss function value and the second loss function value, and continue training until a preset training completion condition is met to obtain a trained first image recognition model.

[0156] The embodiment of the present application provides a model training device, including: inputting a first sample image into a first image recognition model, outputting a first predicted classification result and a first predicted mask; inputting a target feature image into a degradation prediction network, outputting a degradation parameter prediction result; calculating a first loss function value based on the first predicted classification result and the first predicted mask; and calculating a second loss function value based on the degradation parameter and the degradation parameter prediction result; adjusting the model parameters of the first image recognition model based on the above two loss functions, and training the first image recognition model. The embodiment of the present application optimizes network parameters by predicting the loss function of the classification result and the predicted mask and the degradation parameter loss function, so that the image features of the classification and the mask can be taken into account at the same time during the model training process, and the image features that can be restored to the pre-degradation state under different degradation degrees can be extracted, thereby improving the recognition accuracy of the model.

[0157] In some embodiments, the input module 1201 is further used to:

[0158] The second sample image is input into a second image recognition model, and a second prediction classification result and a second prediction mask are output:

[0159] The calculation module 1202 is further used to calculate the third loss function of the second image recognition model based on the second predicted classification result and the second predicted mask.

[0160] In some embodiments, the adjustment module 1203 is specifically configured to:

[0161] Determine the weight coefficients corresponding to the first loss function value, the second loss function value, and the third loss function value respectively;

[0162] Determine a total loss function value of the first image recognition model based on the first loss function value, the second loss function value, and the third loss function value and their corresponding weight coefficients;

[0163] The model parameters of the first image recognition model are adjusted based on the total loss function value, and the training is continued until a preset training completion condition is met to obtain a trained first image recognition model.

[0164] In some embodiments, the input module 1201 is specifically used to:

[0165] Inputting the first sample image into the backbone network of the image recognition model, and introducing an attention mechanism in a preset downsampling layer during a downsampling operation on the first sample image using multiple downsampling layers of the backbone network to obtain first feature images of different scales;

[0166] Using the backbone network, the first feature image with the smallest scale is input into the fusion network of the first object detection network, and second feature images of different scales are output;

[0167] Using the fusion network, input the second feature images of different scales into the detection network of the first image recognition model, and output a target area map of the target object;

[0168] The detection network is used to input the target area map into the classification output network and the mask output network of the first image recognition model respectively, and a first predicted classification result and a first predicted mask are output.

[0169] In some embodiments, the input module 1201 is further specifically configured to:

[0170] In the process of performing an upsampling operation on the first feature image with the smallest scale by using the first fusion network of the fusion network, a third feature image obtained by each upsampling operation is fused with the first feature image with the same scale to obtain a fourth feature image of a different scale;

[0171] In the process of downsampling the fourth feature image with the smallest scale by using the second fusion network of the fusion network, the fifth feature image obtained by each downsampling operation is fused with the first feature image and the fourth feature image of the same scale to obtain a second feature image of a different scale.

[0172] In some embodiments, the input module 1201 is further specifically configured to:

[0173] Inputting a first target feature image output by a first preset downsampling layer in the first image recognition model and a second target feature image output by a second preset downsampling layer in the second image recognition model into the degradation prediction network, and using the degradation prediction network to obtain a first feature vector corresponding to the first target feature image and a second feature vector corresponding to the second target feature image;

[0174] Using the degradation prediction network to concatenate the first feature vector and the second feature vector to obtain difference information between the first feature vector and the second feature vector, wherein the difference information includes the size of the predicted motion blur parameter, the direction of the predicted motion blur parameter, and noise information;

[0175] Passing the concatenated feature vector through multiple fully connected layers in sequence to obtain the number of predicted degradation parameters;

[0176] A degradation parameter prediction result is obtained based on the difference information and the number of the degradation parameters.

[0177] In some embodiments, the above-mentioned device further includes: an acquisition module and a scaling module.

[0178] An acquisition module is used to obtain a training set corresponding to a target object;

[0179] The scaling module is used to scale the multiple original images in the training set according to preset ratios to obtain second sample images corresponding to the multiple original images.

[0180] The present application also provides an image recognition device, which is used to execute the image recognition method provided in any of the above embodiments. Fig.13 As shown, the device includes: an input module 1301 , a determination module 1302 , an identification module 1303 and an output module 1304 .

[0181] The input module 1301 is used to input the image to be recognized into the first image recognition model, where the first image recognition model is trained using the model training method provided by any of the above embodiments.

[0182] The determination module 1302 is used to determine, by using the first image recognition model, an equivariant representation feature image of the feature image corresponding to the image to be recognized before degradation.

[0183] The recognition module 1303 is used to recognize the target object in the image to be recognized in the equivariant representation feature image by using the first image recognition model.

[0184] The output module 1304 is used to output the classification result of the target object in the image to be identified.

[0185] The present application also provides an electronic device to perform the above model training method. Fig.14 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Fig.14 As shown, the electronic device 7 includes: a processor 700, a memory 701, a bus 702 and a communication interface 703, and the processor 700, the communication interface 703 and the memory 701 are connected via the bus 702; the memory 701 stores a computer program that can be run on the processor 700, and when the processor 700 runs the computer program, it executes the model training method or image recognition method provided in any of the aforementioned embodiments of the present application.

[0186] The memory 701 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 703 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0187] The bus 702 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 701 is used to store programs, and the processor 700 executes the program after receiving the execution instruction. The model training method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 700, or implemented by the processor 700.

[0188] The processor 700 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 700. The above processor 700 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 701, and the processor 700 reads the information in the memory 701 and completes the steps of the above method in combination with its hardware.

[0189] The electronic device provided in the embodiments of the present application and the model training method or target detection provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0190] The present application also provides a computer-readable storage medium corresponding to the model training method provided in the above embodiment. Fig.15 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, it executes the model training method or image recognition method provided by any of the aforementioned embodiments.

[0191] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0192] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the model training method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0193] It should be noted that:

[0194] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0195] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0196] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: include: Inputting a first sample image into a first image recognition model, and outputting a first prediction classification result and a first prediction mask, wherein the first sample image is generated by adding a degradation parameter to the second sample image; Inputting a first target feature image output by a first image recognition model and a second target feature image output by a second image recognition model into a degradation prediction network, and outputting a degradation parameter prediction result, wherein the input of the second image recognition model is the second sample image; Calculating a first loss function value of the first image recognition model based on the first predicted classification result and the first predicted mask; Calculating a second loss function value based on the degradation parameter and the degradation parameter prediction result; The model parameters of the first image recognition model are adjusted based on the first loss function value and the second loss function value, and the training is continued until a preset training completion condition is met to obtain a trained first image recognition model.

2. The method according to claim 1, characterized in that: The method further comprises: The second sample image is input into a second image recognition model, and a second prediction classification result and a second prediction mask are output: Based on the second predicted classification result and the second predicted mask, a third loss function of the second image recognition model is calculated.

3. The method according to claim 2, characterized in that The adjusting the model parameters of the first image recognition model based on the first loss function value and the second loss function value, continuing the training until a preset training completion condition is met, and obtaining a trained first image recognition model includes: Determine the weight coefficients corresponding to the first loss function value, the second loss function value, and the third loss function value respectively; Determine a total loss function value of the first image recognition model based on the first loss function value, the second loss function value, the third loss function value, and their corresponding weight coefficients; The model parameters of the first image recognition model are adjusted based on the total loss function value, and the training is continued until a preset training completion condition is met to obtain a trained first image recognition model.

4. The method according to claim 1, characterized in that: The step of inputting the first sample image into a first image recognition model and outputting a first prediction classification result and a first prediction mask comprises: Inputting the first sample image into the backbone network of the image recognition model, and introducing an attention mechanism in a preset downsampling layer during a downsampling operation on the first sample image using multiple downsampling layers of the backbone network to obtain first feature images of different scales; Using the backbone network, the first feature image with the smallest scale is input into the fusion network of the first object detection network, and second feature images of different scales are output; Using the fusion network, input the second feature images of different scales into the detection network of the first image recognition model, and output a target area map of the target object; The detection network is used to input the target area map into the classification output network and the mask output network of the first image recognition model respectively, and a first predicted classification result and a first predicted mask are output.

5. The method according to claim 4, characterized in that The step of using the backbone network to input the first feature image with the smallest scale into a fusion network and outputting second feature images of different scales comprises: In the process of performing an upsampling operation on the first feature image with the smallest scale by using the first fusion network of the fusion network, a third feature image obtained by each upsampling operation is fused with the first feature image with the same scale to obtain a fourth feature image of a different scale; In the process of downsampling the fourth feature image with the smallest scale by using the second fusion network of the fusion network, the fifth feature image obtained by each downsampling operation is fused with the first feature image and the fourth feature image of the same scale to obtain a second feature image of a different scale.

6. The method according to claim 1, characterized in that The step of inputting the first target feature image output by the first image recognition model and the second target feature image output by the second image recognition model into the degradation prediction network and outputting the degradation parameter prediction result comprises: Inputting a first target feature image output by a first preset downsampling layer in the first image recognition model and a second target feature image output by a second preset downsampling layer in the second image recognition model into the degradation prediction network, and using the degradation prediction network to obtain a first feature vector corresponding to the first target feature image and a second feature vector corresponding to the second target feature image; Using the degradation prediction network to concatenate the first feature vector and the second feature vector to obtain difference information between the first feature vector and the second feature vector, wherein the difference information includes the size of the predicted motion blur parameter, the direction of the predicted motion blur parameter, and noise information; Passing the concatenated feature vector through multiple fully connected layers in sequence to obtain the number of predicted degradation parameters; A degradation parameter prediction result is obtained based on the difference information and the number of the degradation parameters.

7. The method according to claim 1, characterized in that Before inputting the first sample image into the first image recognition model and outputting the first prediction classification result and the first prediction mask, the method further includes: Get the training set corresponding to the target object; The plurality of original images in the training set are scaled according to preset ratios to obtain second sample images corresponding to the plurality of original images.

8. An image recognition method, characterized in that: include: Inputting the image to be recognized into a first image recognition model, wherein the first image recognition model is trained using the model training method according to any one of claims 1 to 7; Determine, by using the first image recognition model, an equivariant representation feature image of the feature image corresponding to the image to be recognized before degradation; Using the first image recognition model to recognize a target object in the image to be recognized in the equivariant representation feature image; Output the classification result of the target object in the image to be identified.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images