Image data classification method and related products

Abstract

The present invention discloses a classification method for image data and related products. By inserting an adaptive perturbation module, the robustness of the image classification model to data is enhanced, and the performance stability of the image classification model is ensured when facing unknown out-of-distribution data and large domain offsets.

Description

Technical Field

[0001] The invention relates to a classification method for image data and related products. Background Art

[0002] Domain generalization refers to the ability of a machine learning model to perform well in different domains after being trained on data from a specific domain. The goal is to make the model robust to changes in the input data distribution that may occur in different real-world scenarios or domains. Domain generalization is important in various fields such as computer vision, natural language processing, robotics, and healthcare, where models need to adapt to diverse and evolving environments.

[0003] Data in real environments often cause domain shift due to changes in lighting conditions, background settings, sensor types, etc. At the same time, the data used to train the model may come from a specific distribution domain, but when the model is deployed to a new domain, it may encounter data distributions that have never been encountered, resulting in the model specifically for the training domain not being able to generalize well to the new domain.

[0004] Chinese invention patent application CN114091597A proposes an adversarial training method, device and equipment based on adaptive group sample perturbation constraints. In this method, the training image is input into the initial network model to obtain the network output vector and predicted category corresponding to the training image; if the classification result of the training image is determined to be wrong based on the predicted category and the actual category of the training image, the training image is determined to be a natural sample image; if the classification result of the training image is determined to be correct based on the predicted category and the actual category, the target adaptive group corresponding to the training image is determined based on the network output vector; the target perturbation vector corresponding to the training image is determined based on the target sample perturbation constraint corresponding to the target adaptive group, and a perturbed sample image is generated based on the target perturbation vector and the training image; the initial network model is trained based on the natural sample image and the perturbed sample image to obtain the target network model. Through the technical solution of this application, the anti-interference ability of the target network model for attack samples is significantly improved. Summary of the invention

[0005] The present invention provides an image data classification method and related products to improve the robustness of the model in visual classification tasks and ensure the performance stability of the model when facing unknown out-of-distribution data and large domain offsets.

[0006] In a first aspect, a method for classifying image data is provided, comprising:

[0007] Input the acquired training samples into the initially constructed image classification model, output the corresponding sample prediction results, and generate the corresponding sample intermediate layer features under a preset semantic preservation mechanism; the image classification model includes: a main network and an auxiliary network; the main network and the auxiliary network are composed of convolutional networks with residual connections, and each convolutional block in the convolutional network of the main network is inserted with an adaptive perturbation module; the sample intermediate layer features include original sample features generated by the auxiliary network and perturbed sample features generated by the main network;

[0008] Based on the sample labels corresponding to the training samples and the sample prediction results, determining the total model loss; the total model loss is used to update the parameters of the image classification model;

[0009] Determining the perturbation loss of the adaptive perturbation module based on the original sample features, the perturbation sample features and the sample labels corresponding to the training samples; the perturbation loss is used to update the parameters of the adaptive perturbation module;

[0010] Using the optimal disturbance parameters updated by the adaptive disturbance module to generate a target adaptive disturbance module;

[0011] Using the target adaptive perturbation module and the optimal network parameters obtained by updating the image classification model, a target image classification model is generated;

[0012] When receiving an image classification instruction, obtaining image data to be classified;

[0013] The image data to be classified is input into the target image classification model to obtain a classification prediction result.

[0014] In a second aspect, a device for classifying image data is provided, comprising:

[0015] A prediction unit is used to input the acquired training samples into the initially constructed image classification model, output the corresponding sample prediction results, and generate corresponding sample intermediate layer features under a preset semantic preservation mechanism; the image classification model includes: a main network and an auxiliary network; the main network and the auxiliary network are composed of convolutional networks with residual connections, and each convolutional block in the convolutional network of the main network is inserted with an adaptive perturbation module; the sample intermediate layer features include original sample features generated by the auxiliary network, and perturbed sample features generated by the main network;

[0016] A model loss determination unit, used to determine a total model loss based on the sample labels corresponding to the training samples and the sample prediction results; the total model loss is used to update the parameters of the image classification model;

[0017] A disturbance loss determination unit, used to determine the disturbance loss of the adaptive disturbance module based on the original sample features, the disturbed sample features and the sample labels corresponding to the training samples; the disturbance loss is used to update the parameters of the adaptive disturbance module;

[0018] A target module generating unit, configured to generate a target adaptive disturbance module by using the optimal disturbance parameters updated by the adaptive disturbance module;

[0019] A target model generation unit, configured to generate a target image classification model by using the target adaptive perturbation module and the optimal network parameters obtained by updating the image classification model;

[0020] A to-be-classified data acquisition unit, used to acquire the to-be-classified image data when receiving an image classification instruction;

[0021] The prediction unit is used to input the image data to be classified into the target image classification model to obtain a classification prediction result.

[0022] According to a third aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps or all steps involving model training in the aforementioned method are executed.

[0023] In a fourth aspect, a program product is provided, which, when running on a processor, executes the steps or all steps involved in model training in the aforementioned method.

[0024] By inserting an adaptive perturbation module, the robustness of the image classification model to the data is enhanced, ensuring the performance stability of the image classification model when facing unknown out-of-distribution data and large domain offsets. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of a first embodiment of a method for classifying image data according to the present invention.

[0026] Figure 2 Schematic diagram of an adaptive perturbation module in Embodiment 2 of a method for classifying image data of the present invention.

[0027] Figure 3 Schematic diagram of the semantic preservation mechanism based on frequency domain amplitude and phase information reorganization of the present invention.

[0028] Figure 4 This is a complete algorithm framework of Embodiment 2 of an image data classification method of the present invention.

[0029] Figure 5 The present invention is a structural block diagram of an embodiment of an image data classification device of the present invention. DETAILED DESCRIPTION

[0030] The present invention is further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto.

[0031] Embodiment 1

[0032] See also Figure 1 , Figure 1 This is a flowchart of a first embodiment of a method for classifying image data according to the present invention.

[0033] Step S101, input the acquired training samples into the initially constructed image classification model, output the corresponding sample prediction results, and generate the corresponding sample intermediate layer features under a preset semantic preservation mechanism; the image classification model includes: a main network and an auxiliary network; the main network and the auxiliary network are composed of convolutional networks with residual connections, and each convolutional block in the convolutional network of the main network is inserted with an adaptive perturbation module; the sample intermediate layer features include the original sample features generated by the auxiliary network, and the perturbed sample features generated by the main network.

[0034] In the embodiment of the present invention, the main network and auxiliary network are both composed of convolutional networks with residual connections, and an adaptive perturbation module is inserted into each convolutional block in the convolutional network of the main network. The adaptive perturbation module can generate a specific adaptive perturbation amount according to the input features to maximize the flexibility and pertinence of the perturbation.

[0035] In a specific implementation, the learnable parameters included in the adaptive perturbation module involved in the embodiment of the present invention are mainly composed of convolution and multi-layer perceptron, with a small number of parameters and no need for too much additional training. This can overcome the deficiency of fixed strategies that it is difficult to change the style of the entire feature as much as possible while ensuring that the content remains unchanged, help predict the optimal perturbation direction and perturbation amplitude, and make the perturbation style develop towards the style that is most sensitive to the current model parameters.

[0036] At the same time, the adaptive perturbation module involved in the embodiment of the present invention generates a specific perturbation for each different training sample through an adaptive perturbation amount generation mechanism, thereby overcoming the problem that the existing method uses the same set of perturbations for each batch of data and does not consider the differences between samples in the batch, and generates a specific perturbation for each different training sample.

[0037] Step S102, determining the total model loss based on the sample labels corresponding to the training samples and the sample prediction results; the total model loss is used to update the parameters of the image classification model.

[0038] In an embodiment of the present invention, the total model loss is determined by comparing sample labels and sample prediction results, and the parameters of the image classification model are updated through a back propagation algorithm based on the total model loss to obtain the optimal network parameters.

[0039] Step S103, determining the perturbation loss of the adaptive perturbation module based on the original sample features, the perturbed sample features and the sample labels corresponding to the training samples; the perturbation loss is used to update the parameters of the adaptive perturbation module.

[0040] In an embodiment of the present invention, the perturbation loss of the adaptive module is determined by comparing the error between the original sample features and the perturbed sample features, as well as the error between the prediction result corresponding to the training sample and the sample label, and the parameters of the adaptive perturbation module are updated based on the perturbation loss through the back propagation algorithm to obtain the optimal perturbation parameters.

[0041] Step S104: using the optimal disturbance parameters updated by the adaptive disturbance module to generate a target adaptive disturbance module.

[0042] In the embodiment of the present invention, the target adaptive disturbance module is generated by using the optimal disturbance parameters.

[0043] Step S105, using the target adaptive perturbation module and the optimal network parameters obtained by updating the image classification model to generate a target image classification model.

[0044] In an embodiment of the present invention, the target image classification model is generated by using optimal network parameters and a target adaptive perturbation module.

[0045] Step S106, when receiving an image classification instruction, obtaining image data to be classified.

[0046] Step S107, inputting the image data to be classified into the target image classification model to obtain a classification prediction result.

[0047] In the embodiment of the present invention, the target image classification model processes the input image data to be classified and then gives a corresponding classification prediction result, which is usually the category label or category probability distribution to which the image belongs, and is used to identify the content or features of the image.

[0048] By inserting an adaptive perturbation module, the robustness of the image classification model to the data is enhanced, ensuring the performance stability of the image classification model when facing unknown out-of-distribution data and large domain offsets.

[0049] Embodiment 2

[0050] Embodiment 2 provides a method for classifying image data.

[0051] Step S201, input the acquired training samples into the initially constructed image classification model, output the corresponding sample prediction results, and generate the corresponding sample intermediate layer features under a preset semantic preservation mechanism; the image classification model includes: a main network and an auxiliary network; the main network and the auxiliary network are composed of convolutional networks with residual connections, and each convolutional block in the convolutional network of the main network is inserted with an adaptive perturbation module; the sample intermediate layer features include the original sample features generated by the auxiliary network, and the perturbed sample features generated by the main network; the sample prediction results include: the main network sample prediction results and the auxiliary network sample prediction results.

[0052] The perturbation is applied to the global statistics of the features, and the direction and amplitude of the perturbation are given by a perturbation generation module containing learnable parameters. For different input samples, the perturbation generation module can give an adaptive perturbation amount.

[0053] In the specific implementation, the mean and variance of the input features are first calculated channel by channel according to the feature mean determination expression and the feature variance determination expression.

[0054] It should be understood that in statistics, mean and variance are important statistics that describe the distribution of data in a data set. The mean represents the center position of the data set, while the variance represents the degree of discreteness of the data. In the adaptive perturbation network included in the adaptive perturbation module of an embodiment of the present invention, by calculating the mean and variance of the input features, the feature distribution of the input samples can be better understood, thereby generating an adaptive perturbation amount, that is, a slight change or interference to the original sample to enhance the diversity and challenge of adversarial samples. Among them, these input features include feature maps output by each layer of the convolutional neural network, and each feature map represents a different level of abstract representation of the input data.

[0055] In the process of generating the perturbation, the scaling factor and the offset play a role in adjusting the amplitude and direction of the perturbation - they adjust the statistics of the original sample features to the statistics after perturbation through linear transformation, ensuring that the generated adversarial samples have obvious differences in features from the original samples while retaining the semantic information of the samples. In addition, the original sample features of the embodiment of the present invention are obtained after Instance norm. The feature mean determination expression is:

[0056] ;

[0057] in, is the feature map to be averaged, is the mean of the feature map, is the width of the feature map, is the height of the feature map, is the index of the sample to which the current input belongs, is the dimension index, and They are the height position index and width position index of the pixel in the feature map, respectively. It is the feature map of the pixel corresponding to the subscript.

[0058] The characteristic variance determination expression is:

[0059] ;

[0060] in, is the variance of the input feature map, and the other symbols are defined the same as above.

[0061] Taking into account the statistical differences between samples of different categories and samples of different fields, the convolutional neural network is combined with the multi-layer perceptron to construct an adaptive perturbation network. The network takes the sample intermediate layer features of the training sample as input and outputs the perturbation amount for the input sample. The specific process is shown in the following expression:

[0062] ;

[0063] in, is the scaling factor, is the offset, is the original sample feature of the input, is the convolution operation on the original sample features. is the maximum pooling operation applied to the convolution result, {●} is a multi-layer perceptron.

[0064] Subsequently, the perturbation is applied to the statistics of the original sample features through linear transformation to obtain the perturbed statistics. Finally, the statistics of the original sample features are adjusted to the perturbed statistics through scaling and offset using linear transformation, as shown in the following formula:

[0065] ;

[0066] in, is the sample feature after disturbance, is the variance after disturbance, is the original sample feature, is the mean after disturbance.

[0067] Finally, the perturbed sample features are used as the output of the adaptive perturbation module. The complete process is as follows: Figure 2 As shown in the figure, it can be seen that the adaptive perturbation module includes all structures including the adaptive perturbation network for completing the adaptive adjustment of the original sample features.

[0068] In an optional embodiment, the acquired training samples are input into the initially constructed image classification model, the corresponding sample prediction results are output, and the corresponding sample intermediate layer features are generated under a preset semantic preservation mechanism, including:

[0069] Using bidirectional KL divergence to measure the KL divergence difference between the prediction result of the main network sample and the prediction result of the auxiliary network sample; the KL divergence difference is used for the adaptive perturbation module to perform parameter update;

[0070] After respectively decomposing the original sample feature and the perturbed sample feature into an amplitude map and a phase map, recombining the phase map of the original sample feature and the amplitude map of the perturbed sample feature;

[0071] The frequency domain expression of the reassembled amplitude-phase image is restored to the spatial domain expression through inverse Fourier transform to obtain the sample characteristics after disturbance.

[0072] Ensuring that the semantics of features remain unchanged before and after disturbance is the key to image classification models capturing generalized semantic patterns. The embodiment of the present invention mentions two complementary semantic preservation mechanisms: the first is to apply regularization at the prediction result level to ensure that the image classification model gives consistent prediction results for the same input training sample when disturbance is applied or not. The second is to operate on features in the frequency domain, taking into account that the amplitude and phase maps in the frequency domain have clear physical meanings corresponding to the spatial domain, that is, the amplitude map determines the style of the image in the spatial domain, such as brightness, color, and contrast, and the phase map determines the content of the image in the spatial domain, such as shape and texture. Therefore, this part decomposes and reorganizes the features in the frequency domain. This operation makes the output features have both the content before the disturbance and the style after the disturbance.

[0073] In the specific implementation, consistency regularization is applied to the two prediction results with and without disturbance, as shown in the following expression:

[0074] ;

[0075] in, is the KL divergence of the auxiliary network sample prediction result relative to the main network sample prediction result, is the KL divergence of the prediction results of the main network samples relative to the prediction results of the auxiliary network samples, is the consistency loss, i.e., the KL divergence difference.

[0076] For the adaptive perturbation module, the introduction of this mechanism can not only suppress the content distortion caused by the perturbation to a great extent, so that the applied perturbation will not affect the image classification model's understanding and perception of the input semantics, but also for the image classification model itself, this mechanism further improves the model's robustness to perturbations. Even if the perturbation is generated in a targeted manner, the model can well avoid its impact on the prediction.

[0077] See also Figure 3 , Figure 3 This is a schematic diagram of the semantic preservation mechanism based on the reorganization of frequency domain amplitude and phase information of the present invention. In the figure, for the input image data, a two-dimensional Fourier transform is used to decompose the sample features before and after the disturbance into an amplitude map for characterizing the style of the image data and a phase map for characterizing the content of the image data, and the phase map of the sample features before the disturbance and the amplitude map of the sample features after the disturbance are recombined, and then the inverse Fourier transform is used to restore the reorganized frequency domain expression to the spatial domain expression.

[0078] Step S202, comparing the prediction result of the main network sample and the prediction result of the auxiliary network sample to determine the consistency loss.

[0079] Step S203, determining the main network cross entropy loss and the auxiliary network cross entropy loss according to the training error between the main network sample prediction result and the sample label, and the training error between the auxiliary network sample prediction result and the sample label, respectively.

[0080] Step S204, according to the pre-set loss weight, combining the consistency loss, the main network cross entropy loss and the auxiliary network cross entropy loss, to determine the total model loss; the total model loss is used to update the parameters of the image classification model.

[0081] Step S205, using a preset disturbance amount determination expression and combining the original sample characteristics to determine the disturbed sample characteristics.

[0082] Step S206, using a preset style similarity loss function, combining the perturbed sample features with the original sample features, to determine the style similarity loss.

[0083] In the embodiment of the present invention, the style information of the feature map is modeled by calculating the inner product of the activation values ​​between the channels of the feature, and the style similarity loss is constructed based on this to measure the style difference between different features.

[0084] In the specific implementation, according to the task objectives, the applied disturbance should meet the following requirements: First, it should be ensured that the feature styles before and after the disturbance have obvious differences to ensure the style richness brought by the disturbance operation. Secondly, the direction of the disturbance is not random, but should target the style to which the current network parameters are most sensitive. In view of this, the present invention designs two exclusive loss functions for the learnable adaptive perturbation network part, namely the style similarity loss function and the adversarial perturbation loss function, which correspond to the above two requirements respectively. Among them, the style similarity loss function is:

[0085] ;

[0086] ;

[0087] in, is the element in the style matrix of the original sample feature, is the element in the style matrix of the sample feature after perturbation, and Represent the channel index in the style matrix, is the dimension of the feature map, is the number of channels of the feature map, For Channel The corresponding feature map is For Channel The corresponding feature map is It is the element-wise product operation.

[0088] This loss first models the style information of the feature map and encodes the correlation between channels into a The matrix is ​​denoted as the style matrix . Then, the style matrices of the two feature maps are subtracted, and the mean square error (MSE) is used to calculate the mean square error between the two as a measure of feature similarity. During the training phase, the style similarity loss is calculated using the sample features before and after the perturbation, and the parameters of the adaptive perturbation network are updated to maximize the loss, which can maximize the style difference brought by the perturbation operation.

[0089] Step S207, determining the adversarial perturbation loss according to a preset adversarial perturbation loss function and combining the training error between the main network sample prediction result and the sample label.

[0090] The frequency domain decomposition and reorganization operation is designed based on the two-dimensional Fourier transform, which realizes the semantic protection of the style editing operation process. Specifically, the frequency domain information of the feature is decomposed into amplitude and phase based on the two-dimensional Fourier transform, which realizes the decoupling of the spatial domain style and spatial domain semantic content of the feature in the frequency domain. The phase information of the feature before style editing and the amplitude information after style editing are recombined to ensure that the style transfer process does not cause distortion of the image content.

[0091] In the module training process, the adversarial perturbation loss introduced in the embodiment of the present invention causes the module to produce an erroneous main network prediction result during the training process, and causes the input training sample to be misclassified as a category different from the true label, denoted as , follow the following formula to update the parameters:

[0092] ;

[0093] ;

[0094] in, is the internal parameter of the adaptive disturbance module, is the learning rate, is the adversarial perturbation loss, is the adversarial perturbation loss about The gradient of (●) is the symbol for calculating cross entropy loss, is the sample size, is the sample index, is the sample feature after disturbance, is the category label.

[0095] Step S208: using the optimal disturbance parameters updated by the adaptive disturbance module to generate a target adaptive disturbance module.

[0096] The embodiment of the present invention constructs a feature adaptive perturbation module and adds it to a specific position of the selected visual backbone network to inject adaptive perturbations into the network's middle layer features. During the perturbation parameter update process, the model parameters are divided into two groups, the parameters in the adaptive perturbation module as one group, and all other learnable parameters as one group, and each group of parameters is updated separately using a step-by-step update mechanism. The update process of the parameters in the adaptive perturbation module is provided with a gradient by the style similarity loss function, and the semantic preservation mechanism based on the reorganization of frequency domain amplitude and phase information is connected to the perturbation operation to ensure that the semantics before and after the perturbation remain unchanged.

[0097] Step S209, using the target adaptive perturbation module and the optimal network parameters obtained by updating the image classification model to generate a target image classification model;

[0098] In the embodiment of the present invention, the adaptive perturbation module is updated based on the KL divergence difference, the adaptive perturbation module parameters are determined, and the target adaptive perturbation module is obtained. At the same time, based on the total model loss, the image classification model parameters are updated to obtain the optimal network parameters.

[0099] In the specific implementation, experiments are conducted on multiple data sets to determine the optimal network hyperparameters, including the optimal insertion position of the adaptive perturbation module in various commonly used visual backbone networks. Then, the optimal network parameters and optimal network hyperparameters, as well as the target adaptive perturbation module, are used to generate a modular image classification model.

[0100] See also Figure 4 , Figure 4 This is a complete algorithm framework of Embodiment 2 of a method for classifying image data of the present invention. In the figure, Fc It is the classification loss function of the classifier in the image classification model. It is used to supervise the loss function of the model for classification tasks. The loss function types include adversarial loss L adv , cross entropy loss L cls , consistency loss L consis and style similarity loss L style . These loss functions can be used alone or in combination. All components in the model are trained in an end-to-end manner in the embodiment of the present invention, that is, all components from the input end to the output end, including: the main network and the auxiliary network, are trained together, rather than dividing the model into multiple parts for separate training, and then the network parameters are optimized using the Adam optimizer. The training process first defines the main network and the auxiliary network. The main network is mainly composed of a convolutional network (Rsenet) with residual connections, and the adaptive perturbation module designed by the embodiment of the present invention is inserted at the optimal position represented by the optimal network hyperparameters. The auxiliary network removes the adaptive perturbation module and the frequency domain decomposition and reorganization processing on the basis of the main network. At the end of the convolution layer of the main network, after the convolution layer of the main network generates a feature map containing the original sample features of the image data, the adaptive perturbation module is used to inject perturbations into the intermediate layer features, and then the above-mentioned frequency domain decomposition and reorganization method is used to process the obtained features to obtain feature information containing different frequency domains. The algorithm is trained in a distributed training form, including: the first step is to input the obtained training samples into the main network and the auxiliary network at the same time, obtain the prediction results of the main network samples and the auxiliary network samples, and then determine the consistency loss by comparing the prediction results of the main network samples and the auxiliary network samples. L consis, and determine the main network cross entropy loss L according to the training error between the main network sample prediction result and the sample label, and the training error between the auxiliary network sample prediction result and the sample label. cls1 and auxiliary network cross entropy loss L cls2 , weight these three losses in a certain proportion to obtain the total loss and update the network parameters; the second step uses the original sample features output by the main network and the perturbed sample features output by the auxiliary network to determine the style similarity loss L style , and use the main network sample prediction results of the main network to calculate the adversarial loss L adv , weight the two in a certain proportion, and use the weighted loss to update the parameters of the adaptive perturbation module.

[0101] Step S210, when receiving an image classification instruction, obtaining image data to be classified;

[0102] Step S211, inputting the image data to be classified into the target image classification model to obtain a classification prediction result.

[0103] In the embodiment of the present invention, after the trained target image classification model is obtained, the image data to be classified can be identified based on the target image classification model. The image data to be classified can be photos or screenshots (for example, video screenshots, web page screenshots, chat record screenshots, etc.), and the objects contained in the photos or screenshots can be traffic signs, license plates, users, animals, plants, or buildings, etc.

[0104] The present invention discloses a method for classifying image data, comprising: inputting the acquired training samples into an initially constructed image classification model, outputting corresponding sample prediction results, and generating corresponding sample intermediate layer features under a preset semantic preservation mechanism; the image classification model comprises: a main network and an auxiliary network; the main network and the auxiliary network are composed of convolutional networks with residual connections, and each convolutional block in the convolutional network of the main network is inserted with an adaptive perturbation module; the sample intermediate layer features include original sample features generated by the auxiliary network and perturbed sample features generated by the main network; based on the sample labels corresponding to the training samples and the sample prediction results, determining the total loss of the model; loss; the total model loss is used to update the parameters of the image classification model; based on the original sample features, the perturbed sample features and the sample labels corresponding to the training samples, the perturbation loss of the adaptive perturbation module is determined; the perturbation loss is used to update the parameters of the adaptive perturbation module; the optimal perturbation parameters updated by the adaptive perturbation module are used to generate the target adaptive perturbation module; the target adaptive perturbation module and the optimal network parameters and optimal network hyperparameters updated by the image classification model are used to generate the target image classification model; when receiving an image classification instruction, the image data to be classified is obtained; the image data to be classified is input into the target image classification model to obtain the classification prediction result. Based on feature perturbation and adversarial training technology, a plug-and-play network module and a simple and efficient training strategy are designed. By inserting an adaptive perturbation module, the robustness of the image classification model to the data is enhanced, and the performance stability of the image classification model is ensured when facing unknown out-of-distribution data and large domain offsets.

[0105] Embodiment 3

[0106] See also Figure 5 , Figure 5 This is a structural block diagram of an embodiment of an image data classification device of the present invention, the device comprising:

[0107] The prediction unit 301 is used to input the acquired training samples into the initially constructed image classification model, output the corresponding sample prediction results, and generate the corresponding sample intermediate layer features under the preset semantic preservation mechanism; the image classification model includes: a main network and an auxiliary network; the main network and the auxiliary network are composed of convolutional networks with residual connections, and each convolutional block in the convolutional network of the main network is inserted with an adaptive perturbation module; the sample intermediate layer features include original sample features generated by the auxiliary network, and perturbed sample features generated by the main network;

[0108] A model loss determination unit 302 is used to determine a total model loss based on the sample labels corresponding to the training samples and the sample prediction results; the total model loss is used to update the parameters of the image classification model;

[0109] The perturbation loss determination unit 303 is used to determine the perturbation loss of the adaptive perturbation module based on the original sample features, the perturbation sample features and the sample labels corresponding to the training samples; the perturbation loss is used to update the parameters of the adaptive perturbation module;

[0110] A target module generating unit 304 is used to generate a target adaptive disturbance module using the optimal disturbance parameters updated by the adaptive disturbance module;

[0111] A target model generation unit 305 is used to generate a target image classification model by using the target adaptive perturbation module and the optimal network parameters and optimal network hyperparameters obtained by updating the image classification model;

[0112] The to-be-classified data acquisition unit 306 is used to acquire the to-be-classified image data when receiving the image classification instruction;

[0113] The prediction unit 307 is used to input the image data to be classified into the target image classification model to obtain a classification prediction result.

[0114] In an optional embodiment, the sample prediction result includes: the main network sample prediction result and the auxiliary network sample prediction result; the total model loss is determined by the consistency loss and the cross entropy loss; the cross entropy loss includes: the main network cross entropy loss and the auxiliary network cross entropy loss; the model loss determination unit 302 includes:

[0115] A consistency loss determination subunit, used to compare the prediction result of the main network sample and the prediction result of the auxiliary network sample to determine the consistency loss;

[0116] A cross entropy loss determination subunit, used to determine the main network cross entropy loss and the auxiliary network cross entropy loss according to the training error between the main network sample prediction result and the sample label, and the training error between the auxiliary network sample prediction result and the sample label, respectively;

[0117] The model loss determination subunit is used to determine the total loss of the model according to a preset loss weight, combining the consistency loss, the main network cross entropy loss and the auxiliary network cross entropy loss.

[0118] In an optional embodiment, the prediction unit 301 includes:

[0119] A KL divergence difference determination subunit, used to measure the KL divergence difference between the main network sample prediction result and the auxiliary network sample prediction result using a bidirectional KL divergence; the KL divergence difference is used for the adaptive perturbation module to perform parameter update;

[0120] A recombining subunit, used for decomposing the original sample feature and the perturbed sample feature into an amplitude map and a phase map respectively, and then recombining the phase map of the original sample feature and the amplitude map of the perturbed sample feature;

[0121] The restoration subunit is used to restore the frequency domain expression of the recombined amplitude-phase image to the spatial domain expression through inverse Fourier transform to obtain the perturbed sample features.

[0122] In an optional embodiment, the disturbance loss includes: style similarity loss and adversarial disturbance loss; the disturbance loss determination unit 303 includes:

[0123] A disturbed sample feature determination subunit, used to determine the disturbed sample feature by using a preset disturbance amount determination expression in combination with the original sample feature;

[0124] A style similarity loss determination subunit, configured to determine the style similarity loss by combining the perturbed sample features with the original sample features using a preset style similarity loss function;

[0125] The adversarial perturbation loss determination subunit is used to determine the adversarial perturbation loss according to a preset adversarial perturbation loss function and in combination with the training error between the main network sample prediction result and the sample label.

[0126] In an optional embodiment, the disturbance amount determination expression is:

[0127] ;

[0128] The style similarity loss function is:

[0129] ;

[0130] The adversarial perturbation loss function is:

[0131] ;

[0132] in, is the sample feature after disturbance, is the variance after disturbance, is the mean after disturbance, is the instance normalization of the original sample feature after instance normalization, is the element in the style matrix of the original sample feature, is the element in the style matrix of the sample feature after perturbation, and Represent the channel index in the style matrix, is the dimension of the feature map, is the number of channels of the feature map, Generates internal parameters of the adaptive perturbation module, is the learning rate, is the adversarial perturbation loss about gradient.

[0133] Embodiment 4

[0134] An embodiment of the present invention also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of an image data classification method of any embodiment or only executes the steps involving model training.

[0135] Embodiment 5

[0136] This embodiment provides a program product, which, when running on a processor, executes the steps of an image data classification method of the aforementioned embodiment or only executes the steps involving model training.

[0137] The above-mentioned “units” and “sub-units” may be implemented by hardware, software, or a combination of hardware and software.

[0138] The above “units” and “sub-units” can be deployed on the same computing device or on different computing devices.

[0139] The above memories include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0140] The above processors are, for example, a central processing unit (CPU), a graphics processing unit (GPU), etc. The number of processors can be one or more, and they can be deployed on the same computing device or on different computing devices.

[0141] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments.

[0142] The protection scope of the present invention is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the scope and spirit of the present invention. If these changes and modifications fall within the scope of the claims of the present invention and their equivalents, the present invention is intended to include these changes and modifications.

Claims

1. A method for classifying image data, characterized in that: include: Input the acquired training samples into the initially constructed image classification model, output the corresponding sample prediction results, and generate the corresponding sample intermediate layer features under the pre-set semantic preservation mechanism; The image classification model includes: a main network and an auxiliary network; the main network and the auxiliary network are composed of convolutional networks with residual connections, and each convolutional block in the convolutional network of the main network is inserted with an adaptive perturbation module; the sample intermediate layer features include original sample features generated by the auxiliary network and perturbed sample features generated by the main network; Based on the sample labels corresponding to the training samples and the sample prediction results, determining the total model loss; the total model loss is used to update the parameters of the image classification model; Determining the perturbation loss of the adaptive perturbation module based on the original sample features, the perturbation sample features and the sample labels corresponding to the training samples; the perturbation loss is used to update the parameters of the adaptive perturbation module; Using the optimal disturbance parameters updated by the adaptive disturbance module to generate a target adaptive disturbance module; Using the target adaptive perturbation module and the optimal network parameters obtained by updating the image classification model, a target image classification model is generated; When receiving an image classification instruction, obtaining image data to be classified; Inputting the image data to be classified into the target image classification model to obtain a classification prediction result; The obtained training samples are input into the initially constructed image classification model, the corresponding sample prediction results are output, and the corresponding sample intermediate layer features are generated under the pre-set semantic preservation mechanism, including: The bidirectional KL divergence is used to measure the KL divergence difference between the sample prediction result of the main network and the sample prediction result of the auxiliary network; the KL divergence difference is used for the adaptive perturbation module to perform parameter update; After respectively decomposing the original sample feature and the perturbed sample feature into an amplitude map and a phase map, recombining the phase map of the original sample feature and the amplitude map of the perturbed sample feature; The frequency domain expression of the reassembled amplitude-phase image is restored to the spatial domain expression through inverse Fourier transform to obtain the sample characteristics after disturbance.

2. The image data classification method according to claim 1, characterized in that: The sample prediction results include: the main network sample prediction results and the auxiliary network sample prediction results; the total model loss is determined by the consistency loss and the cross entropy loss; the cross entropy loss includes: the main network cross entropy loss and the auxiliary network cross entropy loss; based on the sample labels corresponding to the training samples and the sample prediction results, the total model loss is determined, including: Comparing the prediction result of the main network sample and the prediction result of the auxiliary network sample to determine the consistency loss; Determining the main network cross entropy loss and the auxiliary network cross entropy loss according to the training error between the main network sample prediction result and the sample label, and the training error between the auxiliary network sample prediction result and the sample label, respectively; According to the preset loss weight, the consistency loss, the main network cross entropy loss and the auxiliary network cross entropy loss are combined to determine the total loss of the model.

3. The image data classification method according to claim 1 or 2, characterized in that: The perturbation loss includes: style similarity loss and adversarial perturbation loss; based on the original sample features, the perturbed sample features and the sample labels corresponding to the training samples, determining the perturbation loss of the adaptive perturbation module includes: Determine the sample characteristics after disturbance by using a preset disturbance amount determination expression and combining the original sample characteristics; Using a preset style similarity loss function, combining the perturbed sample features with the original sample features, to determine the style similarity loss; According to a preset adversarial perturbation loss function, the adversarial perturbation loss is determined in combination with the training error between the main network sample prediction result and the sample label.

4. The image data classification method according to claim 3, characterized in that: The disturbance amount determination expression is: ； The style similarity loss function is: ； The adversarial perturbation loss function is: ； in, is the sample feature after perturbation, is the variance after disturbance, is the mean after disturbance, is the instance normalization of the original sample feature after instance normalization, is the element in the style matrix of the original sample feature, is the element in the style matrix of the sample feature after perturbation, and Represent the channel index in the style matrix, is the dimension of the feature map, is the number of channels of the feature map, Generates internal parameters of the adaptive perturbation module, is the learning rate, is the adversarial perturbation loss about gradient.

5. A device for classifying image data, characterized in that: include: The prediction unit is used to input the acquired training samples into the initially constructed image classification model, output the corresponding sample prediction results, and generate the corresponding sample intermediate layer features under the pre-set semantic preservation mechanism; The image classification model includes: a main network and an auxiliary network; the main network and the auxiliary network are composed of convolutional networks with residual connections, and each convolutional block in the convolutional network of the main network is inserted with an adaptive perturbation module; the sample intermediate layer features include original sample features generated by the auxiliary network and perturbed sample features generated by the main network; A model loss determination unit, used to determine a total model loss based on the sample labels corresponding to the training samples and the sample prediction results; the total model loss is used to update the parameters of the image classification model; A disturbance loss determination unit, used to determine the disturbance loss of the adaptive disturbance module based on the original sample features, the disturbed sample features and the sample labels corresponding to the training samples; the disturbance loss is used to update the parameters of the adaptive disturbance module; A target module generating unit, configured to generate a target adaptive disturbance module by using the optimal disturbance parameters updated by the adaptive disturbance module; A target model generation unit, configured to generate a target image classification model by using the target adaptive perturbation module and the optimal network parameters obtained by updating the image classification model; A to-be-classified data acquisition unit, used to acquire the to-be-classified image data when receiving an image classification instruction; A prediction unit, used for inputting the image data to be classified into the target image classification model to obtain a classification prediction result; The obtained training samples are input into the initially constructed image classification model, the corresponding sample prediction results are output, and the corresponding sample intermediate layer features are generated under the pre-set semantic preservation mechanism, including: The bidirectional KL divergence is used to measure the KL divergence difference between the sample prediction result of the main network and the sample prediction result of the auxiliary network; the KL divergence difference is used for the adaptive perturbation module to perform parameter update; After respectively decomposing the original sample feature and the perturbed sample feature into an amplitude map and a phase map, recombining the phase map of the original sample feature and the amplitude map of the perturbed sample feature; The frequency domain expression of the reassembled amplitude-phase image is restored to the spatial domain expression through inverse Fourier transform to obtain the sample characteristics after disturbance.

6. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the following are executed: Input the acquired training samples into the initially constructed image classification model, output the corresponding sample prediction results, and generate the corresponding sample intermediate layer features under the pre-set semantic preservation mechanism; The image classification model includes: a main network and an auxiliary network; the main network and the auxiliary network are composed of convolutional networks with residual connections, and each convolutional block in the convolutional network of the main network is inserted with an adaptive perturbation module; the sample intermediate layer features include original sample features generated by the auxiliary network and perturbed sample features generated by the main network; Based on the sample labels corresponding to the training samples and the sample prediction results, determining the total model loss; the total model loss is used to update the parameters of the image classification model; Determining the perturbation loss of the adaptive perturbation module based on the original sample features, the perturbation sample features and the sample labels corresponding to the training samples; the perturbation loss is used to update the parameters of the adaptive perturbation module; Using the optimal disturbance parameters updated by the adaptive disturbance module to generate a target adaptive disturbance module; Using the target adaptive perturbation module and the optimal network parameters obtained by updating the image classification model, a target image classification model is generated, and the target image classification model is used to classify the image to be classified; The obtained training samples are input into the initially constructed image classification model, the corresponding sample prediction results are output, and the corresponding sample intermediate layer features are generated under the pre-set semantic preservation mechanism, including: The bidirectional KL divergence is used to measure the KL divergence difference between the sample prediction result of the main network and the sample prediction result of the auxiliary network; the KL divergence difference is used for the adaptive perturbation module to perform parameter update; After respectively decomposing the original sample feature and the perturbed sample feature into an amplitude map and a phase map, recombining the phase map of the original sample feature and the amplitude map of the perturbed sample feature; The frequency domain expression of the reassembled amplitude-phase image is restored to the spatial domain expression through inverse Fourier transform to obtain the sample characteristics after disturbance.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 4 is executed.

8. A program product, characterized in that It is executed by the processor runtime: Input the acquired training samples into the initially constructed image classification model, output the corresponding sample prediction results, and generate the corresponding sample intermediate layer features under the pre-set semantic preservation mechanism; The image classification model includes: a main network and an auxiliary network; the main network and the auxiliary network are composed of convolutional networks with residual connections, and each convolutional block in the convolutional network of the main network is inserted with an adaptive perturbation module; the sample intermediate layer features include original sample features generated by the auxiliary network and perturbed sample features generated by the main network; Based on the sample labels corresponding to the training samples and the sample prediction results, determining the total model loss; the total model loss is used to update the parameters of the image classification model; Determining the perturbation loss of the adaptive perturbation module based on the original sample features, the perturbation sample features and the sample labels corresponding to the training samples; the perturbation loss is used to update the parameters of the adaptive perturbation module; Using the optimal disturbance parameters updated by the adaptive disturbance module to generate a target adaptive disturbance module; Using the target adaptive perturbation module and the optimal network parameters obtained by updating the image classification model, a target image classification model is generated, and the target image classification model is used to classify the image to be classified; The obtained training samples are input into the initially constructed image classification model, the corresponding sample prediction results are output, and the corresponding sample intermediate layer features are generated under the pre-set semantic preservation mechanism, including: The bidirectional KL divergence is used to measure the KL divergence difference between the sample prediction result of the main network and the sample prediction result of the auxiliary network; the KL divergence difference is used for the adaptive perturbation module to perform parameter update; After respectively decomposing the original sample feature and the perturbed sample feature into an amplitude map and a phase map, recombining the phase map of the original sample feature and the amplitude map of the perturbed sample feature; The frequency domain expression of the reassembled amplitude-phase image is restored to the spatial domain expression through inverse Fourier transform to obtain the sample characteristics after disturbance.

9. A program product, characterized in that When executed by a processor, the method according to any one of claims 1 to 4 is executed.

Images