Training method and device of image classification model, equipment and storage medium

Abstract

Embodiments of the present specification provide a training method and device of an image classification model, an apparatus and a storage medium. The training method of the image classification model comprises: performing occlusion processing on a first sample image according to a first classification prediction result of the first sample image to obtain a second sample image; the first classification prediction result is obtained by inputting the first sample image into a first image classification model for classification prediction processing; the first image classification model comprises N sub-modules; the input of an (i+1)th sub-module is determined by the output of an ith sub-module; the second sample image is input into the first image classification model for iterative training to obtain a second image classification model; the step of generating an intermediate image classification model is performed N times, and the intermediate image classification model generated in the Nth time is determined as a target image classification model. In this way, the robustness and generality of the image classification model are improved.

Description

Technical Field

[0001] This application relates to the field of computer vision computing, and in particular to a training method, apparatus, device, and storage medium for an image classification model. Background Technology

[0002] With the development of electronic technology, the demand for image classification is growing rapidly. Performing image classification tasks using a well-trained image classification model is a common image classification method. On the one hand, some image classification models, such as CNN (Convolutional Neural Network) models, often extract multiple highly correlated features due to the large number of adjustable parameters, resulting in redundant features. This leads to poor generalization and significant differences in classification performance between training and test samples. On the other hand, some image classification models may rely too heavily on a small number of salient features. When the image to be classified is disturbed, affecting the extraction of salient features, the prediction of such models is likely to be incorrect, resulting in low robustness. Summary of the Invention

[0003] This application provides a training method, apparatus, device, and storage medium for an image classification model to improve the robustness and versatility of the image classification model.

[0004] In a first aspect, embodiments of this application provide a method for training an image classification model, comprising:

[0005] Based on the first classification prediction result of the first sample image, the first sample image is occluded to obtain the second sample image; the first classification prediction result is obtained by inputting the first sample image into the first image classification model for classification prediction processing; the first image classification model includes N sub-modules; the input of the (i+1)th sub-module is determined by the output of the ith sub-module;

[0006] The second sample image is input into the first image classification model for iterative training to obtain the second image classification model;

[0007] Perform the step of generating intermediate image classification models N-1 times, and determine the intermediate image classification model generated in the N-1th time as the target image classification model;

[0008] The step of generating the intermediate image classification model for the i-th time includes: freezing the parameters of the i-th sub-module in the second image classification model to obtain the third image classification model; inputting the first sample image into the third image classification model for classification prediction to obtain the second classification prediction result of the first sample image; determining the occluded region corresponding to the feature map group output by the i-th sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain the intermediate image classification model; when i is less than N-1, the intermediate image classification model is used as the second image classification model corresponding to the (i+1)-th step of generating the intermediate image classification model.

[0009] Secondly, embodiments of this application provide an image classification method, including:

[0010] Obtain the target image to be classified;

[0011] The target image is input into an image classification model for classification prediction processing to obtain the classification prediction result of the target image; the image classification model is trained by the image classification model training method as described in the first aspect.

[0012] Thirdly, embodiments of this application provide a training apparatus for an image classification model, comprising:

[0013] An occlusion unit is used to occlude the first sample image based on the first classification prediction result of the first sample image to obtain a second sample image; the first classification prediction result is obtained by inputting the first sample image into a first image classification model for classification prediction processing; the first image classification model includes N sub-modules; the input of the (i+1)th sub-module is determined by the output of the ith sub-module;

[0014] The training unit is used to input the second sample image into the first image classification model for iterative training to obtain the second image classification model.

[0015] A generation unit is used to perform N-1 steps of generating intermediate image classification models, and to determine the intermediate image classification model generated in the N-1th step as the target image classification model. The step of generating the i-th intermediate image classification model includes: freezing the parameters of the i-th sub-module in the second image classification model to obtain a third image classification model; inputting the first sample image into the third image classification model for classification prediction processing to obtain a second classification prediction result for the first sample image; determining the occluded region corresponding to the feature map group output by the i-th sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model; when i is less than N-1, the intermediate image classification model is used as the corresponding second image classification model in the (i+1)-th step of generating the intermediate image classification model.

[0016] Fourthly, embodiments of this application provide an image classification apparatus, including:

[0017] The acquisition unit is used to acquire the target image to be classified.

[0018] The classification unit is used to input the target image into the image classification model for classification prediction processing to obtain the classification prediction result of the target image; the image classification model is trained by the image classification model training method as described in the second aspect.

[0019] Fifthly, embodiments of this application provide an electronic device, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform a training method for an image classification model as described in the first aspect, or an image classification method as described in the second aspect.

[0020] In a sixth aspect, embodiments of this application provide a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the training method for the image classification model as described in the first aspect, or the image classification method as described in the second aspect.

[0021] As can be seen, in this embodiment, firstly, based on the first classification prediction result of the first sample image, the first sample image is occluded to obtain the second sample image; the first classification prediction result is obtained by inputting the first sample image into the first image classification model for classification prediction processing; the first image classification model includes N sub-modules; the input of the (i+1)th sub-module is determined by the output of the ith sub-module; secondly, the second sample image is input into the first image classification model for iterative training to obtain the second image classification model; then, the step of generating an intermediate image classification model N-1 times is executed, and the intermediate image classification model generated in the N-1th time is determined as the target image classification model; wherein, the step of generating the ith intermediate image classification model is executed. The steps of the intermediate image classification model include: freezing the parameters of the i-th sub-module in the second image classification model to obtain the third image classification model; inputting the first sample image into the third image classification model for classification prediction to obtain the second classification prediction result of the first sample image; determining the occluded region corresponding to the feature map group output by the i-th sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain the intermediate image classification model; when i is less than N-1, using the intermediate image classification model as the corresponding second image classification model in the (i+1)-th step of generating the intermediate image classification model. Therefore, by occluding the first sample image based on the first classification prediction result, the salient features in the first sample image can be blocked, forcing each sub-module in the first image classification model to mine secondary features other than the salient features. Furthermore, by performing N-1 steps to generate intermediate image classification models, the parameters of the i-th sub-module are frozen during the i-th step of generating intermediate image classification models, which can fix the model parameters of the already trained i-th sub-module layer by layer. Based on the second classification prediction result, the feature map set output by the i-th sub-module is occluded, blocking the salient features in the feature map set output by the i-th sub-module. This forces the (i+1)-th sub-module following the i-th sub-module to mine secondary features other than the salient features in the feature map set output by the i-th sub-module. Thus, in the target image classification model, the first sub-module can mine rich features from the image input to the target image classification model, and each sub-module other than the first sub-module can mine rich features from the feature map set output by the previous sub-module. This improves the overall feature mining performance of the target image classification model and enhances its robustness and generalization. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating a training method for an image classification model provided in this application embodiment;

[0024] Figure 2 This is a schematic diagram of the structure of an image classification model provided in an embodiment of this application;

[0025] Figure 3 A schematic diagram illustrating a training method for an image classification model provided in an embodiment of this application;

[0026] Figure 4 A schematic diagram illustrating a training method for a ResNet18 model provided in an embodiment of this application;

[0027] Figure 5 A flowchart illustrating an image classification method provided in this application embodiment;

[0028] Figure 6 A schematic diagram of a training device for an image classification model provided in an embodiment of this application;

[0029] Figure 7 A schematic diagram of an image classification device provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this application.

[0032] In practical applications, CNN models often consist of multiple sequentially connected sub-modules, with the feature map set input to each subsequent sub-module becoming the output feature map set of the preceding sub-module. On one hand, during the training phase of a CNN model, each sub-module involves multiple parameters to be trained, resulting in a massive number of parameters to be trained in the entire CNN model. Image classification models can easily extract multiple highly correlated features, forming redundant features and reducing the generalization ability of the CNN model. Even if relatively accurate image classification predictions are obtained for training samples during the model training phase, the prediction results may significantly decrease after replacing the training samples with test samples. On the other hand, each sub-module in a CNN model may rely on a small number of salient features. If the image to be classified input to the CNN model is disturbed, making it difficult to extract these salient features normally, the image classification prediction results of the CNN model are very likely to be incorrect, resulting in low robustness of the CNN model. To overcome the above problems, this application provides a training method for an image classification model.

[0033] The training method for the image classification model proposed in this application can be executed by an electronic device, specifically by a processor within that device. The electronic device mentioned here can be a terminal device, such as a smartphone, tablet, desktop computer, intelligent voice interaction device, wearable device, robot, or in-vehicle terminal, etc.; alternatively, the electronic device can also be a server, such as a standalone physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing.

[0034] The training method of the image classification model proposed in this application will be described in detail below through several embodiments.

[0035] Reference Figure 1 This is a flowchart illustrating a training method for an image classification model provided in an embodiment of this application. Figure 1 As shown, the training method for the image classification model provided in this application embodiment may specifically include the following steps:

[0036] Step S102: Based on the first classification prediction result of the first sample image, the first sample image is occluded to obtain the second sample image; the first classification prediction result is obtained by inputting the first sample image into the first image classification model for classification prediction processing; the first image classification model includes N sub-modules; the input of the (i+1)th sub-module is determined by the output of the ith sub-module.

[0037] The first sample images can be a set of pictures used for model training. The first sample images can be facial images, trademark images, car images, etc. This application embodiment does not impose special restrictions on the number and content of the first sample images.

[0038] The first image classification model can be used for image classification. Specifically, it can be used to detect whether a face image is fake, to identify the authenticity of a trademark, to identify the car model in an image, and so on.

[0039] The second sample image can be the first sample image after salient features have been occluded. The salient features in the first sample image can be determined based on the first classification prediction result.

[0040] In practice, the first sample image can be pre-input into the first image classification model for classification prediction processing to obtain the first classification prediction result of the first sample image. Then, based on the first classification prediction result, the region containing significant features in the first sample image is determined, and the region is occluded to obtain the second sample image.

[0041] The first image classification model may include N sub-modules and M data processing layers outside the sub-modules. Each sub-module may include one or more convolutional layers. Each sub-module may also include an activation function. The data processing layers may be modules that do not include parameters to be trained, such as pooling layers, reshape layers, shuffle layers, etc.

[0042] N can be any natural number greater than 1, and M can be any natural number greater than or equal to 0.

[0043] In some embodiments of the first image classification model, N sub-modules can be sequentially connected to M data processing layers, for example, sub-module 1 → sub-module 2 → sub-module 3 → sub-module 4 → data processing layer 1 → data processing layer 2.

[0044] In other embodiments of the first image classification model, the connection relationship between N sub-modules and M data processing layers can be freely set according to the model requirements. For example, sub-module 1 → data processing layer 1 → sub-module 2 → sub-module 3 → sub-module 4 → data processing layer 2.

[0045] This specification does not impose any special restrictions on the connection relationship between the N sub-modules and the M data processing layers in the first image classification model.

[0046] The first image classification model can be a CNN model or any other model that includes N sub-modules. Taking a CNN model as an example, each sub-module can be a stage in the CNN model. For example, stage1 only includes the first sub-module and does not include the data processing layer. Each sub-module can also be combined with one or more data processing layers to form a stage in the CNN model. For example, stage2 includes the second sub-module and pooling layer 1.

[0047] In the N submodules, adjacent submodules can be connected by a data processing layer. For example, the output of the i-th submodule is the input of data processing layer 1, and the output of data processing layer 1 is the input of the (i+1)-th submodule.

[0048] The data processing layer between two adjacent sub-modules can be a pooling layer. A common pooling layer is actually a form of downsampling. There are many different forms of non-linear pooling functions, among which "max pooling" is the most common.

[0049] It is important to emphasize that the pooling layer does not include the parameters to be trained. If the output of the i-th submodule is the input of pooling layer 1, and the output of pooling layer 1 is the input of the (i+1)-th submodule, then pooling layer 1 can be used to perform pooling operations on the feature map set output by the i-th submodule, and output the pooled feature map set to the (i+1)-th submodule. In this case, the input of the (i+1)-th submodule is determined by the output of the i-th submodule, which can be the feature map set input to the (i+1)-th submodule as the pooled feature map set output by the i-th submodule.

[0050] In N submodules, two adjacent submodules can be two submodules connected sequentially. For example, the output of the i-th submodule is the input of the (i+1)-th submodule. In this case, the input of the (i+1)-th submodule is determined by the output of the i-th submodule, which can be the feature map set input to the (i+1)-th submodule and the feature map set output by the i-th submodule.

[0051] A feature map set is a tensor (i.e., a group of feature maps) composed of feature maps extracted by a neural network. Its dimensions are typically represented by (H, W, C), where H is the height, W is the width, and C is the number of channels. Each feature map in a feature map set has the same width (W) and the same height (H). The number of feature maps is the same as the number of channels; therefore, a feature map set contains a total of C feature maps.

[0052] The following can be combined Figure 2 The structure of the first image classification model is illustrated by way of example. Figure 2 This is a schematic diagram of the structure of an image classification model provided in an embodiment of this application. Figure 2 The image classification model shown can be a specific implementation of the first image classification model.

[0053] Specifically, Figure 2 The image classification model shown can be a CNN model that includes convolutional layers, pooling layers, and fully connected layers. For example... Figure 2As shown, 55x55x96 is used to represent a feature map group with size parameters H=W=55. The number of feature maps included in this feature map group is 96. The meanings of 27x27x256, 13x13x384, and 13x13x256 can be referred to 55x55x96.

[0054] It should be noted that the 227x227x3 below the Input Image indicates that the input image includes a three-channel image with size parameters H=W=227, which can be equivalently regarded as the input image including three single-channel images with size parameters H=W=227.

[0055] `Conv` represents a convolutional layer, and `w / max pooling` represents a pooling layer. Therefore, in the image classification model, Stage 202 includes one convolutional layer and one pooling layer, Stage 204 includes one convolutional layer and one pooling layer, and Stage 206 includes three convolutional layers and one pooling layer. The feature map set output by Stage 202 can be represented as 55x55x96 pixels and is passed to Stage 204. The feature map set output by Stage 204 can be represented as 27x27x256 pixels and is passed to Stage 206. The feature map set output by Stage 206 can be represented as 13x13x256 pixels. The output of Stage 206 is the input to fully connected layer 208, and the output of fully connected layer 208 is the input to fully connected layer 210. The output of fully connected layer 210 is the input to classification layer 212. The output of classification layer 212 is the first classification prediction result corresponding to the input image.

[0056] The first image classification model can be pre-configured with multiple preset image categories. The first classification prediction result for the first sample image can be the predicted probability value of the first sample image corresponding to each of these preset image categories. This predicted probability value can be obtained by inputting the first sample image into the first image classification model for image classification prediction processing.

[0057] For example, if the first sample image is a trademark image and the preset image categories include "genuine trademark", "counterfeit trademark" and "other", then the first classification prediction result of the first sample image can include the predicted probability value p1 that the first sample image belongs to "genuine trademark", the predicted probability value p2 that the first sample image belongs to "counterfeit trademark", and the predicted probability value p3 that the first sample image belongs to "other".

[0058] For example, if the first sample image is a car image, and the preset image categories include "Type 1 car", "Type 2 car", ... "Type k car", then the first classification prediction result for the first sample image can include the predicted probability value p1 of the car in the first sample image belonging to "Type 1 car", the predicted probability value p2 of the car in the first sample image belonging to "Type 2 car", ... the predicted probability value pk of the car in the first sample image belonging to "Type k car". k .

[0059] The first image classification model can be obtained by iteratively training an initial image classification model beforehand.

[0060] It should be emphasized that the initial image classification model, the first image classification model, the second image classification model, and the third image classification model have the same structure in this specification.

[0061] For example, if the initial image classification model includes N sub-modules, then the first image classification model, the second image classification model, and the third image classification model all include N sub-modules, and the connection relationship of each sub-module is the same as the connection relationship of each sub-module in the initial image classification model.

[0062] The initial image classification model can include N sub-modules. The difference between the initial image classification model and the first image classification model is that the training parameters in each sub-module of the initial image classification model take pre-configured initial values, while the training parameters in each sub-module of the first image classification model take the corresponding values ​​obtained after model training.

[0063] In one specific implementation, the first sample image carries a target classification label; the target classification label is one of a variety of preset classification labels; the first classification prediction result includes: the first prediction probability value of the target classification label being each of the multiple preset classification labels; the second classification prediction result includes: the second prediction probability value of the target classification label being each of the multiple preset classification labels.

[0064] The target classification label can be a label used to characterize the true classification of the first sample image. Taking a car image as an example, the first sample image can include various preset classification labels such as "first type of car", "second type of car", ... "kth type of car". In a set of car images, if the car in car image 1 belongs to the fourth type of car, then car image 1 carries the target classification label "fourth type of car"; if the car in car image 2 belongs to the first type of car, then car image 2 carries the target classification label "first type of car"; if the car in car image 3 belongs to the fifth type of car, then car image 3 carries the target classification label "fifth type of car", and so on.

[0065] The first classification prediction result includes the first predicted probability value of the target classification label being each of the various preset classification labels. Taking the first sample image as a trademark image as an example, the various preset classification labels may include "genuine trademark", "counterfeit trademark" and "other". Then, by inputting the trademark image into the first image classification model for classification prediction processing, the first predicted probability value of the target classification label being "genuine trademark", the first predicted probability value of the target classification label being "counterfeit trademark", and the first predicted probability value of the target classification label being "other" can be obtained.

[0066] The second classification prediction result includes the second predicted probability value of the target classification label being each of the various preset classification labels. Taking the first sample image as a face image as an example, the various preset classification labels may include "real face", "face swap", and "generated face". Then, by inputting the face image into the third image classification model for classification prediction processing, we can obtain the second predicted probability value of the target classification label of the face image being "real face", the second predicted probability value of the target classification label of the face image being "face swap", and the second predicted probability value of the target classification label of the trademark image being "generated face".

[0067] The definitions of the first and second classification prediction results are similar, but they are generated in different ways. The first classification prediction result is obtained by inputting the first sample image carrying the target classification label into the first image classification model; the second classification prediction result is obtained by inputting the first sample image carrying the target classification label into the third image classification model.

[0068] In one specific implementation, the first sample image is occluded based on the first classification prediction result of the first sample image to obtain a second sample image, including: determining the core feature region in the first sample image based on the target classification label and the first classification prediction result; generating a noisy occluded image based on the brightness information value of each pixel in the core feature region; and replacing the sub-image corresponding to the core feature region in the first sample image with the noisy occluded image.

[0069] The first sample image may include multiple pixels, and each pixel in the first sample image can be regarded as a pixel point. The brightness information of each pixel in the first sample image is a known value. For example, the brightness information of each pixel can be represented by any natural number in the range [0, 255].

[0070] For example, the first sample image includes 2x2 pixels, and the brightness information of this first sample image can be represented by a vector. It means, a 11 =122, a 12=135, a 21 =110, a 22 =40.

[0071] The first sample image can be a multi-channel image, such as an RGB three-channel image. It can be equivalently regarded as the first sample image including the sample image corresponding to each of the three channels.

[0072] Based on the target classification label and the first classification prediction result, the core feature regions in the first sample image are determined. This can include: generating an activation map for each channel based on the target classification label and the first classification prediction result; and determining the core feature regions corresponding to each channel in the first sample image based on the activation map for each channel. An activation map is a category response map that can be generated from a classification model, which can be an image classification model or a classification model used to classify other objects. Activation maps generated based on image classification models can be used to roughly locate discriminative object regions in an image based on the image category label and annotation information.

[0073] An activation map may include multiple points. For example, the response value of each point in the activation map can be determined using the brightness information of that point. The response value of each point in the activation map can reflect a feature of each pixel in the corresponding input image, and there is a mapping relationship between the response value of each point and that feature of each pixel.

[0074] Based on the activation map of each channel, the core feature region corresponding to each channel in the first sample image is determined, which may include: for each channel of the first sample image, based on the brightness information of each point in the activation map, determining the first response point with the largest brightness information and the second response point with the largest brightness information outside the first response point; generating the corresponding first region based on the first response point; generating the corresponding second region based on the second response point; and determining the union of the first region and the second region as the core feature region corresponding to the activation map.

[0075] In the activation map of channel i of the first sample image, if the value of the brightness information of any point is greater than a preset brightness threshold, it indicates that the point's location in the activation map belongs to a strongly activated region. In the activation map, points with higher brightness information have higher response values. There can be a positional mapping relationship between each point in the activation map of channel i and each pixel in the sample image of channel i in the first sample image. For example, if point P1 in the activation map has a positional mapping relationship with pixel P2 in the sample image, and the higher the brightness information value of P1, the higher the response value of P1, and therefore, P2, which has a mapping relationship with P1, is more likely to be a source of significant features.

[0076] For each channel of the first sample image, based on the brightness information of each point in the activation map, determine the first response point with the largest brightness information in the activation map and the second response point with the largest brightness information outside the first response point. Alternatively, for each channel of the first sample image, based on the response values ​​of each point in the activation map, determine the first response point with the largest response value in the activation map and the second response point with the largest response value outside the first response point.

[0077] The first region is generated based on the first response point, which can be a first random region generated according to a preset generation rule with the first response point as the center; the second region is generated based on the second response point, which can be a second random region generated according to a preset generation rule with the second response point as the center.

[0078] The shapes of the first and second random regions can be the same or different. The sizes of the first and second random regions can be the same or different.

[0079] For example, both the first random region and the second random region can be rectangular in shape.

[0080] The first and second random rectangles may or may not overlap.

[0081] In another implementation, the first region is generated based on the first response point, or multiple sub-regions are pre-divided in the activation graph, and the sub-region to which the first response point belongs is determined as the first region; the second region is generated based on the second response point, or multiple sub-regions are pre-divided in the activation graph, and the sub-region to which the second response point belongs is determined as the second region.

[0082] Determining the union of the first region and the second region as the core feature region corresponding to the activation map can be achieved by determining the union of the first random region and the second random region as the core feature region corresponding to the activation map; the union of the first random region and the second random region includes the overlapping region and the non-overlapping region of the first random region and the second random region.

[0083] When the first sample image input to the first image classification model is a multi-channel image, it can be assumed that the channels of the multi-channel image are coupled with each other, that is, the activation maps of each channel are similar. In this case, only one activation map needs to be generated, and the core feature region corresponding to the first sample image is determined based on the activation map.

[0084] When the first sample image input to the first image classification model is a multi-channel image, the first image classification model can also determine whether there is coupling between any two channels.

[0085] Given that the first image classification model determines whether there is coupling between any two channels, the first sample image of the multi-channel is input into the first image classification model to generate an activation map for each channel.

[0086] For example, image I can be an RGB three-channel image with dimensions (H, W, C), where H represents the image height, W represents the image width, and C = 3, representing the number of image channels. By calculating an activation map for each of the three channels of image I, a total of three (H, W, 1) activation maps can be obtained.

[0087] When the coupling between any two channels is determined by the first image classification model, if the first image classification model determines that two channels are coupled, then the brightness information of pixels at the same position in the activation maps of the two coupled channels is similar, and the activation maps will respond at similar positions. Consequently, the positions of the core feature regions corresponding to each channel determined based on the two coupled activation maps are similar. If the first image classification model determines that the coupling between two channels is low, then the brightness information of pixels at the same position in the activation maps of the two uncoupled channels is significantly different, and the activation maps of the two channels respond at different positions. Consequently, the positions of the core feature regions corresponding to each channel determined based on the two uncoupled activation maps are significantly different.

[0088] Among them, pixels located at the same position have similar brightness information, which can be that the difference in brightness information between two pixels located at the same position is less than or equal to a preset difference threshold; two activation maps respond at similar positions, which can be that the overlap between the response regions determined by the maximum response point in the two activation maps is greater than a preset overlap threshold; the core feature regions corresponding to each channel are similar in position, which can be that the overlap between the core feature regions in the activation maps of each channel is greater than a preset overlap threshold.

[0089] In another embodiment, the size parameters and shape of the target response region can be preset. Then, based on the activation map of each channel, the core feature region corresponding to each channel in the first sample image can be determined. This can include: determining a first region based on the brightness information of each point in the activation map; the first region is the target response region with the largest average brightness information of each point; determining a second region based on the brightness information of each point in the activation map; the second region is the target response region with the largest average brightness information of each point other than the first region.

[0090] Based on the brightness information value of each pixel in the core feature region of each channel of the first sample image, a noise occlusion image corresponding to each channel can be generated. After obtaining the noise occlusion image corresponding to each channel, the sub-image corresponding to the core feature region of each channel can be replaced with the noise occlusion image corresponding to each channel.

[0091] In one specific implementation, a noise occlusion image is generated based on the brightness information value of each pixel in the core feature region, including: calculating the average brightness and brightness variance of the core feature region based on the brightness information value of each pixel in the core feature region; generating a Gaussian noise image based on the size parameter of the core feature region, the average brightness and brightness variance of the core feature region, and determining the Gaussian noise image as the noise occlusion image.

[0092] The core feature region can include multiple pixels. By averaging the brightness information values ​​of each pixel in the core feature region, the average brightness of the core feature region can be calculated; by calculating the variance of the brightness information values ​​of each pixel in the core feature region, the variance of the brightness of the core feature region can be calculated.

[0093] Based on the size parameters of the core feature region, the average brightness and brightness variance of the corresponding core feature region, a Gaussian noise image with a size matching the core feature region can be generated.

[0094] The reason for generating Gaussian noise images based on the average brightness and variance of the core feature regions is as follows: In order to extend the technical solution of occluding salient features in the first sample image with Gaussian noise images to occluding salient features in the feature map group with Gaussian noise, since many image classification models use normalization functions (such as batch normalization, group normalization, etc.), these normalization functions will consider the mean and variance of the feature map group. Therefore, generating Gaussian noise images based on the average brightness and variance of the core feature regions can avoid the negative impact of noise on the mean and variance of the feature map group.

[0095] Step S104: Input the second sample image into the first image classification model for iterative training to obtain the second image classification model. In specific implementation, the first image classification model can also be configured such that the output of the first submodule is the input of the regularization loss function. That is, the features in the feature map group output by the first submodule are constrained by the regularization loss function. The purpose is to fully mine from the feature map group after occlusion multiplication, obtain different features, and reduce redundant features.

[0096] Step S106: Perform the step of generating intermediate image classification models N-1 times, and determine the intermediate image classification model generated in the N-1th time as the target image classification model; wherein, the step of generating intermediate image classification models in the i-th time includes: freezing the parameters of the i-th sub-module in the second image classification model to obtain the third image classification model; inputting the first sample image into the third image classification model for classification prediction processing to obtain the second classification prediction result of the first sample image; determining the occluded region corresponding to the feature map group output by the i-th sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain the intermediate image classification model; when i is less than N-1, the intermediate image classification model is used as the second image classification model corresponding to the step of generating intermediate image classification models in the (i+1)-th time.

[0097] i can be any natural number greater than or equal to 1 and less than or equal to N-1.

[0098] When i is less than N-1, the intermediate image classification model is used as the second image classification model in the (i+1)th step of generating the intermediate image classification model; when i is equal to N-1, the intermediate image classification model is determined as the target image classification model.

[0099] It should be emphasized that the structures of the first image classification model, the second image classification model, the third image classification model, and the intermediate image classification model in this specification are completely identical. Therefore, the i-th sub-module in the first image classification model, the second image classification model, the third image classification model, and the intermediate image classification model refers to the same sub-module in terms of structure.

[0100] Freezing the parameters of the i-th submodule can be done by fixing the model parameters to be trained in the i-th submodule. After freezing the parameters of the i-th submodule, the values ​​of the parameters to be trained in the i-th submodule will not change no matter how many times training is performed.

[0101] The first sample image is input into the third image classification model for classification prediction, and the second classification prediction result of the first sample image is obtained.

[0102] The first sample image carries a target classification label; the target classification label is one of several preset classification labels; the second classification prediction result includes: the second predicted probability value of the target classification label being each of the several preset classification labels. Taking the first sample image as a car image as an example, the several preset classification labels may include "first type of car", "second type of car", ... "kth type of car". k can be any natural number greater than 1. In a set of car images, if the car in car image 1 belongs to the fourth type of car, then car image 1 carries the target classification label "fourth type of car"; if the car in car image 2 belongs to the first type of car, then car image 2 carries the target classification label "first type of car"; if the car in car image 3 belongs to the fifth type of car, then car image 3 carries the target classification label "fifth type of car", and so on. For example, by inputting car image 2 into the third image classification model for classification prediction processing, the second predicted probability value of car image 2 carrying the target classification label "first type of car", the second predicted probability value of car image 2 carrying the target classification label "second type of car", ... the second predicted probability value of car image 2 carrying the target classification label "kth type of car" can be obtained.

[0103] The second image classification model can include N sub-modules. The first sub-module can be the first sub-module among the N sub-modules. The second sub-module can be the next sub-module after the first sub-module according to the connection relationship of each sub-module. The third sub-module can be the next sub-module after the second sub-module according to the connection relationship of each sub-module. The i-th sub-module is the next sub-module after the (i-1)-th sub-module according to the connection relationship of each sub-module. The N-th sub-module can be the next sub-module after the (N-1)-th sub-module according to the connection relationship of each sub-module.

[0104] There are no one or more submodules between the i-th submodule and the (i-1)-th submodule, but there may be one or more data processing layers.

[0105] In one implementation, the output of the (i-1)th submodule can be the input of the ith submodule; in another implementation, the output of the (i-1)th submodule can be the input of data processing layer j, and the output of data processing layer j can be the input of the ith submodule; in yet another implementation, the output of the (i-1)th submodule can be the input of a data processing module composed of multiple data processing layers, and the output of the data processing module can be the input of the ith submodule. j can be any natural number.

[0106] After step S104 is executed, the first step of generating an intermediate image classification model is performed, including: freezing the parameters of the first sub-module in the second image classification model to obtain a third image classification model; inputting the first sample image into the third image classification model for classification prediction processing to obtain a second classification prediction result for the first sample image; determining the occluded region corresponding to the feature map group output by the first sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model; when 1 is less than N-1, the intermediate image classification model is used as the corresponding second image classification model in the second step of generating an intermediate image classification model.

[0107] In the process of freezing the parameters of the first submodule in the second image classification model to obtain the third image classification model, the first submodule, which comprises N submodules, is in a parameter-frozen state, and the values ​​of the training parameters in the first submodule no longer change. In the step "inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model," the first submodule does not actually participate in the iterative training, while the submodules following the first submodule can participate in iterative training.

[0108] After performing the first step of generating an intermediate image classification model, the second step of generating an intermediate image classification model is performed, including: freezing the parameters of the second sub-module in the second image classification model to obtain a third image classification model; inputting the first sample image into the third image classification model for classification prediction processing to obtain a second classification prediction result for the first sample image; determining the occluded region corresponding to the feature map group output by the second sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model; when 2 is less than N-1, the intermediate image classification model is used as the corresponding second image classification model in the third step of generating an intermediate image classification model.

[0109] In the process of freezing the parameters of the second sub-module in the second image classification model to obtain the third image classification model, the first and second sub-modules in this third model are in a parameter-frozen state, and the values ​​of the training parameters in the first and second sub-modules no longer change. In the step of "inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model", the first and second sub-modules do not actually participate in iterative training, while the sub-modules following the second sub-module can participate in iterative training.

[0110] After performing the second step of generating an intermediate image classification model, the third step of generating an intermediate image classification model is performed, including: freezing the parameters of the third sub-module in the second image classification model to obtain a third image classification model; inputting the first sample image into the third image classification model for classification prediction processing to obtain a second classification prediction result for the first sample image; determining the occluded region corresponding to the feature map group output by the third sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model; when 3 is less than N-1, the intermediate image classification model is used as the corresponding second image classification model in the fourth step of generating an intermediate image classification model.

[0111] In the process of freezing the parameters of the third sub-module in the second image classification model to obtain the third image classification model, the first, second, and third sub-modules in this third image classification model are all in a parameter-frozen state, and the values ​​of the training parameters in the first, second, and third sub-modules no longer change. In the step of "inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model", the first, second, and third sub-modules do not actually participate in iterative training, while the sub-modules after the third sub-module can participate in iterative training.

[0112] ...

[0113] After performing the (i-1)th step of generating the intermediate image classification model, the ith step of generating the intermediate image classification model is performed, including: freezing the parameters of the ith sub-module in the second image classification model to obtain the third image classification model; inputting the first sample image into the third image classification model for classification prediction processing to obtain the second classification prediction result of the first sample image; determining the occluded region corresponding to the feature map group output by the ith sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain the intermediate image classification model; when i is less than N-1, the intermediate image classification model is used as the corresponding second image classification model in the (i+1)th step of generating the intermediate image classification model.

[0114] In the process of freezing the parameters of the i-th sub-module in the second image classification model to obtain the third image classification model, the first, second, ..., i-th sub-modules in this third image classification model are all in a parameter-frozen state, and the values ​​of the training parameters in the first, second, ..., i-th sub-modules no longer change. In the step of "inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model", the first, second, ..., i-th sub-modules do not actually participate in iterative training, while the sub-modules after the i-th sub-module can participate in iterative training.

[0115] ...

[0116] After performing the step of generating the intermediate image classification model for the N-2th time, the step of generating the intermediate image classification model for the N-1th time is performed, including: freezing the parameters of the N-1th sub-module in the second image classification model to obtain the third image classification model; inputting the first sample image into the third image classification model for classification prediction processing to obtain the second classification prediction result of the first sample image; determining the region to be occluded in the feature map group output by the N-1th sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the region to be occluded to obtain the intermediate image classification model, and determining the intermediate image classification model generated for the N-1th time as the target image classification model.

[0117] In this process, after freezing the parameters of the (N-1)th submodule in the second image classification model to obtain the third image classification model, all submodules in the third image classification model except for the Nth submodule are in a parameter-frozen state. The values ​​of the training parameters in the 1st, 2nd, ..., (N-1)th submodules no longer change. In the step "inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model", the 1st, 2nd, ..., (N-1)th submodules do not actually participate in iterative training, while the Nth submodule, which is located after the (N-1)th submodule, can participate in iterative training.

[0118] By performing N-1 steps to generate intermediate image classification models, where the parameters of the i-th sub-module in the second image classification model are frozen during the i-th step of generating the intermediate image classification model, layer-by-layer feature mining can be achieved.

[0119] First, fix the first submodule in the second image classification model. Use gradient activation to locate the activation regions corresponding to salient features in the feature map set output by the first submodule. Apply a mask to these activation regions, forcing the second submodule, which follows the first submodule in the connection order, to extract secondary features from the feature map set output by the first submodule, excluding the activation regions. Second, fix the second submodule in the second image classification model. Then, both the first and second submodules are in a parameter-frozen state. Use gradient activation to locate the activation regions corresponding to salient features in the feature map set output by the second submodule. Apply a mask to these activation regions, forcing the second submodule, which follows the first submodule in the connection order, to extract secondary features from the feature map set output by the first submodule. The third submodule extracts secondary features from the feature map output by the second submodule, excluding the activation regions… Then, the (N-1)th submodule in the second image classification model is fixed, and the first, second…N-1th submodules are all in a parameter-frozen state. Gradient activation is used to locate the activation regions corresponding to the salient features in the feature map output by the (N-1)th submodule. By masking the activation regions, the Nth submodule, which follows the (N-1)th submodule in the connection order of the submodules, is forced to extract secondary features from the feature map output by the (N-1)th submodule, excluding the activation regions. This allows each submodule in the second image classification model, except for the first submodule, to extract rich features from the previous layer. Furthermore, by using… Figure 1 The training method of the image classification model shown can obtain a relatively accurate image classification prediction result even when the image to be classified is disturbed, thus improving the robustness of the target image classification model.

[0120] For example, in face image detection applications, a target image classification model can be used to detect whether a face image is forged. In practice, the face image input to the target image classification model may be subject to some interference, such as the eyes being obscured by sunglasses, or the image being limited by angle and lighting, resulting in lower clarity in a local area of ​​the left side of the face, etc. The target image classification model, in the process of detecting whether a face image is forged, can not rely solely on a small number of salient features, but rather utilize the characteristic that each submodule can extract rich features from the previous layer, to obtain a more accurate prediction of whether a face image is forged.

[0121] With the rapid development of machine learning technology, techniques for forging images or videos using deep learning have become increasingly sophisticated, producing highly realistic results that are becoming increasingly difficult for the naked eye to distinguish from genuine facial images. This poses significant risks to users of facial images in terms of security, finances, and reputation. Accurately detecting forged facial images through target image classification models can easily identify fake faces that are difficult to distinguish with the naked eye, thus reducing the risks associated with forged facial images.

[0122] For example, in trademark image detection applications, target image classification models can be used to authenticate trademarks. In practice, the trademark images input to the target image classification model may be subject to interference, such as the image being captured from a distance, resulting in low image clarity, or localized blurring due to glare. Target image classification models, in the process of authenticating trademarks, can rely not only on a few salient features but also leverage the fact that each submodule can extract rich features from the previous layer, thus obtaining more accurate predictions of trademark authenticity.

[0123] For example, in the application scenario of car image detection, a target image classification model can be used to identify the car model in an image. In practice, the car image input to the target image classification model may be subject to interference. For instance, the area of ​​the car to be identified on the road may be obscured by other vehicles, or visibility may be low due to weather conditions, resulting in poor image quality. The target image classification model, in the process of identifying the car model in an image, can not rely solely on a few salient features, but rather leverage the characteristic that each submodule can extract rich features from the previous layer, to obtain a more accurate prediction of the car model in the image.

[0124] In one specific implementation, the feature map group includes the feature map of each channel; based on the second classification prediction result, the region to be occluded corresponding to the feature map group output by the i-th sub-module is determined, including: based on the second classification prediction result, the core feature region corresponding to each channel in the feature map group is determined, and the core feature region corresponding to each channel in the feature map group is determined as the region to be occluded corresponding to each channel in the feature map group.

[0125] In specific implementation, the feature map group output by the i-th submodule can be a multi-channel feature map group. The number of feature maps included in the feature map group is equal to the number of channels. That is, each channel can correspond to a feature map, and further, each channel can correspond to a core feature region.

[0126] The core feature region can be the region in the feature map of the corresponding channel used to represent significant features.

[0127] For multi-channel feature map sets, the channels are likely to be uncoupled. In this case, the core feature regions in the various feature maps included in the feature map set may be located in different positions. To ensure that the salient features in the feature map set received by the next submodule are masked, forcing the next submodule to mine secondary features other than the salient features, the core feature region of each feature map can be determined as the occlusion region corresponding to each channel.

[0128] For example, the feature map set includes four feature maps: x1, x2, x3, and x4. x1 corresponds to channel I1; x2 corresponds to channel I2; x3 corresponds to channel I3; and x4 corresponds to channel I4. Based on the second classification prediction results, the core feature region s1 can be determined in x1; the core feature region s2 can be determined in x2; the core feature region s3 can be determined in x3; and the core feature region s4 can be determined in x4. s1 is designated as the occluded region corresponding to channel I1; s2 is designated as the occluded region corresponding to channel I2; s3 is designated as the occluded region corresponding to channel I3; and s4 is designated as the occluded region corresponding to channel I4.

[0129] It is important to emphasize that the number of sub-modules in a third-party image classification model can be large. If only the first sample image input to the third-party image classification model is occluded, the multiplicative effect of the occlusion on the input image will gradually weaken as the number of connected sub-modules increases. For example, when the third-party image classification model includes only two sub-modules, the occluded first sample image has a significant impact on these two sub-modules. When the third-party image classification model includes ten sub-modules, the occluded first sample image has a significant impact on the first few sub-modules among these ten sub-modules, while the impact on the later-ranked sub-modules gradually weakens. Therefore, the later-ranked sub-modules may still output redundant features, making it difficult to guarantee the overall feature extraction performance of the third-party image classification model.

[0130] In this embodiment, by determining the core feature region corresponding to each channel in the feature map group, the core feature region corresponding to each channel in the feature map group is determined as the occlusion region corresponding to each channel in the feature map group. Then, during the iterative training process, the feature maps of each channel output by the feature map group are occluded. This can achieve occlusion multiplication of the feature map groups output by each sub-module. Even if the number of sub-modules included in the third image classification model is large, the feature extraction potential of each sub-module can be fully explored, so that each sub-module except the first sub-module can extract rich features from the previous sub-module.

[0131] Furthermore, the core feature region corresponding to each channel in the feature map group is determined, and the core feature region corresponding to each channel in the feature map group is determined as the occluded region corresponding to each channel in the feature map group. The occluded region in the feature map group is not randomly selected, nor is it a partial feature map of each channel in the feature map group output by the i-th sub-module. Instead, the core feature region corresponding to each channel in the feature map group is selected based on the gradient, that is, the local region where the salient feature in the feature map group is located. After occlusion, it is difficult for the sub-modules after the i-th sub-module to use the salient feature to predict the image classification result. Therefore, the (i+1)-th sub-module after the i-th sub-module is forced to mine secondary features from other unoccluded regions in the feature map group.

[0132] Based on the second classification prediction results, the core feature regions corresponding to each channel in the feature map group are determined. This can be done by generating an activation map for each channel in the feature map group based on the second classification prediction results, and then determining the core feature regions corresponding to each channel in the feature map group based on the activation map of each channel.

[0133] For example, the feature map group F can be a 256-channel tensor with dimensions (H, W, 256), where H represents the tensor height, W represents the tensor width, and 256 represents the number of tensor channels. By calculating an activation map for each of the 256 channels of the feature map group F, a total of 256 (H, W, 1) activation maps can be obtained. After obtaining the 256 activation maps, the core feature region corresponding to each channel in the feature map group can be determined based on the activation map of each channel.

[0134] In one specific implementation, the core feature region corresponding to each channel in the feature map group is determined based on the second classification prediction result, including: in the second classification prediction result, determining the second prediction probability value that matches the target classification label as the first probability value; in the second classification prediction result, determining the second prediction probability value with the highest value other than the first probability value as the second probability value; performing gradient calculation processing on the brightness information of the feature map of each channel in the feature map group based on the first probability value and the second probability value to obtain the activation map corresponding to each channel in the feature map group; and determining the core feature region corresponding to each channel in the feature map group based on the activation map corresponding to each channel in the feature map group.

[0135] In the second classification prediction results, the second predicted probability value that matches the target classification label is determined as the first probability value.

[0136] For example, the target classification label is one of three preset classification labels; the second classification prediction result includes: the second predicted probability value that the target classification label is each of the three preset classification labels. In practice, the second predicted response value for each preset classification label can be obtained first, and then normalized to obtain the second predicted probability value corresponding to each second predicted response value. Furthermore, among these three second predicted probability values, the second predicted probability value that matches the target classification label is determined as the first probability value, and the second predicted probability value with the highest value other than the first probability value is determined as the second probability value.

[0137] In another implementation, the second predicted probability value can be replaced by a second predicted response value. That is, the second classification prediction result includes multiple second predicted response values; in the second classification prediction result, the second predicted response value that matches the target classification label is determined as the first response value; in the second classification prediction result, the second predicted response value with the highest value other than the first response value is determined as the second response value; based on the first response value and the second response value, gradient calculation processing is performed on the brightness information of the feature map of each channel in the feature map group to obtain the activation map corresponding to each channel in the feature map group; based on the activation map corresponding to each channel in the feature map group, the core feature region corresponding to each channel in the feature map group is determined.

[0138] In the above embodiments, the relationship between the second predicted probability value and the second predicted response value is as follows:

[0139] The first sample image is input into a third image classification model for classification prediction, resulting in a second classification prediction result for the first sample image. This second classification prediction result may include multiple second prediction response values. Normalization is then applied to each second prediction response value to obtain a corresponding second prediction probability value.

[0140] The second predicted probability value can be the probability value obtained after normalizing the second predicted response value. Therefore, in the embodiment where the second classification prediction result includes multiple second predicted response values ​​and the embodiment where the second classification prediction result includes multiple second predicted probability values, the technical concept of how to determine the core feature region corresponding to each channel in the feature map group based on the second classification prediction result is similar.

[0141] For example, the second classification prediction result includes three second predicted response values, namely: Target category label g i =0, then the second predicted response value matches the target classification label. That is, the first response value is 3.4, which is the maximum predicted response outside the target classification label. That is, the second response value is 1.9.

[0142] P G P is used to represent the second predicted response value, i.e., the first response value, corresponding to the target classification label carried by the first sample image in the second classification prediction result. max\G Used to represent F in the second classification prediction result G The second predicted response value, which is the largest value other than the first predicted response value, is the second response value.

[0143] Based on the first and second response values, gradient calculation is performed on the brightness information of the feature map of each channel in the feature map group to obtain the activation map corresponding to each channel in the feature map group. Alternatively, the difference between the first and second response values ​​can be calculated to obtain the first difference |P|. G -P max\G Then, calculate the product of the first difference and the gradient information obtained from the brightness information of each pixel in the feature map of each channel in the feature map group. To obtain the activation map for each channel.

[0144] Used to represent channel I in a feature map group C The gradient information is obtained from the brightness information of each pixel in the corresponding feature map. For example, the number of channels corresponding to a feature map group can be 256, then channel I... C It can be channel I1, channel I2... channel I 256 Any one of them.

[0145] For example, for channel I1, according to the first response value P G and the second response value P max\G The first difference |P can be calculated. G -P max\G Then, calculate the first difference |P G -P max\G |The product of the gradient information obtained from the brightness information of each pixel in the feature map of channel I1. product To generate the activation map of channel I1.

[0146] In one specific implementation, the core feature region corresponding to each channel in the feature map group is determined based on the activation map corresponding to each channel in the feature map group. This includes: for each channel in the feature map group, determining the first response point with the largest brightness information and the second response point with the largest brightness information outside the first response point in the activation map based on the brightness information of each point in the activation map; generating a corresponding first region based on the first response point; generating a corresponding second region based on the second response point; and determining the union of the first region and the second region as the core feature region corresponding to the activation map.

[0147] An activation map can include multiple points. The response value of each point in the activation map reflects a feature of a pixel in the corresponding feature map, and there is a mapping relationship between the response value of each point and that feature of each pixel. A multi-channel feature map group includes multiple feature maps, each corresponding to one channel.

[0148] In the activation map of channel i in the feature map group, if the value of the brightness information at any point is greater than a preset brightness threshold, it indicates that the point's location in the activation map belongs to a strongly activated region. In the activation map, points with higher brightness information have higher response values. There can be a positional mapping relationship between each point in the activation map of channel i and each pixel in the feature map of channel i in the feature map group. For example, if point P1 in the activation map has a positional mapping relationship with pixel P2 in the sample image, and the higher the brightness information value of P1, the higher the response value of P1, and therefore, P2, which has a mapping relationship with P1, is more likely to be a source of significant features.

[0149] For each channel in the feature map group, based on the brightness information of each point in the activation map, determine the first response point with the largest brightness information in the activation map and the second response point with the largest brightness information outside the first response point. Alternatively, for each channel in the feature map group, based on the response values ​​of each point in the activation map, determine the first response point with the largest response value in the activation map and the second response point with the largest response value outside the first response point.

[0150] The first region is generated based on the first response point, which can be a first random region generated according to a preset generation rule with the first response point as the center; the second region is generated based on the second response point, which can be a second random region generated according to a preset generation rule with the second response point as the center.

[0151] The shapes of the first and second random regions can be the same or different. The sizes of the first and second random regions can be the same or different.

[0152] For example, both the first random region and the second random region can be rectangular in shape.

[0153] The first and second random rectangles may or may not overlap.

[0154] For example, regarding the first random region, the preset generation rule may include one or more of the following sub-rules:

[0155] (1) The shape of the first random region can be a rectangle.

[0156] (2) The ratio of the length of the first random region to the length of the activation image can be within a first preset ratio range, for example, between 10% and 50%, and the ratio of the width of the first random region to the width of the activation image can be within a second preset ratio range, for example, between 20% and 55%. The first preset ratio range and the second preset ratio range can be the same or different.

[0157] (3) The first random region can be centered on the first response point.

[0158] The preset generation rules for the second random region can be referenced from those for the first random region, and will not be repeated here.

[0159] In the preset generation rules, the generation method of the first random region can be the same as or different from that of the second random region.

[0160] In another implementation, the first region is generated based on the first response point, or multiple sub-regions are pre-divided in the activation graph, and the sub-region to which the first response point belongs is determined as the first region; the second region is generated based on the second response point, or multiple sub-regions are pre-divided in the activation graph, and the sub-region to which the second response point belongs is determined as the second region.

[0161] Determining the union of the first region and the second region as the core feature region corresponding to the activation map can be achieved by determining the union of the first random region and the second random region as the core feature region corresponding to the activation map; the union of the first random region and the second random region includes the overlapping region and the non-overlapping region of the first random region and the second random region.

[0162] In one specific implementation, the third image classification model includes a sample generation sub-model constructed based on the first i sub-modules and a training sub-model constructed based on the last Ni sub-modules; the sample generation sub-model is used to generate a third sample image based on the occluded region corresponding to the feature map group; the training sub-model is used to perform iterative training based on the third sample image to obtain an intermediate image classification model.

[0163] The third image classification model includes N sub-modules. The first sub-module can be the first sub-module among the N sub-modules. The second sub-module can be the next sub-module after the first sub-module according to the connection relationship of each sub-module. The third sub-module can be the next sub-module after the second sub-module according to the connection relationship of each sub-module. The i-th sub-module is the next sub-module after the (i-1)-th sub-module according to the connection relationship of each sub-module. The N-th sub-module can be the next sub-module after the (N-1)-th sub-module according to the connection relationship of each sub-module.

[0164] In the step of “inputting the first sample image into the third image classification model to perform iterative training based on the region to be occluded to obtain an intermediate image classification model”, the first i sub-modules that have undergone parameter freezing are equivalent to not participating in model training during the model training process. The relationship between the input and output of each sub-module that has undergone parameter freezing can be regarded as fixed.

[0165] For example, during the first generation of the intermediate image classification model, the parameters of the first sub-module in the second image classification model are frozen to obtain the third image classification model. The first sample image is input into the third image classification model for classification prediction to obtain the second classification prediction result of the first sample image. Based on the second classification prediction result, the region to be occluded corresponding to the feature map group output by the i-th sub-module is determined. The first sample image is input into the third image classification model for iterative training based on the region to be occluded to obtain the intermediate image classification model. In the process of inputting the first sample image into the third image classification model for iterative training based on the region to be occluded to obtain the intermediate image classification model, the input image of the third image classification model is the first sample image in each training session. Therefore, the input image of the first sub-module, which has undergone parameter freezing, remains unchanged in each training session. Since the first sub-module has undergone parameter freezing, the parameters to be trained are not adjusted after each training session, so the feature map group output by the first sub-module remains unchanged in each training session.

[0166] For example, during the second generation of the intermediate image classification model, the parameters of the second sub-module in the second image classification model are frozen to obtain the third image classification model. The first sample image is input into the third image classification model for classification prediction, resulting in the second classification prediction result for the first sample image. Based on the second classification prediction result, the region to be occluded is determined from the feature map group output by the second sub-module. The first sample image is then input into the third image classification model for iterative training based on the region to be occluded, resulting in the intermediate image classification model. The input of the second sub-module is the output of the first sub-module. In the process of inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model, the input image of the third image classification model is the first sample image during each training session. Therefore, the input image of the first sub-module, which has undergone parameter freezing, remains unchanged during each training session. Since the first sub-module has undergone parameter freezing, the parameters to be trained within it will not be adjusted after each training session. Therefore, the feature map set output by the first sub-module remains unchanged during each training session. Similarly, since the second sub-module has undergone parameter freezing, the parameters to be trained within it will not be adjusted after each training session. The feature map set input into the second sub-module remains unchanged during each training session. Therefore, the feature map set output by the second sub-module remains unchanged.

[0167] When performing the i-th step of generating an intermediate image classification model, the third image classification model may include a sample generation sub-model built based on the first i sub-modules and a training sub-model built based on the last Ni sub-modules.

[0168] When i is less than N-1, the sample generation sub-model can include the 1st sub-module, the 2nd sub-module, ..., the i-th sub-module. The sub-module to be trained can include the (i+1)-th sub-module, the (i+2)-th sub-module, ..., the N-th sub-module.

[0169] When i equals N-1, the sample generation sub-model can include the 1st sub-module, the 2nd sub-module, ..., the (N-1)th sub-module. The sub-module to be trained can include the Nth sub-module.

[0170] The sample generation sub-model is used to generate a third sample image based on the occluded region corresponding to the feature map group. Specifically, the sample generation sub-model can perform convolution calculations on the first sample image to obtain the feature map group corresponding to the first sample image, and then perform occlusion processing on the feature map group corresponding to the first sample image based on the occluded region corresponding to each channel of the feature map group to obtain the third sample image.

[0171] For example, when i=1, the sample generation sub-model may include a first sub-module. The first sub-module can be the first module in the third image classification model, meaning the input image to the third image classification model is equivalent to the input image to the first sub-module. The first sub-module can perform convolution calculations on the input first sample image to obtain a feature map set corresponding to the first sample image. The sample generation sub-model can also be used to occlude each feature map in the feature map set based on the occlusion region corresponding to each channel, obtaining an occluded feature map set, which is then determined as the third sample image.

[0172] For example, when i=2, the sample generation sub-model can include a first sub-module and a second sub-module, with the input of the second sub-module being the output of the first sub-module. The first sub-module can be the first module in the third image classification model, meaning the input image to the third image classification model is equivalent to the input image to the first sub-module. The first sub-module can perform convolution calculations on the input first sample image to obtain feature map group 1 corresponding to the first sample image. The second sub-module can perform convolution calculations on feature map group 1 to obtain feature map group 2 corresponding to feature map group 1. The sample generation sub-model can also be used to occlude each feature map in feature map group 2 based on the occlusion region corresponding to each channel, obtaining an occluded feature map group 2, which is then determined as the third sample image.

[0173] In specific implementation, based on the occluded area corresponding to each channel of the feature map group, the feature map group corresponding to the first sample image is occluded. This can be done by generating a noisy occlusion image for each channel of the feature map group according to the brightness information value of each pixel in the occluded area; and replacing the sub-image corresponding to the occluded area in the feature map with the noisy occlusion image.

[0174] In specific implementation, a noisy occlusion image is generated based on the brightness information value of each pixel in the area to be occluded, including: calculating the average brightness and brightness variance of the area to be occluded based on the brightness information value of each pixel in the area to be occluded; generating a Gaussian noise image based on the size parameters of the area to be occluded, the average brightness and brightness variance of the area to be occluded, and determining the Gaussian noise image as the noisy occlusion image.

[0175] The sub-model to be trained is used for iterative training based on the third sample image to obtain an intermediate image classification model. Specifically, the sub-model to be trained can be used for iterative training based on the third sample image to obtain a trained sub-module to be trained, which in turn can obtain an intermediate image classification model, which consists of a sample generation sub-model and a trained sub-module to be trained.

[0176] In one specific implementation, the loss function of the third image classification model is determined by the regularization loss function and the cross-entropy loss function; the regularization loss function is determined by the feature map of each channel in the feature map group output by the (i+1)th submodule; and the cross-entropy loss function is determined by the target classification label and the second classification prediction result.

[0177] During the step of generating the intermediate image classification model for the i-th time, in the third image classification model, the parameters of the i-th sub-module have been frozen, and the regularization loss function is determined by the feature map of each channel in the feature map group output by the (i+1)-th sub-module.

[0178] For example, the regularization loss function formula is as follows:

[0179]

[0180]

[0181] Among them, c i It is the feature map i, c corresponding to channel i. j It is the feature map j corresponding to channel j, where i is less than j, therefore c j With c i The channels are different. It is the probability distribution obtained by normalizing ci. It is C j The probability distribution is obtained after normalization. It is then calculated using ci. The process, and, through c j Seeking The process can be referred to formula (2) above. JSD (Jensen-Shannondivergence, JS divergence) is used to measure the similarity between two probability distributions. and Since the values ​​are the same, to avoid duplicate calculations, the constraint i < j is used, making the calculation... No further calculation is needed after the value. Furthermore, the calculations between feature map i and feature map i are unnecessary. min(c) is the pixel with the lowest brightness information in the feature map, c is the pixel in the feature map, and ∑ uv c uv Used to represent the sum of brightness information for each pixel in the feature map. L reg The regularized loss function used to represent the third image classification model.

[0182] In image classification models that include multiple sub-modules to be trained, such as CNN models, due to the large number of parameters to be trained, image classification models often extract multiple highly correlated features, resulting in redundancy. For example, the third sub-module outputs feature map group 3, which includes 256 feature maps. Some of these 256 feature maps have high similarity to each other, forming redundant features. Redundant features may cause the image classification model to overfit to the salient features of the samples during the model training phase, leading to poor generalization. That is, the image classification model performs well during the model training phase, but its performance is unsatisfactory when the model is tested.

[0183] By determining the regularization loss function based on the feature map of each channel in the feature map group output by the (i+1)th submodule, sufficient differences can be constrained between the feature maps in the feature map group output by the (i+1)th submodule, thereby reducing redundant features. In addition, the (i+1)th submodule can be forced to mine as many effective features as possible from the i-th submodule in order to achieve sufficient differences between the feature maps.

[0184] In specific implementation, during the first execution of the intermediate image classification model generation step, a regularization loss function is determined based on the feature map of each channel in the feature map group output by the second submodule. This constrains sufficient differences between the feature maps in the feature map group output by the second submodule, thereby reducing redundant features. It also forces the second submodule to extract as many effective features as possible from the first submodule to achieve sufficient differences between the feature maps. Similarly, during the second execution of the intermediate image classification model generation step, a regularization loss function is determined based on the feature map of each channel in the feature map group output by the third submodule. This constrains sufficient differences between the feature maps in the feature map group output by the third submodule, thereby reducing redundant features. It also forces the third submodule to extract as many effective features as possible from the first submodule to achieve sufficient differences between the feature maps. Sufficient differences are achieved by extracting as many effective features as possible from the second submodule… During the (N-1)th execution of the intermediate image classification model generation step, a regularization loss function is determined based on the feature map of each channel in the feature map group output by the Nth submodule. This constrains the existence of sufficient differences between the feature maps in the feature map group output by the Nth submodule, thereby reducing redundant features. It also forces the Nth submodule to extract as many effective features as possible from the N-1 submodule in order to achieve sufficient differences between the feature maps. In summary, during each execution of the intermediate image classification model generation step, the regularization loss function can be used to constrain the feature maps output by each submodule to be trained to be different, minimizing redundant features and forcing each submodule to extract more effective features from the previous submodule.

[0185] Furthermore, due to fewer redundant features, the accuracy of the prediction results obtained by the target image classification model is almost consistent in the model training stage, model testing stage, and when the model is put into application scenarios. This can avoid the situation where the model performs well during training but frequently makes mistakes when put into use, thus improving the generalization of the target image classification model.

[0186] In addition, in the process of passing through, such as Figure 1The training method of the image classification model shown can be used to obtain a target image classification model to perform image classification tasks. Each sub-module can extract rich features from the previous sub-module, so that the target image classification model can obtain relatively accurate image classification prediction results even when the image to be classified is disturbed, thus improving the robustness of the target image classification model.

[0187] For example, in face image detection applications, a target image classification model can be used to detect whether a face image is forged. In practice, the face image input to the target image classification model may be subject to some interference, such as the eyes being obscured by sunglasses, or the image being limited by angle and lighting, resulting in lower clarity in a local area of ​​the left side of the face, etc. The target image classification model, in the process of detecting whether a face image is forged, can not rely solely on a small number of salient features, but rather utilize the characteristic that each submodule can extract rich features from the previous layer, to obtain a more accurate prediction of whether a face image is forged.

[0188] The cross-entropy loss function can be referred to by the following formula (3):

[0189]

[0190] Among them, L CE The cross-entropy loss function is represented by c, which represents the variable used to iterate through the categories of each preset classification label, B represents the number of samples in the batch, and b represents the variable used to iterate through the number of samples in the batch B. y is used to represent the predicted probability value obtained after normalizing the predicted response value for predicting the target classification label of the b-th sample as c. bc This is used to indicate whether the target classification label of the b-th sample corresponds to category c. If so, then y bc =1; if not, then y bc =0.

[0191] For example, the first sample image includes a set of multi-channel images, the number of which is the number of batch samples.

[0192] The loss function of the third image classification model can be determined by the sum of the regularization loss function and the cross-entropy loss function, or it can be determined based on the adjustment coefficient of at least one loss function, the regularization loss function, and the cross-entropy loss function.

[0193] Figure 3 This is a schematic diagram illustrating a training method for an image classification model provided in an embodiment of this application.

[0194] The following can be combined Figure 3This section explains in detail the steps involved in generating an intermediate image classification model.

[0195] like Figure 3 As shown, in stage (a), the initial image classification model is trained in a conventional manner until convergence, resulting in the first image classification model. The input image 301, i.e., image I, of the initial image classification model can be represented by a rectangle. Each submodule 302 can be represented by a trapezoid. The feature map set 303 output by each submodule can be represented by a cuboid. The classification prediction result 304 output by the initial image classification model can be represented by multiple circles.

[0196] In stage (b), before model training, the input image of the first image classification model, i.e., image I, can be a multi-channel image, and the number of channels can be represented by C. This multi-channel image can be equivalent to C input images. Inputting image I into the first image classification model yields a first classification prediction result, which may include multiple first prediction response values.

[0197] In the first classification prediction result, the first predicted response value that matches the true classification label carried by image I is determined as the third response value; in the first classification prediction result, the first predicted response value with the highest value other than the first response value is determined as the fourth response value; based on the third response value and the fourth response, gradient calculation processing is performed on the brightness information of the input image of each channel in image I to obtain the activation map corresponding to each channel in image I; based on the activation map corresponding to each channel in image I, the core feature region corresponding to each channel in image I is determined.

[0198] In the above embodiments, the relationship between the first predicted probability value and the first predicted response value is as follows:

[0199] Image I is input into the first image classification model for classification prediction processing to obtain the first classification prediction result of image I. The first classification prediction result may include multiple first prediction response values. Normalization processing is performed on each first prediction response value to obtain the first prediction probability value corresponding to each first prediction response value.

[0200] The first predicted probability value can be the probability value obtained after normalizing the first predicted response value. Therefore, the technical concept of how to determine the core feature region corresponding to each channel in image I based on the first classification prediction result is similar to that in the embodiment where the first classification prediction result includes multiple first predicted response values ​​and the embodiment where the first classification prediction result includes multiple first predicted probability values.

[0201] In practice, the first predicted response value P at the location corresponding to the actual classification label G can be used as a reference. G The maximum response prediction P outside the true value max\GFor each channel I of image I c Calculate gradient The activation map corresponding to each channel Ic is obtained.

[0202] P G P is used to represent the first predicted response value, i.e., the third response value, corresponding to the true classification label G in the first classification prediction result. max\G Used to represent P in the first classification prediction result G The first predicted response value, which is the largest value outside of the four predicted values, is the fourth response value.

[0203] Channel I used to represent image I c The gradient information.

[0204] According to each channel I c The corresponding activation map occludes the k regions with the largest responses in image I, i.e., the occluded region 305, and fills it with Gaussian noise. For example, the number of k can be 2.

[0205] Additionally, a regularization loss function L can be added to the feature map set output by the first submodule. REG That is, in each training process, a regularized loss function is generated based on the feature map set output by the first submodule.

[0206] Furthermore, the first image classification model is iteratively trained based on the occluded image I to obtain the second image classification model. After each training iteration, a regularized loss function L can be used... REG The training parameters of each sub-module that has not undergone parameter freezing during the next training iteration are adjusted, i.e., the training parameters of each sub-module in the first image classification model.

[0207] In stage (c), the parameters of the first submodule in the second image classification model are frozen to obtain the third image classification model. The gradient is calculated for each feature map F in the feature map set output by the first submodule. The k regions with the highest response in each feature map F are masked based on the activation map and filled with Gaussian noise. Additionally, a regularization loss function is added to the feature map group output by the second submodule.

[0208] Image I is input into the third image classification model. The occlusion-processed feature map group output by the first submodule is used to iteratively train the third image classification model to obtain an intermediate image classification model. This intermediate image classification model is then determined as the second image classification model in stage (d).

[0209] After freezing the parameters of the first submodule, it can be regarded as using the feature map set output by the first submodule as samples to train the sub-model composed of each submodule after the first submodule.

[0210] Phase (d) is similar in concept to phase (c). While the parameters of the first submodule in the second image classification model are frozen, the parameters of the second submodule are also frozen to obtain the third image classification model. The gradient is calculated for each feature map F of the feature map group output by the second submodule. The k regions with the highest responses are masked from the activation map and filled with Gaussian noise. Additionally, a regularization loss function is added to the feature map set output by the third submodule.

[0211] Image I is input into the third image classification model. The occlusion-processed feature map group output by the second sub-module is used to iteratively train the third image classification model to obtain an intermediate image classification model. This intermediate image classification model is then determined as the second image classification model for the next stage.

[0212] In the case that the third image classification model includes N sub-modules, the subsequent stages can refer to stage (c) until all N sub-modules have been trained.

[0213] In such Figure 1In the illustrated embodiment, firstly, based on the first classification prediction result of the first sample image, the first sample image is occluded to obtain a second sample image; the first classification prediction result is obtained by inputting the first sample image into a first image classification model for classification prediction processing; the first image classification model includes N sub-modules; the input of the (i+1)th sub-module is determined by the output of the ith sub-module; secondly, the second sample image is input into the first image classification model for iterative training to obtain a second image classification model; then, the step of generating an intermediate image classification model N-1 times is executed, and the intermediate image classification model generated in the N-1th time is determined as the target image classification model; wherein, the step of generating the intermediate image model in the ith time is executed. The steps of the image classification model include: freezing the parameters of the i-th sub-module in the second image classification model to obtain the third image classification model; inputting the first sample image into the third image classification model for classification prediction to obtain the second classification prediction result of the first sample image; determining the occluded region corresponding to the feature map group output by the i-th sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain the intermediate image classification model; when i is less than N-1, using the intermediate image classification model as the corresponding second image classification model in the (i+1)-th step of generating the intermediate image classification model. Therefore, by occluding the first sample image based on the first classification prediction result, the salient features in the first sample image can be blocked, forcing each sub-module in the first image classification model to mine secondary features other than the salient features. Furthermore, by performing N-1 steps to generate intermediate image classification models, the parameters of the i-th sub-module are frozen during the i-th step of generating intermediate image classification models, which can fix the model parameters of the already trained i-th sub-module layer by layer. Based on the second classification prediction result, the feature map set output by the i-th sub-module is occluded, blocking the salient features in the feature map set output by the i-th sub-module. This forces the (i+1)-th sub-module following the i-th sub-module to mine secondary features other than the salient features in the feature map set output by the i-th sub-module. Thus, in the target image classification model, the first sub-module can mine rich features from the image input to the target image classification model, and each sub-module other than the first sub-module can mine rich features from the feature map set output by the previous sub-module. This improves the overall feature mining performance of the target image classification model and enhances its robustness and generalization.

[0214] Based on the same technical concept as the aforementioned image classification model training method embodiments, this application embodiment also provides a training method for an image classification model applied to the ResNet18 model. Figure 4 This is a schematic diagram illustrating a training method for a ResNet18 model provided in an embodiment of this application.

[0215] The ResNet18 model can be used in applications involving deepfake image detection. The following section combines... Figure 4 Let's take a closer look at the application scenarios of deepfake image detection.

[0216] The ResNet18 model is a common CNN model that can be used to determine whether a face image belongs to the category of {real face, face swap, or generated face}.

[0217] Among them, a face image belonging to a real face refers to a real face image that has not been modified and is taken from a real person's face; a face image belonging to a face swap refers to a face image that is a fake image obtained by fusing at least two images, i.e., a face swap image; a face image belonging to a generated face refers to an image that is generated by forgery technology, i.e., a generated face image.

[0218] Before training the ResNet18 model, data can be pre-built. Specifically, a large number of real face images can be collected; a large number of face-swapped images can be created using at least one of various face-swapping algorithms, APIs, and apps, such as Deepfakes and Faceswap; and a large number of generated face images can be created using at least one of various face generation algorithms, APIs, and apps. For example, the face generation algorithm could be StyleGAN2.

[0219] The three types of images mentioned above are preprocessed, and face alignment is performed using facial landmark detection to obtain an aligned face image database. The true category corresponding to each face image is recorded, and each face image is 224x224x3 in size. The ground truth labels can be denoted as {real face: 0; face swap: 1; generated face: 2}. In "224x224x3", 224x224 represents the size of the face image, i.e., the height x width value, and 3 represents the number of channels. For example, a face image can be an RGB three-channel image.

[0220] like Figure 4As shown, `image` represents the input image to the ResNet18 model. The ResNet18 model includes a first submodule 402, a pooling layer 404, a second submodule 406, a third submodule 408, a fourth submodule 410, a fifth submodule 412, a pooling layer 414, and a fully connected layer 416, connected sequentially. The output of the first submodule 402 is the input of the pooling layer 404; the output of the pooling layer 404 is the input of the second submodule 406; the output of the second submodule 406 is the input of the third submodule 408; the output of the third submodule 408 is the input of the fourth submodule 410; the output of the fourth submodule 410 is the input of the fifth submodule 412; the output of the fifth submodule 412 is the input of the pooling layer 414; and the output of the pooling layer 414 is the input of the fully connected layer 416.

[0221] like Figure 4 As shown, compared to the common ResNet18 model, Figure 4 The last fully connected layer (FC) of the ResNet18 model provided in the embodiment is modified to 512x3, where 512 represents the number of channels and 3 represents that the image classification result of the ResNet18 model includes 3 predicted values, which correspond to 3 preset image categories in the database, thus obtaining a 3-classification model.

[0222] Next, the ResNet18 model parameters were initialized using the ImageNet pre-trained model weights, and the parameters of the newly added fully connected (FC) layers were randomly initialized. Using the Adam optimizer, an initial learning rate of 0.001 was adopted, with the learning rate decaying according to cosine. Training was performed using batch sampling of B=16 until the ResNet18 model converged. ResNet18 model convergence is characterized by a smoothing of the loss function and a stable classification accuracy on the test set.

[0223] After initializing the ResNet18 model parameters and randomly initializing the parameters of the newly added FC layers, the following steps can be performed:

[0224] Step (1): Step (1) is equivalent to Figure 1 Steps S102 and S104 in the embodiments can be referred to the corresponding description sections.

[0225] For each input image I within a batch of sample sets i Perform the following processing separately:

[0226] Will I i Input the current model to obtain the current ResNet18 model for the input image I.i Three predicted response values This corresponds to three categories. For example:

[0227] Assume I i truth label g i =0, then the truth value g i Predicted response value at the corresponding location Truth value g i Maximum response prediction outside

[0228] This can be assumed to be Let me continue explaining.

[0229] For input image I i The activation map is calculated for each of the C channels.

[0230]

[0231] In formula (4), This refers to the input image I i The gradient information of the Cth channel, This refers to the input image I i The activation map of the Cth channel.

[0232] The activation map of channel c is subjected to a 3x3 mean filter to obtain a smoothed activation map.

[0233] The following is a detailed explanation of mean filtering: If the difference in brightness information between two adjacent pixels in the active image is greater than or equal to a preset threshold, it indicates that the brightness information of one of the pixels may be incorrect. Therefore, a filtering region of 3x3 pixels can be determined with either of the two pixels as the center, the average value of the brightness information of each pixel in the filtering region can be calculated, and the brightness information of the pixel at the center of the filtering region can be updated based on the average value.

[0234] Next, we can determine k regions to be occluded, and generate the occluded input image based on these k regions.

[0235] Generating an occluded input image based on k unoccluded regions can be achieved by determining the union of the k unoccluded regions as the target occlusion region, and then performing occlusion processing on the input image based on the target occlusion region. The following example, using k=2, illustrates the specific process of generating an occluded input image based on k unoccluded regions:

[0236] Select the pixel located at the position of maximum response in the activation map, and randomly generate a rectangle centered on this pixel. The maximum length and width of the rectangle are 50% of the image's width and height, and no less than 10%. The length h and width w of the rectangle are randomly generated.

[0237] h = randint(0.1*H, 0.5*H);

[0238] w = randint(0.1*W, 0.5*W);

[0239] randint() is a function that can be used to generate random numbers. The two parameters in parentheses can be used to represent the maximum and minimum values ​​of the random numbers that can be generated.

[0240] Based on the randomly generated rectangle, for image I i Values ​​within a rectangular region are extracted from channel c, and the mean and variance are calculated to generate a Gaussian noise image, which is then used to replace image I. i The value within the rectangular area of ​​channel c.

[0241] Next, select the pixel located at the second largest response position in the activation map, and randomly generate a rectangle centered on this pixel. The maximum length and width of the rectangle are 50% of the image's width and height, and no less than 10%. The length h and width w of the rectangle are randomly generated.

[0242] h = randint(0.1*H, 0.5*H);

[0243] w = randint(0.1*W, 0.5*W);

[0244] Based on the randomly generated rectangle, for image I i Values ​​within a rectangular region are extracted from channel c, and the mean and variance are calculated to generate a Gaussian noise image, which is then used to replace image I. i The value within the rectangular area of ​​channel c.

[0245] If the second randomly generated rectangle overlaps with the first randomly generated rectangle, only the non-overlapping parts are processed. The mean and variance are also calculated using only the non-overlapping parts. Therefore, an occluded input image is generated based on the two randomly generated rectangles, and the occluded region in this occluded input image is the union of the two randomly generated rectangles.

[0246] When k > 2, the technical concept is similar to that when k = 2, and will not be repeated here.

[0247] For each input image I within a batch of sample sets... iAfter the corresponding occluded input images, each occluded input image can be fed into the current ResNet18 model to obtain a new batch of predicted response values. The new batch of predicted response values ​​is normalized to obtain the predicted probability values, and the cross-entropy loss function is calculated based on the predicted probability values ​​and the true labels.

[0248]

[0249] L CE This refers to the cross-entropy loss function, where c represents the variable used to iterate through the three classification labels, B represents the number of samples in the batch, and b represents the variable used to iterate through the number of samples in the batch B. y is the normalized predicted probability value obtained by representing the predicted response value for predicting the class label of the b-th sample as c. bc The classification label used to indicate whether the b-th sample is category c is used; if so, then y bc =1; if not, then y bc =0.

[0250] In addition, a regularization loss function can be added to the feature map group output by the first submodule 402 to constrain sufficient differences between the channels of the feature map group.

[0251]

[0252] L reg Used to represent the regularization loss function, L reg The input value is the brightness information of the feature map group output by the first submodule 402. The specific definition of this loss function can be found in [reference needed]. Figure 1 The corresponding part in the embodiment.

[0253] The weighted sum of the two yields the loss function L = L CE +λ reg L reg , where λ reg This is the effect of balancing the two loss functions by pre-setting parameters.

[0254] The model parameters are updated by calculating the gradient of the loss function L on the model parameters of the entire ResNet18 model.

[0255] This completes one batch of model training. Multiple batches of training are then performed until the model converges.

[0256] Step (2):

[0257] Step (2) can be referred to Figure 1 In the description of step S106 in the embodiment, step (2) can replace the step of generating the intermediate image classification model for the first time.

[0258] Based on the model parameters obtained from step (1), the training parameters in the first submodule are fixed.

[0259] For each input image I within a batch of sample sets i Perform the following processing separately:

[0260] Will I i Input the current model to obtain the current ResNet18 model for the input image I. i Three predicted response values This corresponds to three categories. For example,

[0261] Assume I i If the truth label gi = 0, then the predicted response value at the position corresponding to the truth label gi is... Truth value g i Maximum response prediction outside

[0262] This can be assumed to be Let me continue explaining.

[0263] Activation maps are calculated for each of the C channels of feature map group F.

[0264]

[0265] This refers to the gradient information of the c-th channel of the feature map group F. FC This refers to the activation map of the c-th channel of feature map group F.

[0266] The activation map of channel c is subjected to a 3x3 mean filter to obtain a smoothed activation map.

[0267] The following is a detailed explanation of mean filtering: If the difference in brightness information between two adjacent pixels in the active image is greater than or equal to a preset threshold, it indicates that the brightness information of one of the pixels may be incorrect. Therefore, a filtering region of 3x3 pixels can be determined with either of the two pixels as the center, the average value of the brightness information of each pixel in the filtering region can be calculated, and the brightness information of the pixel at the center of the filtering region can be updated based on the average value.

[0268] Next, we can determine k regions to be occluded, and generate the occluded input image based on these k regions.

[0269] Generating an occluded input image based on k unoccluded regions can be achieved by determining the union of the k unoccluded regions as the target occlusion region, and then performing occlusion processing on the input image based on the target occlusion region. The following example, using k=2, illustrates the specific process of generating an occluded input image based on k unoccluded regions:

[0270] Select the pixel located at the position of maximum response in the activation map, and randomly generate a rectangle centered on this pixel. The maximum length and width of the rectangle are 50% of the image's width and height, and no less than 10%. The length h and width w of the rectangle are randomly generated.

[0271] h = randint(0.1*H, 0.5*H);

[0272] w = randint(0.1*W, 0.5*W);

[0273] randint() is a function that can be used to generate random numbers. The two parameters in parentheses can be used to represent the maximum and minimum values ​​of the random numbers that can be generated.

[0274] Based on randomly generated rectangles, values ​​within the rectangular regions are extracted from channel c of feature map group F. The mean and variance are calculated to generate a Gaussian noise image, which is then used to replace image I. i The value within the rectangular area of ​​channel c.

[0275] Next, select the pixel located at the second largest response position in the activation map, and randomly generate a rectangle centered on this pixel. The maximum length and width of the rectangle are 50% of the image's width and height, and no less than 10%. The length h and width w of the rectangle are randomly generated.

[0276] h = randint(0.1*H, 0.5*H);

[0277] w = randint(0.1*W, 0.5*W);

[0278] Based on the randomly generated rectangle, extract the values ​​within the rectangular region of channel c of feature map group F, calculate the mean and variance, and generate a Gaussian noise image. This Gaussian noise image is used to replace the values ​​within the rectangular region of channel c of feature map group F.

[0279] If the second randomly generated rectangle overlaps with the first randomly generated rectangle, only the non-overlapping parts are processed. The mean and variance are also calculated using only the non-overlapping parts. Therefore, an occluded input image is generated based on the two randomly generated rectangles, and the occluded region in this occluded input image is the union of the two randomly generated rectangles.

[0280] When k>2, the technical concept is similar to that when k=2, and will not be repeated here.

[0281] For each input image I within a batch of sample sets... i After the corresponding occluded feature map group F, it can be input into the unfrozen part of the current ResNet18 model. That is, the occluded feature map group F is input into the training sub-module consisting of pooling layer 404, sub-module 2 406, sub-module 3 408, sub-module 410, sub-module 5 412, pooling layer 414, and fully connected layer 416 to obtain a new batch of predicted response values. The new batch of predicted response values ​​is normalized to obtain the predicted probability values, and the cross-entropy loss function is calculated based on the aforementioned predicted probability values ​​and true value labels.

[0282]

[0283] L CE This refers to the cross-entropy loss function, where c represents the variable used to iterate through the three classification labels, B represents the number of samples in the batch, and b represents the variable used to iterate through the number of samples in the batch B. y is the normalized predicted probability value obtained by representing the predicted response value for predicting the class label of the b-th sample as c. bc The classification label used to indicate whether the b-th sample is category c is used; if so, then y bc =1; if not, then y bc =0.

[0284] At the same time, a regularization loss function is added to the feature map group output by the second submodule to constrain the feature maps of each channel of the feature map group to have sufficient differences.

[0285]

[0286] L reg Used to represent the regularization loss function, L reg The input value is the brightness information of the feature map group output by the second submodule. The specific definition of this loss function can be found in [reference needed]. Figure 1 The corresponding part in the embodiment.

[0287] The weighted sum of the two yields the loss function L = L CE +λ reg L reg , where λ reg This is the effect of balancing the two loss functions by pre-setting parameters.

[0288] The gradient of the unfrozen model parameters is calculated based on the loss function L, and the model parameters are updated.

[0289] This completes one batch of model training. Multiple batches of training are then performed until the model converges.

[0290] The following steps are similar to step (2): parameter freezing is performed on the parameters of each sub-module sequentially; activation maps for each channel of the current feature map are calculated based on the gradient; and the k largest activation regions of the corresponding feature map channels are occluded based on the activation maps. (Refer to...) Figure 1 The following is an explanation of step S106 in the embodiment. The occluded feature map is fed into the subsequent model. A regularization loss function is added to the feature map group in the next stage (if there is a next stage). The parameters of the subsequent model are updated by combining the regularization loss function and the classification loss function until the training of the 5th submodule 412 is completed. Then, the 1st submodule 402, 2nd submodule 406, 3rd submodule 408, 4th submodule 410, and 5th submodule 412 included in the trained ResNet18 model have all completed model training. Therefore, the trained ResNet18 model can be used to perform image classification tasks.

[0291] For example, in face image detection applications, the trained ResNet18 model can be used to detect whether a face image is forged. In practice, the face image input to the trained ResNet18 model may be subject to some interference, such as the eyes being obscured by sunglasses, or the image being limited by angle and lighting, resulting in lower clarity in a local area of ​​the left side of the face, etc. The trained ResNet18 model, in the process of detecting whether a face image is forged, can not rely solely on a few salient features, but rather leverage the characteristic that each submodule can extract rich features from the previous layer, to obtain a more accurate prediction of whether a face image is forged.

[0292] Based on the same technical concept as the aforementioned image classification model training method embodiments, this application embodiment also provides an image classification method.

[0293] Figure 5 This is a flowchart illustrating an image classification method provided in an embodiment of this application.

[0294] Step S502: Obtain the target image to be classified.

[0295] The target image can be an image that requires image classification, such as a face image, a trademark image, a car image, etc. A face image can be a face image to be detected as fake. A trademark image can be a trademark image to be authenticated. A car image can be a car image to be identified based on its model.

[0296] Step S504: Input the target image into the image classification model for classification prediction processing to obtain the classification prediction result of the target image; the image classification model is trained by the image classification model training method provided in the aforementioned image classification model training method embodiment.

[0297] For example, if the target image is a face image, the classification prediction result of the target image may include a prediction probability value p1 that the face in the image is a real face, a prediction probability value p2 that the face in the image is a face swapped face, and a prediction probability value p3 that the face in the image is a generated face.

[0298] After obtaining the classification prediction results of the target image, the final classification prediction result of the target image can be determined based on the comparison of the magnitudes of each prediction probability value. For example, assuming p1>p2>p3, the final classification prediction result of the target image can be determined as a real human face.

[0299] For example, if the target image is a trademark image, the classification prediction result of the target image may include a prediction probability value p1 that the target image is a genuine trademark, a prediction probability value p2 that the target image is a counterfeit trademark, and a prediction probability value p3 that the target image is something else.

[0300] After obtaining the classification prediction results of the target image, the final classification prediction result of the target image can be determined based on the comparison results between each predicted probability value and the preset probability value. For example, if p1 < preset probability value x, p2 > preset probability value x, and p3 < preset probability value x, then the final classification prediction result of the target image can be determined as counterfeit trademark.

[0301] For example, the target image is a car image, and the classification prediction result of the target image can include the predicted probability value p1 that the car in the target image is a first type of car, the predicted probability value p2 that the car in the target image is a second type of car, and so on, the predicted probability value pk that the car in the target image is a k-th type of car. k .

[0302] After obtaining the classification prediction results of the target image, the prediction probability values ​​that satisfy the preset classification conditions can be determined based on each prediction probability value. The image category corresponding to the prediction probability value that satisfies the preset classification conditions is then determined as the final classification prediction result of the target image. For example, suppose p1, p2...p kIf there is at least one positive prediction probability value, and the number of such prediction probability values ​​is one, then the image category corresponding to the prediction probability value is determined as the final classification prediction result of the target image; if the number of such prediction probability values ​​is greater than one, and the difference between the largest and second largest prediction probability values ​​is greater than a preset threshold, then the image category corresponding to the largest prediction probability value is determined as the final classification prediction result of the target image.

[0303] In the above embodiments, a training method for an image classification model is provided, and correspondingly, a training device for an image classification model is also provided, which will be described below with reference to the accompanying drawings.

[0304] Figure 6 This is a schematic diagram of a training device for an image classification model provided in an embodiment of this application.

[0305] This embodiment provides a training device 600 for an image classification model, comprising:

[0306] The occlusion unit 601 is used to occlude the first sample image according to the first classification prediction result of the first sample image to obtain the second sample image; the first classification prediction result is obtained by inputting the first sample image into the first image classification model for classification prediction processing; the first image classification model includes N sub-modules; the input of the (i+1)th sub-module is determined by the output of the ith sub-module;

[0307] Training unit 602 is used to input the second sample image into the first image classification model for iterative training to obtain the second image classification model;

[0308] The generation unit 603 is used to perform the step of generating intermediate image classification models N-1 times, and to determine the intermediate image classification model generated in the N-1th time as the target image classification model. The step of generating the intermediate image classification model in the i-th time includes: freezing the parameters of the i-th sub-module in the second image classification model to obtain a third image classification model; inputting the first sample image into the third image classification model for classification prediction processing to obtain a second classification prediction result for the first sample image; determining the occluded region corresponding to the feature map group output by the i-th sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model; when i is less than N-1, the intermediate image classification model is used as the corresponding second image classification model in the (i+1)-th step of generating the intermediate image classification model.

[0309] Optionally, the feature map group includes a feature map for each channel; the generation unit 603 includes:

[0310] The sub-unit group is determined based on the second classification prediction result to determine the core feature region corresponding to each channel in the feature map group, and the core feature region corresponding to each channel in the feature map group is determined as the occluded region corresponding to each channel in the feature map group.

[0311] Optionally, the first sample image carries a target classification label; the target classification label is one of a variety of preset classification labels;

[0312] The first classification prediction result includes: the first predicted probability value of the target classification label being each of the multiple preset classification labels;

[0313] The second classification prediction result includes: the second predicted probability value of the target classification label being each of the multiple preset classification labels.

[0314] Optionally, the sub-unit group is determined, including:

[0315] The first determining subunit is used to determine the second predicted probability value that matches the target classification label as the first probability value in the second classification prediction result;

[0316] The second determining subunit is used to determine the second predicted probability value, which is the highest value other than the first probability value, as the second probability value in the second classification prediction result.

[0317] The computational subunit is used to perform gradient calculation on the brightness information of the feature map of each channel in the feature map group based on the first probability value and the second probability value, so as to obtain the activation map corresponding to each channel in the feature map group.

[0318] The third determining subunit is used to determine the core feature region corresponding to each channel in the feature map group based on the activation map corresponding to each channel in the feature map group.

[0319] Optionally, the third determining sub-unit is specifically used for:

[0320] For each channel in the feature map group, based on the brightness information of each point in the activation map, determine the first response point with the largest brightness information and the second response point with the largest brightness information outside the first response point in the activation map;

[0321] A first region is generated based on the first response point; a second region is generated based on the second response point.

[0322] The union of the first and second regions is determined as the core feature region corresponding to the activation map.

[0323] Optionally, the third image classification model includes a sample generation sub-model built based on the first i sub-modules and a training sub-model built based on the last Ni sub-modules;

[0324] The sample generation sub-model is used to generate a third sample image based on the occluded region corresponding to the feature map group;

[0325] The sub-model to be trained is used for iterative training based on the third sample image to obtain an intermediate image classification model.

[0326] Optionally, the loss function of the third image classification model is determined by the regularization loss function and the cross-entropy loss function;

[0327] The regularization loss function is determined by the feature map of each channel in the feature map group output by the (i+1)th submodule;

[0328] The cross-entropy loss function is determined by the target classification label and the second classification prediction result.

[0329] Optionally, the occlusion unit 601 includes:

[0330] The fourth determining subunit is used to determine the core feature regions in the first sample image based on the target classification label and the first classification prediction result;

[0331] The generation subunit is used to generate a noisy occlusion image based on the brightness information value of each pixel in the core feature region;

[0332] The replacement sub-unit is used to replace the sub-image corresponding to the core feature region in the first sample image with the noise-occluded image.

[0333] Optionally, sub-units are generated, specifically for:

[0334] Based on the brightness information value of each pixel in the core feature region, calculate the average brightness and brightness variance of the core feature region.

[0335] Based on the size parameters of the core feature region, the average brightness and the brightness variance of the corresponding core feature region, a Gaussian noise image is generated, and the Gaussian noise image is identified as the noise occlusion image.

[0336] The image classification model training device provided in this application includes an occlusion unit, a training unit, and a generation unit. The occlusion unit is used to occlude the first sample image based on a first classification prediction result of the first sample image to obtain a second sample image. The first classification prediction result is obtained by inputting the first sample image into the first image classification model for classification prediction processing. The first image classification model includes N sub-modules; the input of the (i+1)th sub-module is determined by the output of the i-th sub-module. The training unit is used to input the second sample image into the first image classification model for iterative training to obtain a second image classification model. The generation unit is used to perform N-1 steps of generating intermediate image classification models and determine the intermediate image classification model generated in the N-1th generation. The target image classification model is defined as follows: the step of generating an intermediate image classification model for the i-th time includes: freezing the parameters of the i-th sub-module in the second image classification model to obtain a third image classification model; inputting the first sample image into the third image classification model for classification prediction to obtain a second classification prediction result for the first sample image; determining the occluded region corresponding to the feature map group output by the i-th sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain an intermediate image classification model; when i is less than N-1, the intermediate image classification model is used as the corresponding second image classification model in the (i+1)-th step of generating the intermediate image classification model. Therefore, by occluding the first sample image based on the first classification prediction result, the salient features in the first sample image can be blocked, forcing each sub-module in the first image classification model to mine secondary features other than the salient features. Furthermore, by performing N-1 steps to generate intermediate image classification models, the parameters of the i-th sub-module are frozen during the i-th step of generating intermediate image classification models, which can fix the model parameters of the i-th sub-module layer by layer. By occluding the feature map group output by the i-th sub-module based on the second classification prediction result, the salient features in the feature map group output by the i-th sub-module can be blocked, forcing the i+1 sub-module after the i-th sub-module to mine secondary features other than the salient features. Thus, in the intermediate image classification model generated in the N-1th step, the first sub-module can mine rich features from the image input to the target image classification model, and each sub-module other than the first sub-module can mine rich features from the feature map group output by the previous sub-module, improving the overall feature mining performance of the target image classification model and enhancing the robustness and generalization of the target image classification model.

[0337] In the above embodiments, an image classification method is provided, and correspondingly, an image classification device is also provided, which will be described below with reference to the accompanying drawings.

[0338] Figure 7 This is a schematic diagram of an image classification device provided in an embodiment of this application.

[0339] This embodiment provides an image classification device 700, including:

[0340] The acquisition unit is used to acquire the target image to be classified.

[0341] The classification unit is used to input the target image into the image classification model for classification prediction processing to obtain the classification prediction result of the target image; the image classification model is trained by the image classification model training method provided in the aforementioned image classification model training method embodiment.

[0342] The image classification model training apparatus provided in this application includes an acquisition unit and a classification unit. The acquisition unit acquires the target image to be classified; the classification unit inputs the target image into the image classification model for classification prediction processing to obtain the classification prediction result of the target image. The image classification model is trained using the image classification model training method provided in the aforementioned embodiment. Therefore, the image classification model trained using this method has better feature extraction performance, can extract rich features, and makes the classification result of the target image more accurate.

[0343] Corresponding to the image classification model training method described above, or corresponding to the image classification method described above, based on the same technical concept, this application embodiment also provides an electronic device for executing the image classification model training method or the image classification method provided above. Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0344] like Figure 8 As shown, electronic devices can vary considerably due to differences in configuration or performance. They may include one or more processors 801 and memories 802, with the memory 802 storing one or more application programs or data. The memory 802 can be temporary or persistent storage. The application programs stored in the memory 802 may include one or more modules (not shown), each module including a series of computer-executable instructions within the electronic device. Furthermore, the processor 801 may be configured to communicate with the memory 802, executing the series of computer-executable instructions stored in the memory 802 on the electronic device. The electronic device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input / output interfaces 805, one or more keyboards 806, etc.

[0345] In one specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0346] Based on the first classification prediction result of the first sample image, the first sample image is occluded to obtain the second sample image; the first classification prediction result is obtained by inputting the first sample image into the first image classification model for classification prediction processing; the first image classification model includes N sub-modules; the input of the (i+1)th sub-module is determined by the output of the ith sub-module;

[0347] The second sample image is input into the first image classification model for iterative training to obtain the second image classification model;

[0348] Perform the step of generating intermediate image classification models N-1 times, and determine the intermediate image classification model generated in the N-1th time as the target image classification model;

[0349] The step of generating the intermediate image classification model for the i-th time includes: freezing the parameters of the i-th sub-module in the second image classification model to obtain the third image classification model; inputting the first sample image into the third image classification model for classification prediction to obtain the second classification prediction result of the first sample image; determining the occluded region corresponding to the feature map group output by the i-th sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain the intermediate image classification model; when i is less than N-1, the intermediate image classification model is used as the second image classification model corresponding to the (i+1)-th step of generating the intermediate image classification model.

[0350] In another specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0351] Obtain the target image to be classified;

[0352] The target image is input into the image classification model for classification prediction processing to obtain the classification prediction result of the target image; the image classification model is trained using the image classification model training method.

[0353] Corresponding to the image classification model training method described above, or corresponding to the image classification method described above, based on the same technical concept, this application embodiment also provides a computer-readable storage medium.

[0354] In one specific embodiment, the computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed by a processor, implement the following process:

[0355] Based on the first classification prediction result of the first sample image, the first sample image is occluded to obtain the second sample image; the first classification prediction result is obtained by inputting the first sample image into the first image classification model for classification prediction processing; the first image classification model includes N sub-modules; the input of the (i+1)th sub-module is determined by the output of the ith sub-module;

[0356] The second sample image is input into the first image classification model for iterative training to obtain the second image classification model;

[0357] Perform the step of generating intermediate image classification models N-1 times, and determine the intermediate image classification model generated in the N-1th time as the target image classification model;

[0358] The step of generating the intermediate image classification model for the i-th time includes: freezing the parameters of the i-th sub-module in the second image classification model to obtain the third image classification model; inputting the first sample image into the third image classification model for classification prediction to obtain the second classification prediction result of the first sample image; determining the occluded region corresponding to the feature map group output by the i-th sub-module based on the second classification prediction result; inputting the first sample image into the third image classification model for iterative training based on the occluded region to obtain the intermediate image classification model; when i is less than N-1, the intermediate image classification model is used as the second image classification model corresponding to the (i+1)-th step of generating the intermediate image classification model.

[0359] In another specific embodiment, the computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed by a processor, implement the following process:

[0360] Obtain the target image to be classified;

[0361] The target image is input into the image classification model for classification prediction processing to obtain the classification prediction result of the target image; the image classification model is trained using the image classification model training method.

[0362] It should be noted that the embodiments of computer-readable storage media in this specification and the embodiments of training methods for image classification models in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding methods described above, and the repeated parts will not be described again.

[0363] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0364] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, this specification can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0365] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0366] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0367] These computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0368] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0369] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0370] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0371] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0372] The embodiments of this application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0373] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0374] The above are merely embodiments of this document and are not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A method for training an image classification model, the method comprising: The method comprises the following steps: obtaining a second sample image by performing occlusion processing on the first sample image according to a first classification prediction result of the first sample image; the first classification prediction result is obtained by inputting the first sample image into a first image classification model for classification prediction processing; the first image classification model comprises N sub-modules; the input of an (i+1)th sub-module is determined by the output of an ith sub-module; performing iterative training on the second sample image by inputting the second sample image into the first image classification model to obtain a second image classification model; performing the step of generating an intermediate image classification model for N-1 times, and determining the intermediate image classification model generated for the N-1th time as a target image classification model; the step of generating an intermediate image classification model for the ith time comprises: performing parameter freezing processing on the parameters of the ith sub-module in the second image classification model to obtain a third image classification model; performing classification prediction processing on the first sample image by inputting the first sample image into the third image classification model to obtain a second classification prediction result of the first sample image; determining a to-be-occluded region corresponding to a feature map group output by the ith sub-module according to the second classification prediction result; inputting the first sample image into the third image classification model to perform iterative training based on the to-be-occluded region to obtain an intermediate image classification model; and when i is less than N-1, taking the intermediate image classification model as the corresponding second image classification model in the step of generating an intermediate image classification model for the (i+1)th time; the feature map group comprises feature maps of each channel; and the step of determining a to-be-occluded region corresponding to the feature map group in the feature map group output by the ith sub-module according to the second classification prediction result comprises:

2. The method of claim 1, wherein, determining a core feature region corresponding to each channel in the feature map group according to the second classification prediction result, and taking the core feature region corresponding to each channel in the feature map group as the to-be-occluded region corresponding to each channel in the feature map group. the first sample image carries a target classification label; and the target classification label is one of a plurality of preset classification labels; the first classification prediction result comprises a first prediction probability value of the target classification label as each of the preset classification labels; 3. The method of claim 2, wherein, the second classification prediction result comprises a second prediction probability value of the target classification label as each of the preset classification labels. the step of determining a core feature region corresponding to each channel in the feature map group according to the second classification prediction result comprises: in the second classification prediction result, taking a second prediction probability value matching the target classification label as a first probability value; in the second classification prediction result, taking a second prediction probability value with the highest value other than the first probability value as a second probability value; performing gradient calculation processing on the brightness information of the feature maps of each channel in the feature map group according to the first probability value and the second probability value to obtain an activation map corresponding to each channel in the feature map group. According to the activation map corresponding to each channel in the feature map group, a core feature region corresponding to each channel in the feature map group is determined.

4. The method of claim 3, wherein, The determination of the core feature region corresponding to each channel in the feature map group according to the activation map comprises: For each channel in the feature map group, a first response point with the maximum brightness information in the activation map and a second response point with the maximum brightness information outside the first response point are determined according to the brightness information of each point in the activation map; A first region corresponding to the first response point is generated based on the first response point, and a second region corresponding to the second response point is generated based on the second response point; The union of the first region and the second region is determined as the core feature region corresponding to the activation map.

5. The method of claim 1, wherein, The third image classification model comprises a sample generation sub-model constructed based on the first i sub-modules and a to-be-trained sub-model constructed based on the last N-i sub-modules; The sample generation sub-model is configured to generate a third sample image based on the to-be-occluded region corresponding to the feature map group; The to-be-trained sub-model is configured to perform iterative training according to the third sample image to obtain an intermediate image classification model.

6. The method of claim 2, wherein, The loss function of the third image classification model is determined by a regularization loss function and a cross-entropy loss function; The regularization loss function is determined by the feature map of each channel in the feature map group output by the i+1th sub-module; The cross-entropy loss function is determined by the target classification label and the second classification prediction result.

7. The method of claim 2, wherein, The occlusion processing of the first sample image according to the first classification prediction result of the first sample image to obtain a second sample image comprises: According to the target classification label and the first classification prediction result, a core feature region in the first sample image is determined; According to the brightness information value of each pixel in the core feature region, a noise occlusion image is generated; The sub-image corresponding to the core feature region in the first sample image is replaced by the noise occlusion image.

8. The method of claim 7, wherein, The generation of the noise occlusion image according to the brightness information value of each pixel in the core feature region comprises: According to the brightness information value of each pixel in the core feature region, the brightness average value and the brightness variance corresponding to the core feature region are calculated; According to the size parameter of the core feature region, the brightness average value and the brightness variance corresponding to the core feature region, a Gaussian noise image is generated, and the Gaussian noise image is determined as the noise occlusion image.

9. An image classification method characterized by, It comprises: obtaining a target image to be classified; inputting the target image into an image classification model for classification prediction processing to obtain a classification prediction result of the target image; The image classification model is trained by the training method of the image classification model according to any one of claims 1-8.

10. A training device for an image classification model, characterized in that, It comprises: an occlusion unit configured to perform occlusion processing on a first sample image according to a first classification prediction result of the first sample image to obtain a second sample image; The first classification prediction result is obtained by inputting the first sample image into a first image classification model for classification prediction processing; the first image classification model comprises N sub-modules; An input of the i+1th sub-module is determined by an output of the ith sub-module; The training unit is configured to input the second sample image into the first image classification model for iterative training to obtain a second image classification model. The generating unit is configured to perform an N-1th step of generating an intermediate image classification model, and determine the N-1th generated intermediate image classification model as the target image classification model. The ith step of generating the intermediate image classification model includes: performing parameter freezing processing on parameters of the ith sub-module in the second image classification model to obtain a third image classification model; inputting the first sample image into the third image classification model for classification prediction processing to obtain a second classification prediction result of the first sample image; determining a to-be-occluded region corresponding to the feature map group in the feature map group output by the ith sub-module according to the second classification prediction result; inputting the first sample image into the third image classification model to perform iterative training based on the to-be-occluded region to obtain an intermediate image classification model; and when i is less than N-1, taking the intermediate image classification model as a corresponding second image classification model in the i+1th step of generating the intermediate image classification model. The feature map group includes feature maps of each channel; and the generating unit includes: A determining sub-unit group is configured to determine a core feature region corresponding to each channel in the feature map group according to the second classification prediction result, and determine the core feature region corresponding to each channel in the feature map group as the to-be-occluded region corresponding to each channel in the feature map group.

11. An electronic device, comprising: comprise: a processor; and a memory configured to store computer-executable instructions that, when executed, cause the processor to perform the training method of the image classification model according to any one of claims 1-8, or the image classification method according to claim 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store computer-executable instructions that, when executed by a processor, implement the training method of the image classification model according to any one of claims 1-8, or the image classification method according to claim 9.

Images