Data augmentation method, model training method, and image processing method and device

Abstract

The present disclosure provides a data enhancement method, a model training method and an image processing method and device, relates to the technical field of artificial intelligence, in particular to the computer vision technology. The specific implementation scheme is: obtaining a class activation mapping corresponding to a training sample for data enhancement, wherein the class activation mapping is generated based on a sample image of the training sample and a target image processing model trained by the training sample; generating an attention mask matching the size of the sample image based on the class activation mapping corresponding to the training sample; and performing data enhancement on the pre-acquired sample image based on the pre-acquired noise matrix and the attention mask to generate a new training sample corresponding to the training sample. Thus, the adaptability of the data enhancement method to different models can be improved, thereby helping to improve the training effect of the model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, and more particularly to a data augmentation method, a model training method, and an image processing method and device. BACKGROUND

[0002] With the development of artificial intelligence technology, the demand for data augmentation technology is becoming increasingly strong.

[0003] In the field of computer vision, existing data augmentation methods mainly include the following: image horizontal or vertical flipping; image rotation; image scaling; image random cropping; adding noise to the image; image brightness, contrast, and saturation changes; image local distortion and radiation transformation; random region occlusion of the image; and superimposition of two or more images. SUMMARY

[0004] A data augmentation method, a model training method, and an image processing method and device are provided.

[0005] According to a first aspect, a data augmentation method is provided, which includes: obtaining a class activation map corresponding to a training sample for data augmentation, wherein the class activation map is generated based on a sample image of the training sample and a target image processing model trained by the training sample; generating an attention mask matching the size of the sample image based on the class activation map corresponding to the training sample; and performing data augmentation on a pre-acquired sample image based on a pre-acquired noise matrix and the attention mask to generate a new training sample corresponding to the training sample.

[0006] According to a second aspect, a method for training an image processing model is provided, which includes: obtaining an image processing pre-trained model trained based on a training sample; and performing fine training on the image processing pre-trained model based on a new training sample corresponding to the training sample to obtain a trained image processing model, wherein the new training sample is obtained according to the method described in any of the implementations of the first aspect.

[0007] According to a third aspect, a method for image processing is provided, which includes: obtaining an image to be processed; and inputting the image to be processed into a pre-trained image processing model to generate an image processing result, wherein the image processing model is trained according to the method described in any of the implementations of the second aspect.

[0008] According to a fourth aspect, a data augmentation apparatus is provided, which comprises: an acquisition unit configured to acquire a class activation map corresponding to a training sample for data augmentation, wherein the class activation map is generated based on a sample image of the training sample and a target image processing model trained by the training sample; a first generation unit configured to generate an attention mask matching a size of the sample image based on the class activation map corresponding to the training sample; and a second generation unit configured to perform data augmentation on the pre-acquired sample image based on the pre-acquired noise matrix and the attention mask to generate a new training sample corresponding to the training sample.

[0009] According to a fifth aspect, an apparatus for training an image processing model is provided, which comprises: a model acquisition unit configured to acquire an image processing pre-trained model trained based on training samples; and a training unit configured to perform fine training on the image processing pre-trained model based on a new training sample corresponding to a training sample to obtain a trained image processing model, wherein the new training sample is obtained according to the method described in any implementation manner of the first aspect.

[0010] According to a sixth aspect, an apparatus for image processing is provided, which comprises: an image acquisition unit configured to acquire an image to be processed; and a processing unit configured to input the image to be processed into a pre-trained image processing model to generate an image processing result, wherein the image processing model is trained according to the method described in any implementation manner of the second aspect.

[0011] According to a seventh aspect, an electronic device is provided, which comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any implementation manner of the first aspect, the second aspect and the third aspect.

[0012] According to an eighth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, which is used to enable a computer to perform the method described in any implementation manner of the first aspect, the second aspect and the third aspect.

[0013] According to a ninth aspect, a computer program product is provided, which comprises a computer program, and the computer program, when executed by a processor, implements a non-transitory computer-readable storage medium storing computer instructions, which is used to enable a computer to perform the method described in any implementation manner of the first aspect, the second aspect and the third aspect.

[0014] According to the technical solution of the present disclosure, the semantic information contained in the class activation map of the training sample is introduced into the data augmentation method by taking the class activation map generated based on the training sample and the corresponding target image processing model as a reference factor for generating the attention mask, so that the data augmentation method can make targeted adjustments according to specific data and training tasks, thereby improving the adaptability of the data augmentation method to different models, and further helping to improve the training effect of the model. Moreover, the data augmentation method of the present solution does not introduce additional variables and parameters, and can be completed only by using the original structure of the model, and is relatively simple to implement.

[0015] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings are used to better understand the present solution and do not limit the present disclosure. Among them:

[0017] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0018] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0019] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0020] Figure 4 is a schematic diagram of one application scenario of the data augmentation method that can implement the embodiments of the present disclosure;

[0021] Figure 5 is a schematic diagram of a data augmentation device according to an embodiment of the present disclosure;

[0022] Figure 6 is a schematic diagram of a device for training an image processing model according to an embodiment of the present disclosure;

[0023] Figure 7 is a schematic diagram of a device for image processing according to an embodiment of the present disclosure;

[0024] Figure 8 is a block diagram of an electronic device for implementing the data augmentation method, the method for training an image processing model, or the method for image processing according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various details intended to facilitate understanding of the present disclosure. Accordingly, it should be understood that various changes and modifications to the embodiments described herein can be made by those skilled in the art without departing from the scope and spirit of the present disclosure. Also, for the purpose of clarity and a concise description, descriptions of well-known functions and constructions are omitted from the following description.

[0026] Figure 1 is a schematic diagram 100 according to a first embodiment of the present disclosure. The data enhancement method includes the following steps:

[0027] S101, obtaining a class activation mapping corresponding to a training sample for data enhancement.

[0028] In this embodiment, the execution subject of the data enhancement method can obtain the class activation mapping corresponding to the training sample for data enhancement from the local or communicatively connected electronic device through wired or wireless connection. The training sample can generally include a sample image and sample annotation information corresponding to the sample image. The class activation mapping is generally generated based on the sample image of the training sample and the target image processing model trained by the training sample. As an example, the target image processing model can be a model trained by machine learning using the training sample.

[0029] In this embodiment, the class activation mapping (CAM) can be used to represent a heat map obtained by calculating the feature map output by the last feature extraction layer of the target image processing model and the weights of the fully connected layer of the target image processing model. It can reflect the attention of the target image processing model to different positions of the input sample image. Optionally, the class activation mapping can also include other methods based on which optimization is performed, such as Grad-CAM (Gradient-weighted Class Activation Mapping).

[0030] As an example, the class activation mapping can be represented by the following formula (1):

[0031]

[0032] wherein C can be used to represent the generated class activation mapping corresponding to the current input sample image. F cThe c-th channel of the feature map output by the last feature extraction layer of the target image processing model corresponding to the current input sample image can be used to represent the c-th channel of the feature map output by the last feature extraction layer of the target image processing model. The n can be used to represent the total number of channels of the feature map output by the last feature extraction layer of the target image processing model. The ω lc The weight value of the c-th channel in the weight vector of the l-th class corresponding to the current input sample image represented by the fully connected layer of the target image processing model can be used to represent the weight value of the c-th channel in the weight vector of the l-th class corresponding to the current input sample image represented by the fully connected layer of the target image processing model.

[0033] Therefore, the class activation map is usually the same size as the feature map output by the last feature extraction layer, and can reflect the attention of the target image processing model to different positions of the input sample image.

[0034] It should be noted that the execution subject can also obtain a class activation map set composed of class activation maps corresponding to a plurality of training samples. Each class activation map in the class activation map set can correspond to a training sample.

[0035] S102, generating an attention mask matching the size of the sample image based on the class activation map corresponding to the training sample.

[0036] In this embodiment, based on the training sample obtained in the above step S101, the execution subject can generate an attention mask matching the size of the sample image in various ways. The size matching usually refers to the same length-width ratio. Alternatively, the size matching can also refer to the same size.

[0037] In this embodiment, the distribution of attention weights contained in the attention mask is usually consistent with the distribution of values in the class activation map. As an example, for the position of the input sample image to which the target image processing model pays more attention embodied in the class activation map, the value of the element corresponding to the position in the attention mask is also higher.

[0038] S103, based on the pre-acquired noise matrix and the attention mask, data augmentation is performed on the pre-acquired sample image to generate new training samples corresponding to the training samples.

[0039] In this embodiment, based on the pre-acquired noise matrix and the attention mask generated in step S102, the execution subject can perform data augmentation on the pre-acquired sample image in various ways to generate new training samples corresponding to the training samples. The noise matrix can be used to represent the noise (such as white noise, Gaussian noise, etc.) to be added to the sample image of the training sample.

[0040] As an example, the execution subject can fuse the pre-acquired noise matrix with the attention mask generated in step S102 and the pre-acquired sample image to generate a new sample image. Then, the execution subject can combine the generated new sample image and the sample label information of the corresponding training sample into a new training sample.

[0041] The method provided by the above embodiments of the present disclosure can introduce the semantic information contained in the class activation map of the training sample into the data augmentation method by taking the class activation map generated based on the training sample and the corresponding target image processing model as a reference factor for generating the attention mask, so that the data augmentation method can make targeted adjustments according to specific data and training tasks, thereby improving the adaptability of the data augmentation method to different models, and further helping to improve the training effect of the model. Moreover, the data augmentation method of the present solution does not introduce additional variables and parameters, and can be completed only by using the original structure of the model, and the implementation is also relatively simple.

[0042] In some optional implementations of the present embodiment, based on the class activation map corresponding to the training sample, the execution subject can generate an attention mask matching the size of the sample image by the following steps:

[0043] S1021, performing a scale transformation on the class activation map corresponding to the training sample to generate an attention heat map consistent in size with the sample image.

[0044] In these implementations, the execution subject can perform a scale transformation (such as an image enlargement or reduction method) on the class activation map corresponding to the training sample in various ways to generate an attention heat map consistent in size with the sample image.

[0045] S1022, generating an attention mask consistent in size with the attention heat map based on a comparison between the values in the attention heat map and a target value.

[0046] In these implementations, the execution subject can compare the values in the attention heat map generated in step S1021 with a target value, and according to different comparison results, the execution subject can generate an attention mask consistent in size with the attention heat map. The target value can be a pre-set numerical value or a value determined according to a rule (such as a local area average value), which is not limited herein.

[0047] Based on the above optional implementations, the present solution can generate an attention mask by comparing the attention heat map consistent in size with the sample image with a target value, thereby enriching the generation method of the attention mask.

[0048] Optionally, based on the optional implementation above, the target value can include an average value determined based on the attention heat map. As an example, the target value can be an average value of each pixel in the attention heat map. As another example, the target value can also be an average value of each pixel in the attention heat map consistent with the size of the sample image.

[0049] Therefore, the present scheme can determine the size of the value of each element in the attention mask based on the global average value of the attention heat map, so that the generated attention mask only pays attention to the area with higher importance in the image, thereby improving the effect of the attention mask.

[0050] Optionally, based on the optional implementation above, based on the comparison between the value in the attention heat map and the target value, the execution subject can generate an attention mask consistent with the size of the attention heat map by the following steps:

[0051] First, in response to determining that the value in the attention heat map is less than the target value, a preset first value is determined as the corresponding value in the attention mask.

[0052] Second, in response to determining that the value in the attention heat map is greater than or equal to the target value, a preset second value is determined as the corresponding value in the attention mask.

[0053] In these implementations, the preset first value is generally less than the preset second value. As an example, the preset first value can be 0, and the preset second value can be 1.

[0054] As an example, the attention mask can be represented by the following formula (2):

[0055]

[0056] wherein i∈[1,p], j∈[1,q]. p and q can be used to represent the width and length of the generated attention mask, respectively. M i,j may be used to represent the value of the element of the generated attention mask at the (i,j) position. y0 and y1 can be used to represent the preset first value and the preset second value, respectively. C i ′ ,j may be used to represent the value of the element of the attention heat map consistent with the size of the input sample image at the (i,j) position. A can be used to represent the target value.

[0057] Based on the optional implementation above, the present scheme can generate an attention mask using a preset first value and a preset second value, thereby distinguishing the attention given to different important areas in the image, and thereby improving the effect of the attention mask.

[0058] In some optional implementations of the present embodiment, the size of the attention mask is generally consistent with the size of the sample image. Based on the pre-acquired noise matrix and the attention mask, the execution body can perform data augmentation on the pre-acquired sample image to generate new training samples corresponding to the training samples by the following steps:

[0059] S1031, multiply the pre-acquired noise matrix and the attention mask which are consistent with the size of the sample image element by element to generate a first noise matrix.

[0060] S1032, based on the fusion of the first noise matrix and the sample image, generate a new sample image corresponding to the sample image.

[0061] In these implementations, the execution body can fuse the first noise matrix generated in step S1031 with the pre-acquired sample image in various ways to generate a new sample image corresponding to the sample image.

[0062] S1033, based on the new sample image and the sample annotation information of the training sample, generate a new training sample.

[0063] In these implementations, based on the new sample image generated in step S1032 and the sample annotation information of the training sample, the execution body can generate a new training sample in various ways. As an example, the execution body can associate the new sample image generated in step S1032 with the sample annotation information of the corresponding training sample as a new training sample.

[0064] Based on the above optional implementation, the present scheme provides a scheme for generating new training samples using pre-acquired noise matrices, attention masks, and sample images, thereby realizing data augmentation associated with training samples and model training.

[0065] Optionally, based on the above optional implementation, the execution body can add the first noise matrix and the sample image element by element to generate a new sample image.

[0066] Thus, the present scheme provides a new sample image generation method that introduces additive noise, thereby enriching the data augmentation method associated with training samples and model training.

[0067] Optionally, based on the above optional implementation, based on the fusion of the first noise matrix and the sample image, the execution body can generate a new sample image corresponding to the sample image according to the following steps:

[0068] First, correct the elements belonging to the preset value in the first noise matrix to 1 to generate a second noise matrix.

[0069] In these implementations, the preset value can be flexibly set according to the needs of the actual application scenario. It can be related to the value in the attention mask. As an example, when the value in the attention mask is discrete (for example, only contains 0 and 1), the preset value can be a numerical value, for example, it can be 0. As an example, when the value in the attention mask is continuous (for example, takes a value between 0 and 1), the preset value can also be a range, for example, it can be less than 0.5.

[0070] Secondly, the second noise matrix is multiplied with the sample image element by element to generate a new sample image.

[0071] Based on the above optional implementation, the scheme provides a new sample image generation method introducing multiplicative noise, thereby enriching the data enhancement method associated with the training sample and model training.

[0072] In some optional implementations of the present embodiment, the parameters of the target image processing model can be the saved model parameters with the best effect on the validation set corresponding to the training sample after training based on the training sample.

[0073] In these implementations, as an example, the execution subject for training the image processing model can first use the training set containing the training sample to train the model. Then, the execution subject can use the validation set corresponding to the training set to verify the effect of the trained image processing model, and select the image processing model with the best effect as the target image processing model.

[0074] Thus, by selecting the target image processing model with good effect and high accuracy as the basis for data enhancement, the data enhancement method can be more targeted for the model, which helps further training of the model.

[0075] Continuing to refer to Figure 2 , Figure 2 is a schematic diagram 200 according to the second embodiment of the present disclosure. The method for training an image processing model includes the following steps:

[0076] S201, obtaining an image processing pre-trained model trained based on a training sample.

[0077] In the present embodiment, the execution subject of the method for training an image processing model can obtain an image processing pre-trained model trained based on a training sample from a local or communicatively connected electronic device through wired or wireless connection. The image processing pre-trained model can include an image processing model trained using the training sample of the training set.

[0078] S202, fine-tune the image processing pre-trained model based on new training samples corresponding to the training samples to obtain a trained image processing model.

[0079] In the embodiment, the execution subject can obtain new training samples corresponding to the training samples in step S201 in advance. Then, based on the image processing pre-trained model obtained in step S201, the execution subject can adjust the parameters of the image processing pre-trained model using the new training samples, stop training when the training end condition is met, and thus obtain a trained image processing model. The new training samples can be obtained according to the data augmentation method described in any of the preceding embodiments, which will not be described here.

[0080] From Figure 2 It can be seen that the flow 200 of the method for training an image processing model in the embodiment embodies the steps of pre-training an image processing model using training samples and fine-tuning the image processing model using a data augmentation method associated with the training samples and the model. Thus, the scheme described in the embodiment can further improve the training effect of the image processing model.

[0081] Next, referring to Figure 3 , Figure 3 is a schematic diagram 300 according to the third embodiment of the disclosure. The method for image processing includes the following steps:

[0082] S301, obtaining an image to be processed.

[0083] In the embodiment, the execution subject of the method for image processing can obtain the image to be processed in various ways. The image to be processed can include various images that can be processed by the image processing model, which is not limited here.

[0084] In the embodiment, as an example, the image to be processed can be an image containing a person. As another example, the image to be processed can be a road condition image taken by an autonomous vehicle. The execution subject can obtain the image to be processed from a local or communicatively connected electronic device.

[0085] S302, inputting the image to be processed into a pre-trained image processing model to generate an image processing result.

[0086] In the embodiment, the execution subject can input the to-be-processed image obtained in step S301 into the pre-trained image processing model to generate an image processing result corresponding to the to-be-processed image. The image processing model can be trained by the method for training an image processing model described in any of the foregoing embodiments. The image processing result can correspond to the image processing model. For example, when the image processing model is a face recognition model, the image processing result can be used to represent information of a person displayed in the face image. For another example, when the image processing model is a lane detection model, the image processing result can be used to indicate a position of a lane line displayed in the image.

[0087] It can be seen from Figure 3 that the flow 300 of the method for processing an image in the embodiment embodies a step of processing an image by using an image processing model trained by the method for training an image processing model. Thus, the scheme described in the embodiment can improve the effect of image processing by using an image processing model trained by new training samples obtained by using the data enhancement method associated with a training sample and a model.

[0088] Continuing to refer to Figure 4 , Figure 4 is an application scenario of the data enhancement method according to an embodiment of the present disclosure. In the application scenario of Figure 4 , an execution subject (for example, a server) for data enhancement can first obtain a class activation map 401 corresponding to a training sample for data enhancement. The class activation map 401 can be generated based on a sample image 4011 of the training sample and a target image processing model 4012 trained by using the training sample containing the sample image 4011. Then, the execution subject can generate an attention mask 402 matching the size of the sample image 4011 based on the class activation map 401. Finally, based on a pre-obtained noise matrix 403 and the generated attention mask 402, the execution subject can perform data enhancement on the pre-obtained sample image 4011 to generate a new sample image 404 corresponding to the sample image 4011, thereby generating a new training sample.

[0089] Currently, the prior art usually adopts image flipping, rotation, adding noise, image random area shielding and the like to perform data enhancement. Since the above data enhancement methods are all changes without distinction to the whole image or enhancement to some random areas, the data enhancement method cannot make targeted adjustment according to specific data and tasks. Moreover, since the above data enhancement methods do not utilize the information learned by the model to be trained in training and are completely independent of the model to be trained, they lack relatively clear semantic information and cannot be well combined with the model. The method provided by the above embodiments of the present disclosure can introduce the semantic information contained in the class activation map of the training sample into the data enhancement method by taking the class activation map generated based on the training sample and the corresponding target image processing model as a reference factor for generating the attention mask, so that the data enhancement method can make targeted adjustment according to specific data and training tasks, thereby improving the adaptability of the data enhancement method to different models and further helping to improve the training effect of the model. Moreover, since the data enhancement method of the present solution does not introduce additional variables and parameters and only utilizes the original structure of the model, the implementation is relatively simple.

[0090] Further reference Figure 5 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a data enhancement device, which corresponds to the method embodiment shown in Figure 1 , and the device can be specifically applied to various electronic devices.

[0091] As shown in Figure 5 , the data enhancement device 500 provided by the present embodiment includes an acquisition unit 501, a first generation unit 502 and a second generation unit 503. The acquisition unit 501 is configured to acquire a class activation map corresponding to a training sample for data enhancement, wherein the class activation map is generated based on a sample image of the training sample and a target image processing model trained by the training sample; the first generation unit 502 is configured to generate an attention mask matching the size of the sample image based on the class activation map corresponding to the training sample; and the second generation unit 503 is configured to perform data enhancement on the pre-acquired sample image based on the pre-acquired noise matrix and the attention mask to generate a new training sample corresponding to the training sample.

[0092] In the present embodiment, the specific processing of the acquisition unit 501, the first generation unit 502 and the second generation unit 503 in the data enhancement device 500 and the technical effects brought by the same can be respectively referred to the related descriptions of steps S101, S102 and S103 in the Figure 1 corresponding embodiments, which will not be described herein again.

[0093] In some optional implementations of the present embodiment, the first generation unit 502 can include a first generation module (not shown in the figure) configured to scale the class activation map corresponding to the training sample to generate an attention heat map consistent with the size of the sample image; and a second generation module (not shown in the figure) configured to generate an attention mask consistent with the size of the attention heat map based on a comparison between the values in the attention heat map and the target value.

[0094] In some optional implementations of the present embodiment, the target value can include an average value determined based on the attention heat map.

[0095] In some optional implementations of the present embodiment, the second generation module can be further configured to: determine a preset first value as the corresponding value in the attention mask in response to determining that the value in the attention heat map is less than the target value; and determine a preset second value as the corresponding value in the attention mask in response to determining that the value in the attention heat map is greater than or equal to the target value, wherein the preset first value is less than the preset second value.

[0096] In some optional implementations of the present embodiment, the size of the attention mask can be consistent with the size of the sample image. The second generation unit 502 can include a third generation module (not shown in the figure) configured to multiply the noise matrix pre-acquired and consistent with the size of the sample image with the attention mask element by element to generate a first noise matrix; a fourth generation module (not shown in the figure) configured to generate a new sample image corresponding to the sample image based on a fusion of the first noise matrix and the sample image; and a fifth generation module (not shown in the figure) configured to generate a new training sample based on the new sample image and the sample label information of the training sample.

[0097] In some optional implementations of the present embodiment, the fourth generation module can be further configured to add the first noise matrix and the sample image element by element to generate the new sample image.

[0098] In some optional implementations of the present embodiment, the fourth generation module can be further configured to correct the elements belonging to a preset value in the first noise matrix to 1 to generate a second noise matrix; and multiply the second noise matrix and the sample image element by element to generate the new sample image.

[0099] In some optional implementations of the present embodiment, the parameters of the target image processing model can be the saved model parameters with the best effect on the validation set corresponding to the training sample after being trained based on the training sample.

[0100] The device provided by the above embodiments of the present disclosure can introduce semantic information contained in the class activation map of the training sample into the data enhancement method, so that the data enhancement method can make targeted adjustments according to specific data and training tasks. Thus, the adaptability of the data enhancement method to different models can be improved, and the training effect of the model can be improved. Moreover, the data enhancement method of the present solution does not introduce additional variables and parameters, and can be completed only by using the original structure of the model, and the implementation is relatively simple.

[0101] With further reference to Figure 6 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a device for training an image processing model, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied in various electronic devices.

[0102] As shown in Figure 6 , the device for training an image processing model provided by the present embodiment includes a model obtaining unit 601 and a training unit 602. The model obtaining unit 601 is configured to obtain an image processing pre-trained model trained based on a training sample. The training unit 602 is configured to finely train the image processing pre-trained model based on a new training sample corresponding to the training sample to obtain a trained image processing model. The new training sample is obtained according to the data enhancement method described in the foregoing embodiments.

[0103] In the present embodiment, in the device 600 for training an image processing model: the specific processing of the model obtaining unit 601 and the training unit 602 and the technical effects brought by the specific processing can be respectively referred to the related descriptions of steps S201 and S202 in the corresponding embodiments, which will not be repeated here. Figure 2

[0104] The device provided by the above embodiments of the present disclosure can introduce semantic information contained in the class activation map of the training sample into the data enhancement method, so that the data enhancement method can make targeted adjustments according to specific data and training tasks. Thus, the adaptability of the data enhancement method to different models can be improved, and the training effect of the model can be improved. Moreover, the data enhancement method of the present solution does not introduce additional variables and parameters, and can be completed only by using the original structure of the model, and the implementation is relatively simple.

[0105] With further reference to Figure 7 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a device for training an image processing model, which corresponds to the method embodiment shown in Figure 3 ​The method embodiment shown corresponds to the device, which can be specifically applied to various electronic devices.

[0106] As Figure 7 The device for image processing 700 provided in this embodiment includes an image acquisition unit 701 and a processing unit 702. The image acquisition unit 701 is configured to acquire an image to be processed. The processing unit 702 is configured to input the image to be processed into a pre-trained image processing model to generate an image processing result, wherein the image processing model is obtained through the method for training an image processing model as described in the foregoing embodiments.

[0107] In this embodiment, the specific processing of the image acquisition unit 701 and the processing unit 702 in the device for image processing 700 and the technical effects brought by the specific processing can be respectively referred to the specific processing of the image acquisition unit 701 and the processing unit 702 and the technical effects brought by the specific processing in the method for training an image processing model. Figure 3 The related description of steps S301 and S302 in the corresponding embodiments will not be repeated here.

[0108] The device provided in the foregoing embodiments of the present disclosure can use the image processing model obtained through the method for training an image processing model to process the image acquired by the image acquisition unit 701. Since the image processing model is trained using the new training samples obtained through the data enhancement method associated with the training samples and the model, the effect of image processing can be improved.

[0109] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information involved comply with relevant laws and regulations, necessary security measures are taken, and the public order and good customs are not violated.

[0110] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0111] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0112] As Figure 8As shown, the device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0113] A plurality of components in the device 800 are connected to the I / O interface 805, including an input unit 806 such as a keyboard, a mouse, etc., an output unit 807 such as various types of displays, speakers, etc., a storage unit 808 such as a magnetic disk, an optical disk, etc., and a communication unit 809 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0114] The computing unit 801 can be various general and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as a data augmentation method, a method for training an image processing model, or a method for image processing. For example, in some embodiments, the data augmentation method, the method for training an image processing model, or the method for image processing can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data augmentation method, the method for training an image processing model, or the method for image processing described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the data augmentation method, the method for training an image processing model, or the method for image processing by any other appropriate means, such as by means of firmware.

[0115] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0116] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0117] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0118] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0119] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0120] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0121] It should be understood that various forms of flow shown above can be used, with steps reordered, added, or removed. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without limitation herein, so long as the desired results of the technology disclosed in the present disclosure are achieved.

[0122] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, changes, improvements, and the like that come within the spirit and principles of the present disclosure should be considered within the scope of the present disclosure.

Claims

1. A data enhancement method, comprising: obtaining a class activation map corresponding to a training sample for data enhancement, wherein the class activation map is generated based on a sample image of the training sample and a target image processing model trained by the training sample; generating an attention mask matching a size of the sample image based on the class activation map corresponding to the training sample, comprising: performing a scale transformation on the class activation map corresponding to the training sample to generate an attention heat map consistent with the size of the sample image; generating an attention mask consistent with the size of the attention heat map based on a comparison between values in the attention heat map and target values, wherein the target values include an average value determined based on each pixel in the attention heat map, and a distribution of attention weights included in the attention mask is consistent with a distribution of values in the class activation map; performing data enhancement on the pre-obtained sample image based on a pre-obtained noise matrix and the attention mask to generate a new training sample corresponding to the training sample, comprising: multiplying the pre-obtained noise matrix consistent with the size of the sample image and the attention mask element by element to generate a first noise matrix, wherein the size of the attention mask is consistent with the size of the sample image; generating a new sample image corresponding to the sample image based on fusion of the first noise matrix and the sample image; and generating the new training sample based on the new sample image and sample annotation information of the training sample.

2. The method of claim 1, wherein, The generating of the attention mask consistent with the size of the attention heat map based on the comparison between the values in the attention heat map and the target values comprises: in response to determining that a value in the attention heat map is less than the target value, determining a preset first value as a corresponding value in the attention mask; in response to determining that a value in the attention heat map is greater than or equal to the target value, determining a preset second value as a corresponding value in the attention mask, wherein the preset first value is less than the preset second value.

3. The method of claim 1, wherein, The generating of the new sample image corresponding to the sample image based on the fusion of the first noise matrix and the sample image comprises: adding the first noise matrix and the sample image element by element to generate the new sample image.

4. The method of claim 1, wherein, The generating of the new sample image corresponding to the sample image based on the fusion of the first noise matrix and the sample image comprises: correcting elements belonging to a preset value in the first noise matrix to 1 to generate a second noise matrix; multiplying the second noise matrix and the sample image element by element to generate the new sample image.

5. The method according to one of claims 1 to 4, wherein Parameters of the target image processing model are saved model parameters with the best effect on a validation set corresponding to the training sample after being trained based on the training sample.

6. A method for training an image processing model, comprising: obtaining an image processing pre-training model trained based on a training sample; performing fine training on the image processing pre-training model based on a new training sample corresponding to the training sample to obtain a trained image processing model, wherein the new training sample is obtained by performing data enhancement on a pre-obtained sample image based on a pre-obtained noise matrix and an attention mask. The method according to any one of claims 1-5.

7. A method for image processing, comprising: obtaining an image to be processed; inputting the image to be processed into a pre-trained image processing model to generate an image processing result, wherein the image processing model is trained according to the method of claim 6.

8. A data enhancement apparatus, comprising: an obtaining unit configured to obtain a class activation map corresponding to a training sample for data enhancement, wherein the class activation map is generated based on a sample image of the training sample and a target image processing model trained by the training sample; a first generating unit configured to generate an attention mask matching a size of the sample image based on the class activation map corresponding to the training sample; a second generating unit configured to perform data enhancement on the pre-obtained sample image based on a pre-obtained noise matrix and the attention mask to generate a new training sample corresponding to the training sample; the first generating unit comprises a first generating module configured to: perform a scale transformation on the class activation map corresponding to the training sample to generate an attention heat map consistent with the size of the sample image; and generate an attention mask consistent with the size of the attention heat map based on a comparison between values in the attention heat map and target values, wherein the target values include an average value determined based on each pixel in the attention heat map, and a distribution of attention weights contained in the attention mask is consistent with a distribution of values in the class activation map; the second generating unit comprises: a third generating module configured to multiply a pre-obtained noise matrix consistent with the size of the sample image and the attention mask element by element to generate a first noise matrix, wherein the size of the attention mask is consistent with the size of the sample image; a fourth generating module configured to generate a new sample image corresponding to the sample image based on fusion of the first noise matrix and the sample image; and a fifth generating module configured to generate the new training sample based on the new sample image and sample annotation information of the training sample.

9. An apparatus for training an image processing model, comprising: a model obtaining unit configured to obtain an image processing pre-trained model trained based on a training sample; a training unit configured to perform fine training on the image processing pre-trained model based on a new training sample corresponding to the training sample to obtain a trained image processing model, wherein the new training sample the method according to any one of claims 1-5.

10. An apparatus for image processing, comprising: an image obtaining unit configured to obtain an image to be processed; a processing unit configured to input the image to be processed into a pre-trained image processing model to generate an image processing result, wherein the image processing model is trained according to the method of claim 6.

11. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are for causing the computer to perform the method of any one of claims 1-7.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-7.

Images