Image enhancement method and system based on label relevance

Abstract

The application discloses an image enhancement method and system based on label correlation degree; wherein the method comprises: obtaining a training set; inputting the original image in the training set after preprocessing into a feature extraction network to output a feature map; inputting the feature map into an attention activation module to perform F times convolution operation to obtain an attention activation map; the attention activation map has F channels; the jth channel is selected from the F channels; an area associated with a label is found from the attention activation map of the jth channel, and the pixel value of the associated area is set to zero; the attention activation map of the jth channel after the zero setting operation is used as a mask map; the original image and the mask map are operated to finally obtain an enhanced original image; other images in the training set are enhanced in the same way to obtain an enhanced training set; the enhanced training set is input into a classification network to train the classification network to obtain a trained classification network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to an image enhancement method and system based on label relevance. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Image recognition technology based on convolutional neural network model has been widely applied to our life, but due to the complex and changeable scene in real life, the robustness of the model is greatly challenged, and most recognition models will have poor recognition effect when facing these noise data, in view of this phenomenon, researchers propose to add these disturbances in the data processing stage, and then use the enhanced data to train the model, so as to improve the robustness and generalization of the model.

[0004] Data enhancement is an effective regularization method, compared with other regularization methods, data enhancement has the advantages of easy implementation, no additional calculation and strong portability.

[0005] At present, the data enhancement method commonly used in convolutional neural network includes disturbance of image pixel space in addition to traditional geometric transformation, in recent years, a data enhancement method based on information erasure has been proposed, which has become a research hotspot, the principle of this method is to delete the continuous pixel information on the input image through a certain strategy, although a large number of experiments have proved that these methods can significantly improve the generalization and robustness of deep learning model, but they cannot avoid the problems of over-erasure and under-erasure, over-erasure refers to the fact that the erasure area is too large, so that the target is completely erased, and the remaining image information is not enough to classify it, which will bring noise to the model, under-erasure refers to the fact that the background area unrelated to the target is erased, which cannot bring effect improvement to the model.

[0006] Therefore, in subsequent research, some researchers divide the picture into a 4*4 size image block and randomly select to erase each block area with a probability of 0.5, although it has been improved, but still cannot solve the two problems. Some researchers try to avoid this problem by extracting a saliency map, the specific method is: first, extract the key features of the target in the picture through the saliency map, then set a threshold when selecting the erasure area, the area below the threshold can be erased, so as to avoid the problem of over-erasure, but this method does not solve the problem of under-erasure. SUMMARY

[0007] In order to solve the problem of how to select a suitable erasure area in information erasure in the prior art, the present application provides an image enhancement method and system based on label relevance.

[0008] In a first aspect, the present application provides an image enhancement method based on label relevance;

[0009] The image enhancement method based on label relevance comprises:

[0010] obtaining a training set; the training set comprises original images with known classification labels;

[0011] after preprocessing the original images in the training set, inputting the original images into a feature extraction network to output a feature map; the feature map comprises C channels; C is a positive integer;

[0012] inputting the feature map into an attention activation module to perform F times of convolution operation to obtain an attention activation map; the attention activation map comprises F channels; F is a positive integer; selecting a jth channel from the F channels; wherein j is greater than or equal to 1 and less than or equal to F; finding a region associated with a label from the jth channel of the attention activation map, setting pixel values of the associated region to zero, and taking the jth channel of the attention activation map after the zero setting operation as a mask map;

[0013] performing operation on the original image and the mask map to finally obtain an enhanced original image; enhancing other images in the training set in the same way to obtain an enhanced training set;

[0014] inputting the enhanced training set into a classification network to train the classification network to obtain a trained classification network.

[0015] In a second aspect, the present application provides an image enhancement system based on label relevance;

[0016] The image enhancement system based on label relevance comprises:

[0017] an obtaining module configured to obtain a training set; the training set comprises original images with known classification labels;

[0018] a feature extraction module configured to preprocess the original images in the training set, input the original images into a feature extraction network, and output a feature map; the feature map comprises C channels; C is a positive integer;

[0019] an attention activation module configured to input the feature map into the attention activation module to perform F times of convolution operation to obtain an attention activation map; the attention activation map comprises F channels; F is a positive integer; select a jth channel from the F channels; wherein j is greater than or equal to 1 and less than or equal to F; find a region associated with a label from the jth channel of the attention activation map, set pixel values of the associated region to zero, and take the jth channel of the attention activation map after the zero setting operation as a mask map;

[0020] The image enhancement module is configured to: operate the original image and the mask image, and finally obtain an enhanced original image; and enhance other images in the training set in the same manner to obtain an enhanced training set.

[0021] The training module is configured to: input the enhanced training set into the classification network to train the classification network, and obtain a trained classification network.

[0022] In a third aspect, the present application further provides an electronic device comprising:

[0023] a memory for non-transiently storing computer readable instructions; and

[0024] a processor for running the computer readable instructions,

[0025] When the computer readable instructions are run by the processor, the method of the first aspect is executed.

[0026] In a fourth aspect, the present application further provides a storage medium for non-transiently storing computer readable instructions, wherein when the non-transient computer readable instructions are executed by a computer, the instructions of the method of the first aspect are executed.

[0027] Compared with the prior art, the present application has the following beneficial effects:

[0028] The feature erasing method based on image classification of the present application measures the label correlation degree between each feature in the image through an attention activation map, so as to ensure that the erased region is a feature related to the target, and most of the key features are retained. It is ensured that the training data has certain interference, so as to solve the problem of overfitting of the convolutional neural network, so as to improve the robustness and generalization of the model. BRIEF DESCRIPTION OF DRAWINGS

[0029] The drawings accompanying the specification of the present application form a part thereof and serve to provide further understanding of the present application, the illustrative embodiments of the present application and its description serve to explain the present application, and do not constitute an improper limitation of the present application.

[0030] Figure 1 The method flowchart of the first embodiment is shown in the following figure;

[0031] Figure 2 The mask image generation schematic diagram of the first embodiment is shown in the following figure. DETAILED DESCRIPTION

[0032] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0033] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and "comprising", when used in this specification, specify the presence of stated features, integers, steps, or components, but do not preclude the presence or addition of one or more other features, integers, steps, components, or groups thereof.

[0034] The embodiments in the present application and the features in the embodiments can be combined with each other in the case of no conflict.

[0035] All data acquisition of the embodiments is based on the compliance with laws and regulations and user consent, and the legal application of data.

[0036] Embodiment one

[0037] The embodiment provides an image enhancement method based on label relevance;

[0038] The image enhancement method based on label relevance comprises the following steps:

[0039] S101: acquiring a training set; the training set comprises original images with known classification labels;

[0040] S102: after the original images in the training set are preprocessed, the original images are input into a feature extraction network, and a feature map is output; the feature map comprises C channels; C is a positive integer;

[0041] S103: the feature map is input into an attention activation module to perform F times of convolution operation, and an attention activation map is obtained; the attention activation map comprises F channels; F is a positive integer;

[0042] A jth channel is selected from the F channels; wherein j is greater than or equal to 1 and less than or equal to F; an area associated with a label is found from the jth channel of the attention activation map, pixel values of the associated area are set to zero, and the jth channel of the attention activation map after the zero setting operation is taken as a mask map;

[0043] S104: the original image and the mask map are operated, and finally an enhanced original image is obtained; other images of the training set are enhanced in the same way, and an enhanced training set is obtained;

[0044] S105: the enhanced training set is input into a classification network to train the classification network, and a trained classification network is obtained.

[0045] Further, the feature extraction network is implemented by using a convolutional neural network.

[0046] Further, the data in the training set is preprocessed.

[0047] Further, the S102: the original image in the training set is preprocessed and input into the feature extraction network, and a feature map is output; specifically including:

[0048] In each training step, an original image in the training set is input into the feature extraction network, and a feature map is output; wherein the feature map contains C channels, and the number of channels is determined by the number of convolutions.

[0049] Further, the S103: the attention activation module is implemented by using a 1*1 convolution layer.

[0050] The 1*1 convolution layer is used to combine the activation values of each channel of the feature map to obtain an attention activation map.

[0051] Further, the jth channel is selected from the F channels; wherein the selection is performed by using a random selection method.

[0052] Further, the region associated with the label is found in the attention activation map of the jth channel, the pixel values of the associated region are set to zero, and the attention activation map of the jth channel after the zero setting operation is used as a mask map; specifically including:

[0053] A rectangular box with a size of a*b is set; the rectangular box slides in the attention activation map of the jth channel according to a set step size;

[0054] Each sliding calculates the average pixel value of all pixel points in the image in the rectangular box;

[0055] If the average pixel value of all pixel points in the image in the rectangular box is greater than a set threshold, it indicates that the image in the rectangular box is related to the label, and the pixels of all pixel points in the image in the rectangular box are set to zero, and the attention activation map of the jth channel after the zero setting operation is used as a mask map.

[0056] If the average pixel value of all pixel points in the image in the rectangular box is less than a set threshold, it indicates that the image in the rectangular box is less related to the label, and the original value of all pixel points in the image in the rectangular box is retained.

[0057] The size of the rectangular box is set to 20% of the size of the feature map.

[0058] Further, the S104: the original image and the mask image are operated, and finally the enhanced original image is obtained;Specifically, the original image and the mask image are operated, and the enhanced original image is obtained.

[0059] Further, the S105: the enhanced training set is input into the classification network to train the classification network, and the trained classification network is obtained;The classification network is a convolutional neural network.

[0060] The feature erasing method based on image classification of the application utilizes the attention activation map to find the correlation of each feature and label of the image target, and then selectively performs feature erasing, which comprises: in the data preprocessing stage, the data is enhanced to avoid overfitting. The original data is trained by the neural network to obtain the attention activation map. The mask map is generated according to the attention activation map, and the mask map is overlaid on the original map, so that the relevant features are erased.

[0061] Figure 1 For the model framework, the original training data is subjected to convolution operation to obtain a feature map, and then the attention activation map is obtained by activating each channel. A mask map is generated by randomly selecting a significant area of one channel and is overlaid on the original map to finally obtain enhanced data.

[0062] Figure 2 After the picture is subjected to feature extraction, a multi-channel feature map is obtained, and each channel feature map extracts a significant feature. Then, the feature activation is performed by the attention activation module to obtain an activation map. Since the shape of the significant area of the activation map is irregular, a mask map is formed by setting a rectangular frame in advance. Finally, the generated mask map is up-sampled and overlaid on the original map.

[0063] First, the original sample is preprocessed and input into the neural network to obtain the attention activation map. Since each channel corresponds to a certain visual feature in the original image, a 1*1 convolution layer is added after the last convolution layer of the convolutional neural network to detect the features corresponding to different semantics in the channel aspect, and the full connection layer is removed, so that the position information can be retained.

[0064] The label correlation can be obtained by the attention activation map to select the features to be erased. Assuming that the feature map output by the last convolution layer of the original CNN is F C×H×W , wherein C, H and W represent the number of channels of the feature map, the width and the height of the feature map respectively, the attention activation map can be represented as A C×H×WThat is, A has F channels, and the width and height are the same as the feature map F. The area with a larger response in the activation map is generally an area that significantly helps classification. A channel in the activation map is selected, a rectangular frame is slid in the activation map of the selected channel, and the average value of the pixels in the rectangular frame is compared with a set threshold value. A mask map is generated by measuring the importance of the rectangular area.

[0065] The sample of the erased part of the feature is added to the training sample. Dot product operation is performed on the generated mask map and the original map to obtain enhanced data. The enhanced data is used for training of the convolutional neural network, thereby improving the robustness and generalization of the model.

[0066] Embodiment two

[0067] The embodiment provides an image enhancement system based on label relevance;

[0068] The image enhancement system based on label relevance comprises:

[0069] An acquisition module configured to acquire a training set; the training set comprising original images with known classification labels;

[0070] A feature extraction module configured to input the original images in the training set into a feature extraction network after preprocessing, and output a feature map; the feature map comprising C channels; C is a positive integer;

[0071] An attention activation module configured to input the feature map into the attention activation module for F times of convolution operation to obtain an attention activation map; the attention activation map comprising F channels; F is a positive integer; a jth channel is selected from the F channels; wherein j is greater than or equal to 1 and less than or equal to F; an area associated with a label is found from the attention activation map of the jth channel, and the pixel value of the associated area is set to zero; and the attention activation map of the jth channel after the zero setting operation is used as a mask map;

[0072] An image enhancement module configured to perform operation on the original image and the mask map to finally obtain an enhanced original image; and other images in the training set are enhanced in the same way to obtain an enhanced training set;

[0073] A training module configured to input the enhanced training set into a classification network to train the classification network, and obtain a trained classification network.

[0074] It should be noted that the above acquisition module, feature extraction module, attention activation module, image enhancement module and training module correspond to steps S101 to S105 in Embodiment One, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment One. It should be noted that the above modules can be executed in a computer system such as a set of computer executable instructions as part of a system.

[0075] The description of each embodiment in the above embodiments has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0076] The proposed system can be implemented in other ways. For example, the system embodiments described above are only illustrative, for example, the division of the above modules is only a logical function division, and in actual implementation, there can be another division method, for example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0077] Embodiment Three

[0078] The embodiment also provides an electronic device, including one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method described in Embodiment One.

[0079] It should be understood that in the embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0080] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0081] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software.

[0082] The method in the embodiment one can be directly embodied as being completed by a hardware processor, or being completed by a combination of hardware and software modules in the processor. The software modules can be located in a storage medium in the art, such as a random access memory, a flash memory, a read only memory, a programmable read only memory, an electrically erasable programmable memory, a register, or the like. The storage medium is located in the storage, and the processor reads information in the storage, and combines the hardware to complete the steps of the above method. To avoid repetition, no further detailed description is given here.

[0083] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0084] Embodiment four

[0085] The embodiment also provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in embodiment one.

[0086] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image enhancement method based on label relevance, characterized in that, The method comprises the following steps: obtaining a training set; the training set comprises original images with known classification labels; after preprocessing the original images in the training set, inputting the original images into a feature extraction network to output a feature map; the feature map comprises C channels; C is a positive integer; inputting the feature map into an attention activation module to perform F times of convolution operation to obtain an attention activation map; the attention activation map comprises F channels; F is a positive integer; selecting the jth channel from the F channels; wherein j is greater than or equal to 1 and less than or equal to F; finding a region associated with the label from the attention activation map of the jth channel, setting the pixel values of the associated region to zero, and taking the attention activation map of the jth channel after the zero setting operation as a mask map, which specifically comprises: setting a rectangular box; the rectangular box slides in the attention activation map of the jth channel according to a set step size; each time the rectangular box slides, the average pixel value of all pixel points in the image in the rectangular box is calculated; if the average pixel value of all pixel points in the image in the rectangular box is greater than a set threshold value, it indicates that the image in the rectangular box is highly relevant to the label, and the pixels of all pixel points in the image in the rectangular box are set to zero, and the attention activation map of the jth channel after the zero setting operation is taken as the mask map; if the average pixel value of all pixel points in the image in the rectangular box is less than a set threshold value, it indicates that the image in the rectangular box is less relevant to the label, and the original values of all pixel points in the image in the rectangular box are retained; performing operation on the original image and the mask map to finally obtain an enhanced original image; other images in the training set are enhanced in the same way to obtain an enhanced training set; inputting the enhanced training set into a classification network to train the classification network, and obtaining a trained classification network.

2. The image enhancement method based on label relevance as claimed in claim 1, wherein, The feature extraction network is implemented by using a convolutional neural network.

3. The image enhancement method based on label relevance as claimed in claim 1, wherein, After preprocessing the original images in the training set, wherein the preprocessing refers to size normalization processing of each original image in the training set.

4. The image enhancement method based on label relevance as claimed in claim 1, wherein, After preprocessing the original images in the training set, inputting the original images into the feature extraction network to output a feature map; specifically comprising: in each training step, inputting an original image in the training set into the feature extraction network to output a feature map; wherein the feature map comprises C channels, and the number of channels is determined by the number of convolutions.

5. The image enhancement method based on label relevance as claimed in claim 1, wherein, The attention activation module is implemented by using a 1*1 convolution layer.

6. The image enhancement method based on label relevance as claimed in claim 1, wherein, performing operation on the original image and the mask map to finally obtain an enhanced original image; specifically, performing dot product operation on the original image and the mask map to obtain the enhanced original image; inputting the enhanced training set into a classification network to train the classification network, and obtaining a trained classification network; wherein the classification network is a convolutional neural network.

7. An image enhancement system based on tag relevance, characterized in that, The method comprises the following steps: an obtaining module configured to obtain a training set; the training set comprises original images with known classification labels; a feature extraction module configured to preprocess the original images in the training set, input the original images into a feature extraction network, and output a feature map; the feature map comprises C channels; C is a positive integer; The attention activation module is configured to input the feature map into the attention activation module for F times of convolution operation to obtain an attention activation map; the attention activation map has F channels; F is a positive integer; a jth channel is selected from the F channels; wherein j is greater than or equal to 1 and less than or equal to F; an area associated with the label is found from the attention activation map of the jth channel, and pixel values of the associated area are set to zero; the attention activation map of the jth channel after the zero setting operation is taken as a mask map, and specifically includes: a rectangular box is set; the rectangular box slides in the attention activation map of the jth channel according to a set step length; each time the sliding is performed, a pixel average value of all pixel points in the image in the rectangular box is calculated; if the pixel average value of all pixel points in the image in the rectangular box is greater than a set threshold value, it is indicated that the image in the rectangular box is highly relevant to the label, and pixels of all pixel points in the image in the rectangular box are subjected to a zero setting operation; and the attention activation map of the jth channel after the zero setting operation is taken as the mask map; if the pixel average value of all pixel points in the image in the rectangular box is less than the set threshold value, it is indicated that the image in the rectangular box is less relevant to the label, and original values of all pixel points in the image in the rectangular box are retained; the image enhancement module is configured to perform operation on the original image and the mask map to finally obtain an enhanced original image; other images in a training set are enhanced in the same manner to obtain an enhanced training set; the training module is configured to input the enhanced training set into a classification network to train the classification network, and obtain a trained classification network. 8.An electronic device, comprising: a memory for non-transiently storing computer readable instructions; and a processor for running the computer readable instructions, wherein the computer readable instructions are run by the processor to perform the method of any one of claims 1-6.

9. A storage medium characterized by, non-transiently storing computer readable instructions, wherein when the non-transient computer readable instructions are executed by a computer, instructions of the method of any one of claims 1-6 are executed.

Images