An image labeling method based on neural network interpretability

Abstract

The application discloses an image labeling method based on neural network interpretability, comprising: classifying an obtained image to obtain a correct class label of the image; training a convolutional neural network by using the image with the class label; taking the trained convolutional neural network as a neural network to be explained, taking an image to be labeled and the correct class label of the image as input, and using a neural network interpretability algorithm to explain the trained convolutional neural network; obtaining a heat map related to the input image and the image class after the explanation process is completed; and obtaining a semantic segmentation label and a target detection label of the image to be labeled based on the heat map. The application can generate the target detection label and the semantic segmentation label according to the class label, thereby greatly reducing the labeling workload of the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning computer vision, and in particular to an image labeling method based on neural network interpretability. BACKGROUND

[0002] Deep learning has developed rapidly in recent years, and convolutional neural networks are a kind of deep learning, which plays an important role in the field of computer vision. Common convolutional neural networks include AlexNet, VGG, ResNet, etc. The basic functions of these neural networks are feature extraction and classification. Convolutional neural networks are also increasingly applied to target detection and semantic segmentation tasks, such as Yolo, SSD, etc.

[0003] Convolutional neural networks require a large amount of data sets to complete model training, so data needs to be labeled according to the type of task. Labeling class labels is the simplest, which only needs to give the class of the image, such as: "cat", "dog", "rabbit"; labeling target detection labels needs to label the position of the target object from the image, which is generally a rectangular box; and labeling semantic segmentation labels needs to label the boundary of the target object with an irregular polygon. Obviously, the workload of labeling semantic labels is several times that of labeling class labels.

[0004] There have been many research results on neural network interpretability in recent years, among which the Shapley value method, the saliency map method, and the class activation map method can effectively give the attention distribution of the convolutional neural network to a specific input image, and give it in the form of a heat map. In particular, the class activation map method outperforms the other two methods. Among the class activation map methods, Grad-CAM has gained the recognition of many researchers. The heat map output by these interpretability methods can to some extent reflect the position and boundary of the target object. Figure One

[0005] Existing data labeling methods mostly rely on manual work, especially the labeling of semantic labels. Since the outline of the target object needs to be labeled completely, the operation complexity is high, and it takes a lot of time and effort for labeling engineers to complete it. When a large number of samples are needed, a large number of workers are often involved in data labeling work. SUMMARY

[0006] The present application provides an image labeling method based on neural network interpretability to solve the technical problem of low efficiency in labeling target detection labels and semantic segmentation labels for images in the prior art.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] ​In one aspect, the present application provides an image labeling method based on neural network interpretability, comprising:

[0009] classifying the obtained images to obtain correct class labels of the images;

[0010] training a preset convolutional neural network using images with class labels;

[0011] using the trained convolutional neural network as the neural network to be explained, using the image to be labeled and its corresponding correct class label as input, using a preset neural network interpretability algorithm to interpret the trained convolutional neural network, and obtaining a heat map related to the input image and the image class after the interpretation process is completed;

[0012] obtaining semantic segmentation labels and object detection labels of the image to be labeled based on the heat map.

[0013] Further, the correct class label of the image is a correct classification label obtained by manual labeling, each class has a unique label, and the labels are in a parallel relationship without a belonging relationship.

[0014] Further, the convolutional neural network is composed of a convolutional layer and a classification layer; wherein the convolutional layer includes at least one neural network layer composed of a convolutional kernel, and the classification layer includes at least one fully connected layer.

[0015] Further, the training of the preset convolutional neural network using images with class labels comprises:

[0016] dividing the images and class labels into a training set and a validation set according to a preset ratio, and enhancing the training set using a preset data enhancement method; wherein the data enhancement method includes image inversion, image rotation, channel equalization, brightness adjustment, sharpening, blurring, and random cropping;

[0017] training the preset convolutional neural network using the enhanced training set; after completing the training, measuring the accuracy of the convolutional neural network on the validation set, and when the accuracy is lower than a preset accuracy threshold, increasing the training set and adjusting the network hyperparameters to improve the accuracy of the convolutional neural network.

[0018] Further, after training the preset convolutional neural network using the enhanced training set, the training of the preset convolutional neural network using images with class labels further comprises:

[0019] merging the enhanced validation set into the enhanced training set after enhancing the validation set using the preset data enhancement method, and retraining the convolutional neural network to obtain a final trained convolutional neural network.

[0020] Further, based on the heat map, semantic segmentation labels and object detection labels of the image to be labeled are obtained, comprising:

[0021] The heat map is converted into semantic segmentation labels using a threshold method.

[0022] The smallest rectangle containing the semantic segmentation labels is found as the object detection label using a minimum rectangle method.

[0023] Further, the heat map is converted into semantic segmentation labels using a threshold method, comprising:

[0024] A threshold is set to convert the heat map into a binary image.

[0025] The binary image is directly taken as the semantic segmentation label of the corresponding input image, or the semantic segmentation label of the corresponding input image is obtained after subsequent processing of the binary image.

[0026] Further, the threshold is set to be the average of the minimum value and the maximum value of the heat map.

[0027] Further, the smallest rectangle containing the semantic segmentation labels is found using a minimum rectangle method, comprising:

[0028] The largest independent region in the semantic label is found.

[0029] The minimum value of the largest independent region on the x-axis and the y-axis is found to obtain a coordinate as a starting point, and the maximum value of the largest independent region on the x-axis and the y-axis is found to obtain a coordinate as an ending point.

[0030] The starting point and the ending point are taken as two vertices of a rectangle to obtain a smallest rectangle containing the semantic segmentation labels and having edges parallel to the input image.

[0031] Further, after obtaining the smallest rectangle containing the semantic segmentation labels and having edges parallel to the input image, the smallest rectangle containing the semantic segmentation labels is found using a minimum rectangle method, further comprising:

[0032] The smallest rectangle is expanded by 1 pixel outward, and the coordinates of the expanded rectangle are taken as the object detection label.

[0033] In another aspect, the present application also provides an electronic device comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above method.

[0034] In another aspect, the present application also provides a computer readable storage medium, wherein the storage medium stores at least one instruction, which is loaded and executed by the processor to implement the above method.

[0035] The technical solution provided by the present application can generate target detection labels and semantic labels by only classifying images and using a convolutional neural network and a neural network explainability method. Optionally, the technical solution provided by the present application can also improve the quality of the labels by post-processing the generated labels according to a heat map. The beneficial effects brought by the technical solution include at least:

[0036] 1. The neural network explainability method used in the present application can convert a semantic segmentation labeling task into a class labeling task, thereby greatly reducing the workload of manual labeling.

[0037] 2. The present application can not only output semantic segmentation labels, but also output target detection labels. DETAILED DESCRIPTION

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 is an execution flow diagram of the image labeling method based on neural network explainability provided by the first embodiment of the present application;

[0040] Figure 2 is an execution flow diagram of the image labeling method based on neural network explainability provided by the second embodiment of the present application;

[0041] Figure 3 is a flow diagram of converting a heat map into a label provided by the second embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0043] First Embodiment

[0044] The present embodiment provides an image labeling method based on neural network explainability, which can be implemented by an electronic device. The execution flow of the method is shown in Figure 1 , which includes the following steps:

[0045] S1, classifying the obtained images to obtain correct class labels of the images;

[0046] It should be noted that the correct class label of the image is the correct classification label annotated by a human, each class has a unique label, and there is no parent-child relationship between the labels.

[0047] S2, training a preset convolutional neural network using images with class labels;

[0048] The convolutional neural network is composed of a convolutional layer and a classification layer; the convolutional layer includes at least one neural network layer composed of a convolutional kernel, and the classification layer includes at least one fully connected layer.

[0049] The process of training the preset convolutional neural network using images with class labels specifically includes:

[0050] S21, dividing the images and class labels into a training set and a validation set according to a preset ratio, and enhancing the training set using a preset data enhancement method; the data enhancement method includes image inversion, image rotation, channel equalization, brightness adjustment, sharpening, blurring, and random cropping;

[0051] S22, training the preset convolutional neural network using the enhanced training set; after the training is completed, the accuracy of the convolutional neural network is measured on the validation set, and when the accuracy is lower than a preset accuracy threshold, the accuracy of the convolutional neural network is improved by increasing the training set and adjusting the network hyperparameters.

[0052] Further, after training the preset convolutional neural network using the enhanced training set, to improve the network performance, the training process of the convolutional neural network can further include:

[0053] S23, merging the enhanced validation set into the enhanced training set after enhancing the validation set using the preset data enhancement method, and retraining the convolutional neural network to obtain a final trained convolutional neural network.

[0054] S3, using the trained convolutional neural network as the neural network to be explained, using the image to be annotated and its corresponding correct class label as input, and using a preset neural network explainability algorithm to explain the trained convolutional neural network, to obtain a heat map related to the input image and the image class after the explanation process is completed.

[0055] The neural network explainability algorithm includes three types of methods: SHAP, SaliencyMap, and CAM, and gives the explanation result in the form of a heat map (also known as a saliency map or attention map). The CAM method includes CAM, GradCAM, GradCAM++, SmoothGradCAM, ScoreCAM, and GroupCAM.

[0056] Furthermore, it should be noted that the heatmap changes with the input image, the selected neural network interpretability method, and the interpreted neural network layer; when generating the heatmap, the category labels input into the neural network interpretability algorithm are the correct category labels, not the prediction results of the convolutional neural network; and the convolutional neural network layer corresponding to the heatmap can be any convolutional layer of a trained convolutional neural network and its subsequent activation layers; the resolution of the heatmap can be lower than that of the input image. If the resolution of the heatmap is different from that of the input image, the heatmap needs to be scaled to the same resolution as the input image.

[0057] S4. Based on the heatmap, obtain the semantic segmentation label and object detection label of the image to be labeled.

[0058] Specifically, in this embodiment, S4 includes:

[0059] S41, The heatmap is converted into semantic segmentation labels using a thresholding method;

[0060] The thresholding method converts a heatmap into a binary image by setting a threshold. This binary image can be used directly as a semantic segmentation label for the corresponding input image, or it can be processed appropriately to obtain a semantic segmentation label. The threshold is the average of the minimum and maximum values ​​of the heatmap.

[0061] S42, use the minimum rectangle method to find the minimum rectangle containing the semantic segmentation label as the target detection label;

[0062] The minimum rectangle method takes semantic segmentation labels as input and outputs the position of a rectangle, where the sides of the rectangle are parallel to the input image. The process is as follows: First, find the largest independent region in the semantic labels. Then, find the minimum value of this independent region on the x-axis and y-axis to obtain a coordinate as the starting point. Next, find the maximum value of this independent region on the x-axis and y-axis to obtain a coordinate as the ending point. The two vertices of the minimum rectangle are the starting point and the ending point, respectively. At this time, the minimum rectangle overlaps with the target in the label by 1 pixel. Therefore, the minimum rectangle can be expanded outward by 1 ring (1 pixel) to obtain the final target detection label.

[0063] In summary, this embodiment provides an image annotation method based on neural network interpretability. By employing neural network interpretability, only image classification is required, and then object detection labels and semantic labels can be generated using convolutional neural networks and neural network interpretability methods, thus improving annotation efficiency. Furthermore, the generated labels can be post-processed based on heatmaps to further enhance label quality.

[0064] Second Embodiment

[0065] The embodiment combines an actual application example to illustrate the implementation process of the image labeling method based on neural network explainability of the application, as shown in Figure 2 The execution flow includes the following steps:

[0066] S101, label the acquired image class label;

[0067] Specifically, the image data set prepared in the embodiment includes two types of images, one is 297 corrosion metal images, and the other is 113 uncorroded metal images; the total number of the two types of data is 410. The labels have been labeled as “Corrosion” and “Uncorrosion”.

[0068] S102, divide the image and the class label into a training set and a validation set, and enhance the training set;

[0069] In the embodiment, the image data set is randomly divided into a training set and a validation set according to a ratio of 8:2, and the training set is enhanced by using image inversion, desaturation, rotation, channel equalization, brightness adjustment, smoothing, sharpening and blurring, random cropping and the like, so as to obtain a total of 6213 images for training. The validation set is not enhanced. The image inversion includes left-right inversion, up-down inversion and center inversion; the brightness adjustment means that the original image and a pure black image are mixed according to a certain proportion; the random cropping only crops images with a resolution greater than the default input resolution of VGG16 (the convolutional neural network used in the embodiment is VGG16, which will be described in detail below), and the center area is reserved each time to ensure that the core content in the center of the image is not lost or less lost.

[0070] S103, according to the number of images and the number of classes, select a suitable convolutional neural network, use the enhanced training set for training, and measure the accuracy of the convolutional neural network on the validation set after training is completed;

[0071] The higher the accuracy is, the better it is, and when the accuracy is lower than a threshold, the accuracy needs to be improved by increasing the training set, adjusting the hyperparameters and the like before the next step. When the accuracy is high enough, it means that the current convolutional neural network has learned enough knowledge. Moreover, in order to further improve the performance of the convolutional neural network, the validation set can be enhanced according to the method described in S102, and then merged into the training set, and the convolutional neural network is retrained to obtain a final trained convolutional neural network model.

[0072] Specifically, the embodiment selects VGG16 neural network, changes the number of neurons in the last layer output layer to 2, corresponding to the number of categories of the data set (2 categories). Then set the loss function as cross-entropy function, use SGD algorithm for training, learning rate is set to 0.0001, momentum value 0.9. After reading the image using OpenCV, convert it to NumPy format, and subtract 128.0 from each pixel of each channel. After 60 rounds of iteration, the training is ended, and the accuracy is tested on the validation set, with an accuracy of more than 91%, meeting the standard;

[0073] Based on the above, the embodiment does not use the method of retraining, but directly uses the VGG16 convolutional neural network trained in S103 for subsequent steps.

[0074] S104, set the neural network to be explained as the VGG16 convolutional neural network trained in S103, input the image and the corresponding classification label, and use the Shapley value method, saliency map method or class activation map method to explain the different layers of the convolutional neural network to obtain the heat map.

[0075] Among them, the neural network layer to be explained is the relatively late layer in the VGG16 convolutional layer.

[0076] Specifically, the embodiment uses the Grad-CAM neural network explainability method to explain the neural network, and sets the neural network layer to be explained as "features.15". Then input the original 410 images and their correct classification categories into the Grad-CAM explainability algorithm batch by batch to obtain 410 heat maps.

[0077] S105: set a threshold to convert the heat map into a binary image, which is the semantic segmentation label corresponding to the input image.

[0078] Among them, the threshold is the average of the minimum and maximum values of the heat map. Specifically, the threshold of the embodiment is set to 0.5, and the threshold method is used to convert the heat map obtained in S104 into a binary image, i.e. semantic segmentation label. Specifically, all pixels exceeding 0.5 are set to the maximum value 1.0, and all pixels less than 0.5 are set to the minimum value 0.0. Due to the low quality of part of the semantic segmentation label, a small amount of manual correction work is required.

[0079] S106, use the minimum rectangle method to convert semantic segmentation into target detection label.

[0080] Among them, the minimum rectangle method is: find the largest independent area in the semantic segmentation label, find the minimum value of the independent area on the x-axis and y-axis, and get the coordinates as the starting point, and then find the maximum value of the independent area on the x-axis and y-axis, and get the coordinates as the end point. Then the two vertices of the minimum rectangle are the starting point and the end point respectively; in addition, the four edges of the minimum rectangle are parallel to the edges of the heat map, so a minimum rectangle can be obtained, and the coordinates of the rectangle are the target detection label.

[0081] In addition, it needs to be pointed out that since there is 1 pixel overlap between the minimum rectangle and the target in the label at this time, the minimum rectangle can also be expanded outward by 1 circle (1 pixel) to obtain the final target detection label.

[0082] S107, the poor quality target detection label and the semantic segmentation label are appropriately post-processed to obtain the final label, which requires some manual operation.

[0083] After the above steps, the target detection label and the semantic segmentation label of all images can be obtained, and these two types of labels indicate the position of the corrosion area to different degrees. As shown in Figure 3 The first image from left is a randomly selected image from the data set, which is the input image, the second image is the heat map corresponding to the input image, and after simple repair, the semantic segmentation label corresponding to the input image is shown in the third image. The target detection label corresponding to the input image is shown in the rectangular frame in the fourth image.

[0084] Third embodiment

[0085] The embodiment provides an electronic device, which comprises a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to realize the method of the first embodiment.

[0086] The electronic device can have relatively large differences due to different configurations or performances, and can comprise one or more processors (central processing units, CPUs) and one or more memories, wherein the memory stores at least one instruction, which is loaded and executed by the processor to realize the above method.

[0087] Fourth embodiment

[0088] The embodiment provides a computer readable storage medium, which stores at least one instruction, which is loaded and executed by the processor to realize the method of the above first embodiment. The computer readable storage medium can be a ROM, a random access memory, a CD-ROM, a magnetic tape, a floppy disk and an optical data storage device, etc. The instructions stored therein can be loaded and executed by the processor in the terminal to realize the above method.

[0089] Furthermore, it is to be understood that the application can be provided as a method, an apparatus, or a computer program product. Accordingly, the application embodiments can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the application embodiments can take the form of a computer program product on one or more computer-usable storage media (including disks, diskettes, tapes, optical, solid storage, RAM, ROM, etc.) embodying computer-readable instructions.

[0090] The application embodiments are described herein with reference to the flowchart and / or block diagram illustrations of the methods, terminal devices (systems), and computer program products according to the application embodiments. It is to be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks.

[0091] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing terminal devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks.

[0092] It should also be understood that, throughout this document, use of terms such as "including", "comprising", or "having" are open-ended, and are intended to permit but not require the inclusion of additional elements, for example, other elements, or the exclusion of additional elements, for example, other elements. Furthermore, the words "a" or "an", as used in the context of this document, are understood to be open-ended, and are intended to mean "one or more". The words "comprise", "comprising", "include", and "including" as well as variations such as "comprises", "comprising", "includes" and "including" will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

[0093] Finally, it should be noted that the above description is of preferred embodiments of the application, and that although preferred embodiments of the application have been described, numerous changes and modifications can be made to the preferred embodiments without departing from the principles of the application, and that these changes and modifications are intended to be included within the scope of the application. Accordingly, the appended claims are intended to cover all such changes and modifications as fall within the scope of the application.

Claims

1. An image annotation method based on neural network interpretability, characterized in that, include: The acquired images are classified to obtain the correct category labels. Train a pre-defined convolutional neural network using images with category labels; The trained convolutional neural network is used as the neural network to be interpreted. The image to be labeled and its corresponding correct category label are used as input. The pre-set neural network interpretability algorithm is used to interpret the trained convolutional neural network. After the interpretation process is completed, a heat map related to the input image and image category is obtained. Based on the heatmap, semantic segmentation labels and object detection labels for the image to be labeled are obtained; The convolutional neural network consists of convolutional layers and classification layers; wherein, the convolutional layer includes at least one neural network layer composed of convolutional kernels, and the classification layer includes at least one fully connected layer; The method of training a predefined convolutional neural network using images with category labels includes: The images and category labels are divided into a training set and a validation set according to a preset ratio, and the training set is augmented using a preset data augmentation method. The predefined convolutional neural network is trained using an enhanced training set. After training, the accuracy of the convolutional neural network is measured on the validation set. When the accuracy is lower than a predefined accuracy threshold, the accuracy of the convolutional neural network is improved by increasing the training set and adjusting the network hyperparameters. Based on the heatmap, semantic segmentation labels and object detection labels for the image to be labeled are obtained, including: The heatmap was converted into semantic segmentation labels using a thresholding method. The minimum rectangle method is used to find the minimum rectangle containing the semantic segmentation label as the target detection label; Finding the smallest rectangle containing the semantic segmentation label using the minimum rectangle method includes: Find the largest independent region in the semantic tags; Find the minimum value of the largest independent region on the x-axis and y-axis, and use this coordinate as the starting point. Then find the maximum value of the largest independent region on the x-axis and y-axis, and use this coordinate as the ending point. Using the starting point and the ending point as the two vertices of a rectangle, a minimum rectangle containing the semantic segmentation label and whose sides are parallel to the input image is obtained.

2. The image annotation method based on neural network interpretability as described in claim 1, characterized in that, The correct category label for an image is a label obtained through manual annotation to correctly classify the image. Each category has a unique label, and the labels are parallel to each other without any hierarchical relationship.

3. The image annotation method based on neural network interpretability as described in claim 1, characterized in that, The data augmentation methods include: image inversion, image rotation, channel equalization, brightness adjustment, sharpening, blurring, and random cropping.

4. The image annotation method based on neural network interpretability as described in claim 3, characterized in that, After training the preset convolutional neural network using the enhanced training set, the step of training the preset convolutional neural network using images with category labels further includes: The validation set is augmented using a preset data augmentation method and then merged into the augmented training set. The convolutional neural network is then retrained to obtain a final trained convolutional neural network.

5. The image annotation method based on neural network interpretability as described in claim 1, characterized in that, The step of converting the heatmap into semantic segmentation labels using a thresholding method includes: Set a threshold to convert the heatmap into a binary image; The binary image can be directly used as the semantic segmentation label of the corresponding input image, or the semantic segmentation label of the corresponding input image can be obtained after further processing of the binary image.

6. The image annotation method based on neural network interpretability as described in claim 5, characterized in that, The threshold is set as the average of the minimum and maximum values ​​of the heatmap.

7. The image annotation method based on neural network interpretability as described in claim 1, characterized in that, After obtaining a minimum rectangle containing the semantic segmentation label and whose edges are parallel to the input image, the step of finding the minimum rectangle containing the semantic segmentation label using the minimum rectangle method further includes: Expand the smallest rectangle outward by 1 pixel, and the coordinates of the expanded rectangle are the target detection label.

Images