An interactive image annotation method based on deep learning

Through the interactive image labeling method based on deep learning, the thermal distribution map assists the labeler to identify surface defects of industrial products, solving the problems of defect location randomness and category openness in the existing technology, and achieving efficient and accurate labeling and detection , reducing data dependency and cost.

CN114202518BActive Publication Date: 2024-11-19TSINGHUA SHENZHEN INTERNATIONAL GRADUATE SCHOOL
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202111449025.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-01
Publication Date
2024-11-19
Estimated Expiration
2041-12-01

AI Technical Summary

Technical Problem

Existing deep learning-based 2D image-assisted labeling tools cannot effectively deal with defect location randomness and defect category openness in surface defect detection of industrial products, resulting in inaccuracy and inefficiency of detection, and require a large amount of high-quality labeled data for training. .

Method used

Using an interactive image annotation method based on deep learning, a thermal distribution map is generated by training model comparison and standard image, and auxiliary labelers quickly and accurately label defect areas, and use abnormal pictures for pre-pre-order using abnormal pictures during the training stage. Training to reduce dependence on labeled data.

Benefits of technology

Improve the speed and accuracy of image annotation, can identify image anomalies in zero annotation, reduce system development costs and technical thresholds, enhance adaptability to new anomalies categories and shapes, and provide high-quality data support for industrial Quality control and algorithm optimization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114202518B_ABST
    Figure CN114202518B_ABST
Patent Text Reader

Abstract

An interactive image annotation method based on deep learning includes the following steps: S1, using a model trained by a deep learning algorithm to compare the difference between a to-be-tagged image and a standard part image, and determining the difference value of each region between the to-be-tagged image and the standard part image; S2, the model generates a thermal distribution map with color and depth changes of each region according to the difference value of each region between the to-be-tagged image and the standard part image; S3, presenting the to-be-tagged image and its corresponding thermal distribution map on a terminal interface, so as to mark defects in the to-be-tagged image with reference to the thermal distribution map. The present invention realizes automatic detection of abnormal areas in images and presents them through thermal distribution maps, which serve as a reference for labelers to mark defects, thereby improving the speed and accuracy of image annotation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to digital image processing, and in particular to an interactive image annotation method based on deep learning. Background Art

[0002] Surface defect detection of industrial products is an indispensable part of quality control before products leave the factory. For a long time, enterprises have relied on workers to find abnormal areas with their naked eyes. This method is difficult to maintain stability in detection accuracy and speed, which restricts the level of product quality control of enterprises. Limited by the quality of photos, naked eye vision and concentration, labelers often miss or mislabel areas when repeating work. For photos with small defect areas and inconspicuous defects, labelers need to stare at the entire picture many times before making a decision to label or let go, which is very time-consuming. Labelers lack professional knowledge and background of specific products in the industrial field. When faced with seemingly correct areas in the picture, they do not know how to deal with them and usually regard them as non-defective, causing factories and algorithm providers to lose valuable first-hand information. In addition, the economic cost of manual inspection and the health threats to workers' eyes and respiratory tract have also prompted the industry to look for more advanced detection methods.

[0003] The most commonly used 2D image annotation tools are labelMe and labelImg, both of which are open source image polygon annotation tools based on Python. They can be used to manually annotate images for model training of computer vision tasks such as object detection and image classification. After the image is annotated, they can export annotations in the format required for PASCAL VOC, YOLO and COCO training. LabelMe supports annotating objects with multiple rectangles, circles, points or lines, while LabelImg only supports creating rectangular boxes around objects. The common advantages of the two are: lightweight and cross-platform. The common disadvantages are: the image cannot be pre-processed, and the annotator needs to look for abnormal areas in the original image.

[0004] In recent years, driven by the development of deep learning and the digital transformation of enterprises, some people have used deep learning-based algorithms to automatically detect surface defects of products. In order to achieve algorithms with higher detection accuracy, a large amount of high-quality labeled data is required during training.

[0005] A 2D image assisted annotation tool based on deep learning adopts the method of using a part of purely manually annotated data as training samples to train a model that can recognize objects or text in the picture, and then use the trained model to make inferences on unannotated pictures and mark the preliminary results. Tools using this method are mainly divided into two categories: one is to extract text, key fields, etc. at fixed positions in the picture. This type of invention cannot extract abnormal areas that appear randomly in the picture. The other type uses a rectangular box to mark the target in the image, but the object categories that can be recognized are limited, and it can only cope with closed category scenes. In addition, when training the model, it is necessary to manually check with the tool provider multiple times to check the object categories and difficult samples that may appear in the picture.

[0006] There are two difficult problems to be overcome in the detection of surface defects of industrial products on real production lines: first, the position of defects in photos is random, and the purpose of defect detection cannot be achieved by scanning pictures to fix the position and extract fixed features; second, the defect categories, shapes, etc. are open, and it is impossible to list all possible shapes and categories in advance through exhaustive methods.

[0007] Existing auxiliary annotation tools cannot deal with the above two problems well. The method of scanning images and extracting fixed position image information cannot deal with the randomness of defect locations. Algorithms that pre-set defect categories and shapes are not robust and cannot deal with new abnormal categories and shapes. In addition, such methods have high requirements for the accuracy of pre-trained models, because if the pre-trained model has false positives, it is more time-consuming to manually modify the wrong labels.

[0008] In summary, existing deep learning-based 2D image-assisted annotation tools are designed according to the logic of closed scenes and lack robustness for complex and changeable industrial product defects. In addition, the algorithms used in these methods require a certain number of high-quality annotated images as training samples.

[0009] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0010] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide an interactive image annotation method based on deep learning to improve the speed and accuracy of image annotation.

[0011] To achieve the above object, the present invention adopts the following technical solutions:

[0012] An interactive image annotation method based on deep learning includes the following steps:

[0013] S1. Use a model trained with a deep learning algorithm to compare the difference between the image to be marked and the image of the standard part, and determine the difference value of each area between the image to be marked and the image of the standard part;

[0014] S2, the model generates a thermal distribution map with color and depth changes of each area according to the difference value of each area of ​​the image to be marked and the image of the standard part;

[0015] S3. Presenting the image to be marked and its corresponding thermal distribution map on the terminal interface so as to mark defects in the image to be marked with reference to the thermal distribution map.

[0016] Further:

[0017] Step S1 also includes determining whether the maximum difference value of each area between the image to be marked and the standard part image exceeds a set abnormality threshold, if so, proceeding to the next step, otherwise determining that the image to be marked has no abnormality.

[0018] In step S3, the image to be marked and its corresponding thermal distribution map are presented horizontally in one window at the same time.

[0019] After step S3, the following steps are also included:

[0020] S4. Classify, count and analyze the marked defects.

[0021] The classification statistics and analysis include statistics on the number of defects of each category, the proportion of false alarms in pre-inspection, and the proportion of difficult-to-classify images to qualified products.

[0022] In the training phase of the model, the training samples are all pictures without abnormalities, and the features extracted by the pre-trained neural network are stored; in the testing phase, the nearest neighbors are found for the patch features of the test pictures, and their Euclidean distance is the condition for judging abnormalities.

[0023] In the algorithm, the relationship between block features and anomaly metrics is as follows:

[0024] M=<f1,f2,f3,…f N >

[0025] f i is the regional feature of the normal sample block during training, and M is the set of regional features of all normal sample blocks;

[0026]

[0027] F is the feature of the test image extracted by the neural network during testing, with a total of n×n block features, Φ(X test ) represents the space composed of all regional features of the test image;

[0028]

[0029] d i is the average Euclidean distance between the block feature and the K nearest neighbor block features in M;

[0030]

[0031] D img Anomaly score for test image level

[0032]

[0033] HM means drawing a heat map, and all block features constitute an abnormal heat distribution map.

[0034] In step S4, the pictures of the new defect category are also stored in the standard defect database.

[0035] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the interactive image annotation method based on deep learning is implemented.

[0036] The present invention has the following beneficial effects:

[0037] The present invention proposes an interactive image annotation method based on deep learning, which can use a model trained under zero annotation to identify abnormal areas of the image and use a heat map to intuitively present the abnormal areas, thereby assisting annotators to annotate pictures accurately and quickly.

[0038] The present invention can provide a semi-automatic interactive auxiliary image annotation system, which can realize automatic detection of abnormal areas in images and present them through heat maps, which can serve as a reference for annotators to annotate defects, improve the speed and accuracy of image annotation, and further provide high-quality data for deep learning-based defect detection algorithms, and also provide data support for the formulation of quality control standards and process improvements.

[0039] The present invention can overcome the problems of low speed and low accuracy when simply using open source tools such as LabelMe and LabelImg for labeling. When faced with new abnormal categories and new shapes, the present invention is significantly better than systems designed based on closed category logic and has stronger adaptability. The preferred left-right distributed heat map matches the original image, and the design of automatically recording the labeling information and automatically popping up the next photo after labeling greatly improves the labeling speed and accuracy of the labeler. In addition, the present invention has a very distinct advantage: no labeling data is required when training the pre-inspection model required by the system, and only a small number of defect-free standard product photos are needed to complete the training. This greatly reduces the cost of system development, greatly enhances the flexibility of the system to migrate to new business scenarios, and reduces the financial and technical barriers for small and medium-sized enterprises to use computer vision technology for surface defect detection of industrial products.

[0040] The advantages of the embodiments of the present invention are specifically reflected in the following aspects:

[0041] First, without any pre-labeled images, the algorithm can be used to perform preliminary anomaly detection on product photos, so that areas that may contain defects can be quickly, accurately, and intuitively presented using heat maps to assist labelers in labeling images. Second, photos containing abnormal areas are recorded and stored in a database, which can be analyzed by quality control experts with industrial knowledge to find process improvements and update quality control standards. Third, by mining the abnormal areas marked by the algorithm and the manually labeled data, the samples and corresponding parameters during model training can be updated, allowing the model to be continuously optimized during iterations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a module composition diagram of an interactive auxiliary image annotation system according to an embodiment of the present invention.

[0043] Figure 2 A diagram of a model structure for implementing abnormal area pre-detection according to an embodiment of the present invention.

[0044] Figure 3 The figure is a flow chart of an interactive image annotation method according to an embodiment of the present invention.

[0045] Figure 4 This is a thermal distribution diagram of an example of the present invention.

[0046] Figure 5 This is a thermal distribution diagram of another example of the present invention.

[0047] Figure 6 This is a schematic diagram of a thermal distribution diagram of another example of the present invention and real-time statistical annotation results stored in a database by category. DETAILED DESCRIPTION

[0048] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0049] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for fixing as well as for coupling or communication.

[0050] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0051] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0052] See also Figures 1 to 6 The embodiment of the present invention provides an interactive image annotation method based on deep learning, comprising the following steps:

[0053] S1. Use a model trained with a deep learning algorithm to compare the difference between the image to be marked and the image of the standard part, and determine the difference value of each area between the image to be marked and the image of the standard part;

[0054] S2. The model generates a thermal distribution map (hereinafter referred to as a thermal map) with color and depth changes of each region according to the difference value of each region of the image to be marked and the image of the standard part;

[0055] S3. Presenting the image to be marked and its corresponding thermal distribution map on the terminal interface so as to mark defects in the image to be marked with reference to the thermal distribution map.

[0056] In a preferred embodiment, step S1 further includes determining whether the maximum difference value of each area between the image to be marked and the image of the standard part exceeds a set abnormality threshold, if so, proceeding to the next step, otherwise determining that the image to be marked has no abnormality.

[0057] In a preferred embodiment, in step S3, the image to be marked and its corresponding thermal distribution map are presented horizontally in one window at the same time.

[0058] In a preferred embodiment, step S3 further includes the following steps:

[0059] S4. Classify, count and analyze the marked defects.

[0060] In a preferred embodiment, the classification statistics and analysis include statistics on the number of defects of each category, the proportion of false positives in pre-inspection, and the proportion of difficult-to-classify images to qualified products.

[0061] In a preferred embodiment, in step S4, the pictures of the new defect category are also stored in the standard defect database.

[0062] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the interactive image annotation method based on deep learning.

[0063] The present invention proposes an interactive image annotation method based on deep learning, which can use a model trained under zero annotation to identify abnormal areas of the image and use a heat map to intuitively present the abnormal areas, thereby assisting annotators to annotate pictures accurately and quickly.

[0064] The present invention can overcome the problems of low speed and low accuracy when simply using open source tools such as LabelMe and LabelImg for labeling. When faced with new abnormal categories and new shapes, the present invention is significantly better than systems designed based on closed category logic and has stronger adaptability. The preferred left-right distributed heat map matches the original image, and the design of automatically recording the labeling information and automatically popping up the next photo after labeling greatly improves the labeling speed and accuracy of the labeler. In addition, the present invention has a very distinct advantage: no labeling data is required when training the pre-inspection model required by the system, and only a small number of defect-free standard product photos are needed to complete the training. This greatly reduces the cost of system development, greatly enhances the flexibility of the system to migrate to new business scenarios, and reduces the financial and technical barriers for small and medium-sized enterprises to use computer vision technology for surface defect detection of industrial products.

[0065] The specific embodiments of the present invention are further described below.

[0066] According to the method of the present invention, a semi-automatic interactive auxiliary image annotation system can be provided to realize automatic detection of abnormal areas in images and present them through heat maps, which can serve as a reference for annotators to annotate defects, thereby improving the speed and accuracy of image annotation, and further providing high-quality data for defect detection algorithms based on deep learning, and also providing data support for the formulation of quality control standards and process improvements.

[0067] The system can include three modules, namely: abnormal area detection module, interactive image annotation module, and statistical analysis module. The abnormal area detection module can quickly identify the parts of the image that may contain defects and present them with a heat map according to the degree of abnormality. The module supports setting abnormal thresholds on demand and only transmits images with abnormal values ​​exceeding the threshold to the image annotation module; the interactive image annotation module can automatically match the image to be marked and its corresponding heat map, and present both horizontally in one window at the same time, so that the annotator can focus on the abnormal area according to the prompts of the heat map and then make marks on the original image; the statistical analysis module can classify and record the number of each defect and the original image, and can cross-analyze the annotations made by different markers.

[0068] Figure 1 The module composition of the interactive auxiliary image annotation system is shown. The core modules of the system are the abnormal area pre-inspection module and the interactive picture annotation module. The abnormal area pre-inspection module is a model trained based on deep learning algorithms. Only standard parts without defects are used as samples during training. The module can compare the difference between the input image and the standard part, and output a heat map with different color depths according to the size of the difference value. The interactive picture annotation module can automatically match the image to be annotated with its corresponding heat map, and use bright colors to prompt the annotator to focus on the area, thereby helping the annotator to annotate the image accurately and quickly. The statistical analysis module can count the number of various defects in real time, and record the annotation speed, annotation results, etc. of the annotator.

[0069] The detailed introduction of each module of the system is as follows:

[0070] Figure 2 The structure of the model for pre-checking abnormal areas is shown. In the training phase, the training samples are all pictures without abnormalities, and the features extracted by the pre-trained neural network are stored in the memory. In the testing phase, the nearest neighbor of the patch feature of the test picture is found in the memory, and its Euclidean distance is the condition for judging abnormalities. The relationship between the block feature and the abnormality metric is as follows:

[0071] M=<f1,f2,f3,…f N >

[0072] f i is the regional feature of the normal sample block during training, and M is the set of regional features of all normal sample blocks;

[0073]

[0074] F is the feature of the test image extracted by the neural network during testing, with a total of n×n block features, Φ(X test ) represents the space composed of all regional features of the test image;

[0075]

[0076] d i is the average Euclidean distance between the block feature and the K nearest neighbor block features in M;

[0077]

[0078] D img Anomaly score for test image level

[0079]

[0080] HM draws a thermal map, and all block features constitute an abnormal thermal distribution map.

[0081] After the pre-check module, the original image and the heat map will be transmitted to the interactive annotation module. The structure of the interactive annotation module and its integration with the other two modules are shown in Figure 3 Flowchart shown.

[0082] When labelers label images with the help of heat maps, they will encounter two situations: one is defects with known clear category attributes, which will be directly stored in the standard defect database for training of algorithms such as image classification and target detection. For those that do not belong to known clear categories, they are further divided into three situations: the first is that the labeler discovers a new defect category. Such pictures will be stored in another database, and after the experts in the quality control department determine their categories, they will be stored in the standard defect database. The second is that the labeler cannot determine whether there are defects in the picture. This type will be handed over to a senior expert group for review. The third is that the labeler finds that the pre-inspection module misjudged and judged OK products as abnormal. Such pictures will be fed back to the algorithm module personnel for algorithm improvement. Finally, all the labeling results and pictures will be transmitted to the analysis module. The analysis module can be flexibly designed. Commonly used ones include statistics on the number of various defects, the proportion of false alarms in pre-inspection, and the proportion of difficult-to-classify photos to OK products.

[0083] The present invention tests the pre-inspection and marking effects of the system on multiple photos of magnetic tiles containing defects. The results are described as follows. Figure 4 As shown in the figure, the left side is the abnormal area heat map given by the pre-inspection system, and the right side is the original image. With the help of the heat map, the labeler can quickly mark the defects. Figure 5As shown in the figure, when there is more than one abnormality in the image, the heat map can also give obvious hints. The defect in the lower right corner of the right image is very small and often ignored. With the hint in the lower right corner of the left image, the annotator will pay special attention to this area, thereby discovering and annotating the defect. Figure 6 As shown in the box part, the statistical analysis module can count the annotation results in real time and store them in the database according to categories.

[0084] In addition to the above, the advantages of the embodiments of the present invention are specifically reflected in the following aspects:

[0085] First, without any pre-labeled images, the algorithm can be used to perform preliminary anomaly detection on product photos, so that areas that may contain defects can be quickly, accurately, and intuitively presented using heat maps to assist labelers in labeling images. Second, photos containing abnormal areas are recorded and stored in a database, which can be analyzed by quality control experts with industrial knowledge to find process improvements and update quality control standards. Third, by mining the abnormal areas marked by the algorithm and the manually labeled data, the samples and corresponding parameters during model training can be updated, allowing the model to be continuously optimized during iterations.

[0086] The background section of the present invention may contain background information about the problem or environment of the present invention, but does not necessarily describe the prior art. Therefore, the content included in the background section is not an admission by the applicant that the prior art is available.

[0087] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, it can also make several substitutions or modifications to these described embodiments, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention. In the description of this specification, the description of the reference terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In the absence of mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. An interactive image annotation method based on deep learning, It is characterized in that The steps include: S1. Use a model trained with a deep learning algorithm to compare the difference between the image to be marked and the image of the standard part, and determine the difference value of each area between the image to be marked and the image of the standard part; S2, the model generates a thermal distribution map with color and depth changes of each area according to the difference value of each area of ​​the image to be marked and the image of the standard part; S3, presenting the image to be marked and its corresponding thermal distribution map on the terminal interface, so as to mark defects in the image to be marked with reference to the thermal distribution map; In the training phase of the model, the training samples are all pictures without abnormalities, and the features extracted by the pre-trained neural network are stored; In the testing phase, the nearest neighbor is found for the patch features of the test image, and its Euclidean distance is used as the condition for judging abnormality; In the algorithm, the relationship between block features and anomaly metrics is as follows: f i is the regional feature of the normal sample block during training, and M is the set of regional features of all normal sample blocks; F is the feature of the test image extracted by the neural network during testing, with a total of n×n block features, Φ(X test ) represents the space composed of all regional features of the test image; d i is the average Euclidean distance between the block feature and the K nearest neighbor block features in M; D img Anomaly score for test image level HM means drawing a heat map, and all block features constitute an abnormal heat distribution map.

2. The interactive image annotation method based on deep learning as claimed in claim 1, It is characterized in that Step S1 also includes determining whether the maximum difference value of each area between the image to be marked and the standard part image exceeds a set abnormality threshold, if so, proceeding to the next step, otherwise determining that the image to be marked has no abnormality.

3. The interactive image annotation method based on deep learning according to claim 1 or 2, It is characterized in that In step S3, the image to be marked and its corresponding thermal distribution map are presented horizontally in one window at the same time.

4. The interactive image annotation method based on deep learning according to any one of claims 1 to 2, It is characterized in that After step S3, the following steps are also included: S4. Classify, count and analyze the marked defects.

5. The interactive image annotation method based on deep learning as claimed in claim 4, It is characterized in that The classification statistics and analysis include statistics on the number of defects of each category, the proportion of false alarms in pre-inspection, and the proportion of difficult-to-classify images to qualified products.

6. The interactive image annotation method based on deep learning as claimed in claim 4, It is characterized in that In step S4, the pictures of the new defect category are also stored in the standard defect database.

7. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the deep learning-based interactive image annotation method according to any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Interactive industrial cold start defect detection system and method

    CN115953385A

  • Interactive labeling method and system based on large model

    CN116912827A

  • Large model-based interactive labeling method and system

    CN116912827B