Defect labeling method and device suitable for defect image, equipment and storage medium

Abstract

The invention discloses a defect labeling method and device suitable for a defect image, equipment and a storage medium, and relates to the field of image processing. According to the scheme, a defect image set and a defect auxiliary image set corresponding to the defect image set are obtained; identifying defect features contained in each defect auxiliary image in the defect auxiliary image set; and correspondingly marking the defect features contained in the defect auxiliary images in the defect auxiliary image set into each defect image in the defect image set. According to the scheme, firstly, defect auxiliary images which contain massive defect images and are in one-to-one correspondence with the defect images are obtained; the defect auxiliary image does not contain the background image but only contains the defect image, so that contour recognition can be directly performed on the defect auxiliary image to determine the defect content, the defect position in the defect image can be determined by directly matching and mapping during labeling, rapid labeling is realized, and rich labeling image data is provided for AI model training.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of document scanning, and in particular to a defect labeling method, device, equipment and storage medium applicable to defect images. Background Art

[0002] In the field of industrial inspection, data enhancement and defect annotation technology for defect images play a key role in training detection algorithms, enhancing data sets, and simulating various defect situations. At present, the main data enhancement method for defect images is to build a data set training set. Building a data set means generating a large number of defect images. The mainstream methods include methods based on generative adversarial networks (GANs), methods based on diffusion models, and traditional image processing methods.

[0003] Whether it is a generative adversarial network, a Diffusion model, or traditional image processing, the generation of defect images requires a complete defect image library or database. In terms of building training sets and test sets, the traditional collection of defect images is not convenient. Among the limited wafer images collected, the number and type of defect images are single, and it is difficult to screen out a rich variety of defect types. The defect annotation of test set images is almost always done manually to ensure the accuracy of the verification set of the AI ​​model. This annotation method is inefficient and the data set is not rich enough. Therefore, how to perform defect annotation and build rich defect data has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a defect annotation method, device, equipment and storage medium applicable to defect images, so as to solve the problems of single type and quantity of existing defect data sets and low efficiency of defect annotation.

[0005] On the one hand, the present application provides a defect annotation method applicable to defect images, the method comprising:

[0006] Acquire a defect image set and a defect auxiliary image set corresponding to the defect image set;

[0007] Identify defect features contained in each defect auxiliary image in the defect auxiliary image set;

[0008] The defect features contained in the defect auxiliary images in the defect auxiliary image set are correspondingly annotated to each defect image in the defect image set.

[0009] Specifically, the defect features contained in each defect auxiliary image in the defect auxiliary image set include:

[0010] Extracting a defect auxiliary image from the defect auxiliary image set, and identifying a contour area of ​​a defect feature in the defect auxiliary image;

[0011] The serial number of the defect auxiliary image is obtained, and the corresponding target defect image in the defect image set is determined according to the serial number; the target defect image has the same size as the defect auxiliary image, and contains the same defect features at the same position.

[0012] Specifically, the step of correspondingly marking defect features contained in the defect auxiliary image in the defect auxiliary image set to each defect image in the defect image set includes:

[0013] Matching the defect auxiliary image with the corresponding defect image, and mapping the defect contour identified in the defect auxiliary image to the defect image;

[0014] Determine the target position of the defect feature in the defect image according to the mapping result of the defect contour;

[0015] The mapped defect contour is marked in the defect image, and a defect label is generated based on the defect contour and the target position.

[0016] Specifically, mapping the defect contour identified in the defect auxiliary image to the defect image includes:

[0017] Determine the image size of the defect auxiliary image, and establish an image mapping frame according to the image size;

[0018] Overlaying the defect auxiliary image and the corresponding defect image in an image mapping frame;

[0019] Hiding the defect image and extracting the defect contour image from the defect auxiliary image;

[0020] The defect image is displayed, and the extracted defect contour map is fixedly displayed above the defect image, and mapped based on the image overlapping area.

[0021] Specifically, before acquiring the defect image set and the defect auxiliary image set corresponding to the defect image set, the method further includes:

[0022] Acquire a preset defect image;

[0023] Extracting defect features and a background image from the preset defect image;

[0024] A set of auxiliary images is generated based on the set of feature changes of the defect features, and the set of defect images is generated based on the set of changes of the defect features and the background image.

[0025] Specifically, generating an auxiliary image set based on the variation set of defect features includes:

[0026] Determine the target size of the preset defect image, and generate a feature trajectory set according to the target size and the feature change set; the feature change set records the rules for generating the feature trajectory;

[0027] A canvas base map of the target size is constructed, and defect features are fused into the canvas base map according to the feature trajectory set to generate an auxiliary image set.

[0028] Specifically, the feature trajectory set is also used to determine the fusion position of the defect feature and the background image; the step of generating the defect image set based on the defect feature change set and the background image includes:

[0029] Determining feature transformation content of defect features based on the feature change set, and performing feature transformation on the defect features according to the feature transformation content to generate a defect feature set; the feature transformation content includes size transformation and angle rotation transformation;

[0030] The defect features in the defect feature set are serially numbered according to the feature trajectory set, and the background image is fused according to the serial numbers to generate the defect image set.

[0031] On the other hand, the present application provides a defect annotation device applicable to defect images, the device comprising:

[0032] An acquisition module, used for acquiring a defect image set and a defect auxiliary image set corresponding to the defect image set;

[0033] A recognition module, used for recognizing defect features contained in each defect auxiliary image in the defect auxiliary image set;

[0034] A labeling module, used to label the defect features contained in the defect auxiliary image in the defect auxiliary image set to each defect image in the defect image set

[0035] On the other hand, the present application provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the defect annotation method applicable to defect images described in the above aspects.

[0036] On the other hand, the present application provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the defect annotation method applicable to defect images described in the above aspects.

[0037] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least: when the present solution is used for defect annotation, a defect image set and a defect auxiliary image set are first obtained. The two sets adopt a completely matching mode, which includes a large number of defect images and defect auxiliary images corresponding to each defect image. The defect auxiliary image does not contain a background image but only contains the defect image, so that the defect auxiliary image can be directly identified by contour to determine the defect content. When annotating, the defect position in the defect image can be determined by directly matching the mapping, so as to achieve rapid annotation and provide rich annotated image data for AI model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of a defect annotation method applicable to defect images provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram showing a defect image and a corresponding defect auxiliary image is exemplarily listed;

[0040] Figure 3 Shown is a schematic diagram of defect annotation of a defect image based on a defect auxiliary image;

[0041] Figure 4 A schematic diagram showing the characteristics of a fusion defect;

[0042] Figure 5 Schematic diagram showing characteristics of different types of defects;

[0043] Figure 6 is a schematic diagram of extracting defect features and background images provided by an exemplary embodiment;

[0044] Figure 7 A schematic diagram of a defect trajectory curve generated in a solder resist area is shown;

[0045] Figure 8 A schematic diagram showing the generation of a defect image sequence by mapping and fusing a feature trajectory sequence with the same background image is shown;

[0046] Fig. 9 is based on Figure 8 Schematic diagram of defect annotation for defect images in FIG.

[0047] Fig.10 A schematic diagram of collecting a rectangular pixel frame to calculate a target pixel point is shown;

[0048] Fig.11 It is a schematic diagram of restoring target pixels of multiple color blocks;

[0049] Fig.12 A structural block diagram of a defect annotation device applicable to defect images provided in an embodiment of the present application is shown;

[0050] Fig.13 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0052] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0053] Figure 1 : is a flow chart of a defect labeling method applicable to defect images provided in an embodiment of the present application, comprising the following steps:

[0054] S1. Acquire a defect image set and a defect auxiliary image set corresponding to the defect image set;

[0055] The defect image set contains a large number of defect images, and the defect auxiliary image set is an image set used to annotate defect features in defect images, including defect auxiliary images corresponding to the defect images. For the definition of defect auxiliary image, this application refers to an image that is the same size as the defect image but only contains the defect image. Figure 2 The schematic diagram of the defect image and the corresponding defect auxiliary image is listed as an example. The background part of the defect image on the left is a partial image of the wafer image or the material board image, and the background image contains at least one defect feature. The defect auxiliary image on the right does not contain the background image, but only contains the defect feature, and the shape and coordinate position of the defect feature are exactly the same as the defect image in the defect image on the left.

[0056] S2, identifying defect features contained in each defect auxiliary image in the defect auxiliary image set;

[0057] This step first requires reading all defect auxiliary images in the defect auxiliary image set one by one, and using the contour recognition method to identify all defect features contained therein. Optionally, this step can also determine the position coordinates of each defect feature for later defect feature matching and defect labeling.

[0058] S3. Correspondingly marking defect features contained in the defect auxiliary images in the defect auxiliary image set to each defect image in the defect image set.

[0059] This step is to extract each defect image and the unique matching defect auxiliary image in the two sets, and then map it to the defect image according to the defect contour previously identified in the defect auxiliary image. In this way, all defect features and corresponding positions in the defect image can be directly determined, and then the defect features can be marked and displayed according to the edge contour.

[0060] Figure 3 What is shown is a schematic diagram of defect labeling of a defect image based on a defect auxiliary image. The defect features in the defect image can be directly determined by feature matching, and the defect contour can be labeled and displayed, which is convenient for subsequent AI model training and manual verification.

[0061] The execution logic of this solution is: AI model training requires a large number of training images, and training must have a test set and a corresponding training set. The training set is important data for measuring AI training results and detection accuracy. The defect image set has been obtained, but the traditional way of making a test set is to directly use manual annotation to operate directly on the defect image. This means that the sample capacity of the test set or training set cannot achieve massive data, which has a direct impact on the accuracy of the AI ​​model. This solution prepares auxiliary defect images that specifically display defect features in advance when making defect images. Before defect annotation, use other AI detection models to extract the defect contours in the defect auxiliary image. From Figure 3 It can be seen intuitively that the defect auxiliary image does not contain the wafer background or material board background, so the defect contour can be directly and accurately identified and extracted. Furthermore, when marking defects, the defect auxiliary image can be directly mapped to the defect image, so there is no need to identify the defect image or manually mark it. This operation method can generate a large number of richly labeled images, providing a rich verification set for training AI models.

[0062] In summary, this solution first obtains a defect image set and a defect auxiliary image set when performing defect annotation. These two sets adopt a completely matching mode, which contains a large number of defect images and defect auxiliary images corresponding to each defect image. The defect auxiliary image does not contain a background image but only contains the defect image. In this way, the defect auxiliary image can be directly identified by contour to determine the defect content. When annotating, the defect position in the defect image can be determined by directly matching the mapping, realizing fast annotation and providing rich annotated image data for AI model training.

[0063] In some embodiments, when identifying each defect auxiliary image in a defect auxiliary image set, in order to ensure that the unique defect image in the defect image set can be matched, it can be performed in a numbered labeling manner. Specifically, the defect auxiliary image is extracted from the defect auxiliary image set, and the contour area of ​​the defect feature in the defect auxiliary image is identified. Then, the number of the defect auxiliary image is obtained, and the corresponding target defect image in the defect image set is determined according to the number. This process needs to be numbered when the defect image set and the defect auxiliary image set are generated. The target defect image with the same number is consistent in size with the defect auxiliary image, and the position of the defect features contained is the same, so that it can be matched according to the number in the subsequent process.

[0064] The defect image annotation process may further include the following:

[0065] A. Matching the defect auxiliary image with the corresponding defect image, and mapping the defect contour identified in the defect auxiliary image to the defect image;

[0066] There are many forms of defect feature mapping process. This application provides two mapping methods:

[0067] 1. Identify the coordinates of the defect contour in the defect auxiliary image, and map the contour coordinates to the defect image one by one, that is, coordinate mapping determines the defect contour in the defect image.

[0068] 2. Mapping according to the defect feature map, that is, determining the image size of the defect auxiliary image and establishing an image mapping frame according to the image size;

[0069] Overlaying the defect auxiliary image and the corresponding defect image in an image mapping frame;

[0070] The defect image is hidden, and the defect contour image is extracted from the defect auxiliary image;

[0071] The defect image is displayed, and the extracted defect contour map is fixedly displayed above the defect image, and mapped based on the image overlapping area.

[0072] The second method does not need to determine the coordinates of the defect contour, and does not require coordinate mapping. This double-layer image overlay mapping method is similar to setting a mask layer. The defect contour map in the mask layer is retained, and the rest is hidden or eliminated, while the defect image in the lower layer is naturally revealed, which is equivalent to mapping the defect contour map in the mask layer to the defect image.

[0073] B. Determine the target position of the defect feature in the defect image according to the mapping result of the defect contour;

[0074] C. Annotate the mapped defect contour in the defect image and generate a defect label based on the defect contour and target position.

[0075] Defect labels are generated for the purpose of later verification of the AI ​​model and to determine the detection accuracy of the trained AI model.

[0076] Of course, the above steps are to directly obtain the defect auxiliary image set corresponding to the defect image set, that is, the defect image and the defect auxiliary image must be matched one by one, so that the labeled image can be generated for the AI ​​model to train and verify the model. This application also provides the specific content of generating the defect image set and the corresponding defect auxiliary image set, which is summarized as follows:

[0077] S1, obtaining a preset defect image;

[0078] The preset defect image described in this embodiment is an image of a wafer or material board obtained after being photographed and screened by a camera, and the image contains at least one determined defect feature (point). These preset defect images with defect features are aggregated to obtain a preset defect image set. In particular, the preset defect image set of this application can be images of different magnifications or different sizes. In order to facilitate unified operation, this application is illustrated by taking the same image size as an example.

[0079] S2, extracting defect features and background images from a preset defect image;

[0080] Different from the traditional generative adversarial network and Diffusion model modes of generating defect images, the images generated by AI in these modes are poor in authenticity. The authenticity requirements can be guaranteed by extracting and regenerating features from existing preset defect images. The core idea is to extract the defect features in the preset defect images one by one, that is, to extract the defect feature map. The extracted defect feature map forms a defect feature set, and the remaining part of the image after extracting the defect features forms the background image. The background image here is a pure image after completely removing the defect features. All background images are gathered to obtain a background image set, which is used for subsequent data enhancement to enrich the database of defect images.

[0081] S3. Generate an auxiliary image set based on a feature variation set of defect features, and generate a defect image set based on the variation set of defect features and the background image.

[0082] Because the preset defect images are carefully selected and have a small amount of data, simply combining the extracted defect features with the background image cannot meet the data richness. Therefore, this application chooses to perform some feature transformations on all the defect features covered in the defect feature set, and the feature change set contains specific feature transformation rules, such as how to enlarge or reduce defect features, and how to integrate defect features into the background image.

[0083] All defect features in the defect feature set are transformed based on the feature change set, that is, each defect feature can be transformed to expand a feature sequence, and the defect feature maps in the feature sequence are obtained after transformation and have different forms. Then these defect feature maps are used as foreground images and combined with each background image in the background image set to generate a corresponding defect image sequence for each defect feature sequence, and all defect feature sequences can be expanded to generate a defect image set to achieve a richer image data set enhancement effect.

[0084] Figure 4 The schematic diagram of fused defect features is shown, where the defect features circled in the upper part of the left figure are the extracted and cut-out parts, while the defect image on the right is obtained after the feature transformation and re-fusion of the background image (the background image can be the same as or different from the original background). After this transformation, the defect points in the preset defect image can be fully utilized and fully expanded to obtain a rich defect image.

[0085] This solution extracts features from all preset defect images in the preset defect image set and aggregates them into a defect feature set and a background image set. For the defect feature set, a preset feature change set is used to transform the features, and each defect feature is expanded into a defect feature sequence to achieve quantitative fission. Then, each defect feature sequence is fully matched and defect-fused with the background image in the background image set to expand and obtain rich defect images, thereby achieving data enhancement effects.

[0086] In some possible implementations, the process of generating the defect feature set and the background image set may be implemented using the following steps:

[0087] a. extracting a preset defect image from a preset defect image set, and identifying a defect contour of defect features contained therein;

[0088] b. Extracting defect features from a preset defect image according to the defect contour to generate a defect feature map; generating a background image based on the image after the defect features are extracted from the preset defect image, and the background image does not contain the defect features;

[0089] Figure 5 A schematic diagram shows different types of defect features, including board loss, electroplating resist, solder mask peeling, solder mask scratches, post-gold scratches, gold surface agglomeration, nickel leakage on gold surface, copper exposure on gold surface, gold surface depression, discoloration, foreign matter on pad, and ink on pad. The screening of these defects can be done manually or by other AI model detection, and then a defect feature set is generated based on the contour range.

[0090] Assuming that the defect feature set includes defects A, B, and C, the feature change set is transformed to form the A sequence {A1, A2, A3, A4}, the B sequence {B1, B2, B3, B4}, and the C sequence {C1, C2, C3, C4}. Each feature in these sequences can be defect-fused with all background images. The A defect image sequence is generated for the A sequence features, the B defect image sequence is generated for the B sequence features, and the C defect image sequence is generated for the C sequence features. Figure 4 It is a schematic diagram of extracting defect features and background images provided by an exemplary embodiment. The background image generation process needs to repair the remaining part after the defect features are extracted, which is equivalent to a normal image.

[0091] c. Extract the preset defect images in the preset defect image set in sequence, extract and generate defect feature maps and corresponding background images respectively, and summarize them into a defect feature set and a background image set.

[0092] Furthermore, performing feature transformation on the defect feature set based on the feature change set may specifically include the following steps:

[0093] d. Determine the feature transformation content of each defect feature image in the defect feature set based on the feature change set content;

[0094] The feature change set specifies the feature transformation of the defect feature set and the rules for defect fusion with the background image set. Based on the rules, it is clear which feature transformations are performed on the defect feature map. In the embodiment of the present application, the feature transformation content includes size transformation and angle rotation transformation, such as enlarging by 2 times, 4 times, reducing by 1 times, image compression, image rotation by 90 degrees, etc.

[0095] e. Extract defect feature maps in sequence, perform size transformation and / or angle rotation according to feature transformation content, and generate a feature map sequence;

[0096] For a selected defect feature map, the corresponding feature map sequence will be generated after feature transformation using the feature transformation content. These feature map sequences contain defect feature maps of different sizes and / or rotation angles. This step is mainly to expand the defect feature content, so as to ensure the diversity of subsequent defect data.

[0097] In some embodiments, the feature change set also records defect fusion rules, and defect images are generated based on these rules instead of randomly generated to avoid duplication. In particular, the fusion position of the sequence feature map directly determines the defect effect after fusion. For this reason, the present application further continues to perform step f.

[0098] f. Obtain the target size of the defect image in the preset defect image set, and determine the feature trajectory sequence of the feature image sequence according to the target size and the feature change set.

[0099] The image sizes in the preset defect image set can be the same or different. Different feature trajectory sequences can be established for different image sizes. When the defect is fused, the matching feature trajectory sequence is selected according to the size of the background image. This application is described with a unified target size. After obtaining the target size, the feature trajectory is generated according to the coverage range of the target size. When fused, the defect feature sequence will be integrated into the target position of the background image in sequence according to the feature trajectory. The process can be further described as follows:

[0100] 1. Determine the number of defect feature graphs in the defect feature set and determine the trajectory points;

[0101] The number of defect feature maps represents the number of trajectory points contained in the feature trajectory. Considering the maximum number, for a selected background image, an N:1 mapping mode will be used for defect fusion, where N is the number of defect feature maps in the defect feature set.

[0102] 2. Generate a defect trajectory frame based on the target size, and generate a trajectory curve within the defect trajectory frame;

[0103] Figure 7 The schematic diagram of the defect trajectory curve generated by the solder mask is shown. The trajectory curve must be drawn within the size range of the defect trajectory box. In fact, the defect trajectory box when drawing the trajectory curve does not consider the background image content, that is, a blank box with the same size as the background image, and then the trajectory curve is drawn in the blank box. Figure 7 What is shown is the path effect after mapping the real background image.

[0104] 3. Select target trajectory points according to the number of defect feature maps, and generate a feature trajectory sequence of the feature map sequence based on the target trajectory points.

[0105] The target trajectory point is the fusion position of the defect feature map on the trajectory curve. After determining the target trajectory point, the target trajectory point (including the coordinates of the target trajectory point in the defect trajectory box) can be associated and matched with the defect feature map, for example, the trajectory point and the feature sequence image are set in a numbering order, and the feature trajectory sequence is generated according to the number matching.

[0106] In particular, a trajectory curve can be set to correspond to a defect feature sequence, and each target trajectory point corresponds to a defect feature map in the defect feature set. Of course, when the background image has a rich solder mask or gold surface, multiple trajectory curves can be set to include multiple defect feature sequences at the same time, or multiple defect feature sequences can be used to draw a trajectory curve together.

[0107] Based on the above embodiments, the process of generating a defect image set by combining a background image set can be achieved by the following steps:

[0108] 1. Select the target background image from the background image set;

[0109] 2. The defect feature images in the defect feature set are numbered in sequence according to the feature trajectory sequence, and the target background images are fused according to the sequence numbers to generate a defect image sequence;

[0110] Note that this step can randomly select different background images of the same size or use the same background image for fusion, that is, N defect feature images of a feature trajectory sequence correspond to N background images, and these N background images can be copied from one target background image, or from several target background images. Figure 7 What is shown in the figure is that a background image is used to map a feature trajectory sequence for display.

[0111] 3. Select different background images from the background image set in turn, and fuse the defect feature maps according to the feature trajectory sequence to obtain a defect image set.

[0112] Figure 8 The schematic diagram of generating a defect image sequence by mapping the fusion feature trajectory sequence with the same background image is shown. In the above technical solution, by extracting a limited number of preset defect images, expanding them with a small number of defect features to the maximum extent, and using the trajectory sequence method to determine the fusion position, the maximum number of defects can be fused for each background image, thereby improving the data richness. Figure 8 Correspondingly, when Figure 8 After annotating the defect image in Fig. 9 , where all defect contours are marked with yellow frames.

[0113] The above parts are all explained using the repaired background image as an example, because this process does not introduce other irrelevant normal images, but directly uses the image after the defect features are removed. In fact, in order to achieve the data enhancement effect, additional images of normal areas can be introduced or photographed, but this method is not conducive to the subsequent defect image annotation, because the produced defect images are used to train AI models. While providing a test training set for the AI ​​model, it is also necessary to ensure that there is a corresponding verification set for verification, that is, to ensure that there is a completely matching defect annotated image. Therefore, this application does not use direct shooting or introduction of other normal images, but uses the image after the defect is removed for repair, so that the fused defect image can be directly used for subsequent generation of annotated images.

[0114] The process of extracting defective images to generate background images can be implemented through the following steps:

[0115] 1. Determine the image coordinates of the defect contour;

[0116] 2. Select blank pixels from the defect outline inward, and use the blank pixels as the corner points of the rectangle to expand the rectangular pixel frame; the rectangular pixel frame must contain at least 4*4 pixels;

[0117] 3. Take the average of the remaining pixels in the rectangular pixel frame, and determine the pixel average as the pixel value of the blank pixel;

[0118] Fig.10 The schematic diagram of collecting rectangular pixel frames to calculate target pixel points is shown. The scheme uses a successive approximation strategy to calculate pixel values ​​inward in sequence. The internal pixels are completely determined by the size of the pixel values ​​in the surrounding area, and there will be no situation where the color difference with the outer contour is too large.

[0119] Fig.11 This is a schematic diagram of restoring target pixels in multiple color blocks. For the case where defect features span multiple color areas at the same time, the mode of first restoring the background contour edge line and then performing pixel value approximation calculation can be adopted. Fig.11 The dotted line inside the defect contour is a virtual contour restored based on the background contour, and then the contour is defined as a color boundary. The S1 and S2 areas on the left and right of the color boundary correspond to two completely different color blocks. For the S1 and S2 blocks, blank pixels are also selected inward according to the curve of the outermost defect contour, and the blank pixels are used as the rectangular corner points to expand the rectangular pixel frame to calculate the pixel value, but the color boundary line is used as a sign to stop the approximation calculation, that is, the S1 area is approximated and calculated from left to right as a whole, and the S2 area is approximated and calculated from right to left as a whole. In this way, when the two areas approach the color boundary line, they stop, thereby restoring two different color blocks. For other forms, such as the case of dividing the color block up and down, this solution is also applicable, and the details will not be repeated.

[0120] 4. Select blank pixels layer by layer along the defect contour and calculate the pixel values ​​until all defect contours are filled to obtain the repaired background image.

[0121] Fig.12 The structure block diagram of the defect annotation device applicable to defect images provided in an embodiment of the present application is shown, and the device includes:

[0122] An acquisition module 1210 is used to acquire a defect image set and a defect auxiliary image set corresponding to the defect image set;

[0123] The identification module 1220 is used to identify defect features contained in each defect auxiliary image in the defect auxiliary image set;

[0124] The labeling module 1230 is used to label the defect features contained in the defect auxiliary images in the defect auxiliary image set to the respective defect images in the defect image set.

[0125] The defect labeling device applicable to defect images provided in the embodiment of the present application can be applied to the defect labeling method applicable to defect images provided in the above embodiment. For relevant details, refer to the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0126] It should be noted that the defect annotation device for defective images provided in the embodiments of the present application is only illustrated by the division of the above-mentioned functional modules / functional units. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the defect annotation device for defective images is divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the defect annotation method for defective images provided in the above-mentioned method embodiment and the implementation method of the defect annotation device for defective images provided in this embodiment belong to the same concept. The specific implementation process of the defect annotation device for defective images provided in this embodiment is detailed in the above-mentioned method embodiment and will not be repeated here.

[0127] Fig.13 The block diagram of a computer device provided by an exemplary embodiment of the present application is shown. It is a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. Among them, the processor and the memory may be connected by a bus or in other ways. Among them, the processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processors (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips.

[0128] The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content that needs to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0129] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method implementation is realized. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0130] In some embodiments, the computer device may further optionally include: a peripheral device interface and at least one peripheral device. The processor, the memory and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit, a display screen and a keyboard.

[0131] The peripheral device interface can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor and the memory. In some embodiments, the processor, the memory, and the peripheral device interface are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor, the memory, and the peripheral device interface can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0132] The display screen is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen is a touch screen, the display screen also has the ability to collect touch signals on the surface or above the surface of the display screen. The touch signal can be input to the processor as a control signal for processing. At this time, the display screen can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen can be one, arranged on the front panel of the computer device; in other embodiments, the display screen can be at least two, respectively arranged on different surfaces of the computer device or in a folding design; in other embodiments, the display screen can be a flexible display screen, arranged on a curved surface or a folding surface of the computer device. Even, the display screen can be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode) and the like.

[0133] The power supply is used to power various components in the computer device. The power supply can be AC, DC, a disposable battery, or a rechargeable battery. When the power supply includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0134] Those skilled in the art will appreciate that the structure shown in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0135] The embodiment of the present application also discloses a computer-readable storage medium. Specifically, the computer-readable storage medium is used to store a computer program, and when the computer program is executed by the processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art can understand that the implementation of all or part of the process in the above-mentioned implementation method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the process of the implementation of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0136] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A defect labeling method applicable to defect images, characterized in that: The method comprises: Acquire a defect image set and a defect auxiliary image set corresponding to the defect image set; Identify defect features contained in each defect auxiliary image in the defect auxiliary image set; The defect features contained in the defect auxiliary images in the defect auxiliary image set are correspondingly annotated to each defect image in the defect image set.

2. The method according to claim 1, characterized in that The defect features contained in each defect auxiliary image in the defect identification auxiliary image set include: Extracting a defect auxiliary image from the defect auxiliary image set, and identifying a contour area of ​​a defect feature in the defect auxiliary image; The serial number of the defect auxiliary image is obtained, and the corresponding target defect image in the defect image set is determined according to the serial number; the target defect image has the same size as the defect auxiliary image, and contains the same defect features at the same position.

3. The method according to claim 2, characterized in that The step of correspondingly marking the defect features contained in the defect auxiliary images in the defect auxiliary image set to each defect image in the defect image set includes: Matching the defect auxiliary image with the corresponding defect image, and mapping the defect contour identified in the defect auxiliary image to the defect image; Determine the target position of the defect feature in the defect image according to the mapping result of the defect contour; The mapped defect contour is marked in the defect image, and a defect label is generated based on the defect contour and the target position.

4. The method according to claim 3, characterized in that The step of mapping the defect contour identified in the defect auxiliary image to the defect image comprises: Determine the image size of the defect auxiliary image, and establish an image mapping frame according to the image size; Overlaying the defect auxiliary image and the corresponding defect image in an image mapping frame; Hiding the defect image and extracting the defect contour image from the defect auxiliary image; The defect image is displayed, and the extracted defect contour map is fixedly displayed above the defect image, and mapped based on the image overlapping area.

5. The method according to claim 1, characterized in that Before acquiring the defect image set and the defect auxiliary image set corresponding to the defect image set, the method further includes: Acquire a preset defect image; Extracting defect features and a background image from the preset defect image; A set of auxiliary images is generated based on the set of feature changes of the defect features, and the set of defect images is generated based on the set of changes of the defect features and the background image.

6. The method according to claim 5, characterized in that Generating an auxiliary image set based on the variation set of defect features includes: Determine the target size of the preset defect image, and generate a feature trajectory set according to the target size and the feature change set; the feature change set records the rules for generating the feature trajectory; A canvas base map of the target size is constructed, and defect features are fused into the canvas base map according to the feature trajectory set to generate an auxiliary image set.

7. The method according to claim 6, characterized in that The feature trajectory set is also used to determine the fusion position of the defect feature and the background image; the generating the defect image set based on the change set of the defect feature and the background image includes: Determining feature transformation content of defect features based on the feature change set, and performing feature transformation on the defect features according to the feature transformation content to generate a defect feature set; the feature transformation content includes size transformation and angle rotation transformation; The defect features in the defect feature set are serially numbered according to the feature trajectory set, and the background image is fused according to the serial numbers to generate the defect image set.

8. A defect labeling device suitable for defect images, characterized in that: The device comprises: An acquisition module, used for acquiring a defect image set and a defect auxiliary image set corresponding to the defect image set; A recognition module, used for recognizing defect features contained in each defect auxiliary image in the defect auxiliary image set; The marking module is used to mark the defect features contained in the defect auxiliary images in the defect auxiliary image set to the respective defect images in the defect image set.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the defect annotation method applicable to defect images as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the defect annotation method applicable to defect images as described in any one of claims 1 to 7.

Images