Virtual defect image generation method and device, equipment and storage medium

Abstract

The invention discloses a virtual defect image generation method and device, equipment and a storage medium, and relates to the field of image processing. Obtaining a contour image and a semantic image of the material plate, marking a target contour object in the contour image, and obtaining use case data for the target contour object; updating a target contour object in the semantic image and / or the contour image based on the use case data to generate a virtual material number graph; setting a defect contour and generating a material label graph in the virtual material number graph, and obtaining virtual defect data used for synthesizing a virtual defect; and synthesizing a virtual defect graph based on the contour labels in the material label graph and the virtual defect data. According to the scheme, the contour map and the semantic map can be made based on the preferential material plate, the virtual material number graph is expanded and generated through object contour mapping, targeted training of all defect types is considered, even a defect graph of a specific neighborhood is generated in a targeted mode for high-incidence defects of a production line, the detection capability of the model in characteristic type deficiency is emphatically optimized, and the detection efficiency is improved. The production and detection efficiency of a production line is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, apparatus, device and storage medium for generating a virtual defect image. Background Art

[0002] In the integrated circuit industry, defect detection for PCBs is a crucial inspection task, focusing on measuring the yield of produced materials and screening, removing, or repairing defective materials. Most defect detection methods rely on CAM image matching and recognition. Some advanced technologies employ AI models trained to identify defects in PCB images. However, AI models rely heavily on the recognition capabilities of the model structure and the training dataset. While augmenting the training dataset can mitigate this issue, the dataset is simply based on CAM images filtered for defective images. This trained model is insensitive to subtle variations within the same PCB type, resulting in poor recognition accuracy. For example, slight deviations, rotations, and dimensional variations in the metal surface, metal wires, or the PCB as a whole cannot be identified, ultimately resulting in substandard PCB production accuracy. Furthermore, when the training set contains insufficient defect image data, the trained AI model's ability to detect PCB images is weak. This is particularly true for defects that occur less frequently during PCB production. Without training on these defects, the AI ​​model often struggles to accurately identify them, impacting yield. Summary of the Invention

[0003] The present application provides a virtual defect image generation method, device, equipment and storage medium, which can simulate material board images under various models and scenarios, and generate corresponding virtual defect images, enrich the defect image training set, and help improve the defect detection rate of the model.

[0004] In one aspect, the present application provides a method for generating a virtual defect image, the method comprising: Acquire a contour image and a semantic image of a material board, annotate a target contour object in the contour image, and acquire use case data for the target contour object; the contour image and the semantic image correspond one-to-one, and the use case data includes processing information for updating the target contour object; updating the target contour object in the semantic image and / or contour image based on the use case data to generate a virtual material number map; Setting a defect outline in the virtual material number map, generating a material label map containing the defect outline, and obtaining virtual defect data for synthesizing a virtual defect; the outline label in the material label map is used to indicate the defect outline type; A virtual defect map is synthesized based on the contour labels in the material label map and the virtual defect data.

[0005] On the other hand, the present application provides a virtual defect image generating device, the device comprising: A contour annotation module is configured to obtain a contour image and a semantic image of a material board, annotate a target contour object in the contour image, and obtain use case data for the target contour object; the contour image and the semantic image correspond one-to-one, and the use case data includes processing information for updating the target contour object; A first image generation module is configured to update the target contour object in the semantic image and / or contour image based on the use case data to generate a virtual material number map; a second image generation module, configured to set a defect outline in the virtual material number image, generate a material label image containing the defect outline, and obtain virtual defect data for synthesizing a virtual defect; the outline label in the material label image is used to indicate the defect outline type; A defect synthesis module is used to synthesize a virtual defect map based on the contour label in the material label map and the virtual defect data.

[0006] On the other hand, the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the virtual defect image generation method described in the above aspect.

[0007] On the other hand, the present application provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the virtual defect image generation method described in the above aspect.

[0008] The technical solutions provided by the embodiments of the present application provide at least the following beneficial effects: by establishing a design database of material contour images and semantic images, identifying targets requiring design modification by annotating target contour objects in the contour images, and designing corresponding use case data. The contour position features in the contour images and the semantic features in the semantic images are fully identified and extracted, and the semantic and contour images are processed in combination with processing information in the use case data to generate a color virtual material number map that meets design requirements, thereby fully expanding the material palette image.

[0009] Compared to traditional defect image training sets collected through photo-based defect recognition on production lines, this solution leverages diffusion model-like image generation techniques to generate more realistic defects, significantly reducing the collection cycle for small-sample defect data and rapidly improving the model's detection rate. Furthermore, this solution allows for targeted training for all defect types, even generating defect images for specific neighborhoods of high-incidence defects on production lines. This prioritizes optimizing the model's detection capabilities for specific defect types, improving production and inspection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flowchart of a method for generating a virtual defect image provided by an embodiment of the present application; Figure 2 A schematic diagram showing a possible form of a selected contour image and a corresponding semantic image is shown; Figure 3 A possible output of a color virtual material number diagram based on the design requirement of replicating a circular gold surface is shown; Figure 4 A schematic diagram showing a possible form of generating a material label diagram; Figure 5 A schematic diagram showing a possible form of generating a virtual defect map based on a label contour map; Figure 6 A flow chart of a method for training a material number synthesis model is shown; Figure 7 A schematic diagram showing the offset processing, angle rotation processing, and size enlargement or reduction processing of the target metal surface in the actual material image; Figure 8 Schematic diagram for generating adjustment contour map and semantic map for enlarging the circular gold surface; Figure 9 This example shows how to perform compound modification on a material diagram to generate a virtual material number diagram. Figure 10 A flow chart showing a method for generating label images of different types of materials is shown; Figure 11 The diagram shows the operation of setting the trajectory defect outline and point defect outline on the material label map; Figure 12 A schematic diagram showing a possible method of generating a material label image by filling the interior of a point defect outline with different colors; Figure 13 Therefore Figure 11 Schematic diagram of a virtual defect map synthesized from a midpoint defect profile or a trajectory defect profile; Figure 14 It shows the virtual defect maps with different defect intensities synthesized and outputted by different generation models; Figure 15The following is a structural block diagram of a virtual defect image generating device provided by an embodiment of the present application; Figure 16 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0011] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0012] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0013] Figure 1 : is a flowchart of a method for generating a virtual defect image provided by an embodiment of the present application, comprising the following steps: S1. Obtain a contour image and a semantic image of a material board, annotate a target contour object in the contour image, and obtain use case data for the target contour object; The material board here refers to the actual produced material board, and the contour image and semantic image of the actual material board are obtained by photographing and processing. The design database is a database specifically used to store geber files and semantic images of the global or local design of material boards in the field of integrated circuits. The geber files are specifically for the circuit board configuration parameters, drilling data, physical properties and layer layout information of printed circuit boards / material boards. In actual production, Gerber files play a vital role, including but not limited to providing accurate manufacturing information, guiding production and manufacturing, ensuring design confidentiality, and promoting supply chain collaboration. A large number of geber file materials constitute the design database. Because it is for defect detection of material boards, this embodiment mainly uses the various components, text, chips and wiring contour information therein as the utilization objects, which are expressed as line contours in the CAD template drawing, so it is called a contour image.

[0014] Semantic images mainly serve AI models. The (semantic) pixel values ​​in the semantic images are obtained based on the panoramic image or local high-definition image of the material board taken by a color camera, and are displayed as grayscale images after grayscale processing. The semantic pixels in them can be used for AI model analysis, such as pixel-level image recognition technology, which is particularly used to identify components, chips and wiring conditions in this embodiment.

[0015] In this embodiment, the design database should contain contour images and semantic images of a large number of material palettes. To facilitate network model processing and virtual image synthesis, the database can be designed as two image sets, and a matching relationship should be established between the contour images and the semantic images. When a contour image is selected, the matching semantic image is directly selected, and vice versa.

[0016] The target contour object is the outline of the real target object in the material plate. For example, if the target object is a gold-plated surface, the target contour object is the outline of the gold-plated surface. Figure 2 A schematic diagram of a selected contour image and a corresponding semantic image in one possible form is shown, where their sizes match each other, and the line contour objects in the contour image correspond one-to-one to the pixel areas in the semantic image. Annotating the target contour objects in the contour image and designing use case data are essentially for inputting design requirements. The contour image is used as the annotation object here, mainly to clearly distinguish between components and contours. In the case of a large material board with complex multi-layer wiring, the semantic image cannot be accurately positioned by relying solely on the distinction between different grayscales, especially complex wiring diagrams. After annotating specific target contour objects in the contour image, the area of ​​the semantic image can actually be determined based on the mapping relationship.

[0017] The use case data is the design requirement for (virtual) generating a material number image, which can be a specific program file that records the description of the target contour object. At least one marked target contour object can refer to the form of components, text, circuits or the entire carrier board in the image. For example Figure 2 The circular gold surface outline indicated by the arrow on the right side of the outline diagram (which can also be bolded or framed) is the marked target outline object. The description in the use case data then represents the operation performed on this circular gold surface. For example, if you create multiple copies of the circular gold surface and place them in designated locations, the generated virtual part number diagram will display a color image containing multiple circular gold surfaces.

[0018] Another important use of generating virtual part number diagrams using use case data is to synthesize a large number of use case images to represent different types of material boards. For example, a version upgrade or iterative update of a material board may show changes in the number, position, shape, and material of some metal surfaces, components, and circuits on the carrier board. Using use case data and a design database, a material template diagram can be quickly synthesized. For a specific material board, this quickly synthesized material template diagram can serve as a massive set of defect images for defect detection. Crucially, use case data is used to construct a variety of virtual part number boards with potentially minor variations, which serve as the basis for generating defect images and defect detection.

[0019] S2. Update the target contour object in the semantic image and / or contour image based on the use case data to generate a virtual material number map; The contour image and semantic image are both non-color images. The generated virtual material number image simulates the color virtual material number image. It does not render it into a specific color, but generates the same effect as the actual defect detection scene, based on the panoramic camera or high-definition camera shooting. Figure 3 This image shows a possible output of a color virtual part number for replicating a circular gold surface. Multiple replicas of the circular gold surface are distributed near the right side of the image. Because actual defect detection involves both dirt and color detection, high-precision color cameras are typically used. Different colors represent different textures and materials, which are key areas of focus during defect detection.

[0020] This process can be implemented using an AI model. Initially, labeled contour images, semantic images, and corresponding real-world material images are used to train an AI model specifically designed to generate color virtual material number maps. This allows for arbitrary use case data to be designed based on a small number of images, synthesizing a vast amount of homologous but distinct material board images, providing a vast amount of data for subsequent material board defect detection.

[0021] In the early training process of the material number synthesis model, the actual color material plate image is used as supervision. Based on the position of the input contour image and the semantic pixel content of the semantic image, the model is trained and regressed by extracting contour position features and semantic features.

[0022] During the actual virtual part number diagram generation phase, the annotated contour image, semantic image, and use case data are imported into the part number synthesis model. This allows the trained model parameters to accurately extract and analyze contour positional and semantic features, outputting a virtual part number diagram that meets the requirements. Figure 3 In the color virtual part number image shown, multiple replicated circular gold surfaces are distributed near the right side of the image. Of course, in some embodiments, all wafers, components, and circuits in the entire image can be rotated or offset before output, simulating the deviation caused by component placement or etching during the material board production process, and using this as a defect image to assist in later high-precision defect detection.

[0023] S3. Setting a defect outline in a virtual material number map, generating a material label map including the defect outline, and obtaining virtual defect data for synthesizing a virtual defect; In S2, a batch processing approach can be used to obtain a virtual material number atlas to store the generated massive virtual material number images. Then, in S3, the virtual material number images are retrieved from the virtual material number atlas one by one as needed. Note that the virtual material number images can be normal images or images containing defects.

[0024] The process of setting the defect outline involves planning the defect shape and size within the target area of ​​the image and assigning a corresponding outline label. The target area here refers to the specific area where the defect is to be generated, either through instructions or manual designation. This area can be any area in the image or an area that is actually prone to defects. This outline label is embedded in the material label image, and subsequent operations determine the defect outline type specified in the image by reading the outline label. Optionally, the specific defect outline can be annotated in the material label image for easy verification and reference by the operator.

[0025] Figure 4 A schematic diagram of generating a material label map in one possible form is shown. The left image is a virtual material number map obtained from the virtual material number map set before processing. The right image is a defect outline of an irregular circular area generated in the specified target area, which is the material label map.

[0026] In some embodiments, in order to obtain as many massive defect image training sets as possible, the present application will poll all virtual material number images in the virtual material number image set, and set the defect contour, contour label, and generate the corresponding defect label image according to the situation. All the generated defect label images are aggregated to construct a material label set. After that, the target material label images can be selected one by one, and then combined with the virtual defect data required to generate the defect, and specific defect pixels are generated in the defect contour.

[0027] The virtual defect data is designed to control the attributes and characteristic information of the output defects, such as text defects, dirt defects, scratch defects, board crack defects, etc. By inputting the attributes and characteristic information of the corresponding defects, a virtual defect map of the artificially controlled material defects can be output.

[0028] S4. Synthesize a virtual defect map based on the contour labels and virtual defect data in the material label map.

[0029] Figure 5A schematic diagram illustrates one possible method for generating a virtual defect map based on a labeled contour map. The left image shows the input material label map with the target contour label. The black area represents the ideal defect contour, while the right image shows the dirt defect generated in this contour area based on the virtual defect data. The white and gray pixels simulate the appearance of a real material board stained with dirt. The red contour line represents the additional labeling process used in this embodiment of the application, which is intended to omit the manual labeling step in the training set generation process. As is well known, the defect detection process relies on the identification of defect images in the training set for model training. This process requires input of labels with the labeled defect types, which is a fundamental step. In reality, all material images in the training set must be manually labeled to ensure that the model is fed with defect recognition capabilities before it can recognize unlabeled defect images. However, this embodiment automatically labels the virtual defect contours and generates defect labels when the practical image generator or AI model outputs the virtual defect map. In other words, the output of this solution is equivalent to directly completing the manual labeling step of establishing the training set, facilitating the subsequent defect training process, shortening the defect detection model development cycle, and indirectly improving defect detection efficiency.

[0030] The virtual defect data in this application can be a data file adapted for a model or system, or it can be displayed through a system interface, with corresponding data inputted through interactive interface operations, and then synthesized into a virtual image by calling a system database. For a specific target material label image, multiple different virtual defect data can be called based on actual conditions to obtain virtual defects under different standards. Combined with the massive label images in the material label set, an exponentially increasing number of virtual defect images can be obtained.

[0031] In summary, the present embodiment establishes a design database of material contour images and semantic images, identifies targets requiring design modification by annotating target contour objects in the contour images, and designs corresponding use case data. The contour position features in the contour images and the semantic features in the semantic images are fully identified and extracted, and the semantic and contour images are processed in conjunction with the processing information in the use case data to generate a color virtual material number map that meets the design requirements, thereby fully expanding the material board image.

[0032] Compared to traditional defect image training sets collected through photo-based defect recognition on production lines, this solution leverages diffusion model-like image generation techniques to generate more realistic defects, significantly reducing the collection cycle for small-sample defect data and rapidly improving the model's detection rate. Furthermore, this solution allows for targeted training for all defect types, even generating defect images for specific neighborhoods of high-incidence defects on production lines. This prioritizes optimizing the model's detection capabilities for specific defect types, improving production and inspection efficiency.

[0033] Because a virtual material number map that conforms to the use case data is generated by importing the material number synthesis model, the function and accuracy of the material number synthesis model are crucial to the output result. For this reason, the present application embodiment provides a training method for generating a material number synthesis model. Figure 6 As shown, the specific steps include: Step 601: Mark the target area in the actual material image with the corresponding texture material label; The material number synthesis model is generated based on image training from a material image training set, a semantic image training set, and a contour image training set. The material image training set contains actual material images with various material information; the semantic image training set contains semantic images corresponding to all actual material images; and the contour image training set contains contour images corresponding to all actual material images. Correspondingly, all images in the material image training set, the semantic image training set, and the contour image training set are mapped and matched. Selecting an image from one set also selects images from the other two sets.

[0034] The training phase relies on labeling images, so manual annotation is required early on. Texture material labels are used to record the texture and material information of various component circuits in color actual material images, such as textured gold-plated, tin-plated, and chrome-plated surfaces, as well as gold or copper wires adapted to different functions. Texture information can include the color, texture, and shading characteristics of a specific material.

[0035] In this step, the manually annotated texture material labels are mainly determined based on the contour image. The actual material map is determined by contour image mapping, and then the texture material labels are annotated based on the mapped target area and used as a model verification set.

[0036] Step 602: aligning the labeled actual material image with the corresponding semantic image and contour image, and mapping the labeled target area to the semantic image and contour image; This step mainly establishes a mapping relationship between the actual material image, semantic image and contour image, and determines the correspondence between the actually selected target object, target contour and target area in the entire image size to facilitate quick matching.

[0037] Step 603, extracting a target contour area and a target semantic area corresponding to the target area from the semantic image and the contour image respectively; In step 604, the pixel content of the target area in the actual material image is used as supervision, the target contour area and the labeled target semantic area are used as sample input, the contour position features in the contour image and the semantic features in the semantic image are extracted, and the material number synthesis model is trained.

[0038] When building an AI model, the loss structure type of the model should be given priority. Because the training is based on contour images and semantic images, and the verification is based on actual material images, it should be considered from two dimensions: target contour position and semantic pixel value. The label loss based on position recognition and the label loss based on pixel-level recognition should be constructed. The loss function It is expressed as follows:

[0039] Among them and represents the loss weight, represents pixel-level loss, represents the label classification loss; Indicates the image size, represents the real pixel value, Represents the model's predicted pixel value; represents the number of samples, Indicates the The true labels of samples, Represents the model prediction label.

[0040] By comparing the predicted label type with the actual label type, the material number synthesis model is obtained after cross-validation.

[0041] In some embodiments, after acquiring the use case data, the material number synthesis model further analyzes and determines the use case data. Specifically, the material number synthesis model determines the target operation image based on the type of information processed in the use case data. As previously mentioned, because the input image is divided into contour images and semantic images, and although the use case data is based on the contour annotation of at least one target contour object, its primary target can be either contour images or semantic images.

[0042] In this embodiment, the processed information may include contour data and texture material data. The contour data includes size and position information of at least one target contour object, and primarily acts on the contour image. The texture material data includes texture information and material information of at least one target contour object, and primarily acts on the semantic image, specifically, the pixel region that matches the target contour object.

[0043] The following discusses contour images and semantic images separately.

[0044] 1. When the target operation image is a contour image, the process of updating the target contour object in the contour image based on the use case data can be summarized as follows: 1) Map the target contour object to the semantic image and determine the target adjustment area and the corresponding texture material label; 2) Adjust the target contour object according to its size and posture information to generate an adjusted contour map; The specific process needs to be determined according to the adjustment requirements, and the texture material label of the target contour object remains unchanged before and after the adjustment. This is the key to generating a qualified color virtual image.

[0045] In this embodiment, the size and posture information includes but is not limited to size, offset, angle, shift, increase or decrease, and scaling ratio. The corresponding operations are to adjust the size, coordinate offset, shift, angle, increase or decrease the target contour object, or scale the target contour object.

[0046] Figure 7 The diagram shows how to perform offset processing, angle rotation processing, and size enlargement or reduction processing on the target metal surface in the actual material image. In actual operation, it is also possible to shift, rotate, and scale up or down the content of multiple or the entire image.

[0047] Of course, in some embodiments, there are situations where text, components, etc. are added or deleted in a specific area (such as a carrier board). In this case, the target outline object can represent the carrier board, and the corresponding operation can be performed in combination with the added or deleted position coordinate information.

[0048] The contour adjustment is based on the original contour image, which is an operation to adjust the target contour line according to the needs. Figure 2 The outline in the figure is enlarged to illustrate the scale. Figure 8 This is a schematic diagram of how to magnify a circular gold face to generate an adjusted contour map and semantic map. The arrows in the left and right figures indicate the display effect after the magnification operation.

[0049] 3) Remap the adjusted contour image to the semantic image, and fill and update the semantic pixel values ​​of the target adjustment area in the semantic image according to the contour position and texture material label in the adjusted contour image.

[0050] Because the newly generated target contour changes, the coordinate data is also updated. This is why it is necessary to synchronize the changes to the semantic information. The inspiration of this synchronization process is pixel recognition and contour filling. The changes in the corresponding pixel area of ​​the original target contour are redefined according to the new target contour area, and then the semantic pixel value is filled in according to the original label information. Figure 8 The display effect is shown in the middle right picture.

[0051] 2. When the target operation image is a semantic image, the process of updating the target contour in the semantic image based on the use case data can be summarized as follows: 1) Map the target contour object to the semantic image and determine the target adjustment area and the corresponding texture material label; 2) According to the size and posture information of the target contour object, the texture information and / or material information of the target adjustment area is modified, a new texture material label is regenerated, and the position is matched with the target contour object in the contour image.

[0052] This step mainly targets changes in materials and textures, such as changing a gold-plated surface to a chrome-plated surface, or changing a copper wire to a gold wire. Such changes are not visible on the contour map, so they can only be located through contours and then modified in the semantic map, that is, updating the semantic pixel values ​​of the target adjustment area.

[0053] Of course, in actual use cases, the size and posture information of multiple targets are often obtained, and multiple modifications to the texture and material are also included to achieve the purpose of generating massive virtual images.

[0054] Figure 9 The schematic diagram of generating a virtual material number image by composite modification of a material image is shown as an example. The leftmost column shows the original material image, the second column shows the semantic image after grayscale processing, and the third column shows the modified semantic image generated based on the use case data. It can be clearly seen in the re-synthesized virtual material number image on the far right that the texture of the rectangular gold surface in the first row has been changed to a solid color, and the material connecting the bent gold wire and the substrate in the second row has been modified. Similarly, the texture material of the inner ring of the circular crystal surface in the fourth row has been replaced. By comparing the slight modification differences before and after the first and fourth columns, it is possible to simulate the local etching error of the material plate, and also to represent the material and size differences of material plates with different material numbers. The defect image training set generated by these material images can provide rich detection experience for subsequent defect detection.

[0055] The following is a detailed introduction to the process of further synthesizing a virtual defect map based on the virtual material number map.

[0056] In some embodiments, the process of generating a material label image based on a virtual material number image requires the targeted generation of defects of a certain size based on instructions or specific requirements. This requires controlling the defect outline type and scale (size). For example, common defects include point defects (spot dirt, perforations, missing parts) and track defects (scratches, large-area dirt, printing defects). Therefore, in this embodiment, it is necessary to first specify the defect outline type, namely point outline and track outline. Point outlines are contour areas generated based on the input point pixel coordinates, while track outlines are contour areas enclosed by closed track lines. Different types of contour defects require specific analysis and synthesis based on shape and scale.

[0057] To this end, this application provides a method for generating different types of material label images, such as Figure 10 As shown, the method may include the following steps: Step 1010, when an editing operation on a target area in the virtual material number map is detected, determining the pixel area of ​​the pixel editing area; This solution can be based on human-computer interaction on the display interface or edit operations determined by command files. For example, clicking a mouse in a target area of ​​an image generates a point defect outline; dragging the mouse across the target area creates a sliding track that represents the defect outline. Command files can contain a series of coordinate points, and the track formed by these coordinate points can be used to represent point operations and sliding track operations.

[0058] For click operations, multiple pixels of the graphic are usually determined. These pixels are then used as the pixel editing area, and a pixel area can be determined. For sliding track operations, when a closed track line is slid out, the enclosed area is the pixel editing area, and a pixel area can also be determined.

[0059] Step 1020 , when the pixel area of ​​the pixel editing region is smaller than the preset pixel area value, it is determined as a point outline editing operation, and a point defect outline of a target shape is generated based on the pixel editing region; Figure 11 This diagram shows how to set a track defect outline and a point defect outline on a material label image. The first pixel area 1110 represents the pixel editing area (dashed circle) for the point outline editing operation. This area represents the pixel point touched by the mouse image or the range of pixels covered by several coordinates in the instruction. The area value is determined by counting the number of pixels. If the pixel area is less than the preset pixel area value, it indicates that the operation is a point outline editing operation.

[0060] Since the pixel points touched by the mouse during point outline editing are small, and the image itself is a high-definition image, the tiny dust particles are actually equivalent to the first pixel area 1110. Therefore, when actually generating defects, it is usually necessary to generate an area larger than the pixel editing area, that is, Figure 11 The target shape of the point defect outline 1130 is generated in the above process. The target shape is usually set by the system and can be an arc, a circle, a rectangle, or other irregular shapes according to the design requirements. Figure 11 The irregular shapes are shown in .

[0061] Step 1030 , when the edited pixel area is not less than the preset area value, it is determined as a track contour editing operation, and the contour enclosed by the pixel editing area is determined as the track defect contour; Figure 11 The second pixel area 1120 represents the pixel editing area enclosed by the trajectory contour editing operation. Since the area generated by sliding the mouse or according to the trajectory coordinates is usually larger, the pixel editing area is the area actually enclosed by the contour line.

[0062] Step 1040 : Modify the internal pixels of the generated point defect outline / track defect outline to target pixels to generate a material label image.

[0063] This step is essentially set up to facilitate system operation and re-inspection operations. For example, when the human-computer interaction interface or automatic operation interface is displayed, in order to facilitate the review of the process and defect design effects, the contour content needs to be filled with a different color. The fill color only needs to be different from the original pixel value inside the contour, such as setting an inverted color fill. Specifically, after determining the contour area, the pixel values ​​within the area are uniformly sampled according to the set sampling density and the average pixel value is calculated. Then, based on the average pixel value, the inverted pixel value (target pixel value) is determined. After that, the pixels inside the point defect contour / track defect contour can be filled according to the inverted pixel value.

[0064] Figure 12 A schematic diagram of generating a material label diagram by filling different colors inside a point defect outline is shown in one possible form. In the figure, the target shape for generating the point defect outline is set to a butterfly shape, and the pixel outline area is filled with green. Then, when subsequently generating defective pixels, operations are also performed within this green-filled area.

[0065] Of course, in some other embodiments, the defect profile type can be determined directly based on an input command. This command directly defines the defect profile or trajectory defect profile and the corresponding coordinate points, which is equivalent to directly determining the pixel editing area and pixel area. Furthermore, the internal pixels of the generated point defect profile / trajectory defect profile are modified to the target pixels to generate a material label image.

[0066] Regardless of whether command input or interface interaction is used, the point defect outline and the pixel editing area do not overlap. In the case of generating an irregular target shape, special provisions need to be made for the position. When a point outline editing operation is determined according to the input command, the point pixel coordinates are used as the pixel center coordinates, and the point defect outline is generated; the point pixel coordinates can be specified in the command, or they can be determined by calculating the center point based on multiple pixel coordinates in the command, and the center point is determined as the point pixel coordinates. When a point outline editing operation is determined according to the editing operation, the pixel center coordinates of the pixel editing area are determined, and the point defect outline of the target shape is generated based on the pixel center coordinates. For example Figure 11 The center coordinates of the first pixel area 1101 are the point pixel coordinates.

[0067] In some embodiments, the process of synthesizing a virtual defect map based on the target contour label and the virtual defect data may be implemented by the following steps: A. Determine the defect contour type and the pixel editing area based on the target contour label; B. Extracting target defect types and target texture material information calibrated in virtual defect data; In step A, the defect profile type is determined, i.e., the point defect profile or the trajectory defect profile, while in step B, the target defect type refers to a specific defect type, such as dirt, board cracks, scratches, or perforations. Different defects will present different textures and colors on the material board. This is the information recorded in the virtual defect data.

[0068] C. Generate defect pixels in the pixel editing area based on the target defect type and target texture material information, and output a virtual defect map with defect pixels.

[0069] In some embodiments, the generation of defective pixels requires combining the specific defect type. The defect type, rather than the defect outline type, is the key data used to distinguish different defect textures and materials. When it is necessary to generate a large number of defective images of different types, the virtual defect data cannot contain the defective pixel data of that type. Therefore, it is necessary to establish an additional database to call the material based on the virtual defect data for generation. To this end, the present application provides a method for calling the defect material library to generate defective pixels. The solution includes the following steps: D. Extract target defect materials that match the target defect type from the defect material library; Theoretically, if you want to generate various virtual defect images, the defect material library should cover all defect materials as much as possible. The defect material can be image data or code metadata stored in the database, etc. The image data can directly extract pixel texture features, and the code metadata can generate pixel texture features and image materials through execution. This application takes the storage of image materials as an example for explanation.

[0070] E. Scale the target defect material based on the pixel area of ​​the pixel editing area and map it to the pixel editing area; This step is to adjust the target defect material to match the pixel editing area.

[0071] F. Match and modify the texture material of the target defect material based on the target texture material information to generate defective pixels.

[0072] The modification of texture materials includes multiple levels, including texture parameter modification and density modification. For example, the texture of the gold-plated surface and the texture of the chrome-plated surface have different pixel values ​​with different thicknesses and contents. If the target defect material is directly enlarged, the texture spacing will change significantly, similar to the effect of the increase in the spacing between the annual rings of a tree trunk. This texture density will change with the size scaling. Therefore, this embodiment first records the texture density of the original material and the defect texture density, and then performs size scaling.

[0073] After the mapping is completed, the original material texture density and defect texture density are obtained, and then the material texture and defect texture density mapped to the pixel editing area after scaling are modified (modified to the original material texture density and defect texture density) based on the original material texture density and defect texture density to regenerate the defective pixels.

[0074] Figure 13 Therefore Figure 11 Schematic diagram of the synthesis of virtual defect maps using midpoint defect contours or trajectory defect contours. The two contour areas correspond to different contour types and defect types, respectively. Two virtual defects of different types and scales are generated based on the original virtual material number map.

[0075] In some embodiments, the above-mentioned method of synthesizing virtual defect images can be completed through an AI model, that is, designing a defect image generator, and then directly inputting the defect image or material label image, and then outputting one or more virtual defect images with different virtual intensities according to needs.

[0076] In one possible implementation, a defect image generator can be provided, and a GAN model based on an adversarial network, a diffusion model based on a diffusion path, and an image generation model, etc., can be built into the defect image generator. The defect image generator receives a material label image and generates at least one virtual defect image output based on all built-in models. Figure 14 Different generative models are used to synthesize and output virtual defect images with varying defect intensities. Different virtual intensities can help defect detection models converge quickly. For example, in the early stages of training, training focuses on images with low virtual intensities to achieve rapid network convergence. Later in the training phase, training focuses on images with high virtual intensities to improve the model's detection accuracy and capabilities.

[0077] In summary, the implementation of this plan can bring the following beneficial effects: Quality: Improve detection accuracy: By generating defect images that are close to reality, the model can learn more defect features, reduce missed detections and false detections, and improve product quality. Improve quality stability: Prepare materials and adjust parameters in advance to reduce quality fluctuations in the production process, making the production process more controllable. Cost: Reduce sample collection costs: Reduce reliance on actual defect samples, saving manpower, material resources, and time costs for sample collection; Reduce production preparation costs: Prepare in advance to reduce additional costs caused by delays in physical material supply and improper parameter adjustments.

[0078] Delivery: Shorten delivery cycle: quickly launch material numbers and improve defect detection efficiency, reduce production downtime, and ensure on-time product delivery; Improve delivery flexibility: Quickly adjust production plans and switch material numbers in a timely manner to meet customer order changes.

[0079] Service: Improve customer satisfaction: High-quality products reduce customer complaints, fast delivery and flexible response to customer needs enhance customer experience; Enhance market competitiveness: With advantages in quality, cost and delivery, establish a good brand image, attract more customers and enhance market competitiveness.

[0080] Figure 15 The structure block diagram of the virtual defect image generation device provided in an embodiment of the present application is shown, and the device includes: The contour annotation module 1510 is configured to obtain a contour image and a semantic image of a material board, annotate a target contour object in the contour image, and obtain use case data for the target contour object; the contour image and the semantic image correspond one-to-one, and the use case data includes processing information for updating the target contour object; A first image generation module 1520 is configured to update the target contour object in the semantic image and / or contour image based on the use case data to generate a virtual material number map; The second image generation module 1530 is configured to set a defect outline in the virtual material number image, generate a material label image including the defect outline, and obtain virtual defect data for synthesizing a virtual defect; the outline label in the material label image is used to indicate the defect outline type; The defect synthesis module 1540 is configured to synthesize a virtual defect map based on the contour labels in the material label map and the virtual defect data.

[0081] The virtual defect image generation device provided in the embodiment of the present application can be applied to the virtual defect image generation method provided in the above embodiment. For relevant details, please refer to the above method embodiment. Its implementation principles and technical effects are similar and will not be repeated here.

[0082] It should be noted that the virtual defect image generation device provided in the embodiments of the present application is only illustrated by the division of the above-mentioned functional modules / functional units. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the virtual defect image generation device can be divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the virtual defect image generation method provided in the above-mentioned method embodiment and the implementation method of the virtual defect image generation device provided in this embodiment belong to the same concept. The specific implementation process of the virtual defect image generation device provided in this embodiment is detailed in the above-mentioned method embodiment and will not be repeated here.

[0083] Figure 16The following is a block diagram of the structure of a computer device provided by an exemplary embodiment of the present application. The computer device may be a desktop computer, a laptop computer, a PDA, or a cloud server. The computer device may include, but is not limited to, a processor and a memory. The processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, graphics processing units (GPUs), embedded neural network processing units (NPUs) or other dedicated deep learning coprocessors, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above chips. The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor can be implemented in at least one of the following hardware forms: DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to handle computing operations related to machine learning.

[0084] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method embodiment is implemented. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0085] In some embodiments, the computer device may optionally include a peripheral device interface and at least one peripheral device. The processor, memory, and peripheral device interface may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface via a bus, signal lines, or circuit boards. Specifically, the peripheral device includes at least one of a radio frequency circuit, a display screen, and a keyboard.

[0086] The peripheral device interface can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor and memory. In some embodiments, the processor, memory, and peripheral device interface are integrated on the same chip or circuit board. In other embodiments, any one or two of the processor, memory, and peripheral device interface can be implemented on separate chips or circuit boards, although this embodiment is not limited to this.

[0087] The display screen is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, or any combination thereof. When the display screen is a touch screen, it also has the ability to capture touch signals on or above the surface of the display screen. The touch signals can be input as control signals to a processor for processing. In this case, the display screen can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there can be one display screen, disposed on the front panel of the computer device; in other embodiments, there can be at least two display screens, disposed on different surfaces of the computer device or in a foldable design; in still other embodiments, the display screen can be a flexible display screen, disposed on a curved or foldable surface of the computer device. Furthermore, the display screen can be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. The display screen can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0088] Those skilled in the art will understand that the structure shown in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0089] The embodiment of the present application also discloses a computer-readable storage medium. Specifically, the computer-readable storage medium is used to store a computer program, and when the computer program is executed by the processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art will understand that all or part of the processes in the above-mentioned method implementation of the present application can be completed by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the implementation of the above-mentioned methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0090] This specific embodiment is merely an explanation of the present invention and is not intended to limit the present invention. After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed. However, as long as such modifications are within the scope of the claims of the present invention, they are protected by patent law.

Claims

1. A method for generating a virtual defect image, characterized in that: The method comprises: Acquire a contour image and a semantic image of a material board, annotate a target contour object in the contour image, and acquire use case data for the target contour object; the contour image and the semantic image correspond one-to-one, and the use case data includes processing information for updating the target contour object; updating the target contour object in the semantic image and / or contour image based on the use case data to generate a virtual material number map; Setting a defect outline in the virtual material number map, generating a material label map containing the defect outline, and obtaining virtual defect data for synthesizing a virtual defect; the outline label in the material label map is used to indicate the defect outline type; A virtual defect map is synthesized based on the contour labels in the material label map and the virtual defect data.

2. The method according to claim 1, characterized in that After obtaining the use case data, the method further includes: Determine a target operation image according to the type of processing information in the use case data; the target operation image is a contour image or a semantic image; The processing information includes contour data and texture material data; the contour data includes size and posture information for operating at least one target contour object, and the texture material data includes texture information and material information for operating at least one target contour object.

3. The method according to claim 2, characterized in that When the target operation image is a contour image, updating the target contour object in the semantic image based on the use case data includes: Mapping the target contour object to a semantic image to determine a target adjustment area and a corresponding texture material label; Adjusting the target contour object according to the size and posture information of the target contour object to generate an adjusted contour map; the texture material label of the target contour object remains unchanged before and after the adjustment; Remapping the adjustment contour image into the semantic image, and filling and updating the semantic pixel values ​​of the target adjustment area in the semantic image according to the contour position and texture material label in the adjustment contour image; Among them, the size and posture information includes at least one of size, offset, angle, shift, increase or decrease, and scaling ratio; the adjustment content includes size adjustment, coordinate offset adjustment, shift adjustment, angle adjustment, increase or decrease of the target contour object, or scaling adjustment of the target contour object.

4. The method according to claim 2, characterized in that When the target operation image is a semantic image, updating the target outline in the semantic image based on the use case data includes: Mapping the target contour object to a semantic image to determine a target adjustment area and a corresponding texture material label; According to the size and posture information of the target contour object, the texture information and / or material information of the target adjustment area is modified, a new texture material label is regenerated, and position matching is performed with the target contour object in the contour image.

5. The method according to claim 3 or 4, characterized in that The generating of the virtual material number map includes: Taking the target contour area and the target semantic area with texture material labels as input, the contour position features in the contour image and the semantic features in the semantic image are extracted, and the virtual material number map is generated by combining the contour position features and the semantic features in the semantic image.

6. The method according to claim 1, characterized in that The defect contour types include point contour and track contour; the point contour is a contour area generated based on the point pixel coordinates, and the track contour is a contour area surrounded by a closed trajectory line; The step of setting a defect outline in the virtual material number map and generating a material label map including the defect outline includes: When an editing operation on a target area in the virtual material number map is detected, determining a pixel area of ​​the pixel editing area; When the pixel area of ​​the pixel editing region is less than a preset pixel area value, it is determined as a point outline editing operation, and a point defect outline of a target shape is generated based on the pixel editing region; when the edited pixel area is not less than the preset area value, it is determined as a track outline editing operation, and the outline enclosed by the pixel editing region is determined as the track defect outline; The internal pixels of the generated point defect outline / track defect outline are modified to target pixels to generate the material label image.

7. The method according to claim 6, characterized in that When the editing operation is determined to be a point outline editing operation, determining the pixel center coordinates of the pixel editing area, and generating the point defect outline of the target shape according to the pixel center coordinates; The target shape of the point defect contour is determined according to the target area of ​​the material plate.

8. The method according to claim 6, characterized in that The synthesizing a virtual defect map based on the contour label in the material label map and the virtual defect data includes: determining a defect contour type and the pixel editing area therein based on the contour label; Extracting target defect type and target texture material information calibrated in the virtual defect data; Defective pixels are generated in the pixel editing area based on the target defect type and target texture material information, and the virtual defect map with the defective pixels is output.

9. The method according to claim 8, characterized in that Generating defective pixels in the pixel editing area based on the target defect type and target texture material information includes: Extracting target defect materials that match the target defect type from a defect material library; Scaling the target defect material based on the pixel area of ​​the pixel editing area and mapping it to the pixel editing area; The texture material of the target defective material is matched and modified based on the target texture material information to generate defective pixels.

10. The method according to claim 9, characterized in that The matching and modifying the texture material of the target defective material based on the target texture material information to generate defective pixels includes: Determine the original material texture density and the defect texture density in the target defect material, modify the material texture and defect texture density mapped to the pixel editing area after resizing based on the original material texture density and the defect texture density, and regenerate the defective pixels.

Images