A defect image generation method and device, computer equipment and storage medium

Abstract

This application relates to a method, apparatus, computer device, and storage medium for generating defect images. The method includes: acquiring a large-scale model trained using a glass industry dataset for identifying glass defects, and constructing an image generation network based on the backbone network of the large-scale model; performing multimodal fusion processing on the acquired mask images of multiple defects, multiple descriptive words for defects, and the defect center position corresponding to each descriptive word to obtain multimodal fusion features; determining the quantized feature parameters of each defect using the image generation network based on the multimodal fusion features; generating a multi-defect feature map using the image generation network based on the multimodal fusion features and the quantized feature parameters of each defect; and fusing the acquired original image of defect-free glass in the target scene with the multi-defect feature map to obtain a target defect sample image matching the target scene. Using this application can effectively improve the generation efficiency of sample images including multiple defects.

Description

Technical Field

[0001] This application relates to the field of sample generation technology, and in particular to a method, apparatus, computer device and storage medium for generating defect images. Background Technology

[0002] In the current field of electronic product manufacturing, glass covers are core structural components, and their macroscopic structure is highly consistent: the outer ring has an ink layer (some of which have a pre-reserved hole structure), and the edges are often treated with a curved edge process; the inside is a uniform viewing area, and the viewing area can use different surface processes such as AG (anti-glare) and frosted according to product requirements.

[0003] In existing technologies, the training of glass cover defect detection models and the acquisition of defect samples mainly rely on the following methods: First, relying on the collection of real defect samples, by screening defective glass covers in actual production and manually labeling them to form training samples; second, using traditional generative models to generate defect samples. However, existing technologies have significant shortcomings in practical applications, especially in the defect sample generation stage. Traditional generative models can only generate a single defect sample at a time. If it is necessary to obtain samples containing multiple defects or multiple defects of the same type, the generation process must be repeated, resulting in extremely low generation efficiency and making it difficult to meet the rapid demand for a large number of diverse defect samples in the early stages of new projects.

[0004] Therefore, a new method for generating defective images is urgently needed. Summary of the Invention

[0005] Therefore, it is necessary to provide a defect image generation method, apparatus, computer device, computer-readable storage medium, and computer program product to address the above-mentioned technical problems, which can effectively improve the generation efficiency of sample images including various defects.

[0006] In a first aspect, this application provides a method for generating defective images, including:

[0007] Obtain a large model for identifying glass defects trained using a glass industry dataset, and construct an image generation network based on the backbone network of the large model.

[0008] Multimodal fusion processing is performed on the mask images of multiple defects, the prompt words describing multiple defects, and the defect center position corresponding to each prompt word to obtain multimodal fusion features;

[0009] The quantitative feature parameters of each defect are determined by using an image generation network based on multimodal fusion features.

[0010] A multi-defect feature map is generated using an image generation network based on multimodal fusion features and the quantized feature parameters of each defect.

[0011] The original image of the defect-free glass in the target scene is fused with the multi-defect feature map to obtain a target defect sample image that matches the target scene.

[0012] Secondly, this application provides a defect image generation apparatus, comprising:

[0013] The network construction module is used to obtain a large model for identifying glass defects trained using a glass industry dataset, and to build an image generation network based on the backbone network of the large model.

[0014] The feature fusion module is used to perform multimodal fusion processing on the mask images of multiple defects, the prompt words describing multiple defects, and the defect center position corresponding to each prompt word to obtain multimodal fusion features;

[0015] The determination module is used to determine the quantization feature parameters of each defect based on multimodal fusion features through the image generation network;

[0016] The generation module is used to generate multi-defect feature maps by using an image generation network, based on multimodal fusion features and quantized feature parameters of each defect;

[0017] The image fusion module is used to fuse the original image of the defect-free glass in the target scene with the multi-defect feature map to obtain a target defect sample image that matches the target scene.

[0018] Thirdly, this application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the method described above.

[0019] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0020] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described above.

[0021] The aforementioned defect image generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product reuse the backbone network of a large-scale defect recognition model trained on a glass industry dataset to construct an image generation network. Through multimodal fusion of multiple defect masks, prompts, and defect center locations, the quantified feature parameters of defects are determined, and multi-defect feature maps are generated. These feature maps are then fused with the original image of defect-free glass to obtain a target defect sample image of the target scene. Through the chain of multimodal fusion, quantified feature parameter determination, feature map generation, and original image fusion, a large number of high-quality defect samples can be generated rapidly, significantly shortening the startup cycle of new projects. It supports the generation of multiple and various types of defects at once, improving the efficiency of defect image generation, reducing the time cost of sample generation, and providing data support for the rapid iteration of defect detection models. Attached Figure Description

[0022] Figure 1 This application environment diagram illustrates a defect image generation method provided in an embodiment of this application.

[0023] Figure 2 A flowchart illustrating a defect image generation method provided in an embodiment of this application;

[0024] Figure 3 A structural block diagram of a defect image generation device provided in an embodiment of this application;

[0025] Figure 4 An internal structural diagram of a computer device provided in an embodiment of this application;

[0026] Figure 5 An internal structural diagram of another computer device provided in an embodiment of this application;

[0027] Figure 6 This is an internal structural diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] The defect image generation method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0030] like Figure 2 As shown, this application provides a method for generating defect images, which is applied to... Figure 1 The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps:

[0031] S201. Obtain a large model for identifying glass defects trained using a glass industry dataset, and construct an image generation network based on the backbone network of the large model.

[0032] The glass industry-specific dataset refers to an industry-level, precisely labeled, multimodal dataset containing tens of thousands of data pairs of glass from various sizes and processes, including mobile phones, tablets, laptops, and televisions. Each data pair includes a glass image, defect annotations, and text comments (recording the defect location, name, and severity), forming the core data foundation for training the large-scale model. The large-scale model used to identify glass defects is an industry-specific model trained on the aforementioned dataset, capable of accurately identifying fine-grained defect information. The backbone network of the large-scale model is the core architecture responsible for feature extraction, carrying industry knowledge such as glass structure, defect features, and process adaptation, and is key to the image generation network's ability to reuse industry-specific data. The image generation network is used to generate glass defect sample images, built upon the backbone network of the large-scale model, and possesses multimodal feature processing and defect feature generation capabilities.

[0033] In this embodiment, the main approach is to first acquire a large industry-specific model with fine-grained glass defect recognition capabilities, extract its core feature extraction architecture (i.e., backbone network) that carries industry knowledge, and then supplement it with functional modules (such as feature fusion, decoding, and generation layers) adapted to the defect generation task to construct an image generation network that can reuse industry knowledge.

[0034] In practice, a large-scale glass defect recognition model trained using a glass industry-specific dataset is obtained, and all parameters of its backbone network are extracted using a model parameter parsing tool. When constructing the image generation network, the structure of its backbone network (number of layers, number of attention heads, feature dimensions) is kept completely consistent with the backbone network of the large-scale model, and the extracted parameters are directly loaded. Then, based on the backbone network, functional modules adapted to the defect generation task are added. The image generation network constructed in this way inherits the industry knowledge of the large-scale model.

[0035] This step avoids the image generation network learning glass industry knowledge from scratch (such as AG frosting process, ink area / window area differences), significantly shortening the training cycle of the generation network and reducing model development costs. Relying on the fine-grained recognition capabilities of the backbone network of the large model, it provides knowledge support for the accurate determination of subsequent defect quantification feature parameters, solving the problem that traditional generation networks do not understand the industry defect patterns.

[0036] S202. Perform multimodal fusion processing on the mask images of multiple defects, the prompts describing multiple defects, and the center position of the defect corresponding to each prompt to obtain multimodal fusion features.

[0037] The mask images for multiple defects are pure black images of the same size as the original defect-free glass image. Pure white regions independently represent the location and shape of multiple defects (e.g., linear pure white regions for scratches in the ink area, circular pure white regions for bubbles in the viewing window area), and the pure white regions of different defects do not overlap. The defect description prompts are structured text composed according to a specified format (e.g., generating a minor ink scratch in the ink area by combining the defect mask). Each prompt corresponds to one defect, clearly specifying the defect's location, severity, and type. Multiple prompts are passed in a list format. The defect center position refers to the pixel-level center coordinates of the defect corresponding to each prompt on the original glass image, used to accurately anchor the defect generation location. The center position coordinates of multiple defects are passed in a list format. Multimodal fusion processing involves transforming the features of the image modality (defect mask), text modality (prompt), and location modality (center coordinates) into a unified dimension, and strengthening the intermodal correlation through an attention mechanism to obtain fused features that include defect location, shape, and semantic requirements.

[0038] In implementing multimodal fusion processing, features of each modality are extracted separately. For example, image-coded features of the defect mask are extracted using an image encoder that is of the same origin as the backbone network of the large model; text-coded features of the prompt are extracted using a text encoder; and the center position coordinates are transformed into position-coded features using a two-dimensional positional encoding algorithm. Then, a cross-attention mechanism is used to complete the fusion of image-coded features, text-coded features, and positional encoding features. It should be noted that, for cases with multiple defects, the three modal features of each defect are independently fused and then concatenated into a unified fused feature vector.

[0039] Understandably, this step breaks down information barriers between different modalities, solving the problem of defect generation bias caused by traditional generative models relying on only a single modality (such as a label), and ensuring the accurate transmission of generation requirements. The modal features of multiple defects are processed independently and then fused, laying the data foundation for generating multiple defects at once and avoiding confusion of information from multiple defects.

[0040] S203. Using an image generation network, determine the quantization feature parameters of each defect based on multimodal fusion features.

[0041] In this application, the backbone network of the image generation network is the feature extraction architecture constructed in step S201, which reuses the knowledge of the large-scale glass defect recognition model. It has the ability to parse multimodal features and output industry-specific defect parameters. The quantified feature parameters are quantifiable defect detail parameters determined based on glass industry knowledge (such as slight ink scratches "width 1-2 pixels, contrast 30%-40%", bubbles in the normal viewing area "diameter 5-6 pixels, roundness ≥0.8"), which are the specific standards for generating defect feature maps.

[0042] In some embodiments, this step primarily involves reusing the industry-knowledge-based backbone network within the image generation network to parse, infer, or match the morphological, semantic, and positional information in the multimodal fusion features, outputting quantitative feature parameters for each defect that conform to glass industry standards and can directly guide defect generation. In specific implementations, the backbone network performs layered processing on the multimodal fusion features. The bottom-layer features parse the pixel-level morphology of the mask image (e.g., the length of scratch lines, the radius of bubbles), the middle-layer features extract semantic constraints of the prompt (e.g., the parameter range corresponding to "slight"), and the high-layer features combine positional information to determine process adaptation requirements (e.g., scratches in the AG frosted area require superimposed particle texture at a 45° angle). The fully connected layers of the backbone network infer the layered features, dynamically calculating the quantitative feature parameters for each defect (e.g., based on the pixel length of the scratch lines in the mask image, combined with the "slight" semantic constraint, calculating a width of 1.2 pixels and a contrast of 32%).

[0043] Understandably, by determining the quantitative feature parameters of each defect based on industry-specific large-scale model knowledge, the problem of unrealistic defects caused by relying on experience to set parameters in traditional generation models is avoided, ensuring that the generated defects conform to industrial reality. It also provides accurate quantitative standards for each defect, enabling the subsequently generated defect feature maps to have pixel-level precision, thus solving the difficulty of fuzzy defect features in traditional generation tasks.

[0044] S204. Using an image generation network, generate a multi-defect feature map based on the multimodal fusion features and the quantized feature parameters of each defect.

[0045] Among them, the multi-defect feature map refers to the feature map with the same size as the original image of the defect-free glass. It carries the quantitative features of multiple defects (such as scratch lines, bubble outlines, and process textures) with pixel-level precision. Each defect feature is independent and does not overlap, and it serves as a defect template for subsequent fusion with the original image.

[0046] In some embodiments, in this step, the image generation network, based on the defect generation requirements determined by multimodal fusion features, and combined with the quantized feature parameters of each defect, independently generates the visual features of each defect on a carrier with the same size as the original image through feature decoding, pixel rendering, or adversarial optimization, forming a multi-defect feature map without background interference and with accurate features. Specifically, the feature decoding layer of the image generation network receives the multimodal fusion features and quantized feature parameters, decodes the fusion features into a defect location mask (marking the pixel range of each defect) with the same size as the original image, and converts the quantized feature parameters into rendering instructions (such as "draw a 1.5-pixel wide line with 35% contrast in the (22-28, 52-58) pixel range"); according to the defect location mask and rendering instructions, pixel-level rendering is performed on a pure black background: each defect is drawn independently (such as the linear outline of a scratch, the circular filling of a bubble), and a process texture (such as AG frosted particles) is superimposed to generate a multi-defect feature map.

[0047] In some implementations, multimodal fusion features and quantized feature parameters are input into the generator of the image generation network (the backbone network is the generator's feature extraction module). The generator generates an initial multi-defect feature map based on the quantized feature parameters. Simultaneously, the initial multi-defect feature map is input into a discriminator (the discriminator's parameters partially reuse the defect recognition capabilities of the large model). The discriminator determines whether the defect features conform to the quantized feature parameters and industry standards. If the discriminator output does not conform (e.g., the bubble's roundness is only 0.7), an adjustment signal is fed back to the generator. The generator corrects the rendering parameters based on the signal (e.g., adjusting the bubble's outline pixels), iterating until the discriminator determines that it conforms, and then outputs the final multi-defect feature map.

[0048] This step enables the simultaneous generation of multiple defects, solving the efficiency problem of traditional defect generation which can only generate one defect at a time. Based on quantified feature parameters, defect features are generated to ensure that the shape, size, and process adaptability of each defect meet the requirements, avoiding the problem of large deviations between the defect features of traditional generation models and expectations.

[0049] S205. The original image of the defect-free glass in the target scene is fused with the multi-defect feature map to obtain a target defect sample image that matches the target scene.

[0050] The target scenario refers to any application scenario including mobile phones, tablets, laptops, automotive, and televisions. The original, defect-free glass image refers to a clean glass image adapted to a specific application scenario (such as mobile phones or automotive), fully presenting the macroscopic structure of the glass (ink area, viewing window area) and its technological features (curved edges, AG frosted finish), serving as the basic carrier for defect fusion. Fusion involves pixel-level overlay and transition processing of defect features from multiple defect feature maps with corresponding areas of the original, defect-free glass image, ensuring seamless integration of defects and the original image in terms of process, lighting, and texture. The target defect sample image refers to defect samples that meet the requirements of the target scenario and glass industry standards, and can be directly used for defect detection model training or dataset construction.

[0051] This step primarily involves aligning the original image of the defect-free glass in the target scene with the multi-defect feature map. Through pixel weighting, process adaptation, or lighting adjustments, the defect features are seamlessly superimposed onto the corresponding areas of the original image to obtain the final defect sample image. Specifically, the original image of the defect-free glass and the multi-defect feature map are aligned in coordinates (ensuring the defect features match the original image's position). Pixels in the defect area of ​​the multi-defect feature map are extracted and compared with corresponding pixels in the original image. A weighted fusion formula is used to calculate the fused pixel values. Simultaneously, the lighting in the defect area (e.g., brightness, shadow direction) is adjusted to match the global lighting of the original image, generating the final defect sample image.

[0052] The defect samples generated in this step are highly matched with the target scene, solving the problem of the disconnect between traditional generated model samples and actual application scenarios, and can directly support the model training of new projects.

[0053] The embodiment of this application reuses the backbone network of a large-scale defect recognition model trained on a glass industry dataset to construct an image generation network. Through multimodal fusion of multiple defect masks, prompts, and defect center locations, quantitative defect feature parameters are determined and multi-defect feature maps are generated. These maps are then fused with the original image of defect-free glass to obtain a target defect sample image of the target scene. By following the chain of multimodal fusion, quantitative feature parameter determination, feature map generation, and original image fusion, a large number of high-quality defect samples can be generated rapidly, significantly shortening the startup cycle of new projects. It supports the generation of multiple and various types of defects at once, improving the efficiency of defect image generation, reducing the time cost of sample generation, and providing data support for the rapid iteration of defect detection models.

[0054] In some embodiments, the glass industry dataset is constructed based on glass image data of different sizes and processes in multiple application scenarios, including mobile phones, tablets, laptops, automotive, and televisions. Each data entry in the glass industry dataset includes a glass image, defect annotations, and defect text description information, including at least the defect location, defect name, and defect severity in the image.

[0055] Understandably, this dataset addresses the pain point of existing technologies where mobile phone cover glass data cannot be reused for watches and laptops. It allows different projects to share industry-level data, avoiding the need for repeated collection of defect samples in new projects, significantly reducing time and material costs, and shortening project implementation cycles. Each dataset contains a glass image, defect annotations, and text descriptions including location, name, and severity. This enables the training of large-scale, industry-specific models with fine-grained recognition capabilities, rather than conventional models that can only identify scratches. This provides core knowledge support for subsequent accurate defect generation and detection. Text descriptions containing defect location, name, and severity replace traditional, vague defect labels, clearly distinguishing defect morphologies in different glass areas (ink area / viewing window area) and of varying severity. This avoids the problem of traditional generation models requiring multiple attempts to generate the target defect, making description-based, accurate defect generation possible.

[0056] In some embodiments, multimodal fusion processing is performed on the acquired mask images of multiple defects, multiple defect description prompts, and the defect center position corresponding to each prompt to obtain multimodal fusion features, including:

[0057] The acquired mask images of multiple defects are encoded to obtain image encoding features;

[0058] The obtained prompts for multiple defect descriptions are processed by text encoding to obtain text encoding features;

[0059] The defect center location corresponding to each prompt word is encoded to obtain the location encoding feature;

[0060] Based on the first cross-attention mechanism, the text encoding features and position encoding features are fused to obtain intermediate features;

[0061] Based on the second cross-attention mechanism, intermediate features and image coding features are fused to obtain multimodal fusion features.

[0062] In specific implementation, an image encoder, a text encoder, and a position encoder are used to encode the mask image, prompt words, and defect center location, respectively, to obtain image-encoded features, text-encoded features, and position-encoded features. A first cross-attention layer is constructed, with text-encoded features as the Query (guiding semantic direction) and position-encoded features as the Key and Value (providing spatial location information). The number of attention heads is set to 8, and the feature dimension is maintained at 512. The text-encoded features and position-encoded features are input into the first cross-attention layer. Within the layer, the similarity matrix between the Query and the Key is first calculated (through dot product operation), and then normalized using the Softmax function to obtain the attention weights. The higher the weight value, the stronger the correlation between text semantics and positional information. Finally, by weighted summation of the weights and Values, the region, degree, and type semantics of the text are bound to the spatial anchor information of the location, and intermediate features are output. A second cross-attention layer is constructed, with intermediate features as the Query (guiding the direction of semantic and positional association) and image-encoded features as the Key and Value (providing the basis for defect morphology). The number of attention heads and feature dimensions are the same as in the first cross-attention layer. First, the dimensions of the image-encoded features are adjusted. Then, the intermediate features and the adjusted image-encoded features are input into the second cross-attention layer to calculate the similarity between the Query and the Key (image patch features). The association between the intermediate features and the corresponding pure white defect region morphology in the mask is strengthened through weight allocation. Finally, after weighted summation, the semantic, positional, and morphological information is integrated to output multimodal fusion features. Each feature vector corresponds to the complete generation requirements of a defect.

[0063] By employing a two-stage cross-attention mechanism, text semantics and location coordinates are first bound together, followed by the binding of semantics, location, and mask shape. This avoids information confusion and lays the foundation for accurate defect generation in the future.

[0064] In some embodiments, an image generation network generates a multi-defect feature map based on multimodal fusion features and quantized feature parameters of each defect, including:

[0065] Based on the severity of defects or the importance of the region, multiple defects are prioritized for generation, and the ranking results are obtained.

[0066] The defects in the sorting results are traversed. For each defect, the defect generation range is determined based on the defect center position and defect shape included in the multimodal fusion features. The defect generation range is then rendered based on the quantized feature parameters of the defect to obtain a multi-defect feature map. The quantized feature parameters include at least the geometric and optical parameters of the defect.

[0067] Specifically, sorting by defect severity refers to referencing the glass industry defect grading standards, assigning values ​​of 3, 2, and 1 to severe, normal, and minor defects respectively (higher values ​​indicate higher priority). For example, severe bubbles (3 points) in the viewing window area have higher priority than minor scratches (1 point) in the ink area, and normal scratches (2 points) in the viewing window area have higher priority than normal inclusions (2 points) in the ink area; if the scores are the same, the viewing window area is given priority by default.

[0068] Prioritize by area importance: Based on the functional priority of the glass cover, assign a value of 2 to the viewing area (which directly affects the display effect) and a value of 1 to the ink area and auxiliary structure area. For example, minor scratches in the viewing area (2 points) have a higher priority than severe inclusions in the ink area (1 point).

[0069] The numerical descending order method is used to sort the assigned values ​​of all defects and generate a priority list. For example, the defect group is: severe air bubbles in the window area (3 points), severe inclusions in the ink area (1 point), and normal scratches in the window area (2 points). The ranking result is: severe air bubbles in the window area > normal scratches in the window area > severe inclusions in the ink area.

[0070] The results are traversed and sorted in descending order of priority. High-priority defects are processed first to avoid low-priority defects from overwriting the features of high-priority defects (e.g., window area defects are generated first to prevent ink area defects from occupying window area pixels). Extract the defect center position (e.g., (25,55)) and defect shape (e.g., linear scratch, circular bubble) from the multimodal fusion features; determine the range by combining the geometric attributes of the defect shape. Specifically, if it is a linear scratch (shape parameters "length 5 pixels, width 1.5 pixels"), take the defect center position as the midpoint and extend along the scratch extension direction (analyzed from the fusion features, such as the horizontal direction) to determine the generation range as "horizontal coordinate 22-28, vertical coordinate 54-56"; if it is a circular bubble (shape parameter "diameter 5 pixels"), draw a circle with a radius of 2.5 pixels and the center position as the center to determine the generation range as "horizontal coordinate 77.5-82.5, vertical coordinate 117.5-122.5" (pixel-level precision); ensure that the generation range falls completely within the pure white area of ​​the defect in the mask, and avoid exceeding the preset shape boundary.

[0071] The geometric parameters are the shape, size, and edge state of the defect, such as "linear scratch (shape), length 5 pixels (size), edge blur radius 1.2 pixels (edge ​​state)"; the optical parameters are the optical properties extracted to match the glass area and process, such as "contrast 35%, color RGB(200,200,200)" for scratches in the ink area, and "transmittance 85%, color RGB(230,230,230)" for bubbles in the viewing window area (based on industry knowledge of large model reuse to ensure compatibility with the background color of the area). Within the defined generation range, the basic shape is drawn according to geometric parameters. For example, linear scratches are connected at both ends of the range with a 1.5-pixel wide line, and the edges are overlaid with a 1.2-pixel Gaussian blur. Circular bubbles are filled with circles with a diameter of 5 pixels, and the edges are not blurred. Pixel attributes are adjusted based on optical parameters. The pixel contrast of the scratch area in the ink area is calculated at 35%, and the pixel transmittance of the bubble area in the viewport area is reduced by 15% (simulating the light-blocking effect of real bubbles). After the current defect is rendered, the process continues to traverse the next defect, repeating the determination of the range and the rendering process until all defects are rendered, forming a multi-defect feature map containing the independent features of all defects.

[0072] In some embodiments, the defect image generation method further includes:

[0073] The target defect sample image is input into a large model for defect identification, and the defect identification result is obtained.

[0074] The accuracy of the target defect sample image is verified based on the defect identification results to obtain the sample verification results.

[0075] Specifically, the target defect samples are preprocessed, and the preprocessed target defect sample images are input into a large glass defect recognition model trained on an industry-specific dataset. This triggers the model's fine-grained recognition process, and finally outputs structured defect recognition results. The defect recognition results must include the location information, type and degree, and quantitative feature parameters of each defect, and the recognition results of each defect are independently labeled.

[0076] From the input data of the target defect sample generation stage, benchmark data for verification is extracted to ensure consistency with the comparison dimensions of the large model recognition results. Specifically, the defect center coordinates and the pure white area range of the mask image used when generating the target defect sample image are extracted as benchmarks for judging the accuracy of the defect location. The prompt words of the defect description during generation are extracted, and each defect's region (ink area / window area), defect type, and severity are decomposed as benchmarks for judging the consistency of type and severity. The pixel deviation between the defect center coordinates in the recognition result and the benchmark coordinates is calculated. If the deviation is less than a threshold, the coordinates are qualified. It is determined whether the defect area range in the recognition result falls completely within the pure white area of ​​the mask image. If so, the area range is qualified, avoiding defects exceeding the preset generation range. The combination of region and type in the recognition result is completely matched with the combination of region and type in the prompt benchmark. If they match, the type and region are qualified. First, determine whether the degree of defect in the identification result (e.g., "minor") is consistent with the degree description in the prompt benchmark; then verify whether the quantitative feature parameters in the identification result fall within the benchmark parameter range. If the degree description is consistent and the parameters are within the range, then the degree is qualified.

[0077] In some embodiments, the defect image generation method further includes:

[0078] By using target defect sample images that match the target scene, the defect detection model to be trained in the target scene is trained to obtain a defect detection model for recognizing various defects in the target scene.

[0079] Because pre-trained large models rely on high-performance GPUs or multi-GPU clusters for operation, inference time for a single image is typically >1 second, requiring gigabytes of video memory. However, the hardware deployed for defect detection in target scenes is mostly edge devices (such as automotive embedded chips and production line inspection terminals). These devices generally suffer from limited storage (e.g., only 8-32GB), weak computing power (no dedicated GPU or only integrated lightweight GPU), and power consumption limitations. Directly using large models can lead to hardware storage overflow, inference lag, or even failure to start, failing to meet real-time detection requirements. Therefore, a separate defect detection model adapted to the target scene needs to be trained, requiring only CPU or lightweight GPU for inference, fully adapting to edge hardware resources. Specifically, following a pre-defined training method, the defect detection model for the target scene is trained based on target defect sample images matched to the target scene.

[0080] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0081] Based on the same inventive concept, this application also provides a defect image generation apparatus. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of one or more defect image generation apparatus embodiments provided below can be found in the limitations of the defect image generation method above, and will not be repeated here.

[0082] like Figure 3 As shown, this application embodiment provides a defect image generation apparatus 300, including:

[0083] Network building module 301 is used to obtain a large model for identifying glass defects trained using a glass industry dataset, and to build an image generation network based on the backbone network of the large model.

[0084] The feature fusion module 302 is used to perform multimodal fusion processing on the mask images of multiple defects, the prompt words describing multiple defects, and the defect center position corresponding to each prompt word to obtain multimodal fusion features;

[0085] The determination module 303 is used to determine the quantization feature parameters of each defect based on the multimodal fusion features through the image generation network;

[0086] The generation module 304 is used to generate a multi-defect feature map by means of an image generation network, based on the multimodal fusion features and the quantized feature parameters of each defect.

[0087] The image fusion module 305 is used to fuse the original image of the defect-free glass in the target scene with the multi-defect feature map to obtain a target defect sample image that matches the target scene.

[0088] In some embodiments, the glass industry dataset is constructed based on glass image data of different sizes and processes in multiple application scenarios, including mobile phones, tablets, laptops, automotive, and televisions. Each data entry in the glass industry dataset includes a glass image, defect annotations, and defect text description information, including at least the defect location, defect name, and defect severity in the image.

[0089] In some embodiments, in performing multimodal fusion processing on the acquired mask images of multiple defects, the prompts describing multiple defects, and the defect center position corresponding to each prompt to obtain multimodal fusion features, the feature fusion module 302 is specifically used for:

[0090] The acquired mask images of multiple defects are encoded to obtain image encoding features;

[0091] The obtained prompts for multiple defect descriptions are processed by text encoding to obtain text encoding features;

[0092] The defect center location corresponding to each prompt word is encoded to obtain the location encoding feature;

[0093] Based on the first cross-attention mechanism, the text encoding features and position encoding features are fused to obtain intermediate features;

[0094] Based on the second cross-attention mechanism, intermediate features and image coding features are fused to obtain multimodal fusion features.

[0095] In some embodiments, in generating a multi-defect feature map using an image generation network based on multimodal fusion features and quantized feature parameters of each defect, the generation module 304 is specifically configured to:

[0096] Based on the severity of defects or the importance of the region, multiple defects are prioritized for generation, and the ranking results are obtained.

[0097] The defects in the sorting results are traversed. For each defect, the defect generation range is determined based on the defect center position and defect shape included in the multimodal fusion features. The defect generation range is then rendered based on the quantized feature parameters of the defect to obtain a multi-defect feature map. The quantized feature parameters include at least the geometric and optical parameters of the defect.

[0098] In some embodiments, the defect image generation apparatus further includes:

[0099] The verification module is used to input the target defect sample image into the large model for defect identification and obtain the defect identification result; based on the defect identification result, the accuracy of the target defect sample image is verified and the sample verification result is obtained.

[0100] In some embodiments, the defect image generation apparatus further includes:

[0101] The training module is used to train the defect detection model to be trained in the target scene using target defect sample images that match the target scene, so as to obtain a defect detection model for recognizing various defects in the target scene.

[0102] Each module in the aforementioned defect image generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0103] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to implementing the defect image generation method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the aforementioned defect image generation method.

[0104] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it performs the steps in the aforementioned defect image generation method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen; the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the casing of the computer device, or external keyboards, touchpads or mice, etc.

[0105] Those skilled in the art will understand that Figure 4 or Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0106] In some embodiments, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0107] In some embodiments, such as Figure 6 The diagram shows the internal structure of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described method embodiments.

[0108] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0109] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0111] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating defect images, characterized in that, include: Obtain a large model for identifying glass defects trained using a glass industry dataset, and construct an image generation network based on the backbone network of the large model. Multimodal fusion processing is performed on the mask images of multiple defects, the prompt words describing multiple defects, and the defect center position corresponding to each prompt word to obtain multimodal fusion features; The image generation network determines the quantization feature parameters of each defect based on the multimodal fusion features. The image generation network generates a multi-defect feature map based on the multimodal fusion features and the quantized feature parameters of each defect. The original image of the defect-free glass in the target scene is fused with the multi-defect feature map to obtain a target defect sample image that matches the target scene.

2. The method according to claim 1, characterized in that, The glass industry dataset is constructed based on glass image data of different sizes and processes in multiple application scenarios, including mobile phones, tablets, laptops, automobiles, and televisions. Each data entry in the glass industry dataset includes a glass image, defect annotations, and defect text description information, including at least the defect location, defect name, and defect severity in the image.

3. The method according to claim 1, characterized in that, The process of performing multimodal fusion processing on the masked images of multiple defects, the prompts describing multiple defects, and the defect center position corresponding to each prompt to obtain multimodal fusion features includes: The acquired mask images of multiple defects are encoded to obtain image encoding features; The obtained prompts for multiple defect descriptions are processed by text encoding to obtain text encoding features; The defect center location corresponding to each prompt word is encoded to obtain the location encoding feature; Based on the first cross-attention mechanism, the text encoding features and the position encoding features are fused to obtain intermediate features; Based on the second cross-attention mechanism, the intermediate features and the image coding features are fused to obtain multimodal fusion features.

4. The method according to claim 1, characterized in that, The step of generating a multi-defect feature map through the image generation network, based on the multimodal fusion features and the quantized feature parameters of each defect, includes: Based on the severity of the defects or the importance of the region, the multiple defects are prioritized for generation, and a ranking result is obtained. The defects in the sorting results are traversed. For each defect, the defect generation range is determined based on the defect center position and defect shape included in the multimodal fusion features. The defect generation range is then rendered based on the quantized feature parameters of the defect to obtain a multi-defect feature map. The quantized feature parameters include at least the geometric and optical parameters of the defect.

5. The method according to claim 1, characterized in that, The method further includes: The target defect sample image is input into the large model for defect identification to obtain the defect identification result; The accuracy of the target defect sample image is verified based on the defect identification results to obtain the sample verification results.

6. The method according to claim 1, characterized in that, The method further includes: Using the target defect sample image that matches the target scene, the defect detection model to be trained in the target scene is trained to obtain a defect detection model for identifying various types of defects in the target scene.

7. A defect image generation apparatus, characterized in that, include: The network construction module is used to acquire a large model for identifying glass defects trained using a glass industry dataset, and to construct an image generation network based on the backbone network of the large model. The feature fusion module is used to perform multimodal fusion processing on the mask images of multiple defects, the prompt words describing multiple defects, and the defect center position corresponding to each prompt word to obtain multimodal fusion features; The determination module is used to determine the quantization feature parameters of each defect based on the multimodal fusion features through the image generation network; The generation module is used to generate a multi-defect feature map through the image generation network, based on the multimodal fusion features and the quantized feature parameters of each defect; The image fusion module is used to fuse the original image of the defect-free glass in the target scene with the multi-defect feature map to obtain a target defect sample image that matches the target scene.

8. The apparatus according to claim 7, characterized in that, The glass industry dataset is constructed based on glass image data of different sizes and processes in multiple application scenarios, including mobile phones, tablets, laptops, automobiles, and televisions. Each data entry in the glass industry dataset includes a glass image, defect annotations, and defect text description information, including at least the defect location, defect name, and defect severity in the image.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images