A Multi-Category Industrial Quality Inspection Method Based on Generative Transfer Learning
By expanding the defect dataset through generative transfer learning and a physical simulation engine, the problem of rapidly adapting to new products and process changes in industrial production lines has been solved, enabling efficient and automated multi-category quality inspection and improving the adaptability and accuracy of the inspection system.
Patent Information
- Application Number
- CN202511387594.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-26
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-09-26
AI Technical Summary
In industrial production lines, product update cycles are rapid and production line switching is frequent. Traditional quality inspection methods cannot quickly adapt to changes in new products or processes, and the generalization and flexibility of defect detection are insufficient, resulting in long-term data collection and annotation training requirements, which affect production efficiency and costs.
By employing a generative transfer learning approach, the defect dataset is expanded using generative tools and a physical simulation engine. Reinforcement learning is then combined to optimize the generator parameters, thereby establishing a multi-category industrial quality inspection platform to achieve automated quality inspection and rapid model training.
It improves the adaptability and efficiency of the quality inspection system, reduces data collection and labeling time, lowers costs, ensures the stability and accuracy of the quality inspection system in rapidly changing environments, and realizes automated quality inspection.
Smart Images

Figure CN121190447B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial production quality inspection technology, and more specifically, to a multi-category industrial quality inspection method based on generative transfer learning. Background Technology
[0002] In industrial production lines, a single line may produce many similar but different products, which may vary in shape, color, and other characteristics. These differences mean that pre-set defect datasets may not meet the quality inspection requirements of new products, resulting in low reusability of the same dataset solution. Furthermore, the limited number of defective samples and the difficulty in detecting extreme defective samples are also significant issues. Deep learning-based solutions have long deployment cycles and require a large number of samples; machine vision-based solutions have poor generalization capabilities, cannot adapt to diverse recognition requirements, and require users to be very familiar with business details, hindering flexible project deployment.
[0003] Furthermore, manufacturing enterprises currently face challenges such as rapid product update cycles and frequent production line switching; the need to quickly establish and verify the reliability of quality inspection systems before launching new products or processes, making it unacceptable to spend excessive time collecting, labeling, and training data; changes in production processes or quality control indicators leading to new defect scenarios requiring rapid application to the quality inspection system; and the need for highly flexible and rapidly iterating quality inspection systems. All of these factors place higher demands on traditional industrial quality inspection models.
[0004] Therefore, in order to address the problems currently faced by manufacturing enterprises, such as rapid product update cycles, frequent production line switching, the need to quickly establish and verify the reliability of quality inspection systems for new products or processes, the inability to accept long periods of data collection, annotation, and training, and the emergence of new defect scenarios due to changes in production processes or quality control indicators, this invention, targeting multi-category industrial quality inspection production lines, adopts a generative transfer learning approach to enhance the adaptability of data samples to different products and increase the number of defect samples. This addresses the needs of existing quality inspection methods in terms of generalization capabilities, timeliness of adaptation to new scenarios, flexibility, and intelligence. Summary of the Invention
[0005] The present invention aims to overcome at least one of the defects (deficiencies) of the prior art and provide a multi-category industrial quality inspection method based on generative transfer learning, which enhances the adaptability of data samples to different products and increases the number of defect samples, so as to solve the requirements of the original quality inspection method in terms of generalization capability, timeliness of adaptation to new scenarios, flexibility and intelligence.
[0006] The technical solution adopted by this invention is a multi-category industrial quality inspection method based on generative transfer learning, the method comprising: A testing model is deployed in a multi-category industrial quality inspection service platform. Then, a camera is used to take pictures of the products to be tested on the production line workbench to obtain sample data of the products to be tested. The sample data of the products to be tested is then transmitted to the industrial control computer. The testing model issued from the multi-category industrial quality inspection service platform is used to perform product quality inspection. Then, the production line workbench automatically packages qualified products and rejects unqualified products based on the test results. By deploying testing models in the industrial quality inspection service platform and combining them with camera imaging technology, real-time monitoring and automatic quality inspection of products to be inspected on the production line can be achieved. This reduces manual intervention, improves production efficiency and stability, optimizes product quality control, effectively reduces production costs, and promotes the intelligent and automated upgrading of industrial production.
[0007] Preferably, the method further includes training the detection model in the multi-category industrial quality inspection service platform, the training steps of which include: S1: Establish a basic defect database; S2: Build a synthetic expanded defect library based on the basic defect library; S3: Generative tools using an additional integrated physical simulation engine are used to port defect targets to new product scenarios to generate defect features that are more in line with the actual scenarios, thereby establishing a porting and expansion library. S4: Combine the basic defect library, the synthetic extended defect library, and the ported extended library to establish a quality inspection dataset. Then, input the quality inspection dataset into the detection model for training to obtain a trained detection model.
[0008] This application proposes a multi-category industrial quality inspection method based on generative transfer learning. Through generative tools and transfer learning, it addresses the challenges faced by quality inspection systems in multi-category production, significantly improving the adaptability, reliability, and efficiency of the system. This method generates synthetic defect data and combines it with a physical simulation engine, enabling rapid response to the launch of new products or processes, reducing data collection and annotation time and lowering costs. The generated expanded defect library enhances the model's generalization ability, allowing it to adapt to the quality inspection needs of different product categories. Simultaneously, this method incorporates multiple defect sample generation methods for dataset preprocessing, improving the scenario adaptability and sample weighting of defect samples, enhancing the model's generalization ability, and enabling flexible responses to changes in production processes and the evolution of defect scenarios, ensuring stable operation of the quality inspection system in rapidly changing production environments. Finally, the detection model trained based on this method can achieve automated quality inspection, improve production line efficiency, reduce labor costs and error rates, and effectively shorten the time to launch new products or processes.
[0009] Preferably, in step S1, the basic defect library includes a complete defect library and a trimmed defect library, wherein: For a complete defect database: Defect image-label pairs are obtained by using existing labeled product defect datasets, manually labeled new product defect datasets, and zero-shot inference to generate labels using multimodal large models. These defect image-label pairs are then used to build a basic defect database. For the cropping defect library: based on the location information stored in the label, the pixel blocks in the corresponding image are cropped out. If the label stores the four-point coordinate information of the corresponding quadrilateral, the image within the quadrilateral formed by the four-point coordinates of the corresponding image is cropped out. If the label stores the multi-point coordinate information of the corresponding polygonal box, the image within the polygonal box formed by the multi-point coordinates of the corresponding image is cropped out. Then, these cropped defect targets are classified and stored according to the label category to build a cropping defect library.
[0010] By establishing a complete defect database and a trimmed defect database, the accuracy and generalization ability of the defect detection system are effectively improved. The complete defect database expands the dataset through multiple data sources (existing labeled datasets, manual annotations, and newly generated data), ensuring its diversity and comprehensiveness, thereby enhancing the model's ability to identify different defect types. The trimmed defect database, on the other hand, precisely trims and categorizes defect regions for storage, allowing the model to focus on training within those regions, reducing interference from redundant information and improving detection efficiency and accuracy. This not only improves data utilization efficiency but also ensures the adaptability and stability of the detection model when facing different products and defect scenarios, demonstrating significant technical advantages.
[0011] Preferably, step S2 includes: S21: Based on the color characteristics of the new product, use a generative adversarial network to change the characteristics of the defect targets in the original complete defect library, generate a new complete defect library, and obtain synthetic expansion library 1; S22: The defect library is cropped by directly changing the color or using a generative adversarial network to change the features of the defect targets in the original defect library. At the same time, the shape of the defect targets is changed by inverting, mirroring, and increasing or decreasing the area and shape of the targets. Then, the defect image-label pairs and the corresponding cropped images are matched with the same name. The newly generated cropped defect library images are overwritten with the corresponding full defect library images according to the label position information in the full defect library, thus obtaining the synthetic expansion library 2.
[0012] By expanding and diversifying the defect database using Generative Adversarial Networks (GANs) and image enhancement techniques, the accuracy and generalization ability of the defect detection system are significantly improved. First, by altering the color features of defect targets in the complete defect database using GANs, a new synthetic expansion database 1 is generated, increasing the diversity of defect data and improving the model's robustness. Second, by changing the color, shape, and size of defect targets in the cropped defect database and combining this with image enhancement techniques, a synthetic expansion database 2 is generated, further enhancing the model's adaptability to different defect morphologies and background variations. This method reduces reliance on large amounts of real-world data, lowers annotation costs, and improves the adaptability and stability of the detection model in actual production.
[0013] Preferably, step S2 further includes setting a generation parameter evolution mechanism to establish a reinforcement learning-based transfer parameter control system to optimize the generative adversarial network; wherein, the transfer parameter control system includes: using a state-space encoder to encode new product features into an N-dimensional vector, and then using the PPO algorithm to dynamically adjust the learning rate, gradient penalty coefficient, and texture fusion intensity of the generator used.
[0014] By introducing a reinforcement learning-based transfer parameter control system, the generation process of a generative adversarial network (GAN) is optimized, thereby improving the accuracy and efficiency of defect database expansion. Specifically, the system utilizes a state-space encoder to encode new product features into N-dimensional vectors and employs the PPO algorithm to dynamically adjust key parameters such as the generator's learning rate, gradient penalty coefficient, and texture fusion intensity to ensure the optimization and stability of the generation process. Through this method, the GAN can adaptively adjust parameters based on real-time feedback, avoiding overfitting and training instability, thus improving the quality and diversity of generated images and enhancing the adaptability and robustness of the quality inspection system in complex production environments.
[0015] Preferably, step S2 further includes setting an automated quality scoring mechanism to calculate the final quality score, and then scoring the generated samples through a pre-trained quality assessment network. Samples with scores greater than a set value are entered into a manual review queue.
[0016] The automated quality scoring mechanism used in this application can effectively improve the quality control level of the generated samples, reduce the burden of manual review, and ensure that the final generated data meets the expected standards, thereby improving the efficiency and accuracy of the entire defect library expansion process.
[0017] Preferably, the automated quality scoring mechanism includes: inputting a generated sample image, then calling a physical simulation module to obtain physical property parameters, then extracting image feature vectors, encoding the process requirement text, performing multimodal feature fusion, and finally calculating the final quality score; wherein, the final quality score formula is:
[0018] in, Indicates the defect type weight; Indicates the severity of the defect; Indicates the number of defect types.
[0019] The above steps and formulas allow for a quick and accurate preliminary evaluation of the generated samples, reducing the workload and time required for manual review, thereby improving quality control efficiency. Furthermore, they ensure that only high-quality samples are selected for manual review, thus enhancing overall data quality.
[0020] Preferably, step S3 includes: S31: Construct a material property database based on the characteristics of different industrial materials; S32: Extract semantic features of surface materials of new products through CLIP model, and automatically match the illumination reflection parameters and texture fusion coefficients during defect transplantation to construct a material property matching module; S33: Through differentiable rendering technology, a geometric deformation field is generated based on the 3D CAD model of the target product, thereby establishing a deformation field generation network. This network is used to calculate and predict based on material properties and in conjunction with a physics engine, so that the transplanted defects adapt to the product surface and generate realistic defect samples of different shapes. S34: Perform environmental consistency verification. Use a physically based rendering engine to render the modified 3D model from step S33 with a multi-light source environment, generating a composite image that includes specular reflection, diffuse reflection, and shadows. S35: The pre-trained Noise2Noise network is used to detect artifacts in the transplanted region. When unnatural boundaries are detected, a second optimization based on Poisson fusion is automatically triggered to establish a dynamic fusion evaluation system. S36: Simultaneously, in multiple camera view scenarios, the defect features appearing in one view are directly transplanted to other different viewpoints to expand the defect samples, thereby establishing a transplanted expansion library.
[0021] The aforementioned series of optimization steps significantly improved the realism and consistency of the generated defect samples. Firstly, by constructing a material property database, extracting semantic features using the CLIP model, generating geometric deformation fields using differentiable rendering technology, and employing adaptive defect transplantation via a physics engine, the generated defect samples were ensured to highly match the surface of the target product. Furthermore, environmental consistency verification was performed using a physical rendering engine, ensuring that the generated synthetic images maintained consistency under different lighting conditions. Combined with a pre-trained Noise2Noise network to automatically detect artifacts and trigger Poisson fusion-based optimization, the naturalness and accuracy of the samples were further improved. A dynamic fusion evaluation system was established, and defect features appearing from one perspective were transplanted to other different perspectives to expand the defect samples, thereby improving the quality of the defect samples, reducing manual intervention, and ensuring efficiency and accuracy in complex application scenarios.
[0022] Preferably, step S33 includes: S331: Convert CAD models into symbolic distance functions or explicit triangular meshes; S332: Model the deformation field by extracting the surface geometric features, then using a graph neural network to process the grid vertex features, then using an MLP network to output displacement vectors to establish the displacement field, and using latent encoding to control the defect type and intensity; S333: Gradient backpropagation is achieved through the integration of a 3D computing framework, supporting end-to-end optimization from vertex position to pixel color for differentiable rendering; S334: Perform physical constraint modeling, establish a material parameter library, and calculate stress distribution using the finite element method or a neurophysics engine; S335: Based on the morphological characteristics of the transplanted defects, the displacement field isosurface is extracted to generate different defects.
[0023] This application utilizes highly integrated technical means to provide a more realistic, accurate, and efficient defect transfer and deformation process, ensuring that the generated defect samples conform to physical and visual consistency, and greatly improving the reliability and accuracy in industrial applications.
[0024] Preferably, step S35 includes: S351: Input the image patch generated in S34 into the pre-trained Noise2Noise network, which extracts features through an encoder-decoder structure to generate a reconstructed image patch after artifact removal; S352: Calculate the pixel-level residual between the original image patch and the reconstructed image patch output by the network, and generate a residual map; S353: Based on the residual map obtained in S352, artifact regions are determined, thereby establishing a dynamic fusion evaluation system.
[0025] By introducing a Noise2Noise network and residual analysis, the accuracy and efficiency of image artifact removal are significantly improved. First, the image patch generated by S34 is input into a pre-trained Noise2Noise network. This network effectively extracts image features through an encoder-decoder structure, generating reconstructed image patches after artifact removal, thus reducing the impact of artifacts. Next, a residual map is generated by calculating the pixel-level residual between the original and reconstructed image patches, used to analyze the accuracy and differences in image reconstruction. Based on the residual map, the system can accurately identify artifact regions and establish a fusion evaluation system based on their dynamic changes, thereby achieving more intelligent and automated artifact removal and image quality improvement. This process effectively improves the visual effect and quality of image reconstruction, making artifact removal more accurate and demonstrating high practicality and robustness.
[0026] Preferably, in step S353, the artifact region determination includes: S3531: Perform threshold segmentation and mark out-of-limit pixels in the residual map as suspected artifact points based on the set empirical threshold. S3532: Perform morphological processing, dilate discrete artifact points, and connect adjacent regions to form an artifact mask. S3533: Perform boundary feature detection, count the artifact mask density near the boundary of the transplanted area, and if the artifact mask density exceeds a preset threshold, it is determined that there is an unnatural boundary, thereby realizing the artifact region determination.
[0027] The above steps effectively improve the accuracy of artifact region identification, ensure the naturalness and realism of the image, and enhance the preservation of details and quality control during the artifact removal process.
[0028] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention addresses the challenges of multi-product industrial quality inspection production lines by employing generative transfer learning to enhance the adaptability of data samples to different products and increase the number of defect samples. It effectively solves problems currently faced by manufacturing enterprises, such as rapid product update cycles, frequent production line transitions, the need to quickly establish and verify the reliability of the quality inspection system before launching new products or processes, the inability to accept long periods of data collection, annotation, and training, and the emergence of new defect scenarios due to changes in production processes or quality control indicators. This invention achieves automated quality inspection, improves production line efficiency, reduces labor costs and error rates, and effectively shortens the time to launch new products or processes. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of the multi-category industrial quality inspection method of the present invention.
[0030] Figure 2This is a schematic diagram of the detection model training method of the present invention. Detailed Implementation
[0031] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention. To better illustrate the following embodiments, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions; it is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0032] Example 1 like Figure 1 As shown, this embodiment provides a multi-category industrial quality inspection method based on generative transfer learning, the method including: A testing model is deployed in a multi-category industrial quality inspection service platform. Then, a camera is used to take pictures of the products to be tested on the production line workbench to obtain sample data of the products to be tested. The sample data of the products to be tested is then transmitted to the industrial control computer. The testing model issued from the multi-category industrial quality inspection service platform is used to perform product quality inspection. Then, the production line workbench automatically packages qualified products and rejects unqualified products based on the test results. Preferably, such as Figure 2 As shown, the method further includes training the detection model in the multi-category industrial quality inspection service platform, and the training steps include: Step S1: Establish a basic defect database; In step S1, the basic defect library includes a complete defect library and a trimmed defect library, wherein: For a complete defect database: Defect image-label pairs are obtained by using existing labeled product defect datasets, manually labeled new product defect datasets, and zero-shot inference to generate labels using multimodal large models. These defect image-label pairs are then used to build a basic defect database. For the cropping defect library: Pixel blocks in the corresponding image are cropped based on the location information stored in the tags. If the tag stores the four-point coordinate information of the corresponding quadrilateral, the image within the quadrilateral formed by the four-point coordinates of the corresponding image is cropped. If the tag stores the multi-point coordinate information of the corresponding polygonal bounding box, the image within the polygonal bounding box formed by the multi-point coordinates of the corresponding image is cropped. These cropped defect targets are then categorized and stored according to the tag category to establish the cropping defect library. In this embodiment, assuming there are category 1 and category 2 tags, images cropped from tag category 1 are stored together, and images cropped from tag category 2 are stored together. The defect image-tag pair and the corresponding cropped image are named with the same name. If a set of defect image-tag pairs contains multiple cropped images, the last digit of the cropped image name is added (e.g., 1 or 2) to distinguish them.
[0033] Step S2: Build a synthetic expanded defect library based on the basic defect library; Step S2 includes: Step S21: In an industrial production line, a single production line may produce many similar but different products. These products may differ in shape, color, etc. These different shapes, colors, and other characteristics will cause the pre-set defect dataset to fail to meet the quality inspection requirements of the new products. Therefore, to improve the model's generalization ability in handling this situation, for the complete defect database, a generative adversarial network is used to modify the features of the defect targets in the original complete defect database based on the color features of the new products, generating a new complete defect database, resulting in the synthetic expansion database 1. Step S22: Since the defect targets may be small and the texture features may be complex, the defect target part may not achieve good results in the above-mentioned generation operation for the complete defect library. Therefore, the defect library is cropped by directly changing the color or using a generative adversarial network to change the features of the defect targets in the original defect library. At the same time, the shape of the defect targets is changed by inverting, mirroring, and increasing or decreasing the area and shape of the targets. Then, the defect image-label pair and the corresponding cropped image are matched with the same name. The newly generated cropped defect library image is overlaid into the position of the corresponding complete defect library image according to the label position information in the complete defect library, thereby obtaining the synthetic expansion library 2.
[0034] Preferably, step S2 further includes setting a generation parameter evolution mechanism to establish a reinforcement learning-based transfer parameter control system to optimize the generative adversarial network; wherein, the transfer parameter control system includes: using a state-space encoder to encode new product features into an N-dimensional vector, and then using the PPO algorithm to dynamically adjust the learning rate, gradient penalty coefficient, and texture fusion intensity of the generator used.
[0035] Specifically, in the task of dynamically adjusting parameters, the objective function formula of the PPO algorithm is:
[0036] in, Indicates the current policy network; Represents the dominance function; Indicates the clipping threshold; Indicates a given state; This represents the adjustment amount of three parameters: generator learning rate, gradient penalty coefficient, and texture fusion intensity.
[0037] Preferably, step S2 further includes setting an automated quality scoring mechanism to calculate the final quality score, and then scoring the generated samples through a pre-trained quality assessment network. Samples with scores greater than a set value are entered into a manual review queue.
[0038] Specifically, the automated quality scoring mechanism includes: Input a sample image to generate; in this embodiment, the sample image is a 512×512 PNG format. Then, the physical simulation module is called to obtain the physical property parameters, and the 32-dimensional physical property vector (surface roughness Ra=0.12μm, refractive index n=1.48, etc.) output by the physical simulation module is compressed to 64 dimensions through a fully connected layer; Next, the image feature vector is extracted. The EfficientNet-B4 network is used. The input is a 512×512 resolution RGB image, and the output is a 1024-dimensional feature vector. Then, the weights of the first 3 convolutional layers are frozen, and the last 2 Bottleneck layers are fine-tuned to achieve detailed adjustments. Secondly, the process requirement text is encoded. The CLIP ViT-B / 32 text encoder is integrated to map the process requirement (such as "no bubbles with a diameter > 1mm") into a 512-dimensional vector. Example input: "No visible scratches on the transparent bottle body, and no more than 3 bubbles with a diameter < 0.5mm are allowed". Finally, multimodal feature fusion is performed to calculate the final quality score; wherein, the formula for the final quality score is:
[0039] in, The defect type weight can be set to 0.15 for scratches, 0.3 for bubbles, and 0.25 for color difference in this embodiment. Indicates the severity of the defect; Indicates the number of defect types.
[0040] Step S3: In order to further overcome the problems of insufficient defect samples and imbalanced samples, the defect targets in the cropped defect library are transplanted into the new product scenario to generate defect features that are more in line with the actual scenario. By using generative tools with additional integrated physical simulation engines (such as the image customization tool Anydoor), defect targets are transplanted into the new product scenario to generate defect features that are more in line with the actual scenario, thereby establishing a transplanted expansion library. Preferably, step S3 specifically includes: Step S31: Construct a material property database based on the characteristics of different industrial materials, such as storing 128-dimensional property vectors of common industrial materials (metal / plastic / ceramic) such as surface tension coefficient and elastic modulus to construct the material property database; Step S32: Extract semantic features of the surface material of the new product through the CLIP model, and automatically match the illumination reflection parameters and texture fusion coefficients during defect transplantation to construct a material property matching module; Specifically, the clip model employs a dual-tower neural network structure, achieving cross-modal semantic alignment between images and text through contrastive learning. It mainly comprises an image encoder, a text encoder, and a shared embedding space. The specific steps are as follows: On the image side, 5-9 local cropping operations are performed on the material surface image, extracting detailed texture features from each, and then inputting these features into the image encoder; on the text side, a common defect description library is established, and these defect descriptions are input into the text encoder; the outputs of the image encoder and text encoder are sorted by feature similarity in the shared embedding space to achieve rapid matching of defect types. Finally, numerical parameters are converted into semantics, such as converting surface roughness Ra=0.8μm into a natural language description like "slightly frosted texture."
[0041] Step S33: Using differentiable rendering technology, a geometric deformation field is generated based on the 3D CAD model of the target product, thereby establishing a deformation field generation network. This network is used to calculate and predict based on material properties and in conjunction with a physics engine, so that the transplanted defects adapt to the product surface and generate realistic defect samples of different shapes. Specifically, step S33 includes: Step S331: Convert the CAD model into a symbolic distance function (SDF) or an explicit triangular mesh; Step S332: Model the deformation field by extracting surface geometric features such as curvature, normal vector, and thickness distribution. Then, use a graph neural network to process the grid vertex features. Next, use an MLP network to output displacement vectors to establish a displacement field and use latent coding to control the defect type and intensity. Step S333: Gradient backpropagation is achieved by integrating 3D computing frameworks such as PyTorch3D, and end-to-end optimization from vertex position to pixel color is supported for differentiable rendering. Step S334: Perform physical constraint modeling, establish a material parameter library (elastic modulus / Poisson's ratio, etc.), and calculate stress distribution using the finite element method or a neurophysics engine; Step S335: Extract the isosurface of the displacement field based on the morphological characteristics of the transplanted defects to generate different defects such as cracks / holes.
[0042] Step S34: Perform environmental consistency verification. Use a physically based rendering (PBR) engine to render the modified 3D model from step S33 using a multi-light source environment, generating a composite image that includes specular reflection, diffuse reflection, and shadows. S35: Artifact detection is performed on the transplanted region using a pre-trained Noise2Noise network. When unnatural boundaries are detected, a second optimization based on Poisson fusion is automatically triggered, thereby establishing a dynamic fusion evaluation system.
[0043] Preferably, step S35 includes: Step S351: Input the image patch generated in S34 into the pre-trained Noise2Noise network, which extracts features through the encoder-decoder structure to generate the reconstructed image patch after artifact removal; The Noise2Noise network is a deep learning-based denoising method whose core idea is to train a denoising model without clean data labels. Its network structure design revolves around a self-supervised learning paradigm of "from noise to noise". Noise2Noise typically employs an encoder-decoder architecture (such as a variant of U-Net).
[0044] Step S352: Calculate the pixel-level residual between the original image patch and the reconstructed image patch output by the network, and generate a residual map; Preferably, the expression for generating the residual map is:
[0045] in, This represents the input pixel value, which is the pixel value of the original image at coordinates (x, y). This represents the output pixel value, which is the pixel value at the same coordinate in the image after processing by an algorithm (such as a denoising network); Absolute difference: calculated by measuring the absolute difference between the pixel values of the two pixels. Eliminate the influence of positive and negative signs to ensure that the residual value is always non-negative; Residual plot: The final generated residual plot It is a matrix with the same size as the input image, where the value at each position represents the intensity of the difference between the corresponding pixel before and after the algorithm processing.
[0046] Step S353: Based on the residual map obtained in step S352, determine the artifact region to establish a dynamic fusion evaluation system.
[0047] Preferably, in step S353, the artifact region determination includes: Step S3521: Perform threshold segmentation, and mark out-of-limit pixels in the residual map as suspected artifact points according to the set empirical threshold (such as residual value > 0.1); Step S3522: Perform morphological processing, dilate discrete artifact points, and connect adjacent regions to form an artifact mask; Step S3523: Perform boundary feature detection, count the artifact mask density near the boundary of the transplanted area (e.g., ±5 pixel bandwidth), and if the artifact mask density exceeds a preset threshold (e.g., 30% of the area is covered), it is determined that there is an unnatural boundary, thereby realizing the artifact region determination.
[0048] Step S36: Simultaneously, in multiple camera view scenarios, directly transplant the defect features appearing in one view to other different view to expand the defect samples, thereby establishing a transplanted expansion library.
[0049] Preferably, step S36 includes: Step S361: Establish a multi-view projection matrix library and construct a perspective projection model for view transformation based on the calibration parameters of the production line camera. Step S362: Combining the perspective projection model, the neural radiation field (NeRF) technology is used to generate a three-dimensional representation of the defective target, achieving consistent rendering of defects from any viewpoint; wherein the neural radiation field (NeRF) technology is to generate a three-dimensional model of the target by taking photos of the target from different angles with the camera position calibrated and using image information from multiple angles. Step S363: Introduce an adversarial perspective verification mechanism to determine the morphological rationality of network detection of porting defects from different perspectives; Specifically, the adversarial perspective verification mechanism includes: First, multi-stream feature extraction is performed: ResNet-50 with shared weights is used to process three viewpoint images (front view, side view, and top view) in parallel. Then, cross-attention fusion is performed: geometric relationships between viewpoints are established through the Transformer module, and its expression formula is as follows:
[0050] in, This indicates a query, representing the current time step or the current sequence. The key represents the feature representation of each input vector; The value represents the weighted average of the values calculated based on the attention weights. This represents the transposed key matrix; This is the scaling factor.
[0051] Step S364: Calculate the geometric deformation consistency, material reflection consistency and edge continuity respectively, and then set the corresponding weights to combine the above calculation results to obtain the final score. Then, determine whether the shape is reasonable based on the score. If not, optimize by transplanting defect features. Specifically, the calculation of geometric deformation consistency involves calculating the Hausdorff distance between adjacent viewpoints, where the distance is <1.2 pixels, and its weight is set to 0.4; the calculation of material reflection consistency involves analyzing the HSV spatial distribution similarity ΔE of specular highlights, where the similarity is <5, and its weight is set to 0.3; the calculation of edge continuity involves detecting a Canny edge matching rate >85% across viewpoints, and its weight is set to 0.3. Therefore, the resulting scoring formula is:
[0052] in, Indicates the geometric deformation consistency score; Indicates the material's reflectivity score; This represents the edge continuity score.
[0053] When S>0.78, it is considered reasonable; otherwise, the following optimization is triggered: retrain the defect generator, optimize by adding a view consistency loss term and correcting the defect deformation parameters based on physical simulation.
[0054] Step S4: Combine the basic defect library, the synthetic extended defect library, and the ported extended library to establish a quality inspection dataset. Then, input the quality inspection dataset into the detection model for training to obtain a trained detection model.
[0055] Specifically, the quality inspection dataset used in practice is composed of the complete defect library in the basic defect library, synthetic expansion library 1 and synthetic expansion library 2 in the synthetic expansion defect library, and the ported expansion library. Then, the quality inspection dataset is trained using mainstream detection frameworks (such as the YOLO series), and the trained detection model is deployed on actual terminal devices for quality inspection inference.
[0056] The model training phase also includes a transfer validity verification step. The learning strategy adopted is to prioritize training models based on a basic defect library, and then gradually add synthetic and ported samples. Establish a dynamic weighted loss function: apply cosine annealing weights to the loss function of the transplanted samples to balance the learning of new and old features.
[0057] Example 2 Based on the method described in Example 1, this embodiment provides a method for inspecting droplets on the surface of a laundry detergent bottle, the method being as follows: In laundry detergent bottling production lines, leaks frequently occur from the filling gun, dripping onto the bottle surface. These drips can appear on any surface of the upper half of the bottle. Furthermore, a single bottling line fills bottles of different brands of laundry detergent, which vary in shape, color, and size, making droplet detection on the bottle surface extremely difficult. Especially when switching to a different type of detergent bottle, the model needs to be readjusted, and data may even need to be re-collected, re-labeled, and a new model trained to detect droplets on the new bottle surface. Specialized technicians are also required to make adjustments on-site, significantly impacting the production line's efficiency. Therefore, to address these issues, a method for detecting droplets on the surface of laundry detergent bottles is proposed.
[0058] First, the basic defect database: By collecting an image dataset of leaking laundry detergent bottles, semantic annotation was performed using a combination of CLIP-RC and manual annotation. Only two categories were labeled: "Laundry detergent bottle" and "Leakage droplets," named "Laundry detergentbottle" and "Leakage droplets," respectively. Simultaneously, leaking droplets were manually added to the laundry detergent bottles to expand the defect database. Since performing this directly on the filling line was inconvenient, the manual droplet collection was conducted in a convenient scenario (referred to as Scenario A). Then, based on the semantic location information of the leaking droplets stored in the labels, the corresponding leaking pixel blocks in the images were cropped. The cropped images were named after the original images. If the original image contained multiple leaking droplets, the cropped image name was prefixed with the numbers 1 and 2 for differentiation. The cropped images were stored in a dedicated folder.
[0059] Secondly, synthesize and expand the defect library: In the laundry detergent bottling production line, different laundry detergent bottles need to be filled with liquid. The different colors and shapes make it extremely difficult to detect transparent leakage droplets on the laundry detergent bottle.
[0060] The current filling production line uses blue laundry detergent bottles. The previous defect library consisted of green laundry detergent bottles. Using the blue laundry detergent bottles as a template, a generative adversarial network was used to transform the semantically labeled green laundry detergent bottle pixels in the original complete defect library to blue, generating a new synthetic expansion library 1.
[0061] Generative adversarial networks (GANs) are used to transform existing transparent green-background leaking droplet features in the cropping defect library into transparent blue-background leaking droplet features. Simultaneously, four new cropping defect libraries are generated by inverting, mirroring, and enlarging / reducing the target area and shape. These defect image-label pairs are then matched with their corresponding cropped images using the same name. The newly generated cropping defect library images are then overlaid onto the corresponding positions in the complete defect library images according to the label position information. Images generated in this way are called the synthetic augmentation library 2.
[0062] The synthetic expanded defect library also includes a generation parameter evolution mechanism: establishing a transfer parameter control system based on reinforcement learning, including: ① a state space encoder: encoding new product features into a 128-dimensional vector (color histogram + surface curvature map + material classification probability); ② using the PPO algorithm to dynamically adjust the generator learning rate, gradient penalty coefficient and texture fusion intensity.
[0063] An automated quality scoring mechanism was also set up, which scores the generated samples through a pre-trained quality assessment network, and only samples with a score > 0.82 are allowed to enter the manual review queue.
[0064] Furthermore, construct a porting expansion library: To facilitate the collection of data on manually added droplets and improve operational efficiency, the manual droplet collection process was not conducted in an actual production line scenario. Using data collected in scenario A would increase the difficulty of reasoning in the actual production line. To overcome this problem, a physical simulation engine was integrated and ported to the image customization tool Anydoor.
[0065] Specifically, the following steps are included: ① Material property database construction: The main object of quality inspection of liquid leakage from laundry detergent bottles is the liquid droplets. This part mainly establishes the gravity characteristics of laundry detergent when it drips from the plastic bottle. ② Construct a material property matching module: Extract semantic features of the surface material of the new product through the CLIP model, and automatically match the illumination reflection parameters and texture fusion coefficients when transplanting defects; ③ Establish an auxiliary mechanism based on a physical simulation engine: Perform 3D modeling of the product, using the scripting interface of Blender, AutoCAD, or SolidWorks, and pre-set a fixed model defect modification program (covering leaking droplets). Key parameters of the defect modification program (such as coverage location, length, and tortuosity) are to be determined. These key parameters are mainly obtained by identifying defect features in a synthetic defect library using a large model, and outputting information such as scratch location, length, and tortuosity according to a specific output text format. These parameters are then input into the defect modification program to modify the 3D model for defects.
[0066] ④ Environmental consistency verification: Use a physically based rendering (PBR) engine to perform multi-light source environment rendering on the transplanted defects, and generate a synthetic leaking droplet image that includes specular reflection, diffuse reflection and shadows; In this way, the leaking droplets collected in scenario A are transplanted to the actual production line operation scenario. The generated images become the transplanted and expanded library; This also includes expanding the defect samples by directly transferring the leaking droplet defect features from one viewpoint to other different viewpoints in multi-camera perspective scenarios. Specifically, this includes: Establish a multi-view projection matrix library: Based on the calibration parameters of the production line camera, construct a perspective projection model for view transformation; The neural radiation field technology is used to generate a three-dimensional representation of the defect target, so as to achieve consistent rendering of defects from any viewpoint. An adversarial perspective verification mechanism is introduced: the morphological rationality of the network detection transplantation defect is judged from different perspectives. Finally, based on the complete defect library, synthetic expansion library 1 and synthetic expansion library 2 in the synthetic expansion defect library, and the ported expansion library, a quality inspection dataset for actual use of laundry detergent bottle leakage droplets is formed. The dataset is then trained using mainstream detection frameworks (such as the YOLO series), and the trained detection model is deployed on actual terminal devices for quality inspection inference.
[0067] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.
Claims
1. A multi-category industrial quality inspection method based on generative transfer learning, the method comprising: A testing model is deployed in a multi-category industrial quality inspection service platform. Then, a camera is used to take pictures of the products to be tested on the production line workbench to obtain sample data of the products to be tested. The sample data of the products to be tested is then transmitted to the industrial control computer. The testing model issued from the multi-category industrial quality inspection service platform is used to perform product quality inspection. Then, the production line workbench automatically packages qualified products and rejects unqualified products based on the test results. The method is characterized in that it further includes training the detection model in the multi-category industrial quality inspection service platform, the training steps of which include: S1: Establish a basic defect database; S2: Build a synthetic expanded defect library based on the basic defect library; S3: Generative tools using an additional integrated physical simulation engine are used to port defect targets to new product scenarios to generate defect features that are more in line with the actual scenarios, thereby establishing a porting and expansion library. Step S3 includes: S31: Construct a material property database based on the characteristics of different industrial materials; S32: Extract semantic features of surface materials of new products through CLIP model, and automatically match the light reflection parameters and texture fusion coefficients during defect transplantation to construct a material property matching module; S33: Through differentiable rendering technology, a geometric deformation field is generated based on the 3D CAD model of the target product, thereby establishing a deformation field generation network. This network is used to calculate and predict based on material properties and in conjunction with a physics engine, so that the transplanted defects adapt to the product surface and generate realistic defect samples of different shapes. S34: Perform environmental consistency verification. Use a physically based rendering engine to render the modified 3D model from step S33 with a multi-light source environment, generating a composite image that includes specular reflection, diffuse reflection, and shadows. S35: The pre-trained Noise2Noise network is used to detect artifacts in the transplanted region. When unnatural boundaries are detected, a second optimization based on Poisson fusion is automatically triggered to establish a dynamic fusion evaluation system. S36: Simultaneously, in multiple camera view scenarios, the defect features appearing in one view are directly transferred to other different viewpoints to expand the defect samples, thereby establishing a transfer and expansion library. S4: Combine the basic defect library, the synthetic extended defect library, and the ported extended library to establish a quality inspection dataset. Then, input the quality inspection dataset into the detection model for training to obtain a trained detection model.
2. The multi-category industrial quality inspection method based on generative transfer learning according to claim 1, characterized in that, In step S1, the basic defect library includes a complete defect library and a trimmed defect library, wherein: For a complete defect database: Defect image-label pairs are obtained by using existing labeled product defect datasets, manually labeled new product defect datasets, and zero-shot inference to generate labels using multimodal large models. These defect image-label pairs are then used to build a basic defect database. For the cropping defect library: based on the location information stored in the label, the pixel blocks in the corresponding image are cropped out. If the label stores the four-point coordinate information of the corresponding quadrilateral, the image within the quadrilateral formed by the four-point coordinates of the corresponding image is cropped out. If the label stores the multi-point coordinate information of the corresponding polygonal box, the image within the polygonal box formed by the multi-point coordinates of the corresponding image is cropped out. Then, these cropped defect targets are classified and stored according to the label category to build a cropping defect library.
3. The multi-category industrial quality inspection method based on generative transfer learning according to claim 2, characterized in that, Step S2 includes: S21: Based on the color characteristics of the new product, use a generative adversarial network to change the characteristics of the defect targets in the original complete defect library, generate a new complete defect library, and obtain synthetic expansion library 1; S22: The defect library is cropped by directly changing the color or using a generative adversarial network to change the features of the defect targets in the original defect library. At the same time, the shape of the defect targets is changed by inverting, mirroring, and increasing or decreasing the area and shape of the targets. Then, the defect image-label pairs and the corresponding cropped images are matched with the same name. The newly generated cropped defect library images are overwritten with the corresponding full defect library images according to the label position information in the full defect library, thus obtaining the synthetic expansion library 2.
4. The multi-category industrial quality inspection method based on generative transfer learning according to claim 3, characterized in that, Step S2 further includes setting a generation parameter evolution mechanism to establish a reinforcement learning-based transfer parameter control system to optimize the generative adversarial network; wherein, the transfer parameter control system includes: using a state space encoder to encode the new product features into an N-dimensional vector, and then using the PPO algorithm to dynamically adjust the learning rate, gradient penalty coefficient and texture fusion intensity of the generator used.
5. A multi-category industrial quality inspection method based on generative transfer learning according to claim 3, characterized in that, Step S2 also includes setting up an automated quality scoring mechanism to calculate the final quality score, and then scoring the generated samples through a pre-trained quality assessment network. Samples with scores greater than a set value are entered into a manual review queue.
6. The multi-category industrial quality inspection method based on generative transfer learning according to claim 5, characterized in that, The automated quality scoring mechanism includes: inputting a generated sample image, then calling a physical simulation module to obtain physical property parameters, then extracting image feature vectors, encoding the process requirement text, performing multimodal feature fusion, and finally calculating the final quality score; wherein, the final quality score formula is: in, Indicates the defect type weight; Indicates the severity of the defect; Indicates the number of defect types.
7. The multi-category industrial quality inspection method based on generative transfer learning according to claim 1, characterized in that, Step S33 includes: S331: Convert CAD models into symbolic distance functions or explicit triangular meshes; S332: Model the deformation field by extracting the surface geometric features, then using a graph neural network to process the grid vertex features, then using an MLP network to output displacement vectors to establish the displacement field, and using latent encoding to control the defect type and intensity; S333: Gradient backpropagation is achieved through the integration of a 3D computing framework, supporting end-to-end optimization from vertex position to pixel color for differentiable rendering; S334: Perform physical constraint modeling, establish a material parameter library, and calculate stress distribution using the finite element method or a neurophysics engine; S335: Based on the morphological characteristics of the transplanted defects, the displacement field isosurface is extracted to generate different defects.
8. The multi-category industrial quality inspection method based on generative transfer learning according to claim 1, characterized in that, Step S35 includes: S351: Input the image patch generated in S34 into the pre-trained Noise2Noise network, which extracts features through an encoder-decoder structure to generate a reconstructed image patch after artifact removal; S352: Calculate the pixel-level residual between the original image patch and the reconstructed image patch output by the network, and generate a residual map; S353: Based on the residual map obtained in S352, artifact regions are determined, thereby establishing a dynamic fusion evaluation system.
9. A multi-category industrial quality inspection method based on generative transfer learning according to claim 8, characterized in that, In step S353, the artifact region determination includes: S3531: Perform threshold segmentation and mark out-of-limit pixels in the residual map as suspected artifact points based on the set empirical threshold. S3532: Perform morphological processing, dilate discrete artifact points, and connect adjacent regions to form an artifact mask. S3533: Perform boundary feature detection, count the artifact mask density near the boundary of the transplanted area, and if the artifact mask density exceeds a preset threshold, it is determined that there is an unnatural boundary, thereby realizing the artifact region determination.
Citation Information
Patent Citations
Method for enhancing surface defect data of metal industrial product
CN114581339A
