Automobile sealant anomaly detection method and system based on deep transfer learning

Abstract

This invention discloses a method and system for detecting anomalies in automotive sealant based on deep transfer learning. The method includes: a virtual sample generation step, generating virtual sealant images containing defects such as sealant breakage, offset, perforation, and abnormal sealant width based on CAD data; a domain adaptation feature mapping step, aligning virtual samples to the feature space of real working conditions through an adversarial domain adaptation network; an anomaly detection classification step, outputting detection results and confidence levels; and a progressive detection refinement step, collecting difficult samples for incremental model updates. This invention requires only 10% of the labeled sample size of traditional methods to achieve comparable detection accuracy, belonging to the field of machine vision technology.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and deep learning technology, specifically to a method and system for detecting abnormalities in automotive sealant based on deep transfer learning. Background Technology

[0002] Automotive lights, serving as tools for illumination and safety signal transmission, are essential components of automobiles. To improve their performance, high-quality sealing is crucial. Poor sealing allows moisture intrusion, affecting illumination, lifespan, and appearance, ultimately jeopardizing driving safety. In automotive light manufacturing, sealant is widely used for sealing due to its strong adhesion. It is applied evenly to the first component, bonding it to the second component through the sealant layer. However, issues such as the inability to precisely control the amount of sealant applied during production make it difficult to guarantee consistent sealing quality. Poor sealing directly impacts equipment performance and lifespan, consequently affecting operational reliability.

[0003] In the modern, high-speed production line environment, the human eye is either unable to effectively identify sealant defects or has low identification efficiency. Furthermore, high labor costs are forcing manufacturers to undertake technological transformation, introducing machine vision technology to improve product quality inspection efficiency. Existing technologies utilize machine vision to inspect the sealing quality of sealants, such as sealant sealing quality inspection solutions based on deep learning and Halcon processing technology.

[0004] The sealant sealing quality inspection method disclosed in Chinese patent CN116912249A has the following technical problems: First, the deep learning-based detection method requires a large amount of labeled datasets during training, and a lot of time is spent on sample labeling, which is time-consuming and labor-intensive; Second, when the production background changes or the product model is switched, data needs to be collected again and the model needs to be trained, which is not very applicable; Third, the existing method cannot effectively solve the problem of scarce defect samples in industrial scenarios, especially for the introduction of new products or small-batch production scenarios, it is difficult to obtain enough defect samples for model training; Fourth, the existing detection model usually maintains fixed parameters after deployment, and cannot continuously optimize the detection performance according to the difficult cases encountered in actual production.

[0005] The aforementioned technical problems result in existing sealant anomaly detection methods having drawbacks such as high sample acquisition costs, weak model generalization ability, poor adaptability to new scenarios, and limited improvement in detection accuracy, which restricts the large-scale application of deep learning technology in the field of automotive sealant quality inspection. Summary of the Invention

[0006] To address the shortcomings of existing technologies, such as insufficient sealant detection samples, weak model transferability, and difficulty in continuously improving detection performance, this invention provides a method and system for anomaly detection of automotive sealant based on deep transfer learning. This method, through an innovative domain-adaptive deep learning framework, achieves efficient knowledge transfer from virtual samples to actual working conditions. It requires only 10% of the labeled sample size of traditional methods to achieve comparable detection accuracy, significantly reducing the deployment cost of the AI ​​system.

[0007] The technical solution adopted by this invention to solve its technical problem is: a method for detecting abnormalities in automotive sealant based on deep transfer learning, comprising the following steps: a virtual sample generation step, acquiring CAD model data of the vehicle to be tested and sealant material characteristic parameters, extracting three-dimensional geometric information of the sealant coating path based on the CAD model data, and generating a virtual sealant image set containing multiple defect types based on material characteristic parameters and preset defect type parameters, including sealant breakage defects, offset defects, perforation defects, and abnormal sealant width defects; a domain adaptation feature mapping step, inputting the virtual sealant image set as source domain data into a feature extraction network to obtain source domain feature vectors, inputting a small number of collected real sealant images as target domain data into a feature extraction network to obtain target domain feature vectors, and aligning the source domain feature vectors and target domain feature vectors in feature space through an adversarial domain adaptation network, which includes a feature extractor and a domain discriminator. The feature extractor extracts domain-invariant features, and the domain discriminator determines whether the features originate from the source or target domain. Domain-adapted feature representations are obtained by minimizing the adversarial loss between the feature extractor and the domain discriminator. In the anomaly detection and classification step, the domain-adapted feature representations are input into the anomaly detection classifier, which outputs detection results and corresponding detection confidence scores based on these representations. Detection results include normal categories and multiple defect categories. In the progressive detection refinement step, the detection confidence score is compared with a preset hard sample threshold. When the detection confidence score is lower than the hard sample threshold, the corresponding real sealant image is marked as a hard sample and stored in a hard sample cache queue. When the number of samples in the hard sample cache queue reaches a preset batch threshold, the feature extraction network and the anomaly detection classifier are incrementally updated and trained using the samples in the hard sample cache queue, and the updated network parameters are fed back to the domain-adapted feature mapping step.

[0008] Preferably, the weighting coefficient of the adversarial loss ranges from 0.1 to 0.5, the hard sample threshold ranges from 0.3 to 0.7, and the batch threshold ranges from 16 to 128.

[0009] Preferably, the number of real sealant images is 5% to 15% of the number of virtual sealant image sets.

[0010] This invention also provides an automotive sealant anomaly detection system based on deep transfer learning, comprising: a virtual sample generation module, used to acquire CAD model data of the vehicle to be inspected and sealant material characteristic parameters, extract three-dimensional geometric information of the sealant coating path based on the CAD model data, and generate a virtual sealant image set including sealant breakage defects, offset defects, perforation defects, and sealant width abnormality defects based on the material characteristic parameters and preset defect type parameters; a domain adaptation feature mapping module, used to use the virtual sealant image set as source domain data and a small number of collected real sealant images as target domain data, perform feature space alignment through an adversarial domain adaptation network including a feature extractor and a domain discriminator, and output the domain-adapted feature representation; an anomaly detection classification module, used to output detection results including normal categories and multiple defect categories and corresponding detection confidence scores based on the domain-adapted feature representation; and a progressive detection refinement module, used to store real sealant images with detection confidence scores lower than a preset hard sample threshold into a hard sample cache queue, and incrementally update and train the domain adaptation feature mapping module and the anomaly detection classification module when the number of samples reaches a batch threshold.

[0011] The beneficial effects of this invention are as follows: First, by using virtual sample generation technology based on CAD data, the problem of scarce sealant defect samples in industrial scenarios is effectively solved, reducing data acquisition and annotation costs. Second, by employing an adversarial domain adaptation network to achieve knowledge transfer from virtual samples to real working conditions, only 10% of the real labeled samples required by traditional methods are needed to achieve comparable detection accuracy. Third, the innovative progressive detection refinement mechanism can continuously collect difficult samples for model optimization, enabling detection performance to steadily improve over time. Fourth, the system can be quickly adapted to different vehicle models and production lines, significantly improving the versatility and deployment efficiency of the detection system. Attached Figure Description

[0012] Figure 1 This is a flowchart of an automotive sealant anomaly detection method based on deep transfer learning provided in an embodiment of the present invention.

[0013] Figure 2 This is an architecture diagram of an automotive sealant anomaly detection system based on deep transfer learning provided in an embodiment of the present invention. Detailed Implementation

[0014] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0015] like Figure 1As shown in the figure, this invention provides an anomaly detection method for automotive sealant based on deep transfer learning. This method achieves efficient knowledge transfer from virtual data to real working conditions through four core steps: virtual sample generation, domain-adaptive feature mapping, anomaly detection classification, and progressive detection refinement. This effectively solves the problem of insufficient sealant detection samples in industrial scenarios.

[0016] Step S1: Virtual sample generation step.

[0017] The core objective of the virtual sample generation step is to automatically generate a large-scale, diverse virtual sealant image dataset based on vehicle CAD data and sealant material properties, providing sufficient source domain samples for subsequent domain adaptation training.

[0018] First, the CAD model data of the vehicle to be tested is acquired. In the automotive manufacturing industry, each vehicle model's headlight assembly has a corresponding 3D CAD design file. These files precisely record the geometry, width parameters, and position information of the sealant application path. The system reads the CAD model file of the target vehicle from the enterprise's product data management platform, supporting mainstream 3D formats such as STEP, IGES, and STL. The CAD model data contains a complete geometric description of the sealant application area, including the centerline coordinate sequence of the application path, the designed sealant width value, and the normal vector information of the application area.

[0019] Secondly, the three-dimensional geometric information of the sealant application path is extracted. The system performs geometric analysis on the imported CAD model to identify the three-dimensional curve corresponding to the sealant application path. Specifically, by traversing the boundary representations in the CAD model, closed contour curves marked as sealant application areas are selected, and the discrete point coordinate sequence of these curves is extracted. Let the sealant application path consist of n sampling points, and the three-dimensional coordinates of the i-th sampling point be denoted as... The sampling interval is adaptively adjusted based on the curvature of the coating path, with smaller intervals used in areas of high curvature to ensure geometric accuracy. Simultaneously, the tangent and normal vectors at each sampling point are calculated to provide the geometric basis for subsequent virtual image rendering.

[0020] Next, the system obtains the material properties of the sealant. The visual appearance of the sealant is affected by its optical properties, including surface reflectivity, roughness, and color. The system reads the optical parameters corresponding to the target sealant model from the material database. Let the diffuse reflectance coefficient of the sealant be... The specular reflectance coefficient is Surface roughness is The RGB components of the inherent color are respectively These parameters are obtained by performing spectral measurements on actual sealant samples beforehand and are stored in a material parameter database for system retrieval.

[0021] Then, a bidirectional reflectance distribution function (BRDF) model of the sealant surface is established based on material property parameters. The bidirectional reflectance distribution function (BRDF) describes the reflection distribution characteristics of light after it strikes an object's surface and is a core model for achieving realistic rendering. This invention uses the Cook-Torrance microplane model to construct the BRDF of the sealant surface, and its mathematical expression is:

[0022] ,

[0023] in, This represents the bidirectional reflectance distribution function value; This represents the direction vector of the incident light; Indicates the direction vector of the emitted light; The formula for calculating the half-range vector is as follows: ; Represents the surface normal vector; Indicates the diffuse reflectance coefficient; The normal distribution function of the microplane is represented by the GGX distribution. The Fresnel reflection function is represented using the Schlick approximation; This represents the geometric occlusion function.

[0024] Next, virtual defect regions are generated based on the defect type parameters. This invention supports the generation of four main sealant defect types: adhesive breakage defects, offset defects, perforation defects, and abnormal adhesive width defects. For each defect type, the system maintains a set of defect generation parameters, including defect location distribution parameters, defect size distribution parameters, and defect morphology parameters.

[0025] The formation process of the sealant breakage defect is as follows: The breakage initiation position is randomly selected along the sealant application path. The breakage positions follow a uniform distribution. Let the normalized position of the breakage initiation point on the path be denoted as . The fracture length follows a log-normal distribution. Let the fracture length be... ,in and Based on statistical data of actual adhesive breakage defects, the sealant geometry within the fracture area is removed from the 3D model to create the adhesive breakage effect.

[0026] The process of generating the offset defect is as follows: Randomly select the start and end positions of the offset along the sealant application path; the length of the offset segment follows a normal distribution; calculate the offset direction vector, which is usually perpendicular to the tangent vector of the application path; the offset distance follows a normal distribution, let the offset distance be... The sealant geometry within the offset section is translated along the offset direction to create an offset effect.

[0027] The formation process of perforation defects is as follows: The center position of the perforation is randomly selected within the sealant coating area; the perforation positions follow a two-dimensional uniform distribution; the perforation diameter follows a log-normal distribution. Let the perforation diameter be... A circular through hole is made on the sealant geometry with the center of the perforation as the center and the diameter of the perforation as the diameter, to create a perforation effect.

[0028] The generation process of the sealant width anomaly defect is as follows: Randomly select the start and end positions of the anomaly along the sealant application path; calculate the sealant width anomaly coefficient, which follows a normal distribution. Let the sealant width anomaly coefficient be... ,in This indicates that the glue width is too wide. This indicates that the sealant width is too narrow; the sealant geometry in the abnormal section is scaled along the normal direction to create an abnormal sealant width effect.

[0029] Finally, a virtual sealant image is generated using the rendering engine. The system employs Physically Based Rendering (PBR) to render the virtual sealant image based on the previously constructed BRDF model, defect geometry model, and preset lighting conditions. During the rendering process, the system simulates the imaging parameters of an industrial camera, including field of view, focal length, and exposure time, to ensure consistency in visual characteristics between the virtual image and the real-world acquired image. Simultaneously, the system adds appropriate Gaussian noise and blurring effects to the rendering result to simulate image noise and optical distortion during the actual acquisition process.

[0030] Through the above steps, the system can automatically generate a large-scale virtual sealant image dataset. In a preferred embodiment of the present invention, for each vehicle model, the system generates 10,000 virtual sealant images, of which normal samples account for 15%, samples with broken sealant defects account for 20%, samples with offset defects account for 25%, samples with perforation defects account for 15%, and samples with abnormal sealant width defects account for 25%.

[0031] Step S2: Domain adaptation feature mapping step.

[0032] The core objective of the domain adaptation feature mapping step is to eliminate the distributional differences between virtual samples (source domain) and real samples (target domain) through an adversarial domain adaptation network, so that the feature representations learned on virtual samples can be effectively transferred to real-world scenarios.

[0033] First, the set of virtual sealant images is used as the source domain data input to the feature extraction network. Let the source domain dataset be... ,in This represents the i-th virtual sealant image. This indicates the corresponding defect category label. This represents the total number of samples in the source domain. The feature extraction network uses a deep convolutional neural network structure, which is mathematically represented as follows: ,in Represents network parameters, Represents the input image space. Represent the feature space. For source domain samples The feature extraction network outputs source domain feature vectors. .

[0034] The feature extraction network employs a multi-scale feature pyramid structure to simultaneously capture both detailed texture features and overall structural features of the image. The network comprises four convolutional blocks and three residual connection modules. The first convolutional block contains a 7×7 convolutional layer, a batch normalization layer, and a ReLU activation function layer, with 64 output channels and the feature map size remaining unchanged. The second convolutional block contains two cascaded 3×3 convolutional layers, with 128 output channels and the feature map size downsampled to half its original size. The third convolutional block contains two cascaded 3×3 convolutional layers, with 256 output channels and the feature map size downsampled to one-quarter its original size. The fourth convolutional block contains two cascaded 3×3 convolutional layers, with 512 output channels and the feature map size downsampled to one-eighth its original size.

[0035] During the multi-scale feature extraction process, the system extracts feature maps from the outputs of the second, third, and fourth convolutional blocks, respectively, denoted as . , and To unify the spatial resolution of feature maps at different scales, the system... 2x upsampling ,right 4x upsampling was performed to obtain , so that it is with They have the same spatial dimensions. Then, the feature maps from the three scales are concatenated along the channel dimension to obtain multi-scale fused features. Its number of channels is Finally, a 1×1 convolutional layer is used to reduce the number of channels in the multi-scale fused features to 256, and then a global average pooling layer is used to obtain a 256-dimensional feature vector.

[0036] Secondly, a small number of real sealant images were collected and used as input to the feature extraction network as target domain data. Let the target domain dataset be... ,in This represents the j-th real sealant image. This represents the total number of samples in the target domain. It should be noted that target domain samples typically do not have label information, or only a small number of samples have label information. In this invention, the number of target domain samples is much smaller than the number of source domain samples, i.e. Preferably for 5% to 15% for target domain samples. The feature extraction network outputs a feature vector of the target domain. .

[0037] Then, an adversarial domain adaptation network is constructed to achieve feature space alignment. The core idea of ​​the adversarial domain adaptation network is to introduce a domain discriminator. The discriminator attempts to distinguish whether the feature vector originates from the source domain or the target domain; simultaneously, the feature extractor... The aim is to deceive the domain discriminator, making it unable to distinguish the source of features. Through this adversarial training, the feature extractor is forced to learn domain-invariant features, meaning that the extracted features have a statistically similar distribution regardless of whether the input is a virtual or real sample.

[0038] Domain discriminator The network structure is as follows: the input layer receives a 256-dimensional feature vector; the first fully connected layer outputs a 128-dimensional vector using the LeakyReLU activation function; the second fully connected layer outputs a 64-dimensional vector using the LeakyReLU activation function; the output layer is a single neuron using the Sigmoid activation function, and the output is the domain discrimination probability. Let the output of the domain discriminator be... ,in Given the input feature vector, an output value close to 1 indicates that it is classified as the source domain, and close to 0 indicates that it is classified as the target domain.

[0039] The training process for adversarial domain adaptation employs a gradient inversion strategy. During the forward propagation phase, the domain discriminator normally receives feature vectors and outputs the domain discrimination result; during the backward propagation phase, the gradient generated by the domain discriminator passes through a gradient inversion layer (GRL) before being passed to the feature extractor. The gradient inversion layer inverts the gradient, i.e., (Forward propagation) (Backpropagation), where This is the gradient reversal coefficient.

[0040] The total loss function for adversarial domain adaptation is defined as:

[0041]

[0042] in, This represents the total loss value; The classification loss is represented by the cross-entropy loss function, which is calculated only for labeled source domain samples and a small number of labeled target domain samples. Indicates a counter-cyclical loss; The weighting coefficient represents the adversarial loss, with a value ranging from 0.1 to 0.5.

[0043] The formula for calculating adversarial loss is:

[0044]

[0045] in, Indicates the adversarial loss value; Indicates the number of samples in the source domain; Indicates the number of samples in the target domain; This represents the i-th source domain sample; This represents the j-th target domain sample; This represents a feature extraction network; Representation domain discriminator.

[0046] Furthermore, this invention introduces a multi-scale domain difference measurement mechanism to more precisely characterize the distributional differences between the source and target domains. The formula for calculating the multi-scale domain difference measurement value is as follows:

[0047]

[0048] in, This represents a multi-scale domain difference measure. This represents the layer index of the feature extraction network, with values ​​of 2, 3, and 4 corresponding to the second, third, and fourth convolutional blocks, respectively. Let the weight coefficients of the l-th layer satisfy the following condition: ; This represents the feature distribution of the source domain samples in the l-th layer; This represents the feature distribution of the target domain samples in the l-th layer; This represents the maximum mean difference function.

[0049] The formula for calculating the maximum mean difference (MMD) is:

[0050]

[0051] in, This represents the maximum difference in mean between distributions P and Q; This represents the number of samples in distribution P; This represents the number of samples in distribution Q; This represents the i-th sample in distribution P; This represents the j-th sample in distribution Q; The kernel mapping function is represented by a Gaussian radial basis kernel; Represents the regenerated nucleus Hilbert space; denoted as Hilbert space norm.

[0052] Step S3: Anomaly detection and classification steps.

[0053] The core objective of the anomaly detection and classification step is to classify the defect type of the sealant image based on the domain-adapted feature representation and output the detection confidence score for subsequent identification of difficult samples.

[0054] First, the domain-adapted feature representation is input into the anomaly detection classifier. The anomaly detection classifier adopts a multi-layer fully connected network structure, with the following architecture: the input layer receives a 256-dimensional domain-adapted feature vector; the first fully connected layer outputs a 128-dimensional vector using ReLU activation and Dropout regularization (Dropout rate 0.5); the second fully connected layer outputs a 64-dimensional vector using ReLU activation and Dropout regularization (Dropout rate 0.3); and the output layer consists of 5 neurons corresponding to 5 categories (normal, glue break, offset, perforation, and glue width abnormality) and using the Softmax activation function.

[0055] Let the output of the classifier be ,in Let represent the probability that a sample belongs to the k-th class, satisfying . The detection result is taken from the category with the highest probability, i.e.:

[0056]

[0057] in, Indicates the predicted category of the detection result; Indicates a category index; This represents the predicted probability of the k-th class.

[0058] The detection confidence score is defined as the probability value of the predicted class, that is:

[0059]

[0060] in, Indicates the confidence level of the detection; This represents the predicted probability of the k-th class. Detection confidence reflects the model's degree of certainty about the current prediction result; a higher confidence score indicates that the model is more certain of the correctness of the current prediction result.

[0061] Step S4: Progressive testing and refining steps.

[0062] The core objective of the incremental detection refinement step is to continuously improve detection performance by incrementally updating and training the model through the continuous collection of difficult samples in the production process.

[0063] First, the detection confidence level is compared with a preset hard sample threshold. Let the hard sample threshold be... The value ranges from 0.3 to 0.7, with a preferred value of 0.5. When the detection confidence level... Below the hard sample threshold At that time, that is This indicates that the model has significant uncertainty in its judgment of the current sample, and the sample is identified as a difficult sample.

[0064] The mathematical expression for the hard sample confidence evaluation algorithm is:

[0065]

[0066] in, The function represents the hard sample label for sample x, where a value of 1 indicates a hard sample and a value of 0 indicates a non-hard sample. This represents the detection confidence level of sample x; This represents the preset threshold for difficult samples.

[0067] Then, the identified difficult samples are stored in a difficult sample cache queue. The difficult sample cache queue is managed using a first-in, first-out (FIFO) strategy; when the queue is full, a newly added difficult sample replaces the oldest added sample. Let the capacity of the difficult sample cache queue be... The current number of samples in the queue is .

[0068] Next, data augmentation is performed on the hard samples to improve sample diversity. Data augmentation operations include: random rotation transformation, with rotation angles randomly sampled within the range of -15° to +15°; random scaling transformation, with scaling ratios randomly sampled within the range of 0.9 to 1.1; and random brightness transformation, with brightness adjustment coefficients randomly sampled within the range of 0.8 to 1.2. The augmented hard samples are stored together with the original hard samples in the hard sample cache queue.

[0069] When the number of samples in the hard sample cache queue reaches the preset batch threshold Incremental training is triggered at a certain time. The batch threshold ranges from 16 to 128, with a preferred value of 32. Incremental training employs an experience replay strategy, randomly selecting a portion of samples from historical training samples and mixing them with difficult samples to form incremental training batches. Let the experience replay ratio be... The number of historical samples in the incremental training batch is The number of difficult samples is ,in The value range is from 0.3 to 0.7.

[0070] Incremental training employs a mini-batch stochastic gradient descent optimizer, with a decaying learning rate. Let the initial learning rate be... The attenuation coefficient is Each completed After one round of training, the learning rate is updated as follows:

[0071]

[0072] in, This represents the updated learning rate; This represents the learning rate before the update; This represents the attenuation coefficient, with a value ranging from 0.9 to 0.99.

[0073] Finally, the updated network parameters are fed back to the domain adaptation feature mapping step, enabling online model updates. Through a progressive detection refinement mechanism, the model can continuously learn new challenging cases from the production process, constantly optimizing its detection performance. Experiments show that after one month of continuous operation, the model's detection accuracy can be improved by 3% to 8%.

[0074] In a preferred embodiment of the present invention, the configuration of each key parameter is as follows: adversarial loss weight coefficient Set to 0.3, the hard sample threshold Set to 0.5, batch threshold Set to 32, experience replay ratio Set the initial learning rate to 0.5. Set to 0.001, learning rate decay coefficient Set to 0.95, gradient reversal coefficient The number of cells increases linearly from 0 to 1 during training.

[0075] To verify the effectiveness of the method of this invention, a comparative experiment was conducted on an automotive headlight production line. The experiment used 10,000 virtual sealant images as source domain data and only 1,000 real sealant images as target domain data (accounting for 10% of the virtual samples). The comparative methods included: traditional object detection methods (requiring training entirely with real samples), direct transfer methods without domain adaptation, and methods using domain adaptation but without progressive refinement.

[0076] Experimental results show that: traditional object detection methods achieve an accuracy of 94.2% when trained with 10,000 real samples; the direct transfer method (using only 10% of real samples) achieves an accuracy of 78.5%; the method using domain adaptation but without progressive refinement achieves an accuracy of 89.3%; and the method of this invention (using both domain adaptation and progressive refinement) achieves an accuracy of 93.8%. This demonstrates that the method of this invention achieves comparable detection accuracy to traditional methods using only 10% of the real sample size. Furthermore, after one month of continuous progressive refinement, the detection accuracy of the method of this invention further improves to 96.5%, surpassing the best performance of traditional methods.

[0077] In terms of detection speed, the method of this invention has a single-frame detection time of 28 milliseconds, which meets the requirements of real-time detection on the production line. Regarding the number of model parameters, the lightweight feature extraction network used in this invention contains only about 3.5 million parameters, which can be deployed on edge computing devices.

[0078] The method of this invention has significant technical advantages compared with existing technologies. Existing sealant detection methods mainly rely on the collection and labeling of a large number of real samples. For example, the sealant sealing quality detection method disclosed in the prior art (patent application CN116912249A) requires acquiring two images before and after sealant application, performing differential processing, using a target detection model to determine whether sealant has been applied to the area, and then judging the defect type such as sealant breakage, overflow, or insufficient sealant based on the sealant contour. Although this method can achieve basic sealant quality detection, it has the following technical drawbacks: First, this method relies on a pre-trained target detection model, and the training of the target detection model requires a large number of labeled samples, making it difficult to obtain sufficient training data in the new product introduction stage or small-batch production scenarios; Second, this method uses a fixed threshold for defect judgment, which cannot adapt to changes in lighting and material differences under different production conditions; Third, the detection model of this method has fixed parameters after deployment and cannot be continuously optimized according to the difficulties encountered in actual production.

[0079] In contrast, the method of this invention utilizes innovative virtual sample generation technology to automatically generate large-scale virtual training samples using vehicle CAD data and material optical parameters, effectively solving the problem of scarce real defect samples. Through an adversarial domain adaptation network, this invention can effectively transfer feature knowledge learned from virtual samples to real-world scenarios, achieving high-precision detection with only a small number of real samples. Through a progressive detection refinement mechanism, this invention can continuously collect difficult samples for incremental model updates, enabling detection performance to steadily improve over time. These technological innovations collectively constitute the core technological contribution of this invention, making its method significantly superior to existing technologies in terms of sample efficiency, detection accuracy, and adaptability.

[0080] In practical applications, the method of this invention also needs to consider the handling of various boundary conditions and abnormal situations. When the input image has severe occlusion or extremely poor image quality, the detection confidence will be significantly reduced, and the system will automatically mark the sample as a difficult sample and issue an alarm to remind the operator to perform manual review. When the difficult sample cache queue has not triggered incremental training for a long time, the system will periodically perform model status checks to evaluate whether the model has experienced performance degradation. If the model performance is detected to be lower than a preset threshold, the system will automatically trigger the model retraining process. In addition, when the system detects that the distribution of the input image differs too much from that of the training data, such as when a new type of sealant material is used or when there is a significant change in the lighting conditions of the production line, the system will automatically prompt the user to update the material parameters or re-collect calibration samples.

[0081] This invention's method exhibits excellent multi-vehicle scalability. Since virtual sample generation is entirely based on CAD data and material parameters, when new vehicle models need support, users only need to import the new model's CAD model file and configure the corresponding sealant material parameters. The system can then automatically generate virtual training samples for the new model and train a dedicated detection model. In a preferred embodiment, the system supports managing detection models for up to 50 vehicle models simultaneously. Each model is stored, updated, and operates independently without interference. When the production line switches vehicle models, the system automatically loads the corresponding detection model based on the model code issued by the upstream production management system. The entire switching process requires no manual intervention and takes no more than 2 seconds.

[0082] like Figure 2 As shown, this embodiment of the invention provides an automotive sealant anomaly detection system based on deep transfer learning. The system includes four core functional modules: a virtual sample generation module, a domain adaptation feature mapping module, an anomaly detection classification module, and a progressive detection refinement module. The modules form a deep coupling and closed-loop collaboration of data flow and control flow.

[0083] The virtual sample generation module is used to acquire the CAD model data and sealant material characteristic parameters of the vehicle model to be tested. Based on the CAD model data, it extracts the three-dimensional geometric information of the sealant application path and generates a set of virtual sealant images containing defects such as glue breakage, offset, perforation, and abnormal glue width based on the material characteristic parameters and preset defect type parameters.

[0084] The virtual sample generation module's internal structure includes a CAD parsing unit, a material modeling unit, a defect generation unit, and an image rendering unit. The CAD parsing unit is responsible for reading and parsing the CAD model file and extracting the geometric information of the sealant coating path. The material modeling unit is responsible for constructing a bidirectional reflection distribution function model based on material property parameters, as described in the method embodiments. The defect generation unit is responsible for generating various defects on the sealant geometric model based on defect type parameters, as described in the method embodiments. The image rendering unit is responsible for generating virtual sealant images based on physically based rendering methods, as described in the method embodiments.

[0085] The output of the virtual sample generation module is a set of virtual sealant images, which is passed as source domain data to the domain adaptation feature mapping module.

[0086] The domain-adaptive feature mapping module uses a set of virtual sealant images as source domain data and a small number of real sealant images as target domain data. It performs feature space alignment through an adversarial domain adaptation network that includes a feature extractor and a domain discriminator, and outputs the domain-adapted feature representation.

[0087] The internal structure of the domain adaptation feature mapping module includes a feature extractor, a domain discriminator, a gradient inversion layer, and a loss calculation unit. The feature extractor employs a multi-scale feature pyramid structure, as described in the method embodiments. The domain discriminator employs a multi-layer fully connected network structure, as described in the method embodiments. The gradient inversion layer implements the gradient inversion operation for adversarial training. The loss calculation unit is responsible for calculating the classification loss, adversarial loss, and total loss.

[0088] The domain-adaptive feature mapping module receives inputs from virtual sealant images from the virtual sample generation module and real sealant images from an industrial camera. Its output is a domain-adapted feature representation, which is then passed to the anomaly detection and classification module. Furthermore, the module receives model update parameters and a set of difficult samples from the progressive detection refinement module, enabling online model updates.

[0089] The anomaly detection and classification module is used to output detection results and corresponding detection confidence scores for normal categories and multiple defect categories based on the domain-adapted feature representation.

[0090] The anomaly detection classification module's internal structure includes a classification network and a confidence calculation unit. The classification network employs a multi-layer fully connected network structure, as described in the method embodiments. The confidence calculation unit is responsible for calculating the detection confidence based on the output probabilities of the classification network, as described in the method embodiments.

[0091] The anomaly detection classification module takes as input the domain-adapted feature representation from the domain-adapted feature mapping module. The output of the anomaly detection classification module includes the detection result and the detection confidence score. The detection result is output to the upper-layer application system for quality assessment, while the detection confidence score is passed to the progressive detection refinement module for hard sample identification.

[0092] The progressive detection refinement module stores real sealant images with a detection confidence level lower than a preset hard sample threshold into a hard sample cache queue. When the number of samples reaches the batch threshold, the domain adaptation feature mapping module and the anomaly detection classification module are incrementally updated and trained.

[0093] The progressive detection and refinement module's internal structure includes a hard sample identification unit, a hard sample cache queue, a data augmentation unit, and an incremental training unit. The hard sample identification unit is responsible for determining whether a sample is a hard sample based on detection confidence and a hard sample threshold, as described in the method embodiments. The hard sample cache queue stores and manages hard samples using a first-in, first-out (FIFO) strategy. The data augmentation unit performs enhancement processing on hard samples, such as random rotation, scaling, and brightness transformation, as described in the method embodiments. The incremental training unit performs incremental update training when the number of hard samples reaches a batch threshold, as described in the method embodiments.

[0094] The progressive detection refinement module takes as input the detection confidence score from the anomaly detection classification module and the corresponding real sealant image. The output of the progressive detection refinement module is the model update parameters, which are fed back to the domain-adaptive feature mapping module to enable online model updates.

[0095] The four core modules of this invention form a deep coupling and closed-loop collaborative relationship: the output of the virtual sample generation module serves as the key input to the domain adaptation feature mapping module, providing large-scale source domain training data; the output of the domain adaptation feature mapping module serves as the input to the anomaly detection classification module, providing the domain-adapted feature representation; the output of the anomaly detection classification module is fed back to the progressive detection refinement module, providing a basis for identifying difficult samples; the output of the progressive detection refinement module is fed back to the domain adaptation feature mapping module and the anomaly detection classification module, enabling online updates of model parameters. This closed-loop collaborative mechanism allows the system to continuously learn from production data and constantly optimize its detection performance.

[0096] This invention's system can be deployed on industrial edge computing devices or run on cloud servers. In edge deployment mode, the system runs on an industrial control computer installed next to the production line, connecting to industrial cameras and PLC controllers via industrial Ethernet to achieve real-time detection and online updates. In cloud deployment mode, the system runs on a cloud server, receiving image data uploaded from the production line via a 5G network, and after completing the detection, sending the results down to the production line control system.

[0097] In edge deployment mode, recommended hardware configurations include: an industrial PC with an Intel Core i7 or equivalent CPU, at least 16GB of RAM, an NVIDIA GeForce RTX 2060 or higher GPU for deep learning inference acceleration, and at least 512GB of SSD storage for model files and hard sample caching. A color camera with a resolution of at least 5 megapixels and a frame rate of at least 30 frames per second is recommended, along with a ring LED light source to ensure uniform illumination. The system interfaces with the production line control system via Gigabit industrial Ethernet, supporting mainstream industrial communication protocols such as OPC UA and Modbus TCP.

[0098] In cloud deployment mode, cloud server instances with GPU acceleration capabilities are recommended. The system supports containerized deployment and can be quickly deployed to Kubernetes clusters via Docker images to achieve elastic scaling and high availability. The cloud system also supports parallel processing of multiple production lines, with a single cloud server capable of serving the testing tasks of up to 10 production lines simultaneously.

[0099] This invention's system supports unified management of multiple vehicle models and production lines. Users can configure CAD models and material parameters for different vehicle models through the system management interface. The system automatically generates corresponding virtual samples and trains dedicated models. When the production line switches vehicle models, the system automatically loads the corresponding model without manual intervention. The system also provides comprehensive monitoring and alarm functions, including statistical analysis of test results, model performance monitoring, and anomaly alarm push notifications, helping users to promptly grasp the production quality status.

[0100] This invention provides a standardized interface specification, facilitating integration with existing enterprise production and quality management systems. The image input interface supports both GigE Vision and USB3 Vision industrial camera interface standards, and also supports receiving image files over a network. The detection result output interface offers both RESTful API and message queue methods, returning data in formats including detailed information such as detection category, confidence level, defect location coordinates, and defect area. The model management interface provides functions such as model upload, download, activation, and backup, and supports remote model updates. The system status interface provides real-time data such as system operating status, performance indicators, and alarm information, supporting integration with the enterprise's unified operation and maintenance monitoring platform.

[0101] The embodiments of the present invention are not limited to the specific embodiments described above. Those skilled in the art can make various equivalent changes or substitutions based on the technical solutions of the present invention, and all such changes or substitutions should be included within the protection scope of the present invention.

Claims

1. A method for detecting anomalies in automotive sealant based on deep transfer learning, characterized in that, Includes the following steps: The virtual sample generation step involves acquiring the CAD model data of the vehicle to be tested and the sealant material characteristic parameters, extracting the three-dimensional geometric information of the sealant application path based on the CAD model data, and generating a set of virtual sealant images containing multiple defect types based on the material characteristic parameters and preset defect type parameters. The defect types include sealant breakage defects, offset defects, perforation defects, and abnormal sealant width defects. The domain-adaptive feature mapping step involves inputting the set of virtual sealant images as source domain data into a feature extraction network to obtain source domain feature vectors, and inputting a small number of collected real sealant images as target domain data into the feature extraction network to obtain target domain feature vectors. An adversarial domain adaptation network is then used to align the feature spaces of the source domain feature vectors and the target domain feature vectors. This adversarial domain adaptation network includes a feature extractor and a domain discriminator. The feature extractor extracts domain-invariant features, and the domain discriminator determines whether the features originate from the source domain or the target domain. The domain-adapted feature representation is obtained by minimizing the adversarial loss between the feature extractor and the domain discriminator. The anomaly detection and classification step involves inputting the domain-adapted feature representation into an anomaly detection classifier. The anomaly detection classifier outputs the detection result and the corresponding detection confidence based on the domain-adapted feature representation. The detection result includes normal categories and multiple defect categories. The progressive detection refinement step compares the detection confidence with a preset hard sample threshold. When the detection confidence is lower than the hard sample threshold, the corresponding real sealant image is marked as a hard sample and stored in a hard sample cache queue. When the number of samples in the hard sample cache queue reaches a preset batch threshold, the feature extraction network and the anomaly detection classifier are incrementally updated and trained using the samples in the hard sample cache queue, and the updated network parameters are fed back to the domain adaptation feature mapping step.

2. The method for detecting abnormalities in automotive sealant based on deep transfer learning according to claim 1, characterized in that, The weighting coefficient of the adversarial loss ranges from 0.1 to 0.5, the hard sample threshold ranges from 0.3 to 0.7, and the batch threshold ranges from 16 to 128.

3. The method for detecting anomalies in automotive sealant based on deep transfer learning according to claim 1, characterized in that, The process of generating a set of virtual sealant images containing multiple defect types based on the material property parameters and preset defect type parameters includes: determining the optical reflectance coefficient and surface roughness coefficient of the sealant according to the material property parameters; establishing a bidirectional reflectance distribution function model of the sealant surface based on the optical reflectance coefficient and the surface roughness coefficient; randomly generating defect regions on the sealant coating path according to the defect location distribution parameters and defect size distribution parameters in the defect type parameters; and performing texture rendering on the defect regions through a rendering engine to obtain virtual sealant images.

4. The method for detecting abnormalities in automotive sealant based on deep transfer learning according to claim 1, characterized in that, The proportion of samples of each defect type in the virtual sealant image set is configured according to the probability of occurrence of each defect type in actual production. Among them, the proportion of samples with broken glue is 15% to 25%, the proportion of samples with offset defects is 20% to 30%, the proportion of samples with perforation defects is 10% to 20%, the proportion of samples with abnormal glue width is 25% to 35%, and the proportion of normal samples is 10% to 20%.

5. The method for detecting abnormalities in automotive sealant based on deep transfer learning according to claim 1, characterized in that, The step of aligning the source domain feature vector and the target domain feature vector using an adversarial domain adaptation network includes: the feature extractor adopts a multi-scale feature pyramid structure, extracting feature maps of different scales in shallow, medium and deep layers respectively, concatenating the feature maps of different scales by channel dimension to obtain multi-scale fused features, the domain discriminator performing binary classification on the multi-scale fused features, and inverting the gradient of the domain discrimination loss through a gradient inversion layer before backfeeding it to the feature extractor.

6. The method for detecting abnormalities in automotive sealant based on deep transfer learning according to claim 5, characterized in that, The feature extractor includes a group of convolutional layers, residual connection modules, and a global average pooling layer connected in sequence. The group of convolutional layers includes four convolutional blocks, each containing a convolutional layer, a batch normalization layer, and an activation function layer. The number of residual connection modules is 3 to 6.

7. The method for detecting abnormalities in automotive sealant based on deep transfer learning according to claim 1, characterized in that, The step of marking the corresponding real sealant image as a difficult sample and storing it in the difficult sample cache queue further includes: performing data augmentation processing on the difficult sample, the data augmentation processing including random rotation transformation, random scaling transformation and random brightness transformation, and storing the augmented difficult sample together with the original difficult sample in the difficult sample cache queue.

8. The method for detecting abnormalities in automotive sealant based on deep transfer learning according to claim 1, characterized in that, The incremental update training of the feature extraction network and the anomaly detection classifier using samples in the hard sample cache queue includes: randomly selecting a portion of samples from historical training samples and mixing them with the hard samples using an experience replay strategy, and performing mini-batch gradient descent training on the mixed samples according to a preset learning rate decay strategy, wherein the learning rate decay strategy is to multiply the learning rate by a decay coefficient after each preset training round.

9. The method for detecting abnormalities in automotive sealant based on deep transfer learning according to claim 1, characterized in that, The number of the small number of real sealant images is 5% to 15% of the number of virtual sealant images, and the real sealant images are acquired on the production line by industrial cameras.

10. A deep transfer learning-based automotive sealant anomaly detection system, used to implement the deep transfer learning-based automotive sealant anomaly detection method according to any one of claims 1-9, characterized in that, include: The virtual sample generation module is used to acquire CAD model data and sealant material characteristic parameters of the vehicle model to be tested, extract three-dimensional geometric information of the sealant coating path based on the CAD model data, and generate a set of virtual sealant images including sealant breakage defects, offset defects, perforation defects and abnormal sealant width defects based on the material characteristic parameters and preset defect type parameters. The domain-adaptive feature mapping module is used to take the set of virtual sealant images as source domain data and a small number of real sealant images as target domain data, and perform feature space alignment through an adversarial domain adaptation network containing a feature extractor and a domain discriminator, and output the domain-adapted feature representation. An anomaly detection and classification module is used to output detection results and corresponding detection confidence scores, including normal categories and multiple defect categories, based on the domain-adapted feature representation. The progressive detection refinement module is used to store real sealant images with a detection confidence level lower than a preset difficult sample threshold into a difficult sample cache queue. When the number of samples reaches the batch threshold, the domain adaptation feature mapping module and the anomaly detection classification module are incrementally updated and trained.

Images