Leak-proof adhesive detection method and system based on image recognition

Abstract

The application provides an image recognition-based glue leakage detection method and system. The method comprises the following steps: preprocessing initial image data based on multiple perspectives; eliminating occlusion errors by calculating the weighted average value of each pixel point in the enhanced image in multiple perspectives; calculating a first loss function of a glue leakage defect detection model based on deep learning, and training the glue leakage defect detection model based on the first loss function; designing a conditional generative adversarial network, calculating a second loss function of the conditional generative adversarial network; designing a lightweight glue leakage defect detection model based on a convolutional neural network, and generating a defect probability value based on the region of the image. The application solves the problems of insufficient precision, poor adaptability, insufficient data and the like in the prior art of glue leakage detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image recognition, and particularly relates to a leakage-proof glue detection method and system based on image recognition. BACKGROUND

[0002] With the continuous development of industrial automation, the glue coating precision in precision manufacturing processes such as coating and assembly has become a key link in product quality control. Especially in the electronic product, automobile manufacturing, optical element and semiconductor industries, the precision of glue coating directly affects the performance and appearance of the final product. For example, in the process of manufacturing smart phones and cars, improper glue coating of waterproof seals and structural components can cause product quality problems such as water leakage and poor air tightness, which often need to be reworked, greatly increasing production costs and time.

[0003] Currently, leakage-proof glue detection technology mainly relies on image processing and machine vision technology. Traditional detection methods include using a single-view camera to capture images of the coating area, and then analyzing and judging through image processing algorithms. Although these methods can preliminarily detect glue leakage, they have many limitations. First, because the glue layer during the coating process can be very thin and transparent, the imaging angle and lighting environment of the traditional camera can easily cause the contrast between the glue leakage area and the surrounding coating area to be low, thereby affecting the accuracy of image recognition. Second, existing image processing algorithms mostly rely on fixed image datasets, and the glue leakage samples in these datasets are often insufficient to cover all the glue leakage types that may be encountered in actual production processes, resulting in detection systems being prone to false positives or false negatives when facing unknown glue leakage patterns.

[0004] In addition, most existing detection technologies use single-view image acquisition, which means that only images at a specific angle are used for glue leakage judgment, and the information on the coating surface cannot be fully obtained. This single-view detection method not only easily overlooks some hidden glue leakage problems, but also is easily disturbed by factors such as environmental changes and coating thickness changes. For example, during the coating process, if some angles cannot collect enough data due to equipment obstruction or the particularity of the workpiece shape, the glue leakage area may be missed, thereby affecting the overall detection effect. In addition, due to the limited nature of the image dataset, existing technologies lack effective data enhancement means and are difficult to cope with various glue coating defects that occur in the production process.

[0005] To overcome the above problems, the industry has begun to try to introduce more advanced algorithms, such as deep learning and generative adversarial networks (GAN), to improve detection accuracy and robustness. Although these methods have improved the accuracy of detection to some extent, most systems still face problems such as insufficient data, incomplete simulation of defect types, and insufficient recognition ability of complex defects. For example, generative adversarial networks (GAN) can effectively generate defect images, but are still limited by the training data of the generator and cannot fully simulate all defect types that may occur during the coating process. In addition, due to the complexity of the GAN model itself, it requires a large amount of computing resources and time during training and application, and for real-time detection requirements in industrial production lines, computational efficiency remains a challenge.

[0006] Therefore, although the existing leak-proof adhesive detection system has made some breakthroughs in some aspects, it still faces problems such as low precision, poor adaptability, and insufficient data, which directly affect the automation level of the production line and product quality. SUMMARY

[0007] The purpose of the present application is to provide an image recognition-based leak-proof adhesive detection method and system to solve the problems of insufficient precision, poor adaptability, and insufficient data in the prior art.

[0008] To achieve the above purpose, in a first aspect of the present application, an image recognition-based leak-proof adhesive detection method is provided, the method comprising:

[0009] The initial image data based on multiple perspectives is preprocessed, and local region image enhancement is performed based on the preprocessed image, and the image after image enhancement is normalized to obtain an enhanced image with normalized pixel values;

[0010] Based on each pixel point of the enhanced image, the occlusion error is eliminated by calculating the weighted average value of each pixel point in multiple perspectives to obtain a three-dimensional point set after weighted averaging, and the normal vector of each pixel point in the three-dimensional space is calculated, the surface is smoothed by combining the curvature information of the neighborhood points, the processed pixel points are weighted and fused, and the three-dimensional geometric structure of the workpiece surface is restored;

[0011] Based on the three-dimensional point set after weighted averaging and the normal vector of each pixel point in the three-dimensional space, a first loss function of a leak-proof adhesive defect detection model based on deep learning is calculated, and the leak-proof adhesive defect detection model is trained based on the first loss function;

[0012] A conditional generative adversarial network is designed for the three-dimensional point set output by the leak-proof adhesive defect detection model, a second loss function of the conditional generative adversarial network is calculated, the conditional generative adversarial network is trained based on the second loss function, realistic leak-proof adhesive images are generated, and the training data set of the leak-proof adhesive defect detection model is enhanced;

[0013] According to the glue leakage defect detection model, a lightweight glue leakage defect detection model based on a convolutional neural network is designed, an image output by the glue leakage defect detection model is used as input, a defect probability value based on a region of the image is generated, a judgment of the glue leakage defect is made based on the defect probability value, and adjustment is made.

[0014] Further, the preprocessing adopts a weighted Gaussian smoothing filter for denoising, and uses a geometric mapping method to map pixel points of each image to a unified three-dimensional space coordinate system for three-dimensional mapping, and an image registration method based on image gradients, to ensure accurate positional relationship of images of each view in the three-dimensional space.

[0015] The local region image enhancement adopts an adaptive enhancement algorithm based on local region statistics, introduces an enhancement factor based on local region to dynamically adjust the enhancement intensity according to the contrast, brightness distribution and texture information of different regions in the image.

[0016] The normalization processing includes size standardization and pixel normalization, and the pixel normalization means that the pixel value of the image after the size standardization processing is normalized to the range of [0, 1].

[0017] Further, the weighted average value is calculated by weighting the weights of the pixel points under the current view to obtain the three-dimensional point set after the weighted average, and the weighted average values of multiple views are combined to eliminate occlusion errors.

[0018] Further, the curvature information of the neighborhood points is combined for surface smoothing, the processed pixel points are weighted and fused, and the three-dimensional geometric structure of the workpiece surface is restored, including:

[0019] A target function is designed based on the gradient of each pixel point and the curvature of the corresponding pixel point, the normal vector of each pixel point in the three-dimensional space is calculated, and the curvature information of the neighborhood points is combined for surface smoothing.

[0020] Each pixel point of each view is weighted and averaged based on the depth value of each view, the contribution degree of each view is weighted through the depth information, and the reconstruction accuracy of the real surface region is enhanced.

[0021] Further, based on the three-dimensional point set, a joint description of local curvature and normal vector is combined for micro-deformation feature extraction, and the depth information of the depth value of each view is fused to adjust the weight based on the depth information in the three-dimensional point set.

[0022] Further, the glue leakage defect detection model based on deep learning combines a convolutional neural network and a graph neural network; a first loss function of the glue leakage defect detection model based on deep learning is calculated, and the glue leakage defect detection model is trained based on the first loss function, including:

[0023] Geometric features of each local region are extracted by the convolutional neural network;

[0024] Each local region is taken as a node in the graph neural network, and a connection relationship between nodes is expressed by an adjacency matrix, wherein the adjacency matrix reflects a spatial distance between each three-dimensional point set in the local region;

[0025] The first loss function includes a weighted cross-entropy loss, a geometric regularization term, and a smoothness regularization term, wherein the weighted cross-entropy loss is used to increase attention to the glue leakage region, the geometric regularization term is used to maintain the geometric consistency of the surface and avoid overfitting, and the smoothness regularization term is used to reduce noise and maintain the smoothness of the prediction result;

[0026] The glue leakage defect detection model is trained using the first loss function.

[0027] Further, the conditional generative adversarial network includes a generator and a discriminator, the task of the generator is to generate a synthetic glue leakage image according to an input three-dimensional point set, and the task of the discriminator is to judge whether the image input to the generator is a real image;

[0028] The generator structure includes a plurality of convolutional layers and deconvolutional layers, which map low-dimensional input features to a high-dimensional image space;

[0029] The discriminator extracts features of the image through a convolutional layer and uses a binary classification output to judge whether the input image is a false image generated by the generator.

[0030] Further, a second loss function of the conditional generative adversarial network is calculated, the conditional generative adversarial network is trained based on the second loss function, a realistic glue leakage image is generated, and the training data set of the glue leakage defect detection model is enhanced, including:

[0031] A second loss function based on a min-max loss is designed based on the conditional generative adversarial network;

[0032] The conditional generative adversarial network is trained according to the second loss function, and a conditional control mechanism is introduced, three-dimensional surface features are added as conditional inputs to the input of the generator, so as to generate more realistic glue leakage defect images; wherein the three-dimensional surface features are three-dimensional point sets, normal vectors, and depth information.

[0033] Furthermore, the lightweight glue leakage defect detection model based on convolutional neural networks includes convolutional layers, pooling layers, and fully connected layers. The convolutional layers extract high-dimensional features from the input image, progressively extracting features from low-level to high-level through multiple convolutional operations. The pooling layers reduce computational complexity and enhance feature abstraction through max pooling or average pooling, extracting key information from the image. The fully connected layers perform image classification and output the probability value of the presence of defects.

[0034] When the defect probability value exceeds the set threshold, a feedback mechanism will be automatically triggered to send an adjustment instruction to the production system.

[0035] Adjust the amount of adhesive sprayed from the production equipment;

[0036] Adjust the operating speed of the production line;

[0037] Adjust the flow rate or temperature of the raw materials.

[0038] In another aspect of the invention, an image recognition-based anti-adhesive leakage detection system is provided, the system comprising:

[0039] The initial image data acquisition unit is used to preprocess the initial image data based on multiple viewpoints, perform local region image enhancement based on the preprocessed image, and normalize the enhanced image to obtain an enhanced image with normalized pixel values.

[0040] The image data enhancement unit is used to eliminate occlusion error by calculating the weighted average of each pixel in the enhanced image from multiple viewpoints, obtain a weighted average three-dimensional point set, calculate the normal vector of each pixel in three-dimensional space, combine the curvature information of the neighboring points to perform surface smoothing, and perform weighted fusion of the processed pixels to restore the three-dimensional geometric structure of the workpiece surface.

[0041] The glue leakage defect analysis unit is used to calculate the first loss function of the deep learning-based glue leakage defect detection model based on the weighted average three-dimensional point set and the normal vector of each pixel in the three-dimensional space, and to train the glue leakage defect detection model based on the first loss function.

[0042] The glue leakage defect enhancement unit is used to design a conditional generative adversarial network for the three-dimensional point set optimized by the glue leakage defect detection model, calculate the second loss function of the conditional generative adversarial network, train the conditional generative adversarial network based on the second loss function, generate realistic glue leakage images, and enhance the training dataset of the glue leakage defect detection model.

[0043] The glue leakage result output unit is used for designing a lightweight glue leakage defect detection model based on a convolutional neural network according to the glue leakage defect detection model, using an image output by the glue leakage defect detection model as input, generating a defect probability value based on a region of the image, judging the glue leakage defect based on the defect probability value, and adjusting.

[0044] The beneficial technical effects of the present application are at least as follows:

[0045] (1) The present application adopts multi-view image synthesis technology, which can simultaneously capture images of the coating area from different angles by multiple cameras, and register and synthesize these images to generate a complete omnidirectional coating image. This method overcomes the angle limitation problem of traditional single-view cameras, ensures that the system can fully obtain all information of the coating area, and avoids the problem of missed detection caused by angle blind area or equipment obstruction. Through the fusion of multi-angle images, the system can more accurately capture the tiny glue leakage defects in the coating process, improving the detection accuracy.

[0046] (2) The present application introduces a generative adversarial network (GAN) as the core technology for data enhancement and defect simulation. Traditional glue leakage detection systems usually rely on fixed data sets, but due to the diversity and complexity of glue leakage defect types, existing training data often cannot cover all possible glue leakage situations, resulting in insufficient detection accuracy. By training the GAN model, the generator can simulate various types of glue leakage defects, such as different glue leakage shapes, sizes, and distributions, enhancing the diversity and comprehensiveness of the data set. The discriminator continuously optimizes the detection model, making it have stronger defect recognition ability. In the present application, GAN not only enhances the training data set, but also further improves the system's ability to recognize complex glue leakage patterns, solving the problem of false detection or missed detection of traditional detection systems when facing complex defects.

[0047] (3) The present application combines multi-view image synthesis with a generative adversarial network to form a complete glue leakage detection closed loop. This system not only effectively overcomes the precision, adaptability, and data problems of existing technologies, but also can real-time feedback the coating quality of the production line and automatically repair, ensuring that the coating quality in the production process always meets the standards. This system significantly improves the robustness of the detection system, reduces manual intervention, and reduces production costs, having wide application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0048] The present application is further illustrated by the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present application. For ordinary skilled in the art, other drawings can be obtained without creative labor on the basis of the following drawings.

[0049] Figure 1 A flow chart of the image recognition-based leak-proof adhesive detection method of an embodiment of the present application is shown.

[0050] Figure 2 A system framework diagram of the image recognition-based leak-proof adhesive detection system of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] Embodiments of the present application are described in detail below with reference to examples shown in the accompanying drawings, in which the same or similar numerals or symbols represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0052] As shown in Figure 1 The image recognition-based leak-proof adhesive detection method provided by an embodiment of the present application includes the following steps S1-S5:

[0053] S1, pre-process the initial image data based on multiple perspectives, perform local region image enhancement based on the pre-processed image, and perform normalization processing on the image-enhanced image to obtain an enhanced image with normalized pixel values.

[0054] The core goal of this step is to improve image quality through advanced image preprocessing technology, and to provide high-quality image data for subsequent multi-perspective three-dimensional reconstruction, deep learning training, and leak-proof adhesive detection algorithms. In particular, the processed image should be able to minimize the interference of light changes, background noise, and physical occlusions on workpiece surface detection, thereby improving detection accuracy.

[0055] Specifically, since image acquisition in an industrial environment is often accompanied by problems such as light changes, sensor noise, lens distortion, etc., the primary task is to perform denoising processing on the image. In order to effectively remove noise and preserve the details of the workpiece, a weighted Gaussian smoothing filter (referred to as an "adaptive denoising filter") is used for denoising.

[0056] Let the image data I i , where i represents the perspective number, and N is the noise. The goal of denoising is to filter out the noise N i by weighted Gaussian filtering to obtain the denoised image F i . Define the adaptive weighted filter The core idea is to determine the filtering strength according to the brightness and texture features of the local region:

[0057]

[0058] where, is an adaptive filter based on local region statistics (e.g. local standard deviation and mean), and is the local noise standard deviation. The adaptive filter can dynamically adjust the denoising strength according to the image characteristics of different regions, avoiding over-smoothing that leads to loss of image details.

[0059] In addition, the denoising process should also consider the preservation of texture details in the image, and the texture enhancement regularization term λ T is used to control the degree of denoising to prevent excessive loss of image details. The overall objective can be represented by the following optimization function:

[0060]

[0061] where ||·||2 represents the L2 norm, represents the L1 norm of the image gradient, and λ T is the regularization term for texture preservation. This can effectively remove noise while preserving the details of the workpiece surface.

[0062] Further, the images taken from multiple perspectives need to be spatially aligned to ensure that the pixel positions of the images between different perspectives correspond consistently. In particular, factors such as physical coordinate deviations between images, differences in camera shooting angles, etc. will affect subsequent three-dimensional reconstruction, so accurate geometric mapping methods must be used to map the pixel points of each image to a unified three-dimensional coordinate system.

[0063] Assume that each pixel point has coordinates (p x ,p y ) on the image plane, and its corresponding three-dimensional space coordinates are P i =(x i ,y i ,z i ). To achieve spatial alignment between multiple perspective images, the present application uses a perspective projection model to map image coordinates to three-dimensional space based on camera calibration information. The specific mapping formula is:

[0064]

[0065] where K i is the intrinsic matrix of the camera, is the two-dimensional image coordinate, and P i is the corresponding three-dimensional space coordinate.

[0066] Further, when performing image alignment, an image registration method based on image gradients can be used to ensure the accurate positional relationship of images from each perspective in three-dimensional space, reducing errors caused by camera angles and image acquisition sequence. The image registration optimization objective is:

[0067]

[0068] wherein is the mapped image, is the target image, the optimization target is to minimize the error between the images, ensuring that all images are accurately aligned in space.

[0069] Further, image enhancement is to improve the visibility of the workpiece surface details. The traditional image enhancement method may not be suitable for complex industrial environments, so the present application designs an adaptive enhancement algorithm based on local region statistics (referred to as "local dynamic enhancement algorithm"). This algorithm takes into account the contrast, brightness distribution and texture information of different regions in the image, and dynamically adjusts the enhancement intensity by introducing an enhancement factor based on local regions:

[0070] The present application first calculates the brightness mean μ(x,y) and standard deviation σ(x,y) of the local region of the image, and then adjusts the brightness of the image based on these local statistics:

[0071]

[0072] where α(x,y) and β(x,y) are factors based on local region contrast and brightness adjustment. In order to avoid excessive saturation caused by excessive enhancement, a local constraint function is defined:

[0073]

[0074] where λ E is a regularization factor that controls the enhancement intensity, and through this regularization term, it ensures that details are not lost during the enhancement process and maintains the local smoothness of the image.

[0075] Further, in order to enable subsequent three-dimensional reconstruction and deep learning models to efficiently process images under different viewing angles, all images need to be size standardized and pixel normalized. In order to adapt to the deep learning network, the pixel values of the image will be normalized to the range [0,1].

[0076] Suppose the image still has different brightness ranges after enhancement, the present application normalizes the pixel values by the following formula:

[0077]

[0078] S2, based on each pixel point of the enhanced image, the weighted average value of each pixel point in multiple viewing angles is calculated to eliminate occlusion errors, and the normal vector of each pixel point in the three-dimensional space is calculated. The curvature information of the neighborhood points is combined to smooth the surface, the processed pixel points are weighted and fused, and the three-dimensional geometric structure of the workpiece surface is restored.

[0079] The goal of this step is to recover the 3D geometry of the workpiece surface based on the multi-view image data through a 3D reconstruction algorithm. In order to improve the reconstruction quality, especially the accuracy of the glue missing detection, this step not only performs standard image fusion, but also fuses depth information and eliminates possible occlusion errors and geometric inaccuracies caused by view angle differences.

[0080] Specifically, in multi-view images, due to occlusion, reflection, and surface discontinuity between different views, some areas of image information may be missing, affecting the reconstruction result. In order to compensate for these missing information, the present application introduces an error compensation algorithm based on adaptive weighted fusion.

[0081] For each pixel point P i , the present application eliminates occlusion errors by calculating its weighted average value in multiple views, and defines the weighted coefficient w i as the visibility or reliability of the pixel point in each view, which is usually dependent on the coverage of the view and the surface reflection characteristics. The 3D point set after weighted averaging The calculation formula is:

[0082]

[0083] where w ij is the weight of the pixel point P i in view j, and P ij is the 3D coordinate point in the jth view. In this way, the present application can effectively compensate for the occluded or partially missing areas.

[0084] Unlike traditional image weighted fusion methods, the present application introduces a view-based weighting strategy, which considers the reliability of each view (such as view overlap, reflected light influence, etc.), and then dynamically adjusts the weight. This method can significantly reduce the reconstruction error caused by view error or occlusion, and improve the reconstruction accuracy.

[0085] Further, in order to ensure that the reconstructed 3D model surface is smooth and continuous, the present application introduces a surface smoothing algorithm based on geometric features to optimize the reconstructed 3D model. This algorithm eliminates noise by analyzing the local geometric features (such as curvature, normal vector) of the 3D model, avoiding geometric defects caused by discontinuity.

[0086] First, the present application calculates the normal vector n i of each pixel point P i in the 3D space, and combines the curvature information of the neighborhood points for surface smoothing. The objective function is as follows:

[0087]

[0088] where, represents the gradient of the pixel point P i , κ i is the curvature of the point, λ curvature is the curvature regularization term, which balances the surface smoothness and geometric accuracy. By optimizing this objective function, the surface will become smoother and effectively remove noise, ensuring the accuracy and reliability of three-dimensional reconstruction.

[0089] This method combines curvature and normal vector to optimize the surface, and based on the traditional smoothing method, the application proposes an adaptive regularization method based on local geometric features. This method can effectively remove surface noise without affecting the details of the model, especially in the edge and detail part of the workpiece, which can better preserve the real geometric shape.

[0090] Further, in addition to the standard multi-view image fusion, the application will also use the depth information of each view for optimization to enhance the quality of surface reconstruction. Through the fusion and optimization processing of the depth map, the detail quality of the three-dimensional model can be further improved, especially in the concave and convex areas and weak parts of the workpiece.

[0091] The application designs a depth map weighted fusion algorithm, which weights the contribution of each view through depth information, thereby enhancing the reconstruction accuracy of the real surface area. Assuming that the depth value of each view is d i , then the weighted surface point The calculation formula is:

[0092]

[0093] Unlike traditional direct application of depth information, the application introduces an adaptive weighting strategy for the depth map, which adjusts the weight through the quality of the depth value during calculation, effectively avoiding errors introduced by inaccurate depth data. Through this method, the reconstruction accuracy of small details on the workpiece surface can be improved.

[0094] S3, based on the weighted average three-dimensional point set and the normal vector of each pixel point in the three-dimensional space, calculate the first loss function of the glue leakage defect detection model based on depth learning, and train the glue leakage defect detection model based on the first loss function.

[0095] In this stage, the application will construct a glue leakage defect detection model based on depth learning based on the three-dimensional points in step S2. Through in-depth analysis of the optimized three-dimensional surface, potential glue leakage defect areas are identified. The following are the detailed steps:

[0096] Specifically, the application first extracts features from the three-dimensional points output by step S2. Each point It contains its three-dimensional spatial coordinate information, and the optimized surface has certain smoothness and geometric features.

[0097] Geometric Feature Extraction: This invention focuses on extracting geometric information such as local curvature and normal vectors at each point. These features help the model identify subtle deformations on the surface, especially areas with adhesive leakage defects. The innovation lies in combining a joint description of local curvature and normal vectors; this combination effectively identifies minute deformations.

[0098]

[0099] in, It represents the local second-order gradient, revealing the change in curvature. The gradient of a point reflects the change in the surface normal vector.

[0100] This combination helps to capture defects that might not be detected by curvature or normal vectors alone.

[0101] Deep information fusion: the depth information d of each point i It will be input into the model along with geometric features. Depth information helps to better understand surface undulations and subtle defects; therefore, this invention defines a weighted depth feature:

[0102]

[0103] Where w depth These are the weighting coefficients for depth information. In this way, the model can adjust the degree of emphasis on depth information according to the specific characteristics of the workpiece.

[0104] Furthermore, to improve the accuracy of adhesive leakage detection, a multi-scale detection model is designed by combining convolutional neural networks (CNN) and graph neural networks (GNN). This combination can simultaneously extract local details and global spatial relationships.

[0105] CNN part: Extracting R for each local region using a convolutional neural network. i Geometric features. Convolutional layers can learn subtle local shape changes on the surface, making them particularly suitable for detecting minute defects such as adhesive leakage.

[0106] GNN Section: To further capture the spatial relationships between regions, a Graph Neural Network (GNN) is introduced. Each region R... i Consider a node in the graph, and the connections between nodes are represented by the adjacency matrix A. ij To express, A ij Reflects the point and points Spatial distance between:

[0107]

[0108] where, is the Euclidean distance between two points; and is the standard deviation of the Gaussian kernel, controlling the influence range of the neighborhood. In this way, the GNN can effectively learn the geometric continuity between different regions, enhancing the spatial correlation of the glue leakage detection.

[0109] Further, in order to solve the class imbalance problem in glue leakage detection and enhance the detection ability of the model for small defects, the application designs an innovative loss function, including weighted cross-entropy, geometric regularization term and robustness smoothness term.

[0110] Weighted cross-entropy loss: Considering the scarcity of glue leakage regions in the data, the application introduces a weighting coefficient w i to increase the attention to glue leakage regions:

[0111]

[0112] where, w i is the class weight for each sample, ensuring that the glue leakage region has a large weight in training. L i is the true label of each region (glue leakage / non-glue leakage). is the defect probability predicted by the model.

[0113] Geometric regularization term: In order to maintain the geometric consistency of the surface and avoid overfitting, the application adds a geometric regularization term. This term helps the model to learn the surface changes more in line with the actual geometric structure of the workpiece, ensuring that the prediction result has physical rationality:

[0114]

[0115] where, and are the second derivatives of the predicted surface and the real surface, respectively, representing the change in local curvature.

[0116] This term helps the model avoid unreasonable geometric distortion and ensures correct positioning of the glue leakage region.

[0117] Smoothness regularization term: To reduce noise and maintain the smoothness of the prediction result, the application introduces a smoothness regularization term, making the prediction result more coherent and easy to generalize:

[0118]

[0119] where, is the gradient of the predicted surface, n iThe real normal vector of the workpiece surface. This regularization term can help the model avoid overfitting when dealing with more complex three-dimensional surfaces, and maintain the robustness of the prediction.

[0120] Further, by combining the weighted cross-entropy, the geometric regularization term and the smoothness regularization term, the final first loss function can be represented as:

[0121]

[0122] Where λ1 and λ2 are the weights of the regularization terms, adjusted through cross-validation to ensure the generalization ability of the model. Through this training method, the model can effectively detect the glue leakage defects, especially when facing complex and small defects on the workpiece surface, it has strong robustness and accuracy.

[0123] S4, for the optimized output three-dimensional point set of the glue leakage defect detection model, design a conditional generative adversarial network, calculate the second loss function of the conditional generative adversarial network, train the conditional generative adversarial network based on the second loss function, generate realistic glue leakage images, and enhance the training data set of the glue leakage defect detection model.

[0124] In this step, the present application generates synthetic glue leakage images by introducing a generative adversarial network (GAN), aiming to enhance the training data set, especially in the case of few glue leakage defect areas and difficult to obtain. By generating synthetic data, the diversity of glue leakage defects can be effectively increased, and the generalization ability of the model can be improved.

[0125] Specifically, the input data: the input of this stage comes from step S3, i.e. the three-dimensional point set with optimized geometric features output by the deep learning model These three-dimensional points contain the position, normal vector and depth information of each surface point, etc.

[0126] For each point Its coordinates represent the three-dimensional position of the workpiece surface, n i is the normal vector, d i is the depth information.

[0127] Objective: The present application aims to generate synthetic images with glue leakage defects based on these inputs. These images can simulate glue leakage defects of different angles, different sizes and different degrees to enhance the training data set.

[0128] Further, the GAN architecture design:

[0129] In order to accurately simulate the details of the glue leakage defect in the synthetic image, the present application designs a conditional generative adversarial network (cGAN), which can generate synthetic images according to the input three-dimensional surface features The generated image is conditioned on the input point cloud (including geometry and depth information) to generate realistic missing glue images.

[0130] Generator: The task of the generator is to generate an image from the input point cloud The generated image is conditioned on the input point cloud (including geometry and depth information) to generate realistic missing glue images.

[0131] The output of the generator is a synthetic image The size of the generated image is consistent with the actual workpiece surface image, and the defect area in the generated image simulates the missing glue phenomenon.

[0132] Discriminator: The task of the discriminator is to determine whether the input image is a real image. The discriminator extracts the features of the image through convolutional layers and uses binary classification output to determine whether the input image is a fake image generated by the generator.

[0133] The input of the discriminator is a real missing glue image or a generated image The output of the discriminator is a probability indicating whether the image is a real image.

[0134] Loss function design: In order to train the generator and the discriminator, the present application designs an adversarial loss function based on the min-max loss, which is suitable for the training of the generative adversarial network. For the loss of the generator, the present application calculates it through the following formula:

[0135]

[0136] wherein, is the probability output by the discriminator, indicating that the image is a real image.

[0137] The generator improves the realism of the generated image by maximizing the probability of the discriminator misclassifying the generated image.

[0138] The loss function of the discriminator is:

[0139]

[0140] wherein, I i is a real missing glue image, and D(I i ) represents the judgment of the discriminator on the real image.

[0141] Further, to ensure that the generated images of glue leakage by the generator are consistent with the actual defects, the present application introduces a conditional control mechanism. The generator not only relies on random noise as input, but also takes three-dimensional surface features as conditional input, so as to generate more realistic images of glue leakage defects.

[0142] Conditional input: each time the generator is trained, it takes the geometric features (position, normal vector, etc.) of each point in the three-dimensional point set as input. By doing so, the generator can generate glue leakage defects corresponding to specific workpiece surface features, enhancing the personalization and realism of the generated images.

[0143] Specifically, the generator input is:

[0144]

[0145] where z i is the conditional input, containing the geometric features and depth information of the surface.

[0146] Image enhancement: To further enhance the diversity of the dataset, the present application can perform random rotation, scaling, affine transformation, etc. on the generated images to simulate different viewing angles and shooting conditions. These transformations will make the model more robust during training, and it can handle more complex actual situations.

[0147] Further, the present application optimizes the entire GAN network by alternately training the generator and the discriminator. In each training iteration, the parameters of the generator and the discriminator are updated, so that the generator can generate more and more realistic images of glue leakage, and the discriminator becomes more and more skilled at distinguishing between true and false images.

[0148] Training objective: the objective of the generator is to maximize the "true" judgment of the discriminator on the generated images, while the objective of the discriminator is to maximize the correct classification of the real images and the correct judgment of the generated images. In this way, the generator and the discriminator constantly play against each other, thus generating high-quality synthetic images of glue leakage.

[0149] Final second loss function: by minimizing the loss functions of the generator and the discriminator, the final optimization objective is:

[0150]

[0151] where λ reg is the weight of the regularization term, used to limit the geometric consistency of the generated images.

[0152] Regularization term: to ensure that the generated images meet the geometric constraints of the workpiece, the present application can add a regularization term to the loss function, such as by penalizing the geometric consistency between the generated images and the real images:

[0153]

[0154] where, and are the second-order gradients of the predicted and real images, respectively, used to measure the curvature difference of the surface. Through such training and optimization, the images generated by the GAN can effectively expand the dataset of glue leakage defects, improving the recognition ability of the trained model for glue leakage defects.

[0155] S5, according to the glue leakage defect detection model, a lightweight glue leakage defect detection model based on convolutional neural network is designed, the image output by the glue leakage defect detection model is used as input, the defect probability value based on the region of the image is generated, and the judgment of the glue leakage defect is made based on the defect probability value and adjusted.

[0156] In this stage, the goal is to apply the model generated and optimized in the previous steps to real-time glue leakage defect detection, and introduce a quality feedback mechanism to achieve automated quality detection and rapid feedback, thereby optimizing the production process and workpiece quality. The system automatically detects glue leakage defects by analyzing images or three-dimensional data in real time, and optimizes production or equipment adjustment through the feedback mechanism.

[0157] Specifically, the present application adopts a lightweight glue leakage defect detection model based on convolutional neural network (CNN). The model extracts features in the input image through multiple convolution operations, and judges whether there is a glue leakage defect through a classification network.

[0158] Model architecture: the detection model includes the following modules:

[0159] Convolutional layer: used to extract high-dimensional features of the input image Through multiple convolution operations, low-level features are gradually extracted to high-level features.

[0160] Pooling layer: through maximum pooling or average pooling, the computational complexity is reduced, the abstraction degree of the features is improved, and the key information of the image is extracted.

[0161] Fully connected layer: finally, the image classification is realized through the fully connected layer, and the probability of defect existence is output.

[0162] The input of the model is The output is the defect probability value of each image region Specifically, the model judges the glue leakage defect through the following classification formula:

[0163]

[0164] where, is the image The probability of detecting defects. W is the weight matrix of the classification layer, and b is the bias term. For the image features extracted, multi-layer convolution and pooling operations are performed. σ is the Sigmoid activation function, and the output is the probability value of the defect, ranging between [0, 1].

[0165] Further, in order to ensure quality control and real-time optimization in the production process, the present application introduces a quality feedback mechanism, which automatically adjusts the production equipment or process parameters according to the results of defect detection. The key to this mechanism is to quickly process the detection results and timely transmit the feedback information to the production system.

[0166] Feedback strategy: when the probability of detecting glue leakage defects exceeds the set threshold θ, the system will automatically trigger the feedback mechanism and send adjustment instructions to the production system. These adjustment instructions may include:

[0167] Adjusting the glue spraying amount of the production equipment;

[0168] Adjusting the running speed of the production line;

[0169] Adjusting the flow or temperature of raw materials and other parameters.

[0170] Further, the core of the feedback mechanism is to optimize the production parameters based on real-time detection results and historical defect patterns. For example, when the probability of glue leakage defects for a certain workpiece category or production batch exceeds the threshold, the system will automatically generate adjustment instructions A i :

[0171]

[0172] where A i is the feedback adjustment instruction for workpiece i, which may include device settings and process adjustments. is the probability of glue leakage defects for the i-th workpiece image. θ is the set threshold for defect probability, and when P defect , the feedback mechanism is triggered. N is the total number of images detected with defects.

[0173] As Figure 2 shown, in another embodiment of the present application, an image recognition-based glue leakage detection system is provided, which comprises:

[0174] An initial image data acquisition unit 501 for pre-processing initial image data based on multiple perspectives, performing local region image enhancement based on the pre-processed images, and performing normalization processing on the image-enhanced images to obtain pixel value normalized enhanced images;

[0175] The image data enhancement unit 502 is configured to eliminate the occlusion error by calculating a weighted average value of each pixel point of the enhanced image in multiple view angles, obtain a three-dimensional point set after weighted averaging, and calculate a normal vector of each pixel point in a three-dimensional space, combine the curvature information of the neighborhood points to smooth the surface, perform weighted fusion on the processed pixel points, and restore the three-dimensional geometric structure of the workpiece surface.

[0176] The glue leakage defect analysis unit 503 is configured to calculate a first loss function of a glue leakage defect detection model based on deep learning based on the three-dimensional point set after weighted averaging and the normal vector of each pixel point in the three-dimensional space, and train the glue leakage defect detection model based on the first loss function.

[0177] The glue leakage defect enhancement unit 504 is configured to optimize the three-dimensional point set output by the glue leakage defect detection model, design a conditional generative adversarial network, calculate a second loss function of the conditional generative adversarial network, train the conditional generative adversarial network based on the second loss function, generate a realistic glue leakage image, and enhance the training data set of the glue leakage defect detection model.

[0178] The glue leakage result output unit 505 is configured to design a lightweight glue leakage defect detection model based on a convolutional neural network according to the glue leakage defect detection model, use the image output by the glue leakage defect detection model as input, generate a defect probability value based on a region of the current image, and make a judgment on the glue leakage defect and make an adjustment based on the defect probability value.

[0179] In addition, technical details not described in detail in the present embodiment can be referred to the parameter operation method provided by any embodiment of the present application, which will not be described here.

[0180] It should be noted that in this paper, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0181] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example method can be realized by means of software and a necessary general hardware platform, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk) and includes a number of instructions for causing an end device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method described in each embodiment of the present application.

[0183] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A leak-proof adhesive detection method based on image recognition, characterized in that, The method includes: The initial image data based on multiple perspectives is preprocessed, and local region image enhancement is performed based on the preprocessed image. The enhanced image is then normalized to obtain an enhanced image with normalized pixel values. Based on each pixel of the enhanced image, the occlusion error is eliminated by calculating its weighted average value in multiple viewpoints, resulting in a weighted average three-dimensional point set. The normal vector of each pixel in three-dimensional space is calculated, and surface smoothing is performed by combining the local curvature information of the neighboring points. The processed pixels are then weighted and fused to restore the three-dimensional geometric structure of the workpiece surface. Based on the weighted average three-dimensional point set and the normal vector of each pixel in the three-dimensional space, the first loss function of the deep learning-based glue leakage defect detection model is calculated, and the glue leakage defect detection model is trained based on the first loss function. For the 3D point set optimized by the glue leakage defect detection model, a conditional generative adversarial network is designed, a second loss function of the conditional generative adversarial network is calculated, and the conditional generative adversarial network is trained based on the second loss function to generate realistic glue leakage images and enhance the training dataset of the glue leakage defect detection model. Based on the glue leakage defect detection model, a lightweight glue leakage defect detection model based on convolutional neural network is designed. The image output by the glue leakage defect detection model is used as input to generate a defect probability value based on the region of the current image. The glue leakage defect is judged and adjusted based on the defect probability value. Specifically, based on the three-dimensional point set, feature extraction of local curvature and normal vector is performed, and micro-deformation feature extraction is performed by combining the joint description of local curvature and normal vector. Depth information fusion is performed for the depth value of each viewpoint, and the weights based on depth information in the three-dimensional point set are adjusted. The deep learning-based adhesive leakage defect detection model combines convolutional neural networks and graph neural networks; then, the first loss function of the deep learning-based adhesive leakage defect detection model is calculated, and the adhesive leakage defect detection model is trained based on the first loss function, including: Geometric features of each local region are extracted using a convolutional neural network; Each local region is treated as a node in a graph neural network, and the connection relationships between nodes are expressed by an adjacency matrix, which reflects the spatial distance between each set of three-dimensional points in the local region. The first loss function includes a weighted cross-entropy loss, a geometric regularization term, and a smoothness regularization term. The weighted cross-entropy loss is used to increase attention to the area of ​​glue leakage. The geometric regularization term is used to maintain the geometric consistency of the surface and avoid overfitting. The smoothness regularization term is used to reduce noise and maintain the smoothness of the prediction results. The glue leakage defect detection model is trained using the first loss function; The conditional generative adversarial network includes a generator and a discriminator. The generator's task is to generate a synthetic image of glue leakage based on the input 3D point set, and the discriminator's task is to determine whether the image input to the generator is a real image. The generator structure includes multiple convolutional and deconvolutional layers, which map low-dimensional input features to a high-dimensional image space. The discriminator extracts image features through a convolutional layer and uses binary classification output to determine whether the input image is a fake image generated by the generator. The second loss function of the conditional generative adversarial network is calculated, and the network is trained based on this second loss function to generate realistic glue leakage images. The training dataset for enhancing the glue leakage defect detection model includes: Design a second loss function based on minimum-maximum loss based on conditional generative adversarial networks; The conditional generative adversarial network is trained according to the second loss function, and a conditional control mechanism is introduced. By adding three-dimensional surface features as conditional inputs to the generator, more realistic images of glue leakage defects are generated. The three-dimensional surface features are three-dimensional point sets, normal vectors, and depth information.

2. The image recognition-based method for detecting adhesive leakage according to claim 1, characterized in that, The preprocessing uses a weighted Gaussian smoothing filter for noise reduction, and a geometric mapping method to map the pixels of each image to a unified three-dimensional coordinate system for three-dimensional mapping, as well as an image registration method based on image gradients to ensure the precise positional relationship of the images from each viewpoint in three-dimensional space. The local region image enhancement adopts an adaptive enhancement algorithm based on local region statistics. According to the contrast, brightness distribution and texture information of different regions in the image, an enhancement factor based on local region is introduced to dynamically adjust the enhancement intensity. The normalization process includes size normalization and pixel normalization. Pixel normalization means that the pixel values ​​of the image after size normalization will be normalized to the range of [0,1].

3. The image recognition-based method for detecting adhesive leakage according to claim 1, characterized in that, The weighted average is calculated by weighting the pixels based on the current viewpoint to obtain a weighted average 3D point set. The weighted average from multiple views is then used to eliminate occlusion errors.

4. The image recognition-based method for detecting adhesive leakage according to claim 3, characterized in that, The process of smoothing the surface by incorporating the local curvature of neighboring points, and then weighting and fusing the processed pixels to restore the three-dimensional geometric structure of the workpiece surface includes: The objective function is designed based on the gradient of each pixel and the local curvature of the corresponding pixel. The normal vector of each pixel in three-dimensional space is calculated, and the surface is smoothed by combining the local curvature of the neighboring points. A weighted average is calculated for each pixel in each viewpoint based on the depth value of each viewpoint. The contribution of each viewpoint is weighted by the depth information, thereby enhancing the reconstruction accuracy of the real surface area.

5. The image recognition-based method for detecting adhesive leakage according to claim 1, characterized in that, The lightweight glue leakage defect detection model based on convolutional neural networks includes convolutional layers, pooling layers, and fully connected layers. The convolutional layers extract high-dimensional features from the input image, progressively extracting features from low-level to high-level through multiple convolutional operations. The pooling layers reduce computational complexity and enhance feature abstraction through max pooling or average pooling, extracting key information from the image. The fully connected layers perform image classification and output the probability value of the presence of defects. When the defect probability value exceeds the set threshold, a feedback mechanism will be automatically triggered to send an adjustment instruction to the production system. Adjust the amount of adhesive sprayed from the production equipment; Adjust the operating speed of the production line; Adjust the flow rate or temperature of the raw materials.

6. A system for performing the image recognition-based anti-leakage adhesive detection method as described in claim 1, characterized in that, The system includes: The initial image data acquisition unit is used to preprocess the initial image data based on multiple viewpoints, perform local region image enhancement based on the preprocessed image, and normalize the enhanced image to obtain an enhanced image with normalized pixel values. The image data enhancement unit is used to eliminate occlusion error by calculating the weighted average of each pixel in the enhanced image from multiple viewpoints, obtain a weighted average three-dimensional point set, calculate the normal vector of each pixel in three-dimensional space, combine the curvature information of the neighboring points to perform surface smoothing, and perform weighted fusion of the processed pixels to restore the three-dimensional geometric structure of the workpiece surface. The glue leakage defect analysis unit is used to calculate the first loss function of the deep learning-based glue leakage defect detection model based on the weighted average three-dimensional point set and the normal vector of each pixel in the three-dimensional space, and to train the glue leakage defect detection model based on the first loss function. The glue leakage defect enhancement unit is used to design a conditional generative adversarial network for the three-dimensional point set optimized by the glue leakage defect detection model, calculate the second loss function of the conditional generative adversarial network, train the conditional generative adversarial network based on the second loss function, generate realistic glue leakage images, and enhance the training dataset of the glue leakage defect detection model. The glue leakage result output unit is used to design a lightweight glue leakage defect detection model based on a convolutional neural network according to the glue leakage defect detection model, use the image output by the glue leakage defect detection model as input, generate a defect probability value for the region based on the image, and judge and adjust the glue leakage defect based on the defect probability value.

Images