Glue leakage prevention detection method and system based on image recognition

Abstract

The invention provides a glue leakage prevention detection method and system based on image recognition. The method comprises the following steps: preprocessing initial image data based on a plurality of visual angles; on the basis of each pixel point of the enhanced image, calculating a weighted average value of the pixel point in a plurality of view angles to eliminate the occlusion error; calculating a first loss function of a glue leakage defect detection model based on deep learning, and training the glue leakage defect detection model based on the first loss function; designing a conditional generative adversarial network, and calculating a second loss function of the conditional generative adversarial network; and designing a lightweight glue leakage defect detection model based on the convolutional neural network, and generating a defect probability value of a region based on the image. The problems of insufficient precision, poor adaptability, insufficient data and the like in the prior art of glue leakage prevention detection are solved.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition, and particularly relates to a glue leakage prevention detection method and system based on image recognition. Background Art

[0002] With the continuous development of industrial automation, the glue coating accuracy in precision manufacturing processes such as coating and assembly has become a key link in product quality control. Especially in the electronics, automotive manufacturing, optical component, and semiconductor industries, the accuracy of glue coating directly affects the performance and appearance of the final product. For example, in the manufacturing processes of smart phones and automobiles, improper glue coating for waterproof sealing and structural components will lead to product quality problems such as water leakage and poor airtightness, which often require rework, greatly increasing the production cost and time.

[0003] Currently, the glue leakage prevention detection technology mainly relies on image processing and machine vision technology. Traditional detection methods include using a single-view camera to capture images of the coating area and then analyzing and judging through image processing algorithms. Although these methods can initially detect glue leakage, they have many limitations. First, since the glue coating layer during the coating process may be very thin and transparent, the imaging angle and lighting environment of traditional cameras can easily result in a low contrast between the glue leakage area and the surrounding coating area, thereby affecting the accuracy of image recognition. Second, most existing image processing algorithms rely on fixed image datasets, and the glue leakage samples in these datasets are often insufficient to cover all possible glue leakage types encountered in the actual production process, resulting in the detection system being prone to false detection or missed detection when facing unknown glue leakage patterns.

[0004] In addition, most existing detection technologies use single-view image acquisition, which means that only images at specific angles are used for glue leakage judgment, and the information on the coating surface cannot be comprehensively obtained. This single-view detection method not only easily ignores some hidden glue leakage problems but is also easily interfered by factors such as environmental changes and coating thickness changes. For example, during the coating process, if insufficient data can be collected at certain angles due to equipment occlusion or the special shape of the workpiece, it may lead to the missed detection of the glue leakage area, thereby affecting the overall detection effect. Moreover, due to the limited nature of the image dataset, existing technologies lack effective data augmentation means and are difficult to cope with various glue coating defects that occur during the production process.

[0005] To overcome the above problems, the industry has begun to attempt to introduce more advanced algorithms, such as deep learning and generative adversarial networks (GANs), to improve detection accuracy and robustness. Although these methods have improved the detection accuracy to a certain extent, most systems still face problems such as insufficient data, incomplete simulation of defect types, and insufficient ability to identify complex defects. For example, generative adversarial networks (GANs) can effectively generate defect images, but they are still limited by the training data of the generator and cannot fully simulate all possible defect types that may occur during the coating process. In addition, due to the complexity of the GAN model itself, a large amount of computing resources and time are required during its training and application processes, and computational efficiency is still a problem for real-time detection requirements in industrial production lines.

[0006] Therefore, although existing anti-leakage glue detection systems have made some breakthroughs in certain aspects, they still face problems such as low accuracy, poor adaptability, and insufficient data, which directly affect the automation level and product quality of the production line. Summary of the Invention

[0007] The object of the present invention is to propose an anti-leakage glue detection method and system based on image recognition to solve the problems of insufficient accuracy, poor adaptability, and insufficient data in the prior art.

[0008] To achieve the above object, in the first aspect of the present invention, an anti-leakage glue detection method based on image recognition is provided. The method includes:

[0009] Preprocess the initial image data based on multiple perspectives, perform local region image enhancement on the preprocessed image, and perform normalization processing on the image after image enhancement to obtain an enhanced image with normalized pixel values;

[0010] Based on each pixel point of the enhanced image, eliminate the occlusion error by calculating its weighted average value in multiple perspectives to obtain a weighted average three-dimensional point set, calculate the normal vector of each pixel point in three-dimensional space, perform surface smoothing in combination with the curvature information of neighboring points, and perform weighted fusion on the processed pixel points to restore the three-dimensional geometric structure of the workpiece surface;

[0011] Calculate the first loss function of the leakage glue defect detection model based on deep learning based on the weighted average three-dimensional point set and the normal vector of each pixel point in three-dimensional space, and train the leakage glue defect detection model based on the first loss function;

[0012] For the three-dimensional point set optimized by the output of the leakage glue defect detection model, design a conditional generative adversarial network, calculate the second loss function of the conditional generative adversarial network, train the conditional generative adversarial network based on the second loss function, generate realistic leakage glue images, and enhance the training data set of the leakage glue defect detection model;

[0013] According to the glue leakage defect detection model, a lightweight glue leakage defect detection model based on a convolutional neural network is designed. Using the image output by the glue leakage defect detection model as input, a defect probability value for the region based on this image is generated, and the glue leakage defect is judged and adjusted based on the defect probability value.

[0014] Further, the preprocessing uses a weighted Gaussian smoothing filter for denoising, and uses a geometric mapping method to map the pixel points of each image to a unified three-dimensional space coordinate system for three-dimensional mapping, as well as an image registration method based on image gradients to ensure the precise positional relationship of the images from each perspective in the three-dimensional space;

[0015] The local region image enhancement uses an adaptive enhancement algorithm based on local region statistics. According to the contrast, brightness distribution, and texture information in different regions of the image, an enhancement factor based on the local region is introduced to dynamically adjust the enhancement intensity;

[0016] The normalization processing includes size standardization and pixel normalization. The pixel normalization means that the pixel values of the image after size standardization processing will be normalized to the range of [0,1].

[0017] Further, the weighted average is calculated by weighted averaging based on the weights of the pixel points under the current perspective to obtain a weighted average three-dimensional point set, and the weighted averages from multiple perspectives are combined to eliminate occlusion errors.

[0018] Further, the surface is smoothed by combining the curvature information of neighboring points, and the processed pixel points are weighted and fused to restore the three-dimensional geometric structure of the workpiece surface, including:

[0019] Design an objective function based on the gradient of each pixel point and the curvature of the corresponding pixel point, calculate the normal vector of each pixel point in the three-dimensional space, and smooth the surface by combining the curvature information of neighboring points;

[0020] Based on the depth value of each perspective, weighted average calculation is performed on each pixel point of each perspective, and the contribution degree of each perspective is weighted by the depth information, so as to enhance the reconstruction accuracy of the real surface area.

[0021] Further, based on the three-dimensional point set, micro-deformation feature extraction is performed by combining the joint description of local curvature and normal vector, and depth information fusion is performed for the depth value of each perspective to adjust the weight based on depth information in the three-dimensional point set.

[0022] Further, the glue leakage defect detection model based on deep learning combines a convolutional neural network and a graph neural network; calculating a first loss function of the glue leakage defect detection model based on deep learning, and training the glue leakage defect detection model based on the first loss function includes:

[0023] Extract the geometric features of each local area through a convolutional neural network;

[0024] Take each local area as a node in the graph neural network, and the connection relationship between nodes is expressed by an adjacency matrix, where the adjacency matrix reflects the spatial distance between each three-dimensional point set in the local area;

[0025] The first loss function includes a weighted cross-entropy loss, a geometric regularization term, and a smoothness regularization term. Among them, the involved weighted cross-entropy loss is used to increase the attention to the glue leakage area, the geometric regularization term is used to maintain the geometric consistency of the surface and avoid overfitting, and the smoothness regularization term is used to reduce noise and maintain the smoothness of the prediction result;

[0026] Use the first loss function to train the glue leakage defect detection model.

[0027] Further, the conditional generative adversarial network includes a generator and a discriminator. The task of the generator is to generate a synthetic glue leakage image according to the input three-dimensional point set, and the task of the discriminator is to judge whether the image input to the generator is a real image;

[0028] Among them, the generator structure includes multiple convolutional layers and deconvolutional layers, which map low-dimensional input features to a high-dimensional image space;

[0029] The discriminator extracts the features of the image through convolutional layers and uses binary classification output to judge whether the input image is a fake image generated by the generator.

[0030] Further, calculating a second loss function of the conditional generative adversarial network, training the conditional generative adversarial network based on the second loss function, generating a realistic glue leakage image, and enhancing the training dataset of the glue leakage defect detection model includes:

[0031] Design a second loss function based on the minimax loss based on the conditional generative adversarial network;

[0032] Train the conditional generative adversarial network according to the second loss function, and introduce a conditional control mechanism. By adding three-dimensional surface features as conditional inputs to the input of the generator, more realistic glue leakage defect images can be generated; among them, the three-dimensional surface features are three-dimensional point sets, normal vectors, and depth information.

[0033] Further, the lightweight glue leakage defect detection model based on a convolutional neural network includes a convolutional layer, a pooling layer, and a fully connected layer; wherein, the convolutional layer is used to extract high-dimensional features of the input image, and through multi-layer convolutional operations, gradually extract from low-level features to high-level features; the pooling layer reduces the computational complexity through max pooling or average pooling, improves the abstraction degree of features, and extracts the key information of the image; the fully connected layer realizes image classification and outputs the defect probability value indicating the existence of a defect.

[0034] When the defect probability value exceeds the set threshold, a feedback mechanism will be automatically triggered to send an adjustment instruction to the production system:

[0035] Adjust the glue spraying amount of the production equipment;

[0036] Adjust the running speed of the production line;

[0037] Adjust the flow rate or temperature of the raw materials.

[0038] In another aspect of the present invention, a glue leakage prevention detection system based on image recognition is provided, and the system includes:

[0039] An initial image data acquisition unit, which is used to preprocess the initial image data based on multiple perspectives, perform local region image enhancement on the preprocessed image, and perform normalization processing on the image after image enhancement to obtain an enhanced image with normalized pixel values;

[0040] An image data enhancement unit, which is used to eliminate occlusion errors for each pixel point of the enhanced image by calculating its weighted average value in multiple perspectives, obtain a weighted average three-dimensional point set, calculate the normal vector of each pixel point in three-dimensional space, perform surface smoothing in combination with the curvature information of neighboring points, and perform weighted fusion on the processed pixel points to restore the three-dimensional geometric structure of the workpiece surface;

[0041] A glue leakage defect analysis unit, which is used to calculate the first loss function of the glue leakage defect detection model based on deep learning based on the weighted average three-dimensional point set and the normal vector of each pixel point in three-dimensional space, and train the glue leakage defect detection model based on the first loss function;

[0042] A glue leakage defect enhancement unit, which is used to design a conditional generative adversarial network for the three-dimensional point set optimally output by the glue leakage defect detection model, calculate the second loss function of the conditional generative adversarial network, train the conditional generative adversarial network based on the second loss function, generate realistic glue leakage images, and enhance the training dataset of the glue leakage defect detection model;

[0043] The glue leakage result output unit is used to design a lightweight glue leakage defect detection model based on a convolutional neural network according to the glue leakage defect detection model, use the image output by the glue leakage defect detection model as input, generate a defect probability value for the region based on the image, and make a judgment on the glue leakage defect based on the defect probability value and make adjustments.

[0044] The beneficial technical effects of the present invention are at least as follows:

[0045] (1) By adopting the multi-view image synthesis technology, the present invention can simultaneously collect images of the coating area from different angles by multiple cameras, register and synthesize these images, so as to generate a complete all-round coating image. This method overcomes the perspective limitation problem existing in traditional single-view cameras, ensures that the system can comprehensively obtain all information of the coating area, and avoids the problem of missed detection caused by perspective blind spots or equipment occlusion. Through the fusion of multi-angle images, the system can more accurately capture the tiny glue leakage defects in the coating process and improve the detection accuracy.

[0046] (2) The present invention introduces the generative adversarial network (GAN) as the core technology for data augmentation and defect simulation. Traditional glue leakage detection systems usually rely on fixed data sets. However, due to the diversity and complexity of glue leakage defect types, the existing training data often cannot cover all possible glue leakage situations, resulting in insufficient detection accuracy. By training the GAN model, the generator can simulate various types of glue leakage defects, such as different glue leakage shapes, sizes, distributions, etc., to enhance the diversity and comprehensiveness of the data set. The discriminator continuously optimizes the detection model to make it have stronger defect recognition ability. In the present invention, GAN is not only used to enhance the training data set, but also further improves the system's ability to identify complex glue leakage patterns, solving the problem of easy misdetection or missed detection of traditional detection systems when facing complex defects.

[0047] (3) By combining multi-view image synthesis with the generative adversarial network, the present invention forms a perfect glue leakage detection closed loop. This system can not only effectively overcome the accuracy, adaptability and data problems in the prior art, but also can real-time feedback the coating quality of the production line and perform automatic repair to ensure that the coating quality in the production process always meets the standards. This system significantly improves the robustness of the detection system, reduces manual intervention, and lowers production costs, and has broad application prospects. Description of the Drawings

[0048] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the following drawings.

[0049] Figure 1 This is a flowchart of the anti-leakage glue detection method based on image recognition according to an embodiment of the present invention.

[0050] Figure 2 This is a framework diagram of the anti-leakage glue detection system based on image recognition according to an embodiment of the present invention. Detailed implementation manners

[0051] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0052] As Figure 1 shown, the anti-leakage glue detection method based on image recognition provided by the embodiment of the present invention includes the following steps S1-S5:

[0053] S1. Preprocess the initial image data based on multiple perspectives, perform local region image enhancement on the preprocessed image, and normalize the enhanced image to obtain an enhanced image with normalized pixel values.

[0054] The core objective of this step is to improve the image quality through advanced image preprocessing techniques and provide high-quality image data for subsequent multi-view three-dimensional reconstruction, deep learning training, and leakage glue detection algorithms. In particular, the processed image should be able to minimize the interference of light changes, background noise, and physical occlusion on the workpiece surface detection, thereby improving the detection accuracy.

[0055] Specifically, since image acquisition in the industrial environment is often accompanied by problems such as light changes, sensor noise, and lens distortion, the primary task is to denoise the image. To effectively remove noise and retain the details of the workpiece, a weighted Gaussian smoothing filter (referred to as an "adaptive denoising filter") is used for denoising.

[0056] Set the image data I i , where i represents the perspective number and N is the noise. The goal of denoising is to filter out the noise N i through weighted Gaussian filtering to obtain the denoised image F i . Define the adaptive weighted filter whose core idea is to determine the filtering intensity according to the brightness and texture features of the local region:

[0057]

[0058] Among them, It is a filter adaptively selected based on local region statistics (such as local standard deviation and mean), where σ represents the noise standard deviation of the local region. The adaptive filter can dynamically adjust the denoising intensity according to the image features of different regions, avoiding the loss of image details caused by over-smoothing.

[0059] In addition, the denoising process should also consider the preservation of texture details in the image, through the texture enhancement regularization term λ T to control the degree of denoising to prevent excessive loss of image details. The overall objective can be represented by the following optimization function:

[0060]

[0061] where ∥·∥2 represents the L2 norm, represents the L1 norm of the image gradient, and λ T is the regularization term for texture preservation. This can effectively remove noise while retaining the surface details of the workpiece.

[0062] Furthermore, the images taken from multiple perspectives need to be spatially aligned to ensure that the pixel positions in the images from different perspectives correspond consistently. In particular, factors such as the physical coordinate deviation between images and the difference in the camera shooting angles will affect the subsequent 3D reconstruction. Therefore, an accurate geometric mapping method must be used to map the pixel points of each image to a unified 3D space coordinate system.

[0063] Assume that each pixel point has coordinates (p x , p y ) on the image plane, and its corresponding 3D space coordinate is P i = (x i , y i , z i ). To achieve the spatial alignment between multi-perspective images, the present invention uses the perspective projection model to map the image coordinates to the 3D space based on the calibration information of the camera. The specific mapping formula is:

[0064]

[0065] where K i is the internal parameter matrix of the camera, is the two-dimensional image coordinate, and P i is the corresponding 3D space coordinate.

[0066] Furthermore, when performing image alignment, an image registration method based on image gradient can be used to ensure the accurate positional relationship of the images from each perspective in the 3D space, reducing the errors caused by the camera angle and the image acquisition order. The optimization objective of image registration is:

[0067]

[0068] Among them is the mapped image, is the target image. The optimization goal is to minimize the error between the images and ensure that all images are accurately aligned spatially.

[0069] Furthermore, image enhancement is to improve the visibility of the surface details of the workpiece. Traditional image enhancement methods may not be able to adapt to complex industrial environments. Therefore, the present invention designs an adaptive enhancement algorithm based on local region statistics (referred to as the "local dynamic enhancement algorithm"). This algorithm takes into account the contrast, brightness distribution, and texture information of different regions in the image, and dynamically adjusts the enhancement intensity by introducing an enhancement factor based on the local region:

[0070] The present invention first calculates the brightness mean μ(x, y) and standard deviation σ(x, y) of the local region of the image, and then adjusts the brightness of the image based on these local statistics:

[0071]

[0072] where α(x, y) and β(x, y) are factors adjusted based on the local region contrast and brightness. To avoid over-saturation caused by over-enhancement, a local constraint function is defined:

[0073]

[0074] where λ E is a regularization factor that controls the enhancement intensity. Through this regularization term, it is ensured that details are not lost during the enhancement process and the local smoothness of the image is maintained.

[0075] Furthermore, in order to enable subsequent 3D reconstruction and deep learning models to efficiently process images from different perspectives, all images need to be size-normalized and pixel-normalized. To adapt to deep learning networks, the pixel values of the images will be normalized to the range [0, 1].

[0076] Assuming that the image still has different brightness ranges after enhancement, the present invention normalizes the pixel values through the following formula:

[0077]

[0078] S2. Based on each pixel point of the enhanced image, the occlusion error is eliminated by calculating the weighted average value of the pixel point in multiple perspectives, and a weighted average 3D point set is obtained. The normal vector of each pixel point in the 3D space is calculated, and the surface is smoothed by combining the curvature information of the neighboring points. The processed pixel points are weighted and fused to restore the 3D geometric structure of the workpiece surface.

[0079] The goal of this step is to restore the three-dimensional geometric structure of the workpiece surface through a three-dimensional reconstruction algorithm based on image data from multiple perspectives. To improve the reconstruction quality, especially for the accuracy of glue leakage detection, this step not only performs standard image fusion but also fuses depth information and eliminates possible geometric inaccuracies caused by occlusion errors and perspective differences.

[0080] Specifically, in multi-perspective images, due to problems such as occlusion, reflection, and surface discontinuity between different perspectives, image information in some areas may be missing, thus affecting the reconstruction result. To compensate for this missing information, the present invention introduces an error compensation algorithm based on adaptive weighted fusion.

[0081] For each pixel point P i , the present invention eliminates occlusion errors by calculating the weighted average of this pixel point in multiple perspectives, and defines the weighting coefficient w i as the visibility or credibility of this pixel point in each perspective, usually depending on the coverage of the perspective and the surface reflection characteristics. The three-dimensional point set after weighted averaging The calculation formula is:

[0082]

[0083] where w ij is the weight of this pixel point P i in the j-th perspective, and P ij is the three-dimensional coordinate point in the j-th perspective. In this way, the present invention can effectively compensate for occluded or partially missing areas.

[0084] Different from traditional image weighted fusion methods, the present invention introduces a perspective-based weighting strategy, comprehensively considering the credibility of each perspective (such as perspective overlap, reflected light influence, etc.), and then dynamically adjusting the weights. This method can significantly reduce the reconstruction error caused by perspective errors or occlusions and improve the reconstruction accuracy.

[0085] Further, to ensure that the surface of the reconstructed three-dimensional model is smooth and continuous, the present invention optimizes the reconstructed three-dimensional model by introducing a surface smoothing algorithm based on geometric features. This algorithm eliminates noise by analyzing the local geometric features (such as curvature, normal vector) of the three-dimensional model and avoids geometric defects caused by discontinuities.

[0086] First, the present invention calculates the normal vector n i of each pixel point P i in three-dimensional space and performs surface smoothing in combination with the curvature information of neighboring points. The objective function is as follows:

[0087]

[0088] where, Denote the gradient of pixel point P i as κ i and the curvature of this point as λ curvature is the curvature regularization term, which is used to balance surface smoothness and geometric accuracy. By optimizing this objective function, the surface will become smoother, effectively removing noise and ensuring the accuracy and reliability of 3D reconstruction.

[0089] This method performs surface optimization by combining curvature and normal vectors. Based on traditional smoothing methods, the present invention proposes an adaptive regularization method based on local geometric features. This method can effectively remove surface noise without affecting model details, especially at the edges and details of the workpiece, where it can better retain the true geometric shape.

[0090] Furthermore, in addition to standard multi-view image fusion, the present invention will also utilize the depth information of each view for optimization to enhance the quality of surface reconstruction. By fusing and optimizing the depth maps, the detail quality of the 3D model can be further improved, especially in the concave and convex regions and weak parts of the workpiece.

[0091] The present invention designs a depth map weighted fusion algorithm, which weights the contribution degree of each view through depth information, thereby enhancing the reconstruction accuracy of the real surface area. Assume that the depth value of each view is d i , then the weighted surface point The calculation formula is:

[0092]

[0093] Different from the direct application of traditional depth information, the present invention introduces an adaptive weighting strategy for depth maps, adjusting the weights during the calculation process according to the quality of depth values, effectively avoiding errors introduced by inaccurate depth data. Through this method, the reconstruction accuracy of the tiny details on the workpiece surface can be improved.

[0094] S3. Calculate the first loss function of the glue leakage defect detection model based on deep learning based on the weighted average 3D point set and the normal vector of each pixel point in 3D space, and train the glue leakage defect detection model based on the first loss function.

[0095] In this stage, the present invention will construct a glue leakage defect detection model based on deep learning based on the 3D points in step S2 . By deeply analyzing the optimized 3D surface, potential glue leakage defect areas are identified. The following are the detailed steps:

[0096] Specifically, the present invention first extracts features from the 3D points output in step S2 . For each point Including its three-dimensional spatial coordinate information, and the optimized surface has a certain smoothness and geometric features.

[0097] Geometric feature extraction: The present invention focuses on extracting geometric information such as the local curvature and normal vector of each point. These features can help the model identify subtle deformations on the surface, especially in the areas of glue leakage defects. The innovation lies in that the present invention combines the joint description of local curvature and normal vector, and this combination can effectively identify tiny deformations:

[0098]

[0099] Among them, represents the local second-order gradient, revealing the change of curvature. represents the gradient of the point, reflecting the change of the surface normal vector.

[0100] This combination helps to capture glue leakage defects that may not be detected by individual curvature or normal vector alone.

[0101] Depth information fusion: The depth information d of each point i will be input into the model together with the geometric features. The depth information helps to better understand the surface undulations and subtle defects. Therefore, the present invention defines a weighted depth feature:

[0102]

[0103] Among them, w depth is the weighting coefficient of the depth information. In this way, the model can adjust the degree of emphasis on the depth information according to the specific characteristics of the workpiece.

[0104] Furthermore, to improve the accuracy of glue leakage detection, a multi-scale detection model is designed by combining a convolutional neural network (CNN) and a graph neural network (GNN). This combination can simultaneously extract local details and global spatial relationships.

[0105] CNN part: Extract the geometric features of each local region R i through a convolutional neural network. The convolutional layer can learn the subtle local shape changes on the surface, which is especially suitable for detecting tiny glue leakage defects.

[0106] GNN part: To further capture the spatial relationships between regions, a graph neural network (GNN) is introduced. Each region R i is regarded as a node in the graph, and the connection relationship between nodes is expressed through an adjacency matrix A ij A ij reflects the spatial distance between point and point :

[0107]

[0108] Among them, is the Euclidean distance between two points; σ is the standard deviation of the Gaussian kernel, which controls the influence range of the neighborhood; in this way, the GNN can effectively learn the geometric continuity between different regions and enhance the spatial correlation of glue leakage detection.

[0109] Furthermore, in order to solve the problem of class imbalance in glue leakage detection and enhance the detection ability of the model for small defects, the present invention designs an innovative loss function, including weighted cross-entropy, geometric regularization term, and robustness smoothness term.

[0110] Weighted cross-entropy loss: Considering the scarcity of glue leakage regions in the data, the present invention introduces a weighted coefficient w i in the loss function to increase the attention to the glue leakage regions:

[0111]

[0112] Among them, w i is the class weight for each sample, ensuring that the glue leakage regions have a larger weight in training. L i is the true label (glue leakage / non-glue leakage) of each region. is the defect probability predicted by the model.

[0113] Geometric regularization term: In order to maintain the geometric consistency of the surface and avoid overfitting, the present invention adds a geometric regularization term. This term helps the model learn the surface changes more in line with the actual geometric structure of the workpiece and ensures that the prediction results have physical rationality:

[0114]

[0115] Among them, and are the second-order derivatives of the predicted surface and the true surface respectively, representing the change of local curvature.

[0116] This term helps the model avoid unreasonable geometric distortions and ensures the correct positioning of the glue leakage regions.

[0117] Smoothness regularization term: To reduce noise and maintain the smoothness of the prediction results, the present invention introduces a smoothness regularization term, making the prediction results more coherent and easy to generalize:

[0118]

[0119] Among them, is the gradient of the predicted surface, n iis the true normal vector of the workpiece surface. This regularization term can help the model avoid overfitting when dealing with complex three-dimensional surfaces and maintain the robustness of the prediction.

[0120] Furthermore, by combining the weighted cross-entropy, geometric regularization term, and smoothness regularization term, the final first loss function can be expressed as:

[0121]

[0122] where λ1 and λ2 are the weights of the regularization terms, adjusted by cross-validation to ensure the generalization ability of the model. Through this training method, the model can effectively detect glue leakage defects, especially when facing complex and tiny defects on the workpiece surface, with strong robustness and accuracy.

[0123] S4. For the three-dimensional point set optimized by the glue leakage defect detection model, design a conditional generative adversarial network, calculate the second loss function of the conditional generative adversarial network, and train the conditional generative adversarial network based on the second loss function to generate realistic glue leakage images and enhance the training data set of the glue leakage defect detection model.

[0124] In this step, the present invention introduces a generative adversarial network (GAN) to generate synthetic glue leakage images, aiming to enhance the training data set, especially in the case where the glue leakage defect area is small and difficult to obtain. By generating synthetic data, the diversity of glue leakage defects can be effectively increased, and the generalization ability of the model can be improved.

[0125] Specifically, input data: The input at this stage comes from step S3, that is, the three-dimensional point set with optimized geometric features output by the deep learning model. These three-dimensional points contain the position, normal vector, depth information, etc. of each surface point.

[0126] For each point its coordinates represent the three-dimensional position of the workpiece surface, n i is the normal vector, and d i is the depth information.

[0127] Objective: The present invention hopes to generate synthetic images with glue leakage defects based on these inputs. These images can simulate glue leakage defects at different angles, different sizes, and different degrees to enhance the training data set.

[0128] Furthermore, GAN architecture design:

[0129] To accurately simulate the details of glue leakage defects in the synthetic images, the present invention designs a conditional generative adversarial network (cGAN), which can generate synthetic images according to the input three-dimensional surface features when generating synthetic images. Perform conditional control (including geometric and depth information) to generate realistic glue leakage images.

[0130] Generator: The task of the generator is to generate synthetic glue leakage images based on the input 3D point set. In the present invention, after the 3D point cloud is processed by a convolutional neural network (CNN), it is used as conditional input to the generator. The structure of the generator includes multiple convolutional layers and deconvolutional layers, which map low-dimensional input features to high-dimensional image space.

[0131] The output of the generator is a synthetic image whose size is the same as the actual workpiece surface image, and the defect area therein simulates the glue leakage phenomenon.

[0132] Discriminator: The task of the discriminator is to determine whether the input image is a real image. The discriminator extracts the features of the image through convolutional layers and uses binary classification output to determine whether the input image is a fake image generated by the generator.

[0133] The input of the discriminator is a real glue leakage image or a generated image and its output is the probability indicating whether the image is a real image.

[0134] Loss function design: To train the generator and the discriminator, the present invention designs an adversarial loss function based on the min-max loss, which is suitable for the training of the generative adversarial network. For the loss of the generator, the present invention calculates it through the following formula:

[0135]

[0136] where is the probability output by the discriminator, indicating the probability that the image is a real image.

[0137] The generator improves the authenticity of the synthetic image by maximizing the misclassification probability of the discriminator for the generated image.

[0138] The loss function of the discriminator is:

[0139]

[0140] where I i is a real glue leakage image, and D(I i ) represents the judgment of the discriminator on the real image.

[0141] Furthermore, to ensure that the glue leakage images generated by the generator match the actual defects, the present invention introduces a conditional control mechanism. The generator not only relies on random noise as input but also uses three-dimensional surface features as conditional inputs to generate more realistic glue leakage defect images.

[0142] Conditional input: Each time during training, the generator takes the geometric features (such as position, normal vector, etc.) of each point in the three-dimensional point set as input. By doing so, the generator can generate glue leakage defects corresponding to the surface features of specific workpieces, enhancing the personalization and realism of the generated images.

[0143] Specifically, the input to the generator is:

[0144]

[0145] where z i is the conditional input, which contains the geometric features and depth information of the surface.

[0146] Image enhancement: To further improve the diversity of the dataset, the present invention can perform operations such as random rotation, scaling, and affine transformation on the generated images to simulate different perspectives and shooting conditions. These transformations will make the model more robust during training and capable of handling more complex actual situations.

[0147] Furthermore, the present invention optimizes the entire GAN network by alternately training the generator and the discriminator. In each training iteration, the present invention updates the parameters of the generator and the discriminator so that the generator can generate more and more realistic glue leakage images, while the discriminator becomes better and better at distinguishing between real and fake images.

[0148] Training objective: The objective of the generator is to maximize the discriminator's "real" judgment of the generated images, while the objective of the discriminator is to maximize the correct classification of real images and the correct determination of generated images. In this way, the generator and the discriminator continuously play a game, thereby generating high-quality synthetic glue leakage images.

[0149] Final second loss function: By minimizing the loss functions of the generator and the discriminator, the final optimization objective is:

[0150]

[0151] where λ reg is the weight of the regularization term, which is used to limit the geometric consistency of the generated images.

[0152] Regularization term: To ensure that the generated images conform to the geometric constraints of the workpiece, the present invention can add a regularization term to the loss function, such as by penalizing the geometric consistency between the generated images and the real images:

[0153]

[0154] Among them, and are the second-order gradients of the predicted image and the real image respectively, which are used to measure the curvature difference of the surface. Through such training and optimization, the images generated by the GAN can effectively expand the dataset of glue leakage defects and improve the recognition ability of the training model for glue leakage defects.

[0155] S5. According to the glue leakage defect detection model, design a lightweight glue leakage defect detection model based on a convolutional neural network, use the image output by the glue leakage defect detection model as input, generate the defect probability value of the region based on this image, and make a judgment on the glue leakage defect based on the defect probability value and make adjustments.

[0156] In this stage, the goal is to apply the model generated and optimized in the previous steps to real-time glue leakage defect detection, and introduce a quality feedback mechanism to achieve automated quality detection and rapid feedback, thereby optimizing the production process and workpiece quality. The system automatically detects glue leakage defects by analyzing images or three-dimensional data in real time, and optimizes production or equipment adjustment through the feedback mechanism.

[0157] Specifically, the present invention adopts a lightweight glue leakage defect detection model based on a convolutional neural network (CNN). This model extracts features in the input image through multiple convolutional operations and judges whether there is a glue leakage defect through a classification network.

[0158] Model architecture: This detection model includes the following modules:

[0159] Convolutional layer: Used to extract high-dimensional features of the input image Through multiple convolutional operations, features are gradually extracted from low-level features to high-level features.

[0160] Pooling layer: Reduces the computational complexity through max pooling or average pooling, improves the abstraction degree of features, and extracts the key information of the image.

[0161] Fully connected layer: Finally, image classification is achieved through the fully connected layer, and the probability of the existence of defects is output.

[0162] The input of the model is The output is the defect probability value of each image region Specifically, the model makes a judgment on the glue leakage defect through the following classification formula:

[0163]

[0164] Among them, is the image The detection probability of defects. W is the weight matrix of the classification layer, and b is the bias term. is the feature extracted from the image through multiple convolutional and pooling operations. σ is the Sigmoid activation function, which outputs the probability value of defects, ranging from [0, 1].

[0165] Furthermore, to ensure quality control and real-time optimization in the production process, the present invention introduces a quality feedback mechanism that automatically adjusts production equipment or process parameters according to the results of defect detection. The key to this mechanism lies in quickly processing the detection results and timely transmitting the feedback information to the production system.

[0166] Feedback strategy: When the probability of detecting glue leakage defects exceeds the set threshold θ, the system will automatically trigger the feedback mechanism and send adjustment instructions to the production system. These adjustment instructions may include:

[0167] Adjust the glue spraying amount of the production equipment;

[0168] Adjust the running speed of the production line;

[0169] Adjust parameters such as the flow rate or temperature of the raw materials.

[0170] Furthermore, the core of the feedback mechanism is to optimize production parameters based on real-time detection results and historical defect patterns. For example, when the probability of glue leakage defects for a certain workpiece category or production batch exceeds the threshold, the system will automatically generate an adjustment instruction A through the following optimization formula i :

[0171]

[0172] where A i is the feedback adjustment instruction for workpiece i, which may include equipment settings and process adjustments. is the probability of glue leakage defects in the i-th workpiece image. θ is the set threshold of the defect probability. When P defect , the feedback mechanism is triggered. N is the total number of images with detected defects.

[0173] As Figure 2 shown, in another embodiment of the present invention, a glue leakage prevention detection system based on image recognition is provided. The system includes:

[0174] An initial image data acquisition unit 501, which is used to preprocess the initial image data based on multiple perspectives, perform local area image enhancement on the preprocessed image, and perform normalization processing on the image after image enhancement to obtain an enhanced image with normalized pixel values;

[0175] The image data enhancement unit 502 is configured to, based on each pixel point of the enhanced image, eliminate occlusion errors by calculating the weighted average value of the pixel point in multiple viewpoints, obtain a weighted average three-dimensional point set, calculate the normal vector of each pixel point in the three-dimensional space, perform surface smoothing by combining the curvature information of neighboring points, perform weighted fusion on the processed pixel points, and restore the three-dimensional geometric structure of the workpiece surface;

[0176] The glue leakage defect analysis unit 503 is configured to calculate a first loss function of the glue leakage defect detection model based on deep learning based on the weighted average three-dimensional point set and the normal vector of each pixel point in the three-dimensional space, and train the glue leakage defect detection model based on the first loss function;

[0177] The glue leakage defect enhancement unit 504 is configured to design a conditional generative adversarial network for the three-dimensional point set optimally output by the glue leakage defect detection model, calculate a second loss function of the conditional generative adversarial network, train the conditional generative adversarial network based on the second loss function, generate realistic glue leakage images, and enhance the training data set of the glue leakage defect detection model;

[0178] The glue leakage result output unit 505 is configured to design a lightweight glue leakage defect detection model based on a convolutional neural network according to the glue leakage defect detection model, use the image output by the glue leakage defect detection model as input, generate a defect probability value for the region of the current image, and perform judgment and adjustment on the glue leakage defect based on the defect probability value.

[0179] In addition, for the technical details not described in detail in this embodiment, reference may be made to the parameter operation method provided in any embodiment of the present invention, which will not be elaborated here.

[0180] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or system including the element.

[0181] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0182] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a read-only memory / random access memory, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0183] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A glue leakage prevention detection method based on image recognition, characterized in that, The method includes: Preprocessing the initial image data based on multiple perspectives, performing local region image enhancement on the preprocessed image, and normalizing the enhanced image to obtain an enhanced image with normalized pixel values; Based on each pixel point of the enhanced image, eliminating occlusion errors by calculating the weighted average in multiple perspectives to obtain a weighted average three-dimensional point set, calculating the normal vector of each pixel point in three-dimensional space, performing surface smoothing by combining the local curvature information of neighboring points, and performing weighted fusion on the processed pixel points to restore the three-dimensional geometric structure of the workpiece surface; Based on the weighted average three-dimensional point set and the normal vector of each pixel point in three-dimensional space, calculating the first loss function of the glue leakage defect detection model based on deep learning, and training the glue leakage defect detection model based on the first loss function; For the three-dimensional point set optimized and output by the glue leakage defect detection model, designing a conditional generative adversarial network, calculating the second loss function of the conditional generative adversarial network, training the conditional generative adversarial network based on the second loss function to generate realistic glue leakage images, and enhancing the training data set of the glue leakage defect detection model; According to the glue leakage defect detection model, designing a lightweight glue leakage defect detection model based on a convolutional neural network, using the image output by the glue leakage defect detection model as input, generating the defect probability value of the region based on the current image, and making a judgment on the glue leakage defect based on the defect probability value and making adjustments.

2. The anti-leakage glue detection method based on image recognition according to claim 1, wherein The preprocessing uses a weighted Gaussian smoothing filter for denoising, uses a geometric mapping method to map the pixel points of each image to a unified three-dimensional space coordinate system for three-dimensional mapping, and an image registration method based on image gradient to ensure the precise positional relationship of the images in three-dimensional space for each perspective; The local region image enhancement uses an adaptive enhancement algorithm based on local region statistics, and according to the contrast, brightness distribution and texture information of different regions in the image, introducing an enhancement factor based on the local region to dynamically adjust the enhancement intensity; The normalization processing includes size standardization and pixel normalization, and the pixel normalization means that the pixel values of the image after size standardization processing will be normalized to the range of [0,1].

3. The leak-proof glue detection method based on image recognition according to claim 1, characterized in that, The weighted average is calculated by weighted averaging based on the weights of the pixel points in the current perspective to obtain a weighted average three-dimensional point set, and the weighted average in combination with multiple perspectives is used to eliminate occlusion errors.

4. The leak-proof glue detection method based on image recognition according to claim 3, characterized in that, The combining of the local curvature of neighboring points for surface smoothing and the weighted fusion of the processed pixel points to restore the three-dimensional geometric structure of the workpiece surface includes: Designing an objective function based on the gradient of each pixel point and the local curvature of the corresponding pixel point, calculating the normal vector of each pixel point in three-dimensional space, and performing surface smoothing by combining the local curvature of neighboring points; Performing weighted average calculation on each pixel point of each perspective based on the depth value of each perspective, and weighting the contribution degree of each perspective through depth information, so as to enhance the reconstruction accuracy of the real surface region.

5. The anti-leakage glue detection method based on image recognition according to any one of claims 1 to 4, characterized in that, Based on the three-dimensional point set, extract the features of local curvature and normal vector, combine the joint description of local curvature and normal vector to extract the micro-deformation features, and perform depth information fusion for the depth value of each perspective, and adjust the weight based on depth information in the three-dimensional point set.

6. The leak-proof glue detection method based on image recognition according to claim 5, wherein The glue leakage defect detection model based on deep learning combines a convolutional neural network and a graph neural network; then calculate the first loss function of the glue leakage defect detection model based on deep learning, and train the glue leakage defect detection model based on the first loss function, including: Extract the geometric features of each local area through a convolutional neural network; Take each local area as a node in the graph neural network, and the connection relationship between nodes is expressed by an adjacency matrix, where the adjacency matrix reflects the spatial distance between each three-dimensional point set in the local area; The first loss function includes a weighted cross-entropy loss, a geometric regularization term, and a smoothness regularization term. Among them, the involved weighted cross-entropy loss is used to increase the attention to the glue leakage area, the geometric regularization term is used to maintain the geometric consistency of the surface and avoid overfitting, and the smoothness regularization term is used to reduce noise and maintain the smoothness of the prediction result; Use the first loss function to train the glue leakage defect detection model.

7. The leak-proof glue detection method based on image recognition according to claim 1, characterized in that The conditional generative adversarial network includes a generator and a discriminator. The task of the generator is to generate a synthetic glue leakage image according to the input three-dimensional point set, and the task of the discriminator is to judge whether the image input to the generator is a real image; Among them, the structure of the generator includes multiple convolutional layers and deconvolutional layers, which map low-dimensional input features to high-dimensional image space; The discriminator extracts the features of the image through convolutional layers and uses binary classification output to judge whether the input image is a fake image generated by the generator.

8. The method for leak-proof glue detection based on image recognition according to claim 7, characterized in that Calculate the second loss function of the conditional generative adversarial network, and train the conditional generative adversarial network based on the second loss function to generate a realistic glue leakage image and enhance the training dataset of the glue leakage defect detection model, including: Design a second loss function based on the minimax loss based on the conditional generative adversarial network; Train the conditional generative adversarial network according to the second loss function, and introduce a conditional control mechanism. Add three-dimensional surface features as conditional inputs to the input of the generator to generate more realistic glue leakage defect images; among them, the three-dimensional surface features are three-dimensional point sets, normal vectors, and depth information.

9. The method for leak-proof glue detection based on image recognition according to claim 8, wherein The lightweight glue leakage defect detection model based on a convolutional neural network includes convolutional layers, pooling layers, and fully connected layers; among them, the convolutional layers are used to extract high-dimensional features of the input image, and through multi-layer convolutional operations, gradually extract from low-level features to high-level features; the pooling layers reduce the computational complexity through max pooling or average pooling, improve the abstraction degree of features, and extract the key information of the image; the fully connected layers implement image classification and output the defect probability value indicating the existence of defects; When the defect probability value exceeds the set threshold, a feedback mechanism will be automatically triggered to send an adjustment instruction to the production system: Adjust the glue spraying amount of the production equipment; Adjust the running speed of the production line; Adjust the flow rate or temperature of the raw materials.

10. A glue leakage prevention detection system based on image recognition, characterized in that, The system includes: An initial image data acquisition unit, which preprocesses the initial image data based on multiple perspectives, enhances the local area image based on the preprocessed image, and normalizes the enhanced image to obtain an enhanced image with normalized pixel values; An image data enhancement unit, which eliminates occlusion errors by calculating the weighted average of each pixel point in the enhanced image in multiple perspectives to obtain a weighted average three-dimensional point set, calculates the normal vector of each pixel point in three-dimensional space, performs surface smoothing in combination with the curvature information of neighboring points, and performs weighted fusion on the processed pixel points to restore the three-dimensional geometric structure of the workpiece surface; A glue leakage defect analysis unit, which calculates the first loss function of the glue leakage defect detection model based on deep learning based on the weighted average three-dimensional point set and the normal vector of each pixel point in three-dimensional space, and trains the glue leakage defect detection model based on the first loss function; A glue leakage defect enhancement unit, which designs a conditional generative adversarial network for the three-dimensional point set optimally output by the glue leakage defect detection model, calculates the second loss function of the conditional generative adversarial network, trains the conditional generative adversarial network based on the second loss function to generate realistic glue leakage images, and enhances the training data set of the glue leakage defect detection model; A glue leakage result output unit, which designs a lightweight glue leakage defect detection model based on a convolutional neural network according to the glue leakage defect detection model, uses the image output by the glue leakage defect detection model as input, generates a defect probability value for the area based on the image, and judges and adjusts the glue leakage defect based on the defect probability value.

Images