Welding spot visual inspection classification method and system

Abstract

The invention relates to the technical field of welding spot visual inspection and identification, in particular to a welding spot visual inspection and classification method and system. The invention discloses a welding spot visual inspection classification method which comprises the following steps: acquiring original image data, and preprocessing to obtain standard image data; geometric features of the welding spots are extracted through an edge detection algorithm, and a data set is constructed based on standard image data and the geometric features; constructing a neural network model, performing feature extraction on the standard image data, structuring geometric features, fusing the extracted features and the structured geometric features to form feature vectors, and classifying the welding spots; training a neural network model based on the data set; and detecting and classifying the welding spots through the trained neural network model. The method has the advantages that the welding spots are comprehensively and accurately classified, the welding spots are detected in real time, the welding spot defects are automatically and comprehensively detected and classified, the detection and classification precision is high, the method adapts to complex light reflection conditions of the surfaces of the welding spots, the false detection probability and the missed detection probability are reduced, and robustness is good.

Description

Technical Field

[0001] The present invention relates to the technical field of visual inspection and recognition of solder joints, and in particular to a method and system for visual inspection and classification of solder joints. Background Art

[0002] Flat wire motors are widely used in high-end applications such as new energy vehicles. However, the excellent performance and reliable operation of flat wire motors are deeply dependent on the quality of their welds. In order to ensure a firm connection between the coil and the stator, high-precision processes such as laser welding or resistance welding are usually used.

[0003] In the actual production process, due to the diversity of material properties, the volatility of the processing environment and the many challenges of equipment stability, solder joints will inevitably have a series of defects such as cold solder joints, leaking solder joints, blown solder joints, and flying solder joints. These defects not only significantly reduce the mechanical strength of the solder joints, but may also gradually lead to the decline of electrical performance. In severe cases, they may even induce short circuits or cause failures. Therefore, efficient and accurate quality inspection of solder joints has become a crucial part of ensuring the reliability of flat wire motors.

[0004] The traditional manual inspection method relies on the operator's naked eye observation to evaluate the quality of solder joints. This method has many disadvantages that are difficult to overcome. First, its inspection efficiency is low and it is difficult to adapt to the fast-paced needs of mass production in modern manufacturing. Second, manual inspection is highly subjective, and different inspectors may have different identification standards for solder joint defects, which seriously affects the reliability and consistency of the inspection results. Third, operators are prone to fatigue after long hours of solder joint inspection, which further increases the risk of false detection.

[0005] In order to improve this situation, some production lines have adopted basic sensor detection devices, such as using infrared sensors to detect the temperature of solder joints, and using laser ranging sensors to measure the height of solder joints. Although these methods can provide certain quantitative indicators, their detection range and accuracy are still greatly limited because the sensors cannot capture the subtle shapes of solder joints. Specifically, for complex defects such as blown welds and flying welds, simple sensor detection methods are often unable to effectively identify them; at the same time, for flat wire motors of different specifications or solder joint shapes, the sensor settings need to be adjusted frequently, which greatly affects production efficiency.

[0006] At present, although some companies have tried to introduce machine vision technology to perform solder joint inspection, most of the existing visual systems are still at the level of surface inspection, and can only identify simple solder joint locations or appearance problems, and cannot comprehensively and accurately classify various complex defects of solder joints. At the same time, the algorithm accuracy of the existing visual system needs to be further improved, especially when the solder joint surface has high gloss and there is reflective interference, the false detection rate is still high. Summary of the invention

[0007] In view of the problem that the existing technology cannot comprehensively and accurately detect and classify various complex defects of flat wire motor solder joints, especially under the interference of reflection, the false detection rate is high. The present invention proposes a solder joint visual detection and classification method and system, which collects the original image data containing solder joint reflection on the production line in real time, pre-processes the original image data to obtain standard image data, obtains the geometric features of the solder joint through the edge detection algorithm, and extracts the depth features of the standard image data through the constructed neural network model, structures the geometric features, and fuses the extracted features with the structured geometric features to form a feature vector, so as to comprehensively and accurately classify the solder joints, and has the advantages of real-time detection of solder joints, automatic and comprehensive detection and classification of solder joint defects, high detection and classification accuracy, adapting to the complex reflection conditions on the solder joint surface, reducing the probability of false detection and missed detection, and good robustness.

[0008] The present invention discloses a method for visual inspection and classification of solder joints, comprising the following steps:

[0009] Acquire the original image data containing the reflection of the welding point, and perform preprocessing to obtain the standard image data;

[0010] Extracting geometric features of solder joints in the standard image data by an edge detection algorithm, and constructing a labeled data set based on the standard image data and the corresponding geometric features;

[0011] Constructing a neural network model for solder joint detection and classification, wherein the neural network model is used to perform deep feature extraction on the standard image data, and to structure geometric features corresponding to the standard image data, and to fuse the extracted features with the structured geometric features to form a feature vector, and to classify solder joints based on the feature vector;

[0012] Training the neural network model based on the labeled data set;

[0013] The weld points are detected and classified using the trained neural network model.

[0014] In a preferred embodiment, the obtaining of the original image data containing the reflection of the welding point specifically comprises the following steps:

[0015] Based on the reflective characteristics of the solder joints, the angle and brightness of the red, green and blue LED lights are dynamically adjusted through coding;

[0016] The industrial camera is controlled to take pictures of each welding spot on the assembly line one by one to obtain the original image data including the reflection of the welding spot.

[0017] In a preferred embodiment, the preprocessing to obtain standard image data specifically includes the following steps:

[0018] Setting a Roi detection area in the original image data, wherein the Roi detection area includes a welding point and its edge;

[0019] Clipping the original image data based on the Roi detection area to obtain welding point image data;

[0020] The solder joint image data is subjected to image enhancement processing to obtain the standard image data.

[0021] In a preferred embodiment, the image enhancement process includes an image grayscale operation, an image dilation operation or a Gaussian filter process, and the image grayscale operation is implemented by the following calculation method:

[0022] ,

[0023] in, Represents the grayscale value at the (x, y) position in the grayscale image, Represents the pixel value of the red channel at position (x,y) in the color image, Represents the red RGB weight coefficient, Represents the pixel value of the green channel at the (x,y) position in the color image, represents the green RGB weight coefficient, Represents the pixel value of the blue channel at position (x,y) in the color image, Represents the blue RGB weight coefficient;

[0024] The image expansion operation is implemented using the following calculation method:

[0025] ,

[0026] in, Represents the pixel value at the (x,y) position in the input image data, Represents a structural element, Represents structural elements The offset of each element in Represents the image after dilation The pixel value at position (x,y) in the image;

[0027] The Gaussian filtering process is implemented by the following calculation method:

[0028] ,

[0029] in, Represents the pixel value at the (x,y) position in the input image data, is the weight of the Gaussian kernel at position (i, j), k represents the radius of the Gaussian filter, The new pixel value after filtering.

[0030] In a preferred embodiment, the step of extracting geometric features of solder joints in the standard image data by using an edge detection algorithm specifically comprises the following steps:

[0031] Obtain edge points of welding spots in the standard image data by using a Canny edge detection algorithm;

[0032] Obtain circle parameters that best match the edge point distribution through Hough circle transform, wherein the circle parameters include circle center coordinates (a, b) and circle radius r;

[0033] Based on the circle parameters and the edge point distribution, geometric features of the solder joint are extracted.

[0034] In a preferred embodiment, the geometric features of the solder joint include edge continuity, solder joint area S, solder joint perimeter C and circularity e;

[0035] The edge continuity of the solder joint is judged by the edge point distribution. If the edge point distribution is continuous, the solder joint edge is judged to be continuous; otherwise, the solder joint edge is judged to be discontinuous.

[0036] According to the edge point distribution, the number of pixel points in the edge is calculated as the welding point area S;

[0037] Calculate the circumference C of the welding point by the radius of the circle parameter;

[0038] The circularity e is calculated by the solder joint area S and the solder joint perimeter C, specifically using the following calculation method:

[0039] .

[0040] In a preferred embodiment, the step of constructing a labeled data set based on the standard image data and corresponding geometric features comprises the following steps:

[0041] Based on the continuity of the edge points, the state of the solder joint is judged. If the edge of the solder joint is discontinuous, a label of a false solder joint or a leak solder joint is established using the corresponding standard image data and geometric features;

[0042] If the edge of the weld is continuous, the weld state is judged by the weld area S. If the weld area S is outside the standard weld area parameter range, a label of blown weld is established with the corresponding standard image data and geometric features;

[0043] If the solder joint area S is within the standard solder joint area parameter range, the solder joint state is judged by the circularity e; if the circularity e is outside the standard solder joint circularity parameter range, a label of flying soldering is established with the corresponding standard image data and geometric features;

[0044] If the circularity e is within the standard welding spot circularity parameter range, a normal label is established for the corresponding standard image data and geometric features.

[0045] In a preferred embodiment, the neural network model at least includes:

[0046] The feature extraction layer includes several convolutional layers for performing deep feature extraction on the standard image data, which is specifically implemented by the following calculation method:

[0047] ,

[0048] in, Represents the value of channel k at position (i, j) of the extracted feature. represents the pixel value of channel c at position (i+m,j+n) of the input image, i and j represent the spatial positions of the extracted features, and k represents the output channel index corresponding to an output channel. represents the weight of the convolution kernel, m, n represent the spatial size of the convolution kernel, c is the channel index of the convolution kernel corresponding to the input image, k is the index of the convolution kernel, representing the kth convolution kernel, is the bias, used to adjust the convolution result, represents the activation function;

[0049] The fully connected layer is used to structure the geometric features corresponding to the standard image data, and is specifically implemented by the following calculation method:

[0050] ,

[0051] in, represents the output of layer l, represents the activation function, represents the weight matrix, Indicates bias;

[0052] A feature fusion layer, the input end of which is connected to the feature extraction layer and the output end of the fully connected layer, and is used to fuse the features extracted by the feature fusion layer and the geometric features structured by the fully connected layer to form a feature vector;

[0053] The feature classification layer is used to generate a category probability distribution based on the feature vector through a Softmax activation function, thereby classifying the solder joints.

[0054] The present invention also provides a system using any of the above-mentioned solder joint visual inspection and classification methods, characterized in that it at least comprises:

[0055] An image acquisition module, the image acquisition module at least includes an industrial camera for real-time acquisition of original image data of dynamic solder joints in a production line, and a light source system adapted to the industrial camera for changing the reflective properties of the solder joints, the light source system at least including red, green and blue LED lights;

[0056] An image preprocessing module, used to preprocess the original image data of the welding spot to obtain standard image data, extract the geometric features of the welding spot in the standard image data by an edge detection algorithm, and construct a labeled data set based on the standard image data and the corresponding geometric features;

[0057] A model building module, used to build a neural network model for solder joint detection and classification, and train the neural network model through the labeled data set to obtain a trained neural network model;

[0058] The classification module is used to detect and classify solder joints using the trained neural network model.

[0059] A preferred embodiment further includes:

[0060] The control module is used to control the industrial camera to track the position of dynamic welding points in the production line in real time, control the angle and brightness of the red, green and blue LED lights in the light source system, and generate a test report based on the classification result of the classification module and send it to the production control system of the production line.

[0061] Compared with the prior art, the solder joint visual inspection and classification method and system disclosed in the present invention have the following beneficial effects:

[0062] (1) The present invention discloses a method and system for visual inspection and classification of solder joints, which obtains raw image data containing solder joint reflections and performs preprocessing to obtain standard image data. The raw image data contains complex reflection information on the solder joint surface. The use of raw image data containing reflection information of solder joints can enhance the generalization ability of the neural network model, and can effectively remove reflection interference and improve the accuracy of solder joint detection. The trained neural network model can not only process solder joint images under normal lighting conditions, but also adapt to solder joint images under complex reflection conditions, thereby improving the applicability and reliability of the model. The geometric features of solder joints in the standard image data are extracted through an edge detection algorithm, providing a reliable basis for subsequent data analysis and solder joint detection, and a labeled data set is constructed based on the standard image data and the corresponding geometric features. The data set not only contains the visual appearance information of the solder joint, but also incorporates its geometric properties such as shape, size, and position. This fusion of multi-dimensional information provides a solid foundation for comprehensive analysis and accurate classification of solder joints. The dataset with comprehensive multi-dimensional information can train a more generalized and robust neural network model for solder joint detection and classification. Such a neural network model can not only process solder joint images under normal conditions, but also maintain stable performance under complex conditions such as illumination changes and noise interference, thereby improving the accuracy and reliability of solder joint detection. The constructed neural network model is used to extract deep features from standard image data, structure the geometric features corresponding to the standard image data, and fuse the extracted features with the structured geometric features to form a feature vector, and classify solder joints based on the feature vector. On the one hand, deep feature extraction can automatically extract complex and abstract features from image data, and on the other hand, it can capture subtle differences in solder joint images, such as tiny defects and shape changes. These features are crucial for accurately classifying solder joints. Through deep feature extraction, the neural network model can more accurately identify different types of solder joints. On the one hand, structuring geometric features can clarify key information such as the shape, size, and position of solder joints. These geometric features are of great significance for understanding the physical properties of solder joints and judging their quality. On the other hand, they enhance the interpretability of the model. Through structured geometric features, the decision-making process of the model is more transparent, which helps to understand why the model makes a specific classification, which is very helpful for quality control and troubleshooting in industrial applications.

[0063] (2) The present invention discloses a method and system for visual inspection and classification of solder joints. Based on the continuity of edge points, the solder joint status is judged. If the solder joint edge is discontinuous, the corresponding standard image data and geometric features are used to establish a label of false solder joint or leak solder joint. If the solder joint edge is continuous, the solder joint status is judged by the solder joint area S. If the solder joint area S is outside the standard solder joint area parameter range, the corresponding standard image data and geometric features are used to establish a label of blown solder joint. If the solder joint area S is within the standard solder joint area parameter range, the solder joint status is judged by the circularity e. If the circularity e is outside the standard solder joint circularity parameter range, the corresponding standard image data and geometric features are used to establish a label of flying solder joint. If the circularity e is within the standard solder joint circularity parameter range, the corresponding standard image data and geometric features are used to establish a normal label. On the one hand, not only the intuitive feature of the continuity of the solder joint edge is considered, but also more detailed geometric features such as solder joint area and circularity are introduced. This multi-dimensional feature parameter analysis makes the judgment of the solder joint status more comprehensive and accurate. By comprehensively considering multiple feature parameters, it can more comprehensively reflect the quality and status of solder joints, and reduce misjudgments or missed judgments caused by errors in the judgment of a single feature parameter; on the other hand, it can automatically establish corresponding labels for standard image data and geometric features based on the judgment results of the solder joint status. This automated label establishment process greatly improves the efficiency and accuracy of data processing. Through automated label establishment, this method can generate a large number of data sets with accurate labels, providing rich sample resources for the subsequent training of solder joint detection and classification models. The feature parameters and judgment criteria can be adjusted and optimized according to actual needs, and the parameter ranges such as the standard solder joint area and circularity can be adjusted according to different welding processes and materials. It can be extended to other types of solder joint detection and classification tasks, and only the feature parameters and judgment criteria need to be adjusted according to specific needs.

[0064] (3) The present invention discloses a method and system for visual inspection and classification of weld spots, which at least include an image acquisition module, an image preprocessing module, a model building module, a classification module and a control module. The control module is used to generate an inspection report based on the classification result of the classification module and send it to the production control system of the production line. On the one hand, it adapts to the high-speed beat of modern production lines and completes real-time online inspection; on the other hand, it dynamically adjusts welding process parameters through data analysis and process feedback to achieve closed-loop control of the welding process, thereby helping to improve production quality and trace faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of a flow chart of an embodiment of a solder joint visual inspection method and system of the present invention; Figure 2 A normal solder joint of an embodiment of a solder joint visual inspection method and system of the present invention includes original image data of solder joint reflection; Figure 3 The original image data of the unsoldered solder joints including the solder joint reflections of the embodiments of the solder joint visual inspection method and system of the present invention; Figure 4 A cold solder joint of an embodiment of a solder joint visual inspection method and system of the present invention includes original image data of solder joint reflection; Figure 5 The flying solder joints of the embodiments of the solder joint visual inspection method and system of the present invention include original image data of solder joint reflection; Figure 6 A blown weld, which is an embodiment of a weld visual inspection method and system of the present invention, includes original image data of weld reflection. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0067] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "or / and" used herein includes any and all combinations of one or more related listed items.

[0069] In addition, in the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes, and do not specifically refer to the order or sequence, nor are they used to limit the present invention. They are only used to distinguish components or operations described with the same technical terms, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0070] A solder joint visual inspection classification method of this embodiment is as follows: Figure 1 As shown, at least the following steps are included:

[0071] Step S1, obtaining the original image data containing the reflection of the welding point, and performing preprocessing to obtain the standard image data. Figure 2-Figure 6 As shown, Figure 2 The original image data of normal solder joints including solder joint reflections; Figure 3 The original image data of the unwelded solder joints including the solder joint reflections; Figure 4 The original image data of the cold solder joint including the reflection of the solder joint; Figure 5 The original image data of the flying solder joints including the solder joint reflection; Figure 6 The original image data of the fried weld includes the reflection of the weld.

[0072] The original image data contains complex reflective information on the surface of the solder joint. Using the original image data containing reflective information of the solder joint can enhance the generalization ability of the neural network model, effectively remove reflective interference, and improve the accuracy of solder joint detection. The trained neural network model can not only process solder joint images under normal lighting conditions, but also adapt to solder joint images under complex reflective conditions, improving the applicability and reliability of the model. The geometric features of the solder joints in the standard image data are extracted through the edge detection algorithm, providing a reliable basis for subsequent data analysis and solder joint detection. Based on the standard image data and the corresponding geometric features, a labeled data set is constructed. The data set not only contains the visual appearance information of the solder joint, but also incorporates its geometric properties such as shape, size, and position. This fusion of multi-dimensional information provides a solid foundation for the comprehensive analysis and accurate classification of solder joints. The data set with comprehensive multi-dimensional information can train a more generalized and robust solder joint detection classification neural network model. Such a neural network model can not only process solder joint images under normal conditions, but also maintain stable performance under complex conditions such as lighting changes and noise interference, improving the accuracy and reliability of solder joint detection.

[0073] The specific steps include:

[0074] Step S101, based on the reflective properties of the solder joints, the angles and brightness of the three color LED lights of red, green and blue are dynamically adjusted by coding, and the number of LED lights of each color is at least 1. Since solder joints may have different reflective properties due to factors such as materials, surface conditions, shapes, etc., dynamically adjusting the lighting can ensure that each solder joint can be photographed under optimal lighting conditions, thereby improving the quality and clarity of the image. The use of LED lights of multiple colors also allows color analysis, which helps to distinguish the difference between the solder joint and its surrounding environment, further enhancing the accuracy of detection.

[0075] Step S102, controlling the industrial camera to photograph each solder joint on the assembly line one by one, and obtaining the original image data including the reflection of the solder joint. On the one hand, this automated process greatly improves the detection efficiency, reduces the need for manual intervention, reduces labor costs, and speeds up the operation of the production line; on the other hand, it ensures that each solder joint can be photographed evenly and consistently, improving the consistency and reliability of detection.

[0076] Step S103, setting a Roi detection area in the original image data, the Roi detection area includes the solder joint and its edge, so that the subsequent image processing is more focused and efficient, and by limiting the processing area, the interference of irrelevant information is reduced, and the accuracy and speed of image processing are improved.

[0077] Step S104, the original image data is clipped based on the Roi detection area to obtain solder joint image data. The clipped image data is more compact and contains key information of the solder joint, which is convenient for subsequent image analysis and processing.

[0078] Step S105, performing image enhancement processing on the solder joint image data to obtain standard image data. The image enhancement processing includes image graying operation, image dilation operation or Gaussian filtering processing.

[0079] In this embodiment, the standard image data is obtained by sequentially performing image graying operation, image dilation operation and Gaussian filtering. First, the image graying operation is performed on the solder joint image data, and the weighted sum of the red, green and blue channel values ​​of each pixel in the solder joint image data is calculated by the following method:

[0080] ,

[0081] in, Represents the grayscale value at the (x, y) position in the grayscale image, Represents the pixel value of the red channel at position (x,y) in the color image, Represents the red RGB weight coefficient, Represents the pixel value of the green channel at the (x,y) position in the color image, represents the green RGB weight coefficient, Represents the pixel value of the blue channel at position (x,y) in the color image, Represents the blue RGB weight coefficient. The solder joint image data is converted into a grayscale image through the image grayscale operation, which simplifies the computational complexity of the subsequent image processing algorithm, reduces the storage and transmission costs, improves the detection speed, and is conducive to real-time detection and classification of solder joints.

[0082] The image dilation operation is performed on the grayscale image after the image grayscale operation, and the following calculation method is used to achieve it:

[0083] ,

[0084] in, Represents the pixel value at the (x,y) position in the input image data, Represents a structural element, Represents structural elements The offset of each element in Represents the image after dilation The pixel value at the (x, y) position in the image. The image dilation operation can effectively expand the boundaries of the brighter areas (such as solder joints) in the image, making the edges of the solder joints clearer and more obvious. At the same time, since the dilation operation selects the maximum grayscale value in the neighborhood as the result, it can also remove or weaken the influence of the background on the contour and features of the solder joint to a certain extent.

[0085] The image after the image dilation operation is subjected to Gaussian filtering to obtain standard image data, which is achieved using the following calculation method:

[0086] ,

[0087] in, Represents the pixel value at the (x,y) position in the input image data, is the weight of the Gaussian kernel at position (i, j), k represents the radius of the Gaussian filter, The new pixel value after filtering. Through the Gaussian filtering operation, high-frequency noise in the image, such as Gaussian noise, salt and pepper noise, etc., is removed, and the image effect is enhanced by removing noise on details and edges, which is convenient for the following edge detection operation.

[0088] Step S2, extracting the geometric features of the solder joints in the standard image data through the edge detection algorithm, and constructing a labeled data set based on the standard image data and the corresponding geometric features. The data set not only contains the visual appearance information of the solder joints, but also incorporates their geometric properties such as shape, size, and position. This fusion of multi-dimensional information provides a solid foundation for the comprehensive analysis and accurate classification of solder joints. The data set with comprehensive multi-dimensional information can train a more generalized and robust solder joint detection classification neural network model. Such a neural network model can not only process solder joint images under normal conditions, but also maintain stable performance under complex conditions such as lighting changes and noise interference, thereby improving the accuracy and reliability of solder joint detection.

[0089] The specific steps include:

[0090] Step S201, using the Canny edge detection algorithm, obtain the edge points of the solder joints in the standard image data. The Sobel operator is used to respectively calculate the derivatives of the standard image data in the x direction and the y direction, and the convolution kernel corresponding to the vertical Sobel operator in the x direction is implemented by the following calculation method:

[0091] ,

[0092] The convolution kernel corresponding to the horizontal Sobel operator in the y direction is implemented using the following calculation method:

[0093] ,

[0094] The edge point strength is represented by the gradient amplitude, which is calculated as follows:

[0095] ,

[0096] The direction of the edge point is represented by the direction of the gradient, which is calculated using the following method:

[0097] ,

[0098] in, represents the magnitude of the gradient, represents the derivative of the standard image data in the x direction, represents the derivative of the standard image data in the y direction, Represents the gradient direction. Through the convolution operation, the gradient G of the standard image data in the x-axis and y-axis directions can be extracted. x and G y, and then calculate the edge strength and direction of each pixel. In the generated gradient amplitude image, the non-maximum suppression technology is used to retain the local gradient maximum and suppress the non-maximum, so as to achieve the refined processing of the edge. By setting two thresholds: one is a high threshold and the other is a low threshold. For points whose pixel values ​​exceed the high threshold, they are judged as strong edges; while points whose pixel values ​​are lower than the low threshold are eliminated and are not considered edge points. For those points whose pixel values ​​are between the two thresholds, if they are connected to the strong edge, we judge them as weak edges. Finally, according to the areas where the grayscale value changes more drastically in the image, the edge points of the solder joints are accurately located and extracted. Although the Canny algorithm involves multiple steps, its computational complexity is relatively small, which is suitable for real-time edge detection tasks and can quickly process large amounts of image data. Moreover, by adjusting the parameters, the influence of noise and pseudo-edges can be reduced while maintaining edge accuracy, thereby improving the performance of edge detection.

[0099] Step S202, obtain the circle parameters that best match the edge point distribution through Hough circle transform, the circle parameters include the coordinates of the circle center (a, b), and the circle radius r. By defining a three-dimensional accumulator array to store the votes for each possible combination of circle center and radius, for each edge point, traverse the possible radius values, calculate the possible circle center coordinates (a, b), and vote for the corresponding position in the accumulator. Specifically, for each edge point (x i ,y i ) and radius r, calculate the possible center coordinates (a, b) that satisfy:

[0100] ,

[0101] Increase the number of votes for (a, b, r) in the accumulator. The (a, b, r) with the highest number of votes is used as the circle parameters that best fit the distribution of edge points.

[0102] Step S203: extracting the geometric features of the solder joint based on the circle parameters and the edge point distribution.

[0103] The geometric features of the solder joint include edge continuity, solder joint area S, solder joint perimeter C and circularity e. The edge continuity of the solder joint is judged by the edge point distribution. If the edge point distribution is continuous, the solder joint edge is judged to be continuous, otherwise the solder joint edge is judged to be discontinuous.

[0104] Through the edge point distribution, the number of pixels within the edge is calculated as the soldering point area S;

[0105] Calculate the circumference C of the solder joint through the radius of the circle parameter;

[0106] The circularity e is calculated by the solder joint area S and solder joint perimeter C. The specific calculation method is as follows:

[0107] .

[0108] Step S204, judging the state of the solder joint based on the continuity of the edge points, if the solder joint edge is discontinuous, creating a label of a false solder joint or a leak solder joint using the corresponding standard image data and geometric features;

[0109] If the edge of the weld is continuous, the weld state is judged by the weld area S. If the weld area S is outside the standard weld area parameter range, the corresponding standard image data and geometric features are used to create a label of weld explosion.

[0110] If the solder joint area S is within the standard solder joint area parameter range, the solder joint state is judged by the circularity e. If the circularity e is outside the standard solder joint circularity parameter range, the corresponding standard image data and geometric features are used to create a label for flying soldering.

[0111] If the circularity e is within the range of the standard solder joint circularity parameter, the corresponding standard image data and geometric features are labeled normally. On the one hand, not only the intuitive feature of the continuity of the solder joint edge is considered, but also more detailed geometric features such as the solder joint area and circularity are introduced. This multi-dimensional feature parameter analysis makes the judgment of the solder joint state more comprehensive and accurate. By comprehensively considering multiple feature parameters, the quality and state of the solder joint can be more comprehensively reflected, and the misjudgment or missed judgment caused by the misjudgment of a single feature parameter can be reduced; on the other hand, according to the judgment result of the solder joint state, the corresponding labels can be automatically established for the standard image data and geometric features. This automated label establishment process greatly improves the efficiency and accuracy of data processing. Through automated label establishment, this method can generate a large number of data sets with accurate labels, providing rich sample resources for the training of subsequent solder joint detection classification models. Among them, the feature parameters and judgment criteria can be adjusted and optimized according to actual needs, and the parameter ranges such as the standard solder joint area and circularity can be adjusted according to different welding processes and materials. It can be extended to other types of solder joint detection and classification tasks, and only the feature parameters and judgment criteria need to be adjusted according to specific needs.

[0112] Step S3, construct a neural network model for solder joint detection and classification. The neural network model is used to extract deep features from standard image data, and to structure the geometric features corresponding to the standard image data, and to fuse the extracted features with the structured geometric features to form a feature vector, and to classify solder joints based on the feature vector. Deep feature extraction can automatically extract complex and abstract features from image data on the one hand, and can capture subtle differences in solder joint images, such as tiny defects and shape changes on the other hand. These features are crucial for accurate classification of solder joints. Through deep feature extraction, the neural network model can more accurately identify different types of solder joints. Structuring geometric features can clarify key information such as the shape, size, and position of solder joints on the one hand. These geometric features are of great significance for understanding the physical properties of solder joints and judging their quality, and on the other hand, enhance the interpretability of the model. Through structured geometric features, the decision-making process of the model is more transparent, which helps to understand why the model makes a specific classification, which is very helpful for quality control and troubleshooting in industrial applications.

[0113] In this embodiment, the neural network model at least includes:

[0114] Feature extraction layer, the feature extraction layer includes several convolutional layers for deep feature extraction of standard image data, which is implemented by the following calculation method:

[0115] ,

[0116] in, Represents the value of channel k at position (i, j) of the extracted feature. represents the pixel value of channel c at position (i+m,j+n) of the input image, i and j represent the spatial position (height and width) of the extracted features, k represents the output channel index, and each convolution kernel corresponds to an output channel. represents the weight of the convolution kernel, m, n represent the spatial size of the convolution kernel, c is the channel index of the convolution kernel corresponding to the input image, k is the index of the convolution kernel, representing the kth convolution kernel, Bias is used to adjust the convolution results and provide additional flexibility. Represents the activation function, which is used to introduce nonlinear characteristics;

[0117] Fully connected layer: The fully connected layer is used to structure the geometric features corresponding to the standard image data. The specific calculation method is as follows:

[0118] ,

[0119] in, represents the output of layer l, represents the activation function, represents the weight matrix, Indicates bias;

[0120] A feature fusion layer, the input end of the feature fusion layer is connected to the feature extraction layer and the output end of the fully connected layer, and is used to fuse the features extracted by the feature fusion layer and the geometric features structured by the fully connected layer to form a feature vector;

[0121] Feature classification layer, the feature classification layer is used to generate category probability distribution based on feature vectors through Softmax activation function, so as to classify solder joints. Softmax activation function uses the following calculation method to generate category probability distribution:

[0122] ,

[0123] in, represents the kth element of the eigenvector, Represents the kth element of the output vector, that is, the classification probability of the kth category.

[0124] In this embodiment, the features extracted by the feature fusion layer and the geometric features structured by the fully connected layer are fused using the following calculation method:

[0125] ,

[0126] in, represents the feature vector, represents the features extracted by the feature fusion layer, Represents the structured geometric features of the fully connected layer.

[0127] Step S4: training the neural network model based on the labeled data set.

[0128] In this embodiment, the data set is divided into a training set and a test set. The neural network model uses a cross entropy loss function to perform classification training in the training set, calculates the difference between the predicted probability of the neural network model and the true label encoding, and optimizes the neural network model parameters through a gradient descent algorithm and an Adam optimizer.

[0129] The trained neural network model is used to make predictions on the test set, and the performance of the model is evaluated by the classification accuracy, precision, recall, and F1 score indicators.

[0130] Step S5, detecting and classifying the solder joints through the trained neural network model. In order to apply this technology to actual production, it is necessary to integrate the trained neural network model into a complete solder joint detection system. This system is not only responsible for capturing the image of the solder joint, but also preprocessing the image, inputting it into the neural network model, and accurately interpreting the output of the model to achieve accurate classification and detection of the solder joint. By integrating the neural network model into the solder joint detection system, automated and accurate detection and classification of solder joints can be achieved. This system not only improves detection efficiency and accuracy, but also reduces the cost and risk of manual detection.

[0131] This embodiment also provides a system using any of the above-mentioned solder joint visual inspection and classification methods, comprising at least:

[0132] An image acquisition module, the image acquisition module at least includes an industrial camera for real-time acquisition of original image data of dynamic solder joints in the production line, and a light source system adapted to the industrial camera for changing the reflective properties of the solder joints, the light source system including at least red, green and blue LED lights;

[0133] An image preprocessing module is used to preprocess the original image data of the solder joint to obtain standard image data, extract the geometric features of the solder joint in the standard image data through an edge detection algorithm, and construct a labeled data set based on the standard image data and the corresponding geometric features;

[0134] A model building module, used to build a neural network model for solder joint detection classification, and train the neural network model through the labeled data set to obtain a trained neural network model;

[0135] The classification module is used to detect and classify solder joints through the trained neural network model.

[0136] Preferably, the solder joint visual inspection and classification system further comprises:

[0137] The control module is used to control the industrial camera to track the position of dynamic welding points in the production line in real time, control the angle and brightness of the red, green and blue LED lights in the light source system, and generate a test report based on the classification results of the classification module and send it to the production control system of the production line. On the one hand, it adapts to the high-speed beat of modern production lines and completes real-time online detection; on the other hand, it dynamically adjusts welding process parameters through data analysis and process feedback, realizes closed-loop control of the welding process, and helps improve production quality and trace faults.

[0138] The embodiment shown in the present invention, by adopting advanced deep learning algorithms, has a detection accuracy rate of more than 98%, and can accurately and correctly identify a variety of solder joint defects to ensure product quality. Combined with the optimized application of image enhancement technology and intelligent light sources, the false detection rate is significantly reduced in complex optical environments, and the reliability of detection is improved. The detection process of each solder joint is fast and efficient, taking no more than 0.5 seconds, which fully meets the needs of modern production lines for high-tempo production. In addition, the automated solder joint visual inspection and classification system has excellent compatibility, can easily and seamlessly connect with a variety of production equipment, and can be quickly deployed, bringing a plug-and-play convenient experience to the production line.

[0139] The test results are fed back to the production line in real time through the production control system (MES system) of the production line, so that the welding process parameters can be dynamically adjusted according to the actual test situation, further optimizing the production process. The system also comprehensively records all test data, providing strong support for traceability and analysis, helping enterprises to continuously improve production efficiency and product quality.

[0140] It is worth mentioning that the solder joint visual inspection and classification system is highly adaptable and can be flexibly adjusted according to different types of welding processes and solder joint specifications. The modular design provides unlimited possibilities for system expansion. For example, it can easily adapt to the solder joint inspection of other types of motors to meet the diverse production needs of enterprises.

[0141] In short, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A solder joint visual inspection classification method, characterized in that: At least the following steps are included: Acquire the original image data containing the reflection of the welding point, and perform preprocessing to obtain the standard image data; Extracting geometric features of solder joints in the standard image data by an edge detection algorithm, and constructing a labeled data set based on the standard image data and the corresponding geometric features; Constructing a neural network model for solder joint detection and classification, wherein the neural network model is used to perform deep feature extraction on the standard image data, and to structure geometric features corresponding to the standard image data, and to fuse the extracted features with the structured geometric features to form a feature vector, and to classify solder joints based on the feature vector; Training the neural network model based on the labeled data set; The weld points are detected and classified using the trained neural network model.

2. A solder joint visual inspection and classification method according to claim 1, characterized in that: The obtaining of the original image data containing the reflection of the welding point specifically comprises the following steps: Based on the reflective characteristics of the solder joints, the angle and brightness of the red, green and blue LED lights are dynamically adjusted through coding; The industrial camera is controlled to take pictures of each welding spot on the assembly line one by one to obtain the original image data including the reflection of the welding spot.

3. A solder joint visual inspection and classification method according to claim 1 or 2, characterized in that: The preprocessing to obtain standard image data specifically includes the following steps: Setting a Roi detection area in the original image data, wherein the Roi detection area includes a welding point and its edge; Clipping the original image data based on the Roi detection area to obtain welding point image data; The solder joint image data is subjected to image enhancement processing to obtain the standard image data.

4. A solder joint visual inspection and classification method according to claim 3, characterized in that: The image enhancement process includes image graying operation, image dilation operation and Gaussian filtering process. The image graying operation is implemented by the following calculation method: ， in, Represents the grayscale value at the (x, y) position in the grayscale image, Represents the pixel value of the red channel at position (x,y) in the color image, Represents the red RGB weight coefficient, Represents the pixel value of the green channel at the (x,y) position in the color image, represents the green RGB weight coefficient, Represents the pixel value of the blue channel at position (x,y) in the color image, Represents the blue RGB weight coefficient; The image expansion operation is implemented using the following calculation method: ， in, Represents the pixel value at the (x,y) position in the input image data, Represents a structural element, Represents structural elements The offset of each element in Represents the image after dilation The pixel value at position (x,y) in the image; The Gaussian filtering process is implemented by the following calculation method: ， in, Represents the pixel value at the (x,y) position in the input image data, is the weight of the Gaussian kernel at position (i, j), k represents the radius of the Gaussian filter, The new pixel value after filtering.

5. A solder joint visual inspection and classification method according to any one of claims 1-2 or 4, characterized in that: The step of extracting the geometric features of the solder joints in the standard image data by using an edge detection algorithm specifically includes the following steps: Obtain edge points of welding spots in the standard image data by using a Canny edge detection algorithm; Obtain circle parameters that best match the edge point distribution through Hough circle transform, wherein the circle parameters include circle center coordinates (a, b) and circle radius r; Based on the circle parameters and the edge point distribution, geometric features of the solder joint are extracted.

6. A solder joint visual inspection and classification method according to claim 5, characterized in that: The geometric features of the solder joint include edge continuity, solder joint area S, solder joint perimeter C and circularity e; The edge continuity of the solder joint is judged by the edge point distribution. If the edge point distribution is continuous, the solder joint edge is judged to be continuous; otherwise, the solder joint edge is judged to be discontinuous. According to the edge point distribution, the number of pixel points in the edge is calculated as the welding point area S; Calculate the circumference C of the welding point by the radius of the circle parameter; The circularity e is calculated by the solder joint area S and the solder joint perimeter C, specifically using the following calculation method: 。 7. A solder joint visual inspection and classification method according to claim 6, characterized in that: The step of constructing a labeled data set based on the standard image data and the corresponding geometric features comprises the following steps: Based on the continuity of the edge points, the state of the solder joint is judged. If the edge of the solder joint is discontinuous, a label of a false solder joint or a leak solder joint is established using the corresponding standard image data and geometric features; If the edge of the weld is continuous, the weld state is judged by the weld area S. If the weld area S is outside the standard weld area parameter range, a label of a blown weld is established with the corresponding standard image data and geometric features; If the solder joint area S is within the standard solder joint area parameter range, the solder joint state is judged by the circularity e; if the circularity e is outside the standard solder joint circularity parameter range, a label of flying soldering is established with the corresponding standard image data and geometric features; If the circularity e is within the standard welding spot circularity parameter range, a normal label is established for the corresponding standard image data and geometric features.

8. A solder joint visual inspection and classification method according to any one of claims 1-2, 4 or 6-7, characterized in that: The neural network model at least includes: The feature extraction layer includes several convolutional layers for performing deep feature extraction on the standard image data, which is specifically implemented by the following calculation method: ， in, Represents the value of channel k at position (i, j) of the extracted feature. represents the pixel value of channel c at position (i+m,j+n) of the input image, i and j represent the spatial positions of the extracted features, and k represents the output channel index. represents the weight of the convolution kernel, m, n represent the spatial size of the convolution kernel, c is the channel index of the convolution kernel corresponding to the input image, k is the index of the convolution kernel, representing the kth convolution kernel, is the bias, used to adjust the convolution result, represents the activation function; The fully connected layer is used to structure the geometric features corresponding to the standard image data, and is specifically implemented by the following calculation method: ， in, represents the output of layer l, represents the activation function, represents the weight matrix, Indicates bias; A feature fusion layer, the input end of the feature fusion layer is connected to the feature extraction layer and the output end of the fully connected layer, and is used to fuse the features extracted by the feature fusion layer and the geometric features structured by the fully connected layer to form a feature vector; The feature classification layer is used to generate a category probability distribution based on the feature vector through a Softmax activation function, thereby classifying the solder joints.

9. A system using a solder joint visual inspection and classification method as claimed in any one of claims 1 to 8, characterized in that: At least: An image acquisition module, the image acquisition module at least includes an industrial camera for real-time acquisition of original image data of dynamic solder joints in a production line, and a light source system adapted to the industrial camera for changing the reflective properties of the solder joints, the light source system at least including red, green and blue LED lights; An image preprocessing module, used to preprocess the original image data of the welding spot to obtain standard image data, extract the geometric features of the welding spot in the standard image data by an edge detection algorithm, and construct a labeled data set based on the standard image data and the corresponding geometric features; A model building module, used to build a neural network model for solder joint detection and classification, and train the neural network model through the labeled data set to obtain a trained neural network model; The classification module is used to detect and classify solder joints using the trained neural network model.

10. A solder joint visual inspection and classification system according to claim 9, characterized in that: Also includes: The control module is used to control the industrial camera to track the position of dynamic welding points in the production line in real time, control the angle and brightness of the red, green and blue LED lights in the light source system, and generate a test report based on the classification result of the classification module and send it to the production control system of the production line.

Images