A method and system for visual inspection and classification of solder joints

Abstract

The present application relates to the technical field of visual detection and recognition of welding spots, and particularly relates to a visual detection and classification method and system for welding spots.The visual detection and classification method for welding spots disclosed by the present application obtains original image data and pre-processes the original image data to obtain standard image data; extracts welding spot geometric features through an edge detection algorithm, constructs a data set based on the standard image data and the geometric features; constructs a neural network model, extracts features from the standard image data, structures the geometric features, fuses the extracted features and the structured geometric features to form a feature vector, and classifies the welding spots; trains the neural network model based on the data set; and detects and classifies the welding spots through the trained neural network model.The present application can comprehensively and accurately classify welding spots, and can detect welding spots in real time, automatically and comprehensively detect and classify welding spot defects, has high detection and classification accuracy, is suitable for complex reflection conditions on the surface of welding spots, reduces the probability of false detection and missed detection, and has good robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual detection and recognition of welding points, and particularly relates to a visual detection and classification method and system for welding points. BACKGROUND

[0002] Flat wire motors are widely used in high-end application fields such as new energy vehicles. However, the excellent performance and reliable operation of the flat wire motor are deeply dependent on the quality of the welding points. In order to ensure the firm connection between the coil and the stator, high-precision processes such as laser welding or resistance welding are usually adopted.

[0003] In the actual production process, due to the diversity of material properties, the volatility of processing environment and the stability of equipment, a series of defects such as virtual welding, missed welding, explosive welding and flying welding will inevitably occur in the welding points. These defects not only significantly reduce the mechanical strength of the welding points, but also may gradually lead to the decline of electrical performance, and even induce short circuit or cause failure in serious cases. Therefore, efficient and accurate quality detection of the welding points has become a crucial link to ensure the reliability of the flat wire motor.

[0004] The traditional manual detection method relies on the naked eye observation of the operator to evaluate the quality of the welding points, and this method has many insurmountable drawbacks. First, its detection efficiency is low, which is difficult to adapt to the fast-paced demand of batch production in modern manufacturing industry; second, the subjectivity of manual detection is strong, and there may be differences in the identification standards of welding point defects among different detection personnel, which seriously affects the reliability and consistency of the detection results; third, the operator is prone to fatigue after a long time of welding point detection work, which further increases the risk of misdiagnosis.

[0005] In order to improve this situation, some production lines have adopted basic sensor detection devices, such as using infrared sensors to detect the temperature of the welding points and using laser ranging sensors to measure the height of the welding points. Although these methods can provide certain quantitative indicators, the detection range and accuracy are still greatly limited because the sensors cannot capture the subtle morphology of the welding points. Specifically, for complex defects such as explosive welding and flying welding, simple sensor detection methods often cannot effectively identify them; at the same time, for different specifications of flat wire motors or welding point morphology, the sensor settings need to be frequently adjusted, which greatly affects the production efficiency.

[0006] At present, although some enterprises have tried to introduce machine vision technology for welding point detection, the existing visual systems mostly still remain at the level of surface detection, and can only identify simple welding point positions or appearance problems, and cannot comprehensively and accurately classify various complex defects of the welding points. At the same time, the algorithm accuracy of the existing visual systems needs to be further improved, especially in the case of high glossiness of the welding point surface and existence of light interference, the misdiagnosis rate is still high. SUMMARY

[0007] In view of the problems that the prior art cannot comprehensively and accurately detect and classify various complex defects of the welding points of the flat wire motor, especially under the reflection interference, the high false detection rate, the welding point visual detection and classification method and system are provided, the original image data containing the reflection of the welding point is collected in real time, the standard image data is obtained by preprocessing the original image data, the geometric features of the welding point are obtained by the edge detection algorithm, the deep features of the standard image data are extracted by the constructed neural network model, the geometric features are structured, and the extracted features and the structured geometric features are fused to form a feature vector, so that the welding point is comprehensively and accurately classified, the welding point is detected in real time, the welding point defects are automatically and comprehensively detected and classified, the detection and classification accuracy is high, the complex reflection conditions on the surface of the welding point are adapted, the false detection and missed detection probabilities are reduced, and the robustness is good.

[0008] The welding point visual detection and classification method disclosed in the application comprises the following steps:

[0009] Obtain the original image data containing the reflection of the welding point, and preprocess the original image data to obtain standard image data;

[0010] Extract the geometric features of the welding point in the standard image data by an edge detection algorithm, and construct a data set with labels based on the standard image data and the corresponding geometric features;

[0011] Construct a neural network model for welding point detection and classification, the neural network model is used for deep feature extraction of the standard image data, and the geometric features corresponding to the standard image data are structured, and the extracted features and the structured geometric features are fused to form a feature vector, and the welding point is classified based on the feature vector;

[0012] Train the neural network model based on the data set with labels;

[0013] Detect and classify the welding point by the trained neural network model.

[0014] A preferred embodiment, the original image data containing the reflection of the welding point is obtained, and the specific steps are as follows:

[0015] Based on the reflection characteristics of the welding point, the angles and brightness of the red, green and blue three-color LED lamps are dynamically adjusted by an encoding mode;

[0016] Control the industrial camera to take pictures of each welding point one by one on the assembly line to obtain the original image data containing the reflection of the welding point.

[0017] A preferred embodiment, the pre-processing standard image data, specifically including the following steps:

[0018] In the original image data set Roi detection area, the Roi detection area includes the welding point and its edge;

[0019] Based on the Roi detection area on the original image data to get the welding point image data clipping;

[0020] The welding point image data is enhanced to get the standard image data.

[0021] A preferred embodiment, the image enhancement processing includes image gray operation, image dilation operation or Gaussian filter processing, the image gray operation, using the following calculation method is realized:

[0022] ,

[0023] Wherein, Indicates the gray value of the (x, y) position in the gray image, Indicates the pixel value of the red channel of the (x, y) position in the color image, Indicates the red RGB weight coefficient, Indicates the pixel value of the green channel of the (x, y) position in the color image, Indicates the green RGB weight coefficient, Indicates the pixel value of the blue channel of the (x, y) position in the color image, Indicates the blue RGB weight coefficient;

[0024] The image dilation operation, using the following calculation method is realized:

[0025] ,

[0026] Wherein, Indicates the pixel value of the (x, y) position in the input image data, Indicates the structure element, Indicates the offset of each element in the structure element , Indicates the pixel value of the (x, y) position in the expanded image ;

[0027] The Gaussian filter processing, using the following calculation method is realized:

[0028] ,

[0029] Wherein, Indicates the pixel value of the (x, y) position in the input image data, is the weight of the Gaussian kernel at position (i,j), k represents the radius of the Gaussian filter, the filtered new pixel value.

[0030] A preferred embodiment, the geometric features of the solder joint in the standard image data are extracted by an edge detection algorithm, specifically comprising the following steps:

[0031] By Canny edge detection algorithm, the edge points of the solder joint in the standard image data are obtained;

[0032] The circular parameters most consistent with the edge point distribution are obtained by Hough circle transformation, including the center coordinates (a, b) and the radius r of the circle;

[0033] Based on the circular parameters and the edge point distribution, the geometric features of the solder joint are extracted.

[0034] A preferred embodiment, the geometric features of the solder joint include edge continuity, solder joint area S, solder joint perimeter C and circularity e;

[0035] The edge continuity of the solder joint is judged by the edge point distribution. If the edge point distribution is continuous, it is judged that the solder joint edge is continuous, otherwise it is judged that the solder joint edge is discontinuous;

[0036] The number of pixel points in the edge is calculated as the solder joint area S by the edge point distribution;

[0037] The solder joint perimeter C is calculated by the radius of the circular parameters;

[0038] The circularity e is calculated by the solder joint area S and the solder joint perimeter C, specifically using the following calculation method:

[0039] .

[0040] A preferred embodiment, the labeled data set is constructed based on the standard image data and the corresponding geometric features, comprising the following steps:

[0041] Based on the edge point continuity, the solder joint state is judged. If the solder joint edge is discontinuous, the corresponding standard image data and geometric features are labeled as false or missed welding;

[0042] If the solder joint edge is continuous, the solder joint state is judged by the solder joint area S. If the solder joint area S is outside the standard solder joint area parameter range, the corresponding standard image data and geometric features are labeled as explosive welding;

[0043] If the welding spot area S is within the standard welding spot area parameter range, the welding spot state is judged by the circularity e, and if the circularity e is outside the standard welding spot circularity parameter range, the corresponding standard image data and geometric features are labeled as flying welding;

[0044] If the circularity e is within the standard welding spot circularity parameter range, the corresponding standard image data and geometric features are labeled as normal.

[0045] A preferred embodiment, the neural network model at least includes:

[0046] The feature extraction layer includes a plurality of convolutional layers for deep feature extraction of the standard image data, and is implemented by the following calculation method:

[0047] ,

[0048] wherein, represents the value of the extracted feature at position (i,j) in channel k, represents the pixel value of the input image at position (i+m,j+n) in channel c, i,j represents the spatial position of the extracted feature, k represents the output channel index corresponding to an output channel, represents the weight of the convolution kernel, m,n represents the spatial size of the convolution kernel, c is the channel index of the input image corresponding to the convolution kernel, k is the index of the convolution kernel, and represents the kth convolution kernel, is a bias for adjusting the convolution result, represents an activation function;

[0049] The fully connected layer is used to structure the geometric features corresponding to the standard image data, and is implemented by the following calculation method:

[0050] ,

[0051] wherein, represents the output of the lth layer, represents an activation function, represents a weight matrix, represents a bias;

[0052] The feature fusion layer is connected to the output end of the feature extraction layer and the fully connected layer, and is used to fuse the features extracted by the feature fusion layer and the geometric features structured by the fully connected layer to form a feature vector;

[0053] a feature classification layer, configured to generate a class probability distribution based on the feature vector by a Softmax activation function, so as to classify the welding spot.

[0054] The application further provides a system using the welding spot visual detection and classification method, and the system comprises at least:

[0055] an image acquisition module, comprising at least an industrial camera configured to acquire real-time dynamic welding spot original image data in a production line, and a light source system configured to change the reflection characteristics of the welding spot, wherein the light source system comprises at least red, green and blue LED lights;

[0056] an image preprocessing module, configured to preprocess the welding spot original image data to obtain standard image data, extract geometric features of the welding spot in the standard image data by an edge detection algorithm, and construct a labeled data set based on the standard image data and the corresponding geometric features;

[0057] a model building module, configured to build a neural network model for welding spot detection and classification, and train the neural network model by the labeled data set to obtain a trained neural network model;

[0058] a classification module, configured to detect and classify the welding spot by the trained neural network model.

[0059] A preferred embodiment further comprises:

[0060] a control module, configured to control the industrial camera to track the position of the dynamic welding spot in the production line in real time, control the angle and brightness of the red, green and blue LED lights in the light source system, and send a detection report of the classification result of the classification module to a production control system of the production line.

[0061] Compared with the prior art, the welding spot visual detection and classification method and system have the following beneficial effects:

[0062] (1) The welding point visual detection and classification method and system disclosed by the application obtains original image data containing welding point reflection and performs preprocessing to obtain standard image data. The original image data contains complex reflection information of the welding point surface. Using the original image data containing the welding point reflection information can enhance the generalization ability of the neural network model, effectively remove reflection interference, and improve the accuracy of welding point detection. The trained neural network model can not only process welding point images under normal lighting conditions, but also adapt to welding point images under complex reflection conditions, improving the applicability and reliability of the model. The edge detection algorithm is used to extract the geometric features of the welding points in the standard image data, providing a reliable basis for subsequent data analysis and welding point detection. A labeled data set is constructed based on the standard image data and corresponding geometric features. The data set not only contains the visual appearance information of the welding points, but also integrates their geometric properties such as shape, size, and position. The fusion of this multi-dimensional information provides a solid foundation for comprehensive analysis and accurate classification of welding points. The data set integrating multi-dimensional information can train a more generalized and robust welding point detection and classification neural network model. Such a neural network model can not only process welding point images under normal conditions, but also maintain stable performance under complex conditions such as light changes and noise interference, improving the accuracy and reliability of welding point detection. The constructed neural network model is used for deep feature extraction of the standard image data and structuring of the geometric features corresponding to the standard image data. The extracted features and structured geometric features are fused to form a feature vector, and the welding points are classified based on the feature vector. Deep feature extraction can automatically extract complex and abstract features from image data and capture subtle differences in welding point images, such as minor defects and shape changes. These features are crucial for accurate classification of welding points. Through deep feature extraction, the neural network model can more accurately identify different types of welding points. Structuring geometric features can clarify key information such as shape, size, and position of the welding points. These geometric features are important for understanding the physical properties of the welding points and judging their quality. On the other hand, structuring geometric features enhances model interpretability. Through structured geometric features, the model's decision-making process is more transparent, which helps to understand why the model makes a particular classification. This is very helpful for quality control and troubleshooting in industrial applications.

[0063] (2) The welding point visual detection classification method and system disclosed by the application judges the welding point state based on the continuity of the edge points, and if the welding point edge is discontinuous, the corresponding standard image data and geometric features are labeled as false welding or missed welding; if the welding point edge is continuous, the welding point state is judged through the welding point area S, and if the welding point area S is outside the standard welding point area parameter range, the corresponding standard image data and geometric features are labeled as explosive welding; if the welding point area S is within the standard welding point area parameter range, the welding point state is judged through the circularity e, and if the circularity e is outside the standard welding point circularity parameter range, the corresponding standard image data and geometric features are labeled as flying welding; if the circularity e is within the standard welding point circularity parameter range, the corresponding standard image data and geometric features are labeled as normal. On the one hand, not only the continuity of the welding point edge is considered, but also more detailed geometric features such as the welding point area and the circularity are introduced. This multi-dimensional feature parameter analysis makes the judgment of the welding point state more comprehensive and accurate. By comprehensively considering various feature parameters, the quality and state of the welding point can be more comprehensively reflected, and the misjudgment or missed judgment caused by the judgment error of a single feature parameter can be reduced; on the other hand, according to the judgment result of the welding point state, the corresponding label can be automatically established for the standard image data and the geometric features. This automatic label establishment process greatly improves the efficiency and accuracy of data processing. Through the automatic label establishment, the method can generate a large number of data sets with accurate labels, providing rich sample resources for the training of subsequent welding point detection classification models. The feature parameters and the judgment standard can be adjusted and optimized according to actual needs. The standard welding point area and the circularity parameter range can be adjusted according to different welding processes and materials, and the method can be applied to other types of welding point detection classification tasks. It only needs to adjust the feature parameters and the judgment standard according to specific needs.

[0064] (3) The welding point visual detection classification method and system disclosed by the application at least includes an image acquisition module, an image preprocessing module, a model building module, a classification module and a control module. The control module is used for generating a detection report of the classification result of the classification module and sending the detection report to a production control system of a production line. On the one hand, the control module is suitable for the high-speed rhythm of the modern production line and completes real-time online detection; on the other hand, through data analysis and process feedback, the welding process parameters are dynamically adjusted to realize closed-loop control of the welding process and help improve production quality and fault tracing. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 It is a flowchart of an embodiment of the welding point visual detection method and system of the application;

[0066] Figure 2The normal solder joint contains original image data of solder joint reflection light for an embodiment of a solder joint visual inspection method and system of the present application;

[0067] Figure 3 The un-soldered solder joint contains original image data of solder joint reflection light for an embodiment of a solder joint visual inspection method and system of the present application;

[0068] Figure 4 The virtual solder joint contains original image data of solder joint reflection light for an embodiment of a solder joint visual inspection method and system of the present application;

[0069] Figure 5 The flying solder joint contains original image data of solder joint reflection light for an embodiment of a solder joint visual inspection method and system of the present application;

[0070] Figure 6 The blown solder joint contains original image data of solder joint reflection light for an embodiment of a solder joint visual inspection method and system of the present application. DETAILED DESCRIPTION

[0071] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0072] Embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0074] Furthermore, in this invention, the use of terms such as "first" and "second" is for descriptive purposes only and does not specifically refer to any order or sequence, nor is it intended to limit the invention. They are merely used to distinguish components or operations described using the same technical terms, and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but only if they are feasible for those skilled in the art. If a combination of technical solutions is contradictory or impossible to implement, such a combination should be considered nonexistent and not within the scope of protection claimed by this invention.

[0075] This embodiment presents a visual inspection and classification method for solder joints, such as... Figure 1 As shown, it includes at least the following steps:

[0076] Step S1: Acquire raw image data containing solder joint reflections and preprocess it to obtain standard image data. For example... Figures 2-6 As shown, Figure 2 The original image data for normal solder joints includes the reflections from the solder joints; Figure 3 The original image data for unsoldered solder joints includes the reflection of the solder joint; Figure 4 The original image data for the solder joint containing the reflection of the solder joint is used for the cold solder joint; Figure 5 The original image data for the fly solder joint includes the reflection of the solder joint; Figure 6 The original image data for the weld spatter includes the reflection of the weld spatter.

[0077] The raw image data contains complex reflective information about the solder joint surface. Using raw image data containing solder joint reflective information can enhance the generalization ability of the neural network model, effectively remove reflective interference, and improve the accuracy of solder joint detection. The trained neural network model can not only handle solder joint images under normal lighting conditions but also adapt to solder joint images under complex reflective conditions, improving the model's applicability and reliability. Geometric features of solder joints in standard image data are extracted using edge detection algorithms, providing a reliable foundation for subsequent data analysis and solder joint detection. A labeled dataset is constructed based on the standard image data and corresponding geometric features. This dataset not only contains the visual appearance information of the solder joints but also incorporates their geometric attributes such as shape, size, and position. This fusion of multi-dimensional information provides a solid foundation for comprehensive analysis and accurate classification of solder joints. The dataset integrating multi-dimensional information can train a more generalized and robust solder joint detection and classification neural network model. Such a neural network model can not only handle solder joint images under normal conditions but also maintain stable performance under complex conditions such as lighting changes and noise interference, improving the accuracy and reliability of solder joint detection.

[0078] Specifically comprising the following steps:

[0079] Step S101, based on the reflective characteristics of the welding points, dynamically adjusting the angle and brightness of red, green and blue color LED lights through coding, and the number of each color LED light is at least 1. Since the welding points may have different reflective properties due to factors such as material, surface state, shape, etc., dynamic adjustment of lighting can ensure that each welding point is photographed under optimal lighting conditions, thereby improving the quality and clarity of the image. The use of multiple color LED lights also allows for color analysis, which helps to distinguish the differences between the welding points and their surroundings, further enhancing the accuracy of detection.

[0080] Step S102, control the industrial camera to take pictures of each welding point one by one on the assembly line, and obtain raw image data containing the reflection of the welding points. On the one hand, such an automated process greatly improves detection efficiency, reduces the need for human intervention, reduces labor costs, and speeds up the operation of the production line; on the other hand, it ensures that each welding point can be uniformly and consistently photographed, improving the consistency and reliability of detection.

[0081] Step S103, setting a Roi detection area in the raw image data, the Roi detection area including the welding points and their edges. This allows subsequent image processing to be more focused and efficient, reducing irrelevant information interference and improving image processing accuracy and speed by limiting the processing area.

[0082] Step S104, based on the Roi detection area, the raw image data is cropped to obtain welding point image data. The cropped image data is more compact and contains key information about the welding points, making it easier for subsequent image analysis and processing.

[0083] Step S105, performing image enhancement processing on the welding point image data to obtain standard image data. Image enhancement processing includes image graying operation, image dilation operation or Gaussian filter processing.

[0084] In this embodiment, image graying operation, image dilation operation and Gaussian filter processing are used in sequence to obtain standard image data. First, perform image graying operation on the welding point image data, and weight the sum of the red, green and blue channel values of each pixel in the welding point image data, using the following calculation method:

[0085] ,

[0086] wherein, represents the gray value of the (x, y) position in the gray image, represents the pixel value of the red channel at the (x, y) position in the color image, represents the red RGB weight coefficient, pixel value of the green channel at position (x, y) in the color image, represents a green RGB weight coefficient, pixel value of the blue channel at position (x, y) in the color image, represents a blue RGB weight coefficient. By converting the solder joint image data into a grayscale image through the image graying operation, the calculation complexity of the subsequent image processing algorithm is simplified, the storage and transmission costs are reduced, the detection speed is improved, and real-time detection and classification of the solder joint are facilitated.

[0087] An image dilation operation is performed on the grayscale image after the image graying operation, and the following calculation method is adopted to realize it:

[0088] ,

[0089] wherein, pixel value of the green channel at position (x, y) in the color image, represents a structural element, represents a structural element offset of each element in the structural element, pixel value of the green channel at position (x, y) in the color image, Through the image dilation operation, the boundary of the brighter area (such as the solder joint) in the image can be effectively expanded, so that the edge of the solder joint becomes clearer and more obvious. At the same time, since the dilation operation selects the maximum grayscale value in the neighborhood as the result, it can also remove or weaken the influence of the background on the solder joint contour and features to a certain extent.

[0090] A Gaussian filtering operation is performed on the image after the image dilation operation to obtain standard image data, and the following calculation method is adopted to realize it:

[0091] ,

[0092] wherein, pixel value of the green channel at position (x, y) in the color image, is the weight of the Gaussian kernel at position (i, j), and k represents the radius of the Gaussian filter, new pixel value after filtering. Through the Gaussian filtering operation, high-frequency noise such as Gaussian noise, salt and pepper noise, etc. in the image is removed, and through the removal of detail and edge noise, the image effect is enhanced, which is convenient for the following edge detection operation.

[0093] Step S2, the geometric features of the welding points in the standard image data are extracted by an edge detection algorithm, and a labeled data set is constructed based on the standard image data and the corresponding geometric features. The data set not only contains the visual appearance information of the welding points, but also integrates their geometric properties such as shape, size, position, etc. The integration of such multi-dimensional information provides a solid foundation for comprehensive analysis and accurate classification of welding points. The data set integrating multi-dimensional information can train a more generalized and robust welding point detection and classification neural network model. Such a neural network model can not only process welding point images under normal conditions, but also maintain stable performance under complex conditions such as changes in illumination and noise interference, thereby improving the accuracy and reliability of welding point detection.

[0094] Specifically includes the following steps:

[0095] Step S201, the edge points of the welding points in the standard image data are obtained by the Canny edge detection algorithm. The derivatives of the standard image data in the x direction and the y direction are obtained by using the Sobel operator, the convolution kernel corresponding to the vertical Sobel operator in the x direction is used, and the following calculation method is used to realize it:

[0096] ,

[0097] The convolution kernel corresponding to the horizontal Sobel operator in the y direction is used, and the following calculation method is used to realize it:

[0098] ,

[0099] The intensity of the edge point is represented by the amplitude of the gradient, and the following calculation method is used to realize it:

[0100] ,

[0101] The direction of the edge point is represented by the direction of the gradient, and the following calculation method is used to realize it:

[0102] ,

[0103] wherein, represents the amplitude of the gradient, represents the derivative of the standard image data in the x direction, represents the derivative of the standard image data in the y direction, represents the gradient direction. By convolution operation, the gradients G x and G y, and then the edge intensity and the direction of the edge of each pixel point are calculated. In the generated gradient amplitude image, a non-maximum suppression technique is adopted to retain the local maximum gradient value and suppress the non-maximum value, so as to realize fine processing of the edge. Two threshold values are set: one is a high threshold value, and the other is a low threshold value. For a point with a pixel value higher than the high threshold value, the point is determined as a strong edge; and for a point with a pixel value lower than the low threshold value, the point is removed and is not used as an edge point. For a point with a pixel value between the two threshold values, if the point has a connection relationship with the strong edge, the point is determined as a weak edge. Finally, the edge points of the welding spot are accurately positioned and extracted according to the region with a relatively large change in the gray value in the image. Although the Canny algorithm involves multiple steps, the calculation amount is relatively small, and the Canny algorithm is suitable for real-time edge detection tasks and can quickly process a large amount of image data. Moreover, by adjusting the parameters, the influence of noise and pseudo-edges can be reduced while the accuracy of the edge is maintained, and the performance of the edge detection is improved.

[0104] In step S202, the circular parameters most consistent with the edge point distribution are obtained through the Hough circle transformation. The circular parameters include the center coordinates (a, b) and the radius r. A three-dimensional accumulator array is defined to store the number of votes for each possible combination of the center and the radius. For each edge point, the possible center coordinates (a, b) are calculated for each possible radius value, and the corresponding position in the accumulator is voted. Specifically, for each edge point (x i , y i ) and the radius r, the possible center coordinates (a, b) satisfy:

[0105] ,

[0106] The number of votes for (a, b, r) in the accumulator is increased. The (a, b, r) with the highest number of votes is taken as the circular parameters most consistent with the edge point distribution.

[0107] In step S203, the geometric features of the welding spot are extracted based on the circular parameters and the edge point distribution.

[0108] The geometric features of the welding spot include the edge continuity, the area S of the welding spot, the circumference C of the welding spot, and the circularity e. The edge continuity of the welding spot is determined based on the edge point distribution. If the edge point distribution is continuous, it is determined that the edge of the welding spot is continuous, otherwise, it is determined that the edge of the welding spot is discontinuous.

[0109] The number of pixel points in the edge is calculated as the area S of the welding spot based on the edge point distribution.

[0110] The circumference C of the welding spot is calculated based on the radius of the circular parameters.

[0111] The circularity e is calculated based on the area S and the circumference C of the welding spot. The calculation method is as follows:

[0112] .

[0113] In step S204, the state of the welding point is judged based on the continuity of the edge points. If the edge of the welding point is discontinuous, the corresponding standard image data and geometric features are labeled as false welding or missed welding.

[0114] If the edge of the welding point is continuous, the state of the welding point is judged by the welding point area S. If the welding point area S is outside the standard welding point area parameter range, the corresponding standard image data and geometric features are labeled as blown welding.

[0115] If the welding point area S is within the standard welding point area parameter range, the state of the welding point is judged by the circularity e. If the circularity e is outside the standard welding point circularity parameter range, the corresponding standard image data and geometric features are labeled as flying welding.

[0116] If the circularity e is within the standard welding point circularity parameter range, the corresponding standard image data and geometric features are labeled as normal. On the one hand, not only the continuity of the welding point edge is considered, but also more detailed geometric features such as welding point area and circularity are introduced. This multi-dimensional feature parameter analysis makes the judgment of the state of the welding point more comprehensive and accurate. By considering a variety of feature parameters, the quality and state of the welding point can be more comprehensively reflected, and the misjudgment or omission caused by the failure of a single feature parameter judgment can be reduced. On the other hand, according to the judgment result of the state of the welding point, the corresponding label can be automatically established for the standard image data and geometric features. This automatic label establishment process greatly improves the efficiency and accuracy of data processing. Through automatic label establishment, the method can generate a large number of data sets with accurate labels, providing rich sample resources for the training of subsequent welding point detection classification models. The feature parameters and judgment standards can be adjusted and optimized according to actual needs. The standard welding point area and circularity parameters can be adjusted according to different welding processes and materials. The method can be applied to other types of welding point detection classification tasks by adjusting the feature parameters and judgment standards according to specific needs.

[0117] Step S3, a neural network model for solder joint detection classification is constructed, the neural network model is used for deep feature extraction of standard image data, and the geometric features corresponding to the standard image data are structured, and the extracted features and the structured geometric features are fused to form a feature vector, and the solder joint is classified based on the feature vector. Deep feature extraction can automatically extract complex and abstract features from image data, and can capture subtle differences in solder joint images, such as small defects and shape changes. These features are crucial for accurate classification of solder joints. Through deep feature extraction, the neural network model can more accurately identify different types of solder joints. Structuring geometric features can clearly indicate the shape, size, position and other key information of the solder joint. These geometric features are important for understanding the physical properties of the solder joint and judging its quality. On the other hand, structured geometric features enhance model interpretability, making the model's decision-making process more transparent and helping to understand why the model makes a particular classification, which is very helpful for quality control and troubleshooting in industrial applications.

[0118] In this embodiment, the neural network model at least includes:

[0119] The feature extraction layer includes a plurality of convolution layers for deep feature extraction of standard image data, and the following calculation method is used to realize it:

[0120] ,

[0121] wherein, represents the value of the extracted feature at position (i,j) in channel k, represents the pixel value of the input image at position (i+m,j+n) in channel c, i,j represents the spatial position (height and width) of the extracted feature, k represents the output channel index, each convolution kernel corresponds to an output channel, represents the weight of the convolution kernel, m,n represents the spatial size of the convolution kernel, c is the channel index of the input image corresponding to the convolution kernel, k is the index of the convolution kernel, and represents the kth convolution kernel, is a bias used to adjust the convolution result and provide additional flexibility, represents an activation function used to introduce non-linear characteristics;

[0122] The fully connected layer is used to structure the geometric features corresponding to the standard image data, and the following calculation method is used to realize it:

[0123] ,

[0124] wherein, represents the output of the lth layer, represents an activation function, denotes a weight matrix, denotes a bias;

[0125] a feature fusion layer, an input end of the feature fusion layer being connected with an output end of the feature extraction layer and the fully connected layer, the feature fusion layer being configured to fuse features extracted by the feature fusion layer and geometric features structured by the fully connected layer to form a feature vector;

[0126] a feature classification layer, the feature classification layer being configured to generate a class probability distribution based on the feature vector by using a Softmax activation function, so as to classify the welding points. The Softmax activation function is configured to generate the class probability distribution by using the following calculation method:

[0127] ,

[0128] wherein, denotes a kth element of the feature vector, denotes a kth element of an output vector, that is, a classification probability of a kth class.

[0129] In the embodiment, the features extracted by the feature fusion layer and the geometric features structured by the fully connected layer are fused by using the following calculation method:

[0130] ,

[0131] wherein, denotes a feature vector, denotes features extracted by the feature fusion layer, denotes geometric features structured by the fully connected layer.

[0132] Step S4: training the neural network model based on a data set with labels.

[0133] In the embodiment, the data set is divided into a training set and a test set, the neural network model is classified and trained in the training set by using a cross-entropy loss function, a difference between a predicted probability of the neural network model and a real label code is calculated, and a neural network model parameter is optimized by using a gradient descent algorithm and an Adam optimizer.

[0134] The trained neural network model is predicted in the test set, and a performance of the model is evaluated by using a classification accuracy, a precision, a recall and an F1 score index.

[0135] Step S5, detecting and classifying the welding points through the trained neural network model. In order to apply this technology to actual production, it is necessary to integrate the trained neural network model into a complete welding point detection system. This system is not only responsible for capturing the image of the welding point, but also for pre-processing the image, inputting it into the neural network model, and accurately interpreting the output of the model to achieve accurate classification and detection of the welding point. By integrating the neural network model into the welding point detection system, automated and accurate detection and classification of welding points can be achieved. Such a system not only improves detection efficiency and accuracy, but also reduces the cost and risk of manual detection.

[0136] The embodiment also provides a system using any of the welding point visual detection and classification methods described above, comprising at least:

[0137] An image acquisition module, which at least includes an industrial camera for acquiring real-time dynamic welding point raw image data in a production line, and a light source system adapted to the industrial camera for changing the reflection characteristics of the welding point, the light source system at least including red, green and blue LED lights;

[0138] An image preprocessing module for preprocessing the welding point raw image data to obtain standard image data, extracting the geometric features of the welding point in the standard image data through an edge detection algorithm, and constructing a labeled data set based on the standard image data and the corresponding geometric features;

[0139] A model building module for building a neural network model for welding point detection and classification, and training the neural network model through the labeled data set to obtain a trained neural network model;

[0140] A classification module for detecting and classifying the welding points through the trained neural network model.

[0141] Preferably, the welding point visual detection and classification system further comprises:

[0142] A control module for controlling the industrial camera to track the position of the dynamic welding point in the production line in real time, controlling the angle and brightness of the red, green and blue LED lights in the light source system, and sending the classification results of the classification module to the production control system of the production line to generate a detection report. On the one hand, it adapts to the high speed of modern production lines, and completes real-time online detection; on the other hand, through data analysis and process feedback, dynamically adjusts the welding process parameters, realizes closed-loop control of the welding process, and helps to improve production quality and fault tracing.

[0143] The embodiments of the present application can accurately identify various welding point defects with an accuracy rate of more than 98% by using advanced deep learning algorithms, ensuring product quality. Combined with image enhancement technology and the optimized application of intelligent light sources, the false detection rate is significantly reduced in complex optical environments, improving the reliability of detection. The detection process of each welding point is rapid and efficient, taking no more than 0.5 seconds, fully meeting the needs of high-speed production of modern production lines. In addition, the automatic welding point visual detection and classification system has excellent compatibility and can easily integrate with various production equipment, enabling rapid deployment and providing a plug-and-play experience for production lines.

[0144] The detection results are fed back to the production line in real time through the production control system (MES system) of the production line, so that the welding process parameters can be dynamically adjusted according to the actual detection situation, further optimizing the production process. The system also records all detection data comprehensively, providing strong support for tracing and analysis, helping enterprises to continuously improve and constantly improve production efficiency and product quality.

[0145] It is worth mentioning that the welding point visual detection and classification system has high adaptability and can be flexibly adjusted according to different types of welding processes and welding point specifications. Modular design provides unlimited possibilities for system expansion, such as easily adapting to welding point detection of other types of motors to meet the diverse production needs of enterprises.

[0146] In summary, the above only describes the preferred embodiments of the present application and does not limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of solder joint visual inspection classification, characterized by, At least comprising the following steps: Obtaining original image data containing weld reflection and preprocessing to obtain standard image data, specifically, based on the reflection characteristics of the weld, the angles and brightness of the red, green and blue color LED lights are dynamically adjusted by encoding, the industrial camera is controlled to take pictures of each weld one by one on the assembly line, and the original image data containing the weld reflection is obtained; The geometric features of the weld in the standard image data are extracted by an edge detection algorithm, including: The edge points of the weld in the standard image data are obtained by the Canny edge detection algorithm; The circular parameters most consistent with the edge point distribution are obtained by the Hough circle transformation, including the center coordinates and the radius r; Based on the circular parameters and the edge point distribution, the geometric features of the weld are extracted; The geometric features of the weld include edge continuity, weld area S, weld circumference C and circularity e; The edge continuity of the weld is judged by the edge point distribution, if the edge point distribution is continuous, it is judged that the weld edge is continuous, otherwise it is judged that the weld edge is discontinuous; The number of pixel points in the edge is calculated as the weld area S by the edge point distribution; The weld circumference C is calculated by the radius of the circular parameters; The circularity e is calculated by the weld area S and the weld circumference C, specifically using the following calculation method: And based on the standard image data and the corresponding geometric features, a labeled data set is constructed, specifically, based on the edge point continuity, the weld state is judged, if the weld edge is discontinuous, the corresponding standard image data and geometric features are labeled as false or missed welding; If the weld edge is continuous, the weld state is judged by the weld area S, if the weld area S is outside the standard weld area parameter range, the corresponding standard image data and geometric features are labeled as explosive welding; If the weld area S is within the standard weld area parameter range, the weld state is judged by the circularity e, if the circularity e is outside the standard weld circularity parameter range, the corresponding standard image data and geometric features are labeled as flying welding; If the circularity e is within the standard weld circularity parameter range, the corresponding standard image data and geometric features are labeled as normal; A neural network model for weld detection classification is constructed, which is used to extract deep features of the standard image data, structure the geometric features corresponding to the standard image data, fuse the extracted features and structured geometric features to form a feature vector, and classify the weld based on the feature vector; The neural network model is trained based on the labeled data set; The weld is detected and classified by the trained neural network model.

2. The method of claim 1, wherein the method further comprises: The standard image data is obtained by preprocessing, specifically including the following steps: A Roi detection area is set in the original image data, which includes the weld and its edge; Crop the original image data based on the Roi detection area to obtain welding point image data; Perform image enhancement processing on the welding point image data to obtain the standard image data.

3. The method of claim 2, wherein: The image enhancement processing includes image graying operation, image dilation operation and Gaussian filter processing, the image graying operation is implemented by using the following calculation method: ， wherein, represents a gray value of a (x, y) position in a gray scale image, represents a pixel value of a red channel of a (x, y) position in a color image, represents a red RGB weight coefficient, represents a pixel value of a green channel of a (x, y) position in a color image, represents a green RGB weight coefficient, represents a pixel value of a blue channel of a (x, y) position in a color image, represents a blue RGB weight coefficient; The image dilation operation is implemented by using the following calculation method: ， wherein, represents a pixel value at position (x, y) in the input image data, represents a structuring element, represents a structuring element an offset of each element in the structuring element, represents a dilated image a pixel value at position (x, y) in the dilated image; The Gaussian filter processing is implemented by using the following calculation method: ， wherein, represents the pixel value at position (x, y) in the input image data, is the weight of the Gaussian kernel at position (i, j), k denotes the radius of the Gaussian filter, the filtered new pixel value.

4. A method of visually inspecting and classifying a solder joint according to any one of claims 1 to 3, characterized in that: The neural network model at least includes: A feature extraction layer, the feature extraction layer includes a plurality of convolution layers for deep feature extraction of the standard image data, and the convolution layers are implemented by using the following calculation method: ， wherein, represents the value of the extracted feature at position (i, j) for channel k, represents the pixel value of the input image at position (i+m, j+n) for channel c, i, j represent the spatial position of the extracted feature, k represents the output channel index, represents the weight of the convolution kernel, m, n represent the spatial size of the convolution kernel, c is the channel index of the input image corresponding to the convolution kernel, k is the index of the convolution kernel, represents the kth convolution kernel, is a bias used to adjust the convolution result, represents an activation function; A fully connected layer, the fully connected layer is used for structuring the geometric features corresponding to the standard image data, and the fully connected layer is implemented by using the following calculation method: ， wherein, represents the output of the l-th layer, represents an activation function, represents a weight matrix, represents a bias; A feature fusion layer, the input end of the feature fusion layer is connected with the output ends of the feature extraction layer and the fully connected layer, and the feature fusion layer is used for fusing the features extracted by the feature extraction layer and the geometric features structured by the fully connected layer to form a feature vector; A feature classification layer, the feature classification layer is used for generating a class probability distribution by a Softmax activation function based on the feature vector, so as to classify the welding points.

5. A system for using a method of visual inspection and classification of a solder joint according to any one of claims 1 to 4, characterized in that, At least include: An image acquisition module, the image acquisition module at least includes an industrial camera used for acquiring dynamic welding point original image data in a production line in real time, and a light source system matched with the industrial camera and used for changing the reflection characteristics of the welding points, the light source system at least includes red, green and blue three-color LED lamps; An image preprocessing module, used for preprocessing the welding point original image data to obtain standard image data, extracting geometric features of the welding points in the standard image data by an edge detection algorithm, and constructing a labeled data set based on the standard image data and the corresponding geometric features; A model building module, used for building a neural network model for welding point detection and classification, and training the neural network model by the labeled data set to obtain a trained neural network model; A classification module, used for detecting and classifying welding points by the trained neural network model.

6. A system for vision-based inspection and classification of solder joints as recited in claim 5, wherein Further include: A control module, used for controlling the industrial camera to track the position of dynamic welding points in a production line in real time, controlling the angle and brightness of the red, green and blue three-color LED lamps in the light source system, and sending a detection report generated by the classification result of the classification module to a production control system of the production line.

Images