Welding spot defect detection method and device, electronic equipment and storage medium

Abstract

The invention provides a welding spot defect detection method. The welding spot defect detection method comprises the following steps: S1, acquiring point cloud data of a welding spot; s2, filtering and multi-view point cloud registration are carried out on the obtained point cloud data to obtain complete point cloud data of the welding spots; s3, determining the surface curvature of each point in the complete point cloud data, representing the surface curvature in the complete point cloud data in a color form, and converting the complete point cloud data of the welding spot into an image containing welding spot defect information; and S4, performing welding spot defect identification on the image containing the welding spot defect information by using a welding spot defect detection model based on deep learning to obtain a final detection result. The terminal welding spot defect detection method aims at terminal welding spot defect detection of the aerospace cable connector, can effectively avoid the light reflection problem which cannot be solved by a traditional detection method, and has the advantages of high identification precision and high detection efficiency.

Description

Technical Field

[0001] The present disclosure belongs to the field of machine vision and relates to a method, device, electronic equipment and storage medium for detecting solder joint defects of aerospace cable terminals. Background Art

[0002] Cables are an indispensable infrastructure in modern society, and they cover a wide range of applications. In the energy field, cables can be used to transmit signals between voltage and current conversion; in the communication field, cables can be used for optical fiber communication and network cable transmission; in the power transmission field, cables are mainly used for high-voltage cable transmission, railway power high-power transmission, etc. In addition, cables are also widely used in other industries, such as automatic control, aerospace, medical equipment, etc. Cable network is the central nervous system of many important systems, and its quality is an important factor in determining product performance and reliability. With the accelerated advancement of modern construction, the level of industrialization and automation has steadily improved, and the production of cable networks has shown the characteristics of large batches, short cycles, high integration, and strict requirements. The production and assembly process of cable networks is long, and there are a large number of measurement and inspection links. At present, it still mainly relies on visual assessment of defects, tape measurement of length, and manual recording of data. The detection time accounts for more than 15% of the total production cycle. The strength is high, the efficiency is low, the accuracy is poor, and the misdetection and missed detection occur from time to time. It is difficult to meet the production requirements of cable networks with multiple varieties, large batches, and high quality, which seriously affects the delivery progress of model products. The quality of solder joints will seriously affect the assembly quality, safety and life of the cable, so it is necessary to carefully inspect from multiple angles to ensure the quality of solder joints. However, current inspections are still mainly done manually, which affects the consistency of the quality of solder joints in aerospace cable products and has low inspection efficiency. There are mainly the following types of existing technologies for the inspection of solder joints in aerospace cable terminals:

[0003] 1. Manual measurement method

[0004] Workers need to use a fixture to hold the connector, then visually evaluate the amount of solder, and check whether there are solder joint defects such as cold solder joints, over solder joints, leaky solder joints, tip extraction, pinholes, etc., and record the data manually. However, since solder joints have many types of defects and a large number of them, ordinary manual visual inspections are easily interfered by other factors, have a high time cost, and have low detection efficiency.

[0005] 2. Solder spot defect detection technology based on 2D vision

[0006] 2D vision-based solder joint defect detection technology is a method that ultimately identifies solder joint defects through image acquisition and processing.

[0007] The 2D vision system mainly includes industrial cameras, optical lenses and light sources. This method first obtains a 2D image of the solder joint through the visual system, and then uses a multi-exposure image fusion algorithm to fuse the solder joint image in the process of image acquisition to solve the problem of uneven exposure. The fused image is then preprocessed and feature extracted. Then, the image is denoised, enhanced, threshold segmented, morphologically processed and edge contour extracted through image preprocessing. The shape features, texture features and directional gradient histogram feature parameters of the solder joint image are extracted. Finally, a neural network is used to classify defective solder joints and normal solder joint images. For example, Zhou Changjun et al. proposed a solder joint defect detection method based on improved ConvNeXt V2 and Faster-RCNN (CN116645342A). The 2D machine vision detection method loses the information of the third dimension, which easily leads to missed defects and cannot effectively calculate the amount of solder, which is inconsistent with the detection target.

[0008] 3. Solder spot defect detection technology based on 3D vision

[0009] For example, Hao Kuangrong et al. proposed an efficient and fast binocular 3D point cloud solder joint defect detection method (CN114092411A), and established a binocular vision system to efficiently and quickly collect visual information of solder joints. Figure 1 And Vision Figure 2 point cloud, obtain complete point cloud data through point cloud registration method, and finally predict the key area of ​​solder joints through defect detection technology based on fine-grained method, and extract the features of key areas for defect classification. At the same time, they also proposed a 3D point cloud solder joint defect detection method based on deep network (CN112614105A), which can effectively locate and detect solder joints in circuit boards, and help improve the accuracy of solder joint quality detection. However, the above method is only applicable to PCB solder joints. Summary of the invention

[0010] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0011] To this end, the present disclosure proposes a solder joint defect detection method, device, electronic device and storage medium, which takes the terminal solder joints of aerospace cable connectors as the object, can effectively avoid the reflection problem that traditional detection methods cannot solve, and at the same time has high recognition accuracy and high detection efficiency.

[0012] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0013] A first aspect of the present disclosure provides a method for detecting solder joint defects, which comprises the following steps:

[0014] Step S1, obtaining point cloud data of welding points;

[0015] Step S2, filtering the acquired point cloud data and performing multi-view point cloud registration to obtain complete point cloud data of the welding point;

[0016] Step S3, determining the surface curvature of each point in the complete point cloud data, and representing it in the complete point cloud data in the form of color, thereby converting the complete point cloud data of the solder joint into an image containing solder joint defect information;

[0017] Step S4: Use a deep learning-based solder joint defect detection model to perform solder joint defect recognition on the image containing solder joint defect information to obtain a final detection result.

[0018] In some embodiments, in step S1, a multi-projection method is used to capture multiple groups of point cloud data in different projection directions to form point cloud data of the welding point.

[0019] In some embodiments, in step S2, a straight-through filter and a statistical filter are used to filter the acquired point cloud data.

[0020] In some embodiments, in step S2, the multi-view point cloud registration of the acquired point cloud data includes: firstly performing a coarse registration of the multi-view point clouds based on a spanning tree, and then performing a fine registration of the multi-view point clouds based on an iterative closest point algorithm.

[0021] In some embodiments, in step S3, determining the surface curvature of each point in the complete point cloud data specifically includes:

[0022] For each scan point q in the complete point cloud data of the weld obtained in step S2, a search radius R is set, and all K adjacent points whose distance to the scan point q is less than or equal to R are searched. A local plane P is fitted based on all adjacent points of the scan point q. The local plane P is expressed as:

[0023]

[0024] in, is the normal vector of the local plane P, d is the distance from the local plane P to the origin of the global consistent coordinate system, p k is the distance between the scan point q and the kth neighbor point of the nearest neighbor;

[0025] The principal component analysis method is used to calculate the covariance matrix M of the normal vector of the local plane P. The calculation formula is as follows:

[0026]

[0027] Among them, p 0 is the center point of the local plane P;

[0028] Perform eigenvalue decomposition on the covariance matrix M to obtain the eigenvalues ​​λ of the covariance matrix M 0 ,λ 1 ,λ 2 , and the three eigenvalues ​​satisfy λ 0 ≤λ 1 ≤λ 2 , then the surface curvature of point q is calculated as follows:

[0029]

[0030] Where δ is the surface curvature at the scanning point q.

[0031] In some embodiments, after obtaining the surface curvature of each point in the point cloud data, the following normalization process is performed:

[0032]

[0033] Among them, δ' is the surface curvature of the normalized scanning point q, the normalized data range is [0,1], min(δ) and max(δ) are the minimum and maximum values ​​of the surface curvature of all points in the point cloud data, respectively.

[0034] The surface curvature of each point after normalization is represented in the complete point cloud data in the form of color, thereby converting the complete point cloud data of the welding point into an image containing welding point defect information.

[0035] In some embodiments, in step S4, the solder joint defect detection model adopts the YOLOv5s model.

[0036] A second aspect of the present disclosure provides a solder joint defect detection device, comprising:

[0037] The first module is used to obtain the point cloud data of the welding point;

[0038] The second module is used to filter the acquired point cloud data and perform multi-view point cloud registration to obtain complete point cloud data of the welding point;

[0039] The third module is used to determine the surface curvature of each point in the complete point cloud data, and represent it in the complete point cloud data in the form of color, so as to convert the complete point cloud data of the solder joint into an image containing solder joint defect information;

[0040] The fourth module is used to use a deep learning-based solder joint defect detection model to perform solder joint defect recognition on the image containing solder joint defect information to obtain a final detection result.

[0041] A third aspect of the present disclosure provides an electronic device, comprising:

[0042] at least one processor, and a memory communicatively coupled to the at least one processor;

[0043] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the solder joint defect detection method according to any embodiment of the first aspect of the present disclosure.

[0044] A third aspect of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the solder joint defect detection method described in any embodiment of the first aspect of the present disclosure.

[0045] The present disclosure has the following characteristics and beneficial effects:

[0046] The present invention first completes the point cloud data collection of solder joints under multiple projection angles; then, through point cloud preprocessing, fused point cloud data with higher accuracy and less noise is obtained, which is more conducive to the subsequent defect detection of solder joints; then, the surface curvature characteristics of solder joints are extracted using three-dimensional point cloud data, and the curvature characteristics are converted into two-dimensional image characteristics; finally, the solder joint defects are accurately identified using a neural network based on the two-dimensional image characteristics. The embodiment of the present invention can effectively avoid the reflection problem that cannot be solved by traditional detection methods, and at the same time, it has high recognition accuracy and high detection efficiency, and can complete the defect detection of solder joints of aerospace cable terminals, realizing the intelligence of the production and manufacturing detection link. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is an overall flow chart of a solder joint defect detection method provided by an embodiment of the present disclosure.

[0048] Figure 2 It is the point cloud data of the welding point obtained in step S1 of the detection method provided in the embodiment of the present disclosure.

[0049] Figure 3 It is the complete point cloud data of the welding point obtained after being processed in step S2 of the detection method provided in the embodiment of the present disclosure.

[0050] Figure 4 It is a three-dimensional representation of the curvature of the solder joint obtained after processing in step S3 of the detection method provided in the embodiment of the present disclosure.

[0051] Figure 5 This is a schematic diagram of the final result obtained through neural network training and testing.

[0052] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] On the contrary, the present application covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present application as defined by the claims. Further, in order to make the public have a better understanding of the present application, some specific details are described in detail in the detailed description of the present application below. Those skilled in the art can fully understand the present application without the description of these details.

[0055] See also Figure 1 The first aspect of the present disclosure provides a method for detecting solder joint defects, comprising:

[0056] Step S1, obtaining point cloud data of welding points;

[0057] Step S2, filtering the acquired point cloud data and performing multi-view point cloud registration to obtain complete point cloud data of the welding point;

[0058] Step S3, determining the surface curvature of each point in the complete point cloud data, and representing it in the complete point cloud data in the form of color, thereby converting the complete point cloud data of the solder joint into a two-dimensional image containing solder joint defect information;

[0059] Step S4: Use the deep learning-based solder joint defect detection model to identify solder joint defects in the image containing solder joint defect information to obtain the final detection result.

[0060] In some embodiments, step S1 obtains point cloud data of solder joints through a point cloud acquisition device. The point cloud acquisition device used in this embodiment has an imaging system of an industrial camera with a resolution of 5328×3040. The industrial camera is fixed by a bracket, and the lens is perpendicular to the horizontal plane. The bracket is equipped with a horizontally movable clamp, and a connector is fixed by the clamp. The point cloud data of the solder joint is obtained by horizontally moving the connector. In order to avoid eliminating image shadows and blind spots caused by a single projection direction, and at the same time solve the 3D reconstruction distortion problems such as single-sided flying points and holes that are easy to appear on the surface of the solder joint, this embodiment adopts a multi-projection method to shoot multiple groups of point cloud data in different projection directions, see Figure 2 .

[0061] In some embodiments, since the point cloud data collected by the point cloud collection device has noise points, and due to occlusion and other problems, incomplete data may not be obtained, in order to obtain higher quality point cloud data, it is necessary to perform preprocessing work in step S2, including:

[0062] Step S21: Point cloud filtering

[0063] Since laser scanning usually produces point cloud datasets with uneven density, and errors in measurement can also produce sparse outliers, the estimation of local point cloud features (such as the normal vector or curvature change rate at the sampling point) is computationally complex, which can lead to erroneous values, which in turn can cause failures in subsequent processing such as point cloud registration.

[0064] In order to remove noise, the embodiment of the present disclosure uses a straight-through filter and a statistical filter to filter the point cloud data obtained in step S1.

[0065] The function of the straight-through filter is to filter out points whose values ​​in the specified dimension direction are not in the given value range. The straight-through filter can filter out the background in the point cloud data, thereby retaining the point cloud data of the welding point;

[0066] The function of statistical filter is to remove obvious outliers, which are often introduced by measurement noise and are characterized by sparse distribution in space. Considering the characteristics of outliers, point clouds with point cloud density less than the set density threshold are defined as invalid point clouds and deleted. The specific steps are as follows:

[0067] Calculate the average distance from each point to its nearest k points after the straight-through filtering. The distances of all points in the point cloud should form a Gaussian distribution, whose shape is determined by the mean μ and the standard deviation σ. According to the mean μ and the standard deviation σ, the points outside the n standard deviation σ can be eliminated. Let the coordinates of the i-th point in the point cloud be P i (X i ,Y i ,Z i ), i∈[1,N], from this point to any point P j (X j ,Y j ,Z j ) is S ij , j∈[1,k] and j≠i, N is the number of points in the point cloud data whose distance to this point is less than or equal to S ij The total number of points, S ij The calculation formula is as follows:

[0068]

[0069] The formula for calculating the average distance between each point and any point, that is, the mean μ, is:

[0070]

[0071] The calculation formula of standard deviation σ is:

[0072]

[0073] Assume that the multiple of the standard deviation is std. When using the statistical filter to remove invalid point clouds, only two thresholds, k and std, need to be input. When the average distance of a point to the k points is within the standard range (μ-σ·std, μ+σ·std), the point is retained. If it is not within the range, it is defined as an outlier point and deleted.

[0074] Step S22: Multi-view point cloud registration

[0075] In order to avoid incomplete point cloud data due to occlusion and other problems, the disclosed embodiment uses a multi-view point cloud registration method to process the filtered point cloud to obtain high-precision complete point cloud data. Point cloud registration aligns point cloud data acquired in different coordinate systems to a reference coordinate system by establishing associations between point clouds. The multi-view point cloud registration method used in this embodiment is to first perform a coarse multi-view point cloud registration and then perform a fine multi-view point cloud registration. Specifically:

[0076] Step S221, perform coarse registration of multi-view point clouds based on the spanning tree. Coarse registration is a registration method performed when any initial relative position of the source point cloud and the target point cloud is completely unknown. The main purpose of this method is to quickly estimate a rough point cloud registration matrix when the initial conditions are unknown. For two groups of point clouds in any initial state, the two groups of point clouds are roughly aligned to provide initial values ​​for the rotation matrix R and the translation vector T. The core idea of ​​coarse registration of multi-view point clouds based on the spanning tree is to regard each group of point clouds to be registered as a corresponding node. First, a spanning tree is constructed according to the relationship between the nodes. Any two groups of point clouds can be directly or indirectly registered according to the spanning tree; then a reference coordinate system is selected, and the point cloud data of all views are registered to the reference coordinate system through the motion transformation between the corresponding point clouds. The specific steps are: select a set of the most complete point clouds as the root node of the spanning tree, set its coordinate system as the reference coordinate system, and then perform point cloud registration of the remaining point clouds with the root node in turn, so that the distance between the corresponding points tends to zero. If there are nodes that cannot be successfully registered with the root node, wait until all nodes are registered with the root node, randomly select a node directly connected to the root node as a new root node, and continue to register the point clouds of the two views until the spanning tree is established. Compared with the method of performing a multi-view rough registration based on the spanning tree by exhaustively enumerating the point cloud registration of the two views, the method used in this embodiment can greatly reduce the number of registrations, thereby greatly reducing the time consumption of rough registration.

[0077] Step S222: Perform fine registration of multiple views based on the iterative closest point algorithm (ICP algorithm). Fine registration is a more accurate and detailed registration based on the rough registration. This example uses the known initial transformation matrix to obtain a more accurate solution through the ICP algorithm. Specifically, the rotation and translation matrix R is constructed by calculating the distance between the corresponding points of the source point cloud and the target point cloud. T , through R TTransform the source point cloud and calculate the mean square error after transformation. If the mean square error meets the threshold condition, the algorithm ends, otherwise it continues to iterate until the error meets the threshold condition or the number of iterations ends. Figure 3 .

[0078] In some embodiments, step S3 aims to extract defects with obvious sudden changes in curvature, such as bubbles and pinholes, based on the curvature characteristics of the point cloud, and represent them in the complete point cloud data in the form of color, thereby converting the complete point cloud data of the solder joint into a two-dimensional image containing solder joint defect information. Compared with directly inputting the two-dimensional information of the solder joint, this can effectively avoid problems such as reflection and achieve fast and accurate defect identification. The specific implementation process of step S3 in this embodiment is as follows:

[0079] For each scan point q in the registered point cloud, set the search radius R, search for all K adjacent points whose distance is less than or equal to R, and then use the least squares method to fit the local plane P of these adjacent points. The local plane P can be expressed as:

[0080]

[0081] in, is the normal vector of the local plane P, d is the distance from the local plane P to the origin of the global consistent coordinate system, p k is the distance between the scan point q and its nearest neighbor, the kth neighbor point.

[0082] The covariance matrix of the normal vector of the local plane P is calculated using principal component analysis (PCA), and the calculation formula is as follows:

[0083]

[0084] Among them, p 0 is the center point of the local plane P.

[0085] Perform eigenvalue decomposition on the covariance matrix M in the above formula to obtain the eigenvalues ​​λ of the covariance matrix M 0 ,λ 1 ,λ 2 , the three eigenvalues ​​satisfy λ 0 ≤λ 1 ≤λ 2 , then the surface curvature of point q is calculated as follows:

[0086]

[0087] After obtaining the surface curvature of each point in the point cloud data, normalization is performed as follows:

[0088]

[0089] Among them, δ' is the surface curvature of point q after normalization, the normalized data range is [0,1], min(δ) and max(δ) are the minimum and maximum values ​​of the surface curvature of all points in the point cloud data, respectively.

[0090] In this embodiment, the normalized curvature data is represented in the form of color in the complete point cloud data, such as Figure 4 shown.

[0091] In some embodiments, in step S4, in order to ensure both the accuracy and speed of defect detection, the lightweight model YOLOv5s in the classic YOLOv5 series model is used as the solder joint defect detection model, and a large amount of two-dimensional solder joint image data containing defect information obtained by processing steps S1, S2, and S3 is made into a training set.

[0092] The YOLOv5s model consists of a backbone network (Backbone), a neck network (Neck), and a detection head (DetectHead). Among them, the YOLOv5s model uses a cross-stage partial network (CSP) as the backbone to extract image features, solving the problem of gradient information duplication in network optimization in other large convolutional neural network frameworks. The YOLOv5s model uses a path aggregation network (PAN) as the neck to aggregate the features of the feature image. The Detect Head of the YOLOv5s model is used to finally detect the feature image after feature aggregation, and generate a final output vector with two category probabilities of defective and non-defective, object scores, and bounding boxes.

[0093] Assume that the loss function of the YOLOv5 model is L, which is the positioning loss L box , classification loss L cls and confidence loss L obj The linear addition of is expressed as:

[0094] L=α box ×L box +α cls ×L cls +α obj ×L obj

[0095] Among them, α box is the weight coefficient of positioning loss, and its value is 0.05; α cls is the weight coefficient of classification loss, its value is 0.5; α obj is the weight coefficient of confidence loss, and its value is 0.05.

[0096] The Complete IoU loss algorithm is used as the positioning loss L of the YOLOv5 model. box , expressed as:

[0097]

[0098] In the formula, R IoU is the intersection and union ratio of the labeled rectangle and the predicted rectangle; Center of the label rectangle and the predicted rectangle center The Euclidean distance of the annotation rectangle and the predicted rectangle; dis is the diagonal distance between the smallest circumscribed rectangle of the annotation rectangle and the predicted rectangle; α is the weight coefficient; v is the consistency coefficient of the measured aspect ratio. The expression of α is:

[0099]

[0100] Finally, the YOLOv5s model can achieve high-precision solder joint defect detection, such as Figure 5 The y area shown in .

[0101] In summary, the solder joint defect detection method provided by the embodiment of the present disclosure first uses the point cloud acquisition device to complete the solder joint point cloud data acquisition under multiple projection angles; then, through point cloud preprocessing, the fused point cloud data with higher accuracy and less noise is obtained, which is more conducive to the subsequent solder joint defect detection; then, the surface curvature characteristics of the solder joint are extracted using the three-dimensional point cloud data, and the curvature characteristics are converted into two-dimensional image characteristics; finally, the solder joint defects are accurately identified using the neural network based on the two-dimensional image characteristics. The embodiment of the present disclosure can effectively avoid the reflection problem that cannot be solved by traditional detection methods, and at the same time, it has high recognition accuracy and high detection efficiency, and can complete the solder joint defect detection of aerospace cable terminals, realizing the intelligentization of the production and manufacturing detection link.

[0102] A second aspect of the present disclosure provides a solder joint defect detection device, comprising:

[0103] The first module is used to obtain the point cloud data of the welding point;

[0104] The second module is used to filter the acquired point cloud data and perform point cloud registration of multiple views to obtain complete point cloud data of the welding point;

[0105] The third module is used to determine the surface curvature of each point in the complete point cloud data and represent it in the complete point cloud data in the form of color, thereby converting the complete point cloud data of the solder joint into a two-dimensional image containing solder joint defect information;

[0106] The fourth module is used to use the deep learning-based solder joint defect detection model to identify solder joint defects in images containing solder joint defect information to obtain the final detection results.

[0107] It should be noted that the aforementioned explanations and descriptions of the embodiments of the solder joint defect detection method are also applicable to the solder joint defect detection device of this embodiment, and will not be repeated here.

[0108] In order to implement the above embodiment, the embodiment of the present disclosure further proposes a computer-readable storage medium, on which a computer program is stored. The program is executed by a processor to execute the solder joint defect detection method of the above embodiment.

[0109] Reference below Figure 6 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. It should be noted that the electronic device in the embodiments of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, servers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0110] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 to a random access memory (RAM) 103. In the RAM 103, various programs and data required for the operation of the electronic device are also stored. The processing device 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0111] Typically, the following devices may be connected to the I / O interface 105: an input device 106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; an output device 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 109. The communication device 109 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0112] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present embodiment includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 109, or installed from the storage device 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0113] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0114] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0115] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the solder joint defect detection method.

[0116] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, python, and conventional procedural programming languages, such as "C-" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0118] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0119] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0120] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0121] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0122] A person of ordinary skill in the art may understand that all or part of the steps carried out in the above-mentioned embodiment method may be completed by instructing the relevant hardware through a program, and the developed program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0123] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0124] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for detecting solder joint defects, characterized in that: The following steps are involved: Step S1, obtaining point cloud data of welding points; Step S2, filtering the acquired point cloud data and performing multi-view point cloud registration to obtain complete point cloud data of the welding point; Step S3, determining the surface curvature of each point in the complete point cloud data, and representing it in the complete point cloud data in the form of color, thereby converting the complete point cloud data of the solder joint into an image containing solder joint defect information; Step S4: Use a deep learning-based solder joint defect detection model to perform solder joint defect recognition on the image containing solder joint defect information to obtain a final detection result.

2. The solder joint defect detection method according to claim 1, characterized in that: In step S1, a multi-projection method is adopted to capture multiple groups of point cloud data in different projection directions to form point cloud data of the welding point.

3. The solder joint defect detection method according to claim 1, characterized in that: In step S2, a straight-through filter and a statistical filter are used to filter the acquired point cloud data.

4. The solder joint defect detection method according to claim 1, characterized in that: In step S2, the multi-view point cloud registration is performed on the acquired point cloud data, including: firstly, coarse registration of the multi-view point clouds is performed based on a spanning tree, and then fine registration of the multi-view point clouds is performed based on an iterative closest point algorithm.

5. The solder joint defect detection method according to claim 1, characterized in that: In step S3, the surface curvature of each point in the complete point cloud data is determined, specifically including: For each scan point q in the complete point cloud data of the weld obtained in step S2, a search radius R is set, and all K adjacent points whose distance to the scan point q is less than or equal to R are searched. A local plane P is fitted based on all adjacent points of the scan point q. The local plane P is expressed as: in, is the normal vector of the local plane P, d is the distance from the local plane P to the origin of the global consistent coordinate system, p k is the distance between the scan point q and the kth neighbor point of the nearest neighbor; The principal component analysis method is used to calculate the covariance matrix M of the normal vector of the local plane P. The calculation formula is as follows: Where p0 is the center point of the local plane P; The covariance matrix M is subjected to eigenvalue decomposition to obtain the eigenvalues ​​λ0, λ1, and λ2 of the covariance matrix M, and the three eigenvalues ​​satisfy λ0≤λ1≤λ2. Then the surface curvature of point q is calculated according to the following formula: Where δ is the surface curvature at the scanning point q.

6. The solder joint defect detection method according to claim 5, characterized in that: After obtaining the surface curvature of each point in the point cloud data, the following normalization process is performed: Among them, δ ‘ is the surface curvature of the normalized scanning point q. The normalized data range is [0,1]. min(δ) and max(δ) are the minimum and maximum surface curvatures of all points in the point cloud data, respectively. The surface curvature of each point after normalization is represented in the complete point cloud data in the form of color, thereby converting the complete point cloud data of the welding point into an image containing the welding point defect information.

7. The solder joint defect detection method according to claim 1, characterized in that: In step S4, the solder joint defect detection model adopts the YOLOv5s model.

8. A solder joint defect detection device, characterized in that: include: The first module is used to obtain the point cloud data of the welding point; The second module is used to filter the acquired point cloud data and perform multi-view point cloud registration to obtain complete point cloud data of the welding point; The third module is used to determine the surface curvature of each point in the complete point cloud data, and represent it in the complete point cloud data in the form of color, so as to convert the complete point cloud data of the solder joint into an image containing solder joint defect information; The fourth module is used to use a deep learning-based solder joint defect detection model to perform solder joint defect recognition on the image containing solder joint defect information to obtain a final detection result.

9. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the solder joint defect detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the solder joint defect detection method according to any one of claims 1 to 7.

Images