A welding spot defect detection method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a solder joint defect detection method, comprising the following steps: S1, acquiring point cloud data of a solder joint; S2, filtering and multi-view point cloud registration on the acquired point cloud data to obtain complete point cloud data of the solder joint; S3, determining the surface curvature of each point in the complete point cloud data, and representing it in the form of color in the complete point cloud data, so as to convert the complete point cloud data of the solder joint into an image containing solder joint defect information; S4, using a solder joint defect detection model based on deep learning to identify the solder joint defect of the image containing the solder joint defect information to obtain the final detection result. The present disclosure can effectively avoid the reflection problem that cannot be solved by the traditional detection method, and has the advantages of high recognition accuracy and high detection efficiency.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the field of machine vision, and relates to a solder joint defect detection method and device for an aerospace cable terminal, an electronic device and a storage medium. BACKGROUND

[0002] Cables are indispensable infrastructure in modern society, and their application fields are very wide. In the energy field, cables can be used to transmit signals between voltage and current conversion; in the communication field, cables can be used for optical fiber communication and network transmission; in the power transmission field, cables are mainly used for high-voltage cable power transmission, railway power transmission, etc. In addition, cables are also widely used in other industries, such as automatic control, aerospace, medical equipment, etc. As the central nervous system of many important systems, the quality of the cable network is an important factor in determining the performance and reliability of the product. With the acceleration of modern construction, the industrialization and automation level is steadily improving, and the production of the cable network presents the characteristics of large batch, short cycle, high integration, and strict requirements. The cable network production assembly process is long, and there are a large number of measurement and inspection links. At present, the defects are still mainly evaluated by visual assessment, the length is measured by tape measure, and the data is recorded manually. The detection time accounts for more than 15% of the total production cycle, which is labor-intensive, low in efficiency, poor in accuracy, and prone to missed detection, which cannot meet the production requirements of the cable network with multiple varieties, large batches, and high quality, and seriously affects the delivery schedule of the product. The quality of the solder joint will seriously affect the assembly quality, safety and service life of the cable, so it needs to be carefully inspected from multiple angles to ensure the quality of the solder joint. However, the current detection is still mainly manual, which affects the consistency of the solder joint quality of the aerospace cable product and is low in inspection efficiency. For the detection of the solder joint of the aerospace cable terminal, there are mainly the following existing technologies:

[0003] 1. Manual measurement method

[0004] The worker needs to use a clamp to hold the connector, then visually assess the solder amount, and check whether there are solder joint defects such as virtual welding, overwelding, missed welding, tip pulling, pinhole, etc., and manually record the data. However, since the solder joint has multiple defect types and a large number, ordinary visual inspection is easily disturbed by other factors, and has high time cost and low detection efficiency.

[0005] 2. Solder joint defect detection technology based on 2D vision

[0006] The solder joint defect detection technology based on 2D vision is a method of ultimately identifying solder joint defects through image acquisition and processing.

[0007] The 2D vision system mainly includes an industrial camera, an optical lens, and a light source. This method first obtains a 2D image of the welding spot through the vision system, and then, in the image acquisition process, in view of the phenomenon of uneven exposure, the welding spot image is fused by using a multi-exposure image fusion algorithm, and then the fused image is preprocessed and feature extracted. Then, the image is denoised, enhanced, threshold segmented, morphologically processed, and edge contour extracted through image preprocessing, the shape feature, texture feature, and direction gradient histogram feature parameters of the welding spot image are extracted, and finally the neural network is used to realize the classification of the defective welding spot and the normal welding spot image. For example, Zhou Changjun et al. proposed a welding spot defect detection method based on improved ConvNeXt V2 and Faster-RCNN (CN116645342A). The 2D machine vision detection method loses the information of the third dimension, is easy to cause defect omission, and cannot effectively calculate the soldering amount, which does not match the detection target.

[0008] 3. Welding spot defect detection technology based on 3D vision

[0009] For example, Hao Mingrong et al. proposed a high-efficiency and fast binocular 3D point cloud welding spot defect detection method (CN114092411A), established a binocular vision system to efficiently and quickly collect the visual Figure 1 and visual Figure 2 point cloud of the welding spot, obtained complete point cloud data through point cloud registration, and finally predicted the key area of the welding spot through a defect detection technology based on a fine-grained method, and extracted the features of the key area for defect classification. Meanwhile, they also proposed a 3D point cloud welding spot defect detection method based on a deep network (CN112614105A), which can effectively locate and detect the welding spot in the circuit board, and is helpful to improve the accuracy of welding spot quality detection. However, the above methods are only applicable to PCB welding spots. SUMMARY

[0010] The present disclosure aims to at least partially solve one of the technical problems in the related art.

[0011] To this end, the present disclosure proposes a welding spot defect detection method, device, electronic equipment and storage medium, which takes the terminal welding spot of an aerospace cable connector as the object, can effectively avoid the reflection problem that cannot be solved by traditional detection methods, and has high recognition accuracy and high detection efficiency.

[0012] To achieve the above purpose, the present disclosure adopts the following technical solutions:

[0013] The first aspect of the present disclosure provides a welding spot defect detection method, which comprises the following steps:

[0014] Step S1, obtaining point cloud data of the welding spot;

[0015] Step S2, filtering the obtained point cloud data and multi-view point cloud registration to obtain complete point cloud data of the solder joint;

[0016] Step S3, determining the surface curvature of each point in the complete point cloud data, and representing it in the form of color in the complete point cloud data, so as to convert the complete point cloud data of the solder joint into an image containing solder joint defect information;

[0017] Step S4, using a solder joint defect detection model based on deep learning to perform solder joint defect recognition on the image containing solder joint defect information to obtain a final detection result.

[0018] In some embodiments, in step S1, a multi-projection mode is adopted to shoot a plurality of groups of point cloud data in different projection directions to constitute the point cloud data of the solder joint.

[0019] In some embodiments, in step S2, a straight-through filter and a statistical filter are used to filter the obtained point cloud data.

[0020] In some embodiments, in step S2, the multi-view point cloud registration performed on the obtained point cloud data includes: first, performing coarse registration of multi-view point cloud based on a spanning tree, and then performing fine registration of multi-view point cloud based on an iterative closest point algorithm.

[0021] In some embodiments, in step S3, the surface curvature of each point in the complete point cloud data is determined, specifically including:

[0022] For each scanning point q in the complete point cloud data of the solder joint obtained in step S2, a search radius R is set, all K adjacent points with a distance less than or equal to R from the scanning point q are searched, and a local plane P is fitted according to all the adjacent points of the scanning point q, and the local plane P is expressed as:

[0023]

[0024] wherein, is the normal vector of the local plane P, d is the distance from the local plane P to the coordinate origin of the global consistent coordinate system, and p k is the distance between the scanning point q and the kth adjacent point closest to the scanning point q;

[0025] The covariance matrix M of the normal vector of the local plane P is calculated by using a principal component analysis method, and the calculation formula is as follows:

[0026]

[0027] wherein, p0 is the center point of the local plane P;

[0028] Eigenvalue decomposition is performed on the covariance matrix M to obtain eigenvalues λ0, λ1, λ2 of the covariance matrix M, and the three eigenvalues satisfy λ0≤λ1≤λ2, and the surface curvature of the point q is calculated according to the following formula:

[0029]

[0030] wherein δ is the surface curvature of the scanning point q.

[0031] In some embodiments, after obtaining the surface curvature of each point in the point cloud data, the following normalization processing is performed:

[0032]

[0033] wherein δ' is the surface curvature of the scanning point q after normalization processing, the normalized data range is [0, 1], and min(δ) and max(δ) are the minimum and maximum values of the surface curvature of all points in the point cloud data.

[0034] The surface curvature of each point after normalization processing is represented in the form of color in the complete point cloud data, so as to convert the complete point cloud data of the solder joint into an image containing solder joint defect information.

[0035] In some embodiments, in step S4, the solder joint defect detection model adopts a YOLOv5s model.

[0036] The second aspect of the present disclosure provides a solder joint defect detection device, comprising:

[0037] A first module is configured to obtain point cloud data of a solder joint.

[0038] A second module is configured to filter and perform multi-view point cloud registration on the obtained point cloud data to obtain complete point cloud data of the solder joint.

[0039] A third module is configured to determine the surface curvature of each point in the complete point cloud data, and represent it in the form of color in the complete point cloud data, so as to convert the complete point cloud data of the solder joint into an image containing solder joint defect information.

[0040] A fourth module is configured to use a deep learning-based solder joint defect detection model to perform solder joint defect recognition on the image containing solder joint defect information to obtain a final detection result.

[0041] The third aspect of the present disclosure provides an electronic device, comprising:

[0042] At least one processor, and a memory in communication connection with the at least one processor;

[0043] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the welding point defect detection method according to any one of the embodiments of the first aspect of the present disclosure.

[0044] The third aspect of the present disclosure provides a computer-readable storage medium, which stores computer instructions for causing the computer to execute the welding point defect detection method according to any one of the embodiments of the first aspect of the present disclosure.

[0045] The present disclosure has the following characteristics and beneficial effects:

[0046] The present disclosure first completes the welding point cloud data collection under multiple projection angles; then through point cloud preprocessing, fusion point cloud data with higher precision and fewer noise points are obtained, which is more conducive to subsequent welding point defect detection; subsequently, the surface curvature features of the welding points are extracted using three-dimensional point cloud data, and the curvature features are converted into two-dimensional image features; finally, the neural network accurately identifies the welding point defects according to the two-dimensional image features. The embodiment of the present disclosure can effectively avoid the reflection problem that cannot be solved by the traditional detection method, and has high recognition accuracy and high detection efficiency, and can complete the welding point defect detection of the spaceflight cable terminal, and realize the intelligentization of the production and manufacturing detection link. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a whole flowchart of a welding point defect detection method provided by the embodiment of the present disclosure.

[0048] Figure 2 is the point cloud data of the welding point obtained in step S1 of the detection method provided by the embodiment of the present disclosure.

[0049] Figure 3 is the complete point cloud data of the welding point obtained after step S2 of the detection method provided by the embodiment of the present disclosure.

[0050] Figure 4 is the welding point curvature three-dimensional representation diagram obtained after step S3 of the detection method provided by the embodiment of the present disclosure.

[0051] Figure 5 is a result schematic diagram finally obtained by neural network training and detection.

[0052] Figure 6 is a structural schematic diagram of an electronic device provided by the embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0054] On the contrary, the present application covers any alternative, modification, equivalent method and scheme defined by the claims within the essence and scope of the present application. Further, in order to make the public better understand the present application, some specific details are described in detail in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0055] Referring to Figure 1 The first aspect embodiment of the present disclosure provides a welding point defect detection method, comprising:

[0056] Step S1, acquiring point cloud data of a welding point;

[0057] Step S2, filtering and multi-view point cloud registration on the acquired point cloud data to obtain complete point cloud data of the welding point;

[0058] Step S3, determining the surface curvature of each point in the complete point cloud data, and representing it in the form of color in the complete point cloud data, so as to convert the complete point cloud data of the welding point into a two-dimensional image containing welding point defect information;

[0059] Step S4, using a welding point defect detection model based on deep learning to recognize the welding point defects of the image containing the welding point defect information to obtain the final detection result.

[0060] In some embodiments, the point cloud data of the welding point is acquired by a point cloud acquisition device in step S1. The imaging system of the point cloud acquisition device used in the present embodiment is an industrial camera with a resolution of 5328x3040. The industrial camera is fixed by a support, the lens is perpendicular to the horizontal plane, the support is provided with a clamping piece movable horizontally, a connector is fixed by the clamping piece, and the point cloud data of the welding point is obtained by horizontally moving the connector. In order to eliminate the image shadow and blind area caused by a single projection direction, and at the same time solve the 3D reconstruction distortion problems such as single-sided flying point and hole easily appearing on the surface of the welding point, a multi-projection mode is adopted in the present embodiment to shoot multiple groups of point cloud data in different projection directions. Referring to Figure 2 .

[0061] In some embodiments, since the point cloud data acquired by the point cloud acquisition device contains noise, and complete data may not be obtained due to occlusion and other problems, in order to obtain higher quality point cloud data, the preprocessing work of step S2 is needed, including:

[0062] Step S21, point cloud filtering

[0063] Since laser scanning usually produces point cloud data sets with uneven density, and errors in additional measurements also produce sparse outliers, estimating local point cloud features (such as normal vectors or curvature rates at sampling points) is computationally complex, which can lead to incorrect values and in turn cause later processing such as point cloud registration to fail.

[0064] To remove noise points, the embodiments of the present disclosure use a pass-through filter and a statistical filter to filter the point cloud data obtained in step S1. Wherein:

[0065] The pass-through filter filters out points whose values in the specified dimension direction are not in the given value range. By using the pass-through filter, the background in the point cloud data can be filtered out, thereby retaining the point cloud data of the welding spot;

[0066] The statistical filter removes obvious outliers. Outliers are often introduced by measurement noise, and their characteristics are sparse distribution in space. Considering the characteristics of outliers, points with a point cloud density less than a set density threshold are defined as invalid point clouds and are deleted. The specific steps are as follows:

[0067] Calculate the average distance of each point after pass-through filtering to its nearest k points. Then, the distances of all points in the point cloud should form a Gaussian distribution, the shape of which is determined by the mean μ and the standard deviation σ. According to the mean μ and the standard deviation σ, points outside n standard deviations σ can be removed. Let the coordinates of the i-th point in the point cloud be P i (X i ,Y i ,Z i ), i∈[1,N], the distance between this point and any point P j (X j ,Y j ,Z j ) is S ij , j∈[1,k] and j≠i, N is the total number of points in the point cloud data whose distance from this point is less than or equal to S ij , and the calculation formula of S ij is as follows:

[0068]

[0069] The formula for calculating the average value of the distance between each point and any point, i.e. the mean μ, is:

[0070]

[0071] The formula for calculating the standard deviation σ is:

[0072]

[0073] The standard deviation is set as std, and the two thresholds of k and std are input to remove the invalid point cloud by using the statistical filter. When the average distance of a point to k points is within the standard range (μ-σ·std, μ+σ·std), the point is retained, and if the point is not within the range, the point is defined as an outlier and is removed.

[0074] Step S22, multi-view point cloud registration

[0075] To avoid the incomplete point cloud data caused by occlusion and the like, the multi-view point cloud registration method is used to process the filtered point cloud to obtain high-precision complete point cloud data. The point cloud registration is to register the point cloud data obtained in different coordinate systems to a reference coordinate system by establishing the association between the point clouds. The multi-view point cloud registration method used in this embodiment is to first perform multi-view point cloud coarse registration and then perform multi-view point cloud fine registration. Specifically:

[0076] Step S221, coarse registration of multi-view point cloud based on spanning tree. The coarse registration is a registration method performed without knowing any initial relative position between the source point cloud and the target point cloud. The main purpose of this method is to quickly estimate a rough point cloud registration matrix in the case of unknown initial conditions. For two groups of point clouds in any initial state, a rotation matrix R and a translation vector T are provided as initial values to make the two groups of point clouds roughly aligned. The core idea of the coarse registration of multi-view point cloud based on spanning tree is to regard each group of point clouds to be registered as a corresponding node. First, a spanning tree is constructed according to the relationship between the nodes, and any two groups of point clouds can be directly or indirectly registered according to the spanning tree. Then, a reference coordinate system is selected, and the point cloud data of all views is registered to the reference coordinate system through the motion transformation between the corresponding point clouds. The specific steps are as follows: a group of most complete point clouds is selected as the root node of the spanning tree, and the coordinate system thereof is set as the reference coordinate system. Then, the remaining point clouds are sequentially registered with the root node in pairs of view point clouds, so that the distance of the corresponding points tends to zero. If there is a node that cannot be successfully registered with the root node, a node directly connected to the root node is randomly selected as a new root node after all nodes are registered with the root node, and the registration of the two view point clouds is continued, until the spanning tree is established. Compared with the method of performing multi-view coarse registration based on spanning tree by enumerating all pairs of view point clouds, the method used in this embodiment can greatly reduce the number of registrations, thereby greatly reducing the time consumption of coarse registration.

[0077] Step S222, fine registration of multi-view based on iterative closest point algorithm (ICP algorithm). The fine registration is a more accurate and detailed registration based on the coarse registration. In this embodiment, the ICP algorithm is used to obtain a more accurate solution based on the known initial transformation matrix. Specifically, a rotation and translation matrix R is constructed by calculating the distance between the corresponding points of the source point cloud and the target point cloud. T T ​For the source point cloud transformation, the mean square error after transformation is calculated. If the mean square error meets the threshold condition, the algorithm ends, otherwise it continues to repeat iteration until the error meets the threshold condition or the iteration number is terminated. The final registration result of this embodiment is shown in Figure 3 .

[0078] In some embodiments, step S3 aims to extract defects such as bubbles and pinholes with obvious curvature mutations according to the curvature characteristics of the point cloud, and represent them in the form of color in the complete point cloud data, thereby converting the complete point cloud data of the welding spot into a two-dimensional image containing welding spot defect information. In this way, compared with directly inputting two-dimensional information of the welding spot, the problem of reflection and the like can be effectively avoided, and rapid and accurate defect identification can be achieved. The specific implementation process of step S3 of this embodiment is as follows:

[0079] For each scanning point q in the registered point cloud, a search radius R is set, all K neighboring points with a distance less than or equal to R are searched, and then the local plane P of these neighboring points is fitted by using the least square method. The local plane P can be represented as:

[0080]

[0081] wherein, is the normal vector of the local plane P, d is the distance from the local plane P to the coordinate origin of the global consistent coordinate system, and p k is the distance between the scanning point q and the nearest neighboring kth neighboring point.

[0082] The covariance matrix of the normal vector of the local plane P is calculated by using principal component analysis (PCA), and the calculation formula is as follows:

[0083]

[0084] wherein, p0 is the center point of the local plane P.

[0085] Eigenvalue decomposition is performed on the covariance matrix M in the above formula, and each eigenvalue λ0, λ1, λ2 of the covariance matrix M is obtained. The three eigenvalues satisfy λ0≤λ1≤λ2, and the surface curvature of the point q is calculated according to the following formula:

[0086]

[0087] After obtaining the surface curvature of each point in the point cloud data, normalization processing is performed as follows:

[0088]

[0089] Wherein, δ' is the surface curvature of the q-point after normalization processing, the normalized data range is [0, 1], min(δ), max(δ) are the minimum and maximum of the surface curvature of all points in the point cloud data.

[0090] The normalized curvature data is represented in the form of color in the complete point cloud data as shown in the following figure. Figure 4

[0091] In some embodiments, in step S4, in order to ensure the accuracy and speed of defect detection at the same time, a lightweight model YOLOv5s in the classic YOLOv5 series model is used as a solder defect detection model, and a large amount of two-dimensional solder image data containing defect information obtained through steps S1, S2 and S3 is used as a training set.

[0092] The YOLOv5s model is composed of a backbone network (Backbone), a neck network (Neck) and a detection head (DetectHead). Among them, the YOLOv5s model uses a cross-stage local network (Cross Stage Partial Networks, CSP) as the Backbone to extract image features, which solves the problem of repeated gradient information optimization in the network optimization of other large convolutional neural network frameworks Backbone. The YOLOv5s model uses a path aggregation network (Path Aggregation Network, PAN) as the Neck to aggregate the features of the feature image. The Detect Head of the YOLOv5s model is used to finally detect the feature- aggregated feature image, and generate the final output vector with two categories of probability of having defects and no defects, object score and bounding box.

[0093] Let the loss function of the YOLOv5 model be L, which is the linear addition of the positioning loss L box , the classification loss L cls and the confidence loss L obj , and the expression is:

[0094] L = α box × L box + α cls × L cls + α obj × L obj

[0095] Wherein, α box is the weight coefficient of the positioning loss, whose value is 0.05; α cls is the weight coefficient of the classification loss, whose value is 0.5; α obj is the weight coefficient of the confidence loss, whose value is 0.05.

[0096] ​The Complete IoU loss algorithm is used as the positioning loss L of the YOLOv5 model box , and is expressed as:

[0097]

[0098] In the formula, R IoU is the intersection over union of the labeled rectangle and the predicted rectangle; is the Euclidean distance between the center of the labeled rectangle and the center of the predicted rectangle; dis is the diagonal distance of the minimum circumscribed rectangle of the labeled rectangle and the predicted rectangle; α is a weight coefficient; and v is a consistency coefficient for measuring the aspect ratio. The expression of α is:

[0099]

[0100] Finally, the YOLOv5s model can achieve high-precision welding point defect detection, such as the y region shown in FIG. 8. Figure 5

[0101] In summary, the welding point defect detection method provided by the embodiments of the present disclosure first completes the welding point point cloud data collection under multiple projection angles by using a point cloud collection device; then, through point cloud preprocessing, fusion point cloud data with higher precision and fewer noise points are obtained, which is more conducive to subsequent welding point defect detection; subsequently, the surface curvature features of the welding point are extracted by using the three-dimensional point cloud data, and the curvature features are converted into two-dimensional image features; finally, the neural network accurately identifies the welding point defects according to the two-dimensional image features. The embodiments of the present disclosure can effectively avoid the reflection problem that cannot be solved by traditional detection methods, and have high recognition accuracy and high detection efficiency, and can complete the welding point defect detection of spaceflight cable terminals, and realize the intelligentization of the production and manufacturing detection link.

[0102] The welding point defect detection device provided by the second aspect of the embodiments of the present disclosure comprises:

[0103] A first module is configured to obtain point cloud data of a welding point.

[0104] A second module is configured to filter and perform multi-view point cloud registration on the obtained point cloud data to obtain complete point cloud data of the welding point.

[0105] A third module is configured to determine the surface curvature of each point in the complete point cloud data, and represent the surface curvature in the complete point cloud data in the form of color, so as to convert the complete point cloud data of the welding point into a two-dimensional image containing welding point defect information.

[0106] A fourth module is configured to use a welding point defect detection model based on deep learning to perform welding point defect recognition on the image containing the welding point defect information, and obtain a final detection result. ​

[0107] It should be noted that the foregoing embodiment of the solder joint defect detection method is also applicable to the solder joint defect detection device of the present embodiment, and will not be repeated here.

[0108] To implement the above-mentioned embodiments, the present embodiment also proposes a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to execute the solder joint defect detection method of the above-mentioned embodiments.

[0109] Reference will now be made to the drawings, in which Figure 6 which shows a structural schematic diagram of an electronic device suitable for implementing the embodiments of the present disclosure. It should be noted that the electronic device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet PCs), PMPs (Portable Multimedia Players), vehicle terminals (such as vehicle navigation terminals), and the like, as well as fixed terminals such as digital TVs, desktop computers, servers, and the like. Figure 6 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0110] As shown in Figure 6 , the electronic device can include a processing device (such as a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 102 or loaded from a storage device 108 into a random access memory (RAM) 103. In the RAM 103, various programs and data required for the operation of the electronic device are also stored. The processing device 101, the ROM 102, and the RAM 103 are connected to each other through a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0111] Generally, the following devices can be connected to the I / O interface 105: input devices 106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, and the like; output devices 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 108 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 109. The communication devices 109 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device with various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices can be implemented or provided instead.

[0112] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the present embodiments include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication device 109, or installed from the storage device 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the methods of the present disclosure are executed.

[0113] It should be noted that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take on many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer a program for use by or in connection with an instruction execution system, apparatus or device. Program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination of the above.

[0114] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and not be assembled into the electronic device.

[0115] The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to execute the above-mentioned solder defect detection method.

[0116] Computer program code for carrying out operations of the present disclosure can be written in any one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++, Python, conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0117] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0118] In addition, the terms "first", "second" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0119] Any process or method descriptions or descriptions of the flow diagrams in the specification or elsewhere in this document, can be understood as representing the steps of the code of the modules, segments or portions of the code for implementing specific logic functions or steps in the process, and the scope of the preferred embodiments of the present application includes additional implementation in which the steps are performed in different order, including an essentially simultaneous performance of the functions according to the involved functions, or in reverse order, which should be understood by those skilled in the art of the embodiments of the present application.

[0120] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer- readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (a magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0121] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0122] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the developed programs can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0123] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0124] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method of detecting a solder joint defect, characterized by, The method comprises the following steps: Step S1, acquiring point cloud data of a welding spot; Step S2, filtering and multi-view point cloud registration are performed on the acquired point cloud data to obtain complete point cloud data of the welding spot; Step S3, the surface curvature of each point in the complete point cloud data is determined, and the following normalization processing is performed: Wherein, δ is the surface curvature of the scanning point q, δ' is the surface curvature of the scanning point q after normalization processing, the data range of the normalized data is [0, 1], min(δ) and max(δ) are the minimum value and the maximum value of the surface curvature of all points in the point cloud data respectively; The surface curvature of each point after normalization processing is represented in the form of color in the complete point cloud data, so as to convert the complete point cloud data of the welding spot into an image containing welding spot defect information; Step S4, a welding spot defect detection model based on deep learning is used to perform welding spot defect identification on the image containing welding spot defect information to obtain a final detection result.

2. The inspection method of claim 1, wherein In step S1, multiple projection modes are adopted to shoot multiple groups of point cloud data in different projection directions to form the point cloud data of the welding spot.

3. The inspection method of claim 1, wherein In step S2, a straight-through filter and a statistical filter are used to filter the acquired point cloud data.

4. The inspection method of claim 1, wherein In step S2, the multi-view point cloud registration performed on the acquired point cloud data comprises the following steps: firstly, coarse registration of multi-view point cloud is performed based on a spanning tree, and then fine registration of multi-view point cloud is performed based on an iterative closest point algorithm.

5. The inspection method of claim 1, wherein In step S3, the surface curvature of each point in the complete point cloud data is determined, and specifically comprises the following steps: For each scanning point q in the complete point cloud data of the welding spot obtained in step S2, a search radius R is set, all K adjacent points with a distance less than or equal to R from the scanning point q are searched, a local plane P is fitted according to all adjacent points of the scanning point q, and the local plane P is represented as: wherein, is a normal vector of the local plane P, d is a distance of the local plane P to the coordinate origin of the global uniform coordinate system, p k is a distance of the scanning point q to the kth nearest neighbor point A principal component analysis method is used to calculate a covariance matrix M of the normal vector of the local plane P, and the calculation formula is as follows: Wherein, p0 is the center point of the local plane P; The covariance matrix M is subjected to eigenvalue decomposition to obtain each eigenvalue λ0, λ1, λ2 of the covariance matrix M, and the three eigenvalues satisfy λ0≤λ1≤λ2, and the surface curvature of the point q is calculated according to the following formula: Wherein, δ is the surface curvature of the scanning point q.

6. The inspection method of claim 1, wherein In step S4, the welding spot defect detection model adopts a YOLOv5s model.

7. A solder joint defect detection apparatus characterized by comprising: Comprise: The first module is used for acquiring point cloud data of a welding spot; The second module is used for filtering and multi-view point cloud registration on the acquired point cloud data to obtain complete point cloud data of the welding spot; The third module is used for determining the surface curvature of each point in the complete point cloud data, and the following normalization processing is performed: Wherein, δ is the surface curvature of the scanning point q, δ' is the surface curvature of the scanning point q after normalization processing, the data range of the normalized data is [0, 1], min(δ) and max(δ) are the minimum value and the maximum value of the surface curvature of all points in the point cloud data respectively; The surface curvature of each point after normalization processing is represented in the form of color in the complete point cloud data, so as to convert the complete point cloud data of the welding spot into an image containing welding spot defect information; A fourth module is configured to utilize the deep learning-based welding point defect detection model to perform welding point defect identification on the image containing the welding point defect information, and obtain a final detection result.

8. An electronic device, comprising: The method comprises the following steps: at least one processor, and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to perform the welding point defect detection method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the computer to perform the welding point defect detection method of any one of claims 1-6.

Images