Ar-based circuit board solder assembly quality analysis method and system

Abstract

The application provides an AR-based circuit board welding assembly quality analysis method and system, a storage medium and electronic equipment, and relates to the technical field of circuit board welding assembly quality analysis.In the application, a circuit board image is collected based on an AR device;position and category information of electronic components on the circuit board image are identified according to a lightweight electronic component detection model deployed on the AR device;based on a pre-constructed assembly template, at least assembly quality analysis results of circuit board mis-welding and missed welding are obtained according to the identification results of the electronic components; and the assembly quality analysis results are displayed in a mixed reality space based on the AR device.The target detection technology based on deep learning is adopted to overcome the poor robustness of electronic component detection in traditional image processing; and the lightweight electronic component detection model is directly deployed on the AR device, thereby avoiding time delay and data transmission when transmitting to the background.

Description

Technical Field

[0001] This invention relates to the field of circuit board soldering assembly quality analysis technology, and specifically to an AR-based circuit board soldering assembly quality analysis method, system, storage medium, and electronic device. Background Technology

[0002] Circuit boards are an important basic component in the assembly of electromechanical systems. With the rapid development of microelectronics and computer technology, electronic components have become more miniaturized and diversified. Circuit boards with hundreds of complex soldering points are prone to missoldering and missing soldering, which can lead to accidents and economic losses. Therefore, it is necessary to perform assembly quality analysis on the soldered circuit boards to ensure product performance.

[0003] Early detection of incorrect or missing solder joints on circuit boards relied mainly on manual visual inspection. However, this method is highly dependent on the experience of the inspectors and is prone to missed detection due to visual fatigue. Therefore, there is a real need to develop automated methods for detecting incorrect or missing solder joints based on computer vision.

[0004] Electronic component detection algorithms are mainly divided into two categories: one is based on traditional image classification methods, which first input an image, then extract features from the image through a complex process, and finally classify the target based on the extracted features. The other is based on deep learning methods, which first input an image, then automatically extract features through a convolutional neural network to generate feature maps, and finally locate and classify the target based on the feature maps.

[0005] However, detection methods based on traditional image processing are not robust and are prone to false positives and false negatives, while deep learning-based methods require high computing power and have poor real-time performance. Furthermore, current methods that use industrial cameras to photograph circuit boards are not conducive to displaying assembly quality analysis results. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides an AR-based method, system, storage medium, and electronic device for analyzing the welding and assembly quality of circuit boards, solving the technical problem of the inability to simultaneously achieve robustness and real-time performance.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] An AR-based method for analyzing the soldering and assembly quality of circuit boards includes:

[0011] AR device-based circuit board image acquisition;

[0012] Based on the lightweight electronic component detection model deployed on the AR device, the location and category information of electronic components on the circuit board image are identified;

[0013] Based on the identification results of the electronic components, and using a pre-built assembly template, at least the assembly quality analysis results of mis-soldering and missing soldering on the circuit board are obtained;

[0014] The assembly quality analysis results are displayed in a mixed reality space using the AR device.

[0015] Preferably, the AR-based circuit board soldering assembly quality analysis method further includes: issuing an alarm for abnormal assembly quality analysis results; and at least recording the abnormal assembly quality analysis results in the server's device archive.

[0016] Preferably, the lightweight electronic component detection model adopts an end-to-end YOLO V4 network model, and the feature extraction network of the YOLO V4 network model is a lightweight Mobilenet network.

[0017] Preferably, the process of obtaining the assembly quality analysis results of mis-soldering and missing soldering on the circuit board includes:

[0018] Based on the principle of minimizing Euclidean distance, the identification results of the electronic components are matched with the electronic components of the assembly template; if the detected label is inconsistent with the assembly template, it is judged as mis-soldering; if the assembly template contains electronic components but no corresponding label is detected, it is judged as missing soldering.

[0019] Preferably, after correcting the identification results of the electronic components, the assembly quality analysis results of the circuit board with missoldering and missing soldering are obtained based on the corrected identification results.

[0020] Preferably, the correction of the identification result of the electronic component includes:

[0021] The pose information of the circuit board image is obtained based on the auxiliary positioning marks around the positioning frame; the positioning frame is the area where the circuit board is placed horizontally.

[0022] Based on the pose information, the planar coordinates of each electronic component on the circuit board are obtained using affine transformation;

[0023] The shooting angle of the circuit board image is unified based on the planar coordinates of the electronic components.

[0024] An AR-based circuit board soldering assembly quality analysis system includes:

[0025] The acquisition module is used to acquire circuit board images based on AR devices;

[0026] The identification module is used to identify the location and category information of electronic components on the circuit board image based on a lightweight electronic component detection model deployed on the AR device.

[0027] The analysis module is used to obtain assembly quality analysis results of at least mis-soldering and missing soldering on the circuit board based on the identification results of the electronic components and a pre-built assembly template.

[0028] The display module is used to display the analysis results of the assembly quality in a mixed reality space based on the AR device.

[0029] Preferably, the circuit board welding assembly quality analysis system further includes:

[0030] The alarm module is used to issue alerts for abnormal assembly quality analysis results.

[0031] The recording module shall at least record the abnormal assembly quality analysis results in the server's device archive.

[0032] A storage medium storing a computer program for AR-based circuit board solder assembly quality analysis, wherein the computer program causes a computer to execute the circuit board solder assembly quality analysis method as described above.

[0033] An electronic device, comprising:

[0034] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing circuit board solder assembly quality analysis as described above.

[0035] (III) Beneficial Effects

[0036] This invention provides an AR-based method, system, storage medium, and electronic device for analyzing the soldering and assembly quality of circuit boards. Compared with existing technologies, it has the following advantages:

[0037] In this invention, circuit board images are acquired using an AR device; the location and category information of electronic components on the circuit board image are identified based on a lightweight electronic component detection model deployed on the AR device; based on the identification results of the electronic components and a pre-built assembly template, at least the assembly quality analysis results of mis-soldering and missing soldering on the circuit board are obtained; and the assembly quality analysis results are displayed in a mixed reality space using the AR device. Deep learning-based target detection technology is employed to overcome the poor robustness of electronic component detection in traditional image processing; the lightweight electronic component detection model is directly deployed on the AR device, avoiding time delays and data transmission to the backend. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating an AR-based circuit board soldering assembly quality analysis method provided in an embodiment of the present invention;

[0040] Figure 2 A block diagram illustrating an AR-based circuit board soldering assembly quality analysis method provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of an electronic component detection model based on Mobilenet, provided in an embodiment of the present invention.

[0042] Figure 4 This invention provides a schematic diagram of a circuit board image.

[0043] Figure 5 This is a schematic diagram of a circuit board for viewing AR glasses, provided as an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] This application provides an AR-based method, system, storage medium, and electronic device for analyzing the welding and assembly quality of circuit boards, solving the technical problem of being unable to balance robustness and real-time performance.

[0046] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0047] The circuit board soldering assembly quality analysis method provided in this invention is based on an AR device front-end (such as AR glasses) and a back-end server. By directly deploying a lightweight electronic component detection model onto the AR glasses, this architecture reduces both the cumbersome process of transmitting image data to the back-end server and the time consumption for soldering assembly quality analysis.

[0048] like Figure 1 As shown, the technical solution can be summarized as follows: acquiring circuit board images based on AR glasses; extracting features of the type and location of electronic components on the circuit board using a lightweight electronic component detection algorithm running on the AR glasses; analyzing the electronic component identification results based on circuit board assembly to discover assembly problems such as incorrect soldering and missing soldering; and using the AR glasses to display and alert on the analysis results of incorrect soldering and missing soldering in the mixed reality space, and uploading them to the server to be recorded in the software's device archive.

[0049] By employing a lightweight end-to-end target detection algorithm, the robustness of deep learning detection is achieved while reducing computational deployment requirements and increasing detection speed. In addition, displaying assembly quality analysis results using AR glasses facilitates guidance for welding personnel to carry out repairs.

[0050] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0051] Example:

[0052] like Figure 2 As shown, this embodiment of the invention provides an AR-based method for analyzing the soldering and assembly quality of circuit boards, including:

[0053] S1. Acquire circuit board images using AR devices;

[0054] S2. Based on the lightweight electronic component detection model deployed on the AR device, identify the location and category information of electronic components on the circuit board image;

[0055] S3. Based on the identification results of the electronic components and the pre-constructed assembly template, obtain at least the assembly quality analysis results of mis-soldering and missing soldering on the circuit board;

[0056] S4. Display the assembly quality analysis results in a mixed reality space based on the AR device;

[0057] S5. Issue an alarm for abnormal assembly quality analysis results;

[0058] S6. At least record the abnormal assembly quality analysis results in the server's device archive.

[0059] The embodiments of this invention effectively solve the problem of visual fatigue in the current manual circuit board welding and assembly quality analysis; they adopt deep learning-based target detection technology to overcome the problem of poor robustness of electronic component detection in traditional image processing; they directly deploy lightweight electronic component detection models on AR devices to avoid time delays and data transmission to the backend; and the interactive display and problem recording based on AR devices help improve the user experience.

[0060] The following will detail each step of the above technical solution:

[0061] In step S1, the circuit board image is acquired based on the AR device.

[0062] In this step, the camera equipped with AR glasses uses high-definition video monitoring technology to achieve high-definition acquisition, encoding, transmission, storage, and display of video image information.

[0063] In step S2, the location and category information of electronic components on the circuit board image are identified based on the lightweight electronic component detection model deployed on the AR device.

[0064] It is easy to understand that misassembly and omission of parts are common assembly problems in electrical assembly processes. In order to achieve automatic identification of abnormal assembly in electrical assembly processes, it is first necessary to extract the assembly features of electrical assembly processes. Among them, the detection of electronic components such as resistors, capacitors, and chips is the key, which requires obtaining the type and location information of electronic components.

[0065] Because typical circuit boards are densely packed with electronic components of varying sizes, smaller components present significant detection challenges. Furthermore, to ensure a good user experience, real-time feedback on electronic component detection is crucial. Therefore, this step utilizes the end-to-end YOLO V4 algorithm for electronic component detection. Hierarchical feature utilization ensures universality for components of different sizes, and the end-to-end algorithm design meets the detection requirements. Additionally, the lightweight MobileNet algorithm replaces the feature extraction network in YOLO V4, further improving detection speed while reducing the model's hardware requirements. All of these factors facilitate the deployment of the detection model on front-end AR glasses.

[0066] The structure of the lightweight electronic component detection model is as follows: Figure 3 As shown, it specifically includes:

[0067] First, the circuit board image captured by the AR glasses in step S1 is scaled to a specified size and input into the network. Then, the MobileNet network is used for hierarchical feature extraction. Since MobileNet uses depthwise separable convolutions instead of ordinary convolutions, it significantly reduces the computational load. Finally, referring to the YOLO V4 algorithm, the upper-layer features extracted by MobileNet are upsampled and fused with the lower-layer features. This fusion structure is used three times consecutively to obtain the corresponding three-layer features. All three layers of features are then detected to obtain the location and category information of electronic components.

[0068] In step S3, based on the identification results of the electronic components and the pre-constructed assembly template, at least the assembly quality analysis results of mis-soldering and missing soldering on the circuit board are obtained.

[0069] Specifically, before performing the assembly quality analysis in step S3, this embodiment of the invention further includes correcting the identification results of the electronic components, and then obtaining the assembly quality analysis results of the circuit board for mis-soldering and missing soldering based on the corrected identification results.

[0070] This is because the electronic components on the circuit board do not present any regular arrangement, so it is necessary to use the spatial positioning capability of AR glasses, combined with some coordinate transformation calculations between 3D and 2D projection, to determine the accurate location of mis-soldering and missing solder joints.

[0071] Specifically, the correction of the identification results of the electronic components includes:

[0072] S10. Based on the auxiliary positioning marks around the positioning frame (markers of a certain special shape, such as...) Figure 4 As shown, the material can be a triangular or circular non-reflective sticker, etc., to obtain the pose information of the circuit board image; the positioning frame is the area where the circuit board is placed horizontally.

[0073] S20. Based on the pose information, obtain the planar coordinates of each electronic component on the circuit board using affine transformation.

[0074] like Figure 5 As shown, different staff members wear AR headsets to inspect circuit boards at varying angles and distances. Affine transformation can counteract the resulting image distortion and scaling.

[0075] For example, the 3D to 2D projective coordinate transformation matrix operation is as follows:

[0076]

[0077] Noted as Z 2D =E×Z′ 3DSince the thickness of the circuit board is almost negligible, z′3 can be considered as 0. Therefore, E is reduced from a 2x3 matrix to a 2x1 matrix, denoted as Z. 2D .

[0078] S30. Based on the planar coordinates of the electronic components, unify the shooting angle of the circuit board image.

[0079] After obtaining the planar coordinates of each electronic component on the circuit board, the circuit board is corrected using the more obvious electronic components on the circuit board (such as the largest electronic component) to avoid the circuit board in the acquired image being upside down or left and right compared to the circuit board in the template library.

[0080] The process of obtaining the assembly quality analysis results of the circuit board for mis-soldering and missing solder joints after obtaining the corrected identification results includes:

[0081] Based on the principle of minimizing Euclidean distance, the identification results of the electronic components are matched with the electronic components of the assembly template; if the detected label is inconsistent with the assembly template, it is judged as mis-soldering; if the assembly template contains electronic components but no corresponding label is detected, it is judged as missing soldering.

[0082] In step S4, the assembly quality analysis results are displayed in a mixed reality space based on the AR device.

[0083] In step S5, an alarm is triggered based on the abnormal assembly quality analysis results.

[0084] Steps S4 to S5 can utilize the sensor technology and mixed reality technology configured in AR glasses to display and alert on the identified circuit board soldering anomalies.

[0085] In step S6, the abnormal assembly quality analysis results are recorded at least in the server's device archive.

[0086] This step actually provides an information-based means of equipment management, which can not only effectively solve the information management problems of each piece of equipment, but also speed up the efficiency of handling equipment (circuit board) assembly abnormalities.

[0087] For example, in a specific application, this embodiment of the invention can be used for software development based on AR devices and the Unity3D development platform. Specifically, the AR glasses front-end system is developed using the C# language, and the program is ultimately deployed to the AR glasses. The Microsoft Unity3D plugin MRTK (Mixed Reality Toolkit) is used, which provides components and functions to accelerate application development, including providing an input system and building block for spatial interaction and UI interfaces, and enabling rapid prototyping within the editor. The client-side is primarily developed using the MRTK plugin in Unity3D. After the application is created, it is published to the UWP platform, and then deployed to the AR glasses using Visual Studio 2019.

[0088] This invention provides an AR-based circuit board soldering assembly quality analysis system, comprising:

[0089] The acquisition module is used to acquire circuit board images based on AR devices;

[0090] The identification module is used to identify the location and category information of electronic components on the circuit board image based on a lightweight electronic component detection model deployed on the AR device.

[0091] The analysis module is used to obtain assembly quality analysis results of at least mis-soldering and missing soldering on the circuit board based on the identification results of the electronic components and a pre-built assembly template.

[0092] The display module is used to display the analysis results of the assembly quality in a mixed reality space based on the AR device;

[0093] The alarm module is used to issue alerts for abnormal assembly quality analysis results.

[0094] The recording module shall at least record the abnormal assembly quality analysis results in the server's device archive.

[0095] This invention provides a storage medium storing a computer program for AR-based circuit board soldering assembly quality analysis, wherein the computer program causes a computer to execute the circuit board soldering assembly quality analysis method as described above.

[0096] This invention provides an electronic device, comprising:

[0097] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing circuit board solder assembly quality analysis as described above.

[0098] It is understood that the AR-based circuit board welding assembly quality analysis system, storage medium, and electronic device provided in the embodiments of the present invention correspond to the AR-based circuit board welding assembly quality analysis method provided in the embodiments of the present invention. The explanations, examples, and beneficial effects of the relevant contents can be referred to the corresponding parts of the circuit board welding assembly quality analysis method, and will not be repeated here.

[0099] In summary, compared with existing technologies, it has the following beneficial effects:

[0100] 1. The embodiments of the present invention effectively solve the problem of visual fatigue in the current analysis of the welding and assembly quality of artificial circuit boards.

[0101] 2. Deep learning-based target detection technology was adopted to overcome the problem of poor robustness in electronic component detection in traditional image processing.

[0102] 3. Lightweight detection model deployment was achieved using the Mobilenet base, which allows the detection model to be deployed directly to AR glasses, avoiding time delays and data transmission to the backend.

[0103] 4. Interactive displays and problem recording based on AR devices can improve the user experience; for example, displaying assembly quality analysis results based on AR glasses can also help guide welding personnel to make repairs.

[0104] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0105] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AR-based method for analyzing the soldering and assembly quality of circuit boards, characterized in that, include: AR device-based circuit board image acquisition; Based on the lightweight electronic component detection model deployed on the AR device, the location and category information of electronic components on the circuit board image are identified; wherein, the lightweight electronic component detection model adopts an end-to-end YOLO V4 network model, and the feature extraction network of the YOLO V4 network model is a lightweight Mobilenet network. Based on the identification results of the electronic components, and using a pre-built assembly template, at least the assembly quality analysis results of mis-soldering and missing soldering on the circuit board are obtained; The assembly quality analysis results are displayed in real time in a mixed reality space based on the AR device. The process involves correcting the identification results of the electronic components, and then obtaining assembly quality analysis results for mis-soldering and missing soldering on the circuit board based on the corrected identification results. The correction of the identification results of the electronic components includes: The pose information of the circuit board image is obtained based on the auxiliary positioning marks around the positioning frame; the positioning frame is the area where the circuit board is placed horizontally. Based on the pose information, the planar coordinates of each electronic component on the circuit board are obtained using affine transformation; The shooting angle of the circuit board image is unified according to the planar coordinates of the electronic components; The process of obtaining the assembly quality analysis results of the circuit board with mis-soldering and missing solder joints includes: Based on the principle of minimizing Euclidean distance, the identification results of the electronic components are matched with the electronic components of the assembly template; if the detected label is inconsistent with the assembly template, it is judged as mis-soldering; if the assembly template contains electronic components but no corresponding label is detected, it is judged as missing soldering.

2. The circuit board welding assembly quality analysis method as described in claim 1, characterized in that, Also includes: An alert will be issued for abnormal assembly quality analysis results; And at least the results of the abnormal assembly quality analysis shall be recorded in the device archive of the server.

3. An AR-based circuit board soldering assembly quality analysis system, characterized in that, The method for performing circuit board soldering assembly quality analysis as described in claim 1 includes: The acquisition module is used to acquire circuit board images based on AR devices; The identification module is used to identify the location and category information of electronic components on the circuit board image based on a lightweight electronic component detection model deployed on the AR device. The analysis module is used to obtain assembly quality analysis results of at least mis-soldering and missing soldering on the circuit board based on the identification results of the electronic components and a pre-built assembly template. The display module is used to display the assembly quality analysis results in a mixed reality space based on the AR device.

4. The circuit board welding assembly quality analysis system as described in claim 3, characterized in that, Also includes: The alarm module is used to issue alerts for abnormal assembly quality analysis results. The recording module is used at least to record the abnormal assembly quality analysis results in the server's device archive.

5. A storage medium, characterized in that, It stores a computer program for AR-based circuit board soldering assembly quality analysis, wherein the computer program causes a computer to execute the circuit board soldering assembly quality analysis method as described in claim 1 or 2.

6. An electronic device, characterized in that, include: One or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the one or more programs are executed by the one or more processors, the circuit board soldering assembly quality analysis method as described in claim 1 or 2 is implemented.

Images