Component tilt detection method, apparatus, system, and readable storage medium

Abstract

The application relates to a component tilt detection method, device, system and readable storage medium. The method comprises the following steps: extracting a plurality of component images corresponding to components from a to-be-detected two-dimensional circuit board image of a to-be-detected circuit board, and extracting a first component point cloud matched with the plurality of component images from a to-be-detected three-dimensional point cloud of the to-be-detected circuit board; performing clustering processing on the to-be-detected three-dimensional point cloud to obtain a plurality of second component point clouds; detecting a target component type corresponding to the plurality of second component point clouds according to the component types corresponding to the plurality of component images and the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds; for each component, generating tilt influence parameter information corresponding to the component based on the target component type and the second component point cloud, and performing tilt detection on the component according to the tilt influence parameter information. The method can accurately detect the tilt of the components on the circuit board.

Description

Technical Field

[0001] This application relates to the field of intelligent manufacturing technology, and in particular to a method, apparatus, system and readable storage medium for detecting component tilt. Background Technology

[0002] Printed circuit boards (PCBs), as core components of modern electronic devices, are manufactured using electronic printing technology and serve the crucial function of electrical connections between electronic components. With the rapid development of electronic technology, PCB designs are becoming increasingly complex. Their quality not only directly affects the performance and reliability of electronic products but also places higher demands on manufacturing processes and testing technologies. Therefore, it is essential to perform tilt detection on the various types of components soldered onto the PCB.

[0003] Existing methods for detecting the tilt of various types of components on circuit boards still rely on manual inspection, primarily visual inspection. Workers need to carefully observe the orientation of various components on the soldered circuit board and then judge whether each component is tilted based on experience. Alternatively, there are methods that detect the tilt of various types of components on circuit boards using two-dimensional images. These methods train an image segmentation model using deep learning, then use the image segmentation model to segment the pixel blocks corresponding to each component from the circuit board image, and determine whether the component is tilted based on the area of ​​the pixel blocks.

[0004] However, current methods for detecting component tilt are still not accurate enough. Summary of the Invention

[0005] Therefore, it is necessary to provide an accurate method, apparatus, system, computer-readable storage medium, and computer program product for detecting component tilt in order to address the aforementioned technical problems.

[0006] In a first aspect, this application provides a method for detecting component tilt, including:

[0007] Acquire a two-dimensional image of the circuit board under test and a three-dimensional point cloud of the circuit board under test, wherein multiple components are mounted on the surface of the circuit board under test;

[0008] Extract the component images corresponding to multiple components from the two-dimensional circuit board image under test, and extract the first component point cloud matching the multiple component images from the three-dimensional point cloud under test.

[0009] Clustering is performed on the 3D point cloud to be tested to obtain multiple point clouds of second components;

[0010] Based on the component types corresponding to multiple component images, and the matching degree between multiple first component point clouds and multiple second component point clouds, the target component type corresponding to multiple second component point clouds is detected.

[0011] For each component, based on the target component type and the point cloud of the second component, tilt influence parameter information corresponding to the component is generated, and tilt detection of the component is performed based on the tilt influence parameter information.

[0012] In one embodiment, based on the target component type and the point cloud of the second component, tilt influence parameter information corresponding to the component is generated, including:

[0013] When the target component type is an LED bead, a plane fitting is performed on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane.

[0014] Detect the projection area information of the point cloud of the second component onto the fitting plane of the circuit board;

[0015] Based on the projected area information, generate tilt influence parameter information for the corresponding components.

[0016] In one embodiment, based on the target component type and the point cloud of the second component, tilt influence parameter information corresponding to the component is generated, including:

[0017] When the target component type is non-planar, plane fitting is performed on the 3D point cloud to be measured to obtain the circuit board fitting plane.

[0018] Obtain the first angle information between the preset direction vector of the point cloud of the second component and the normal vector of the fitting plane of the circuit board;

[0019] Based on the first included angle information, generate the tilt influence parameter information corresponding to the component.

[0020] In one embodiment, based on the target component type and the point cloud of the second component, tilt influence parameter information corresponding to the component is generated, including:

[0021] When the target component type is planar, the point cloud of the second component is fitted with a plane to obtain the component fitting plane, and the 3D point cloud to be tested is fitted with a plane to obtain the circuit board fitting plane.

[0022] Obtain the second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane;

[0023] Based on the second included angle information, the tilt influence parameter information corresponding to the component is generated.

[0024] In one embodiment, tilt detection of the component is performed based on tilt influence parameter information, including:

[0025] For each component, obtain the preset tilt level range information that matches the parameter type of the tilt influence parameter information;

[0026] The tilt level of the component is detected based on the preset tilt level range information and tilt influence parameter information.

[0027] In one embodiment, the three-dimensional point cloud to be measured is clustered to obtain multiple second component point clouds, including:

[0028] Perform plane fitting on the 3D point cloud to be measured to obtain the circuit board fitting plane;

[0029] Based on the circuit board fitting plane, the surface of the circuit board to be tested in the three-dimensional point cloud to be tested is separated from the multiple components on the surface to obtain the overall point cloud of the components.

[0030] Clustering is performed on the overall point cloud of the components to obtain multiple second component point clouds.

[0031] In one embodiment, based on the component types corresponding to the multiple component images and the pairwise matching degree between the multiple first component point clouds and the multiple second component point clouds, the target component type corresponding to the multiple second component point clouds is detected, including:

[0032] Detect the centroid distance between each pair of the centroids of the point clouds of multiple first components and the point clouds of multiple second components;

[0033] Based on multiple centroid distances, the matching degree between each pair of point clouds of multiple first components and multiple point clouds of multiple second components is detected.

[0034] Based on the component types corresponding to multiple component images, detect the component types to be matched in multiple first component point clouds;

[0035] For each second component point cloud, the target component type of the second component point cloud is detected based on the matching degree between the first component point cloud and multiple second component point clouds, as well as the component types to be matched in the multiple first component point clouds.

[0036] Secondly, this application also provides a component tilt detection device, comprising:

[0037] The data acquisition module is used to acquire the two-dimensional image of the circuit board under test and the three-dimensional point cloud of the circuit board under test. Multiple components are mounted on the surface of the circuit board under test.

[0038] The first point cloud generation module is used to extract the component images corresponding to multiple components from the two-dimensional circuit board image to be tested, and to extract the first component point cloud matching multiple component images from the three-dimensional point cloud to be tested.

[0039] The second point cloud generation module is used to perform clustering processing on the 3D point cloud to be tested to obtain multiple second component point clouds.

[0040] The type matching module is used to detect the target component type corresponding to multiple second component point clouds based on the component types corresponding to multiple component images and the matching degree between multiple first component point clouds and multiple second component point clouds.

[0041] The tilt detection module is used to generate tilt influence parameter information for each component based on the target component type and the point cloud of the second component, and to perform tilt detection on the component based on the tilt influence parameter information.

[0042] Thirdly, this application also provides a component tilt detection system, the system comprising:

[0043] A vision sensor is used to acquire two-dimensional images of the circuit board under test and three-dimensional point clouds of the circuit board under test.

[0044] Controller, used for:

[0045] The test involves acquiring a two-dimensional image of the circuit board under test and a three-dimensional point cloud of the circuit board under test, which are collected by a vision sensor. Multiple components are mounted on the surface of the circuit board under test.

[0046] Extract the component images corresponding to multiple components from the two-dimensional circuit board image under test, and extract the first component point cloud matching the multiple component images from the three-dimensional point cloud under test.

[0047] Clustering is performed on the 3D point cloud to be tested to obtain multiple point clouds of second components;

[0048] Based on the component types corresponding to multiple component images, and the matching degree between multiple first component point clouds and multiple second component point clouds, the target component type corresponding to multiple second component point clouds is detected.

[0049] For each component, based on the target component type and the point cloud of the second component, tilt influence parameter information corresponding to the component is generated, and tilt detection of the component is performed based on the tilt influence parameter information.

[0050] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0051] Acquire a two-dimensional image of the circuit board under test and a three-dimensional point cloud of the circuit board under test, wherein multiple components are mounted on the surface of the circuit board under test;

[0052] Extract the component images corresponding to multiple components from the two-dimensional circuit board image under test, and extract the first component point cloud matching the multiple component images from the three-dimensional point cloud under test.

[0053] Clustering is performed on the 3D point cloud to be tested to obtain multiple point clouds of second components;

[0054] Based on the component types corresponding to multiple component images, and the matching degree between multiple first component point clouds and multiple second component point clouds, the target component type corresponding to multiple second component point clouds is detected.

[0055] For each component, based on the target component type and the point cloud of the second component, tilt influence parameter information corresponding to the component is generated, and tilt detection of the component is performed based on the tilt influence parameter information.

[0056] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0057] Acquire a two-dimensional image of the circuit board under test and a three-dimensional point cloud of the circuit board under test, wherein multiple components are mounted on the surface of the circuit board under test;

[0058] Extract the component images corresponding to multiple components from the two-dimensional circuit board image under test, and extract the first component point cloud matching the multiple component images from the three-dimensional point cloud under test.

[0059] Clustering is performed on the 3D point cloud to be tested to obtain multiple point clouds of second components;

[0060] Based on the component types corresponding to multiple component images, and the matching degree between multiple first component point clouds and multiple second component point clouds, the target component type corresponding to multiple second component point clouds is detected.

[0061] For each component, based on the target component type and the point cloud of the second component, tilt influence parameter information corresponding to the component is generated, and tilt detection of the component is performed based on the tilt influence parameter information.

[0062] Currently, the aforementioned component tilt detection methods, devices, systems, computer-readable storage media, and computer program products rely on manual tilt detection of components on circuit boards, which has poor accuracy. Furthermore, when using image segmentation models for component tilt detection, the segmented images may contain noise, resulting in inaccuracy. Therefore, in this application, on the one hand, multiple component images corresponding to various components are identified from the two-dimensional circuit board image of the circuit board under test, and a first component point cloud matching the multiple component images is extracted from the three-dimensional point cloud of the circuit board under test. On the other hand, the three-dimensional point cloud of the circuit board under test is clustered into multiple second component point clouds. Since the clustered second component point clouds are more accurate, the process of obtaining component types based on image extraction is more accurate. Therefore, the target component type corresponding to the multiple second component point clouds can be detected based on the component types corresponding to the multiple component images and the pairwise matching degree between the multiple first component point clouds and the multiple second component point clouds. Then, for each component, the tilt influence parameter information generated based on the target component type and the second component point cloud is used to accurately detect the component tilt. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is an application environment diagram of a component tilt detection method in one embodiment;

[0065] Figure 2 This is a flowchart illustrating a component tilt detection method in one embodiment;

[0066] Figure 3 This is a flowchart illustrating the component tilt detection method in another embodiment;

[0067] Figure 4 This is a flowchart illustrating the component tilt detection method in yet another embodiment;

[0068] Figure 5 This is a schematic diagram of the projected area of ​​the LED beads in a specific application embodiment;

[0069] Figure 6 This is an overall schematic diagram of a component tilt detection system in one embodiment;

[0070] Figure 7 This is a front view of the hardware structure of a component tilt detection system in one embodiment;

[0071] Figure 8 This is a left view of the hardware structure of a component tilt detection system in one embodiment;

[0072] Figure 9 This is a top view of the hardware structure of a component tilt detection system in one embodiment;

[0073] Figure 10 This is a structural block diagram of a component tilt detection device in one embodiment;

[0074] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0076] Printed circuit boards (PCBs), as core components of modern electronic devices, are manufactured using electronic printing technology and play a crucial role in the electrical connection between electronic components. Therefore, it is essential to perform tilt detection on the various types of components soldered onto the PCBs.

[0077] Existing methods for detecting the tilt of various types of components on circuit boards still rely on manual inspection, primarily visual inspection. Workers need to carefully observe the orientation of various components on the soldered circuit board and then judge whether each component is tilted based on experience. Alternatively, there are methods that detect the tilt of various types of components on circuit boards using two-dimensional images. These methods train an image segmentation model using deep learning, then use the image segmentation model to segment the pixel blocks corresponding to each component from the circuit board image, and determine whether the component is tilted based on the area of ​​the pixel blocks.

[0078] However, current methods for detecting component tilt on circuit boards, particularly manual visual inspection, have significant limitations. They are susceptible to fluctuations in the inspector's condition, experience differences, and subjective biases, leading to compromised accuracy and stability, and ultimately failing to accurately detect component tilt. While methods using two-dimensional images for component tilt detection can replace manual inspection, the pixel blocks extracted by image segmentation models may not represent the complete pixels of the corresponding components, and the segmented images may contain noise, resulting in inaccurate calculation results.

[0079] To this end, this application provides an accurate component tilt detection method. On the one hand, it identifies component images corresponding to multiple components from the two-dimensional circuit board image of the circuit board under test, and extracts first component point clouds matching the multiple component images from the three-dimensional point cloud under test. On the other hand, it clusters the three-dimensional point cloud of the circuit board under test into multiple second component point clouds. Since the multiple second component point clouds obtained by clustering are more accurate, the process of obtaining component types based on image extraction is more accurate. Therefore, it is possible to detect the target component type corresponding to the multiple second component point clouds based on the component types corresponding to the multiple component images, as well as the matching degree between the multiple first component point clouds and the multiple second component point clouds. Then, for each component, it accurately detects the tilt of the component based on the tilt influence parameter information generated by the target component type and the second component point clouds.

[0080] The component tilt detection method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with controller 104 via a network. Controller 104 also communicates with vision sensor 106, which acquires two-dimensional images and three-dimensional point clouds of the circuit board under test 108. Multiple components 110 are mounted on the surface of the circuit board under test 108. A data storage system can store the data that controller 104 needs to process. The data storage system can be integrated into controller 104 or placed in the cloud or on another network server.

[0081] When a user triggers a component tilt detection control on the component tilt detection interface of terminal 102, terminal 102 responds to the trigger request of the component tilt detection control, generates a component tilt detection request, and sends the component tilt detection request to controller 104.

[0082] At this time, the controller 104 can control the vision sensor to detect the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board under test 108 in real time, and can also obtain the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board under test 108 from the database.

[0083] Furthermore, the controller 104 extracts component images corresponding to multiple components 110 from the two-dimensional circuit board image under test, and extracts first component point clouds matching the multiple component images from the three-dimensional point cloud under test; performs clustering processing on the three-dimensional point cloud under test to obtain multiple second component point clouds; detects the target component type corresponding to the multiple second component point clouds based on the component types corresponding to the multiple component images and the matching degree between the multiple first component point clouds and the multiple second component point clouds; for each component 110, generates tilt influence parameter information corresponding to the component 110 based on the target component type and the second component point cloud, and performs tilt detection on the component based on the tilt influence parameter information.

[0084] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses.

[0085] In one exemplary embodiment, such as Figure 2 As shown, a component tilt detection method is provided, which is applied to... Figure 1 The following explanation uses controller 104 as an example. Wherein:

[0086] S100: Acquire the two-dimensional image of the circuit board under test and the three-dimensional point cloud of the circuit board under test.

[0087] The circuit board under test (PCB) has multiple components mounted on its surface, such as LEDs, planar components, and non-planar components. The PCB is essentially a printed circuit board (PCB). A two-dimensional image is a pixel array on a plane, where each pixel contains color, brightness, or grayscale values, used to represent the projection of a scene from a specific viewpoint. A three-dimensional point cloud is a collection of discrete points in space, where each point contains three-dimensional coordinates (X, Y, Z) and optional attributes (color, reflectivity, etc.), used to accurately describe the three-dimensional structure of an object or scene.

[0088] Specifically, the circuit board to be tested is clamped by a circuit board clamping mechanism, and the two-dimensional circuit board image and the three-dimensional point cloud of the clamped circuit board are detected by a vision sensor. The two-dimensional circuit board image includes an overall two-dimensional image of multiple components, and the three-dimensional point cloud includes a three-dimensional combined point cloud of multiple components.

[0089] A visual sensor can be a depth sensor, such as a scanning camera, used to detect point clouds of an object, or an image sensor used to capture images of an object. A visual sensor can also be used simultaneously to detect point clouds and capture images of an object. The two-dimensional image and three-dimensional point cloud of the circuit board under test detected by the visual sensor can be fed back to the controller in real time when the controller sends a component detection request, or they can be stored in a database for subsequent component tilt detection of the circuit board under test.

[0090] In other words, the controller can control the vision sensor to detect the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board under test in real time, and can also obtain the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board under test from the database.

[0091] In one exemplary embodiment, the vision sensor is mounted on a sliding guide rail, which drives the vision sensor to move, so as to accurately acquire the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board under test, thereby reducing the occurrence of situations where the vision sensor cannot detect the complete image or point cloud.

[0092] S200 extracts component images corresponding to multiple components from the two-dimensional circuit board image under test, and extracts the first component point cloud matching multiple component images from the three-dimensional point cloud under test.

[0093] Specifically, the image of the two-dimensional circuit board under test is segmented to obtain multiple component images, and each component image corresponds to a component type. The component types corresponding to different component images can be the same or different. Here, the component type of the component image refers to the surface type of the component corresponding to the component image.

[0094] For each component image, a point cloud index is obtained based on the pixel coordinates of the component image, and the first component point cloud that matches the point cloud index is queried from the 3D point cloud to be tested.

[0095] In an exemplary embodiment, image segmentation of the two-dimensional circuit board image under test can be achieved through deep learning methods, such as using a deep learning-based image segmentation model to segment the image of the two-dimensional circuit board image under test; alternatively, the image of the two-dimensional circuit board under test can be compared with the theoretical model of the circuit board to perform image segmentation.

[0096] The S300 performs clustering processing on the 3D point cloud to be measured, resulting in multiple point clouds of second components.

[0097] Specifically, unlike the previous method of first performing image segmentation and then mapping the segmented two-dimensional component images onto the three-dimensional point cloud to be tested, resulting in a first component point cloud matching each component image, this step directly clusters the three-dimensional point cloud to be tested into multiple second component point clouds. In practical applications, the clustering methods for the three-dimensional point cloud to be tested can include Euclidean clustering, region growing clustering, and density-based clustering, etc.

[0098] In an exemplary embodiment, before performing clustering processing on the 3D point cloud to be tested, it is necessary to filter the 3D point cloud to be tested in order to remove discrete points in the 3D point cloud to be tested. The filtering method can be statistical filtering, voxel grid filtering, clustering filtering, and radius filtering, etc.

[0099] Among them, statistical filtering refers to the method based on statistical methods, assuming that the neighboring points of each point in the point cloud should meet certain distribution rules, and identifying and removing points (noise) that deviate too much from the mean by calculating the average distance from each point to its neighboring points; voxel grid filtering refers to dividing the point cloud into a regular three-dimensional voxel grid, and replacing all points in each voxel with a representative point (such as the centroid), thereby downsampling the point cloud; radius filtering refers to counting the number of neighboring points within the radius of each point, and if the number of neighboring points is less than a threshold, it is regarded as a noise point and removed.

[0100] S400, based on the component types corresponding to multiple component images and the matching degree between multiple first component point clouds and multiple second component point clouds, detects the target component type corresponding to multiple second component point clouds.

[0101] Specifically, a matching degree analysis is performed on multiple first component point clouds obtained through image segmentation and multiple second component point clouds obtained through clustering to obtain the pairwise matching degree between multiple first component point clouds and multiple second component point clouds.

[0102] The matching degree analysis can be obtained by the centroid distance between each pair of point clouds of multiple first components and multiple point clouds of multiple second components, or by the point cloud overlap between each pair of point clouds of multiple first components and multiple point clouds of multiple second components. For example, if the point cloud overlap between a certain point cloud of a first component and a certain point cloud of a second component is 90%, then its matching degree can also be considered as 90%.

[0103] Furthermore, each first component point cloud is extracted based on the component image. Therefore, the component type of the component image is the component type of the corresponding first component point cloud. Then, based on the matching degree between multiple first component point clouds and multiple second component point clouds, the target component type corresponding to multiple second component point clouds can be detected.

[0104] It needs to be explained that the reason for not directly using the first component point cloud for component tilt detection is that when segmenting multiple component images from the 2D circuit board image to be tested, inaccurate image segmentation may occur, resulting in noise in the component images. Furthermore, in the process of extracting the first component point cloud matching multiple component images from the 3D point cloud to be tested, although it is possible to accurately obtain the component type corresponding to each first component point cloud based on the component type corresponding to the component image, the noise in the component image will also cause the corresponding first component point cloud to have noise. However, directly clustering the 3D point cloud to be tested into multiple second component point clouds results in less noise. Therefore, on the one hand, multiple component images corresponding to multiple components are extracted from the two-dimensional circuit board image of the circuit board under test, and first component point clouds matching multiple component images are extracted from the three-dimensional point cloud of the circuit board under test. On the other hand, the three-dimensional point cloud of the circuit board under test is clustered into multiple second component point clouds. Since the multiple second component point clouds divided by clustering are more accurate, the process of obtaining the component types based on the multiple component images extracted from the image is more accurate. Therefore, the target component type corresponding to multiple second component point clouds can be detected according to the component types corresponding to multiple component images and the matching degree between multiple first component point clouds and multiple second component point clouds.

[0105] For each component, S500 generates tilt influence parameter information based on the target component type and the point cloud of the second component, and performs tilt detection on the component according to the tilt influence parameter information.

[0106] Specifically, for each component, the target component type is used to determine the parameter type of the tilt influence parameter information. In other words, the tilt influence parameter information is different for different target component types in the point cloud of the second component.

[0107] Therefore, based on the target component type, the parameter type of the tilt influence parameter information corresponding to the component can be determined. Then, based on the parameter type and the point cloud of the second component, the tilt influence parameter information corresponding to the component can be generated. In practical applications, the parameter type of the tilt influence parameter information can be angle information or projected area information, etc.

[0108] Furthermore, generating tilt influence parameter information corresponding to the component based on the parameter type and the second component point cloud also includes: performing plane fitting on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane; generating tilt influence parameter information corresponding to the component based on the circuit board fitting plane and the second component point cloud. For example, the tilt influence parameter information corresponding to the component can be generated based on the projected area of ​​the second component point cloud projected onto the circuit board fitting plane; or the tilt influence parameter information corresponding to the component can be generated based on the angle between the normal vector of the second component point cloud and a vector in a certain direction of the second component point cloud.

[0109] Furthermore, based on the tilt influence parameter information, the tilt detection of the component is performed to obtain the tilt detection result of the component. The tilt detection result is used to characterize whether the component is tilted. Tilt refers to the angle of deviation of an object or plane relative to a reference direction (such as a horizontal or vertical plane).

[0110] Furthermore, based on the tilt influence parameter information, tilt detection of the component can include: acquiring preset tilt threshold information matching the target component type, and performing tilt detection of the component based on the tilt influence parameter information and the preset tilt threshold range; wherein, performing tilt detection of the component based on the tilt influence parameter information and the preset tilt threshold range further includes: when the tilt influence parameter information and the preset tilt threshold range indicate that the tilt influence parameter value is not within the preset tilt threshold range, the component is tilted; when the tilt influence parameter information and the preset tilt threshold range indicate that the tilt influence parameter value is within the preset tilt threshold range, the component is not tilted.

[0111] Currently, the accuracy of relying on manual tilt detection of components on circuit boards is poor, and the segmented images may contain noise and are not accurate enough when using image segmentation models for component tilt detection. Therefore, in this application, on the one hand, multiple component images corresponding to multiple components are identified from the two-dimensional circuit board image of the circuit board under test, and the first component point cloud matching the multiple component images is extracted from the three-dimensional point cloud of the circuit board under test. On the other hand, the three-dimensional point cloud of the circuit board under test is clustered into multiple second component point clouds. Since the multiple second component point clouds divided by clustering are more accurate, the process of obtaining component types based on image extraction is more accurate. Therefore, the target component type corresponding to the multiple second component point clouds can be detected according to the component type corresponding to the multiple component images and the matching degree between the multiple first component point clouds and the multiple second component point clouds. Then, for each component, the tilt influence parameter information generated based on the target component type and the second component point cloud is used to accurately detect the tilt of the component.

[0112] In one exemplary embodiment, the target component type includes at least LED beads, non-planar types, and planar types. It should be noted that LED beads are also a type of component, but because they are made of transparent material, the accuracy of the acquired 3D point cloud is very poor; therefore, they are treated as a separate detection type.

[0113] Therefore, as Figure 3 As shown below, several embodiments will be used to explain in detail how to generate corresponding tilt influence parameter information that matches the target component type when the target component types are different.

[0114] In one exemplary embodiment, it is still as follows Figure 3 As shown, S500 includes:

[0115] S512, when the target component type is LED bead, performs plane fitting on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane.

[0116] S514 detects the projection area information of the point cloud of the second component onto the fitted plane of the circuit board.

[0117] S516 generates tilt influence parameter information for the components based on the projected area information.

[0118] S540 performs tilt detection on components based on tilt influence parameter information.

[0119] Specifically, when the target component type is an LED bead, a plane fitting is performed on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane. The second component point cloud is then projected onto the circuit board fitting plane to obtain the projection area information of the second component point cloud. The projection area information of the second component point cloud is then used as the tilt influence parameter information of the second component point cloud.

[0120] Furthermore, taking the vertical direction as the reference direction for component tilting as an example, tilt detection is performed on the component based on tilt influence parameter information, including:

[0121] When the target component type is an LED chip, the tilt influence parameter information of the second component's point cloud is the projected area information of the second component's point cloud projected onto the circuit board's fitted plane. The projected area is positively correlated with the component's tilt. The tilt detection result is obtained by determining whether the projected area information is greater than a preset projected area threshold. If the projected area is greater than the preset projected area threshold, the component is considered tilted. Figure 5 As shown, the circuit board is a PCB board. When the LED is vertically set on a certain surface of the PCB board, the projected area is the upper surface area of ​​the LED. When the LED is tilted at a certain angle in the vertical direction, the projected area is greater than the upper surface area of ​​the LED.

[0122] In the above embodiments, by detecting the projection area information of the point cloud of the second component onto the fitting plane of the circuit board, the tilt influence parameter information of the lamp bead can be accurately generated.

[0123] In one exemplary embodiment, it is still as follows Figure 3 As shown, S500 includes:

[0124] S522, when the target component type is non-planar, performs planar fitting on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane.

[0125] S524, obtain the first angle information between the preset direction vector of the second component point cloud and the normal vector of the circuit board fitting plane.

[0126] S526, Based on the first included angle information, generate tilt influence parameter information corresponding to the component.

[0127] S540 performs tilt detection on components based on tilt influence parameter information.

[0128] Specifically, when the target component is a non-planar type, a planar fit is performed on the 3D point cloud to be measured to obtain the circuit board fitting plane. Then, the three principal directions of the second component point cloud are obtained by PCA principal component analysis. The smallest principal direction vector is used as the preset direction vector. The first angle information between the preset direction vector of the second component point cloud and the normal vector of the circuit board fitting plane is detected, and the first angle information is used as the tilt influence parameter information of the second component point cloud.

[0129] Taking the vertical direction as the reference direction for the tilt of the component as an example, the first included angle represented by the first included angle information is positively correlated with the tilt of the component. If the first included angle is greater than the preset first included angle threshold, the component is tilted. If the first included angle is less than or equal to the preset first included angle threshold, the component can be regarded as not tilted.

[0130] In the above embodiments, by detecting the first angle information between the preset direction vector of the second component point cloud and the normal vector of the circuit board fitting plane, the tilt influence parameter information of non-planar components can be accurately generated.

[0131] In one exemplary embodiment, S500 includes:

[0132] S532, when the target component type is planar, performs planar fitting on the point cloud of the second component to obtain the component fitting plane, and performs planar fitting on the three-dimensional point cloud to be tested to obtain the circuit board fitting plane.

[0133] S534, obtain the second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane.

[0134] S536, based on the second included angle information, generates tilt influence parameter information corresponding to the component.

[0135] S540 performs tilt detection on components based on tilt influence parameter information.

[0136] Specifically, when the target component type is planar, the point cloud of the second component is fitted with a plane to obtain the component fitting plane. The plane fitting method can be RANSAC plane fitting, least squares plane fitting, PCA plane fitting, etc. Then, the second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane is obtained, and the second angle information is used as the tilt influence parameter information of the second component point cloud.

[0137] Taking the vertical direction as the reference direction for the tilt of the component as an example, the second included angle represented by the second included angle information is positively correlated with the tilt of the component. If the second included angle is greater than the preset second included angle threshold, the component is tilted. If the second included angle is less than or equal to the preset second included angle threshold, the component can be regarded as not tilted.

[0138] In the above embodiments, by detecting the second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane, the tilt influence parameter information of planar components can be accurately generated.

[0139] In an exemplary embodiment, tilt detection of components is performed based on tilt influence parameter information, including:

[0140] For each component, obtain a preset tilt level range that matches the parameter type of the tilt influence parameter information; based on the preset tilt level range information and the tilt influence parameter information, detect the tilt level of the component.

[0141] Specifically, the tilt detection results of the components in this application can not only characterize whether the components are tilted, but also determine the tilt level of the components. More specifically, for each component, a preset tilt level range information matching the parameter type of the tilt influence parameter information is obtained. For example, when the parameter type of the tilt influence parameter information is projected area information, the preset tilt level range information includes a tilt level of level one when the projected area is within a first projected area range, a tilt level of level two when the projected area is within a second projected area range, a tilt level of level three when the projected area is within a third projected area range, and so on. It is then determined whether the tilt influence parameter information is within a certain target level range in the preset tilt level range information. If it is within a certain target level range, the level corresponding to that target level range is determined as the tilt level of the component.

[0142] In this embodiment, by acquiring a preset tilt level range information that matches the parameter type of the tilt influence parameter information, and comparing and analyzing the preset tilt level range information with the tilt influence parameter information, the tilt level of the component can be accurately detected.

[0143] In an exemplary embodiment, the three-dimensional point cloud to be measured is clustered to obtain multiple second component point clouds, including:

[0144] Planar fitting is performed on the 3D point cloud to be tested to obtain the circuit board fitting plane; based on the circuit board fitting plane, the surface of the circuit board to be tested and multiple components on the surface are separated in the 3D point cloud to be tested to obtain the overall point cloud of the components; clustering is performed on the overall point cloud of the components to obtain multiple second component point clouds.

[0145] Specifically, a plane fitting is performed on the three-dimensional point cloud of the circuit board to be tested to obtain the circuit board fitting plane. The plane fitting method can be RANSAC (Random Sample Consensus) fitting plane method, or least squares plane fitting and PCA (Principal Component Analysis) plane fitting method, etc. In this embodiment, the plane fitting method is RANSAC fitting plane method.

[0146] Next, based on the planar parameters of the circuit board fitting plane, the distance from the circuit board fitting plane in the 3D point cloud to be tested is detected. Then, based on the distance from the circuit board fitting plane in the 3D point cloud to be tested and a preset distance threshold, the surface of the circuit board to be tested and multiple components on the surface of the circuit board to be tested are separated to extract the overall point cloud of the components. In practical applications, points in the 3D point cloud to be tested that are at a distance greater than the preset distance threshold are often considered as the overall point cloud of the components.

[0147] At this point, the point cloud is a whole point cloud that does not distinguish multiple components. Therefore, it is necessary to perform clustering processing on the whole point cloud of components to divide the whole point cloud of components into single point clouds of each component, so as to obtain the second component point cloud of multiple components.

[0148] In the above embodiments, by performing planar fitting on the three-dimensional point cloud to be tested, the circuit board to be tested and multiple components on the surface of the circuit board to be tested can be accurately separated in the three-dimensional point cloud to be tested, and the overall point cloud of the components can be obtained. Then, by clustering the overall point cloud of the components, the second component point cloud of multiple components can be accurately obtained.

[0149] In one exemplary embodiment, extracting component images corresponding to multiple components from a two-dimensional circuit board image under test includes:

[0150] Using a pre-trained image segmentation model, component images corresponding to multiple components are extracted from the two-dimensional circuit board image under test.

[0151] Specifically, a pre-trained image segmentation model is used to process the image of the two-dimensional circuit board under test to obtain the component images corresponding to multiple components and the component type of each component image. The pre-trained image segmentation model can be the YOLOv8-seg segmentation model or other deep learning image segmentation models.

[0152] The training process of the image segmentation model includes: scanning multiple sets of circuit boards within a historical time period to obtain multiple sets of two-dimensional historical circuit board images, and determining multiple historical component images in the historical circuit board images and the component type corresponding to each historical component image. The image segmentation model is trained using the measured multiple sets of historical circuit board images, multiple historical component images in the historical circuit board images, and the component type corresponding to each historical component image. The component types include at least LED beads, planar types, and non-planar types. The trained image segmentation model is then used to segment three types of component images: LED beads, planar components, and components with non-planar upper surfaces.

[0153] In the above embodiments, the image segmentation model is trained using historical circuit board images, multiple historical component images in the historical circuit board images, and the component type corresponding to each historical component image, so that the trained image segmentation model can accurately segment three types of component images: LED beads, planar components, and components with non-planar upper surfaces.

[0154] In one exemplary embodiment, such as Figure 4 As shown, S400 includes:

[0155] S410 detects the centroid distance between each pair of the centroids of the point clouds of multiple first components and the point clouds of multiple second components.

[0156] The S420 detects the matching degree between pairs of point clouds of multiple first components and point clouds of multiple second components based on multiple centroid distances.

[0157] S430 detects the type of component to be matched in multiple first component point clouds based on the component types corresponding to multiple component images.

[0158] S440, for each second component point cloud, based on the matching degree between the first component point cloud and multiple second component point clouds, and the component types to be matched in the multiple first component point clouds, detect the target component type of the second component point cloud.

[0159] Specifically, the first component point cloud and the second component point cloud are point clouds detected by the same component in different ways. Therefore, the position difference between the first component point cloud corresponding to the same component and the second component point cloud matched by the first component point cloud will not be large. Taking the centroid of multiple first component point clouds and multiple second component point clouds as the reference point, each first component point cloud is matched with the corresponding second component point cloud according to the principle of closest centroid distance.

[0160] In other words, firstly, the first centroid positions of multiple first component point clouds and the second centroid positions of multiple second component point clouds are detected. Based on the multiple first centroid positions and multiple second centroid positions, the centroid distance between each pair of the centroids of the multiple first component point clouds and the multiple second component point clouds is detected. Then, through the multiple centroid distances, the matching degree between each pair of the multiple first component point clouds and the multiple second component point clouds is detected. For example, if the centroid distance between first component point cloud A and second component point cloud b is greater, the matching degree is lower; if the centroid distance between first component point cloud A and second component point cloud b is closer, the matching degree is higher.

[0161] Furthermore, for each first component point cloud, the second component point cloud with the highest matching degree among multiple second component point clouds is taken as the second component point cloud that matches the first component point cloud.

[0162] Since the first component point cloud is extracted from the 3D point cloud under test through component images, the component types to be matched in multiple first component point clouds are detected based on the component types corresponding to multiple component images. The component types to be matched in the first component point clouds are consistent with the component types in the component images corresponding to the first component point clouds. After determining the second component point cloud that matches each first component point cloud, the component types to be matched in the first component point clouds are taken as the target component types of the second component point clouds.

[0163] In the above embodiments, by detecting the centroid distance between the centroid of the first component point cloud and the centroid of each second component point cloud, the matching degree between multiple first component point clouds and multiple second component point clouds can be detected, thereby accurately determining the second component point cloud that matches the first component point cloud, so as to match the component types among multiple component images, multiple first component point clouds and multiple second component point clouds, thereby improving the accuracy of determining the component type of the second component point cloud.

[0164] In one exemplary embodiment, the component tilt detection method provided in this application can be applied to a component tilt detection system, which consists of a component tilt detection hardware structure 10 and a controller 104. Figure 6 This is a schematic diagram of the overall component tilt detection system. The component tilt detection hardware structure 10 includes a 3D line scan camera 1, a sliding guide rail 2, a circuit board clamping mechanism 3, and a housing 4. The software part is a controller 5. Figure 7 This is a front view of the component tilt detection hardware structure 10. Figure 8 This is a left view of the component tilt detection hardware structure 10. Figure 9 This is a top view of the hardware structure 10 for component tilt detection.

[0165] S1 places the circuit board to be tested on the circuit board clamping mechanism 3, and the 3D line scan camera 1 is moved by the sliding guide rail 2 to scan the component surface of the circuit board to be tested, thereby obtaining the 2D circuit board image and the 3D point cloud of the circuit board to be tested.

[0166] S2 uses a pre-trained YOLOv8-seg segmentation model to infer the segmented pixel blocks of the 2D circuit board image under test, obtaining the segmented pixel blocks of each component. Point cloud indices are obtained based on the pixel coordinates of the segmented pixel blocks of each component. Corresponding points are extracted from the 3D point cloud of the circuit board under test to obtain the first component point cloud of each component, and the first centroid position of the first component point cloud of each component is calculated.

[0167] S3 removes discrete points from the 3D point cloud of the circuit board under test using statistical filtering, and fits the circuit board fitting plane using RANSAC fitting plane method. It extracts the overall point cloud of components above the circuit board fitting plane according to a certain distance threshold beyond the circuit board fitting plane, and uses Euclidean clustering method to divide the overall point cloud of components into single-chip second component point clouds of each component, and calculates the second centroid position of the second component point cloud of each component.

[0168] For each component, S4 calculates the centroid distance between the first centroid position of the first component point cloud and the second centroid position of the second component point cloud. According to the principle of closest centroid distance, the second component point cloud corresponding to the smallest centroid distance is selected as the second component point cloud to match the first component point cloud. For each second component point cloud, the component type label of the first component point cloud is matched, such as three types: LED bead, component type with planar upper surface, and component type with non-planar upper surface.

[0169] The following steps all use the vertical direction as the reference direction for the tilt of the component:

[0170] S5 projects the point cloud of the second component of the LED type onto the circuit board fitting plane, calculates the projected area, sets a projection area threshold, and determines that the LED is tilted if the projected area is greater than the projection area threshold.

[0171] S6 calculates the three principal directions of the point cloud for the second component whose upper surface is a non-planar component type using the PCA principal component analysis method. The angle between the minimum principal direction vector n1 and the normal vector n of the circuit board fitting plane is taken as the first tilt angle. If the first tilt angle is greater than the set first angle threshold, the component whose upper surface is a non-planar component type is tilted.

[0172] S7 uses RANSAC to fit the upper surface plane of the second component point cloud, which has a planar upper surface, to obtain the component fitting plane. The angle between the normal vector n of the circuit board fitting plane and the normal vector n2 of the component fitting plane is taken as the second tilt angle. If the second tilt angle is greater than the set second angle threshold, the component with a planar upper surface is determined to be tilted.

[0173] As can be seen, this application is a high-precision method for detecting the tilt of circuit board components. The circuit board is clamped by a clamping mechanism, and then a 3D line scan camera is driven by a sliding guide rail to scan the component surface of the circuit board to obtain 2D images and 3D point clouds. The image segmentation model is trained using multiple sets of 2D images of circuit boards. The trained model is then used to perform inference segmentation on the 2D images of the circuit board under test. Next, the 3D point cloud of the circuit board under test is filtered and fitted to the plane of the circuit board. The Euclidean clustering results of the point cloud above the fitted plane are matched with the inference segmentation results of the 2D images to obtain the label for each clustered point cloud: LED, component with a planar upper surface, or component with a non-planar upper surface. For the LED point cloud, the tilt angle is determined by calculating the area projected onto the fitted plane of the circuit board. For components with a non-planar upper surface, the tilt angle is determined by the angle between the minimum principal direction vector n1 calculated by PCA principal component analysis and the normal vector n of the fitted plane of the circuit board. For the point cloud of a component with a planar upper surface, the normal vector n2 is obtained by calculating the fitted plane of the planar component point cloud, and the tilt angle is determined by the angle between n2 and the normal vector n of the large plane of the circuit board.

[0174] Compared to manual inspection, this method improves inspection efficiency and avoids the subjectivity of manual inspection. At the same time, compared to 2D image inspection, the method of combining 2D images and 3D point clouds—segmenting three types of components from 2D images and then locating and matching 3D point cloud cluster labels to calculate the tilt angle—greatly improves the detection accuracy of the tilt angle of circuit board components.

[0175] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0176] Based on the same inventive concept, this application also provides a component tilt detection device for implementing the component tilt detection method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more component tilt detection device embodiments provided below can be found in the limitations of the component tilt detection method described above, and will not be repeated here.

[0177] In one exemplary embodiment, such as Figure 10 As shown, a component tilt detection device is provided, comprising: a data acquisition module 100, a first point cloud generation module 200, a second point cloud generation module 300, a type matching module 400, a tilt influence parameter detection module 500, and a tilt detection module 600, wherein:

[0178] The data acquisition module 100 is used to acquire the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board under test. Multiple components are mounted on the surface of the circuit board under test.

[0179] The first point cloud generation module 200 is used to extract component images corresponding to multiple components from the two-dimensional circuit board image to be tested, and to extract the first component point cloud matching multiple component images from the three-dimensional point cloud to be tested.

[0180] The second point cloud generation module 300 is used to perform clustering processing on the three-dimensional point cloud to be measured to obtain multiple second component point clouds.

[0181] The type matching module 400 is used to detect the target component type corresponding to multiple second component point clouds based on the component types corresponding to multiple component images and the matching degree between multiple first component point clouds and multiple second component point clouds.

[0182] The tilt influence parameter detection module 500 is used to generate tilt influence parameter information for each component based on the target component type and the point cloud of the second component.

[0183] The tilt detection module 600 is used to detect the tilt of components based on tilt influence parameter information.

[0184] In one embodiment, the tilt influence parameter detection module 500 is further configured to perform plane fitting on the three-dimensional point cloud to be measured when the target component type is an LED bead, to obtain a circuit board fitting plane; detect the projection area information of the second component point cloud projected onto the circuit board fitting plane; and generate tilt influence parameter information corresponding to the component based on the projection area information.

[0185] In one embodiment, the tilt influence parameter detection module 500 is further configured to perform planar fitting on the three-dimensional point cloud to be measured when the target component type is non-planar, to obtain the circuit board fitting plane; obtain the first angle information between the preset direction vector of the second component point cloud and the normal vector of the circuit board fitting plane; and generate tilt influence parameter information corresponding to the component based on the first angle information.

[0186] In one embodiment, the tilt influence parameter detection module 500 is further configured to, when the target component type is planar, perform planar fitting on the second component point cloud to obtain a component fitting plane, and perform planar fitting on the three-dimensional point cloud to be tested to obtain a circuit board fitting plane; obtain the second included angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane; and generate tilt influence parameter information corresponding to the component based on the second included angle information.

[0187] In one embodiment, the tilt detection module 600 is further configured to acquire, for each component, a preset tilt level range information that matches the parameter type of the tilt influence parameter information; and detect the tilt level of the component based on the preset tilt level range information and the tilt influence parameter information.

[0188] In one embodiment, the second point cloud generation module 300 is further configured to perform planar fitting on the three-dimensional point cloud to be tested to obtain a circuit board fitting plane; based on the circuit board fitting plane, separate the surface of the circuit board to be tested from multiple components on the surface in the three-dimensional point cloud to be tested to obtain a component overall point cloud; and perform clustering processing on the component overall point cloud to obtain multiple second component point clouds.

[0189] In one embodiment, the type matching module 400 is further configured to detect the centroid distance between each pair of the centroids of the multiple first component point clouds and the centroids of the multiple second component point clouds; based on the multiple centroid distances, detect the matching degree between each pair of the multiple first component point clouds and the multiple second component point clouds; detect the component type to be matched in the multiple first component point clouds according to the component type corresponding to the multiple component images; and for each second component point cloud, detect the target component type of the second component point cloud based on the matching degree between each pair of the first component point cloud and the multiple second component point clouds, and the component type to be matched in the multiple first component point clouds.

[0190] Each module in the aforementioned component tilt detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0191] In one exemplary embodiment, a component tilt detection system is also provided, the system comprising:

[0192] A vision sensor is used to acquire two-dimensional images of the circuit board under test and three-dimensional point clouds of the circuit board under test.

[0193] Controller, used for:

[0194] The test involves acquiring a two-dimensional image of the circuit board under test and a three-dimensional point cloud of the circuit board under test, which are collected by a vision sensor. Multiple components are mounted on the surface of the circuit board under test.

[0195] Extract the component images corresponding to multiple components from the two-dimensional circuit board image under test, and extract the first component point cloud matching the multiple component images from the three-dimensional point cloud under test.

[0196] Clustering is performed on the 3D point cloud to be tested to obtain multiple point clouds of second components;

[0197] Based on the component types corresponding to multiple component images, and the matching degree between multiple first component point clouds and multiple second component point clouds, the target component type corresponding to multiple second component point clouds is detected.

[0198] For each component, based on the target component type and the point cloud of the second component, tilt influence parameter information corresponding to the component is generated, and tilt detection of the component is performed based on the tilt influence parameter information.

[0199] Specifically, the vision sensor is used to acquire the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board to be tested, and the controller is used to execute all the steps in the above embodiment of the component tilt detection method, which will not be repeated here.

[0200] Furthermore, the component tilt detection system includes not only a vision sensor and controller, but also a sliding guide rail, a circuit board clamping mechanism, and a housing. The circuit board clamping mechanism is used to hold the circuit board under test, and the sliding guide rail, the circuit board clamping mechanism, and the vision sensor are all housed inside the housing. The vision sensor is mounted on the sliding guide rail and is used to move the vision sensor to accurately acquire the two-dimensional image of the circuit board under test and the three-dimensional point cloud of the circuit board under test.

[0201] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores data such as the two-dimensional image of the circuit board under test and the three-dimensional point cloud data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a component tilt detection method.

[0202] Those skilled in the art will understand that Figure 11 The structure shown is a block diagram of a partial structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0203] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0204] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0205] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0206] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0207] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0208] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting component tilt, characterized in that, The method includes: Acquire a two-dimensional image of the circuit board under test and a three-dimensional point cloud of the circuit board under test, wherein multiple components are mounted on the surface of the circuit board under test; Extract the component images corresponding to the multiple components from the two-dimensional circuit board image to be tested, and extract the first component point cloud matching the multiple component images from the three-dimensional point cloud to be tested; Clustering is performed on the three-dimensional point cloud to be measured to obtain multiple second component point clouds; Based on the component types corresponding to the multiple component images, and the matching degree between each pair of the multiple first component point clouds and the multiple second component point clouds, the target component type corresponding to the multiple second component point clouds is detected. For each of the aforementioned components, tilt influence parameter information corresponding to the component is generated based on the target component type and the second component point cloud, and tilt detection is performed on the component according to the tilt influence parameter information.

2. The method according to claim 1, characterized in that, The step of generating tilt influence parameter information corresponding to the component based on the target component type and the second component point cloud includes: When the target component type is an LED bead, a plane fitting is performed on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane; Detect the projection area information of the point cloud of the second component onto the fitting plane of the circuit board; Based on the projected area information, the tilt influence parameter information corresponding to the component is generated.

3. The method according to claim 1, characterized in that, The step of generating tilt influence parameter information corresponding to the component based on the target component type and the second component point cloud includes: When the target component type is non-planar, the three-dimensional point cloud to be measured is fitted with a plane to obtain the circuit board fitting plane; Obtain the first angle information between the preset direction vector of the second component point cloud and the normal vector of the circuit board fitting plane; Based on the first included angle information, the tilt influence parameter information corresponding to the component is generated.

4. The method according to claim 1, characterized in that, The step of generating tilt influence parameter information corresponding to the component based on the target component type and the second component point cloud includes: When the target component type is planar, the second component point cloud is fitted with a plane to obtain a component fitting plane, and the three-dimensional point cloud to be measured is fitted with a plane to obtain a circuit board fitting plane. Obtain the second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane; Based on the second included angle information, the tilt influence parameter information corresponding to the component is generated.

5. The method according to claim 1, characterized in that, The step of performing tilt detection on the component based on the tilt influence parameter information includes: For each of the aforementioned components, a preset tilt level range information matching the parameter type of the tilt influence parameter information is obtained; The tilt level of the component is detected based on the preset tilt level range information and tilt influence parameter information.

6. The method according to claim 1, characterized in that, The clustering process of the three-dimensional point cloud to be measured yields multiple point clouds of second components, including: The three-dimensional point cloud to be measured is fitted with a plane to obtain the circuit board fitting plane; Based on the circuit board fitting plane, the surface of the circuit board to be tested in the three-dimensional point cloud to be tested is separated from the multiple components on the surface to obtain the overall point cloud of the components. Clustering is performed on the overall point cloud of the components to obtain multiple second component point clouds.

7. The method according to claim 1, characterized in that, The step of detecting the target component type corresponding to the multiple second component point clouds based on the component types corresponding to the multiple component images and the matching degree between each pair of the multiple first component point clouds and the multiple second component point clouds includes: Detect the centroid distance between each pair of the centroids of the point clouds of the first components and the point clouds of the second components; Based on multiple centroid distances, the matching degree between each pair of the first component point cloud and the second component point cloud is detected; Based on the component types corresponding to the multiple component images, detect the component types to be matched in the multiple first component point clouds; For each second component point cloud, based on the matching degree between each pair of the first component point cloud and the multiple second component point clouds, and the component types to be matched in the multiple first component point clouds, the target component type of the second component point cloud is detected.

8. A component tilt detection device, characterized in that, The device includes: The data acquisition module is used to acquire the two-dimensional image of the circuit board under test and the three-dimensional point cloud of the circuit board under test, wherein multiple components are mounted on the surface of the circuit board under test. The first point cloud generation module is used to extract the component images corresponding to the multiple components from the two-dimensional circuit board image to be tested, and to extract the first component point cloud matching the multiple component images from the three-dimensional point cloud to be tested. The second point cloud generation module is used to perform clustering processing on the three-dimensional point cloud to be measured to obtain multiple second component point clouds. The type matching module is used to detect the target component type corresponding to the multiple second component point clouds based on the component types corresponding to the multiple component images and the matching degree between each pair of the multiple first component point clouds and the multiple second component point clouds. The tilt influence parameter detection module is used to generate tilt influence parameter information corresponding to each component based on the target component type and the point cloud of the second component. The tilt detection module is used to perform tilt detection on the component based on the tilt influence parameter information.

9. A component tilt detection system, characterized in that, The system includes: A vision sensor is used to acquire two-dimensional images of the circuit board under test and three-dimensional point clouds of the circuit board under test. Controller, used for: The visual sensor acquires a two-dimensional image of the circuit board under test and a three-dimensional point cloud of the circuit board under test, wherein multiple components are mounted on the surface of the circuit board under test. Extract the component images corresponding to the multiple components from the two-dimensional circuit board image to be tested, and extract the first component point cloud matching the multiple component images from the three-dimensional point cloud to be tested; Clustering is performed on the three-dimensional point cloud to be measured to obtain multiple second component point clouds; Based on the component types corresponding to the multiple component images, and the matching degree between each pair of the multiple first component point clouds and the multiple second component point clouds, the target component type corresponding to the multiple second component point clouds is detected. For each of the aforementioned components, tilt influence parameter information corresponding to the component is generated based on the target component type and the second component point cloud, and tilt detection is performed on the component according to the tilt influence parameter information.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images