Component inclination detection method, device and system and readable storage medium

Abstract

The invention relates to a component inclination detection method, device and system and a readable storage medium. The method comprises the following steps: extracting component images corresponding to a plurality of components from a to-be-detected two-dimensional circuit board image of a to-be-detected circuit board, and extracting a first component point cloud matched with the plurality of component images from a to-be-detected three-dimensional point cloud of the to-be-detected circuit board; performing clustering processing on the to-be-measured three-dimensional point clouds to obtain a plurality of second component point clouds; detecting target component types corresponding to the plurality of second component point clouds according to the component types corresponding to the plurality of component images and the matching degree between every two of the plurality of first component point clouds and the plurality of second component point clouds; and for each component, generating inclination influence parameter information corresponding to the component based on the target component type and the second component point cloud, and performing inclination detection on the component according to the inclination influence parameter information. By adopting the method, the inclination detection of the component on the circuit board can be accurately carried out.

Description

Technical Field

[0001] The present application relates to the field of intelligent manufacturing technology, and in particular to a component tilt detection method, device, system and readable storage medium. Background Art

[0002] Printed circuit boards (PCBs), core components of modern electronic devices, are manufactured using printed circuit board (PCB) technology and serve the critical function of providing electrical connections between electronic components. With the rapid advancement of electronic technology, PCB design has become increasingly complex. Their quality not only directly impacts the performance and reliability of electronic products but also places higher demands on manufacturing processes and testing technologies. Therefore, it is crucial to perform tilt detection on various components soldered onto soldered PCBs.

[0003] Existing methods for detecting the tilt of various types of components on circuit boards still rely on manual inspection, primarily relying on visual inspection. Workers need to carefully observe the posture of various components on the soldered circuit board and then use experience to determine whether each component is tilted. Alternatively, there are methods for detecting the tilt of various types of components on circuit boards using two-dimensional images. This involves training an image segmentation model using deep learning methods, then using the image segmentation model to segment the pixel blocks corresponding to each component from the circuit board image. The tilt of the component is then determined based on the area of ​​the pixel blocks.

[0004] However, current component tilt detection methods are still not accurate enough. Summary of the Invention

[0005] Based on this, it is necessary to provide an accurate component tilt detection method, device, system, computer-readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a component tilt detection method, comprising:

[0007] Acquire a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested, wherein a plurality of components are mounted on a surface of the circuit board to be tested;

[0008] Extracting component images corresponding to a plurality of components from the two-dimensional circuit board image to be tested, and extracting a first component point cloud matching the plurality of component images from the three-dimensional point cloud to be tested;

[0009] Performing clustering processing on the three-dimensional point cloud to be measured to obtain multiple second component point clouds;

[0010] Detecting target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds;

[0011] For each component, based on the target component type and the second component point cloud, the tilt influence parameter information corresponding to the component is generated, and the tilt detection of the component is performed according to the tilt influence parameter information.

[0012] In one embodiment, generating tilt impact parameter information corresponding to a component based on the target component type and the second component point cloud includes:

[0013] When the target component type is a lamp bead, plane fitting is performed on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane;

[0014] Detecting projection area information of the second component point cloud projected onto the circuit board fitting plane;

[0015] According to the projection area information, the tilt impact parameter information corresponding to the component is generated.

[0016] In one embodiment, generating tilt impact parameter information corresponding to a component based on the target component type and the second component point cloud includes:

[0017] When the target component type is non-planar, plane fitting is performed on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane;

[0018] Obtaining first angle information between a preset direction vector of the second component point cloud and a normal vector of the circuit board fitting plane;

[0019] According to the first angle information, tilt influence parameter information corresponding to the component is generated.

[0020] In one embodiment, generating tilt impact parameter information corresponding to a component based on the target component type and the second component point cloud includes:

[0021] When the target component type is a plane type, plane fitting is performed on the second component point cloud to obtain a component fitting plane, and plane fitting is performed on the three-dimensional point cloud to be measured to obtain a circuit board fitting plane;

[0022] Obtaining second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane;

[0023] According to the second angle information, tilt influence parameter information corresponding to the component is generated.

[0024] In one embodiment, performing tilt detection on a component based on tilt-affecting parameter information includes:

[0025] For each component, obtaining preset tilt level range information that matches the parameter type of the tilt influencing parameter information;

[0026] The tilt level of the component is detected according to the preset tilt level range information and tilt influencing parameter information.

[0027] In one embodiment, clustering is performed on the three-dimensional point cloud to obtain a plurality of second component point clouds, including:

[0028] Perform plane fitting on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane;

[0029] Separating the surface of the circuit board to be tested from the multiple components on the surface in the three-dimensional point cloud according to the circuit board fitting plane to obtain an overall point cloud of the components;

[0030] The overall point cloud of the component is clustered to obtain multiple second component point clouds.

[0031] In one embodiment, detecting target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and a matching degree between the plurality of first component point clouds and the plurality of second component point clouds includes:

[0032] Detecting the distance between the centroids of the plurality of first component point clouds and the centroids of the plurality of second component point clouds;

[0033] Based on the multiple centroid distances, detecting the matching degree between the multiple first component point clouds and the multiple second component point clouds;

[0034] Detecting component types to be matched in the plurality of first component point clouds according to component types corresponding to the plurality of component images;

[0035] For each second component point cloud, a target component type of the second component point cloud is detected based on the matching degree between the first component point cloud and the plurality of second component point clouds, and the component types to be matched of the plurality of first component point clouds.

[0036] In a second aspect, the present application further provides a device for detecting component tilt, comprising:

[0037] A data acquisition module is used to acquire a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested, wherein a plurality of components are mounted on the surface of the circuit board to be tested;

[0038] A first point cloud generation module is configured to extract component images corresponding to a plurality of components from the two-dimensional circuit board image to be tested, and to extract a first component point cloud matching the plurality of component images from the three-dimensional point cloud to be tested;

[0039] A second point cloud generation module is used to perform clustering processing on the three-dimensional point cloud to be measured to obtain multiple second component point clouds;

[0040] a type matching module, configured to detect target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and a degree of matching between the plurality of first component point clouds and the plurality of second component point clouds;

[0041] The tilt detection module is used to generate tilt influence parameter information corresponding to each component based on the target component type and the second component point cloud, and perform tilt detection on the component according to the tilt influence parameter information.

[0042] In a third aspect, the present application further provides a component tilt detection system, the system comprising:

[0043] A visual sensor is used to collect a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested;

[0044] Controller for:

[0045] Acquire a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested collected by a visual sensor, wherein a plurality of components are mounted on a surface of the circuit board to be tested;

[0046] Extracting component images corresponding to a plurality of components from the two-dimensional circuit board image to be tested, and extracting a first component point cloud matching the plurality of component images from the three-dimensional point cloud to be tested;

[0047] Performing clustering processing on the three-dimensional point cloud to be measured to obtain multiple second component point clouds;

[0048] Detecting target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds;

[0049] For each component, based on the target component type and the second component point cloud, the tilt influence parameter information corresponding to the component is generated, and the tilt detection of the component is performed according to the tilt influence parameter information.

[0050] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0051] Acquire a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested, wherein a plurality of components are mounted on a surface of the circuit board to be tested;

[0052] Extracting component images corresponding to a plurality of components from the two-dimensional circuit board image to be tested, and extracting a first component point cloud matching the plurality of component images from the three-dimensional point cloud to be tested;

[0053] Performing clustering processing on the three-dimensional point cloud to be measured to obtain multiple second component point clouds;

[0054] Detecting target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds;

[0055] For each component, based on the target component type and the second component point cloud, the tilt influence parameter information corresponding to the component is generated, and the tilt detection of the component is performed according to the tilt influence parameter information.

[0056] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0057] Acquire a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested, wherein a plurality of components are mounted on a surface of the circuit board to be tested;

[0058] Extracting component images corresponding to a plurality of components from the two-dimensional circuit board image to be tested, and extracting a first component point cloud matching the plurality of component images from the three-dimensional point cloud to be tested;

[0059] Performing clustering processing on the three-dimensional point cloud to be measured to obtain multiple second component point clouds;

[0060] Detecting target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds;

[0061] For each component, based on the target component type and the second component point cloud, the tilt influence parameter information corresponding to the component is generated, and the tilt detection of the component is performed according to the tilt influence parameter information.

[0062] The above-mentioned component tilt detection method, device, system, computer-readable storage medium and computer program product currently rely on manual tilt detection of components on circuit boards with poor accuracy. Moreover, when component tilt detection is performed using an image segmentation model, the segmented image may contain noise and be inaccurate. Therefore, in the present application, on the one hand, component images corresponding to multiple components are identified from a two-dimensional circuit board image of the circuit board to be tested, and first component point clouds matching the multiple component images are extracted from the three-dimensional point cloud to be tested. On the other hand, the three-dimensional point cloud to be tested of the circuit board to be tested is clustered into multiple second component point clouds. Since the clustered multiple second component point clouds are more accurate, the process of obtaining component types based on image extraction is more accurate. Therefore, based on the component types corresponding to the multiple component images and the matching degree between the multiple first component point clouds and the multiple second component point clouds, the target component type corresponding to the multiple second component point clouds can be detected. Then, for each component, the component tilt is accurately detected based on the tilt influence parameter information generated by the target component type and the second component point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0064] Figure 1 A diagram showing an application environment of a component tilt detection method according to an embodiment;

[0065] Figure 2 1 is a flow chart of a component tilt detection method according to an embodiment;

[0066] Figure 3 2 is a flow chart of a component tilt detection method according to another embodiment;

[0067] Figure 4 1 is a flow chart of a component tilt detection method in another embodiment;

[0068] Figure 5 This is a schematic diagram of the projection area of ​​a lamp bead in a specific application embodiment;

[0069] Figure 6 is an overall schematic diagram of a component tilt detection system in one embodiment;

[0070] Figure 7 A front view of the hardware structure of a component tilt detection system in one embodiment;

[0071] Figure 8 It is a left side view of the hardware structure of the component tilt detection system in one embodiment;

[0072] Figure 9 A top view of the hardware structure of a component tilt detection system in one embodiment;

[0073] Figure 10 2 is a structural block diagram of a device for detecting component tilt in one embodiment;

[0074] Figure 11 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are used to explain this application and are not intended to limit this application.

[0076] Printed circuit boards (PCBs), core components of modern electronic devices, are manufactured using printed circuit board technology and serve as the key electrical connection between electronic components. Detecting the tilt of various components soldered onto a soldered PCB is crucial.

[0077] Existing methods for detecting the tilt of various types of components on circuit boards still rely on manual inspection, primarily relying on visual inspection. Workers need to carefully observe the posture of various components on the soldered circuit board and then use experience to determine whether each component is tilted. Alternatively, there are methods for detecting the tilt of various types of components on circuit boards using two-dimensional images. This involves training an image segmentation model using deep learning methods, then using the image segmentation model to segment the pixel blocks corresponding to each component from the circuit board image. The tilt of the component is then determined based on the area of ​​the pixel blocks.

[0078] However, current methods for detecting component tilt, such as manual visual inspection, have significant limitations. These methods are susceptible to fluctuations in the inspector's state, experience, and subjective bias, resulting in reduced accuracy and stability, making accurate component tilt detection impossible. While methods for detecting component tilt on circuit boards using two-dimensional images can replace manual inspection, the component pixel blocks extracted by image segmentation models may not represent the complete pixel count of the corresponding component. The segmented image may contain noise, leading to biased calculation results.

[0079] To this end, the present application provides an accurate component tilt detection method. On the one hand, component images corresponding to multiple components are identified from the two-dimensional circuit board image of the circuit board to be tested, and first component point clouds matching multiple component images are extracted from the three-dimensional point cloud to be tested. On the other hand, the three-dimensional point cloud to be tested of the circuit board to be tested is clustered into multiple second component point clouds. Since the clustered multiple second component point clouds are more accurate, the component type acquisition process based on image extraction is more accurate. Therefore, according to the component types corresponding to the multiple component images and the matching degree between the multiple first component point clouds and the multiple second component point clouds, the target component types corresponding to the multiple second component point clouds can be detected. Then, for each component, the tilt of the component can be accurately detected based on the tilt influencing parameter information generated by the target component type and the second component point cloud.

[0080] The component tilt detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, a terminal 102 communicates with a controller 104 via a network. Controller 104 also communicates with a visual sensor 106. Visual sensor 106 is used to capture a two-dimensional image of a circuit board 108 under test and a three-dimensional point cloud. Multiple components 110 are mounted on the surface of circuit board 108 under test. A data storage system can store data that controller 104 needs to process. The data storage system can be integrated with controller 104 or placed on a cloud or other network server.

[0081] The user triggers the component tilt detection control on the component tilt detection interface of the terminal 102 . The terminal 102 responds to the trigger request of the component tilt detection control, generates a component tilt detection request, and sends the component tilt detection request to the controller 104 .

[0082] At this time, the controller 104 can control the visual sensor to detect the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board 108 in real time, and can also obtain the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board 108 from the database.

[0083] Furthermore, the controller 104 extracts component images corresponding to multiple components 110 from the two-dimensional circuit board image to be tested, and extracts first component point clouds matching multiple component images from the three-dimensional point cloud to be tested; clusters the three-dimensional point cloud to be tested to obtain multiple second component point clouds; detects target component types corresponding to the multiple second component point clouds based on the component types corresponding to the multiple component images and the matching degree between the multiple first component point clouds and the multiple second component point clouds; for each component 110, generates tilt influence parameter information corresponding to the component 110 based on the target component type and the second component point cloud, and performs tilt detection on the component based on the tilt influence parameter information.

[0084] The terminal 102 may be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, and projectors. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Head-mounted devices may include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like.

[0085] In an exemplary embodiment, Figure 2 As shown, a component tilt detection method is provided, which is applied to Figure 1 The controller 104 in FIG. 1 is used as an example for explanation.

[0086] S100, obtaining a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested.

[0087] The surface of the circuit board under test is mounted with multiple components, such as LEDs, planar components, and non-planar components. The circuit board under test is essentially a printed circuit board (PCB). A two-dimensional image is an array of pixels on a plane. Each pixel contains a color, brightness, or grayscale value, representing the projection of a scene from a specific perspective. A three-dimensional point cloud is a collection of discrete points in space. Each point contains three-dimensional coordinates (X, Y, Z) and optional attributes (color, reflectivity, etc.), used to accurately describe the three-dimensional structure of an object or scene.

[0088] Specifically, the circuit board to be tested is clamped by a circuit board clamping mechanism, and the two-dimensional circuit board image to be tested and the three-dimensional point cloud to be tested of the clamped circuit board to be tested are detected by a visual sensor. The two-dimensional circuit board image to be tested of the circuit board to be tested includes an overall two-dimensional image of multiple components, and the three-dimensional point cloud to be tested of the circuit board to be tested includes a three-dimensional combined point cloud of multiple components.

[0089] The visual sensor can be a depth sensor such as a scanning camera for detecting an object point cloud, or an image capture sensor for capturing an object image. The visual sensor can also be used for both detecting an object point cloud and capturing an object image. The two-dimensional circuit board image and three-dimensional point cloud of the circuit board under test obtained by the visual sensor can be fed back to the controller in real time when the controller sends a component inspection request. They can also be stored in a database for subsequent access during component tilt inspection of the circuit board under test.

[0090] That is, the controller controls the visual sensor to detect the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board in real time, and can also obtain the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board from the database.

[0091] In an exemplary embodiment, the visual sensor is disposed on a sliding guide rail, which drives the visual sensor to move so as to accurately collect the two-dimensional circuit board image and the three-dimensional point cloud of the circuit board to be tested, thereby reducing the occurrence of the situation where the visual sensor cannot detect the complete image or point cloud.

[0092] S200 , extracting component images corresponding to a plurality of components from the two-dimensional circuit board image to be tested, and extracting a first component point cloud matching the plurality of component images from the three-dimensional point cloud to be tested.

[0093] Specifically, the 2D PCB image to be tested is segmented to obtain multiple component images, each corresponding to a component type. The component types corresponding to different component images can be the same or different. The component type of a component image refers to the surface type of the component corresponding to the component image.

[0094] Then, for each component image, a point cloud index is obtained according to the pixel coordinates of the component image, and the first component point cloud that matches the point cloud index is searched from the three-dimensional point cloud to be measured.

[0095] In an exemplary embodiment, image segmentation of the two-dimensional circuit board image to be tested can be achieved through a deep learning method. For example, image segmentation of the two-dimensional circuit board image to be tested can be performed through an image segmentation model based on deep learning; or the two-dimensional circuit board image to be tested can be compared with a theoretical model of the circuit board to perform image segmentation of the two-dimensional circuit board image to be tested, etc.

[0096] S300: performing clustering processing on the three-dimensional point cloud to be measured to obtain a plurality of second component point clouds.

[0097] Specifically, unlike the aforementioned method of first segmenting the image and then mapping the resulting 2D component images to the 3D point cloud to be measured to obtain a first component point cloud matching each component image, in this step, the 3D point cloud to be measured can be directly clustered into multiple second component point clouds. In practical applications, the clustering method for the 3D point cloud to be measured can include Euclidean clustering, region growing clustering, and density-based clustering.

[0098] In an exemplary embodiment, before clustering the three-dimensional point cloud to be measured, it is necessary to filter the three-dimensional point cloud to be measured to remove discrete points in the three-dimensional point cloud to be measured. The filtering method can be statistical filtering, voxel grid filtering, clustering filtering, radius filtering, etc.

[0099] Among them, the statistical filtering method refers to a statistical method based on the assumption that the neighborhood points of each point in the point cloud should satisfy a certain distribution law. By calculating the average distance from each point to its neighborhood points, points (noise) that deviate too much from the mean are identified and removed; the voxel grid filtering method refers to dividing the point cloud into a regular three-dimensional voxel grid, replacing all points in each voxel with a representative point (such as the center of mass), thereby downsampling the point cloud; the radius filtering method refers to counting the number of neighborhood points within the radius of each point. If the number of neighborhood points is less than the threshold, it is regarded as a noise point and removed.

[0100] S400 , detecting target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and matching degrees between the plurality of first component point clouds and the plurality of second component point clouds.

[0101] Specifically, a matching degree analysis is performed on the plurality of first component point clouds obtained by image segmentation and the plurality of second component point clouds obtained by clustering to obtain the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds.

[0102] Among them, the matching degree analysis can be obtained through the center of mass distance between multiple first component point clouds and multiple second component point clouds, or through the point cloud overlap between multiple first component point clouds and multiple second component point clouds. For example, if the point cloud overlap between a first component point cloud and a second component point cloud is 90%, then the matching degree can also be considered to be 90%.

[0103] Furthermore, each first component point cloud is extracted based on the component image. Therefore, the component type of the component image is the component type of the corresponding queried first component point cloud. Then, based on the matching degree between multiple first component point clouds and multiple second component point clouds, the target component types corresponding to multiple second component point clouds can be detected.

[0104] It needs to be explained that the reason why the first component point cloud is not directly used to perform component tilt detection is: when multiple component images are segmented from the two-dimensional circuit board image to be tested, inaccurate image segmentation may occur, resulting in noise in the component image. In the process of extracting the first component point cloud that matches multiple component images from the three-dimensional point cloud to be tested, although the component type corresponding to each first component point cloud can be accurately obtained based on the component type corresponding to the component image, the first component point cloud corresponding to the component image will also have noise due to the noise in the component image. Directly clustering the three-dimensional point cloud to be tested into multiple second component point clouds will have less noise. Therefore, on the one hand, component images corresponding to multiple components are extracted from the two-dimensional circuit board image of the circuit board to be tested, and first component point clouds matching multiple component images are extracted from the three-dimensional point cloud to be tested. On the other hand, the three-dimensional point cloud to be tested of the circuit board to be tested is clustered into multiple second component point clouds. Since the clustered multiple second component point clouds are more accurate, the component type acquisition process based on the multiple component images extracted from the image is more accurate. Therefore, the target component type corresponding to the multiple second component point clouds can be detected according to the component types corresponding to the multiple component images and the matching degree between the multiple first component point clouds and the multiple second component point clouds.

[0105] S500 , for each component, generating tilt influence parameter information corresponding to the component based on the target component type and the second component point cloud, and performing tilt detection on the component according to the tilt influence parameter information.

[0106] Specifically, for each component, the target component type is used to determine the parameter type of the tilt influence parameter information. That is, different target component types of the second component point cloud have different corresponding tilt influence parameter information.

[0107] Therefore, the parameter type of the tilt-affecting parameter information corresponding to the target component can be determined based on the target component type. Based on the parameter type and the second component point cloud, the tilt-affecting parameter information corresponding to the component can be generated. In practical applications, the parameter type of the tilt-affecting parameter information can be angle information, projected area information, etc.

[0108] Furthermore, generating the tilt influence parameter information corresponding to the component according to the parameter type and the second component point cloud also includes: performing plane fitting on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane; generating the tilt influence parameter information corresponding to the component according to the circuit board fitting plane and the second component point cloud, for example, the tilt influence parameter information corresponding to the component can be generated according to the projected area of ​​the second component point cloud projected onto the circuit board fitting plane; and the tilt influence parameter information corresponding to the component can also be generated according to the angle between the normal vector of the second component point cloud and a vector in a certain direction of the second component point cloud.

[0109] Furthermore, based on the tilt influencing parameter information, the component is tilted and a tilt detection result of the component is obtained. The tilt detection result is used to characterize whether the component is tilted. Tilt refers to the deviation angle of an object or plane relative to a reference direction (such as a horizontal plane or a vertical plane).

[0110] Furthermore, performing tilt detection on components based on the tilt influencing parameter information may include: obtaining preset tilt threshold information matching the target component type, and performing tilt detection on the component based on the tilt influencing parameter information and the preset tilt threshold range; wherein, performing tilt detection on the component based on the tilt influencing parameter information and the preset tilt threshold range also includes: when the tilt influencing parameter information and the preset tilt threshold range indicate that the tilt influencing parameter value is not within the preset tilt threshold range, the component is tilted; when the tilt influencing parameter information and the preset tilt threshold range indicate that the tilt influencing parameter value is within the preset tilt threshold range, the component is not tilted.

[0111] In the above-mentioned component tilt detection method, currently, the accuracy of manual tilt detection of components on the circuit board is poor, and when the component tilt detection is performed through the image segmentation model, the segmented image may have noise and is not accurate enough; therefore, in the present application, on the one hand, component images corresponding to multiple components are identified from the two-dimensional circuit board image of the circuit board to be tested, and a first component point cloud matching the multiple component images is extracted from the three-dimensional point cloud to be tested; on the other hand, the three-dimensional point cloud to be tested of the circuit board to be tested is clustered into multiple second component point clouds. Since the clustered multiple second component point clouds are more accurate, the component type acquisition process based on image extraction is more accurate. Therefore, according to the component types corresponding to the multiple component images and the matching degree between the multiple first component point clouds and the multiple second component point clouds, the target component types corresponding to the multiple second component point clouds can be detected, and then for each component, the tilt influence parameter information generated based on the target component type and the second component point cloud can be used to accurately detect the component tilt.

[0112] In an exemplary embodiment, the target component types include at least lamp beads, non-planar types, and planar types. It should be noted that lamp beads are also a type of component, but due to their transparent material, the accuracy of the collected 3D point cloud is very poor, so they are separately detected as a type of component.

[0113] Therefore, if Figure 3 As shown, multiple embodiments will be used below to explain in detail how to generate corresponding tilt influence parameter information that matches a component with a target component type when the target component type of the component is different.

[0114] In an exemplary embodiment, Figure 3 As shown, S500 includes:

[0115] S512: When the target component type is a lamp bead, plane fitting is performed on the three-dimensional point cloud to obtain a circuit board fitting plane.

[0116] S514 , detecting projection area information of the second component point cloud projected onto the circuit board fitting plane.

[0117] S516: Generate tilt impact parameter information corresponding to the component according to the projection area information.

[0118] S540 , performing tilt detection on components according to the tilt influencing parameter information.

[0119] Specifically, when the target component type is a lamp bead, plane fitting is performed on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane, the second component point cloud is projected onto the circuit board fitting plane to obtain the projection area information of the second component point cloud, and the projection area information of the second component point cloud is used as the tilt influence parameter information of the second component point cloud.

[0120] Furthermore, taking the vertical direction as an example, the component tilt reference direction is detected based on the tilt influencing parameter information, including:

[0121] When the target component type is a lamp bead, the tilt-affecting parameter information of the second component point cloud is the projection area information of the second component point cloud projected onto the circuit board fitting plane. The projection area is positively correlated with the tilt of the component. The tilt detection result of the component is obtained by judging whether the projection area represented by the projection area information is greater than the preset projection area threshold. If the projection area is greater than the preset projection area threshold, the component is tilted. Figure 5 As shown, the circuit board is a PCB board. When the lamp bead is vertically arranged on a certain surface of the PCB board, the projection area is the upper surface area of ​​the lamp bead. When the lamp bead is tilted at a certain angle to the vertical direction, the projection area is larger than the upper surface area of ​​the lamp bead.

[0122] In the above embodiment, by detecting the projection area information of the second component point cloud projected onto the circuit board fitting plane, the tilt influence parameter information of the lamp bead can be accurately generated.

[0123] In an exemplary embodiment, Figure 3 As shown, S500 includes:

[0124] S522: When the target component type is a non-planar type, plane fitting is performed on the three-dimensional point cloud to be measured to obtain a circuit board fitting plane.

[0125] S524 , obtaining first angle information between a preset direction vector of the second component point cloud and a normal vector of the circuit board fitting plane.

[0126] S526: Generate tilt impact parameter information corresponding to the component according to the first angle information.

[0127] S540 , performing tilt detection on components according to the tilt influencing parameter information.

[0128] Specifically, if the target component type is non-planar, a plane fitting is performed on the three-dimensional point cloud to be measured to obtain the circuit board fitting plane. The three principal directions of the second component point cloud are then obtained through PCA principal component analysis. The smallest principal direction vector is used as the preset direction vector. The first angle between the preset direction vector of the second component point cloud and the normal vector of the circuit board fitting plane is detected, and this first angle information is used as the tilt influence parameter information of the second component point cloud.

[0129] Taking the vertical direction as the reference direction for the tilt of the component as an example, at this time, the first angle represented by the first angle information is positively correlated with the tilt of the component. If the first angle is greater than the preset first angle threshold, the component is tilted. If the first angle is less than or equal to the preset first angle threshold, the component can be regarded as not tilted.

[0130] In the above embodiment, by detecting the first angle information between the preset direction vector of the second component point cloud and the normal vector of the circuit board fitting plane, the tilt influence parameter information of the non-planar component can be accurately generated.

[0131] In an exemplary embodiment, S500 includes:

[0132] S532 , when the target component type is a plane type, plane fitting is performed on the second component point cloud to obtain a component fitting plane, and plane fitting is performed on the three-dimensional point cloud to be measured to obtain a circuit board fitting plane.

[0133] S534 , obtaining second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane.

[0134] S536: Generate tilt impact parameter information corresponding to the component according to the second angle information.

[0135] S540 , performing tilt detection on components according to the tilt influencing parameter information.

[0136] Specifically, when the target component type is a plane type, plane fitting is performed on the second component point cloud to obtain a component fitting plane, wherein the plane fitting method can be a RANSAC plane fitting method, or a least squares plane fitting method, or a PCA plane fitting method; then the second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane is obtained, and the second angle information is used as the tilt influence parameter information of the second component point cloud.

[0137] Taking the vertical direction as the reference direction for the tilt of the component as an example, at this time, the second angle represented by the second angle information is positively correlated with the tilt of the component. If the second angle is greater than the preset second angle threshold, the component is tilted. If the second angle is less than or equal to the preset second angle threshold, the component can be regarded as not tilted.

[0138] In the above embodiment, by detecting the second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane, the tilt influence parameter information of the planar component can be accurately generated.

[0139] In an exemplary embodiment, performing tilt detection on a component according to tilt-affecting parameter information includes:

[0140] For each component, preset tilt level range information matching the parameter type of the tilt influencing parameter information is obtained; and the tilt level of the component is detected according to the preset tilt level range information and the tilt influencing parameter information.

[0141] Specifically, the tilt detection result of the components of the present application can not only characterize whether the components are tilted, but also determine the tilt level of the components. More specifically, for each component, a preset tilt level range information that matches the parameter type of the tilt-affecting parameter information is obtained. For example, when the parameter type of the tilt-affecting parameter information is projection area information, the preset tilt level range information includes that when the projection area is within the first projection area range, the tilt level is level one, when the projection area is within the second projection area range, the tilt level is level two, when the projection area is within the third projection area range, the tilt level is level three, etc.; determine whether the tilt-affecting parameter information is within a target level range in the preset tilt level range information. If it is within a target level range, the level corresponding to the target level range is determined as the tilt level of the component.

[0142] In this embodiment, the tilt level of the component can be accurately detected by obtaining preset tilt level range information that matches the parameter type of the tilt influencing parameter information and comparing and analyzing the preset tilt level range information with the tilt influencing parameter information.

[0143] In an exemplary embodiment, clustering is performed on the three-dimensional point cloud to be measured to obtain a plurality of second component point clouds, including:

[0144] A plane fitting is performed on the three-dimensional point cloud to obtain a circuit board fitting plane; based on the circuit board fitting plane, a surface of the circuit board to be measured and multiple components on the surface in the three-dimensional point cloud to be measured are separated to obtain an overall point cloud of the components; and a clustering process is performed on the overall point cloud of the components to obtain multiple second component point clouds.

[0145] Specifically, a plane fitting is performed on the three-dimensional point cloud to be tested of the circuit board to obtain a circuit board fitting plane. The plane fitting method can be a RANSAC (Random Sample Consensus) plane fitting method, or a least squares plane fitting method and a PCA (Principal Component Analysis) plane fitting method. In this embodiment, the plane fitting method is the RANSAC plane fitting method.

[0146] Based on the plane parameters of the circuit board fitting plane, the distance from the circuit board fitting plane in the 3D point cloud to be measured is detected. Based on the distance from the circuit board fitting plane in the 3D point cloud to the 3D point cloud and a preset distance threshold, the surface of the circuit board to be measured and the multiple components on the surface of the circuit board to be measured are separated in the 3D point cloud to extract the overall component point cloud. In practical applications, the points in the 3D point cloud to be measured whose distance from the circuit board fitting plane is greater than the preset distance threshold are often regarded as the overall component point cloud.

[0147] The point cloud at this time is an overall point cloud that does not distinguish between multiple components. Therefore, it is necessary to cluster the overall point cloud of the components to divide the overall point cloud of the components into a single point cloud for each component to obtain a second component point cloud of multiple components.

[0148] In the above embodiment, by performing plane fitting on the three-dimensional point cloud to be measured, the circuit board to be measured and the multiple components on the surface of the circuit board to be measured in the three-dimensional point cloud to be measured can be accurately separated to obtain the overall point cloud of the components, and then the overall point cloud of the components is clustered to accurately obtain the second component point cloud of multiple components.

[0149] In an exemplary embodiment, extracting component images corresponding to a plurality of components from a two-dimensional circuit board image to be tested includes:

[0150] Utilizing a pre-trained image segmentation model, component images corresponding to multiple components are extracted from the two-dimensional circuit board image to be tested.

[0151] Specifically, a pre-trained image segmentation model is used to process the image of the two-dimensional circuit board to be tested to obtain component images corresponding to multiple components and the component type of each component image. The pre-trained image segmentation model can be a YOLOv8-seg segmentation model or other deep learning image segmentation models.

[0152] The training process of the image segmentation model includes: scanning multiple groups of circuit boards within a historical time period to obtain multiple groups of two-dimensional historical circuit board images, and determining multiple historical component images in the historical circuit board images and the component type corresponding to each historical component image, and using the measured multiple groups of historical circuit board images, multiple historical component images in the historical circuit board images, and the component type corresponding to each historical component image to train the image segmentation model, wherein the component types include at least lamp beads, plane types, and non-plane types, etc. At this time, the trained image segmentation model is used to segment the three types of component images: lamp beads, plane components, and components with non-plane upper surfaces.

[0153] In the above embodiment, the image segmentation model is trained using historical circuit board images, multiple historical component images in the historical circuit board images, and the component type corresponding to each historical component image, so that the trained image segmentation model can accurately segment three types of component images: lamp beads, planar components, and components with non-planar upper surfaces.

[0154] In an exemplary embodiment, Figure 4 As shown, S400 includes:

[0155] S410 , detecting the distance between the centroids of the plurality of first component point clouds and the centroids of the plurality of second component point clouds.

[0156] S420 : Based on the multiple centroid distances, detect the matching degree between the multiple first component point clouds and the multiple second component point clouds.

[0157] S430 : Detect component types to be matched in a plurality of first component point clouds according to component types corresponding to a plurality of component images.

[0158] S440 , for each second component point cloud, based on the matching degree between the first component point cloud and the plurality of second component point clouds, and the component types to be matched of the plurality of first component point clouds, detecting the target component type of the second component point cloud.

[0159] Specifically, the first component point cloud and the second component point cloud of the component are point clouds detected by different methods for the same component. Therefore, the position difference between the first component point cloud corresponding to the same component and the second component point cloud matched with the first component point cloud will not be very large. Taking the centroid of multiple first component point clouds and multiple second component point clouds as the reference point, each first component point cloud is matched with the corresponding second component point cloud according to the principle of the shortest distance between the centroids.

[0160] That is to say, first, the first center of mass positions of multiple first component point clouds and the second center of mass positions of multiple second component point clouds are detected, and based on the multiple first center of mass positions and the multiple second center of mass positions, the center of mass distances between the multiple first component point clouds and the multiple second component point clouds are detected, and then the matching degree between the multiple first component point clouds and the multiple second component point clouds is detected through the multiple center of mass distances. For example, if the center of mass distance between the first component point cloud A and the second component point cloud b is longer, the matching degree is lower, and if the center of mass distance between the first component point cloud A and the second component point cloud b is closer, the matching degree is higher.

[0161] Furthermore, for each first component point cloud, the second component point cloud with the highest matching degree with the first component point cloud among the plurality of second component point clouds is used as the second component point cloud matching the first component point cloud.

[0162] Since the first component point cloud is extracted from the 3D point cloud to be measured using component images, the component types to be matched for the multiple first component point clouds are detected based on the component types corresponding to the multiple component images. The component types to be matched for the first component point clouds are consistent with the component types of the component images corresponding to the first component point clouds. After determining the second component point cloud that matches each first component point cloud, the component type to be matched for the first component point cloud is used as the target component type for the second component point cloud.

[0163] In the above embodiment, by detecting the center of mass distance between the center of mass of the first component point cloud and the center of mass of each second component point cloud, the matching degree between multiple first component point clouds and multiple second component point clouds can be detected, and then the second component point cloud matching the first component point cloud can be accurately determined, so as to match the component types between multiple component images, multiple first component point clouds and multiple second component point clouds, thereby improving the accuracy of determining the component type of the second component point cloud.

[0164] In an exemplary embodiment, the component tilt detection method provided in the embodiment of the present application can be applied to a component tilt detection system. The component tilt detection system is composed of a component tilt detection hardware structure 10 and a controller 104. Figure 6 The overall schematic diagram of the component tilt detection system is shown in Figure 1. The component tilt detection hardware structure 10 includes a 3D line scan camera 1, a sliding guide rail 2, a circuit board clamping mechanism 3, and a box 4. The software part is the controller 5. Figure 7 It is a front view of the component tilt detection hardware structure 10, Figure 8 This is a left view of the component tilt detection hardware structure 10. Figure 9 FIG. 1 is a top view of the component tilt detection hardware structure 10 .

[0165] S1 places the circuit board to be tested on the circuit board clamping mechanism 3, and the sliding guide rail 2 drives the 3D line scan camera 1 to move to scan the component surface of the circuit board to be tested, and obtains the 2D circuit board image and 3D point cloud of the circuit board to be tested.

[0166] S2 uses the pre-trained YOLOv8-seg segmentation model to infer the 2D PCB image of the PCB under test, obtaining segmented pixel blocks for each component. Based on the pixel coordinates of each segmented pixel block, it obtains a point cloud index, extracts corresponding points from the 3D point cloud of the PCB under test, obtains the first component point cloud for each component, and calculates the first centroid position of each component's first component point cloud.

[0167] S3 removes discrete points from the 3D point cloud of the circuit board to be tested by a statistical filtering method, fits the circuit board fitting plane of the circuit board to be tested by the RANSAC fitting plane method, extracts the overall point cloud of the components above the circuit board fitting plane according to a certain distance threshold exceeding the circuit board fitting plane, uses the Euclidean clustering method to divide the overall point cloud of the components into a single-piece second component point cloud of each component, and calculates the second centroid position of the second component point cloud of each component.

[0168] S4 calculates the centroid distance between the first centroid position of the first component point cloud and the second centroid position of the second component point cloud for each component. According to the principle of the closest centroid distance, the second component point cloud corresponding to the smallest centroid distance is used as the second component point cloud matching the first component point cloud, and each second component point cloud is matched with the component type label of the first component point cloud, such as three types: lamp beads, components with a planar upper surface, and components with a non-planar upper surface.

[0169] The following steps all take the vertical direction as the reference direction for component tilt:

[0170] S5 projects the second component point cloud of the lamp bead type onto the circuit board fitting plane, calculates the projection area, sets the projection area threshold, and determines that the lamp bead is tilted when the projection area is greater than the projection area threshold.

[0171] S6 calculates the three main directions of the point cloud for the second component point cloud whose upper surface is a non-planar component type through the PCA principal component analysis method, and takes the angle between the minimum main direction vector n1 and the normal vector n of the circuit board fitting plane as the first tilt angle. When the first tilt angle is greater than the set first angle threshold, it is determined that the component with the non-planar upper surface is tilted.

[0172] S7 is for the second component point cloud whose upper surface is a planar type. The upper surface plane of the component is fitted by RANSAC to obtain the component fitting plane. The angle between the normal vector n of the circuit board fitting plane and the normal vector n2 of the component fitting plane is used as the second tilt angle. When the second tilt angle is greater than the set second angle threshold, it is determined that the component with a planar upper surface is tilted.

[0173] It can be seen that this application is a high-precision circuit board component tilt detection method. The circuit board is clamped by using a clamping mechanism, and then the sliding guide rail drives the 3D line scan camera to scan the component surface of the circuit board to obtain a 2D image and a 3D point cloud. The image segmentation model is trained using multiple sets of 2D images of circuit boards, and the trained image segmentation model is used to infer and segment the 2D images of the circuit board to be tested. Then, the 3D point cloud of the circuit board to be tested is filtered and fitted to the plane of the circuit board. The Euclidean clustering results of the point cloud above the fitting plane of the circuit board are matched with the 2D image inference segmentation results to obtain the label of each clustered point cloud: lamp beads, planar components on the upper surface, or non-planar components. For the lamp bead point cloud, the inclination is determined by calculating the area projected onto the fitting plane of the circuit board. For components with non-planar upper surfaces, the angle between the minimum principal direction vector n1 calculated by PCA principal component analysis and the normal vector n of the fitting plane of the circuit board is the inclination angle. For the point cloud of components with planar upper surfaces, the normal vector n2 is obtained by calculating the fitting plane of the planar component point cloud, and the inclination angle is obtained by taking the angle between n2 and the normal vector n of the large plane of the circuit board.

[0174] Compared with manual inspection, the detection efficiency is improved and the subjectivity of manual inspection is avoided. At the same time, compared with 2D image detection, the combined processing of 2D images and 3D point clouds: segmenting three types of components from 2D images, and then locating and matching 3D point cloud cluster labels to calculate the tilt angle method, greatly improves the detection accuracy of the tilt angle of circuit board components.

[0175] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0176] Based on the same inventive concept, embodiments of the present application also provide a component tilt detection device for implementing the aforementioned component tilt detection method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more component tilt detection device embodiments provided below can be found in the limitations of the component tilt detection method described above and will not be further elaborated here.

[0177] In an exemplary embodiment, Figure 10 As shown, a device for detecting component tilt is provided, comprising: a data acquisition module 100, a first point cloud generation module 200, a second point cloud generation module 300, a type matching module 400, a tilt influence parameter detection module 500, and a tilt detection module 600, wherein:

[0178] The data acquisition module 100 is used to acquire a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested, wherein a plurality of components are mounted on the surface of the circuit board to be tested;

[0179] A first point cloud generation module 200 is configured to extract component images corresponding to a plurality of components from the two-dimensional circuit board image to be tested, and to extract a first component point cloud matching the plurality of component images from the three-dimensional point cloud to be tested;

[0180] The second point cloud generating module 300 is used to perform clustering processing on the three-dimensional point cloud to be measured to obtain a plurality of second component point clouds;

[0181] A type matching module 400 is configured to detect target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds;

[0182] The tilt detection module 500 is used to generate tilt influence parameter information corresponding to each component based on the target component type and the second component point cloud, and perform tilt detection on the component according to the tilt influence parameter information.

[0183] In one embodiment, the tilt detection module 500 is also used to perform plane fitting on the three-dimensional point cloud to be measured when the target component type is a lamp bead, to obtain a circuit board fitting plane; detect the projection area information of the second component point cloud projected onto the circuit board fitting plane; and generate the tilt influence parameter information corresponding to the component based on the projection area information.

[0184] In one embodiment, the tilt detection module 500 is also used to perform plane fitting on the three-dimensional point cloud to be measured when the target component type is a non-planar type to obtain a circuit board fitting plane; obtain the first angle information between the preset direction vector of the second component point cloud and the normal vector of the circuit board fitting plane; and generate the tilt influence parameter information corresponding to the component based on the first angle information.

[0185] In one embodiment, the tilt detection module 500 is also used to perform plane fitting on the second component point cloud to obtain a component fitting plane when the target component type is a plane type, and to perform plane fitting on the three-dimensional point cloud to be measured to obtain a circuit board fitting plane; obtain the second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane; and generate the tilt influence parameter information corresponding to the component based on the second angle information.

[0186] In one embodiment, the tilt detection module 500 is further configured to obtain, for each component, preset tilt level range information matching the parameter type of the tilt influencing parameter information; and detect the tilt level of the component based on the preset tilt level range information and the tilt influencing parameter information.

[0187] In one embodiment, the second point cloud generation module 300 is also used to perform plane fitting on the three-dimensional point cloud to be measured to obtain a circuit board fitting plane; according to the circuit board fitting plane, the surface of the circuit board to be measured in the three-dimensional point cloud to be measured and the multiple components on the surface are separated to obtain an overall point cloud of the components; the overall point cloud of the components is clustered to obtain multiple second component point clouds.

[0188] In one embodiment, the type matching module 400 is also used to detect the center of mass distances between the center of mass of multiple first component point clouds and the center of mass of multiple second component point clouds; based on the multiple center of mass distances, detect the matching degree between the multiple first component point clouds and the multiple second component point clouds; detect the component types to be matched of the multiple first component point clouds according to the component types corresponding to the multiple component images; for each second component point cloud, detect the target component type of the second component point cloud based on the matching degree between the first component point cloud and the multiple second component point clouds, and the component types to be matched of the multiple first component point clouds.

[0189] Each module in the aforementioned component tilt detection device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor within a computer device in hardware form, or stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.

[0190] In an exemplary embodiment, a component tilt detection system is also provided, the system comprising:

[0191] A visual sensor is used to collect a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested;

[0192] Controller for:

[0193] Acquire a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested collected by a visual sensor, wherein a plurality of components are mounted on a surface of the circuit board to be tested;

[0194] Extracting component images corresponding to a plurality of components from the two-dimensional circuit board image to be tested, and extracting a first component point cloud matching the plurality of component images from the three-dimensional point cloud to be tested;

[0195] Performing clustering processing on the three-dimensional point cloud to be measured to obtain multiple second component point clouds;

[0196] Detecting target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds;

[0197] For each component, based on the target component type and the second component point cloud, the tilt influence parameter information corresponding to the component is generated, and the tilt detection of the component is performed according to the tilt influence parameter information.

[0198] Specifically, the visual sensor is used to collect the two-dimensional circuit board image to be tested and the three-dimensional point cloud to be tested of the circuit board, and the controller is used to execute all the steps in the above-mentioned component tilt detection method embodiment, which will not be repeated here.

[0199] Furthermore, the component tilt detection system includes not only a visual sensor and controller, but also a sliding rail, a circuit board clamping mechanism, and a housing. The circuit board clamping mechanism is used to clamp the circuit board under test, and the sliding rail, circuit board clamping mechanism, and visual sensor are all located within the housing. The visual sensor is mounted on the sliding rail, driving its movement to accurately capture a 2D image of the circuit board under test and a 3D point cloud of the circuit board under test.

[0200] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 11As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a two-dimensional circuit board image to be tested and a three-dimensional point cloud to be tested of the circuit board to be tested. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a component tilt detection method is implemented.

[0201] Those skilled in the art will understand that Figure 11 The structure shown in the figure is a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0202] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0203] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0204] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0205] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Among them, any reference to memory, database or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, artificial intelligence (AI) processors, and the like.

[0206] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0207] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A component tilt detection method, characterized in that: The method comprises: Acquire a two-dimensional circuit board image and a three-dimensional point cloud of a circuit board to be tested, wherein a plurality of components are mounted on a surface of the circuit board to be tested; Extracting component images corresponding to the plurality of components from the two-dimensional circuit board image to be tested, and extracting a plurality of first component point clouds matching the component images from the three-dimensional point cloud to be tested; performing clustering processing on the three-dimensional point cloud to be measured to obtain a plurality of second component point clouds; Detecting target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds; For each of the components, tilt influence parameter information corresponding to the component is generated based on the target component type and the second component point cloud, and tilt detection is performed on the component according to the tilt influence parameter information.

2. The method according to claim 1, characterized in that The generating, based on the target component type and the second component point cloud, tilt influence parameter information corresponding to the component includes: When the target component type is a lamp bead, performing plane fitting on the three-dimensional point cloud to be measured to obtain a circuit board fitting plane; Detecting projection area information of the second component point cloud projected onto the circuit board fitting plane; Tilt influence parameter information corresponding to the component is generated according to the projection area information.

3. The method according to claim 1, characterized in that The generating, based on the target component type and the second component point cloud, tilt influence parameter information corresponding to the component includes: In the case where the target component type is a non-planar type, performing plane fitting on the three-dimensional point cloud to be measured to obtain a circuit board fitting plane; Acquire first angle information between a preset direction vector of the second component point cloud and a normal vector of the circuit board fitting plane; Tilt influence parameter information corresponding to the component is generated according to the first angle information.

4. The method according to claim 1, wherein The generating, based on the target component type and the second component point cloud, tilt influence parameter information corresponding to the component includes: When the target component type is a plane type, performing plane fitting on the second component point cloud to obtain a component fitting plane, and performing plane fitting on the three-dimensional point cloud to be measured to obtain a circuit board fitting plane; Acquire second angle information between the normal vector of the component fitting plane and the normal vector of the circuit board fitting plane; Tilt influence parameter information corresponding to the component is generated according to the second angle information.

5. The method according to claim 1, characterized in that The step of performing tilt detection on the component according to the tilt influencing parameter information includes: For each of the components, obtaining preset tilt level range information that matches the parameter type of the tilt influencing parameter information; The tilt level of the component is detected according to the preset tilt level range information and the tilt influencing parameter information.

6. The method according to claim 1, characterized in that The clustering process of the three-dimensional point cloud to be measured to obtain a plurality of second component point clouds includes: Performing plane fitting on the three-dimensional point cloud to be measured to obtain a circuit board fitting plane; Separating the surface of the circuit board to be tested from the plurality of components on the surface in the three-dimensional point cloud to obtain an overall point cloud of the components according to the circuit board fitting plane; Clustering is performed on the overall point cloud of the component to obtain a plurality of second component point clouds.

7. The method according to claim 1, characterized in that The detecting target component types corresponding to the plurality of second component point clouds according to the component types corresponding to the plurality of component images and the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds comprises: Detecting the distance between the centroids of the plurality of first component point clouds and the centroids of the plurality of second component point clouds; Based on the plurality of centroid distances, detecting a degree of matching between a plurality of the first component point clouds and a plurality of the second component point clouds; Detecting component types to be matched in a plurality of first component point clouds according to component types corresponding to a plurality of component images; For each of the second component point clouds, based on the matching degree between the first component point cloud and multiple second component point clouds, and the component types to be matched of the multiple first component point clouds, the target component type of the second component point cloud is detected.

8. A component tilt detection device, characterized in that: The device comprises: a data acquisition module, configured to acquire a two-dimensional circuit board image and a three-dimensional point cloud of a circuit board to be tested, wherein a plurality of components are mounted on a surface of the circuit board to be tested; A first point cloud generating module is configured to extract component images corresponding to the plurality of components from the two-dimensional circuit board image to be tested, and extract a plurality of first component point clouds matching the component images from the three-dimensional point cloud to be tested; A second point cloud generation module is used to perform clustering processing on the three-dimensional point cloud to be measured to obtain a plurality of second component point clouds; a type matching module, configured to detect target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and a degree of matching between the plurality of first component point clouds and the plurality of second component point clouds; The tilt detection module is used to generate tilt influence parameter information corresponding to each component based on the target component type and the second component point cloud, and perform tilt detection on the component according to the tilt influence parameter information.

9. A component tilt detection system, characterized in that: The system comprises: A visual sensor is used to collect a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested; Controller for: Acquire a two-dimensional circuit board image and a three-dimensional point cloud of the circuit board to be tested collected by the visual sensor, wherein a plurality of components are mounted on a surface of the circuit board to be tested; Extracting component images corresponding to the plurality of components from the two-dimensional circuit board image to be tested, and extracting a plurality of first component point clouds matching the component images from the three-dimensional point cloud to be tested; performing clustering processing on the three-dimensional point cloud to be measured to obtain a plurality of second component point clouds; Detecting target component types corresponding to the plurality of second component point clouds based on component types corresponding to the plurality of component images and the matching degrees between the plurality of first component point clouds and the plurality of second component point clouds; For each of the components, tilt influence parameter information corresponding to the component is generated based on the target component type and the second component point cloud, and tilt detection is performed on the component according to the tilt influence parameter information.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images