Multi-direction projection AOI point cloud fusion method and system

Abstract

The invention discloses a multi-direction projection AOI point cloud fusion method and system, and relates to a surface mounting technology (SMT) and a data processing technology, and the method comprises the steps: carrying out the collection and processing based on a plurality of projectors, and obtaining four groups of point cloud data; screening according to the obtained multiple groups of point cloud data; for the screened multiple groups of point cloud data, for any camera coordinate, finding corresponding points in the multiple groups of point cloud data, calculating a fusion coordinate according to a set logic sequence, and adding the fusion coordinate to a fusion point cloud; for the fused point cloud, performing data filling on an invalid region; performing neighborhood connected region search on the screened multiple groups of point cloud data based on the fused point cloud after data filling; and according to the searched points in the multiple groups of screened point cloud data, carrying out neighborhood connected region search again, and adding the points meeting the conditions into the fused point cloud. According to the method and the device, multiple groups of point cloud data can be quickly fused with high quality, and the detection accuracy and efficiency of the AOI equipment can be improved.

Description

Technical Field

[0001] The present application relates to the fields of surface mount (SMT) technology and data processing technology, and in particular to a multi-directional projection AOI point cloud fusion method and system. Background Art

[0002] In SMT (surface mount technology) scenarios, components within the surface mount components on PCBs (printed circuit boards) need to be inspected. 3D AOI (automated optical inspection) equipment is an automated inspection device used for quality control during the PCB manufacturing process. The 3D imaging system in 3D AOI equipment typically consists of four projectors and cameras. The camera is positioned downward from the center, and four projectors are placed at similar horizontal heights around the camera in four directions. They project downward and inward, creating a common field of view for the projectors, cameras, and other five devices at the center. The projectors sequentially project a series of structured light stripe grating images onto the PCB surface. The camera then captures the grating images, which have been modulated by the PCB surface components. After image processing, the 3D point cloud data of the PCB surface components is reconstructed. Anomalies in the components are detected by performing algorithmic calculations and comparisons with the 3D point cloud data of qualified samples.

[0003] In the three-dimensional imaging method of the prior art, any one of the four projectors and the central camera can form a binocular stereo vision system, and a set of point cloud data of the object can be obtained using the binocular stereo vision three-dimensional reconstruction method. Therefore, the entire 3D imaging system can obtain four sets of point cloud data, which need to be fused. Due to the presence of highly occluded, black, or reflective areas in PCB components, each set of point cloud data contains some invalid areas and noise points. If a simple point cloud fusion algorithm is used, the quality of the fused point cloud data obtained is low, and there are invalid areas, noise points, and erroneous point clouds, which affect the effect of the subsequent defect detection algorithm and reduce the accuracy of AOI detection. If the four sets of point cloud data are first post-processed and then fused, the processing time will be increased and the efficiency of AOI detection will be reduced. Summary of the Invention

[0004] The embodiments of the present application provide a multi-directional projection AOI point cloud fusion method and system, which can quickly and high-quality fuse multiple sets of point cloud data, which is conducive to improving the accuracy and efficiency of AOI equipment detection.

[0005] The present application provides a multi-directional projection AOI point cloud fusion method, which is applied to a 3D AOI device imaging system. The 3D AOI device imaging system includes a camera, a lens, and multiple projectors, wherein the camera is placed at a central position, the lens is installed at the front end of the camera, and the multiple projectors are placed in multiple directions around the camera and are centrally symmetrical about the camera optical axis. The projector optical axis and the camera optical axis are arranged at an angle. The AOI point cloud fusion method includes:

[0006] Multiple sets of point cloud data are acquired through collection and processing based on multiple projectors;

[0007] Screening is performed based on the multiple sets of point cloud data obtained;

[0008] For the filtered multiple sets of point cloud data, for any camera coordinate, find the corresponding points in the multiple sets of point cloud data, calculate the fusion coordinates according to the set logical order, and add them to the fused point cloud;

[0009] For the fused point cloud, data is filled in the invalid areas;

[0010] Based on the fused point cloud after data filling, the neighborhood connected area search is performed on the filtered multiple groups of point cloud data;

[0011] According to the points in the multiple sets of filtered point cloud data after the search, the neighborhood connected area search is performed again, and those that meet the conditions are added to the fused point cloud.

[0012] The present application also provides a multi-directional projection AOI point cloud fusion system, which is applied to a 3D AOI device imaging system. The 3D AOI device imaging system includes a camera, a lens, and multiple projectors, wherein the camera is placed at a central position, the lens is installed at the front end of the camera, and the multiple projectors are placed in multiple directions around the camera and are centrally symmetrical about the camera optical axis. The projector optical axis and the camera optical axis are arranged at an angle.

[0013] The multi-directional projection AOI point cloud fusion system includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the aforementioned four-directional projection AOI point cloud fusion method are implemented.

[0014] There's no need for complex post-processing of the original multi-set point cloud data; the fusion algorithm can be directly applied to raw point cloud data containing invalid regions and noise. This application performs a small amount of simple post-processing on the fused point cloud during the point cloud fusion process, which speeds up the algorithm. Furthermore, the algorithm steps in this application are suitable for GPU acceleration, which helps improve the efficiency of the overall AOI inspection operation.

[0015] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0017] Figure 1 This is a schematic diagram of the 3D AOI equipment imaging system according to an embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of the basic process of the multi-directional projection AOI point cloud fusion method according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0020] like Figure 1 As shown, the 3D AOI equipment imaging system consists of a camera, a lens, and multiple projectors. The camera is placed in the center, and the lens is mounted in front of the camera. Projectors are placed in multiple directions around the camera, symmetrically about the camera's optical axis, with the projector optical axes angled to the camera's. The fields of view of the camera and the multiple projectors overlap in front of the camera lens, and the overlapping fields of view constitute the measurement area. The multiple projectors all use Sham projection, so that the focal planes of the multiple projectors overlap with the focal plane of the camera. The projectors above and below the camera are designated as Projector 1 and Projector 2; the projectors to the left and right of the camera are designated as Projector 3 and Projector 4. In the embodiments of this application, a 3D AOI equipment imaging system including four projectors is used as an example for illustration. However, the methods of this application can be applied by analogy to AOI imaging systems using more projectors, such as six- or eight-way projection. The projectors of the 3D AOI equipment imaging system of this application can use either Sham projection or conventional projection.

[0021] The embodiment of the present application provides a multi-directional projection AOI point cloud fusion method, which is applied to a 3D AOI device imaging system. The 3D AOI device imaging system includes a camera, a lens, and multiple projectors, wherein the camera is placed at a central position, the lens is installed at the front end of the camera, and the multiple projectors are placed in different directions around the camera and are centrally symmetrical about the camera optical axis. The projector optical axis and the camera optical axis are set at an angle, such as Figure 2 As shown, the AOI point cloud fusion method includes:

[0022] In step S201, four sets of point cloud data are acquired and processed based on multiple projectors. In a specific example, the process includes: controlling multiple projectors to sequentially project a set of structured light stripe grating images onto the surface of the object being measured, and controlling the computer to sequentially acquire the images to obtain multiple sets of grating images modulated by the surface of the object being measured;

[0023] For a set of raster images, image processing and stereo vision 3D reconstruction are performed to obtain a set of original point cloud data; for multiple sets of raster images, 3D reconstruction is repeated to obtain multiple sets of point cloud data.

[0024] For example, according to Figure 1 Projectors 1-4 sequentially project a set of structured light fringe grating images onto the surface of the object being measured. Cameras then capture these images, producing four sets of grating images modulated by the surface of the object being measured. For each set of grating images, conventional image processing methods and stereoscopic 3D reconstruction methods are used to calculate a set of raw point cloud data. This point cloud data is an ordered point cloud. This process is repeated for each of the four sets of grating images, resulting in four sets of point cloud data.

[0025] The four sets of point cloud data are fused to obtain the final point cloud. The subsequent examples in this application further illustrate the fusion process.

[0026] In step S202, the multiple sets of point cloud data are screened to remove outliers.

[0027] In step S203 , for each camera coordinate in the filtered multiple sets of point cloud data, corresponding points are found in each set of point cloud data, and the fused coordinates are calculated according to the set logical order and added to the fused point cloud P.

[0028] In step S204 , for the fused point cloud P, data padding is performed on the invalid areas.

[0029] In step S205 , a neighborhood connected area search is performed on the filtered multiple groups of point cloud data based on the fused point cloud after data filling.

[0030] In step S206, based on the points in the multiple groups of point cloud data that have been screened after the search, a neighborhood connected area search is performed again, and those that meet the conditions are added to the fused point cloud.

[0031] In some embodiments, screening based on the obtained multiple sets of point cloud data includes:

[0032] For multiple sets of point cloud data, check the grayscale value of the camera image corresponding to each point. If the grayscale value is less than the grayscale threshold I t, then remove the point. In this specific example, this embodiment still uses four projectors as an example, and the four sets of original point cloud data are recorded as P1, P2, P3, and P4. For each set of original point cloud data, check the grayscale value of the camera image corresponding to each point. If the grayscale value is less than the threshold It (It = 3 in this embodiment), then remove the point.

[0033] For each point, count the adjacent valid points in a small neighborhood. If the total number of adjacent valid points is less than half of the total number of adjacent points, mark the point as an outlier (this embodiment uses a 3x3 area and the total number of valid points threshold is 4). And,

[0034] Calculate the average distance from the point to all adjacent valid points within the same neighborhood If the average value is greater than the distance threshold d t (The present invention uses d t =1mm), then mark the point as an outlier.

[0035] Remove all outliers.

[0036] In some embodiments, when the 3D AOI equipment imaging system includes four projectors, for the filtered multiple sets of point cloud data P1, P2, P3, and P4, for any camera coordinate, find corresponding points in the multiple sets of point cloud data, record them as p1, p2, p3, and p4, calculate the fused coordinates according to the set logical order, and add them to the fused point cloud P (the initial fused point cloud P is empty). The specific steps include:

[0037] The filtered point cloud data are recorded as P1, P2, P3, and P4 respectively;

[0038] For the corresponding points p1, p2, p3, and p4 found, if points p1, p2, p3, and p4 are all valid, and the distance between p1 and p2 is less than the threshold pt, and the distance between p3 and p4 is less than pt (in this embodiment, pt = 0.2 mm), then the average value of p1, p2, p3, and p4 is used as the coordinate of the fused point and added to the fused point cloud P;

[0039] If points p1 and p2 are valid and the distance between p1 and p2 is less than pt, the average value of p1 and p2 is used as the coordinate of the fused point and added to the fused point cloud P.

[0040] If points p3 and p4 are valid and the distance between p3 and p4 is less than pt, the average value of p3 and p4 is used as the coordinate of the fused point and added to the fused point cloud P.

[0041] If points p1 and p3 are valid, and the distance between p1 and p3 is less than pt, the average value of p1 and p3 is used as the coordinate of the fused point and added to the fused point cloud P;

[0042] If points p1 and p4 are valid, and the distance between p1 and p4 is less than pt, the average value of p1 and p4 is used as the coordinate of the fused point and added to the fused point cloud P;

[0043] If points p2 and p3 are valid, and the distance between p2 and p3 is less than pt, the average value of p2 and p3 is used as the coordinate of the fused point and added to the fused point cloud P;

[0044] If points p2 and p4 are valid and the distance between p2 and p4 is less than pt, use the average of p2 and p4 as the coordinates of the fused point and add it to the fused point cloud P. For point cloud P, remove outliers according to the above steps.

[0045] In some embodiments, for the fused point cloud, filling data in the invalid area includes:

[0046] Scan each point and check for invalid points;

[0047] For any invalid point, count the adjacent valid points within its neighborhood;

[0048] If the total number of adjacent valid points is greater than the specified proportion of the total number of adjacent points, for example, greater than 20% of the total number of adjacent points, calculate the coordinates:

[0049]

[0050] Among them, n is the total number of adjacent valid points, p i For each adjacent valid point coordinate, w i The weight of this valid point q0 is the pixel coordinate value of the invalid point, q i is the pixel coordinate value of the adjacent valid point, and h is the Gaussian parameter used to adjust the weight of the adjacent valid point.

[0051] Add the calculated coordinates p to the fused point cloud P.

[0052] In some embodiments, performing a neighborhood connected area search on the filtered multiple groups of point cloud data based on the data-filled fused point cloud includes:

[0053] For the four sets of filtered point cloud data, if the pixel coordinates corresponding to the point already have coordinate values ​​in point cloud P, they are removed from P1, P2, P3, and P4. A simple average fusion is performed on the processed point clouds P1 and P2 to obtain point cloud P12. A simple average fusion is performed on the point clouds P3 and P4 to obtain point cloud P34.

[0054] For all points within P12 and P34, a neighborhood connected region search is performed, for example, an 8-neighborhood connected region search. A depth-first search algorithm or other connected region search algorithm can also be used. If a connected region is adjacent to any point in the fused point cloud P, the points in the connected region are added to the fused point cloud P and the points in the connected region are removed from P1, P2, P3, and P4. In this specific example, a breadth-first connected region search is used, and the connected region search radius is 5 pixels.

[0055] In some embodiments, based on the points in the multiple sets of filtered point cloud data after the search, searching for neighboring connected areas again and adding the points that meet the conditions to the fused point cloud includes:

[0056] For the points P1, P2, P3, and P4 after the neighborhood connected area search, the neighborhood connected area search is performed again, for example, an 8-neighborhood connected area search is performed here, or a depth-first search algorithm or other connected area search algorithm is used.

[0057] If the connected region is adjacent to any point in the fused point cloud P, then the point in the connected region is added to the fused point cloud P;

[0058] If there are multiple points that meet the conditions in P1, P2, P3, and P4 for a certain pixel coordinate, take the average and add them to the fused point cloud P. Then, for point cloud P, follow step 2 to remove outliers.

[0059] The method of this application does not require complex post-processing of the original four point cloud data sets. The fusion algorithm can be directly applied to raw point cloud data containing invalid areas and noise. Only a small amount of simple post-processing is required during the point cloud fusion process, which improves the algorithm's execution speed. Furthermore, the method steps of this application are suitable for GPU acceleration, which helps improve the efficiency of the overall AOI inspection operation.

[0060] This application gradually adds point clouds to the final fused point cloud based on their quality priority, and uses high-quality point clouds as a guide to assist in the fusion of lower-quality point clouds, ensuring the quality of the final fused point cloud. Compared to simpler fusion algorithms, this method provides better 3D reconstruction of PCB components, which helps improve the accuracy of subsequent defect detection.

[0061] The present application also provides a multi-directional projection AOI point cloud fusion system, which is applied to a 3D AOI device imaging system. The 3D AOI device imaging system includes a camera, a lens, and multiple projectors, wherein the camera is placed at a central position, the lens is installed at the front end of the camera, and the multiple projectors are placed in multiple directions around the camera and are centrally symmetrical about the camera optical axis. The projector optical axis and the camera optical axis are arranged at an angle.

[0062] The multi-directional projection AOI point cloud fusion system includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the aforementioned four-directional projection AOI point cloud fusion method are implemented.

[0063] It should be noted that, in the various embodiments of the present application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0064] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0065] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0066] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are protected by this application.

Claims

1. A multi-directional projection AOI point cloud fusion method, characterized in that: Applied to a 3D AOI device imaging system, the 3D AOI device imaging system includes a camera, a lens, and multiple projectors, wherein the camera is placed at a central position, the lens is installed at the front end of the camera, and the multiple projectors are placed in different directions around the camera and are centrally symmetrical about the camera optical axis. The projector optical axis and the camera optical axis are set at an angle. The AOI point cloud fusion method includes: Four sets of point cloud data are acquired and processed based on multiple projectors; Screening is performed based on the multiple sets of point cloud data obtained; For the filtered multiple sets of point cloud data, for any camera coordinate, find the corresponding points in the multiple sets of point cloud data, calculate the fusion coordinates according to the set logical order, and add them to the fused point cloud; For the fused point cloud, data is filled in the invalid areas; Based on the fused point cloud after data filling, the neighborhood connected area search is performed on the filtered multiple groups of point cloud data; According to the points in the multiple sets of filtered point cloud data after the search, the neighborhood connected area search is performed again, and those that meet the conditions are added to the fused point cloud.

2. The multi-directional projection AOI point cloud fusion method according to claim 1, characterized in that: Multiple sets of point cloud data are collected and processed based on multiple projectors, including: Control multiple projectors to sequentially project a set of structured light stripe grating images onto the surface of the object being measured, and control the computer to sequentially collect them to obtain multiple sets of grating images modulated by the surface of the object being measured; For a set of raster images, image processing and stereo vision 3D reconstruction are performed to obtain a set of original point cloud data; Multiple sets of raster images are repeatedly reconstructed into three dimensions to obtain multiple sets of point cloud data.

3. The multi-directional projection AOI point cloud fusion method according to claim 2, characterized in that: Based on the multiple sets of point cloud data obtained, screening includes: For multiple sets of point cloud data, check the grayscale value of the camera image corresponding to each point. If the grayscale value is less than the grayscale threshold I t , then remove the point; For each point, count the number of valid adjacent points in its neighborhood. If the total number of valid adjacent points is less than half of the total number of adjacent points, mark the point as an outlier; and Calculate the average distance from the point to all adjacent valid points within the same neighborhood If the average value is greater than the distance threshold d t , then mark the point as an outlier; Remove all outliers.

4. The multi-directional projection AOI point cloud fusion method according to claim 2, characterized in that: In the case where the 3D AOI equipment imaging system includes four projectors, for the filtered multiple sets of point cloud data, for any camera coordinate, respectively find corresponding points in the multiple sets of point cloud data, calculate the fused coordinates according to the set logical order, and add them to the fused point cloud, including: The filtered point cloud data are recorded as P1, P2, P3, and P4 respectively; For the corresponding points p1, p2, p3, p4 found, if points p1, p2, p3, p4 are all valid, and the distance between p1 and p2 is less than the threshold pt, and the distance between p3 and p4 is less than pt, then use the average of p1, p2, p3, p4 as the coordinate of the fused point and add it to the fused point cloud P; If points p1 and p2 are valid, and the distance between p1 and p2 is less than pt, the average value of p1 and p2 is used as the coordinate of the fused point and added to the fused point cloud P; If points p3 and p4 are valid, and the distance between p3 and p4 is less than pt, the average value of p3 and p4 is used as the coordinate of the fused point and added to the fused point cloud P; If points p1 and p3 are valid, and the distance between p1 and p3 is less than pt, the average value of p1 and p3 is used as the coordinate of the fused point and added to the fused point cloud P; If points p1 and p4 are valid, and the distance between p1 and p4 is less than pt, the average value of p1 and p4 is used as the coordinate of the fused point and added to the fused point cloud P; If points p2 and p3 are valid, and the distance between p2 and p3 is less than pt, the average value of p2 and p3 is used as the coordinate of the fused point and added to the fused point cloud P; If points p2 and p4 are valid and the distance between p2 and p4 is less than pt, the average value of p2 and p4 is used as the coordinate of the fused point and added to the fused point cloud P.

5. The multi-directional projection AOI point cloud fusion method according to claim 4, characterized in that: For the fused point cloud, data filling for invalid areas includes: Scan each point and check for invalid points; For any invalid point, count the adjacent valid points within its neighborhood; If the total number of adjacent valid points is greater than the total number of adjacent points in the specified ratio, calculate the coordinates: Among them, n is the total number of adjacent valid points, p i For each adjacent valid point coordinate, w i The weight of this valid point q0 is the pixel coordinate value of the invalid point, q i is the pixel coordinate value of the adjacent valid point, and h is the Gaussian parameter used to adjust the weight of the adjacent valid point; Add the calculated coordinates p to the fused point cloud P.

6. The multi-directional projection AOI point cloud fusion method according to claim 5, characterized in that: The search for the neighborhood connected area of ​​the filtered multiple point cloud data based on the fused point cloud after data filling includes: For the four sets of filtered point cloud data, if the pixel coordinates corresponding to the point already have coordinate values ​​in point cloud P, they are removed from P1, P2, P3, and P4. The processed point clouds P1 and P2 are averaged and fused to obtain point cloud P12. The point clouds P3 and P4 are averaged and fused to obtain point cloud P34. For all points in P12 and P34, a neighborhood connected area search is performed. If the connected area is adjacent to any point in the fused point cloud P, the points in the connected area are added to the fused point cloud P, and the points in this connected area are removed from P1, P2, P3, and P4.

7. The multi-directional projection AOI point cloud fusion method according to claim 6, characterized in that: According to the points in the filtered point cloud data after the search, the neighborhood connected area search is performed again, and the qualified points are added to the fused point cloud, including: For the points P1, P2, P3, and P4 after the neighborhood connected area search, the neighborhood connected area search is performed again; If the connected region is adjacent to any point in the fused point cloud P, then the point in the connected region is added to the fused point cloud P; If there are multiple points that meet the conditions in P1, P2, P3, and P4 for a certain pixel coordinate, the average is taken and added to the fused point cloud P.

8. A multi-directional projection AOI point cloud fusion system, characterized in that: Applied to 3D AOI equipment imaging system, the 3D AOI equipment imaging system includes a camera, a lens, and multiple projectors, wherein the camera is placed in the center, the lens is installed in front of the camera, and the multiple projectors are placed in multiple directions around the camera and are centrally symmetrical about the camera optical axis, and the projector optical axis and the camera optical axis are set at an angle; The multi-directional projection AOI point cloud fusion system includes a processor and a memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the steps of the four-directional projection AOI point cloud fusion method according to any one of claims 1 to 7 are implemented.

Images