3D solder paste height detection instrument and detection method based on two-view projection

Abstract

The invention discloses a 3D solder paste height detection instrument and method based on two-view projection, and belongs to the field of multi-view structured light three-dimensional reconstruction. The 3D solder paste height detection device solves the problems that an existing 3D solder paste height detection instrument is complex in structure, long in height detection time and low in detection precision. The method comprises the following steps: firstly, calibrating a center camera and projection light machines on two sides by means of a calibration plate, and establishing an absolute phase compensation amount calculation model based on a calibration result; then projecting fringes to the solder paste to be measured by using the projection light machines on the two sides and collecting fringe patterns, further recovering absolute phases under views of the projection light machines on the two sides and performing distortion removal processing, performing phase fusion on the distortionless absolute phases of the two projection light machines under a view of the reference projection light machine, and recovering a distortionless absolute phase pattern of a full view; and finally, recovering the solder paste height map based on the full-view distortionless absolute phase map, and carrying out median filtering denoising and complementation to obtain a processed solder paste height map. The method can be applied to 3D solder paste height detection.

Description

Technical Field

[0001] This invention relates to the field of multi-view structured light 3D reconstruction, specifically to a 3D solder paste height detection instrument and method based on two-view projection. Background Technology

[0002] Surface mount technology (SMT) is the mainstream process in printed circuit board (PCB) manufacturing today. A complete SMT production line mainly includes solder paste printing, pre-reflow inspection, component placement, reflow soldering, and post-reflow inspection. Among these, the quality of solder paste printing has a significant impact on other stages of the SMT process and the quality of the finished PCB, necessitating defect detection of the printed solder paste. Solder paste height is a crucial indicator in solder paste defect detection. Since 2D inspection methods struggle to obtain height information, 3D solder paste height detection instruments are required.

[0003] Structured light-based 3D reconstruction is a widely used non-contact measurement method in industrial inspection, offering advantages such as high robustness, high precision, and high speed. Currently, existing 3D solder paste height detection instruments primarily employ multi-view projection structured light imaging systems. While this structure effectively overcomes limitations in the reconstruction field of view and reconstruction shadows, its main drawbacks include overly complex structures, high equipment costs, difficulties in imaging system calibration, and time-consuming inspection processes. Furthermore, several issues cannot be ignored with multi-view 3D reconstruction. First, due to calibration and system errors, the height information obtained from each view often differs, making it difficult to fuse the results and affecting the reliability of the reconstruction. Second, the 3D reconstruction results are affected by the imaging system, especially the projection optical engine distortion. To meet the demands of real-time industrial inspection, an effective and efficient distortion correction method is needed. Third, influenced by factors such as substrate material, ambient light conditions, and imaging system hardware, imaging inevitably contains areas with insufficient modulation or overexposure, leading to missing heights or incorrect height calculations in those areas in the results.

[0004] In summary, existing 3D solder paste height detection instruments still suffer from complex structures and long detection times. Furthermore, due to the distortion of the imaging system and the influence of external noise, the reliability of the 3D reconstruction results of solder paste is poor, resulting in low accuracy of height detection. Therefore, it is essential to design a new 3D solder paste height detection instrument and method to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to solve the problems of complex structure, long height detection time, and low height detection accuracy caused by the poor reliability of the 3D solder paste reconstruction results obtained by existing 3D solder paste height detection instruments. Therefore, this invention proposes a 3D solder paste height detection instrument and detection method based on two-view projection.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] According to one aspect of the present invention, a 3D solder paste height detection instrument based on two-view projection is provided, the instrument comprising a light source, a structured light imaging system and an industrial control computer;

[0008] The structured light imaging system includes a central camera and projection optical engines placed on both sides of the central camera, and the lens of the central camera is a telecentric lens.

[0009] The industrial control computer is used to control the central camera and light source to collect color images of 3D solder paste. The industrial control computer is also used to control the projection optical engine to cooperate with the central camera to collect stripe patterns of 3D solder paste.

[0010] According to another aspect of the present invention, a 3D solder paste height detection method based on two-view projection is provided, the method specifically including the following steps:

[0011] Step 1: Use a calibration board to calibrate the structured light imaging system and obtain the intrinsic, extrinsic, and distortion parameters of the central camera, as well as the intrinsic, extrinsic, and distortion parameters of the projection optical engines on both sides.

[0012] Step 2: Based on the intrinsic parameters of the central camera, the intrinsic and extrinsic parameters of the two projection optical engines, and the distortion parameters, construct the distortion-free absolute phase map of the plane at different heights, and recover the distortion-free absolute phase map corresponding to the plane at different heights.

[0013] Using the absolute phase of the plane at different heights and the x and y coordinates of the point in the world coordinate system An absolute phase compensation calculation model was established, and the model parameters of the absolute phase compensation calculation model were optimized based on the distortion-free absolute phase map.

[0014] Step 3: The i-th projection optical engine projects phase-shifted fringes and Gray code coded fringes onto the solder paste to be tested, and uses the central camera to acquire the fringe pattern of the 3D solder paste. Then, the absolute phase under the view of the i-th projection optical engine is recovered by calculating the wrapped phase and unwrapping of the fringe pattern, i=1,2.

[0015] Based on the absolute phase compensation calculation model established in step 2, the absolute phase under each projection optical engine view is distorted to obtain the distortion-free absolute phase under each projection optical engine view.

[0016] Step 4: Select a projection optical engine view as a reference projection optical engine view. Based on the central camera internal parameters, projection optical engine internal parameters, and projection optical engine external parameters obtained in Step 1, transform the distortion-free absolute phase under the other projection optical engine view to the reference projection optical engine view. Perform phase fusion on the distortion-free absolute phase of the two projection optical engines under the reference projection optical engine view to restore the distortion-free absolute phase map of the entire field of view under the reference projection optical engine view.

[0017] Step 5: Under the reference projection optical engine view, use the distortion-free absolute phase map of the entire field of view, the internal parameters of the central camera, the internal parameters of the projection optical engine, and the external parameters of the projection optical engine to recover the solder paste height, and save the solder paste height in the form of an image.

[0018] Step 6: Perform median filtering denoising and completion processing on the solder paste height map saved in Step 5 to obtain the processed solder paste height map, thus completing the 3D solder paste height detection.

[0019] The beneficial effects of this invention are:

[0020] The 3D solder paste height detection instrument provided by this invention has a simple structure, reducing the difficulty of system calibration. Addressing the projection optical engine distortion problem in the imaging system of the detection instrument, a global distortion correction parameter calculation method based on a virtual absolute phase plane is proposed. This method only needs to be calculated once after the imaging system is calibrated, and subsequently, the absolute phase compensation amount can be calculated using 10 distortion correction parameters under each projection optical engine view, achieving absolute phase distortion correction at any height and any pixel position within a certain height range. The model exhibits good generalization and ensures high efficiency in distortion correction. Regarding the absolute phase fusion problem under different views, the method proposed in this invention not only considers the absolute phase under each view but also fully considers the difference in absolute phase between two views, improving the reliability of the phase fusion result between the two views. Addressing the noise and missing height issues in the obtained height image, a GPU-accelerated median filter is proposed. Compared with traditional median filtering, this method can more quickly and effectively correct discrete noise in the height image, is unaffected by invalid values ​​in the image, and better preserves the texture details of the height map. A weighted filtering-based completion method is proposed, which takes into account the similarity of neighborhood heights and adaptively adjusts the filter kernel size according to the similarity of neighborhood heights. This method can achieve good completion results even for large gaps, and preserves the edge details of the height map, thus ensuring the reliability of the solder paste 3D reconstruction results and the accuracy of height detection.

[0021] Furthermore, the absolute phase calculation, absolute phase distortion correction, and height calculation stages fully consider parallelism and utilize GPU acceleration, significantly improving the height detection speed. In addition, this invention is not limited to solder paste height detection; it is also fully applicable to the height detection of other objects within a certain height range, making it widely applicable. Attached Figure Description

[0022] Figure 1 This is a flowchart of a 3D solder paste height detection method based on two-view projection according to the present invention;

[0023] Figure 2 This is a schematic diagram of the calibration plate used in the calibration of this invention;

[0024] Figure 3(a) is a planar detection effect diagram of the projection optical engine before distortion correction according to the present invention;

[0025] Figure 3(b) is a diagram of the planar detection effect after distortion correction by the projection optical engine of the present invention;

[0026] Figure 4(a) is a height diagram before filtering, denoising, and completion processing in the detection process of this invention;

[0027] Figure 4(b) is a height map after filtering, denoising, and completion processing in the detection process of this invention;

[0028] Figure 5(a) is a diagram showing the solder paste detection height of the present invention;

[0029] Figure 5(b) is a three-dimensional point cloud diagram of the solder paste detection height. Detailed Implementation

[0030] Specific Implementation Method 1: The 3D solder paste height detection instrument based on two-view projection described in this implementation method includes a light source, a structured light imaging system, and an industrial control computer;

[0031] The structured light imaging system includes a central camera and projection optical engines placed on both sides of the central camera, and the lens of the central camera is a telecentric lens.

[0032] The industrial control computer is used to control the central camera and light source to collect color images of 3D solder paste. The industrial control computer is also used to control the projection optical engine to cooperate with the central camera to collect stripe patterns of 3D solder paste.

[0033] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the light source has four illumination modes: red, green, blue, and white.

[0034] The other steps and parameters are the same as in Specific Implementation Method 1.

[0035] The specific process of acquiring color images of 3D solder paste in conjunction with the light source and the central camera is as follows:

[0036] First, the brightness of each light source is set. Then, the industrial control computer sends a trigger signal to the light source to light up the red (R) lamp. The trigger signal is delayed by 15ms (to ensure the light source is fully lit). The industrial control computer then triggers the central camera to take a picture. After the image is saved to the buffer, the industrial control computer sends a shutdown signal to the light source. The shutdown signal is delayed by 10ms (to ensure the light source is completely turned off). This is one complete cycle of acquiring the red (R) channel image. Similarly, the green (G) and blue (B) channel images are acquired separately. Finally, the three single-channel images are combined into a three-channel RGB image to obtain the color image of the 3D solder paste. The white light (W) mode is only used for supplementary lighting when necessary. For example, when shooting the red (R) channel image, sometimes a small amount of white light is turned on at the same time as the red light source to ensure the brightness and saturation of the monochromatic light.

[0037] Specific implementation method three: Combining Figure 1 This embodiment describes a 3D solder paste height detection method based on two-view projection, which specifically includes the following steps:

[0038] Step 1: Use a calibration board to calibrate the structured light imaging system and obtain the intrinsic, extrinsic, and distortion parameters of the central camera, as well as the intrinsic, extrinsic, and distortion parameters of the projection optical engines on both sides.

[0039] Step 2: Based on the intrinsic parameters of the central camera, the intrinsic and extrinsic parameters of the two projection optical engines, and the distortion parameters, construct the distortion-free absolute phase map of the plane at different heights, and recover the distortion-free absolute phase map corresponding to the plane at different heights.

[0040] Using the absolute phase of the plane at different heights and the x and y coordinates of the point in the world coordinate system An absolute phase compensation calculation model was established, and the model parameters of the absolute phase compensation calculation model were optimized based on the distortion-free absolute phase map.

[0041] Step 3: The i-th projection optical engine projects phase-shifted fringes and Gray code coded fringes onto the solder paste to be tested, and uses the central camera to acquire the fringe pattern of the 3D solder paste. Then, the absolute phase under the view of the i-th projection optical engine is recovered by calculating the wrapped phase and unwrapping of the fringe pattern, i=1,2.

[0042] Based on the absolute phase compensation calculation model established in step 2, the absolute phase under each projection optical engine view is distorted to obtain the distortion-free absolute phase under each projection optical engine view.

[0043] Step 4: Select a projection optical engine view as a reference projection optical engine view. Based on the central camera internal parameters, projection optical engine internal parameters, and projection optical engine external parameters obtained in Step 1, transform the distortion-free absolute phase under the other projection optical engine view to the reference projection optical engine view. Perform phase fusion on the distortion-free absolute phase of the two projection optical engines under the reference projection optical engine view to restore the distortion-free absolute phase map of the entire field of view under the reference projection optical engine view.

[0044] Step 5: Under the reference projection optical engine view, use the distortion-free absolute phase map of the entire field of view, the internal parameters of the central camera, the internal parameters of the projection optical engine, and the external parameters of the projection optical engine to recover the solder paste height, and save the solder paste height in the form of an image.

[0045] Step 6: Perform median filtering denoising and completion processing on the solder paste height map saved in Step 5 to obtain the processed solder paste height map, thus completing the 3D solder paste height detection.

[0046] Specific Implementation Method Four: This implementation method differs from Specific Implementation Method Three in that it adopts the following... Figure 2 The calibration plate shown is used to calibrate the structured light imaging system. The calibration plate has several equally spaced marker circles. The specific process of step 1 is as follows:

[0047] Step 11: Under the current orientation of the calibration board, use the first projection optical engine to project vertical stripes onto the calibration board and trigger the central camera to acquire an image of the calibration board under its current orientation.

[0048] The first projection optical engine projects horizontal stripes onto the calibration board and triggers the central camera to capture an image of the calibration board in its current orientation.

[0049] Similarly, the second projection optical engine projects vertical and horizontal stripes onto the calibration plate, and acquires an image of the calibration plate in its current orientation.

[0050] All images acquired in the current pose are used as a set of images;

[0051] Step 12: Continuously change the orientation of the calibration board. Perform the process of step 11 under each orientation of the calibration board to obtain several sets of images (the specific number of sets can be set according to the actual situation; 15 sets are used in this invention).

[0052] Step 13: Use all the acquired images to calibrate the central camera and the projection engines on both sides respectively, and obtain the intrinsic parameters, extrinsic parameters and distortion parameters of the central camera, as well as the intrinsic parameters, extrinsic parameters and distortion parameters of the projection engines on both sides.

[0053] Then, by minimizing the reprojection error, the intrinsic and extrinsic parameters of the central camera and the two side projection engines are globally optimized to obtain the globally optimized intrinsic and extrinsic parameters of the central camera and the two side projection engines.

[0054] The other steps and parameters are the same as in Specific Implementation Method 3.

[0055] In this embodiment, the internal and external parameters of the globally optimized central camera and the two side projection optical engines are used as the final internal and external parameters and are then used in subsequent processing.

[0056] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods Three or Four in that the specific process of step 2 is as follows:

[0057] Step 21: Select the height interval [d, d'], specify the height change step size Δd, and then construct a series of equally spaced height planes. Record the height of the j-th height plane as d. j For example, the height of the first height plane is d, and the height of the second height plane is d + Δd.

[0058] Step 22: Reconstruct the distorted absolute phase map corresponding to each height plane, specifically as follows:

[0059] Set the world coordinate system at the extension of the central camera coordinate system along the Z-axis, that is, set the origin of the world coordinate system on the positive half of the Z-axis of the central camera coordinate system, and ensure that the three axes of the world coordinate system point in the same direction as the three axes of the central camera coordinate system. At this point, the x and y coordinates of the 3D world coordinate system... Normalized planar coordinates of the center camera Similarly, the three-dimensional coordinates of a point on the j-th height plane in the world coordinate system are calculated based on the intrinsic parameters of the central camera. , , )for:

[0060]

[0061]

[0062]

[0063] in, The pixel coordinates representing the imaging plane of the central camera. The coordinates representing the center of distortion of the central camera lens. and These represent the magnification along the x-axis and y-axis of the world coordinate system, respectively. It is the angle between the horizontal and vertical sides of the imaging element of the central camera, ideally... , Is with Correlation coefficient Proportional to ,Right now = ;

[0064] Based on the external parameters of the two projection engines, calculate the three-dimensional coordinates of a point on the j-th height plane and its normalized coordinates on the i-th projection engine normalized plane. :

[0065]

[0066]

[0067] in, Let represent the element in the m-th row and n-th column of the i-th projection optical engine extrinsic parameter rotation matrix, where i=1,2, m=1,2,3, n=1,2,3. This represents the translation of the i-th projection optical engine extrinsic parameter along the x-axis of the world coordinate system. This represents the translation of the i-th projection optical engine extrinsic parameter along the y-axis of the world coordinate system. This represents the translation of the i-th projection optical engine extrinsic parameter along the z-axis of the world coordinate system. The normalized coordinates obtained at this time are distortion-free coordinates.

[0068] Normalized coordinates are obtained by using the distortion parameters of the i-th projection engine. By adding radial and tangential distortion, we obtain the normalized coordinates on the i-th projection optical engine normalized plane after adding distortion. :

[0069]

[0070] in, and Represents the radial distortion coefficient, m''=1,2. and Represents the tangential distortion coefficient;

[0071]

[0072] Calculate the normalized coordinates considering the tilt of the projection optical engine imaging plane. :

[0073]

[0074] in, This represents the rotation angle of the projection optical engine imaging plane along the x-axis of the tilt-free optical engine coordinate system. The angle representing the rotation of the projection optical engine imaging plane along the y-axis of the tilt-free optical engine coordinate system is s, where s is the scaling factor.

[0075] Based on the intrinsic parameters and normalized coordinates of the i-th projection engine Restore the absolute phase map of the j-th height plane band distortion:

[0076]

[0077] in, Let T represent the absolute phase map of the j-th height plane band distortion recovered under the i-th projected optical engine view, where T represents the fringe period of the projected optical engine projection. This represents the effective focal length of the i-th projection engine. This represents the x-coordinate of the i-th projection optical engine distortion center, thus achieving a height of d. j Absolute phase recovery of high-plane band distortion;

[0078] Step 23: Recover the distortion-free absolute phase map corresponding to the j-th height plane, specifically as follows:

[0079]

[0080] in, This is the distortion-free absolute phase map of the j-th height plane under the i-th projected optical engine view. For coefficients, , , , , , , ;

[0081] Step 24: Use the absolute phase map of the distortion zone in the j-th height plane and the x and y coordinates of the point in the world coordinate system. The calculation model for the absolute phase compensation is established as follows:

[0082] Under the i-th projected optical engine view, the calculation model for the absolute phase compensation amount of the j-th height plane is as follows:

[0083]

[0084] in, This represents the parameters to be optimized. It is the absolute phase compensation amount of the j-th height plane under the i-th projection optical engine view.

[0085] The other steps and parameters are the same as in specific implementation methods three or four.

[0086] Similarly, the method in steps 22 to 24 of this embodiment can be used to obtain the calculation model of the absolute phase compensation amount for each height plane.

[0087] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods Three to Five in that the parameters to be optimized are optimized using the following optimization model:

[0088]

[0089] Where D represents the number of height planes selected, and N represents the number of sampling points for each height plane. and Representing the i-th projection optical engine view respectively The absolute phase with distortion and the absolute phase without distortion at the n'th sampling point on the height plane. It is the i-th projection optical engine view. The absolute phase compensation amount at the n'th sampling point on the height plane.

[0090] The other steps and parameters are the same as those in one of the specific implementation methods three to five.

[0091] In this embodiment, each sampling point is a pixel. Ten parameters for global absolute phase distortion correction are obtained through optimization under the i-th projected optical-mechanical view. To visually demonstrate the distortion correction effect of this invention, a planar reconstruction was performed using the system platform and system calibration parameters of this invention, and the side views of the reconstructed plane were observed, as shown in Figures 3(a) and 3(b). Figure 3(a) represents the side view of the reconstructed plane without distortion correction, and Figure 3(b) represents the side view of the reconstructed plane after distortion correction.

[0092] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods Three to Six in that the projected phase-shifted stripes and Gray code encoded stripes adopt a projection mode of 4-step phase shift + 7-bit Gray code + 1 complement code, and only vertical stripes are projected.

[0093] The other steps and parameters are the same as those in one of the specific implementation methods three to six.

[0094] The process of step 3 of this invention is accelerated by GPU.

[0095] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods Three to Seven in that the specific process of step 4 is as follows:

[0096] Step 41: Select any one of the two projection optical engine views as the reference view, and transform the distortion-free absolute phase under the other projection optical engine view to the reference projection optical engine view:

[0097]

[0098] Where ref represents the reference projection optical engine view, and f represents the non-reference projection optical engine view (i.e., another projection optical engine view). Represents the absolute phase after the non-reference projection optical engine view is transformed into the reference projection optical engine view (ref). This represents the distortion-free absolute phase in the non-reference projected optical-mechanical view. , and All are coefficients, where i takes values ​​of f and ref;

[0099]

[0100]

[0101]

[0102] in, The calculation method is the same as the coefficient calculation in step 23;

[0103] The two absolute phase maps, converted to the reference projection optical engine view, are respectively used... and This indicates that the absolute phase diagram of the reference projected optical engine view utilizes... and Any representative is acceptable;

[0104] Step 42: Fuse the two absolute phase maps transformed to the reference projection optical-mechanical view to obtain the fused absolute phase map under the reference projection optical-mechanical view; specifically:

[0105] Step 421: For any pixel position in the absolute phase map , pixel position In absolute phase diagram The corresponding absolute phase is denoted as , pixel position In absolute phase diagram Absolute phase is denoted as ;

[0106] Step 422, if and If all values ​​are invalid, proceed to step 423; if and If a valid value exists, proceed to step 424; if and If all values ​​are valid, proceed to step 425.

[0107] Step 423, Pixel Position The fusion phase is also an invalid value;

[0108] Step 424: Use the absolute phase diagram containing the effective values... This indicates that the absolute phase diagram containing invalid values ​​is represented by... Indicated by pixel position Construct a 5×5 neighborhood around the center. For any pixel within the neighborhood... :

[0109] like and All are effective and , As the absolute phase difference threshold, and Store in collection and for pixels Assign weights :

[0110]

[0111] in, This represents the position of pixel p. This represents the position of pixel q. Represents the variance of spatial location. Represents phase variance. It is the base of the natural logarithm. Pixel position In absolute phase diagram The corresponding absolute phase, The pixel position q in the absolute phase map The corresponding absolute phase, The pixel position q in the absolute phase map The corresponding absolute phase;

[0112] like and All are effective and ,but and No need to store in a collection ;

[0113] like Effective, and If invalid, then Store in collection and for pixels Assign weights :

[0114]

[0115] like Invalid and If it is effective, then and No need to store in a collection ;

[0116] like and If all are invalid, then it is not necessary to... and Store in collection ;

[0117] Traverse by pixel position After considering all pixels within a 5×5 neighborhood centered on the set The absolute phase value within the cell is used to calculate the pixel position using the assigned weights. Fusion phase :

[0118]

[0119] in, It is a set Elements within;

[0120] Step 425, Calculation and absolute phase difference ;

[0121] like Then the pixel position The fusion phase is:

[0122]

[0123] in, The modulation degree represents the pixel position p under the reference projected optical engine view. The modulation degree representing the pixel position p under another projected optical engine view;

[0124] like Construct a 5×5 neighborhood centered at pixel position p. For any pixel q within the neighborhood:

[0125] If pixel q satisfies the condition: and All are effective and ,Will and Store in collection And assign weights to pixel q. :

[0126]

[0127] If pixel q does not meet the condition: and All are effective and Then there is no need to and Store in collection ;

[0128] After traversing all pixels within a 5×5 neighborhood centered at pixel position p, based on the set The absolute phase value within the range is used to calculate the fused phase using the corresponding weights assigned. :

[0129]

[0130] Step 426: Similarly, after traversing each pixel position in the absolute phase map (i.e., performing steps 421 to 425 for each pixel position), the absolute phase fusion result under the reference projection optical engine view is completed, that is, the distortion-free absolute phase map of the entire field of view under the reference projection optical engine view is restored.

[0131] The other steps and parameters are the same as those in one of the specific implementation methods three to seven.

[0132] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods Three to Eight in that, in step 5, the solder paste height is recovered using a distortion-free absolute phase map of the entire field of view, the intrinsic parameters of the central camera, the intrinsic parameters of the projection optical engine, and the extrinsic parameters of the projection optical engine, under the reference projection optical engine view; the specific process is as follows:

[0133] Restore the three-dimensional coordinates of each point on the solder paste to be tested. (That is, representing height):

[0134]

[0135]

[0136]

[0137] in, Represents the pixel coordinates of the central camera's imaging plane. The coordinates representing the center of distortion of the central camera lens. The coefficient is calculated in the same way as the coefficient calculation in step 23 (specifically, the calculation is performed using calibration parameters, and the coefficient of the reference projection optical engine is used here). It is a distortion-free absolute phase map of the entire field of view.

[0138] The other steps and parameters are the same as those in specific implementation methods three to eight.

[0139] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods Three to Nine in that the specific process of step 6 is as follows:

[0140] Step 61: Perform median filtering on the height values ​​at each pixel location in the solder paste height map; specifically:

[0141] For any pixel position in the solder paste height map, if the height value at that pixel position is a valid value, then all valid height values ​​in the 3×3 neighborhood of that pixel position are sorted, and the median of the sorted sequence is selected as the updated height value at that pixel position; if the height value at that pixel position is an invalid value, then no processing is required.

[0142] Similarly, each pixel position in the solder paste height map is processed separately;

[0143] To ensure efficiency, the filter kernel is 3×3 in size, and each pixel in the height image is processed in parallel by a separate thread. If a pixel is invalid (missing height), it will not participate in the calculation and will exit the thread. The median filtering method can quickly and effectively correct discrete noise in the height image.

[0144] Step 62: For any pixel position in the median-filtered height image If pixel position and pixel position If there are invalid height values ​​among the top, bottom, left, and right pixels, then the pixel position... High similarity coefficient Invalid value;

[0145] If pixel position and pixel position If the height values ​​of the pixels at the top, bottom, left, and right are all valid values, then by using... and located at pixel position The height similarity coefficient is calculated from the height values ​​of the top, bottom, left, and right pixels. :

[0146]

[0147] in, Indicates pixel position The height value, Representing pixel positions The height values ​​of the top, bottom, left, and right adjacent pixels. Representing pixel positions The positions of the top, bottom, left, and right adjacent pixels;

[0148] The filter kernel size will be determined based on the high similarity coefficient. With threshold The ratio is adaptively adjusted based on the high similarity coefficient. Calculate the size ksize of the filter kernel:

[0149]

[0150] in, This indicates rounding down, % represents the modulo operation, and L represents the base size of the filter kernel, which must be an odd number. For the set threshold, Represents the minimum value operation; NAN represents an invalid value.

[0151] like This indicates the pixel position. If the neighborhood has good similarity, then the pixel position is used. Pixels within the filter kernel centered on The weights are:

[0152]

[0153] in, Representing pixel positions and pixel position Location, Represents the variance of spatial location;

[0154] like or An invalid value indicates the pixel position. Low similarity at certain heights, or pixel positions The heights of its upper, lower, left, and right adjacent pixels contain invalid values. Then by pixel position Pixels within the filter kernel centered on The weights are:

[0155]

[0156] in, The gray level of pixel position p in a color image representing 3D solder paste. The gray level of pixel position q in a color image representing 3D solder paste. Represents the luminance variance; the color image of the 3D solder paste is acquired through a central camera and a light source.

[0157] Step 63: The height value of pixel position p after completion is:

[0158]

[0159] in, Represented by pixel position The set of pixels with valid height values ​​within the center of the filter kernel. Representative set Middle pixel position The height value, This represents the height value of the pixel position p after completion.

[0160] The other steps and parameters are the same as those in one of the specific implementation methods three to nine.

[0161] The completion method of the present invention is performed in the order of pixel position from left to right and from top to bottom. The height map obtained in step 5 is shown in Figure 4(a), and the height map after filtering and completion is shown in Figure 4(b). This shows that filtering, denoising and completion processing of the solder paste height map saved in step 5 can further improve the three-dimensional reconstruction effect.

[0162] Finally, the solder paste height detection results are shown in Figure 5(a), which is a solder paste height map, and Figure 5(b), which is a three-dimensional point cloud map corresponding to Figure 5(a).

[0163] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A 3D solder paste height detection instrument based on two-view projection, characterized in that, The 3D solder paste height detection instrument includes a light source, a structured light imaging system, and an industrial control computer; The structured light imaging system includes a central camera and projection optical engines placed on both sides of the central camera, and the lens of the central camera is a telecentric lens. The industrial control computer is used to control the central camera and light source to collect color images of 3D solder paste. The industrial control computer is also used to control the projection optical engine to cooperate with the central camera to collect stripe patterns of 3D solder paste.

2. The 3D solder paste height detection instrument based on two-view projection according to claim 1, characterized in that, The light source has four illumination modes: red, green, blue, and white.

3. A 3D solder paste height detection method based on two-view projection, characterized in that, The method specifically includes the following steps: Step 1: Use a calibration board to calibrate the structured light imaging system and obtain the intrinsic, extrinsic, and distortion parameters of the central camera, as well as the intrinsic, extrinsic, and distortion parameters of the projection optical engines on both sides. Step 2: Based on the intrinsic parameters of the central camera, the intrinsic and extrinsic parameters of the two projection optical engines, and the distortion parameters, construct the distortion-free absolute phase map of the plane at different heights, and recover the distortion-free absolute phase map corresponding to the plane at different heights. Using the absolute phase of the plane at different heights and the x and y coordinates of the point in the world coordinate system An absolute phase compensation calculation model was established, and the model parameters of the absolute phase compensation calculation model were optimized based on the distortion-free absolute phase map. Step 3: The i-th projection optical engine projects phase-shifted fringes and Gray code coded fringes onto the solder paste to be tested, and uses the central camera to acquire the fringe pattern of the 3D solder paste. Then, the absolute phase under the view of the i-th projection optical engine is recovered by calculating the wrapped phase and unwrapping of the fringe pattern, i=1,2. Based on the absolute phase compensation calculation model established in step 2, the absolute phase under each projection optical engine view is distorted to obtain the distortion-free absolute phase under each projection optical engine view. Step 4: Select a projection optical engine view as a reference projection optical engine view. Based on the central camera internal parameters, projection optical engine internal parameters, and projection optical engine external parameters obtained in Step 1, transform the distortion-free absolute phase under the other projection optical engine view to the reference projection optical engine view. Perform phase fusion on the distortion-free absolute phase of the two projection optical engines under the reference projection optical engine view to restore the distortion-free absolute phase map of the entire field of view under the reference projection optical engine view. Step 5: Under the reference projection optical engine view, use the distortion-free absolute phase map of the entire field of view, the internal parameters of the central camera, the internal parameters of the projection optical engine, and the external parameters of the projection optical engine to recover the solder paste height, and save the solder paste height in the form of an image. Step 6: Perform median filtering denoising and completion processing on the solder paste height map saved in Step 5 to obtain the processed solder paste height map, thus completing the 3D solder paste height detection.

4. The 3D solder paste height detection method based on two-view projection according to claim 3, characterized in that, The specific process of step 1 is as follows: Step 11: Under the current orientation of the calibration board, use the first projection optical engine to project vertical stripes onto the calibration board and trigger the central camera to acquire an image of the calibration board under its current orientation. The first projection optical engine projects horizontal stripes onto the calibration board and triggers the central camera to capture an image of the calibration board in its current orientation. Similarly, the second projection optical engine projects vertical and horizontal stripes onto the calibration plate, and acquires an image of the calibration plate in its current orientation. All images acquired in the current pose are used as a set of images; Step 12: Continuously change the orientation of the calibration board. Perform the process of step 11 under each orientation of the calibration board to obtain several sets of images. Step 13: Use all the acquired images to calibrate the central camera and the projection engines on both sides respectively, and obtain the intrinsic parameters, extrinsic parameters and distortion parameters of the central camera, as well as the intrinsic parameters, extrinsic parameters and distortion parameters of the projection engines on both sides. Then, the intrinsic and extrinsic parameters of the central camera and the two side projection optical engines are globally optimized to obtain the globally optimized intrinsic and extrinsic parameters of the central camera and the two side projection optical engines.

5. The 3D solder paste height detection method based on two-view projection according to claim 3, characterized in that, The specific process of step 2 is as follows: Step 21: Select the height interval [d, d'], specify the height change step size Δd, and then construct a series of equally spaced height planes. Record the height of the j-th height plane as d. j ; Step 22: Reconstruct the distorted absolute phase map corresponding to each height plane, specifically as follows: Calculate the three-dimensional coordinates of a point on the j-th elevation plane in the world coordinate system based on the intrinsic parameters of the central camera. , , )for: in, The pixel coordinates representing the imaging plane of the central camera. The coordinates representing the center of distortion of the central camera lens. and These represent the magnification along the x-axis and y-axis of the world coordinate system, respectively. It is the angle between the horizontal and vertical sides of the imaging element of the central camera. Is with Correlation coefficient; Based on the external parameters of the two projection engines, calculate the three-dimensional coordinates of a point on the j-th height plane and its normalized coordinates on the i-th projection engine normalized plane. : in, Let represent the element in the m-th row and n-th column of the i-th projection optical engine extrinsic parameter rotation matrix, where i=1,2, m=1,2,3, n=1,2,3. This represents the translation of the i-th projection optical engine extrinsic parameter along the x-axis of the world coordinate system. This represents the translation of the i-th projection optical engine extrinsic parameter along the y-axis of the world coordinate system. This represents the translation of the i-th projection optical engine extrinsic parameter along the z-axis of the world coordinate system; Normalized coordinates are obtained by using the distortion parameters of the i-th projection engine. By adding radial and tangential distortion, we obtain the normalized coordinates on the i-th projection optical engine normalized plane after adding distortion. : in, and Represents the radial distortion coefficient, m''=1,2. and Represents the tangential distortion coefficient; Calculate the normalized coordinates considering the tilt of the projection optical engine imaging plane. : in, This represents the rotation angle of the projection optical engine imaging plane along the x-axis of the tilt-free optical engine coordinate system. The angle representing the rotation of the projection optical engine imaging plane along the y-axis of the tilt-free optical engine coordinate system is s, where s is the scaling factor. Based on the intrinsic parameters and normalized coordinates of the i-th projection engine Restore the absolute phase map of the j-th height plane band distortion: in, Let T represent the absolute phase map of the j-th height plane band distortion recovered under the i-th projected optical engine view, where T represents the fringe period of the projected optical engine projection. This represents the effective focal length of the i-th projection engine. The x-coordinate represents the i-th projection optical engine distortion center; Step 23: Recover the distortion-free absolute phase map corresponding to the j-th height plane, specifically as follows: in, This is the distortion-free absolute phase map of the j-th height plane under the i-th projected optical engine view. For coefficients, , , , , , , ; Step 24: Use the absolute phase map of the distortion zone in the j-th height plane and the x and y coordinates of the point in the world coordinate system. The calculation model for the absolute phase compensation is established as follows: Under the i-th projected optical engine view, the calculation model for the absolute phase compensation amount of the j-th height plane is as follows: in, This represents the parameters to be optimized. It is the absolute phase compensation amount of the j-th height plane under the i-th projection optical engine view.

6. The 3D solder paste height detection method based on two-view projection according to claim 5, characterized in that, The parameters to be optimized are optimized using the following optimization model: Where D represents the number of height planes selected, and N represents the number of sampling points for each height plane. and Representing the i-th projection optical engine view respectively The absolute phase with distortion and the absolute phase without distortion at the n'th sampling point on the height plane. It is the i-th projection optical engine view. The absolute phase compensation amount at the n'th sampling point on the height plane.

7. The 3D solder paste height detection method based on two-view projection according to claim 6, characterized in that, The projected phase-shifted stripes and Gray code encoded stripes adopt a projection mode of 4-step phase shift + 7-bit Gray code + 1 complement code, and only project vertical stripes.

8. The 3D solder paste height detection method based on two-view projection according to claim 7, characterized in that, The specific process of step 4 is as follows: Step 41: Select any one of the two projection optical engine views as the reference view, and transform the distortion-free absolute phase under the other projection optical engine view to the reference projection optical engine view: Where ref represents the reference projection optical-mechanical view, and f represents the non-reference projection optical-mechanical view. Represents the absolute phase after the non-reference projection optical engine view is transformed into the reference projection optical engine view (ref). This represents the distortion-free absolute phase in the non-reference projected optical-mechanical view. , and All are coefficients, where i takes values ​​of f and ref; The two absolute phase maps, converted to the reference projection optical engine view, are respectively used... and express; Step 42: Fuse the two absolute phase maps transformed to the reference projection optical-mechanical view to obtain the fused absolute phase map under the reference projection optical-mechanical view; specifically: Step 421: For any pixel position in the absolute phase map , pixel position In absolute phase diagram The corresponding absolute phase is denoted as , pixel position In absolute phase diagram Absolute phase is denoted as ; Step 422, if and If all values ​​are invalid, proceed to step 423; if and If a valid value exists, proceed to step 424; if and If all values ​​are valid, proceed to step 425. Step 423, Pixel Position The fusion phase is also an invalid value; Step 424: Use the absolute phase diagram containing the effective values... This indicates that the absolute phase diagram containing invalid values ​​is represented by... Indicated by pixel position Construct a 5×5 neighborhood around the center. For any pixel within the neighborhood... : like and All are effective and , As the absolute phase difference threshold, and Store in collection and for pixels Assign weights : in, This represents the position of pixel p. This represents the position of pixel q. Represents the variance of spatial location. Represents phase variance. It is the base of the natural logarithm. Pixel position In absolute phase diagram The corresponding absolute phase, The pixel position q in the absolute phase map The corresponding absolute phase, The pixel position q in the absolute phase map The corresponding absolute phase; like and All are effective and ,but and No need to store in a collection ; like Effective, and If invalid, then Store in collection and for pixels Assign weights : like Invalid and If it is effective, then and No need to store in a collection ; like and If all are invalid, then it is not necessary to... and Store in collection ; Traverse by pixel position After considering all pixels within a 5×5 neighborhood centered on the set The absolute phase value within the cell is used to calculate the pixel position using the assigned weights. Fusion phase : in, It is a set Elements within; Step 425, Calculation and absolute phase difference ; like Then the pixel position The fusion phase is: in, The modulation degree represents the pixel position p under the reference projected optical engine view. The modulation degree representing the pixel position p under another projected optical engine view; like Construct a 5×5 neighborhood centered at pixel position p. For any pixel q within the neighborhood: If pixel q satisfies the condition: and All are effective and ,Will and Store in collection And assign weights to pixel q. : If pixel q does not meet the condition: and All are effective and Then there is no need to and Store in collection ; After traversing all pixels within a 5×5 neighborhood centered at pixel position p, based on the set The absolute phase value within the range is used to calculate the fused phase using the corresponding weights assigned. : Step 426: Similarly, after traversing every pixel position in the absolute phase map, the absolute phase fusion result under the reference projection optical engine view is completed, that is, the distortion-free absolute phase map of the entire field of view under the reference projection optical engine view is restored.

9. The 3D solder paste height detection method based on two-view projection according to claim 8, characterized in that, In step 5, under the reference projection optical engine view, the solder paste height is recovered using the distortion-free absolute phase map of the entire field of view, the intrinsic parameters of the central camera, the intrinsic parameters of the projection optical engine, and the extrinsic parameters of the projection optical engine; the specific process is as follows: Restore the three-dimensional coordinates of each point on the solder paste to be tested: in, Represents the pixel coordinates of the central camera's imaging plane. The coordinates representing the center of distortion of the central camera lens. For coefficients, It is a distortion-free absolute phase map of the entire field of view.

10. A 3D solder paste height detection method based on two-view projection according to claim 9, characterized in that, The specific process of step 6 is as follows: Step 61: Perform median filtering on the height values ​​at each pixel location in the solder paste height map; specifically: For any pixel position in the solder paste height map, if the height value at that pixel position is a valid value, then sort all valid height values ​​in the 3×3 neighborhood of that pixel position, and then select the median of the sorted sequence as the updated height value at that pixel position. If the height value at the pixel location is invalid, no processing is required; Similarly, each pixel position in the solder paste height map is processed separately; Step 62: For any pixel position in the median-filtered height image If pixel position and pixel position If there are invalid height values ​​among the top, bottom, left, and right pixels, then the pixel position... High similarity coefficient Invalid value; If pixel position and pixel position If the height values ​​of the pixels at the top, bottom, left, and right are all valid values, then by using... and located at pixel position The height similarity coefficient is calculated from the height values ​​of the top, bottom, left, and right pixels. : in, Indicates pixel position The height value, Representing pixel positions The height values ​​of the top, bottom, left, and right adjacent pixels. Representing pixel positions The positions of the top, bottom, left, and right adjacent pixels; Based on the high similarity coefficient Calculate the size ksize of the filter kernel: in, This indicates rounding down, % represents the modulo operation, and L represents the reference size of the filter kernel. For the set threshold, Represents the minimum value operation; NAN represents an invalid value. like Then by pixel position Pixels within the filter kernel centered on The weights are: in, Representing pixel positions and pixel position Location, Represents the variance of spatial location; like or If the value is invalid, then use the pixel position. Pixels within the filter kernel centered on The weights are: in, The gray level of pixel position p in a color image representing 3D solder paste. The gray level of pixel position q in a color image representing 3D solder paste. Represents the variance in brightness; Step 63: The height value of pixel position p after completion is: in, Represented by pixel position The set of pixels with valid height values ​​within the center of the filter kernel. Representative set Middle pixel position The height value, This represents the height value of the pixel position p after completion.

Images