Methods, apparatus, equipment and storage media for height detection of electronic product components

Abstract

This invention proposes a method, apparatus, device, and storage medium for detecting the height of electronic product components. The method includes: selecting standard samples and acquiring their three-dimensional point cloud data and two-dimensional images using a 3D camera system as standard point clouds and standard images; marking the overall region of interest (ROI) of the product and the specific ROI of each component on the standard images; extracting the corresponding point cloud from the standard point cloud to calculate the height of each component; acquiring at least one set of different standard samples, obtaining their three-dimensional point cloud data and two-dimensional images, aligning them to the standard point cloud and standard images respectively, and performing non-rigid transformation correction; calculating the height of each component again; determining the average height and standard deviation; selecting samples to be tested, acquiring their three-dimensional point cloud data and two-dimensional images, aligning them to the standard point cloud and standard images respectively, and performing non-rigid transformation correction; calculating the current height of each component; and determining whether it is a defective component based on preset conditions.

Description

Technical Field

[0001] This invention relates to the field of defect detection technology for product components, and in particular to a method, apparatus, equipment, and storage medium for height detection of electronic product components. Background Technology

[0002] In the manufacturing process of electronic products, the installation accuracy of components has a direct impact on the quality and performance of the final device. To ensure product quality, manufacturers typically need to implement rigorous testing procedures to verify that these components are installed correctly.

[0003] Currently, the commonly used inspection methods in the industry mainly include manual visual inspection and automated optical inspection (AOI) based on 2D image processing technology. These two methods can identify obvious defects to a certain extent, such as missing parts or significant positional deviations, but they have limitations when dealing with more complex and subtle problems (such as minute differences in height). This is because the human eye tends to overlook subtle differences, and 2DAOI technology cannot provide sufficient depth information to distinguish these subtle differences.

[0004] While 3DAOI technology, utilizing laser scanning or structured light principles, can simultaneously acquire two-dimensional planar images and three-dimensional spatial information of the object under test, theoretically enabling more accurate assessment of component position accuracy, it suffers from several drawbacks. Firstly, the limited field of view of 3DAOI technology makes it difficult to cover the comprehensive inspection needs of larger products, limiting its effectiveness to localized areas and impacting inspection efficiency. Summary of the Invention

[0005] This invention provides a method, apparatus, equipment, and storage medium for detecting the height of electronic product components, solving the problem of detecting height defects in components of large electronic devices in a continuous production environment on a dynamic conveyor belt.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, a method for detecting the height of electronic product components is provided, including:

[0008] A defect-free electronic product is selected as a standard sample, and three-dimensional point cloud data and two-dimensional images of the standard sample are acquired based on a pre-configured 3D camera system; wherein each pixel of the acquired three-dimensional point cloud data and the two-dimensional image corresponds one-to-one.

[0009] Configure the three-dimensional point cloud data of the standard sample as a standard point cloud; configure the two-dimensional image of the standard sample as a standard image; mark the overall region of interest of the electronic product and the specific region of interest of each component on the standard image;

[0010] For each specific region of interest of the component, the corresponding point cloud is extracted from the standard point cloud to form the component point cloud;

[0011] Calculate the height of each component based on the point cloud of the components;

[0012] At least one set of different standard samples are collected, and their three-dimensional point cloud data and two-dimensional images are acquired based on the 3D camera system. Each set of the acquired three-dimensional point cloud data and two-dimensional images are aligned to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and further correction is performed by non-rigid transformation.

[0013] Based on the specific regions of interest already marked on the standard image, mark the specific regions of interest for each component on each set of acquired two-dimensional images, and extract the corresponding point cloud from the point cloud after the non-rigid transformation, and calculate the height of each component based on the extracted point cloud.

[0014] Calculate the mean height and standard deviation for the multiple sets of height data obtained from the calculation;

[0015] Select a sample to be tested, and acquire its three-dimensional point cloud data and two-dimensional image based on the 3D camera system. These are denoted as the point cloud to be tested and the image to be tested. The acquired point cloud to be tested and the image to be tested are aligned to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and further corrected by non-rigid transformation.

[0016] Based on the specific regions of interest (ROIs) already marked on the standard image, mark the ROIs for each component on the acquired test image. Extract the corresponding point cloud from the point cloud after the non-rigid transformation, and calculate the current height of each component based on the extracted point cloud. If the following conditions are met:

[0017]

[0018] If so, the component is a defective component; otherwise, it is a normal component. At the current altitude, The average height is σ, where t is a threshold parameter. i Let be the standard deviation.

[0019] In a first possible implementation of the first aspect, the 3D camera system includes a high-resolution camera and two tilt-shift optical engines used in conjunction with it.

[0020] In the second possible implementation of the first aspect, the standard point cloud is denoted as (X). g ,Y g Z g The standard image is P. g ;

[0021] The overall region of interest R0 of the electronic product and the specific region of interest R of each component i i = 1, 2, ..., N;

[0022] The component point cloud is

[0023] The calculation of the height of each component based on the point cloud of the components specifically includes:

[0024] Step a, perform point cloud analysis on the components. Perform downsampling, retaining M data points, to obtain

[0025] Step b, process the downsampled component point cloud The least squares method is used to fit the data using the three-dimensional planar formula AX + BY + CZ + D = 0, resulting in a set of parameters A, B, C, and D; the residuals for all data points are then calculated. Sort all residuals and calculate the first quartile Q1, the third quartile Q3, and the interquartile range IQR = Q3 - Q1; mark data points with residuals outside the range [0, Q3 + α × IDR] as outliers and remove them, where α is a threshold parameter;

[0026] Step c: Repeat step b above until there are no outlier data points. Record the plane fitting parameters at this point as A', B', C', D'. Calculate the height of the component using the following formula:

[0027]

[0028] in, After removing outlier data points The average value.

[0029] Based on the second possible implementation of the first aspect, in the third possible implementation of the first aspect, three-dimensional point cloud data and two-dimensional images of K sets of defect-free electronic product samples are collected, and the collected three-dimensional point cloud data is denoted as the test point cloud (X). k ,Y k Z k Let k = 1, 2, ..., K, and let the acquired two-dimensional image be the test image P. k ;

[0030] The process involves acquiring at least one set of different standard samples, obtaining their 3D point cloud data and 2D images based on the 3D camera system, aligning each set of the acquired 3D point cloud data and 2D images to the standard point cloud and the standard image respectively through a 2D affine transformation, and performing further correction through a non-rigid transformation. Specifically, this includes:

[0031] For each set of test images P acquired k Align it with the standard image P by performing a two-dimensional affine transformation. g Let T be the two-dimensional affine transformation matrix.

[0032] Based on the two-dimensional affine transformation matrix T, for each acquired test point cloud (X) k ,Y k Z k ), k=1,2,…,K, perform the same two-dimensional affine transformation on it to align the standard point cloud (X) g ,Y g Z g ), denoted as (X k1 ,Y k1 Z k1 ), k = 1, 2, ..., K;

[0033] The residual formula is defined as follows:

[0034] r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy 2 +c7x 2 +c8x 2 y+c9x 3

[0035] Define the optimization objective function as follows:

[0036]

[0037] Where j = 1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; the coefficients c of the residual formula r(x,y) are obtained by optimizing the objective function. i i = 0, 1, ..., 9;

[0038] For the aligned test point cloud (X) k1 ,Y k1 Z k1 Non-rigid transformations are performed on k = 1, 2, ..., K:

[0039] X k2 =X k1

[0040] Y k2 =Y k1

[0041]

[0042] The transformed coordinates are (X k2 ,Y k2 Z k2 ), k = 1, 2, ..., K.

[0043] Secondly, a height detection device for electronic product components is provided, comprising:

[0044] The first standard sample module is used to select defect-free electronic products as standard samples and acquire three-dimensional point cloud data and two-dimensional images of the standard samples based on a pre-configured 3D camera system; wherein, each pixel of the acquired three-dimensional point cloud data and the two-dimensional image corresponds one-to-one.

[0045] Configure the three-dimensional point cloud data of the standard sample as a standard point cloud; configure the two-dimensional image of the standard sample as a standard image; mark the overall region of interest of the electronic product and the specific region of interest of each component on the standard image;

[0046] For each specific region of interest of the component, the corresponding point cloud is extracted from the standard point cloud to form the component point cloud;

[0047] Calculate the height of each component based on the point cloud of the components;

[0048] The second standard sample module is used to acquire at least one different set of standard samples, obtain their three-dimensional point cloud data and two-dimensional images based on the 3D camera system, align each set of the acquired three-dimensional point cloud data and two-dimensional images to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction.

[0049] Based on the specific regions of interest already marked on the standard image, mark the specific regions of interest for each component on each set of acquired two-dimensional images, and extract the corresponding point cloud from the point cloud after the non-rigid transformation, and calculate the height of each component based on the extracted point cloud.

[0050] Calculate the mean height and standard deviation for the multiple sets of height data obtained from the calculation;

[0051] The test sample detection module is used to select the test sample, acquire its three-dimensional point cloud data and two-dimensional image based on the 3D camera system, denoted as the test point cloud and the test image, and align the acquired test point cloud and test image to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction.

[0052] Based on the specific regions of interest (ROIs) already marked on the standard image, mark the ROIs for each component on the acquired test image. Extract the corresponding point cloud from the point cloud after the non-rigid transformation, and calculate the current height of each component based on the extracted point cloud. If the following conditions are met:

[0053]

[0054] If so, the component is a defective component; otherwise, it is a normal component. At the current altitude, The average height is σ, where t is a threshold parameter. i Let be the standard deviation.

[0055] In a first possible implementation of the second aspect, the 3D camera system includes a high-resolution camera and two tilt-shift optical engines used in conjunction with it.

[0056] In the second possible implementation of the second aspect, the standard point cloud is denoted as (X). g ,Y g Z g The standard image is P. g ;

[0057] The overall region of interest R0 of the electronic product and the specific region of interest R of each component i i = 1, 2, ..., N;

[0058] The component point cloud is

[0059] The first standard sample module is further used for:

[0060] Step a, perform point cloud analysis on the components. Perform downsampling, retaining M data points, to obtain

[0061] Step b, process the downsampled component point cloud The least squares method is used to fit the data using the three-dimensional planar formula AX + BY + CZ + D = 0, resulting in a set of parameters A, B, C, and D; the residuals for all data points are then calculated. Sort all residuals and calculate the first quartile q1, the third quartile Q3, and the interquartile range IQR = Q3 - Q1; mark data points whose residuals are outside the range [0, Q3 + α × IQR] as outliers and remove them, where α is a threshold parameter;

[0062] Step c: Repeat step b above until there are no outlier data points. Record the plane fitting parameters at this point as A', B', C', D'. Calculate the height of the component using the following formula:

[0063]

[0064] in, After removing outlier data points The average value.

[0065] Based on the second possible implementation of the second aspect, in the third possible implementation of the second aspect, three-dimensional point cloud data and two-dimensional images of K groups of defect-free electronic product samples are collected. The collected three-dimensional point cloud data is denoted as the test point cloud (X). k ,Y k Z k Let k = 1, 2, ..., K, and let the acquired two-dimensional image be the test image P. k ;

[0066] The second standard sample module is specifically used for:

[0067] For each set of test images P acquired k Align it with the standard image P by performing a two-dimensional affine transformation. g Let T be the two-dimensional affine transformation matrix.

[0068] Based on the two-dimensional affine transformation matrix T, for each acquired test point cloud (X) k ,Y k Z k ), k=1,2,…,K, perform the same two-dimensional affine transformation on it to align the standard point cloud (X) g ,Y g Z g ), denoted as (X k1 ,Y k1 Z k1 ), k = 1, 2, ..., K;

[0069] The residual formula is defined as follows:

[0070] r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy2 +c7x 2 +c8x 2 y+c9x 3

[0071] Define the optimization objective function as follows:

[0072]

[0073] Where j = 1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; the coefficients c of the residual formula r(x,y) are obtained by optimizing the objective function. i i = 0, 1, ..., 9;

[0074] For the aligned test point cloud (X) k1 ,Y k1 Z k1 Non-rigid transformations are performed on k = 1, 2, ..., K:

[0075] X k2 =X k1

[0076] Y k2 =Y k1

[0077]

[0078] The transformed coordinates are (X k2 ,Y k2 Z k2 ), k = 1, 2, ..., K.

[0079] Thirdly, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the electronic product component height detection method as described in the first aspect.

[0080] Fourthly, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the electronic product component height detection method as described in the first aspect.

[0081] Beneficial effects:

[0082] This invention enables high-precision detection of minute differences in the height direction of components, improving the accuracy of defect detection; it can quickly and accurately detect height direction defects of components in large electronic devices without affecting the production line speed, ensuring product quality and production efficiency. It is suitable for electronic products moving on the production line, does not require additional positioning devices, greatly improves detection efficiency, and is more suitable for modern high-efficiency production environments. Attached Figure Description

[0083] Figure 1 A schematic flowchart illustrating a height detection method for electronic product components provided in this application embodiment;

[0084] Figure 2 A structural block diagram of an electronic product component height detection device provided in this application embodiment;

[0085] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0086] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions in the embodiments of this application are clearly described. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.

[0087] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0088] The steps described in the specification and the flowcharts in the accompanying drawings of this invention are not necessarily strictly executed according to the step numbers; the execution order of the method steps can be changed. Furthermore, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be broken down into multiple steps.

[0089] The following detailed description, in conjunction with the accompanying drawings and preferred embodiments, describes the height detection method, apparatus, and medium for electronic product components provided in this application.

[0090] First, the application scenarios of the height detection method for electronic product components in this application will be described in detail.

[0091] 3D Automated Optical Inspection (3D AOI) technology is an advanced automated optical inspection technology that can simultaneously acquire two-dimensional images and three-dimensional height information of the object to be inspected through laser or structured light scanning. It can be widely used in the inspection of electronic circuit boards, and can accurately identify the position of components, soldering quality, and height differences of components.

[0092] However, existing 3D AOI equipment typically requires the inspected object to be in a fixed position, making it difficult to adapt to the changing positions of products on electronic product manufacturing lines. Furthermore, existing 3D AOI equipment has a small field of view, making it unsuitable for inspecting large products such as laptops, limiting its application to only localized areas.

[0093] In the production of large-sized electronic products such as laptops, especially for some key components such as screws, solid-state drives (SSDs), and batteries, improper installation may lead to equipment failure or malfunction.

[0094] Current market solutions for detecting height defects in electronic products, especially large electronic devices, suffer from problems such as slow detection speed, insufficient accuracy, and difficulty in adapting to rapid assembly line operations. This application provides a method for detecting the height of electronic product components, aiming to quickly and accurately detect height-oriented defects in components of large electronic devices without affecting assembly line production speed, thus ensuring product quality and production efficiency.

[0095] Please see Figure 1 This application provides a method for detecting the height of electronic product components, such as... Figure 1 As shown, the detection method of this application embodiment includes the following steps:

[0096] Step S1: Select a defect-free electronic product as a standard sample, and acquire the three-dimensional point cloud data and two-dimensional image of the standard sample based on a pre-configured 3D camera system; wherein, the 3D camera system includes a high-resolution camera and two tilt-shift optical engines used in conjunction with it; each pixel of the acquired three-dimensional point cloud data and two-dimensional image corresponds one-to-one.

[0097] Specifically, a 3D camera system can consist of a high-resolution camera and two tilt-shift optical engines. The high-resolution camera is centrally located with its optical axis pointing vertically downwards towards the object being measured. The two tilt-shift optical engines are symmetrically placed on either side of the high-resolution camera, with their optical axes intersecting the camera's optical axis. The fields of view of the high-resolution camera and the two tilt-shift optical engines coincide on the plane containing the object being measured, and all three are focused on that plane. Using this 3D camera system, three-dimensional point cloud data and two-dimensional photographs of the object can be obtained, and then each pixel of the three-dimensional point cloud data and the two-dimensional image can be mapped one-to-one.

[0098] Step S2: Configure the 3D point cloud data of the standard sample as a standard point cloud; configure the 2D image of the standard sample as a standard image; mark the overall region of interest of the electronic product and the specific region of interest of each component on the standard image.

[0099] Step S3: For each component's specific region of interest, extract the corresponding point cloud from the standard point cloud to form the component's point cloud.

[0100] Step S4: Calculate the height of each component based on the component point cloud.

[0101] The specific method is as follows:

[0102] For example, the three-dimensional point cloud data of the acquired standard sample is denoted as the standard point cloud (X). g ,Y g Z g The acquired two-dimensional photograph is denoted as standard image P. g .

[0103] Mark the overall region of interest R0 of the electronic product and the specific region of interest R of each component on the two-dimensional image of the standard sample. i ,i=1,2,…,N; where R0 is usually a rectangle, R i i = 1, 2, ..., N can be of any shape and should match the shape of the parts.

[0104] For the region of interest R of each component i For each i = 1, 2, ..., N, the corresponding point cloud portion is extracted from the 3D point cloud data of the standard sample (standard point cloud) to form a separate dataset, denoted as .

[0105] For each extracted component point cloud Calculate the height of the parts This includes:

[0106] Step a, perform point cloud analysis on the components. Perform downsampling, retaining M data points, to obtain

[0107] Step b, process the downsampled component point cloud The least squares method is used to fit the data using the three-dimensional planar formula AX + BY + CZ + D = 0, resulting in a set of parameters A, B, C, and D; the residuals for all data points are then calculated. Sort all residuals and calculate the first quartile Q1, the third quartile Q3, and the interquartile range IQR = Q3 - Q1; mark data points whose residuals are outside the range [0, Q3 + α × IQR] as outliers and remove them, where α is a threshold parameter;

[0108] Step c: Repeat step b above until there are no outlier data points. Record the plane fitting parameters at this point as A', B', C', D'. Calculate the height of the component using the following formula:

[0109]

[0110] in, After removing outlier data points The average value.

[0111] Step S5: Collect at least one set of different standard samples, acquire their three-dimensional point cloud data and two-dimensional images based on the 3D camera system, align each set of three-dimensional point cloud data and two-dimensional images to the standard point cloud and standard image respectively through two-dimensional affine transformation, and perform further correction by non-rigid transformation.

[0112] Specifically, due to the changing positions of products on the electronic product manufacturing line, the position of each collected electronic product sample relative to the conveyor belt is different, thus requiring alignment of the collected point cloud and image data. After aligning each set of data through a two-dimensional affine transformation, some residual errors still remain, so a non-rigid transformation is performed for further correction.

[0113] For example, three-dimensional point cloud data and two-dimensional images of K groups of defect-free electronic product samples are collected. The collected three-dimensional point cloud data is denoted as the test point cloud (X). k ,Y k Z k Let k = 1, 2, ..., K, and let the acquired two-dimensional image be the test image P. k .

[0114] The above step S5 specifically includes:

[0115] Step S501, for each acquired test image P k Align it with the standard image P by performing a two-dimensional affine transformation. gLet T be the two-dimensional affine transformation matrix.

[0116] Step S502, based on the two-dimensional affine transformation matrix T, for each acquired test point cloud (X) k ,Y k Z k ), k=1,2,…,K, perform the same two-dimensional affine transformation on it to align the standard point cloud (X) g ,Y g Z g ), denoted as (X k1 ,Y k1 Z k1 ), k = 1, 2, ..., K.

[0117] Step S503, define the residual formula as follows:

[0118] r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy 2 +c7x 2 +c8x 2 y+c9x 3

[0119] Define the optimization objective function as follows:

[0120]

[0121] Where j = 1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; optimizing the above objective function yields the coefficient c of the residual formula r(x,y). i ,i=0,1,…,9.

[0122] Step S504, perform alignment on the test point cloud (X) k1 ,Y k1 Z k1 Non-rigid transformations are performed on k = 1, 2, ..., K:

[0123] X k2 =X k1

[0124] Y k2 =Y k1

[0125]

[0126] The transformed coordinates are (X k2 ,Y k2 Z k2 ), k = 1, 2, ..., K.

[0127] Step S6: Based on the specific regions of interest already marked on the standard image, mark the specific regions of interest for each component on each set of acquired 2D images, and extract the corresponding point cloud from the point cloud after non-rigid transformation. Calculate the height of each component based on the extracted point cloud. The calculation method is the same as in step S4.

[0128] In step S6, for each component, the region of interest R... i ,i=1,2,…,N, in the transformed point cloud (X k2 ,Y k2 Z k2 Extract the corresponding point cloud portion from k = 1, 2, ..., K, and calculate the height of the component using the same method as in step S4, denoted as . Where k = 1, 2, ..., K represents the K data group numbers, and i = 1, 2, ..., N represents the N regions of interest numbers.

[0129] Step S7: Calculate the average height and standard deviation for the multiple sets of height data obtained.

[0130] For each region of interest, K sets of height data are used to calculate the mean height and standard deviation using the following formulas:

[0131] average height

[0132] Standard deviation

[0133] Calculate the average and standard deviation of the K sets of height data for each component to determine the mean and variability of height within the normal range.

[0134] Step S8: Select the sample to be tested, acquire its 3D point cloud data and 2D image based on the 3D camera system, denoted as the test point cloud and the test image. Align the acquired test point cloud and test image to the standard point cloud and standard image respectively through 2D affine transformation, and perform further correction through non-rigid transformation. The process is the same as step S5.

[0135] Step S9: Based on the specific regions of interest already marked on the standard image, mark the specific regions of interest for each component on the acquired test image, extract the corresponding point cloud from the point cloud after non-rigid transformation, and calculate the current height of each component based on the extracted point cloud; if the condition is met:

[0136]

[0137] If so, the component is a defective component; otherwise, it is a normal component. At the current altitude, The average height is σ, where t is a threshold parameter and σ is the average height. i This represents the standard deviation. The method for calculating height is the same as in step S4.

[0138] For example, when the electronic product is a laptop computer, the parameters used in the above embodiment are as follows: in step S4, M = 40,000 data points are retained, and α = 0.1; in step S5, the number of defect-free data groups collected is K = 20; and in step S9, the threshold parameter t = 3.

[0139] Based on the above technical solution, this invention can detect minute differences in the height direction of components with high precision, thereby improving the accuracy of defect detection. It can quickly and accurately detect height direction defects of components in large electronic devices without affecting the production speed of the assembly line, ensuring product quality and production efficiency. It is suitable for electronic products moving on the assembly line, does not require additional positioning devices, greatly improves detection efficiency, and is more suitable for modern high-efficiency production environments.

[0140] See Figure 2 Corresponding to the above embodiments of the height detection method for electronic product components, this application provides an electronic product component height detection device, which includes:

[0141] The first standard sample module 1001 is used to select a defect-free electronic product as a standard sample and acquire three-dimensional point cloud data and two-dimensional images of the standard sample based on a pre-configured 3D camera system; wherein, each pixel of the acquired three-dimensional point cloud data and the two-dimensional image corresponds one-to-one.

[0142] Configure the three-dimensional point cloud data of the standard sample as a standard point cloud; configure the two-dimensional image of the standard sample as a standard image; mark the overall region of interest of the electronic product and the specific region of interest of each component on the standard image;

[0143] For each specific region of interest of the component, the corresponding point cloud is extracted from the standard point cloud to form the component point cloud;

[0144] Calculate the height of each component based on the point cloud of the components;

[0145] The second standard sample module 1002 is used to acquire at least one different set of standard samples, obtain their three-dimensional point cloud data and two-dimensional images based on the 3D camera system, align each set of the acquired three-dimensional point cloud data and two-dimensional images to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction.

[0146] Based on the specific regions of interest already marked on the standard image, mark the specific regions of interest for each component on each set of acquired two-dimensional images, and extract the corresponding point cloud from the point cloud after the non-rigid transformation, and calculate the height of each component based on the extracted point cloud.

[0147] Calculate the mean height and standard deviation for the multiple sets of height data obtained from the calculation;

[0148] The test sample detection module 1003 is used to select the test sample, acquire its three-dimensional point cloud data and two-dimensional image based on the 3D camera system, and denoted as the test point cloud and the test image. The acquired test point cloud and test image are aligned to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and further corrected by non-rigid transformation.

[0149] Based on the specific regions of interest (ROIs) already marked on the standard image, mark the ROIs for each component on the acquired test image. Extract the corresponding point cloud from the point cloud after the non-rigid transformation, and calculate the current height of each component based on the extracted point cloud. If the following conditions are met:

[0150]

[0151] If so, the component is a defective component; otherwise, it is a normal component. At the current altitude, The average height is σ, where t is a threshold parameter. i Let be the standard deviation.

[0152] Furthermore, the 3D camera system includes a high-resolution camera and two tilt-shift optical engines used in conjunction with it.

[0153] Furthermore, let the standard point cloud be denoted as (X). g ,Y g Z g The standard image is P. g ;

[0154] The overall region of interest R0 of the electronic product and the specific region of interest R of each component i i = 1, 2, ..., N;

[0155] The component point cloud is

[0156] The first standard sample module is further used for:

[0157] Step a, perform point cloud analysis on the components. Perform downsampling, retaining M data points, to obtain

[0158] Step b, process the downsampled component point cloud The least squares method is used to fit the data using the three-dimensional planar formula AX + BY + CZ + D = 0, resulting in a set of parameters A, B, C, and D; the residuals for all data points are then calculated. Sort all residuals and calculate the first quartile Q1, the third quartile Q3, and the interquartile range IQR = Q3 - Q1; mark data points whose residuals are outside the range [0, Q3 + α × IQR] as outliers and remove them, where α is a threshold parameter;

[0159] Step c: Repeat step b above until there are no outlier data points. Record the plane fitting parameters at this point as A', B', C', D'. Calculate the height of the component using the following formula:

[0160]

[0161] in, After removing outlier data points The average value.

[0162] Furthermore, three-dimensional point cloud data and two-dimensional images of K groups of defect-free electronic product samples were collected. The collected three-dimensional point cloud data is denoted as the test point cloud (X). k ,Y k Z k Let k = 1, 2, ..., K, and let the acquired two-dimensional image be the test image P. k ;

[0163] The second standard sample module is specifically used for:

[0164] For each set of test images P acquired k Align it with the standard image P by performing a two-dimensional affine transformation. g Let T be the two-dimensional affine transformation matrix.

[0165] Based on the two-dimensional affine transformation matrix T, for each acquired test point cloud (X) k ,Y k Z k ), k=1,2,…,K, perform the same two-dimensional affine transformation on it to align the standard point cloud (X) g ,Y g Z g ), denoted as (X k1 ,Y k1 Z k1 ), k = 1, 2, ..., K;

[0166] The residual formula is defined as follows:

[0167] r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy 2 +c7x 2 +c8x 2 y+c9x 3

[0168] Define the optimization objective function as follows:

[0169]

[0170] Where j = 1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; the coefficients c of the residual formula r(x,y) are obtained by optimizing the objective function. i i = 0, 1, ..., 9;

[0171] For the aligned test point cloud (X) k1 ,Y k1 Z k1 Non-rigid transformations are performed on k = 1, 2, ..., K:

[0172] X k2 =X k1

[0173] Y k2 =Y k1

[0174]

[0175] The transformed coordinates are (X k2 ,Y k2 Z k2 ), k = 1, 2, ..., K.

[0176] The above-mentioned electronic product component height detection device implements the steps and processes of the above-mentioned electronic product component height detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0177] See Figure 3 Corresponding to the above embodiments of the height detection method for electronic product components, this application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps and processes of the above embodiments of the height detection method for electronic product components and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0178] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback function, image playback function, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0179] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0180] Corresponding to the above embodiments of the height detection method for electronic product components, this application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps and processes of the above embodiments of the height detection method for electronic product components and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0181] The processor is the processor in the electronic device described in the above embodiments of this application. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0182] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0184] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for detecting the height of electronic product components, characterized in that, include: A defect-free electronic product is selected as a standard sample, and three-dimensional point cloud data and two-dimensional images of the standard sample are acquired based on a pre-configured 3D camera system; wherein each pixel of the acquired three-dimensional point cloud data and the two-dimensional image corresponds one-to-one. Configure the three-dimensional point cloud data of the standard sample as a standard point cloud; configure the two-dimensional image of the standard sample as a standard image; mark the overall region of interest of the electronic product and the specific region of interest of each component on the standard image; For each specific region of interest of the component, the corresponding point cloud is extracted from the standard point cloud to form the component point cloud; Calculate the height of each component based on the point cloud of the components; At least one set of different standard samples are collected, and their three-dimensional point cloud data and two-dimensional images are acquired based on the 3D camera system. Each set of the acquired three-dimensional point cloud data and two-dimensional images are aligned to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and further correction is performed by non-rigid transformation. Based on the specific regions of interest already marked on the standard image, mark the specific regions of interest for each component on each set of acquired two-dimensional images, and extract the corresponding point cloud from the point cloud after the non-rigid transformation, and calculate the height of each component based on the extracted point cloud. Calculate the mean height and standard deviation for the multiple sets of height data obtained; Select a sample to be tested, and acquire its three-dimensional point cloud data and two-dimensional image based on the 3D camera system. These are denoted as the point cloud to be tested and the image to be tested. The acquired point cloud to be tested and the image to be tested are aligned to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and further corrected by non-rigid transformation. Based on the specific regions of interest (ROIs) already marked on the standard image, mark the ROIs for each component on the acquired test image. Extract the corresponding point cloud from the point cloud after the non-rigid transformation, and calculate the current height of each component based on the extracted point cloud. If the following conditions are met: If so, the component is a defective component; otherwise, it is a normal component. At the current altitude, The average height is σ, where t is a threshold parameter. i Let be the standard deviation.

2. The method for detecting the height of electronic product components according to claim 1, characterized in that, The 3D camera system includes a high-resolution camera and two tilt-shift optical engines used in conjunction with it.

3. The method for detecting the height of electronic product components according to claim 1, characterized in that, Let the standard point cloud be (X). g ,Y g Z g The standard image is P. g ; The overall region of interest R0 of the electronic product and the specific region of interest R of each component i i = 1, 2, ..., N; The component point cloud is The calculation of the height of each component based on the point cloud of the components specifically includes: Step a, perform point cloud analysis on the components. Perform downsampling, retaining M data points, to obtain Step b, process the downsampled component point cloud The least squares method is used to fit the data using the three-dimensional planar formula AX + BY + CZ + D = 0, resulting in a set of parameters A, B, C, and D; the residuals for all data points are then calculated. Sort all residuals and calculate the first quartile Q1, the third quartile Q3, and the interquartile range IQR = Q3 - Q1; mark data points whose residuals are outside the range [0, Q3 + α × IQR] as outliers and remove them, where α is a threshold parameter; Step c: Repeat step b above until there are no outlier data points. Record the plane fitting parameters at this point as A', B', C', D'. Calculate the height of the component using the following formula: in, After removing outlier data points The average value.

4. The method for detecting the height of electronic product components according to claim 3, characterized in that, In addition, three-dimensional point cloud data and two-dimensional images of K groups of defect-free electronic product samples were collected. The collected three-dimensional point cloud data is denoted as the test point cloud (X). k ,Y k Z k Let k = 1, 2, ..., K, and let the acquired two-dimensional image be the test image P. k ; The process involves acquiring at least one set of different standard samples, obtaining their 3D point cloud data and 2D images based on the 3D camera system, aligning each set of the acquired 3D point cloud data and 2D images to the standard point cloud and the standard image respectively through a 2D affine transformation, and performing further correction through a non-rigid transformation. Specifically, this includes: For each set of test images P acquired k Align it with the standard image P by performing a two-dimensional affine transformation. g Let T be the two-dimensional affine transformation matrix. Based on the two-dimensional affine transformation matrix T, for each acquired test point cloud (X) k ,Y k Z k ), k=1,2,…,K, perform the same two-dimensional affine transformation on it to align the standard point cloud (X) g ,Y g Z g ), denoted as (X k1 ,Y k1 Z K1 (,k=1,2,…,K; The residual formula is defined as follows: r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy 2 +c7x 2 +c8x 2 y+c9x 3 Define the optimization objective function as follows: Where j = 1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; the coefficients c of the residual formula r(x,y) are obtained by optimizing the objective function. i i = 0, 1, ..., 9; For the aligned test point cloud (X) k1 ,Y k1 Z k1 Non-rigid transformations are performed on k = 1, 2, ..., K: X k2 =X k1 AND k2 And k1 The transformed coordinates are (X k2 ,Y k2 Z k2 ), k = 1, 2, ..., K.

5. A height detection device for electronic product components, characterized in that, include: The first standard sample module is used to select defect-free electronic products as standard samples and acquire three-dimensional point cloud data and two-dimensional images of the standard samples based on a pre-configured 3D camera system; wherein, each pixel of the acquired three-dimensional point cloud data and the two-dimensional image corresponds one-to-one. Configure the three-dimensional point cloud data of the standard sample as a standard point cloud; configure the two-dimensional image of the standard sample as a standard image; mark the overall region of interest of the electronic product and the specific region of interest of each component on the standard image; For each specific region of interest of the component, the corresponding point cloud is extracted from the standard point cloud to form the component point cloud; Calculate the height of each component based on the point cloud of the components; The second standard sample module is used to acquire at least one different set of standard samples, obtain their three-dimensional point cloud data and two-dimensional images based on the 3D camera system, align each set of the acquired three-dimensional point cloud data and two-dimensional images to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction. Based on the specific regions of interest already marked on the standard image, mark the specific regions of interest for each component on each set of acquired two-dimensional images, and extract the corresponding point cloud from the point cloud after the non-rigid transformation, and calculate the height of each component based on the extracted point cloud. Calculate the mean height and standard deviation for the multiple sets of height data obtained from the calculation; The test sample detection module is used to select the test sample, acquire its three-dimensional point cloud data and two-dimensional image based on the 3D camera system, denoted as the test point cloud and the test image, and align the acquired test point cloud and test image to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction. Based on the specific regions of interest (ROIs) already marked on the standard image, mark the ROIs for each component on the acquired test image. Extract the corresponding point cloud from the point cloud after the non-rigid transformation, and calculate the current height of each component based on the extracted point cloud. If the following conditions are met: If so, the component is a defective component; otherwise, it is a normal component. At the current altitude, The average height is σ, where t is a threshold parameter. i Let be the standard deviation.

6. The height detection device for electronic product components according to claim 5, characterized in that, The 3D camera system includes a high-resolution camera and two tilt-shift optical engines used in conjunction with it.

7. The height detection device for electronic product components according to claim 5, characterized in that, Let the standard point cloud be (X). g ,Y g Z g The standard image is P. g ; The overall region of interest R0 of the electronic product and the specific region of interest R of each component i i = 1, 2, ..., N; The component point cloud is The first standard sample module is further used for: Step a, perform point cloud analysis on the components. Perform downsampling, retaining M data points, to obtain Step b, process the downsampled component point cloud The least squares method is used to fit the data using the three-dimensional planar formula AX + BY + CZ + D = 0, resulting in a set of parameters A, B, C, and D; the residuals for all data points are then calculated. Sort all residuals and calculate the first quartile Q1, the third quartile Q3, and the interquartile range IQR = Q3 - Q1; mark data points whose residuals are outside the range [0, Q3 + α × IQR] as outliers and remove them, where α is a threshold parameter; Step c: Repeat step b above until there are no outlier data points. Record the plane fitting parameters at this point as A', B', C', D'. Calculate the height of the component using the following formula: in, After removing outlier data points The average value.

8. The height detection device for electronic product components according to claim 7, characterized in that, In addition, three-dimensional point cloud data and two-dimensional images of K groups of defect-free electronic product samples were collected. The collected three-dimensional point cloud data is denoted as the test point cloud (X). k ,Y k Z k Let k = 1, 2, ..., K, and let the acquired two-dimensional image be the test image P. k ; The second standard sample module is specifically used for: For each set of test images P acquired k Align it with the standard image P by performing a two-dimensional affine transformation. g ; Let T be the two-dimensional affine transformation matrix; Based on the two-dimensional affine transformation matrix T, for each acquired test point cloud (X) k ,Y k Z k ), k=1,2,…,K, perform the same two-dimensional affine transformation on it to align the standard point cloud (X) g ,Y g Z g ), denoted as (X k1 ,Y k1 Z k1 ), k = 1, 2, ..., K; The residual formula is defined as follows: r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy 2 +c7x 2 +c8x 2 y+c9x 3 Define the optimization objective function as follows: Where j = 1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; the coefficients c of the residual formula r(x,y) are obtained by optimizing the objective function. i i = 0, 1, ..., 9; For the aligned test point cloud (X) k1 ,Y k1 Z k1 Non-rigid transformations are performed on k = 1, 2, ..., K: X k2 =X k1 AND k2 And k1 The transformed coordinates are (X k2 ,Y k2 Z k2 ), k = 1, 2, ..., K.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the height detection method for electronic product components as described in any one of claims 1 to 4.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the electronic product component height detection method as described in any one of claims 1 to 4.

Images