Electronic product part height detection method, device and equipment and storage medium

Abstract

The invention provides an electronic product part height detection method, device and equipment and a storage medium, and the method comprises the steps: selecting a standard sample, and obtaining three-dimensional point cloud data and a two-dimensional image of the standard sample based on a 3D camera system, and enabling the three-dimensional point cloud data and the two-dimensional image to serve as a standard point cloud and a standard image; marking an integral region of interest of the product and a specific region of interest of each part on the standard image; intercepting a corresponding point cloud from the standard point cloud so as to calculate the height of each part; additionally collecting at least one group of different standard samples, obtaining three-dimensional point cloud data and two-dimensional images of the standard samples, respectively aligning the three-dimensional point cloud data and the two-dimensional images to the standard point cloud and the standard images, and performing non-rigid transformation correction; calculating the height of each part again; determining an average height and a standard deviation; selecting a to-be-tested sample, acquiring three-dimensional point cloud data and a two-dimensional image of the to-be-tested sample, respectively aligning the three-dimensional point cloud data and the two-dimensional image to the standard point cloud and the standard image, and performing non-rigid transformation correction; calculating the current height of each part; and judging whether the parts are defective parts or not according to preset conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of product component defect detection, and in particular to a method, device, equipment and storage medium for height detection of electronic product components. Background Art

[0002] In the production process of electronic products, the installation accuracy of components has a direct impact on the quality and performance of the final device. In order to ensure product quality, manufacturers usually need to implement strict inspection processes to verify whether these components are installed correctly.

[0003] At present, the commonly used inspection methods in the industry mainly include manual visual inspection and automatic optical inspection (AOI) based on 2D image processing technology. These two methods can identify obvious defects to a certain extent, such as missing parts or significant deviations in position, but they are limited when facing more complex and subtle problems (such as slight differences in height direction). This is because the human eye is more likely to ignore subtle differences, and 2D AOI technology cannot provide enough depth information to distinguish these subtle differences.

[0004] In the related technologies, although 3DAOI technology can simultaneously obtain the two-dimensional plane image and three-dimensional spatial information of the object to be measured by using the principle of laser scanning or structured light, it can theoretically more accurately evaluate the position accuracy of components and other aspects. However, such systems often require the object to be measured to remain stationary for accurate measurement, and it is difficult to achieve the desired effect in a continuous production environment on a dynamic conveyor belt; secondly, due to the limited field of view, the 3DAOI technology has a limited field of view and is difficult to cover the comprehensive inspection needs of larger-sized products. It can only realize the inspection of local areas, which affects the inspection efficiency. Summary of the invention

[0005] The present invention provides a method, device, equipment and storage medium for detecting the height of electronic product components, which solves the problem of detecting height defects of components in large electronic equipment under a continuous production environment on a dynamic conveyor belt.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a method for detecting the height of electronic product components is provided, comprising:

[0008] Select an electronic product without defects as a standard sample, and obtain three-dimensional point cloud data and a two-dimensional image of the standard sample based on a pre-configured 3D camera system; wherein the obtained three-dimensional point cloud data and each pixel point of the two-dimensional image correspond one to one;

[0009] The three-dimensional point cloud data of the standard sample is configured as a standard point cloud; the two-dimensional image of the standard sample is configured as a standard image; and the overall area of ​​interest of the electronic product and the specific area of ​​interest of each component are marked on the standard image;

[0010] For each specific region of interest of the component, a corresponding point cloud is intercepted from the standard point cloud to form a component point cloud;

[0011] Calculate the height of each component according to the component point cloud;

[0012] In addition, at least one group of different standard samples is collected, and the three-dimensional point cloud data and the two-dimensional image thereof are obtained based on the 3D camera system, and each group of the three-dimensional point cloud data and the two-dimensional image obtained is respectively aligned to the standard point cloud and the standard image through two-dimensional affine transformation, and further corrected by non-rigid transformation;

[0013] According to the specific regions of interest marked on the standard image, the specific regions of interest of each component are marked on each set of the acquired two-dimensional images, and the corresponding point cloud is intercepted from the point cloud after the non-rigid transformation, and the height of each component is calculated according to the intercepted point cloud;

[0014] Calculate the average height and standard deviation for the multiple sets of height data obtained by calculation;

[0015] Select a sample to be tested, obtain its three-dimensional point cloud data and two-dimensional image based on the 3D camera system, record them as the point cloud to be tested and the image to be tested, align the obtained point cloud to be tested and the image to be tested to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction;

[0016] According to the specific region of interest marked on the standard image, the specific region of interest of each component is marked on the acquired image to be tested, the corresponding point cloud is intercepted from the point cloud after the non-rigid transformation, and the current height of each component is calculated based on the intercepted point cloud; if the conditions are met:

[0017]

[0018] If the component is defective, it is a normal component. is the current height, is the average height, t is a threshold parameter, σ i is the standard deviation.

[0019] In a first possible implementation manner of the first aspect, the 3D camera system includes a high-resolution camera and two tilt-shift optical machines used in conjunction with the high-resolution camera.

[0020] In a second possible implementation manner of the first aspect, the standard point cloud is denoted as (X g ,Y g ,Z g ), the standard image is P g ;

[0021] The overall area of ​​interest R0 of the electronic product and the specific area of ​​interest R of each component i ,i=1,2,…,N;

[0022] The component point cloud is

[0023] The step of calculating the height of each component according to the component point cloud specifically includes:

[0024] Step a: Point cloud of components Downsample and retain M data points to obtain

[0025] Step b: Get the component point cloud after downsampling According to the three-dimensional plane formula AX+BY+CZ+D=0, the least squares method is used to fit a set of parameters A, B, C, D; the residuals of all data points are calculated. Sort all residuals, calculate the first quartile Q1, the third quartile Q3 and the interquartile range IQR = Q3-Q1; mark the data points whose residuals are outside the range [0, Q3+α×IDR] as outliers and remove them, where α is a threshold parameter;

[0026] Step c, repeat the above step b until there are no outlier data points, record the plane fitting parameters at this time as A', B', C', D'; use the following formula to calculate the height of the component:

[0027]

[0028] in, After removing outlier data points, The average value of .

[0029] Based on the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, three-dimensional point cloud data and two-dimensional images of K groups of defect-free electronic product samples are collected, and the collected three-dimensional point cloud data is recorded as the test point cloud (X k ,Y k ,Z k ), k = 1, 2, ..., K, the collected two-dimensional image is called the test image P k ;

[0030] The method of collecting at least one group of different standard samples, acquiring the three-dimensional point cloud data and the two-dimensional image thereof based on the 3D camera system, aligning each group of the three-dimensional point cloud data and the two-dimensional image acquired to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and further correcting by non-rigid transformation, specifically includes:

[0031] For each set of test images P k , perform a two-dimensional affine transformation to align the standard image P g ; Let the two-dimensional affine transformation matrix be T;

[0032] According to the two-dimensional affine transformation matrix T, for each set of test point clouds (X k ,Y k ,Z k ), k = 1, 2, ..., K, and perform the same two-dimensional affine transformation to align the standard point cloud (X g ,Y g ,Z g ), denoted as (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K;

[0033] The residual formula is defined as:

[0034] r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy 2 +c7x 2 +c8x 2 y+c9x 3

[0035] The optimization objective function is defined as:

[0036]

[0037] Wherein, j=1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; the coefficient c of the residual formula r(x, y) is obtained by optimizing the optimization objective function. i ,i=0,1,…,9;

[0038] After alignment, the test point cloud (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K for non-rigid transformation:

[0039] X k2 =X k1

[0040] Y k2 =Y k1

[0041]

[0042] The transformed coordinates are labeled (X k2 ,Y k2 ,Z k2 ),k=1,2,…,K.

[0043] In a second aspect, a device for detecting the height of electronic product components is provided, comprising:

[0044] The first standard sample module is used to select an electronic product without defects as a standard sample, and obtain the three-dimensional point cloud data and two-dimensional image of the standard sample based on a pre-configured 3D camera system; wherein the obtained three-dimensional point cloud data and each pixel point of the two-dimensional image correspond one to one;

[0045] The three-dimensional point cloud data of the standard sample is configured as a standard point cloud; the two-dimensional image of the standard sample is configured as a standard image; and the overall area of ​​interest of the electronic product and the specific area of ​​interest of each component are marked on the standard image;

[0046] For each specific region of interest of the component, a corresponding point cloud is intercepted from the standard point cloud to form a component point cloud;

[0047] Calculate the height of each component according to the component point cloud;

[0048] A second standard sample module is used to collect at least one group of different standard samples, obtain the three-dimensional point cloud data and the two-dimensional image thereof based on the 3D camera system, align each group of the three-dimensional point cloud data and the two-dimensional image obtained to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction;

[0049] According to the specific regions of interest marked on the standard image, the specific regions of interest of each component are marked on each set of the acquired two-dimensional images, and the corresponding point cloud is intercepted from the point cloud after the non-rigid transformation, and the height of each component is calculated according to the intercepted point cloud;

[0050] Calculate the average height and standard deviation for the multiple sets of height data obtained by calculation;

[0051] A sample detection module for testing is used to select a sample for testing, obtain its three-dimensional point cloud data and two-dimensional image based on the 3D camera system, record them as the point cloud for testing and the image for testing, align the obtained point cloud for testing and the image for testing to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction;

[0052] According to the specific region of interest marked on the standard image, the specific region of interest of each component is marked on the acquired image to be tested, the corresponding point cloud is intercepted from the point cloud after the non-rigid transformation, and the current height of each component is calculated based on the intercepted point cloud; if the conditions are met:

[0053]

[0054] If the component is defective, it is a normal component. is the current height, is the average height, t is a threshold parameter, σ i is the standard deviation.

[0055] In a first possible implementation manner of the second aspect, the 3D camera system includes a high-resolution camera and two tilt-shift optical machines used in conjunction with the camera.

[0056] In a second possible implementation manner of the second aspect, the standard point cloud is denoted as (X g ,Y g ,Z g ), the standard image is P g ;

[0057] The overall area of ​​interest R0 of the electronic product and the specific area of ​​interest R of each component i ,i=1,2,…,N;

[0058] The component point cloud is

[0059] The first standard sample module is further specifically used for:

[0060] Step a: Point cloud of components Downsample and retain M data points to obtain

[0061] Step b: Get the component point cloud after downsampling According to the three-dimensional plane formula AX+BY+CZ+D=0, the least squares method is used to fit a set of parameters A, B, C, D; the residuals of all data points are calculated. Sort all residuals, calculate the first quartile q1, the third quartile Q3 and the interquartile range IQR = Q3-Q1; mark the data points whose residuals are outside the range [0, Q3+α×IQR] as outliers and remove them, where α is a threshold parameter;

[0062] Step c, repeat step b above until there are no outlier data points, record the plane fitting parameters at this time as A', B', C', D'; use the following formula to calculate the height of the component:

[0063]

[0064] in, After removing outlier data points, The average value of .

[0065] Based on the second possible implementation manner of the second aspect, in a third possible implementation manner of the second aspect, three-dimensional point cloud data and two-dimensional images of K groups of defect-free electronic product samples are collected, and the collected three-dimensional point cloud data is recorded as the test point cloud (X k ,Y k ,Z k ), k = 1, 2, ..., K, the collected two-dimensional image is called the test image P k ;

[0066] The second standard sample module is specifically used for:

[0067] For each set of test images P k , perform a two-dimensional affine transformation to align the standard image P g ; Let the two-dimensional affine transformation matrix be T;

[0068] According to the two-dimensional affine transformation matrix T, for each set of test point clouds (X k ,Y k ,Z k ), k = 1, 2, ..., K, and perform the same two-dimensional affine transformation to align the standard point cloud (X g ,Y g ,Z g ), denoted as (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K;

[0069] The residual formula is defined as:

[0070] r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy2 +c7x 2 +c8x 2 y+c9x 3

[0071] The optimization objective function is defined as:

[0072]

[0073] Wherein, j=1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; the coefficient c of the residual formula r(x, y) is obtained by optimizing the optimization objective function. i ,i=0,1,…,9;

[0074] After alignment, the test point cloud (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K for non-rigid transformation:

[0075] X k2 =X k1

[0076] Y k2 =Y k1

[0077]

[0078] The transformed coordinates are labeled (X k2 ,Y k2 ,Z k2 ),k=1,2,…,K.

[0079] In a third aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for detecting height of electronic product components as described in the first aspect.

[0080] In a fourth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method for detecting the height of electronic product components as described in the first aspect are implemented.

[0081] Beneficial effects:

[0082] The present application is capable of detecting minute differences in the height direction of components with high precision, thereby improving the accuracy of defect detection; it is capable of quickly and accurately detecting height defects of components in large electronic equipment without affecting the production speed of the assembly line, thereby ensuring product quality and production efficiency, being suitable for electronic products moving on the assembly line, not requiring additional positioning devices, greatly improving detection efficiency, and being more suitable for a modern and efficient production environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A schematic flow chart of a method for detecting the height of electronic product components provided in an embodiment of the present application;

[0084] Figure 2 A structural block diagram of a device for detecting the height of electronic product components provided in an embodiment of the present application;

[0085] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the technical solutions in the embodiments of the present application are clearly described. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present application.

[0087] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0088] The description of the method flow in the specification of the present application and the steps of the flowchart in the drawings of the present specification do not have to be strictly executed according to the step numbers, and the method steps can be executed in a different order. Moreover, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0089] The following is a detailed description of the electronic product component height detection method, device and medium provided in the embodiments of the present application in conjunction with the accompanying drawings and preferred embodiments.

[0090] First, the application scenario of the electronic product component height detection method in the embodiment of the present application is described in detail.

[0091] 3D Automated optical inspection (3D AOI) technology is an advanced automatic optical inspection technology that can simultaneously obtain the two-dimensional image and three-dimensional height information of the object to be inspected through laser or structured light scanning. It can be widely used in component inspection of electronic circuit boards and can accurately identify component position, welding quality and component height differences.

[0092] However, existing 3D AOI equipment usually requires the position of the inspected object to be fixed, which makes it difficult to adapt to the changes in the position of products on the electronic product production line. At the same time, the existing 3D AOI equipment has a small field of view and is difficult to meet the inspection requirements of large-sized products such as laptops. It can only perform partial inspection, which also limits its application in electronic product production lines.

[0093] In the production process of large-size electronic products such as laptops, especially for some key components such as screws, solid-state drives (SSDs), batteries, etc., if their installation does not meet specifications, it may cause equipment failure or malfunction.

[0094] The current technical solutions for height defect detection of components in electronic products, especially large electronic equipment, have problems such as slow detection speed, insufficient accuracy, and difficulty in adapting to fast assembly line operation conditions. This application provides a method for height detection of electronic product components, which aims to quickly and accurately detect height defects of components in large electronic equipment without affecting the production speed of the assembly line, thereby ensuring product quality and production efficiency.

[0095] See also Figure 1 , the present application embodiment provides a method for detecting the height of electronic product components, such as Figure 1 As shown, the detection method of the embodiment of the present application includes the following steps:

[0096] Step S1, select an electronic product without defects as a standard sample, and obtain the three-dimensional point cloud data and two-dimensional image of the standard sample based on a pre-configured 3D camera system; wherein the 3D camera system includes a high-resolution camera and two shift optical machines used in conjunction with it; the obtained three-dimensional point cloud data corresponds to each pixel point of the two-dimensional image one by one.

[0097] Specifically, the 3D camera system can be composed of a high-resolution camera and two shift optical machines. The high-resolution camera is placed in the center, with the optical axis facing vertically downward to the object to be measured, and the two shift optical machines are placed symmetrically on both sides of the high-resolution camera with their optical axes intersecting with the optical axis of the high-resolution camera. The field of view of the high-resolution camera and the two shift optical machines coincide with the plane where the object to be measured is located, and the high-resolution camera and the two shift optical machines focus on the plane where the object to be measured is located. This 3D camera system can be used to take pictures to obtain three-dimensional point cloud data and two-dimensional photos of the object to be measured, and further correspond each pixel point of the three-dimensional point cloud data and the two-dimensional image one by one.

[0098] Step S2, configuring the three-dimensional point cloud data of the standard sample into a standard point cloud; configuring the two-dimensional image of the standard sample into a standard image; marking the overall area of ​​interest of the electronic product and the specific area of ​​interest of each component on the standard image.

[0099] Step S3: for each specific region of interest of a component, the corresponding point cloud is intercepted from the standard point cloud to form a component point cloud.

[0100] Step S4, calculating the height of each component according to the component point cloud.

[0101] The specific method is as follows:

[0102] For example, the three-dimensional point cloud data of the standard sample obtained is recorded as the standard point cloud (X g ,Y g ,Z g ), the collected two-dimensional photo is recorded as the standard image P g .

[0103] Mark the overall area of ​​interest R0 of the electronic product and the specific area of ​​interest R0 of each component on the two-dimensional image of the standard sample. i ,i=1,2,…,N; R0 is usually a rectangle, R i ,i=1,2,…,N can be any shape and should match the appearance of the component.

[0104] For each component's region of interest R i , i = 1, 2, ..., N, the corresponding point cloud part is cut out from the three-dimensional point cloud data of the standard sample (standard point cloud) to form a separate data set, which is recorded as

[0105] For each captured component point cloud Calculate the height of a component This includes:

[0106] Step a: Point cloud of components Downsample and retain M data points to obtain

[0107] Step b: Get the component point cloud after downsampling According to the three-dimensional plane formula AX+BY+CZ+D=0, the least squares method is used to fit a set of parameters A, B, C, D; the residuals of all data points are calculated. Sort all residuals, calculate the first quartile Q1, the third quartile Q3 and the interquartile range IQR = Q3-Q1; mark the data points whose residuals are outside the range [0, Q3+α×IQR] as outliers and remove them, where α is a threshold parameter;

[0108] Step c, repeat step b above until there are no outlier data points, record the plane fitting parameters at this time as A', B', C', D'; use the following formula to calculate the height of the component:

[0109]

[0110] in, After removing outlier data points, The average value of .

[0111] Step S5, collect at least one group of different standard samples, obtain their three-dimensional point cloud data and two-dimensional images based on the 3D camera system, align each group of three-dimensional point cloud data and two-dimensional images to the standard point cloud and standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction.

[0112] Specifically, due to the change of product position on the electronic product production line, the position of the electronic product sample collected each time is different relative to the position of the conveyor belt, so the collected point cloud and image data need to be aligned. After aligning each set of data through two-dimensional affine transformation, there are still some residual errors, which are further corrected by non-rigid transformation.

[0113] For example, three-dimensional point cloud data and two-dimensional images of K groups of defect-free electronic product samples are collected, and the collected three-dimensional point cloud data is recorded as the test point cloud (X k ,Y k ,Z k ), k = 1, 2, ..., K, the collected two-dimensional image is called the test image P k .

[0114] Then the above step S5 specifically includes:

[0115] Step S501: for each set of test images P k , perform a two-dimensional affine transformation to align the standard image P g; Let the two-dimensional affine transformation matrix be T;

[0116] Step S502: for each set of test point clouds (X k ,Y k ,Z k ), k = 1, 2, ..., K, and perform the same two-dimensional affine transformation to align the standard point cloud (X g ,Y g ,Z g ), denoted as (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K.

[0117] Step S503, define the residual formula as:

[0118] r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy 2 +c7x 2 +c8x 2 y+c9x 3

[0119] The optimization objective function is defined as:

[0120]

[0121] Wherein, j=1,2,…,m are all coordinate points in the overall region of interest R0 of the electronic product; optimizing the above optimization objective function can obtain the coefficient c of the residual formula r(x,y) i ,i=0,1,…,9.

[0122] Step S504: align the test point cloud (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K for non-rigid transformation:

[0123] X k2 =X k1

[0124] Y k2 =Y k1

[0125]

[0126] The transformed coordinates are labeled (X k2 ,Y k2 ,Z k2 ),k=1,2,…,K.

[0127] Step S6, based on the specific regions of interest marked on the standard image, mark the specific regions of interest of each component on each set of acquired two-dimensional images, and intercept the corresponding point cloud from the point cloud after non-rigid transformation, and calculate the height of each component based on the intercepted point cloud. The calculation method is the same as step S4.

[0128] In step S6, for each component, the region of interest R i ,i=1,2,…,N, in the transformed point cloud (X k2 ,Y k2 ,Z k2 ), k=1,2,…,K, and the corresponding point cloud part is cut out. The height of the component is calculated using the same method as step S4, which is recorded as Wherein k=1,2,…,K represents K groups of data numbers, and i=1,2,…,N represents N regions of interest numbers.

[0129] Step S7, calculating the average height and standard deviation of the multiple sets of height data obtained by calculation.

[0130] For each K set of height data of the region of interest, the average height and standard deviation are calculated according to the following formula:

[0131] average height

[0132] Standard Deviation

[0133] The K sets of height data for each part were averaged and standard deviations were calculated to determine the mean and variability of height within the normal range.

[0134] Step S8, select a sample to be tested, obtain its three-dimensional point cloud data and two-dimensional image based on the 3D camera system, record them as the point cloud to be tested and the image to be tested, align the obtained point cloud to be tested and the image to be tested to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction. The same as step S5.

[0135] Step S9, according to the specific regions of interest marked on the standard image, mark the specific regions of interest of each component on the acquired image to be tested, intercept the corresponding point cloud in the point cloud after non-rigid transformation, and calculate the current height of each component according to the intercepted point cloud; if the conditions are met:

[0136]

[0137] If the component is defective, it is a normal component. is the current height, is the average height, t is a threshold parameter, σ i The height calculation method is the same as step S4.

[0138] Exemplarily, when the electronic product is a laptop computer, the parameters used in the above embodiment are: M=40,000 data points are retained in step S4, α=0.1; the number of defect-free data groups collected in step S5 is K=20; and the threshold parameter t=3 in step S9.

[0139] Based on the above technical solution, the present application shows that it can detect tiny differences in the height direction of components with high precision, thereby improving the accuracy of defect detection; it can quickly and accurately detect height defects of components in large electronic equipment without affecting the production speed of the assembly line, thereby ensuring product quality and production efficiency, and is suitable for electronic products moving on the assembly line. It does not require additional positioning devices, greatly improves detection efficiency, and is more suitable for modern and efficient production environments.

[0140] See also Figure 2 , corresponding to the above-mentioned electronic product component height detection method embodiment, the present application embodiment provides an electronic product component height detection device, the device comprising:

[0141] The first standard sample module 1001 is used to select an electronic product without defects as a standard sample, and obtain the three-dimensional point cloud data and the two-dimensional image of the standard sample based on a pre-configured 3D camera system; wherein the obtained three-dimensional point cloud data and each pixel point of the two-dimensional image correspond one to one;

[0142] The three-dimensional point cloud data of the standard sample is configured as a standard point cloud; the two-dimensional image of the standard sample is configured as a standard image; and the overall area of ​​interest of the electronic product and the specific area of ​​interest of each component are marked on the standard image;

[0143] For each specific region of interest of the component, a corresponding point cloud is intercepted from the standard point cloud to form a component point cloud;

[0144] Calculate the height of each component according to the component point cloud;

[0145] The second standard sample module 1002 is used to collect at least one group of different standard samples, obtain the three-dimensional point cloud data and the two-dimensional image thereof based on the 3D camera system, align each group of the three-dimensional point cloud data and the two-dimensional image obtained to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction;

[0146] According to the specific regions of interest marked on the standard image, the specific regions of interest of each component are marked on each set of the acquired two-dimensional images, and the corresponding point cloud is intercepted from the point cloud after the non-rigid transformation, and the height of each component is calculated according to the intercepted point cloud;

[0147] Calculate the average height and standard deviation for the multiple sets of height data obtained by calculation;

[0148] The sample detection module 1003 is used to select a sample to be tested, obtain its three-dimensional point cloud data and two-dimensional image based on the 3D camera system, record them as the point cloud to be tested and the image to be tested, align the obtained point cloud to be tested and the image to be tested to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction;

[0149] According to the specific region of interest marked on the standard image, the specific region of interest of each component is marked on the acquired image to be tested, the corresponding point cloud is intercepted from the point cloud after the non-rigid transformation, and the current height of each component is calculated based on the intercepted point cloud; if the conditions are met:

[0150]

[0151] If the component is defective, it is a normal component. is the current height, is the average height, t is a threshold parameter, σ i is the standard deviation.

[0152] Furthermore, the 3D camera system includes a high-resolution camera and two tilt-shift optical machines used in conjunction with the camera.

[0153] Furthermore, the standard point cloud is denoted as (X g ,Y g ,Z g ), the standard image is P g ;

[0154] The overall area of ​​interest R0 of the electronic product and the specific area of ​​interest R of each component i ,i=1,2,…,N;

[0155] The component point cloud is

[0156] The first standard sample module is further specifically used for:

[0157] Step a: Point cloud of components Downsample and retain M data points to obtain

[0158] Step b: Get the component point cloud after downsampling According to the three-dimensional plane formula AX+BY+CZ+D=0, the least squares method is used to fit a set of parameters A, B, C, D; the residuals of all data points are calculated. Sort all residuals, calculate the first quartile Q1, the third quartile Q3 and the interquartile range IQR = Q3-Q1; mark the data points whose residuals are outside the range [0, Q3+α×IQR] as outliers and remove them, where α is a threshold parameter;

[0159] Step c, repeat step b above until there are no outlier data points, record the plane fitting parameters at this time as A', B', C', D'; use the following formula to calculate the height of the component:

[0160]

[0161] in, After removing outlier data points, The average value of .

[0162] Furthermore, three-dimensional point cloud data and two-dimensional images of K groups of defect-free electronic product samples are collected, and the collected three-dimensional point cloud data is recorded as the test point cloud (X k ,Y k ,Z k ), k = 1, 2, ..., K, the collected two-dimensional image is called the test image P k ;

[0163] The second standard sample module is specifically used for:

[0164] For each set of test images P k , perform a two-dimensional affine transformation to align the standard image P g ; Let the two-dimensional affine transformation matrix be T;

[0165] According to the two-dimensional affine transformation matrix T, for each set of test point clouds (X k ,Y k ,Z k ), k = 1, 2, ..., K, and perform the same two-dimensional affine transformation to align the standard point cloud (X g ,Y g ,Z g ), denoted as (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K;

[0166] The residual formula is defined as:

[0167] r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy 2 +c7x 2 +c8x 2 y+c9x 3

[0168] The optimization objective function is defined as:

[0169]

[0170] Wherein, j=1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; the coefficient c of the residual formula r(x, y) is obtained by optimizing the optimization objective function. i ,i=0,1,…,9;

[0171] After alignment, the test point cloud (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K for non-rigid transformation:

[0172] X k2 =X k1

[0173] Y k2 =Y k1

[0174]

[0175] The transformed coordinates are labeled (X k2 ,Y k2 ,Z k2 ),k=1,2,…,K.

[0176] The above-mentioned electronic product component height detection device implements the steps and various processes of the above-mentioned electronic product component height detection method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0177] See also Figure 3 Corresponding to the above-mentioned embodiment of the method for detecting the height of components of electronic products, an embodiment of the present application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps and various processes of the above-mentioned embodiment of the method for detecting the height of components of electronic products are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0178] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0179] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0180] Corresponding to the above-mentioned embodiment of the method for detecting the height of components of electronic products, the embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, the steps of the above-mentioned embodiment of the method for detecting the height of components of electronic products and the various processes of the embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0181] The processor is the processor in the electronic device described in the above embodiment of the present application. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0182] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0183] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0184] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for detecting the height of electronic product components, characterized in that: include: Select an electronic product without defects as a standard sample, and obtain three-dimensional point cloud data and a two-dimensional image of the standard sample based on a pre-configured 3D camera system; wherein the obtained three-dimensional point cloud data and each pixel point of the two-dimensional image correspond one to one; The three-dimensional point cloud data of the standard sample is configured as a standard point cloud; the two-dimensional image of the standard sample is configured as a standard image; and the overall area of ​​interest of the electronic product and the specific area of ​​interest of each component are marked on the standard image; For each specific region of interest of the component, a corresponding point cloud is intercepted from the standard point cloud to form a component point cloud; Calculate the height of each component according to the component point cloud; In addition, at least one group of different standard samples is collected, and the three-dimensional point cloud data and the two-dimensional image thereof are obtained based on the 3D camera system, and each group of the three-dimensional point cloud data and the two-dimensional image obtained is respectively aligned to the standard point cloud and the standard image through two-dimensional affine transformation, and further corrected by non-rigid transformation; According to the specific regions of interest marked on the standard image, the specific regions of interest of each component are marked on each set of the acquired two-dimensional images, and the corresponding point cloud is intercepted from the point cloud after the non-rigid transformation, and the height of each component is calculated according to the intercepted point cloud; Calculate the average height and standard deviation for the multiple sets of height data obtained by calculation; Select a sample to be tested, obtain its three-dimensional point cloud data and two-dimensional image based on the 3D camera system, record them as the point cloud to be tested and the image to be tested, align the obtained point cloud to be tested and the image to be tested to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction; According to the specific region of interest marked on the standard image, the specific region of interest of each component is marked on the acquired image to be tested, the corresponding point cloud is intercepted from the point cloud after the non-rigid transformation, and the current height of each component is calculated based on the intercepted point cloud; if the conditions are met: If the component is defective, it is a normal component. is the current height, is the average height, t is a threshold parameter, σ i is the standard deviation.

2. The method for detecting the height of electronic product components according to claim 1, characterized in that: The 3D camera system includes a high-resolution camera and two tilt-shift optical machines used in conjunction with the camera.

3. The electronic product component height detection method according to claim 1, characterized in that: The standard point cloud is denoted as (X g ,Y g ,Z g ), the standard image is P g ; The overall area of ​​interest R0 of the electronic product and the specific area of ​​interest R of each component i ,i=1,2,…,N; The component point cloud is The step of calculating the height of each component according to the component point cloud specifically includes: Step a: Point cloud of components Downsample and retain M data points to obtain Step b: Get the component point cloud after downsampling According to the three-dimensional plane formula AX+BY+CZ+D=0, the least squares method is used to fit a set of parameters A, B, C, D; the residuals of all data points are calculated. Sort all residuals, calculate the first quartile Q1, the third quartile Q3 and the interquartile range IQR = Q3-Q1; mark the data points whose residuals are outside the range [0, Q3+α×IQR] as outliers and remove them, where α is a threshold parameter; Step c, repeat step b above until there are no outlier data points, record the plane fitting parameters at this time as A', B', C', D'; use the following formula to calculate the height of the component: in, After removing outlier data points, The average value of .

4. The method for detecting the height of electronic product components according to claim 3, characterized in that: In addition, the 3D point cloud data and 2D images of K groups of defect-free electronic product samples are collected, and the collected 3D point cloud data is recorded as the test point cloud (X k ,Y k ,Z k ), k = 1, 2, ..., K, the collected two-dimensional image is called the test image P k ; The method of collecting at least one group of different standard samples, acquiring the three-dimensional point cloud data and the two-dimensional image thereof based on the 3D camera system, aligning each group of the three-dimensional point cloud data and the two-dimensional image acquired to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and further correcting by non-rigid transformation, specifically includes: For each set of test images P k , perform a two-dimensional affine transformation to align the standard image P g ; Let the two-dimensional affine transformation matrix be T; According to the two-dimensional affine transformation matrix T, for each set of test point clouds (X k ,Y k ,Z k ), k = 1, 2, ..., K, and perform the same two-dimensional affine transformation to align the standard point cloud (X g ,Y g ,Z g ), denoted as (X k1 ,Y k1 ,Z K1 (,k=1,2,…,K; The residual formula is defined as: r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy 2 +c7x 2 +c8x 2 y+c9x 3 The optimization objective function is defined as: Wherein, j=1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; the coefficient c of the residual formula r(x, y) is obtained by optimizing the optimization objective function. i ,i=0,1,…,9; After alignment, the test point cloud (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K for non-rigid transformation: X k2 =X k1 AND k2 =And k1 The transformed coordinates are labeled (X k2 ,Y k2 ,Z k2 ),k=1,2,…,K.

5. An electronic product component height detection device, characterized in that: include: The first standard sample module is used to select an electronic product without defects as a standard sample, and obtain the three-dimensional point cloud data and two-dimensional image of the standard sample based on a pre-configured 3D camera system; wherein the obtained three-dimensional point cloud data and each pixel point of the two-dimensional image correspond one to one; The three-dimensional point cloud data of the standard sample is configured as a standard point cloud; the two-dimensional image of the standard sample is configured as a standard image; and the overall area of ​​interest of the electronic product and the specific area of ​​interest of each component are marked on the standard image; For each specific region of interest of the component, a corresponding point cloud is intercepted from the standard point cloud to form a component point cloud; Calculate the height of each component according to the component point cloud; A second standard sample module is used to collect at least one group of different standard samples, obtain the three-dimensional point cloud data and the two-dimensional image thereof based on the 3D camera system, align each group of the three-dimensional point cloud data and the two-dimensional image obtained to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction; According to the specific regions of interest marked on the standard image, the specific regions of interest of each component are marked on each set of the acquired two-dimensional images, and the corresponding point cloud is intercepted from the point cloud after the non-rigid transformation, and the height of each component is calculated according to the intercepted point cloud; Calculate the average height and standard deviation for the multiple sets of height data obtained by calculation; A sample detection module for testing is used to select a sample for testing, obtain its three-dimensional point cloud data and two-dimensional image based on the 3D camera system, record them as the point cloud for testing and the image for testing, align the obtained point cloud for testing and the image for testing to the standard point cloud and the standard image respectively through two-dimensional affine transformation, and perform non-rigid transformation for further correction; According to the specific region of interest marked on the standard image, the specific region of interest of each component is marked on the acquired image to be tested, the corresponding point cloud is intercepted from the point cloud after the non-rigid transformation, and the current height of each component is calculated based on the intercepted point cloud; if the conditions are met: If the component is defective, it is a normal component. is the current height, is the average height, t is a threshold parameter, σ i is the standard deviation.

6. The electronic product component height detection device according to claim 5, characterized in that: The 3D camera system includes a high-resolution camera and two tilt-shift optical machines used in conjunction with the camera.

7. The electronic product component height detection device according to claim 5, characterized in that: The standard point cloud is denoted as (X g ,Y g ,Z g ), the standard image is P g ; The overall area of ​​interest R0 of the electronic product and the specific area of ​​interest R of each component i ,i=1,2,…,N; The component point cloud is The first standard sample module is further specifically used for: Step a: Point cloud of components Downsample and retain M data points to obtain Step b: Get the component point cloud after downsampling According to the three-dimensional plane formula AX+BY+CZ+D=0, the least squares method is used to fit a set of parameters A, B, C, D; the residuals of all data points are calculated. Sort all residuals, calculate the first quartile Q1, the third quartile Q3 and the interquartile range IQR = Q3-Q1; mark the data points whose residuals are outside the range [0, Q3+α×IQR] as outliers and remove them, where α is a threshold parameter; Step c, repeat step b above until there are no outlier data points, record the plane fitting parameters at this time as A', B', C', D'; use the following formula to calculate the height of the component: in, After removing outlier data points, The average value of .

8. The electronic product component height detection device according to claim 7, characterized in that: In addition, the 3D point cloud data and 2D images of K groups of defect-free electronic product samples are collected, and the collected 3D point cloud data is recorded as the test point cloud (X k ,Y k ,Z k ), k = 1, 2, ..., K, the collected two-dimensional image is called the test image P k ; The second standard sample module is specifically used for: For each set of test images P k , perform a two-dimensional affine transformation to align the standard image P g ; Let the two-dimensional affine transformation matrix be T; According to the two-dimensional affine transformation matrix T, for each set of test point clouds (X k ,Y k ,Z k ), k = 1, 2, ..., K, and perform the same two-dimensional affine transformation to align the standard point cloud (X g ,Y g ,Z g ), denoted as (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K; The residual formula is defined as: r(x,y)=c0+c1y+c2y 2 +c3y 3 +c4x+c5xy+c6xy 2 +c7x 2 +c8x 2 y+c9x 3 The optimization objective function is defined as: Wherein, j=1, 2, ..., m are all coordinate points in the overall region of interest R0 of the electronic product; the coefficient c of the residual formula r(x, y) is obtained by optimizing the optimization objective function. i ,i=0,1,…,9; After alignment, the test point cloud (X k1 ,Y k1 ,Z k1 ),k=1,2,…,K for non-rigid transformation: X k2 =X k1 AND k2 =And k1 The transformed coordinates are labeled (X k2 ,Y k2 ,Z k2 ),k=1,2,…,K.

9. An electronic device, characterized in that: The electronic device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for detecting the height of electronic product components as claimed in any one of claims 1 to 4.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the electronic product component height detection method according to any one of claims 1 to 4 are implemented.

Images