Method for measuring the boundaries of a three-dimensional object
By using binocular cameras and depth map processing technology, the problem of measuring the three-dimensional dimensions of moving objects was solved, the accuracy of target extraction was improved and the computational complexity was reduced, and the precise measurement of the boundaries of moving objects was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI SENGO ADVANCED TECHNOLOGY CO LTD
- Filing Date
- 2023-01-31
- Publication Date
- 2026-07-31
AI Technical Summary
Current technology cannot effectively measure the size of a moving three-dimensional object.
A depth map is generated using a binocular camera and convolutional calculation is performed. A prediction model is generated by combining thresholding and an LSTM network to extract the edge information of moving objects. The relative distance between the edges is then calculated using the prediction model.
It improves the accuracy of target extraction of objects moving on an assembly line under changing backgrounds, reduces computing power requirements, and enables easy-to-implement 3D dimension measurement.
Smart Images
Figure CN116071385B_ABST
Abstract
Description
[Technical Field]
[0001] This invention belongs to the field of testing, and specifically relates to a method for measuring the boundary of a three-dimensional object. [Background Technology]
[0002] Machine vision has been widely implemented and applied in the fields of mechanical manufacturing and inspection. Industrial cameras can be used to measure the size of three-dimensional (3D) objects. However, the industry usually requires the object to be in a static state for measurement. How to measure the three-dimensional size of moving objects is a problem that the industry urgently needs to solve. [Summary of the Invention]
[0003] The purpose of this invention is to provide a method for measuring the boundary of a three-dimensional object, in order to solve the problem that existing technologies cannot measure the three-dimensional dimensions of moving objects.
[0004] To achieve the above objectives, the three-dimensional object boundary measurement method of the present invention includes the following steps:
[0005] A depth map of the object being measured is generated using a binocular camera, and then the depth map is convolved.
[0006] Thresholding is performed on the depth map after convolution to obtain the threshold map of the relevant pixels;
[0007] The threshold map obtained from the above steps is checked to see if it contains all the pre-defined labels.
[0008] If so, the depth map data of the threshold map of the relevant pixels obtained in the previous step will be processed to obtain a list of depth level information. Then, the data in this table will be used to generate a prediction model through an LSTM network.
[0009] Multiple images are acquired using a binocular camera while the object under test is in motion to obtain the planar coordinates of the object. Then, the edge information of the object under test is extracted. The edge information of the object under test is inferred by using the prediction model formed in the above steps to obtain the depth information features extracted when the three-dimensional object under test appears. The edge information obtained by the above two methods is matched. If a match is found, the relative distance between the edges of the object under test is calculated using the determined image.
[0010] Based on the above main features, when performing convolution calculations, the 3D input tensor is first converted into columns of a 2D matrix, and then the matrix is multiplied by the convolution weights to obtain the convolution result. The convolution weights are determined through multiple tests.
[0011] Based on the above main features, thresholding is performed on the depth map data after convolution to obtain a threshold map of relevant pixels. This is achieved by setting a threshold and filtering out pixels that are less than or greater than the threshold by a certain amount.
[0012] Based on the above main features, the method for calculating the edge information is as follows: the depth information of the object to be measured is input into the prediction model, with 10 sets of information as one input. The prediction model makes two judgments based on the input: "is an edge feature" and "is not an edge feature". When the judgment is "is an edge feature", the original 10 sets of input information are extracted and the coordinates are transformed according to the pre-calibrated coordinates to obtain the edge information of the object to be measured.
[0013] Compared with existing technologies, the method of this invention can significantly improve the accuracy of target extraction of objects moving on an assembly line under changing backgrounds, and reduces the computing power requirements compared with traditional methods, thus making it easier to implement. [Attached Image Description]
[0014] Figure 1 A schematic diagram illustrating the steps of implementing the three-dimensional object boundary measurement method of the present invention.
[0015] Figure 2 A schematic diagram illustrating the convolution calculation used in the three-dimensional object boundary measurement method of the present invention.
Detailed Implementation Methods
[0016] Please see Figure 1 The diagram shown illustrates the steps of implementing the three-dimensional object boundary measurement method of the present invention. The three-dimensional object boundary measurement method of the present invention includes the following steps:
[0017] Step 1: Use a stereo camera to generate a depth map of the object being measured, and then perform convolution calculations on the depth map. During the convolution calculation, as follows... Figure 2 As shown, the 3D input tensor is first converted into columns of a 2D matrix, and then the matrix is multiplied by convolution weights a1 to a8 to obtain the convolution result. The convolution weights a1 to a8 are determined through multiple tests.
[0018] Step 2: Threshold the depth map data after convolution to obtain a threshold map of relevant pixels. Specifically, a threshold is set to filter out pixels that are less than a certain threshold (such as objects between the measured 3D object and the stereo camera) or pixels that are greater than a certain threshold (such as objects behind the measured 3D object, such as the background), thereby obtaining a threshold map of relevant pixels.
[0019] Step 3: Check whether the threshold map obtained in the previous step meets the requirements based on the pre-set labels. Specifically, the threshold map is verified to meet the requirements based on the labels defined during manual on-site calibration (such as selecting some data as labels from the depth map obtained by the standard test object in a test environment without background and after convolution processing). If the relevant pixel area in the threshold map contains all the pre-set labels, then the threshold processing ends; otherwise, the threshold is adjusted and the process returns to Step 2 for reprocessing.
[0020] Step 4: Perform depth processing on the data obtained in Step 3 to obtain a list of depth level information. Then, use the data in this list to generate a prediction model through an LSTM network model. Use this prediction model to extract the specific depth information when the measured 3D object appears.
[0021] Step 5: When the object under test is moving, multiple images are acquired using a binocular camera. Combined with the two-dimensional images of the X and Y axes, the planar coordinates of the object under test are extracted by a convolutional neural network (in specific implementations, this position data is converted into planar coordinate data through pre-calibration and coordinate transformation. Calibration involves setting some specific positions for coordinate transformation between subsequent multiple images, which is described in many existing technologies and will not be elaborated here). The edge information of the object under test (referring to the edge position of the object to be inspected) is extracted. Then, the edge information of the object under test is inferred by using the prediction model formed in Step 4 to obtain the depth information features extracted when the three-dimensional object under test appears. The edge information obtained by the two methods is matched. When the match is successful, the relative distance between the edges of the object under test is calculated using the determined image (determined to be compared with the image to be calculated last, such as the image at a 90-degree angle to the binocular camera), thus completing the detection task. In specific implementation, the above-mentioned edge information estimation method is as follows: the depth information of the object to be measured is input into the prediction model formed in step four, with 10 sets of information as one input. The prediction model makes two judgments based on the input: "is an edge feature" and "is not an edge feature". When the judgment is "is an edge feature", the original 10 sets of input information are extracted and the coordinates are transformed according to the pre-calibrated coordinates to obtain the edge information of the object to be measured.
[0022] As can be seen from the above, the present invention first establishes a prediction model through steps one to four. This model is formed under static conditions by adjusting the threshold of the object under test and a binocular camera. Then, in step five, this prediction model is used to perform actual measurements on the moving object under test. In addition, if different objects are being tested or in different test environments (such as changing the test position or background), a new prediction model needs to be regenerated using steps one to four.
[0023] To facilitate understanding of the technical solution of this invention, the following uses the measurement of the spacing between 3D twisted cooling tubes as an example to illustrate the specific application process of this invention. During inspection, the three-dimensional product (such as a 3D twisted cooling tube) moves along the X-axis on the production line. The testing device, including the binocular camera, does not stop during inspection and shooting (i.e., the product remains in motion along the X-axis). Simultaneously, to reduce computational load, the original images acquired by the binocular camera need to be processed. Specifically, this is done by using focused convolution that ignores irrelevant pixels (because 2D image data is large, ignoring irrelevant pixels, such as background pixels, reduces the amount of data processing).
[0024] The distinction between relevant and irrelevant pixels is achieved using a depth map. In the depth map, a value represents the depth level of a pixel, indicating its distance from the stereo camera (i.e., the distance of objects in front of the stereo camera from the camera itself). First, a depth map is generated using the stereo camera, and then this depth map is input into a convolutional computation, such as... Figure 2 As shown, a portion of the 3D input tensor is converted into columns of a 2D matrix (where each column corresponds to a 3*3 matrix in the 3D input tensor). The matrix is then multiplied by convolution weights a1 to a8 to obtain the convolution result. The calculated depth map is then thresholded to filter out irrelevant pixels that are too short (e.g., pixels of objects between the 3D twisted cooling pipe and the stereo camera) or too long (e.g., pixels of the background behind the 3D twisted cooling pipe). This process yields a preliminary threshold map of relevant and irrelevant pixels.
[0025] Next, the appropriateness of the threshold map is verified based on the labels defined during manual field calibration (specifically, using a standard 3D twisted cooling pipe as the standard model without background, and manually setting some measurement markers). If the relevant pixel area in the threshold map contains all the labels defined during manual field calibration, then the threshold processing is complete; otherwise, the threshold is expanded, and the verification and generation of a threshold map that meets the requirements are repeated.
[0026] The data obtained in the previous step is then processed to obtain a list of depth level information. The data in this list is then used to generate a prediction model through an LSTM (Long Short-Term Memory) network model. This prediction model can then be used to extract the specific depth value features when the 3D twisted cooling pipe appears.
[0027] For multiple images captured by the binocular camera, the edge information of the 3D twisted cooling tube is extracted by combining the X-axis and Y-axis 2D images with the planar feature coordinates of the 3D twisted cooling tube extracted by the convolutional neural network. Then, it is matched with the depth information features extracted in the previous step. If the match is successful, it is considered that the two are the same tube (thus determining the image to be used, such as the image of the 3D twisted cooling tube at a 90-degree angle to the binocular camera, i.e., the image captured when the 3D twisted cooling tube is directly in front of the binocular camera). Then, the relative distance between the edges of the tube in the 2D image (calculated in the X, Y, and Z axes) is calculated to complete the gap defect detection task.
[0028] Compared with existing technologies, the method of this invention can significantly improve the accuracy of target extraction of objects moving on an assembly line in a changing background, and reduces the required computing power compared with traditional methods, thus making it easier to implement.
[0029] It is understood that those skilled in the art can make equivalent substitutions or modifications to the technical solution and inventive concept of the present invention, and all such substitutions or modifications should fall within the protection scope of the appended claims.
Claims
1. A method of measuring a boundary of a three-dimensional object, characterized by The three-dimensional object boundary measurement method includes the following steps: A depth map of the object being measured is generated using a binocular camera, and then the depth map is convolved. Thresholding is performed on the depth map after convolution to obtain the threshold map of the relevant pixels; The threshold map obtained from the above steps is checked to see if it contains all the pre-defined labels. If so, the depth map data of the threshold map of the relevant pixels obtained in the previous step will be processed to obtain a list of depth level information. Then, the data in this table will be used to generate a prediction model through an LSTM network. Multiple images are acquired using a binocular camera while the object under test is in motion, and the planar coordinates of the object are obtained. Then, the edge information of the object is extracted. The edge information of the object is inferred by using the depth information features extracted when the 3D object appears using the prediction model formed in the above steps. The edge information obtained by the above two methods is matched. If a match is found, the relative distance between the edges of the object under test is calculated using the determined image. The method for inferring the edge information is as follows: the depth information of the object under test is input into the prediction model, with 10 sets of information as one input. The prediction model makes two judgments based on the input: "is an edge feature" and "is not an edge feature". When the judgment is "is an edge feature", the original 10 sets of information are extracted and the coordinates are transformed according to the pre-calibrated coordinates to obtain the edge information of the object under test.
2. The three-dimensional object boundary measurement method according to claim 1, wherein: When performing convolution calculations, the 3D input tensor is first converted into columns of a 2D matrix, and then the matrix is multiplied by the convolution weights to obtain the convolution result. The convolution weights are determined through multiple tests.
3. The three-dimensional object boundary measurement method according to claim 2, wherein: Thresholding is performed on the depth map after convolution to obtain a threshold map of relevant pixels. This is done by setting a threshold and filtering out pixels that are less than or greater than the threshold by a certain amount.