A Method and System for Measuring Profile Cross-Section Dimensions Based on 2D Images and 3D Point Clouds

Abstract

This invention discloses a method and system for measuring the cross-sectional dimensions of profiles based on two-dimensional images and three-dimensional point clouds, relating to the technical field of profile inspection. The method includes acquiring two-dimensional and depth images of the profile cross-section to be measured; generating three-dimensional point cloud data based on the depth image; preprocessing the two-dimensional image and three-dimensional point cloud data respectively to obtain preprocessed two-dimensional images and three-dimensional point cloud data; inputting these preprocessed images into a trained burr detection model to obtain burr regions in the two-dimensional image; removing the burr regions from the two-dimensional image to obtain a burr-free image of the profile cross-section to be measured; calibrating the burr-free image of the profile cross-section to obtain a calibrated image of the profile cross-section to be measured; and measuring dimensional parameters on the calibrated image of the profile cross-section to be measured. This invention can replace the actual deburring process, shortening the time cost of waiting for manual polishing, eliminating errors in the imaging and projection process, ensuring that dimensional parameters are measured on an interference-free image, and significantly improving the accuracy of inspection.

Description

Technical Field

[0001] This invention relates to the technical field of profile inspection, and more specifically, to a method and system for measuring profile cross-sectional dimensions based on two-dimensional images and three-dimensional point clouds. Background Technology

[0002] In the profile manufacturing process, materials are extruded and cut to form specific cross-sectional shapes. To ensure these cross-sectional dimensions meet design specifications and that the extrusion dies do not experience wear that could affect product quality, accurate measurement of the profile cross-section is a crucial step. However, in actual production, the cut profile cross-sections often contain burrs, which significantly interfere with the measurement of cross-sectional dimensions.

[0003] In existing technologies, the method for measuring the cross-sectional dimensions of profiles typically involves manually grinding away burrs before measurement. This method has the following problems: manual grinding is time-consuming and labor-intensive, resulting in high labor costs and making it difficult to meet the needs of modern industrial production. Furthermore, manual measurement relies on the operator's experience and judgment, which is highly subjective and prone to inconsistencies due to individual differences or misjudgments, thus reducing the reliability of the inspection. Summary of the Invention

[0004] One objective of this invention is to provide a method for measuring the cross-sectional dimensions of profiles based on two-dimensional images and three-dimensional point clouds, so as to overcome the above-mentioned defects in the inability to accurately measure the cross-sectional dimensions of profiles due to the influence of burrs in industrial production; the second objective is to provide a system for measuring the cross-sectional dimensions of profiles based on two-dimensional images and three-dimensional point clouds.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] This invention provides a method for measuring the cross-sectional dimensions of profiles based on two-dimensional images and three-dimensional point clouds, including:

[0007] Acquire two-dimensional and depth images of the cross-section of the profile to be measured, and generate three-dimensional point cloud data based on the depth image;

[0008] The two-dimensional image and the three-dimensional point cloud data are preprocessed respectively to obtain the preprocessed two-dimensional image and the three-dimensional point cloud data.

[0009] The preprocessed two-dimensional image and three-dimensional point cloud data are input into the trained burr detection model to obtain the burr region in the two-dimensional image.

[0010] Remove the burr area from the two-dimensional image to obtain a burr-free image of the profile cross-section to be measured;

[0011] The burr-free image of the profile section to be measured is corrected to obtain a corrected image of the profile section to be measured, and the dimensional parameters are measured on the corrected image of the profile section to be measured.

[0012] Preferably, acquiring the two-dimensional image and depth image of the cross-section of the profile to be measured includes:

[0013] Fix the cross-section of the profile to be measured, use a two-dimensional industrial camera to acquire a two-dimensional image of the cross-section of the profile to be measured, use a three-dimensional depth camera to acquire a depth image of the cross-section of the profile to be measured, and save the pose information of the three-dimensional depth camera and the two-dimensional industrial camera.

[0014] Preferably, generating three-dimensional point cloud data based on the depth image includes:

[0015] The formula for generating 3D point cloud data from a depth image is:

[0016]

[0017] In the formula, X i Y i Z i U represents the 3D coordinates of the i-th point in the 3D point cloud data. i v i The depth(u) represents the two-dimensional coordinates of the i-th pixel in the depth image. i v i ) represents the depth value of the i-th pixel, c x c y f represents the optical center coordinates of the 3D depth camera. x f y These represent the focal lengths of the 3D depth camera in the x and y directions, respectively.

[0018] Preferably, the pose information of the three-dimensional depth camera includes a three-dimensional camera rotation matrix and a three-dimensional camera translation vector;

[0019] Based on the relative positional relationship between the 3D depth camera and the 2D industrial camera, the pose information of the 2D industrial camera is calculated, including the 2D camera rotation matrix and the 2D camera translation vector; the calculation formula is as follows:

[0020] R 2D =R rel ·R 3D

[0021] t 2D =R rel ·t 3D +t rel

[0022] In the formula, R 3D , t 3DLet R represent the rotation matrix and translation vector of the 3D camera, respectively. rel , t rel R represents the rotation matrix and translation vector of the relative positions of the 3D depth camera and the 2D industrial camera. 2D , t 2D These represent the rotation matrix and translation vector of the two-dimensional camera, respectively.

[0023] Preferably, the two-dimensional image and the three-dimensional point cloud data are preprocessed respectively to obtain preprocessed two-dimensional image and three-dimensional point cloud data, including:

[0024] The two-dimensional image is sequentially subjected to grayscale conversion, noise reduction, adaptive threshold binarization, and connected component analysis to obtain a preprocessed two-dimensional image.

[0025] The 3D point cloud data is denoised to obtain preprocessed 3D point cloud data.

[0026] Preferably, the calculation formula for grayscale processing is as follows:

[0027]

[0028] Where I(x,y) represents the grayscale value of the pixel at coordinate (x,y), and R(x,y), G(x,y), and B(x,y) represent the red, green, and blue components of the pixel at coordinate (x,y) in the two-dimensional image, respectively.

[0029] Preferably, a median filtering algorithm is used for noise reduction. The calculation formula for median filtering is as follows:

[0030] I′(x,y)=median(I(x+i,y+j)),-k≤i,j≤k

[0031] In the formula, I′(x, y) represents the pixel value of the pixel at coordinate (x, y) after denoising, I(x+i, y+j) represents the pixel value in the (2k+1)×(2k+1) window centered at the pixel at (x, y) in the grayscale 2D image, and median represents the median of all pixel values ​​in the window.

[0032] It is understandable that noise will inevitably exist during the image acquisition process, such as noise caused by uneven lighting. In order to remove this noise while preserving the edge information in the image, the median filtering algorithm is used for noise reduction. The principle of median filtering is to replace the value of each pixel in the image with the median value of its neighboring pixels, thereby effectively removing isolated noise points without blurring the edges.

[0033] Preferably, the adaptive threshold binarization process includes:

[0034] Calculate the adaptive threshold:

[0035]

[0036] In the formula, T(x, y) represents the adaptive threshold of the pixel at coordinates (x, y), ω represents the size of the binarization window, and C represents a constant used to adjust the degree of separation between the foreground and the background;

[0037] Binarization is performed based on an adaptive threshold:

[0038]

[0039] In the formula, I_b(x, y) represents the pixel value after binarization at coordinates (x, y), 0 represents the background pixel, and 255 represents the foreground pixel.

[0040] Preferably, the connected component analysis includes:

[0041] The breadth-first search algorithm is used to label the connected components of the binarized 2D image and identify all connected components.

[0042] Calculate the number of pixels in each connected component, and calculate the area of ​​each connected component based on the number of pixels and the pixel values;

[0043] The connected component with the largest area is retained, and the remaining connected components are set as the background to obtain the preprocessed two-dimensional image.

[0044] Preferably, the formula for calculating the area of ​​a connected region is as follows:

[0045]

[0046] Among them, A i Region represents the area of ​​the i-th connected region. i This represents the set of coordinates of all foreground pixels in the i-th connected component.

[0047] Preferably, the three-dimensional point cloud data is subjected to denoising processing to obtain preprocessed three-dimensional point cloud data, including:

[0048] For each point in the three-dimensional point cloud data, search for a set of neighboring points within a preset radius;

[0049] The formula for calculating the mean distance between a point and its nearest neighbors is as follows:

[0050]

[0051] In the formula, d mean (P i ) represents the mean distance between the i-th point and its neighboring point set, N.i Let P represent the set of neighboring points of the i-th point. j Let j represent the j-th point in the nearest neighbor set, and ||*|| denotes calculating the Euclidean distance.

[0052] Traverse each point in the three-dimensional point cloud data and obtain the mean distance between each point and its neighboring point set;

[0053] Calculate the mean and standard deviation of the global point cloud based on the mean distance between each point and its neighboring point set;

[0054] The threshold is calculated based on the mean and standard deviation of the global point cloud, using the following formula:

[0055] τ=μ+α·f

[0056] In the formula, τ represents the threshold, μ represents the mean of the global point cloud, f represents the standard deviation of the global point cloud, and f represents the scaling factor.

[0057] The average distance between each point and its neighboring points is compared with the threshold. If the average distance between the point and its neighboring points is greater than the threshold, the point is removed as an outlier; otherwise, the point is retained.

[0058] Preferably, obtaining the trained burr detection model includes:

[0059] Acquire three-dimensional point cloud data and corresponding two-dimensional images of several profile cross sections;

[0060] The three-dimensional point cloud data and corresponding two-dimensional images of all the profile cross sections are normalized to obtain the normalized three-dimensional point cloud data and corresponding two-dimensional images of the profile cross sections.

[0061] Each point in the normalized 3D point cloud data of all the profile cross sections is labeled with a category to obtain 3D point cloud data of the profile cross sections with category labels;

[0062] To construct a burr detection model, input 3D point cloud data of profile cross-section with category labels and corresponding 2D images, set a loss function for optimization training, and save the corresponding model parameters when the loss function value reaches the minimum value to obtain the trained burr detection model.

[0063] Preferably, the burr detection model includes a two-dimensional feature extraction unit, a first multilayer perceptron, a first three-dimensional feature downsampling unit, a first feature fusion unit, a second three-dimensional feature downsampling unit, a second feature fusion unit, a third three-dimensional feature downsampling unit, a third feature fusion unit, a fourth three-dimensional feature downsampling unit, a fourth feature fusion unit, a second multilayer perceptron, a first three-dimensional feature upsampling unit, a second three-dimensional feature upsampling unit, a third three-dimensional feature upsampling unit, a fourth three-dimensional feature upsampling unit, and a third multilayer perceptron;

[0064] The output of the two-dimensional feature extraction unit is connected to the first input of the first feature fusion unit, the second feature fusion unit, the third feature fusion unit, and the fourth feature fusion unit, respectively.

[0065] The output of the first multilayer perceptron is connected to the input of the first three-dimensional feature downsampling unit. The output of the first three-dimensional feature downsampling unit is connected to the second input of the first feature fusion unit. The output of the first feature fusion unit is connected to the input of the second three-dimensional feature downsampling unit. The output of the second three-dimensional feature downsampling unit is connected to the second input of the second feature fusion unit. The output of the second feature fusion unit is connected to the input of the third three-dimensional feature downsampling unit. The output of the third three-dimensional feature downsampling unit is connected to the second input of the third feature fusion unit. The output of the third feature fusion unit is connected to the input of the fourth three-dimensional feature downsampling unit. The output of the fourth three-dimensional feature downsampling unit is connected to the second input of the fourth feature fusion unit. The output of the fourth feature fusion unit is connected to the input of the second multilayer perceptron.

[0066] The output of the second multilayer perceptron is connected to the first input of the first three-dimensional feature upsampling unit, the output of the first three-dimensional feature upsampling unit is connected to the first input of the second three-dimensional feature upsampling unit, the output of the second three-dimensional feature upsampling unit is connected to the first input of the third three-dimensional feature upsampling unit, the output of the third three-dimensional feature upsampling unit is connected to the first input of the fourth three-dimensional feature upsampling unit, and the output of the fourth three-dimensional feature upsampling unit is connected to the input of the third multilayer perceptron.

[0067] The output of the first feature fusion unit is also connected to the second input of the fourth three-dimensional feature upsampling unit, the output of the second feature fusion unit is also connected to the second input of the third three-dimensional feature upsampling unit, the output of the third feature fusion unit is also connected to the second input of the second three-dimensional feature upsampling unit, and the output of the fourth feature fusion unit is also connected to the second input of the first three-dimensional feature upsampling unit.

[0068] Preferably, the first three-dimensional feature downsampling unit, the second three-dimensional feature downsampling unit, the third three-dimensional feature downsampling unit, and the fourth three-dimensional feature downsampling unit have the same structure, each including a farthest point sampling layer, a K-nearest neighbor grouping layer, a neighborhood max pooling layer, a first fully connected layer, a first attention layer, and a second fully connected layer connected in sequence; the output of the neighborhood max pooling layer is also connected to the output of the second fully connected layer.

[0069] Preferably, the first feature fusion unit, the second feature fusion unit, the third feature fusion unit and the fourth feature fusion unit have the same structure, each including a first cross attention layer, a first self attention layer, a second cross attention layer, a second self attention layer and a third cross attention layer connected in sequence;

[0070] The output of the two-dimensional feature extraction unit is connected to the first input of the first cross-attention layer and the second cross-attention layer, respectively. The output of the second fully connected layer is connected to the second input of the first cross-attention layer. The output of the first self-attention layer is also connected to the first input of the third cross-attention layer.

[0071] Preferably, the first three-dimensional feature upsampling unit, the second three-dimensional feature upsampling unit, the third three-dimensional feature upsampling unit, and the fourth three-dimensional feature upsampling unit have the same structure, each including a third fully connected layer, a fourth fully connected layer, a trilinear interpolation layer, an element-wise addition layer, a fifth fully connected layer, a third self-attention layer, and a sixth fully connected layer;

[0072] The fourth fully connected layer, the trilinear interpolation layer, the element-wise addition layer, the fifth fully connected layer, the third self-attention layer, and the sixth fully connected layer are connected in sequence.

[0073] The input of the third fully connected layer is connected to the output of the third cross-attention layer, the output of the third fully connected layer is connected to the second input of the element-wise addition layer, and the output of the element-wise addition layer is also connected to the output of the sixth fully connected layer.

[0074] Preferably, measuring dimensional parameters on a calibrated image of the profile section to be measured includes:

[0075] Edge detection is performed on the corrected image of the profile section to be measured using an edge detection algorithm to obtain a set of contour lines of the profile section to be measured;

[0076] Arbitrarily select a preset measurement location, and select two corresponding target contour lines from the set of contour lines of the profile section to be measured;

[0077] Calculate the pixel distance between the two target contour lines in the corrected image of the profile section to be measured;

[0078] Based on the preset scaling factor and the pixel distance, the actual distance between the two target contour lines is calculated, and the actual distance is used as the actual size of the preset measurement part.

[0079] By traversing the preset measurement locations, the actual dimensions of all preset measurement locations are obtained, forming the dimensional parameters of the calibration image of the profile section to be measured.

[0080] The present invention also provides a profile cross-sectional dimension measurement system based on two-dimensional images and three-dimensional point clouds, for implementing the above-mentioned method, including:

[0081] The data acquisition module is used to acquire two-dimensional and depth images of the profile cross-section to be measured, and to generate three-dimensional point cloud data based on the depth image.

[0082] The data processing module is used to preprocess the two-dimensional image and the three-dimensional point cloud data respectively, and obtain the preprocessed two-dimensional image and the three-dimensional point cloud data accordingly.

[0083] The burr detection module is used to input the preprocessed two-dimensional image and three-dimensional point cloud data into the trained burr detection model to obtain the burr region in the two-dimensional image.

[0084] The burr removal module is used to remove the burr area from the two-dimensional image to obtain a burr-free image of the profile cross-section to be measured.

[0085] The dimension measurement module is used to correct the burr-free image of the profile section to be measured, obtain a corrected image of the profile section to be measured, and measure the dimension parameters on the corrected image of the profile section to be measured.

[0086] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0087] This invention first acquires two-dimensional and depth images of the profile cross-section to be measured, and generates three-dimensional point cloud data based on the depth image. By fusing the planar geometric information of the two-dimensional image and the spatial geometric information of the three-dimensional point cloud, it ensures that features of different dimensions can be fully utilized in the subsequent burr detection process, thereby improving detection accuracy. Next, the two-dimensional image and three-dimensional point cloud data are preprocessed to remove irrelevant environmental information. The preprocessed two-dimensional image and three-dimensional point cloud data are then input into a trained burr detection model for automatic burr detection, obtaining the burr region in the two-dimensional image. This replaces the actual deburring process, reducing errors caused by operator experience and subjective judgment, reducing the time cost of manual polishing, and improving the consistency, reliability, and efficiency of burr detection. Finally, the detected burr region is removed from the two-dimensional image to generate a burr-free image, which is then corrected to obtain a calibrated image of the profile cross-section to be measured. Dimensional parameters are measured on this calibrated image, eliminating errors in the imaging and projection process and ensuring that dimensional parameters are obtained based on the interference-free profile cross-section contour, thereby significantly improving detection accuracy and reliability. Attached Figure Description

[0088] Figure 1 This is a flowchart of a method for measuring profile cross-sectional dimensions based on two-dimensional images and three-dimensional point clouds, as described in Example 1.

[0089] Figure 2 This is a schematic diagram of the burr detection model described in Example 2;

[0090] Figure 3 This is a schematic diagram of the structure of the first three-dimensional feature downsampling unit, the second three-dimensional feature downsampling unit, the third three-dimensional feature downsampling unit, and the fourth three-dimensional feature downsampling unit described in Embodiment 2;

[0091] Figure 4 This is a schematic diagram of the structure of the first feature fusion unit, the second feature fusion unit, the third feature fusion unit, and the fourth feature fusion unit described in Embodiment 2;

[0092] Figure 5 This is a schematic diagram of the structure of the first three-dimensional feature upsampling unit, the second three-dimensional feature upsampling unit, the third three-dimensional feature upsampling unit, and the fourth three-dimensional feature upsampling unit described in Embodiment 2;

[0093] Figure 6 This is a schematic diagram of the profile cross-sectional dimension measurement system based on two-dimensional images and three-dimensional point clouds as described in Example 3. Detailed Implementation

[0094] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0095] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0096] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0097] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0098] Example 1

[0099] This embodiment provides a method for measuring the cross-sectional dimensions of profiles based on two-dimensional images and three-dimensional point clouds, such as... Figure 1 As shown, it includes:

[0100] S1: Acquire a two-dimensional image and a depth image of the profile section to be measured, and generate three-dimensional point cloud data based on the depth image;

[0101] S2: Preprocess the two-dimensional image and the three-dimensional point cloud data respectively to obtain the preprocessed two-dimensional image and the three-dimensional point cloud data.

[0102] S3: Input the preprocessed two-dimensional image and three-dimensional point cloud data into the trained burr detection model to obtain the burr region in the two-dimensional image;

[0103] S4: Remove the burr area from the two-dimensional image to obtain a burr-free image of the profile section to be measured;

[0104] S5: Correct the burr-free image of the profile section to be measured to obtain a corrected image of the profile section to be measured, and measure the dimensional parameters on the corrected image of the profile section to be measured.

[0105] In the specific implementation process, this embodiment first acquires a two-dimensional image and a depth image of the profile section to be measured, and generates three-dimensional point cloud data based on the depth image. By fusing the planar geometric information of the two-dimensional image and the spatial geometric information of the three-dimensional point cloud, it ensures that the features of different latitudes can be fully utilized in the subsequent burr detection process, thereby improving the detection accuracy. Then, the two-dimensional image and three-dimensional point cloud data are preprocessed to remove irrelevant environmental information. The preprocessed two-dimensional image and three-dimensional point cloud data are input into a trained burr detection model to automatically perform burr detection, obtaining the burr area in the two-dimensional image. This replaces the actual deburring process, reducing errors caused by differences in operator experience and subjective judgment, reducing the time cost of waiting for manual polishing, and improving the consistency, reliability, and efficiency of burr detection. Finally, the detected burr area is removed from the two-dimensional image to generate a burr-free image, which is then corrected to obtain a corrected image of the profile section to be measured. The dimensional parameters are measured on the corrected image of the profile section to be measured, eliminating errors in the imaging and projection process, and ensuring that the dimensional parameters are obtained based on the profile section contour without interference, thereby significantly improving the accuracy and reliability of the detection.

[0106] Example 2

[0107] This embodiment provides a method for measuring the cross-sectional dimensions of profiles based on two-dimensional images and three-dimensional point clouds, including:

[0108] S1: Acquire a two-dimensional image and a depth image of the profile section to be measured, and generate three-dimensional point cloud data based on the depth image;

[0109] The acquisition of the two-dimensional image and depth image of the cross-section of the profile to be measured includes:

[0110] Fix the cross-section of the profile to be measured, use a two-dimensional industrial camera to acquire a two-dimensional image of the cross-section of the profile to be measured, use a three-dimensional depth camera to acquire a depth image of the cross-section of the profile to be measured, and save the pose information of the three-dimensional depth camera and the two-dimensional industrial camera.

[0111] The pose information of the 3D depth camera includes the 3D camera rotation matrix and the 3D camera translation vector.

[0112] Based on the relative positional relationship between the 3D depth camera and the 2D industrial camera, the pose information of the 2D industrial camera is calculated, including the 2D camera rotation matrix and the 2D camera translation vector; the calculation formula is as follows:

[0113] R 2D =R rel ·R 3D

[0114] t 2D =R rel ·t 3D +t rel

[0115] In the formula, R 3D , t 3D Let R represent the rotation matrix and translation vector of the 3D camera, respectively. rel , t rel R represents the rotation matrix and translation vector of the relative positions of the 3D depth camera and the 2D industrial camera. 2D , t 2D These represent the rotation matrix and translation vector of the two-dimensional camera, respectively.

[0116] The formula for generating 3D point cloud data from a depth image is:

[0117]

[0118] In the formula, X i Y i Z i U represents the 3D coordinates of the i-th point in the 3D point cloud data. i v i The depth(u) represents the two-dimensional coordinates of the i-th pixel in the depth image. i v i ) represents the depth value of the i-th pixel, c x c y f represents the optical center coordinates of the 3D depth camera. x f y These represent the focal lengths of the 3D depth camera in the x and y directions, respectively.

[0119] Understandably, depth images are captured by 3D depth cameras, such as LiDAR or structured light cameras, where each pixel contains the depth value from that point to the camera. By combining these depth values ​​with the camera's intrinsic and extrinsic parameters, the depth image can be converted into 3D point cloud data. Through this conversion process, each pixel in the 2D depth image is mapped to a point in 3D space, thereby generating 3D point cloud data and obtaining the camera's pose information in 3D space: rotation matrix R. 3D Translation vector t 3D .

[0120] Pose information estimation for 2D industrial cameras is usually based on pose information from 3D depth cameras, combined with the relative positional relationship between the 3D depth camera and the 2D industrial camera, including the relative position rotation matrix and the relative position translation vector, to derive the pose information of the 2D industrial camera.

[0121] S2: Preprocess the two-dimensional image and the three-dimensional point cloud data respectively to obtain the preprocessed two-dimensional image and the three-dimensional point cloud data.

[0122] The two-dimensional image is sequentially subjected to grayscale processing, noise reduction processing, adaptive threshold binarization processing, and connected component analysis to obtain a preprocessed two-dimensional image.

[0123] S21: Grayscale processing: The specific calculation formula for the grayscale processing is as follows:

[0124]

[0125] Where I(x,y) represents the grayscale value of the pixel at coordinate (x,y), and R(x,y), G(x,y), and B(x,y) represent the red, green, and blue components of the pixel at coordinate (x,y) in the two-dimensional image, respectively.

[0126] Understandably, two-dimensional images are usually color images containing three color channels (R, G, B). To simplify calculations, the color image first needs to be converted into a grayscale image.

[0127] S22: Denoising Processing: Median filtering algorithm is used for denoising. The calculation formula for median filtering is:

[0128] I′(x,y)=median(I(x+i,y+j)),-k≤i,j≤k

[0129] In the formula, I′(x, y) represents the pixel value of the pixel at coordinate (x, y) after denoising, I(x+i, y+j) represents the pixel value in the (2k+1)×(2k+1) window centered at the pixel at (x, y) in the grayscale 2D image, and median represents the median of all pixel values ​​in the window.

[0130] It is understandable that noise will inevitably exist during the image acquisition process, such as noise caused by uneven lighting. In order to remove this noise while preserving the edge information in the image, the median filtering algorithm is used for noise reduction. The principle of median filtering is to replace the value of each pixel in the image with the median value of its neighboring pixels, thereby effectively removing isolated noise points without blurring the edges.

[0131] S23: Adaptive threshold binarization processing, including:

[0132] Calculate the adaptive threshold:

[0133]

[0134] In the formula, T(x, y) represents the adaptive threshold of the pixel at coordinates (x, y), ω represents the size of the binarization window, and C represents a constant used to adjust the degree of separation between the foreground and the background;

[0135] Binarization is performed based on an adaptive threshold:

[0136]

[0137] In the formula, I_b(x, y) represents the pixel value after binarization at coordinates (x, y), 0 represents the background pixel, and 255 represents the foreground pixel.

[0138] Understandably, the purpose of binarization is to divide the pixels in an image into foreground and background, with the foreground being the cross-section of the profile. The adaptive threshold binarization method dynamically calculates the threshold based on the local region of each pixel, making it suitable for processing images with uneven lighting.

[0139] S24: Connectivity analysis, including:

[0140] The breadth-first search algorithm is used to label the connected components of the binarized 2D image and identify all connected components.

[0141] Calculate the number of pixels in each connected component, and calculate the area of ​​each connected component based on the number of pixels and the pixel values;

[0142] The connected component with the largest area is retained, and the remaining connected components are set as the background to obtain the preprocessed two-dimensional image.

[0143] Preferably, the formula for calculating the area of ​​a connected region is as follows:

[0144]

[0145] Among them, A i Region represents the area of ​​the i-th connected region. i This represents the set of coordinates of all foreground pixels in the i-th connected component;

[0146] Understandably, after adaptive threshold binarization, the image will contain multiple connected regions, where the largest connected region is usually the profile cross-section, and other smaller connected regions may be noise or background objects. In order to retain only the profile cross-section region, connected component analysis is required.

[0147] The 3D point cloud data is denoised to obtain preprocessed 3D point cloud data, including:

[0148] For each point in the three-dimensional point cloud data, search for a set of neighboring points within a preset radius;

[0149] The formula for calculating the mean distance between a point and its nearest neighbors is as follows:

[0150]

[0151] In the formula, d mean (P i ) represents the mean distance between the i-th point and its neighboring point set, N. i Let P represent the set of neighboring points of the i-th point. j Let j represent the j-th point in the nearest neighbor set, and ||*|| denotes calculating the Euclidean distance.

[0152] Traverse each point in the three-dimensional point cloud data and obtain the mean distance between each point and its neighboring point set;

[0153] Calculate the mean and standard deviation of the global point cloud based on the mean distance between each point and its neighboring point set;

[0154] The threshold is calculated based on the mean and standard deviation of the global point cloud, using the following formula:

[0155] τ=μ+α·σ

[0156] In the formula, τ represents the threshold, μ represents the mean of the global point cloud, σ represents the standard deviation of the global point cloud, and σ represents the scaling factor.

[0157] The average distance between each point and its neighboring points is compared with the threshold. If the average distance between the point and its neighboring points is greater than the threshold, the point is removed as an outlier; otherwise, the point is retained.

[0158] S3: Input the preprocessed two-dimensional image and three-dimensional point cloud data into the trained burr detection model to obtain the burr region in the two-dimensional image;

[0159] Obtaining the trained spur detection model includes:

[0160] Acquire three-dimensional point cloud data and corresponding two-dimensional images of several profile cross sections;

[0161] The three-dimensional point cloud data and corresponding two-dimensional images of all the profile cross sections are normalized to obtain the normalized three-dimensional point cloud data and corresponding two-dimensional images of the profile cross sections.

[0162] Each point in the normalized 3D point cloud data of all the profile cross sections is labeled with a category to obtain 3D point cloud data of the profile cross sections with category labels;

[0163] To construct a burr detection model, input 3D point cloud data of profile cross-section with category labels and corresponding 2D images, set a loss function for optimization training, and save the corresponding model parameters when the loss function value reaches the minimum value to obtain the trained burr detection model.

[0164] like Figure 2 As shown, the burr detection model includes a two-dimensional image encoder, a three-dimensional point cloud encoder, and a three-dimensional point cloud decoder. The two-dimensional image encoder includes a two-dimensional feature extraction unit; the three-dimensional point cloud encoder includes a first multilayer perceptron, a first three-dimensional feature downsampling unit, a first feature fusion unit, a second three-dimensional feature downsampling unit, a second feature fusion unit, a third three-dimensional feature downsampling unit, a third feature fusion unit, a fourth three-dimensional feature downsampling unit, and a fourth feature fusion unit; the three-dimensional point cloud decoder includes a second multilayer perceptron, a first three-dimensional feature upsampling unit, a second three-dimensional feature upsampling unit, a third three-dimensional feature upsampling unit, a fourth three-dimensional feature upsampling unit, and a third multilayer perceptron.

[0165] The output of the two-dimensional feature extraction unit is connected to the first input of the first feature fusion unit, the second feature fusion unit, the third feature fusion unit, and the fourth feature fusion unit, respectively.

[0166] The output of the first multilayer perceptron is connected to the input of the first three-dimensional feature downsampling unit. The output of the first three-dimensional feature downsampling unit is connected to the second input of the first feature fusion unit. The output of the first feature fusion unit is connected to the input of the second three-dimensional feature downsampling unit. The output of the second three-dimensional feature downsampling unit is connected to the second input of the second feature fusion unit. The output of the second feature fusion unit is connected to the input of the third three-dimensional feature downsampling unit. The output of the third three-dimensional feature downsampling unit is connected to the second input of the third feature fusion unit. The output of the third feature fusion unit is connected to the input of the fourth three-dimensional feature downsampling unit. The output of the fourth three-dimensional feature downsampling unit is connected to the second input of the fourth feature fusion unit. The output of the fourth feature fusion unit is connected to the input of the second multilayer perceptron.

[0167] The output of the second multilayer perceptron is connected to the first input of the first three-dimensional feature upsampling unit, the output of the first three-dimensional feature upsampling unit is connected to the first input of the second three-dimensional feature upsampling unit, the output of the second three-dimensional feature upsampling unit is connected to the first input of the third three-dimensional feature upsampling unit, the output of the third three-dimensional feature upsampling unit is connected to the first input of the fourth three-dimensional feature upsampling unit, and the output of the fourth three-dimensional feature upsampling unit is connected to the input of the third multilayer perceptron.

[0168] The output of the first feature fusion unit is also connected to the second input of the fourth three-dimensional feature upsampling unit, the output of the second feature fusion unit is also connected to the second input of the third three-dimensional feature upsampling unit, the output of the third feature fusion unit is also connected to the second input of the second three-dimensional feature upsampling unit, and the output of the fourth feature fusion unit is also connected to the second input of the first three-dimensional feature upsampling unit.

[0169] like Figure 3 As shown, the first three-dimensional feature downsampling unit, the second three-dimensional feature downsampling unit, the third three-dimensional feature downsampling unit, and the fourth three-dimensional feature downsampling unit have the same structure, each including a farthest point sampling layer, a K-nearest neighbor grouping layer, a neighborhood max pooling layer, a first fully connected layer, a first attention layer, and a second fully connected layer connected in sequence; the output of the neighborhood max pooling layer is also connected to the output of the second fully connected layer.

[0170] like Figure 4 As shown, the first feature fusion unit, the second feature fusion unit, the third feature fusion unit and the fourth feature fusion unit have the same structure, each including a first cross attention layer, a first self attention layer, a second cross attention layer, a second self attention layer and a third cross attention layer connected in sequence;

[0171] The output of the two-dimensional feature extraction unit is connected to the first input of the first cross-attention layer and the second cross-attention layer, respectively. The output of the second fully connected layer is connected to the second input of the first cross-attention layer. The output of the first self-attention layer is also connected to the first input of the third cross-attention layer.

[0172] like Figure 5 As shown, the first three-dimensional feature upsampling unit, the second three-dimensional feature upsampling unit, the third three-dimensional feature upsampling unit, and the fourth three-dimensional feature upsampling unit have the same structure, each including a third fully connected layer, a fourth fully connected layer, a trilinear interpolation layer, an element-wise addition layer, a fifth fully connected layer, a third self-attention layer, and a sixth fully connected layer.

[0173] The fourth fully connected layer, the trilinear interpolation layer, the element-wise addition layer, the fifth fully connected layer, the third self-attention layer, and the sixth fully connected layer are connected in sequence.

[0174] The input of the third fully connected layer is connected to the output of the third cross-attention layer, the output of the third fully connected layer is connected to the second input of the element-wise addition layer, and the output of the element-wise addition layer is also connected to the output of the sixth fully connected layer.

[0175] Understandably, the 2D feature extraction unit is used to extract 2D features from the input 2D image. In this embodiment, the 2D feature extraction unit is based on the ResNet50 network structure, including an input layer and a convolutional layer connected in sequence. The first perceptron is used to extract high-dimensional features from the input 3D point cloud data. The 3D feature downsampling unit downsamples the extracted high-dimensional features to reduce computational complexity and highlight important features. The 3D feature downsampling unit is based on the PointTransformer network structure, which can handle irregular 3D point cloud data and capture long-distance dependencies between points through a self-attention mechanism. The feature fusion unit is used to fuse 2D image features and high-dimensional features of the 3D point cloud. It captures the correlation between 2D and 3D features through a self-attention mechanism and further enhances the interaction between the two modal features through a cross-attention mechanism, resulting in more comprehensive multimodal feature information after fusion. The 3D feature upsampling unit is used to restore the original 3D point cloud data format. The 3D feature upsampling unit is based on the Point Transformer network structure and combines multi-scale jump connection feature information from the 3D feature downsampling unit during the upsampling process to ensure the integrity of multi-level information. Finally, the third multilayer perceptron outputs the recognition result for each point, i.e., whether it is a burr or a normal cross-section.

[0176] It should be noted that during the training of the spur detection model, the cross-entropy loss function is used to measure the difference between the predicted category and the true category, thereby improving the detection accuracy of the spur detection model. The 3D point cloud data with category labels and the corresponding 2D image are input into the spur detection model. The prediction result is calculated through forward propagation, and the model parameters are optimized through backpropagation to minimize the loss function. In this embodiment, the Adam optimization algorithm is used during training, and a double cosine annealing strategy is employed to ensure model convergence. The double cosine annealing strategy dynamically adjusts the learning rate at different stages of the training process by periodically reducing the learning rate. The specific formula for the learning rate change is:

[0177]

[0178] Where, η t η is the current learning rate. min Learning rate, η max Maximum learning rate, T cur This is the current training round, T max This is the maximum number of training epochs. In this embodiment, the initial learning rate η is... max Set to 0.001, minimum learning rate η min Set to 10 -6 .

[0179] It should be noted that in this embodiment, the number of input channels of the two-dimensional feature extraction unit is 1; the number of input channels of the first multilayer perceptron is 3 and the number of output channels is 32; the number of input channels of the second multilayer perceptron is 512 and the number of output channels is 512; the number of input channels of the third multilayer perceptron is 32 and the number of output channels is 2; the number of neighboring points to be considered for each point in all three-dimensional feature downsampling units is 16. Among them, the first three-dimensional feature downsampling unit has an input dimension of 32 and an output size of 1024×64; the second three-dimensional feature downsampling unit has an input dimension of 64 and an output size of 256×128; the third three-dimensional feature downsampling unit has an input dimension of 128 and an output size of 64×256; and the fourth three-dimensional feature downsampling unit has an input dimension of 256 and an output size of 16×512. The attention layers in all feature fusion units have 8 attention heads. Specifically, each attention head in the first feature fusion unit has a feature dimension of 64, the second feature fusion unit has a feature dimension of 128, the third feature fusion unit has a feature dimension of 256, and the fourth feature fusion unit has a feature dimension of 512. The output size of the first 3D feature upsampling unit is 64×256, the second 3D feature upsampling unit has an output size of 256×128, the third 3D feature upsampling unit has an output size of 1024×64, and the fourth 3D feature upsampling unit has an output size of 4096×32.

[0180] It should be noted that after the preprocessed 2D image and 3D point cloud data are input into the trained burr detection model, the burr label of each point in the preprocessed 3D point cloud data is first identified, indicating whether there is a burr or a normal cross-section. Then, the 3D point cloud data is projected onto the imaging plane of the 2D image in 3D space. Specifically:

[0181] Based on the pose information of the 2D industrial camera, the points in the 3D point cloud data are associated with the pixels in the 2D image through the camera projection equation, achieving preliminary pairing. The camera projection equation is as follows:

[0182] p = K 2D [R 2D |t 2D ]P

[0183] Where p represents the pixel coordinates in the two-dimensional image, K 2D This is the intrinsic parameter matrix of a 2D industrial camera, including focal length and principal point offset, R. 2D and t 2D , where represents the external parameters of the industrial camera, represents the rotation matrix and translation vector of the 2D camera respectively, and P is the homogeneous coordinate of the point in the 3D point cloud data;

[0184] Then, for each projection point of the 3D point cloud data onto the imaging plane, calculate the Euclidean distance to each pixel, find the pixel with the smallest Euclidean distance, and take it as the closest pixel.

[0185] Finally, the burr label of the point cloud point corresponding to the projection point is passed to the nearest pixel point as the burr label of that pixel point. In the end, each pixel point on the two-dimensional graphic has a burr label with burrs or normal cross-section, thus obtaining the burr region in the two-dimensional image.

[0186] S4: Remove the burr area from the two-dimensional image to obtain a burr-free image of the profile section to be measured;

[0187] S5: Correct the burr-free image of the profile section to be measured to obtain a corrected image of the profile section to be measured, and measure the dimensional parameters on the corrected image of the profile section to be measured.

[0188] It should be noted that due to the physical characteristics of camera imaging and environmental factors in the measurement scene, such as the distance and angle between the camera and the target object, a certain degree of geometric distortion will be introduced into the image. To ensure measurement accuracy, a homography matrix is ​​used to correct for ensemble errors. The homography matrix projects one plane from the burr-free image to another plane to obtain a corrected image of the cross-section of the profile to be measured. The transformation relationship is as follows:

[0189]

[0190] Where H is a 3×3 homography matrix, (x,y) and (x ′ ,y ′ () represent the pixel coordinates before and after correction, respectively;

[0191] Then, an edge detection algorithm is used to perform edge detection on the corrected image of the profile section to be measured, so as to obtain the set of contour lines of the profile section to be measured.

[0192] Arbitrarily select a preset measurement location, and select two corresponding target contour lines from the set of contour lines of the profile section to be measured;

[0193] Calculate the pixel distance between the two target contour lines in the corrected image of the profile section to be measured;

[0194] Based on the preset scaling factor and the pixel distance, the actual distance between the two target contour lines is calculated, and the actual distance is used as the actual size of the preset measurement part.

[0195] By traversing the preset measurement locations, the actual dimensions of all preset measurement locations are obtained, forming the dimensional parameters of the calibration image of the profile section to be measured.

[0196] Understandably, the edge detection algorithm produces a series of boundary lines or curves. By performing a Hough transform on these edge results and fitting them to straight lines, a set of contour lines for each key part of the profile section to be measured is obtained. Pixel distance is typically calculated by determining the perpendicular distance between these two target contour lines at a given measurement point. The formula is as follows:

[0197]

[0198] Where A1, B1, C1 and A2, B2, C2 are the parameters of the two target contour lines, and (x, y) are the coordinates of the measurement point;

[0199] Finally, based on the proportional relationship between pixel distance and three-dimensional space, the pixel distance is converted into actual size. The proportional relationship is obtained from calibration experiments, and it represents the actual physical length represented by each pixel. The formula for calculating the actual size is:

[0200] d real =α×d pixels

[0201] In the formula, α represents the proportionality coefficient.

[0202] In practical applications, during production line inspections, it can quickly and automatically measure the cross-sectional dimensions of profiles, significantly reducing measurement errors caused by manual operation. Especially for aluminum profiles with burrs, it can accurately detect and remove burr areas, ensuring the precision of dimensional measurements.

[0203] Example 3

[0204] This embodiment provides a profile cross-sectional dimension measurement system based on two-dimensional images and three-dimensional point clouds, used in the method described in Embodiment 1 or 2, such as... Figure 6 As shown, it includes:

[0205] The data acquisition module is used to acquire two-dimensional and depth images of the profile cross-section to be measured, and to generate three-dimensional point cloud data based on the depth image.

[0206] The data processing module is used to preprocess the two-dimensional image and the three-dimensional point cloud data respectively, and obtain the preprocessed two-dimensional image and the three-dimensional point cloud data accordingly.

[0207] The burr detection module is used to input the preprocessed two-dimensional image and three-dimensional point cloud data into the trained burr detection model to obtain the burr region in the two-dimensional image.

[0208] The burr removal module is used to remove the burr area from the two-dimensional image to obtain a burr-free image of the profile cross-section to be measured.

[0209] The dimension measurement module is used to correct the burr-free image of the profile section to be measured, obtain a corrected image of the profile section to be measured, and measure the dimension parameters on the corrected image of the profile section to be measured.

[0210] The same or similar labels correspond to the same or similar parts;

[0211] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0212] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for measuring the cross-sectional dimensions of profiles based on two-dimensional images and three-dimensional point clouds, characterized in that, include: Acquire two-dimensional and depth images of the cross-section of the profile to be measured, and generate three-dimensional point cloud data based on the depth image; The two-dimensional image and the three-dimensional point cloud data are preprocessed respectively to obtain the preprocessed two-dimensional image and the three-dimensional point cloud data. The preprocessed two-dimensional image and three-dimensional point cloud data are input into the trained burr detection model to obtain the burr region in the two-dimensional image. Remove the burr area from the two-dimensional image to obtain a burr-free image of the profile cross-section to be measured; The burr-free image of the profile section to be measured is corrected to obtain a corrected image of the profile section to be measured, and the dimensional parameters are measured on the corrected image of the profile section to be measured. The burr detection model includes, in sequence, a two-dimensional feature extraction unit, a first multilayer perceptron, a first three-dimensional feature downsampling unit, a first feature fusion unit, a second three-dimensional feature downsampling unit, a second feature fusion unit, a third three-dimensional feature downsampling unit, a third feature fusion unit, a fourth three-dimensional feature downsampling unit, a fourth feature fusion unit, a second multilayer perceptron, a first three-dimensional feature upsampling unit, a second three-dimensional feature upsampling unit, a third three-dimensional feature upsampling unit, a fourth three-dimensional feature upsampling unit, and a third multilayer perceptron; The output of the first feature fusion unit is also connected to the second input of the fourth three-dimensional feature upsampling unit, the output of the second feature fusion unit is also connected to the second input of the third three-dimensional feature upsampling unit, the output of the third feature fusion unit is also connected to the second input of the second three-dimensional feature upsampling unit, and the output of the fourth feature fusion unit is also connected to the second input of the first three-dimensional feature upsampling unit. The first feature fusion unit, the second feature fusion unit, the third feature fusion unit, and the fourth feature fusion unit have the same structure, each including a first cross attention layer, a first self attention layer, a second cross attention layer, a second self attention layer, and a third cross attention layer connected in sequence; The output of the two-dimensional feature extraction unit is connected to the first input of the first cross-attention layer and the second cross-attention layer, respectively; the output of the second fully connected layer is connected to the second input of the first cross-attention layer; the output of the first self-attention layer is also connected to the first input of the third cross-attention layer. The first three-dimensional feature upsampling unit, the second three-dimensional feature upsampling unit, the third three-dimensional feature upsampling unit and the fourth three-dimensional feature upsampling unit have the same structure, each including a third fully connected layer, and a fourth fully connected layer, a trilinear interpolation layer, an element-wise addition layer, a fifth fully connected layer, a third self-attention layer and a sixth fully connected layer connected in sequence. The input of the third fully connected layer is connected to the output of the third cross-attention layer, the output of the third fully connected layer is connected to the second input of the element-wise addition layer, and the output of the element-wise addition layer is also connected to the output of the sixth fully connected layer.

2. The method for measuring profile cross-sectional dimensions based on two-dimensional images and three-dimensional point clouds according to claim 1, characterized in that, The acquisition of the two-dimensional image and depth image of the cross-section of the profile to be measured includes: Fix the cross-section of the profile to be measured, use a two-dimensional industrial camera to acquire a two-dimensional image of the cross-section of the profile to be measured, use a three-dimensional depth camera to acquire a depth image of the cross-section of the profile to be measured, and save the pose information of the three-dimensional depth camera and the two-dimensional industrial camera.

3. The method for measuring profile cross-sectional dimensions based on two-dimensional images and three-dimensional point clouds according to claim 1, characterized in that, Generating 3D point cloud data from the depth image includes: The formula for generating 3D point cloud data from a depth image is: In the formula, Represents the third point in the 3D point cloud data The three-dimensional coordinates of a point cloud. Represents the depth image of the th Two-dimensional coordinates of a pixel Indicates the first The depth value of each pixel. This represents the optical center coordinates of the 3D depth camera. These represent 3D depth cameras. Focal length in direction.

4. The method for measuring profile cross-sectional dimensions based on two-dimensional images and three-dimensional point clouds according to claim 1, characterized in that, Obtaining the trained spur detection model includes: Acquire three-dimensional point cloud data and corresponding two-dimensional images of several profile cross sections; The three-dimensional point cloud data and corresponding two-dimensional images of all the profile cross sections are normalized to obtain the normalized three-dimensional point cloud data and corresponding two-dimensional images of the profile cross sections. Each point in the normalized 3D point cloud data of all the profile cross sections is labeled with a category to obtain 3D point cloud data of the profile cross sections with category labels; To construct a burr detection model, input 3D point cloud data of profile cross-section with category labels and corresponding 2D images, set a loss function for optimization training, and save the corresponding model parameters when the loss function value reaches the minimum value to obtain the trained burr detection model.

5. The method for measuring profile cross-sectional dimensions based on two-dimensional images and three-dimensional point clouds according to claim 1, characterized in that, The first, second, third, and fourth 3D feature downsampling units have the same structure, each including a farthest point sampling layer, a K-nearest neighbor grouping layer, a neighborhood max pooling layer, a first fully connected layer, a first attention layer, and a second fully connected layer connected in sequence; the output of the neighborhood max pooling layer is also connected to the output of the second fully connected layer.

6. The method for measuring profile cross-sectional dimensions based on two-dimensional images and three-dimensional point clouds according to claim 1, characterized in that, Measuring dimensional parameters on the calibrated image of the profile section to be measured includes: Edge detection is performed on the corrected image of the profile section to be measured using an edge detection algorithm to obtain a set of contour lines of the profile section to be measured; Arbitrarily select a preset measurement location, and select two corresponding target contour lines from the set of contour lines of the profile section to be measured; Calculate the pixel distance between the two target contour lines in the corrected image of the profile section to be measured; Based on the preset scaling factor and the pixel distance, the actual distance between the two target contour lines is calculated, and the actual distance is used as the actual size of the preset measurement part. By traversing the preset measurement locations, the actual dimensions of all preset measurement locations are obtained, forming the dimensional parameters of the calibration image of the profile section to be measured.

7. A profile cross-sectional dimension measurement system based on two-dimensional images and three-dimensional point clouds, used to implement the method described in any one of claims 1-6, characterized in that, include: The data acquisition module is used to acquire two-dimensional and depth images of the profile cross-section to be measured, and to generate three-dimensional point cloud data based on the depth image. The data processing module is used to preprocess the two-dimensional image and the three-dimensional point cloud data respectively, and obtain the preprocessed two-dimensional image and the three-dimensional point cloud data accordingly. The burr detection module is used to input the preprocessed two-dimensional image and three-dimensional point cloud data into the trained burr detection model to obtain the burr region in the two-dimensional image. The burr removal module is used to remove the burr area from the two-dimensional image to obtain a burr-free image of the profile cross-section to be measured. The dimension measurement module is used to correct the burr-free image of the profile section to be measured, obtain a corrected image of the profile section to be measured, and measure the dimension parameters on the corrected image of the profile section to be measured.

Images