Method for three-dimensional object size measurement based on a depth vision camera

Abstract

This invention discloses a method for measuring the size of three-dimensional objects based on a depth vision camera. Steps one to two achieve data acquisition and model retraining. Steps three to six process the initial data, including data filtering, 3D edge segmentation, and line fitting, to obtain specific object size parameters. Step seven is a visualization operation, which displays the fitted plane in a two-dimensional image.

Description

Technical Field

[0001] This invention relates to a measurement method, and more specifically to a method for measuring the size of a three-dimensional object based on a depth vision camera. Background Technology

[0002] With the continuous development of science and technology, 3D measurement technology has been widely applied in fields such as industrial manufacturing, medical health, virtual reality, and robot navigation. Traditional 3D measurement methods mainly rely on technologies such as laser scanning, structured light, and stereo vision. However, these methods have some limitations, such as high cost, complex equipment, poor environmental adaptability, and limited measurement accuracy. In the industrial manufacturing field, 3D measurement technology is widely used for quality control, component inspection, and assembly accuracy verification. Although traditional laser scanning technology can provide high-precision 3D data, the equipment is expensive and maintenance costs are high. While structured light technology can achieve rapid measurement, it has high requirements for ambient light and is easily affected by external interference. Stereo vision technology requires multiple cameras to work together, resulting in high system complexity, large computational load, and poor real-time performance.

[0003] Depth vision cameras, as an emerging 3D measurement technology, utilize infrared light and Time-of-Flight (ToF) technology to quickly and accurately acquire 3D information of objects. ToF technology calculates the object's distance by measuring the time it takes for infrared light emitted from the camera to travel to and from the object, thus generating a depth image. In 3D measurement, a corresponding 3D object point cloud can be generated using only depth data and camera intrinsic parameters. Furthermore, depth vision cameras are well-suited for large-scale applications due to their low cost and ease of installation, effectively reducing equipment procurement and maintenance costs. In addition, depth vision cameras offer advantages such as strong real-time performance and good environmental adaptability, operating stably under various lighting conditions and suitable for real-time measurement and feedback in dynamic environments. Object size measurement methods based on depth cameras solve the problems of high cost, complex equipment, poor environmental adaptability, and limited measurement accuracy inherent in traditional 3D measurement methods.

[0004] Current 3D object measurement technologies mainly include: 1. Scanning the object surface with a laser beam to obtain high-precision 3D data, suitable for measuring complex surfaces and large-sized objects, but with high equipment costs and slow scanning speed. 2. Reconstructing 3D shapes by projecting grating patterns and analyzing their deformation, featuring high precision and high-speed scanning, suitable for small to medium-sized objects, but requiring high ambient light. 3. Reconstructing 3D shapes by capturing 2D images of the object from multiple angles and using computer vision algorithms, with low cost and wide applicability, but accuracy depends on image quality and algorithm complexity. None of these existing methods can adapt to complex and variable measurement environments and the demands for high efficiency and high precision. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a method for measuring the size of three-dimensional objects based on a depth vision camera, which achieves the size analysis of the main body in a single RGB-D image through methods such as subject segmentation, point cloud information extraction, data filtering, and data mining.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for measuring the size of a three-dimensional object based on a depth vision camera, characterized by comprising the following steps:

[0007] Step 1: Construct a dataset for image subject segmentation. The dataset includes house walls and columns, as well as their corresponding 3D dimensions and location information.

[0008] Step 2: Construct an image subject segmentation model, and use the constructed image subject segmentation model to segment the image and generate corresponding masks;

[0009] Step 3: Read the mask image segmented in Step 2, then select an appropriate threshold, and then apply the threshold to set the pixel values ​​above the threshold to 1 and the pixel values ​​below the threshold to 0. Finally, save or display the processed binarized image. After obtaining the binarized mask information, first apply it to the depth map to segment and extract the depth information of the main body, and then convert the depth information into a point cloud and perform binarization processing on the image.

[0010] Step four: Filter the point cloud transformed in step three;

[0011] Step 5: Perform data mining on the point cloud data filtered in Step 4. Specifically, after obtaining the point cloud data, use a point cloud segmentation model to segment the point cloud into components, mainly front and side views. Record the edge points of these two sides, and also record the edges where the sides intersect. After recording, merge and output the data to obtain the edge information of the 3D point cloud.

[0012] Step six: Preprocess the edge point cloud data obtained in step five to remove noise and outliers to ensure data accuracy. Then, use common line fitting algorithms to fit these edge point cloud data into straight lines. During the fitting process, the algorithm continuously adjusts the plane parameters to find the best-fitting straight line, minimizing the distance error between the point cloud data and the fitted line. After the line fitting is completed, the edge dimensions of each plane are accurately measured through geometric calculations.

[0013] Step 7: Mapping 3D coordinates to a 2D image. After calculating the object size in the image, the 3D coordinates of the edge points are mapped to the 2D image using camera intrinsics.

[0014] As a further improvement to the present invention, the specific steps for constructing the dataset for image subject segmentation in step one are as follows:

[0015] Step one by one, data acquisition devices are deployed in multiple unfinished houses. The deployed devices need to have sufficient transmission speed, i.e., encoding and decoding capabilities, while being able to read images and depth information.

[0016] Steps one and two involve placing the checkerboard pattern at different angles and positions for photographing, then extracting the checkerboard corner points from each image. Based on the correspondence between these corner points and their actual 3D coordinates, the camera's intrinsic parameters (focal length f) are estimated using the least squares method or a nonlinear optimization method. x ,f y , Photocentric C x ,c y The camera intrinsic parameters matrix, including distortion coefficients and extrinsic parameters, is in the form of:

[0017]

[0018] The relationship between the 3D coordinates (X,Y,Z) and image coordinates (u,v) of the corner points of the chessboard can be expressed as:

[0019]

[0020] Among them, the camera's intrinsic parameter is the focal length f. x and f y , light center c x and c y ;

[0021] Steps 1 and 3: Collect data. The collected data includes images and depth information. The collection angle covers the entire house. The shooting perspective of the subject maintains a 3D shape, and the length, width and height ratios are normal.

[0022] Step 14: Label the data collected in Step 13;

[0023] Step 15 involves data cleaning of the data labeled in Step 14. The data cleaning steps include statistical analysis, rule filtering, missing value handling, duplicate data handling, noise filtering, and subject selection.

[0024] As a further improvement of the present invention, the construction of the image subject segmentation model in step two includes a feature extraction module, a subject selection module, and a decoding module. The feature extraction module is used to extract the overall features of the input image, the subject selection module is used to determine and select the subject features in the image to highlight key information, and the decoding module is mainly responsible for parsing and restoring the extracted feature information to generate a visual output result. The specific steps for constructing the image subject segmentation model are as follows:

[0025] Step 21: Construct a feature extraction module based on a two-dimensional convolutional function. This feature extraction module includes 5 network blocks, each of which has a similar structure, consisting of two convolutional layers, a Leaky ReLU function, and a MaxPooling layer.

[0026] Step 22: First, input the features extracted by the feature extraction module in Step 21 into the backpropagation layer of the Conv2DBackpropInput feature input terminal to calculate the convolution operation. At the same time, input the randomly generated matrix after processing. This matrix is ​​first generated into an original constant tensor by the Const operation, then copied into three copies. Each matrix is ​​sliced, with the slice object being the first 3 matrix elements. Then, the first two elements are divided by 2 and concatenated. The concatenated elements are generated by the pack operation. Then, the concatenated elements and the previous features are put into the backpropagation layer of the convolution operation. The output value is put into a set of standard batch normalization operation layers below to complete the construction of the subject selection module.

[0027] Steps two and three: A feature decoding module is built based on a two-dimensional convolution function. Each decoding layer of this feature decoding module will replace the subject selection module with the Max pooling layer as a manifestation of multiple views.

[0028] Step 24: Build the loss function module. The loss function is as follows:

[0029]

[0030] Where, p i It is the value of the i-th dimension of the prediction distribution, g i is the value of the i-th dimension of the corresponding true value distribution, and d is the dimension of the predicted distribution;

[0031] Step 25: Optimize the model using an inner loop plus an outer loop;

[0032] Step 26: Use transfer learning to transfer the model to category segmentation and retrain the model to adapt to different segmentation objects.

[0033] As a further improvement to the present invention, the specific steps for optimizing the model in step two-five using an inner loop plus an outer loop are as follows:

[0034] First, in the inner loop, the model performs several gradient descent operations using a small amount of training data to quickly adjust the model parameters, setting the initial model parameters to θ. For each task T... i With a small amount of training data Calculate loss Then, gradient descent is performed to update the parameters, and the update formula is as follows:

[0035]

[0036] Where α is the learning rate of the inner loop. For task T i The updated parameters.

[0037] Secondly, in the outer loop, the model updates the meta-model parameters using the validation loss from different tasks. Specifically,

[0038] Use the updated parameters In verifying data Calculate loss The meta-model parameters θ are then updated using these losses. The update formula for the outer loop is:

[0039]

[0040] Where β is the learning rate of the outer loop.

[0041] As a further improvement of the present invention, the specific steps for retraining the model in step 26 are as follows: Step 261, perform main body segmentation output layer conversion, retain the feature extraction layer of the pre-trained model, and use it for category classification task;

[0042] Step 262: Add a new output layer for the new task. The new output layer includes a convolutional layer and a classification layer, which are used to map features to specific class labels.

[0043] Step 263: Retrain the model using the category segmentation dataset.

[0044] As a further improvement of the present invention, when retraining the model using the category segmentation dataset in step 263, the parameters of the feature extraction layer F(·) remain unchanged, and only the parameters of the newly added convolutional layer C(·) and classification layer H(·) are updated. The retraining loss function is defined as a combination of the following two parts:

[0045] Segmentation loss Used to measure the performance of a model on class segmentation tasks:

[0046]

[0047] Where N is the number of samples, M is the number of pixels in each sample, and the Loss() loss function is the improved cross-entropy loss function proposed in Step 4. It is the model's predicted output for the j-th pixel of the i-th sample. It is the true label of the j-th pixel of the i-th sample;

[0048] Classification loss Used to ensure the accuracy of category classification:

[0049]

[0050] The CrossEntropy() function is the basic cross-entropy loss function, and the overall retraining loss function is defined as follows: λ is the balance coefficient, used to adaptively balance the relative importance of the two losses.

[0051] As a further improvement of the present invention, the specific steps for binarizing the image in step three are as follows:

[0052] Step 31: Read the binarized mask and depth map. Specifically, read the binarized mask image and the corresponding depth map. The binarized mask image is obtained by converting the original image into a black and white image. The white area represents the subject of interest, while the black area represents the background or parts of no interest. The mask image can be generated from image processing or segmentation algorithms. Meanwhile, the depth map records the actual distance of each pixel to the camera, usually represented by grayscale values. The higher the value, the farther the distance. These two images need to be read at the same resolution and alignment to ensure correct matching of the mask and depth data.

[0053] Step 32: Use the binarized mask image as a filtering tool to retain only the depth information of the main body. Specifically, for each pixel, check the value of its mask image. If the mask value is white, it means that the pixel belongs to the main body of interest, and the corresponding depth value in the depth map is retained. If the mask value is black, it means that the pixel belongs to the background or an unwanted part, and its depth value is ignored.

[0054] Step 33: Obtain camera intrinsic parameters, specifically by reading or setting the camera's intrinsic parameter matrix;

[0055] Steps three and four involve calculating the point cloud. Specifically, first, depth map data is acquired. Then, the camera intrinsic parameters obtained in step three are used to convert this depth information into 3D point cloud coordinates, as shown in the following formula. For each pixel (u,v) and its corresponding depth value Z, its 3D spatial coordinates (X,Y,Z) can be calculated using the following formula:

[0056]

[0057] Z = depth

[0058] Where (u,v) are pixel coordinates on the image plane, (c x ,c y ) is the coordinate of the optical center, (f x,f y ) is the camera's focal length, and the depth value Z is the depth data of the corresponding pixel.

[0059] As a further improvement of the present invention, the step of filtering the point cloud in step four is as follows:

[0060] First, for each point p i First, construct its neighborhood. The K-nearest neighbor algorithm can be used to find the K nearest points within its neighborhood. Next, calculate the average distance d from this point to all points in its neighborhood. i :

[0061]

[0062] Compare this average distance with the average distance of all points. Compare, if d i If the value exceeds a certain preset threshold, the point is considered an outlier and is removed. The specific steps are as follows:

[0063] 1. Constructing the neighborhood: For each point p in the point cloud i Find the K nearest points {p} in its neighborhood. j |j=1,2,...,K};

[0064] 2. Calculate the average distance: Calculate the distance from point p. i The average distance d to all points in its neighborhood i ;

[0065] 3. Remove outliers: Compare d i With preset threshold d i +ασ, where σ is the average distance of all points, α is the standard deviation, and α is an adjustment parameter. i If the value is greater than the threshold, then remove the point.

[0066] As a further improvement to the present invention, the specific steps for fitting the edge point cloud data into a straight line in step six are as follows:

[0067] Step 61: Perform data preprocessing, including noise removal and outlier filtering of the edge point cloud data, to ensure the accuracy and stability of the data.

[0068] Step 62: The RANSAC algorithm is used to fit a straight line to the processed edge point cloud data. The algorithm continuously adjusts the plane parameters to find the straight line that best matches the point cloud data.

[0069] Step 63: Through geometric calculations, measure the side length, width, and angles and distances between the planes formed by the fitted edge lines.

[0070] Step 64 involves organizing and outputting the calculated dimensional parameters to provide basic data for further analysis and processing.

[0071] The beneficial effects of this invention are as follows: The 3D object size measurement method based on a depth vision camera in this embodiment uses a depth map and camera intrinsic parameters as a foundation. It extracts features from the RGB image through a deep neural network and introduces a few-shot learning strategy, enabling the model to accurately extract subject information with a small number of annotations, identify all possible objects in a given RGB image, and achieve subject segmentation. Finally, a retraining strategy is used to transfer the model task from subject segmentation to category segmentation, thereby adapting to the diverse application scenarios of 3D object size measurement. The segmented objects are then binarized to serve as a depth map mask, which is combined with intrinsic parameters to generate a 3D point cloud of the subject. After appropriate data filtering and plane fitting operations, the size information of the subject is obtained, and the coordinates are mapped to a 2D image for display. The size analysis of the subject in a single RGB-D image is achieved through subject segmentation, point cloud information extraction, data filtering, and data mining. Attached Figure Description

[0072] Figure 1 This is an example diagram of simulated data acquisition using the depth camera of the present invention;

[0073] Figure 2 A schematic diagram of the main body segmentation model architecture;

[0074] Figure 3 This is a schematic diagram of the feature extraction module structure;

[0075] Figure 4 A diagram illustrating the selection of modules for the main body;

[0076] Figure 5 This is a schematic diagram of the feature decoding module;

[0077] Figure 6 Here is a flowchart of the 3D edge segmentation algorithm;

[0078] Figure 7 This is a schematic diagram of the 3D coordinate mapping result. Detailed Implementation

[0079] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0080] Reference Figure 1 As shown in the figure, a method for measuring the size of a three-dimensional object based on a depth camera in this embodiment mainly includes the following steps:

[0081] Construct a dataset for image subject segmentation, which includes house walls and columns, along with their corresponding 3D dimensions and location information.

[0082] Step 1: Equipment Deployment. Deploy acquisition equipment in multiple unfinished rooms. This equipment needs to be able to read images and depth information while also having sufficient transmission speed, i.e., encoding and decoding capabilities.

[0083] Step 2: Camera Calibration. Before acquiring the corresponding data, the camera needs to be calibrated. Calibration is fundamental for accurate measurement and obtaining accurate positional information. Its main function is to establish the relationship between image pixel size and actual size, and to establish the mapping relationship between the image coordinate system and the world coordinate system. A checkerboard calibration method can be used to obtain the camera's intrinsic parameters and distortion coefficients. First, place the checkerboard pattern at different angles and positions for shooting, then extract the checkerboard corner points in each image. Through the correspondence between these corner points and the actual 3D coordinates, the camera's intrinsic parameters (focal length f) are estimated using the least squares method or nonlinear optimization methods. x ,f y , Photocentric C x ,c y The camera intrinsic parameters are: distortion coefficients, and extrinsic parameters (rotation matrix R and translation matrix t). The camera intrinsic parameter matrix is ​​in the form of:

[0084]

[0085] The relationship between the 3D coordinates (X,Y,Z) and image coordinates (u,v) of the corner points of the chessboard can be expressed as:

[0086]

[0087] It should be noted that the intrinsic parameter coefficients of different cameras are different in the same scene, and they need to be calibrated separately.

[0088] Step 3: Data Collection. The data to be collected includes images and depth information. The collection angle should cover the entire room as much as possible. At the same time, it is important to ensure that the subject's perspective maintains its 3D shape and that the length, width, and height ratios are normal to avoid distortion caused by perspective issues, which could affect the measurement results.

[0089] Step 4: Data Annotation. The information to be annotated includes whether the column is truncated or occluded, the object's rotation angle, its 2D bounding box in the image, the column's length, width, and height, its position relative to the camera, the column's angle around the y-axis, and finally, the confidence level of the result. A sample of the final data is shown below. Figure 1 As shown, it includes RGB images and depth images, as well as labeled data.

[0090] Step 5: Data Cleaning. Before obtaining the final data, it needs to be cleaned to filter out data unsuitable for model training. This includes handling outliers, missing values, duplicate data, erroneous data, noisy data, and irrelevant data. The steps of data cleaning include statistical analysis, rule filtering, missing value handling, duplicate data handling, noise filtering, and subject selection. Data cleaning improves data quality, thereby enhancing the training effect and accuracy of the model. The cleaned data will serve as input for model training and validation, ensuring that the final model performance meets the expected goals.

[0091] Image subject segmentation model, model architecture is as follows Figure 2 As shown, the structure is mainly divided into three parts: a feature extraction module, a subject selection module, and a decoding module. The feature extraction module extracts the overall features of the input image, the subject selection module identifies and selects the main features in the image to highlight key information, and the decoding module is responsible for parsing and restoring the extracted feature information to generate a visual output. The specific steps for building this structure are as follows:

[0092] Step 1: Building the feature extraction module. This invention uses a feature extraction module based on a two-dimensional convolution function. The model structure is as follows: Figure 3 As shown, the structure comprises five network blocks, each with a similar structure consisting of two convolutional layers, a Leaky ReLU function, and a MaxPooling layer. The parameter settings for each layer are as follows: Figure 3 As shown.

[0093] Step 2: Construct the subject selection module. This invention employs a strategy of randomly generating matrices for framing and calculating deviations under different depths of view. The model structure is as follows: Figure 4 The process consists of multiple operation layers. First, the features extracted by the feature extraction module in Step 1 are input to the backpropagation layer of the Conv2DBackpropInput feature input layer to calculate the convolution operation. Simultaneously, a randomly generated matrix after processing is input. This matrix is ​​first converted into a raw constant tensor by the Const operation, then copied three times. Each matrix is ​​sliced, with the slice containing the first three matrix elements. The first two elements are then divided by 2 and concatenated using the pack operation. The concatenated elements, along with the previous features, are then fed into the backpropagation layer to calculate the convolution operation. The output value is then fed into a standard batch normalization layer. Note that the backpropagation operation must be stopped before calculating the squared difference to prevent the target value from being updated. The normalized value is then passed to the activation function to obtain the result of one subject selection. It is important to note that the same operation will occur multiple times in the decoding module of Step 3.

[0094] Step 3: Construct the feature decoding module. Similar to the feature extraction module, the feature decoding module is also based on two-dimensional convolutional functions. However, unlike the feature extraction module, each decoding layer replaces the subject selection module with a max pooling layer to represent multiple views. The specific structure of the model is as follows: Figure 5 The structure is similar to Step 1, except that the pooling layer is replaced by a Topic Select layer.

[0095] Step 4: Construct the loss function module. This invention uses an improved cross-entropy loss function. Due to the varying data distribution in the main segmentation, the imbalance of the data and the importance of different classes should be considered when calculating the loss. Traditional cross-entropy loss functions often neglect the minority class when dealing with imbalanced data. In main segmentation tasks, the number of pixels in each class may differ significantly, affecting the model's training performance. Therefore, different weights need to be assigned to each class when calculating the loss to enhance attention to the minority class. Simultaneously, joint loss calculations can be performed using metrics such as the Dice coefficient or IoU (Intersection over Union) to further improve the model's segmentation accuracy in boundaries and small regions. Furthermore, for different application scenarios, the weight coefficients of the loss function may need to be dynamically adjusted to adapt to the specific task requirements. Through these improvements, the data distribution problem in main segmentation can be handled more effectively, improving the accuracy and robustness of segmentation. The specific formula is as follows:

[0096]

[0097] Where p i It is the value of the i-th dimension of the prediction distribution, g i d is the value of the i-th dimension of the corresponding true value distribution, and d is the dimension of the predicted distribution.

[0098] Step 5: Due to the high cost of semantic segmentation annotation, this invention employs a few-shot learning strategy, introducing a meta-learning method to enable the model to accurately extract subject information with limited labeled data. Meta-learning trains a meta-model that can quickly adapt to new tasks, thereby achieving efficient subject information extraction with limited labeled data. Specifically, the training in this invention includes two main optimization processes: an inner loop and an outer loop.

[0099] First, in the inner loop, the model performs several gradient descent operations using a small amount of training data to quickly adjust the model parameters. The initial model parameters are set to θ, and for each task T... i With a small amount of training data Calculate loss Then, gradient descent is used to update the parameters. The update formula is as follows:

[0100]

[0101] Where α is the learning rate of the inner loop. For task T i The updated parameters.

[0102] Secondly, in the outer loop, the model updates the meta-model parameters using the validation loss from different tasks. Specifically...

[0103] In other words, use the updated parameters In verifying data Calculate loss The meta-model parameters θ are then updated using these losses. The update formula for the outer loop is:

[0104]

[0105] Where β is the learning rate of the outer loop.

[0106] This invention simulates multiple tasks during the training phase, enabling the model to learn a good initialization parameter θ. This allows the model to achieve good performance on new tasks with only a small number of gradient updates. This strategy significantly improves the model's performance on small sample datasets and enhances its generalization ability across different tasks.

[0107] Step 6: Model retraining strategy. Since the segmentation requirements of different tasks are different, the model needs to be retrained to meet the requirements of different tasks. This invention adopts the transfer learning method to transfer the subject segmentation model to category segmentation in order to adapt to the situation of different segmentation objects.

[0108] This invention achieves task transformation of the model through a retraining strategy, thereby maintaining high performance in the new task. Specifically, it includes the following steps:

[0109] 1. The output layer transformation for subject segmentation first preserves the feature extraction layer of the pre-trained model and applies it to the class segmentation task. The preserved feature extraction layer contains efficient recognition capabilities for target regions in the image. In the subject segmentation task, the pre-trained model has already learned some general features, which are also useful for the class segmentation task.

[0110] 2. Add new output layers for the class segmentation task. The parameters of these output layers are initialized according to the specific requirements of the class segmentation task. The new output layers typically include a convolutional layer and a classification layer, used to map features to specific class labels.

[0111] 3. Retrain the model using the category segmentation dataset. During retraining, keep the parameters of the feature extraction layer F(·) unchanged, and only update the parameters of the newly added convolutional layer C(·) and classification layer H(·). The retraining loss function can be defined as a combination of the following two parts:

[0112] Segmentation loss Used to measure the performance of a model on class segmentation tasks:

[0113]

[0114] Where N is the number of samples, M is the number of pixels in each sample, and the Loss() loss function is the improved cross-entropy loss function proposed in Step 4. It is the model's predicted output for the j-th pixel of the i-th sample. It is the true label of the j-th pixel of the i-th sample.

[0115] Classification loss Used to ensure the accuracy of category classification:

[0116]

[0117] The CrossEntropy() function is the basic cross-entropy loss function. The overall retraining loss function is defined as: λ is the balance coefficient, used to adaptively balance the relative importance of the two losses.

[0118] Although a mask has been generated after image segmentation, the values ​​of the main body are not yet uniformly set to 1, so binarization is required. The specific steps are as follows: First, read the segmented mask image; then, select an appropriate threshold; next, apply the threshold to set pixel values ​​above the threshold to 1 and pixel values ​​below the threshold to 0; finally, save or display the processed binarized image. After obtaining the binarized mask information, apply it to the depth map to segment and extract the depth information of the main body; then, convert the depth information into a point cloud. The specific steps are as follows:

[0119] Step 1: Read the Binarized Mask and Depth Map. First, you need to read the binarized mask image and its corresponding depth map. The binarized mask image is obtained by converting the original image into a black and white image, where white areas (usually with a value of 255) represent the subject of interest, while black areas (with a value of 0) represent the background or parts of no interest. The mask image can be generated from image processing or segmentation algorithms. Meanwhile, the depth map records the actual distance of each pixel to the camera, usually represented by grayscale values, with higher values ​​indicating greater distance. Both images need to be read at the same resolution and alignment to ensure correct matching of the mask and depth data.

[0120] Step 2: Apply a mask. Applying a mask involves using the binarized mask image as a filter to retain only the depth information of the main subject. Specifically, for each pixel, the value of its mask image is checked. If the mask value is white (255), it indicates that the pixel belongs to the subject of interest, and the corresponding depth value in the depth map is retained; if the mask value is black (0), it indicates that the pixel belongs to the background or an unwanted part, and its depth value is ignored. In this way, the mask filters out the depth information of non-subject parts from the depth map, generating a depth map that only contains the depth information of the main subject. This processing step is crucial for subsequent point cloud generation and subject analysis because it ensures that only the region of interest is further processed.

[0121] Step 3: Obtain Camera Intrinsic Parameters. Camera intrinsic parameters are crucial for converting 2D image coordinates into 3D spatial coordinates, including focal length and optical center coordinates. First, the camera's intrinsic parameter matrix needs to be read or set. The intrinsic parameter matrix typically includes the focal length f. x ,f y , Photocentric C x ,c y Focal length indicates the magnification capability of a camera lens, while the optical center coordinates represent the center point of the image plane. These parameters can be obtained through the camera calibration process, which requires photographing a specific calibration board. These calibration images are then used to calculate an accurate intrinsic parameter matrix using algorithms. These parameters can be retrieved from the camera's calibration file, or, in the case of an application, set by the user through a configuration file or using the SDK's API.

[0122] Step 4: Calculate the point cloud. The point cloud calculation process relies on camera intrinsics and the depth map. First, acquire the depth map data, which records the distance of each pixel to the camera. Using the camera intrinsics, this depth information can be converted into 3D point cloud coordinates. Specifically, for each pixel (u, v) and its corresponding depth value Z, its 3D spatial coordinates (X, Y, Z) can be calculated using the following formula:

[0123]

[0124] Z = depth

[0125] Where (u,v) are pixel coordinates on the image plane, (c x ,c y ) is the coordinate of the optical center, (f x ,f yThe focal length (Z) represents the camera's focal length, and the depth value (Z) represents the depth data of the corresponding pixel. These calculations convert the depth information of each pixel into three-dimensional coordinates, thereby generating a complete three-dimensional point cloud dataset. This point cloud data can be used for further applications such as dimensional analysis and object recognition.

[0126] Outliers are inevitable during point cloud extraction, and these outliers can affect subsequent data processing. Therefore, point cloud filtering is necessary. First, for each point p... i First, construct its neighborhood. The K-nearest neighbor algorithm can be used to find the K nearest points within its neighborhood. Next, calculate the average distance d from this point to all points in its neighborhood. i :

[0127]

[0128] Compare this average distance with the average distance of all points. Compare, if d i If the value exceeds a certain preset threshold, the point is considered an outlier and is removed. The specific steps are as follows:

[0129] 1. Constructing the neighborhood: For each point p in the point cloud i Find the K nearest points {p} in its neighborhood. j |j=1,2,...,K}.

[0130] 2. Calculate the average distance: Calculate the distance from point p. i The average distance d to all points in its neighborhood i .

[0131] 3. Remove outliers: Compare d i With preset threshold d i +ασ(where σ is the average distance of all points, α is the standard deviation, and α is an adjustment parameter. i If the value is greater than the threshold, then remove the point.

[0132] These steps can significantly reduce noise and outliers in point clouds, improve the quality of point cloud data, and provide a more reliable data foundation for subsequent 3D reconstruction, feature extraction, and analysis.

[0133] The filtered point cloud is then subjected to data mining. This step aims to obtain the edge information of the main subject. The specific algorithm flowchart is as follows. Figure 6 As shown, after obtaining the point cloud data, a point cloud segmentation model is first used to segment the point cloud into components, mainly the front and side faces. The edge points of these two faces are recorded, as well as the edges where the faces intersect. After recording, the data is merged and output to obtain the edge information of the 3D point cloud.

[0134] Next, line fitting is performed to calculate the edge dimensions. The key to this step is using a fitting algorithm to fit the extracted edge point cloud data into multiple straight lines, thereby calculating the actual dimensions of each edge. Specifically, the edge point cloud data is first preprocessed to remove noise and outliers to ensure data accuracy. Then, common line fitting algorithms, such as RANSAC (Random Sample Consensus), are used to fit these edge point cloud data into straight lines. During the fitting process, the algorithm continuously adjusts the planar parameters to find the best-fitting line, minimizing the distance error between the point cloud data and the fitted line. After the line fitting is complete, geometric calculations are used to accurately measure the edge dimensions of each plane.

[0135] The detailed steps for calculating edge dimensions using plane fitting are as follows:

[0136] Step 1: Data Preprocessing. Noise removal and outlier filtering are performed on the edge point cloud data to ensure data accuracy and stability.

[0137] Step 2: Plane Fitting. The RANSAC algorithm is used to fit a straight line to the processed edge point cloud data. The algorithm continuously adjusts the plane parameters to find the straight line that best matches the point cloud data.

[0138] Step 3: Dimension Calculation. Through geometric calculations, measure the side length, width, and angles and distances between the planes formed by the fitted edge lines.

[0139] Step 4: Data Output. Organize and output the calculated dimensional parameters to provide basic data for further analysis and processing.

[0140] Mapping 3D coordinates to a 2D image: After calculating the size of an object in an image, the 3D coordinates of edge points can be mapped to the 2D image using camera intrinsics, such as... Figure 7 As shown.

[0141] In summary, this invention achieves data acquisition and model retraining through steps 1) to 2), steps 3) to 6) process the initial data, including data filtering, 3D edge segmentation, line fitting and other operations to obtain specific object size parameters, and step 7) is a visualization operation, which displays the fitted plane in a two-dimensional image.

[0142] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for measuring the size of a three-dimensional object based on a depth vision camera, characterized in that: Includes the following steps: Step 1: Construct a dataset for image subject segmentation. The dataset includes house walls and columns, as well as their corresponding 3D dimensions and location information. Step 2: Construct an image subject segmentation model, and use the constructed image subject segmentation model to segment the image and generate corresponding masks; Step 3: Read the mask image segmented in Step 2, then select an appropriate threshold, and then apply the threshold to set the pixel values ​​above the threshold to 1 and the pixel values ​​below the threshold to 0. Finally, save or display the processed binarized image. After obtaining the binarized mask information, first apply it to the depth map to segment and extract the depth information of the main body, and then convert the depth information into a point cloud and perform binarization processing on the image. Step four: Filter the point cloud transformed in step three; Step 5: Perform data mining on the point cloud that has been filtered in Step 4. Specifically, after obtaining the point cloud data, use a point cloud segmentation model to segment the point cloud into components, dividing them into front and side views. Record the edge points of these two views, and also record the edges where the views intersect. After recording, merge and output the data to obtain the edge information of the 3D point cloud. Step six: Preprocess the edge point cloud data obtained in step five to remove noise and outliers to ensure data accuracy. Then, use common line fitting algorithms to fit these edge point cloud data into straight lines. During the fitting process, the algorithm continuously adjusts the plane parameters to find the best-fitting straight line, minimizing the distance error between the point cloud data and the fitted line. After the line fitting is completed, the edge dimensions of each plane are accurately measured through geometric calculations. Step 7: Mapping 3D coordinates to a 2D image. After calculating the object size in the image, the 3D coordinates of the edge points are mapped to the 2D image using camera intrinsics.

2. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 1, characterized in that: The specific steps for constructing the dataset for image subject segmentation in step one are as follows: Step 1: Deploy acquisition devices in multiple unfinished houses. The deployed devices are capable of reading images and depth information, as well as encoding and decoding. Steps one and two involve placing the checkerboard pattern at different angles and positions for photographing, then extracting the checkerboard corner points from each image. Using the correspondence between these corner points and their actual 3D coordinates, the camera's intrinsic focal length is estimated using the least squares method or a nonlinear optimization method. , , light center The distortion coefficients, extrinsic parameters, and camera intrinsic parameters matrix are in the form of: 3D coordinates of the corner points of the chessboard and image coordinates The relationship can be represented as: Among these, the camera's intrinsic parameters are the focal length. and , light center ; Steps 1 and 3: Collect data. The collected data includes images and depth information. The collection angle covers the entire house. The shooting perspective of the subject maintains a 3D shape, and the length, width and height ratios are normal. Step 14: Label the data collected in Step 13; Step 15 involves data cleaning of the data labeled in Step 14. The data cleaning steps include statistical analysis, rule filtering, missing value handling, duplicate data handling, noise filtering, and subject selection.

3. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 2, characterized in that: Step two, which involves constructing an image subject segmentation model, includes a feature extraction module, a subject selection module, and a decoding module. The feature extraction module extracts the overall features of the input image, the subject selection module identifies and selects the subject features in the image to highlight key information, and the decoding module parses and restores the extracted feature information to generate a visual output. The specific steps for constructing the image subject segmentation model are as follows: Step 21: Build a feature extraction module based on a two-dimensional convolutional function. This feature extraction module includes 5 network blocks, each of which consists of two convolutional layers, a Leaky ReLU function, and a MaxPooling layer. Step 22: First, input the features extracted by the feature extraction module in Step 21 into the backpropagation layer of the Conv2DBackpropInput feature input terminal to calculate the convolution operation. At the same time, input the randomly generated matrix after processing. This matrix is ​​first generated into an original constant tensor by the Const operation, then copied into three copies. Each matrix is ​​sliced, with the slice object being the first 3 matrix elements. Then, the first two elements are divided by 2 and concatenated. The concatenated elements are generated by the pack operation. Then, the concatenated elements and the previous features are put into the backpropagation layer of the convolution operation. The output value is put into a set of standard batch normalization operation layers below to complete the construction of the subject selection module. Steps two and three: A feature decoding module is built based on a two-dimensional convolution function. Each decoding layer of this feature decoding module will replace the subject selection module with the Max pooling layer as a manifestation of multiple views. Step 24: Build the loss function module. The loss function is as follows: in, It is the value of the i-th dimension of the prediction distribution. It is the value of the i-th dimension of the corresponding true value distribution. It is the dimension of the predicted distribution; Step 25: Optimize the model using an inner loop plus an outer loop; Step 26: Use transfer learning to transfer the model to category segmentation and retrain the model to adapt to different segmentation objects.

4. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 3, characterized in that: The specific steps for optimizing the model using an inner loop plus an outer loop in step two-five are as follows: First, in the inner loop, the model performs gradient descent using the training data to adjust the model parameters, setting the initial model parameters as follows: For each task Through training data Calculate the loss Then, gradient descent is performed to update the parameters, and the update formula is as follows: in, The learning rate for the inner loop. In the mission Updated parameters; Secondly, in the outer loop, the model updates the meta-model parameters using the validation loss from different tasks. Specifically, the updated parameters are used... In verifying data Calculate loss And update the meta-model parameters using these losses. The update formula for the outer loop is: in, The learning rate for the outer loop.

5. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 4, characterized in that: The specific steps for retraining the model in step two six are as follows: Step 261: Perform main body segmentation output layer transformation, retain the feature extraction layer of the pre-trained model, and use it for the category classification task; Step 262: Add a new output layer for the new task. The new output layer includes a convolutional layer and a classification layer, which are used to map features to specific class labels. Step 263: Retrain the model using the category segmentation dataset.

6. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 5, characterized in that: When retraining the model using the category segmentation dataset in step 263, the feature extraction layer is preserved. The parameters remain unchanged; only the newly added convolutional layers are updated. and classification layer The parameters, and the loss function for retraining are defined as a combination of the following two parts: Segmentation loss Used to measure the performance of a model on class segmentation tasks: Where N is the number of samples, M is the number of pixels in each sample, and the Loss() loss function is the improved cross-entropy loss function proposed in step two, four. It is the model's predicted output for the j-th pixel of the i-th sample. It is the true label of the j-th pixel of the i-th sample; Classification loss This is used to ensure the accuracy of category classification: in, The () function is the basic cross-entropy loss function, and the overall retraining loss function is defined as: , This is the balance coefficient, used to adaptively balance the importance of the two losses.

7. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 6, characterized in that: The specific steps for binarizing the image in step three are as follows: Step 31: Read the binarized mask and depth map. Specifically, read the binarized mask image and the corresponding depth map. The binarized mask image is obtained by converting the original image into a black and white image. The white area represents the subject of interest, while the black area represents the background or parts of no interest. The mask image can be generated from image processing or segmentation algorithms. Meanwhile, the depth map records the actual distance of each pixel to the camera, usually represented by grayscale values. The higher the value, the farther the distance. These two images need to be read at the same resolution and alignment to ensure correct matching of the mask and depth data. Step 32: Use the binarized mask image as a filtering tool to retain only the depth information of the main body. Specifically, for each pixel, check the value of its mask image. If the mask value is white, it means that the pixel belongs to the main body of interest. At this time, retain the corresponding depth value in the depth map. If the mask value is black, it means that the pixel belongs to the background or an unwanted part, and its depth value is ignored in this case; Step 33: Obtain camera intrinsic parameters, specifically by reading or setting the camera's intrinsic parameter matrix; Steps three and four involve calculating the point cloud. Specifically, first, depth map data is acquired. Then, the camera intrinsic parameters obtained in step three are used to convert this depth information into 3D point cloud coordinates, as shown in the following formula for each pixel. Its corresponding depth value Z can be calculated using the following formula to determine its three-dimensional spatial coordinates. : in, These are pixel coordinates on the image plane. ) are the coordinates of the optical center, ( Z is the camera's focal length, and the depth value Z is the depth data of the corresponding pixel.

8. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 7, characterized in that: The filtering steps for the point cloud in step four are as follows: First, for each point First, construct its neighborhood. The K-nearest neighbor algorithm can be used to find the K nearest points in its neighborhood. Then, calculate the average distance from the point to all points in its neighborhood. : Compare this average distance with the average distance of all points. Compare, if If the value exceeds a certain preset threshold, the point is considered an outlier and is removed. The specific steps are as follows:

1. Constructing a neighborhood: For each point in the point cloud Find the K nearest points in its neighborhood. ; 2. Calculate the average distance: Calculate the points Average distance to all points in its neighborhood ; 3. Remove outliers: Comparison With preset threshold + ,in It is the average distance of all points. It is the standard deviation. It is an adjustable parameter, if If the value is greater than the threshold, then remove the point.

9. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 8, characterized in that: The specific steps for fitting the edge point cloud data into a straight line in step six are as follows: Step 61: Perform data preprocessing, including noise removal and outlier filtering of the edge point cloud data, to ensure the accuracy and stability of the data. Step 62: The RANSAC algorithm is used to fit a straight line to the processed edge point cloud data. The algorithm continuously adjusts the plane parameters to find the straight line that best matches the point cloud data. Step 63: Through geometric calculations, measure the side length, width, and angles and distances between the planes formed by the fitted edge lines. Step 64 involves organizing and outputting the calculated dimensional parameters to provide basic data for further analysis and processing.

Images