Three-dimensional object size measurement method based on depth vision camera

Abstract

The invention discloses a three-dimensional object size measurement method based on a depth vision camera, data acquisition and model retraining are realized through steps 1 to 2, and steps 3 to 6 are initial data processing including data filtering and 3d edge segmentation. And step 7, carrying out visual operation, and displaying a plane obtained by fitting into a two-dimensional image.

Description

Technical Field

[0001] The present invention relates to a measuring method, and more particularly to a three-dimensional object size measuring method based on a depth vision camera. Background Art

[0002] With the continuous development of science and technology, three-dimensional measurement technology has been widely used in industrial manufacturing, medical health, virtual reality, and robot navigation. Traditional three-dimensional measurement methods mainly rely on technologies such as laser scanning, structured light, and stereo vision. However, these methods have some limitations, such as high cost, complex equipment, poor environmental adaptability, and limited measurement accuracy. In the field of industrial manufacturing, three-dimensional measurement technology is widely used in quality control, component inspection, and assembly accuracy verification. Although traditional laser scanning technology can provide high-precision three-dimensional data, the equipment is expensive and the maintenance cost is high. Although structured light technology can achieve fast measurement, it has high requirements for ambient light and is easily interfered by external factors. Stereo vision technology requires multiple cameras to work together, the system complexity is high, the calculation amount is large, and the real-time performance is poor.

[0003] As an emerging 3D measurement technology, the depth vision camera can quickly and accurately obtain the 3D information of an object by using infrared light and time of flight (ToF) technology. ToF technology calculates the distance of the object by measuring the time it takes for infrared light to be emitted from the camera to the object and reflected back, thereby generating a depth image. In 3D measurement, the corresponding 3D object point cloud can be generated only by depth data and camera internal parameters. In addition, due to its low cost and easy installation, the depth vision camera is very suitable for large-scale applications and can effectively reduce equipment procurement and maintenance costs. In addition, the depth vision camera has the advantages of strong real-time performance and good environmental adaptability. It can work stably under various lighting conditions and is suitable for real-time measurement and feedback in dynamic environments. The object size measurement method based on the depth camera solves the problems of high cost, complex equipment, poor environmental adaptability and limited measurement accuracy in traditional 3D measurement methods.

[0004] The current three-dimensional measurement technology of objects mainly includes: 1. Scanning the surface of the object with a laser beam to obtain high-precision three-dimensional data, which is suitable for the measurement of complex surfaces and large-sized objects, but the equipment cost is high and the scanning speed is slow. 2. Reconstructing the three-dimensional shape by projecting a grating pattern and analyzing its deformation, which has the characteristics of high precision and high-speed scanning, and is suitable for small and medium-sized objects, but has high requirements for ambient light. 3. By taking two-dimensional images of the object from multiple angles, three-dimensional reconstruction is performed using computer vision algorithms, which has low cost and wide applicability, but the accuracy depends on the image quality and algorithm complexity. None of these existing methods can adapt to the complex and changeable measurement environment and the needs of high efficiency and high precision. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a three-dimensional object size measurement method based on a depth vision camera, which can realize the size analysis of the main body of a single RGB-D image through methods such as main body segmentation, point cloud information extraction, data filtering, and data mining.

[0006] To achieve the above object, the present invention provides the following technical solution: a three-dimensional object size measurement method based on a depth vision camera, characterized in that it includes the following steps:

[0007] Step 1: Build a dataset for image subject segmentation, which includes house walls and columns, as well as the corresponding three-dimensional size and position information;

[0008] Step 2: construct an image subject segmentation model, and use the constructed image subject segmentation model to segment the image and generate a corresponding mask;

[0009] Step 3: read the mask image segmented in step 2, then select an appropriate threshold, and then apply the threshold to set the pixel values ​​above the threshold to 1, and the pixel values ​​below the threshold to 0, and finally save or display the processed binary image. After obtaining the binary mask information, first apply it to the depth map, segment and extract the depth information of the main part, and then convert the depth information into a point cloud to binarize the image;

[0010] Step 4: Filter the point cloud transformed in step 3;

[0011] Step 5: Data mining is performed on the point cloud filtered in step 4. Specifically, after obtaining the point cloud data, the point cloud segmentation model is used to segment the point cloud components, mainly into the front and side faces, and the edge points of the two faces are recorded. At the same time, the edges of the intersection of the faces are also recorded. After recording, the records are merged and output to obtain the edge information of the three-dimensional point cloud;

[0012] Step 6: Preprocess the edge point cloud data obtained in step 5 to remove noise and outliers to ensure the accuracy of the data. Then, use a common straight line fitting algorithm to fit these edge point cloud data into a straight line. During the fitting process, the algorithm continuously adjusts the plane parameters to find the best fitting straight line so that the distance error between the point cloud data and the fitting straight line is minimized. After the straight line fitting is completed, the edge size of each plane is accurately measured through geometric calculation;

[0013] Step 7: Mapping 3D coordinates to the 2D image. After calculating the size of the object in the image, the 3D coordinates of the edge points are mapped to the 2D image through the camera intrinsic parameters.

[0014] As a further improvement of the present invention, the specific steps of constructing a data set for image body segmentation in step 1 are as follows:

[0015] Step 1: Deploy acquisition equipment in multiple rough rooms. The deployed equipment must be able to read images and depth information and have sufficient transmission speed, that is, the ability to encode and decode;

[0016] Step 1 and 2: Place the checkerboard pattern at different angles and positions for shooting, then extract the checkerboard corner points in each image, and estimate the camera's intrinsic parameters (focal length f) using the least squares method or nonlinear optimization method based on the correspondence between these corner points and the actual three-dimensional coordinates. x ,f y , optical center c x ,c y ) and distortion coefficients, as well as external parameters, the camera internal parameter matrix is ​​in the form of:

[0017]

[0018] The relationship between the three-dimensional coordinates (X, Y, Z) of the checkerboard corner points and the image coordinates (u, v) can be expressed as:

[0019]

[0020] Among them, the intrinsic parameter of the camera is the focal length f x and f y , optical center c x and c y ;

[0021] Step 13: Collect data. The collected data includes image and depth information. The collected angle covers the whole house. The subject is photographed in 3D form with normal length, width and height ratio.

[0022] Step 14: label the data collected in step 13;

[0023] Step 15: Clean the data marked in step 14. The steps of data cleaning include statistical analysis, rule screening, missing value processing, duplicate data processing, noise filtering and subject selection.

[0024] As a further improvement of the present invention, the image subject segmentation model constructed in step 2 includes a feature extraction module, a subject selection module and a decoding module, wherein the feature extraction module is used to extract the overall features of the input image, the subject selection module is used to judge and select the subject features in the image, thereby highlighting the key information, and the decoding module is mainly responsible for parsing and restoring the extracted feature information to generate a visual output result. The specific steps of constructing the image subject segmentation model are as follows:

[0025] Step 21: Build a feature extraction module based on the two-dimensional convolution function. The feature extraction module includes 5 network blocks, each of which has a similar structure, consisting of two convolutional layers, a Leaky Relu function, and a MaxPooling layer.

[0026] Step 22, first input the features extracted by the feature extraction module in step 21 into the feature input terminal Conv2DBackpropInput to calculate the back propagation layer of the convolution operation, and at the same time input the randomly generated matrix after processing. The matrix is ​​first generated by the Const operation to generate the original constant tensor, and then copied into three copies, each matrix is ​​sliced, and the slice object is the first 3 matrix elements. Then, the first two elements are divided by 2 and spliced ​​together, and the elements are generated by the pack operation. Then, the spliced ​​elements and the previous features are put into the back propagation layer of the convolution operation, and the output value is put into the following set of standard batch normalization operation layers to complete the construction of the subject selection module;

[0027] Step 2 and 3: build a feature decoding module based on a two-dimensional convolution function. The subject selection module and the Max pooling layer will be replaced between each decoding layer of the feature decoding module as a manifestation of multi-view.

[0028] Step 24: Build a loss function module. The loss function built is as follows:

[0029]

[0030] Among them, p i is the value of the i-th dimension of the predicted distribution, g i is the value of the i-th dimension of the corresponding true value distribution, and d is the dimension of the predicted distribution;

[0031] Step 25, optimize the model by adding inner loop and outer loop;

[0032] Step 26: Use transfer learning to transfer the model to category segmentation to retrain the model to adapt to different segmentation objects.

[0033] As a further improvement of the present invention, the specific steps of optimizing the model by means of inner loop plus outer loop in step 25 are as follows:

[0034] First, in the inner loop, the model performs several gradient descents through a small amount of training data to quickly adjust the model parameters and set the initial model parameters to θ. For each task T i , with a small amount of training data Calculating Losses And perform gradient descent to update the parameters. The update formula is as follows:

[0035]

[0036] Among them, α is the learning rate of the inner loop, For task T i The updated parameters.

[0037] Second, in the outer loop, the model updates the meta-model parameters through the validation losses of different tasks. Specifically,

[0038] Use the updated parameters In verifying the data Calculate the loss And update the meta-model parameters θ through these losses. The update formula of the outer loop is:

[0039]

[0040] Among them, β is the learning rate of the outer loop.

[0041] As a further improvement of the present invention, the specific steps of retraining the model in step 26 are as follows: step 261, converting the main segmentation output layer, retaining the feature extraction layer of the pre-trained model, and using it for the category classification task;

[0042] Step 262, add an output layer for the new task. Add a new output layer for the category segmentation task. The new output layer includes a convolution layer and a classification layer, which are used to map features to specific category labels.

[0043] Step 263, retrain the model using the category segmentation dataset.

[0044] As a further improvement of the present invention, when the model is retrained using the category segmentation dataset in step 263, the parameters of the feature extraction layer F(·) are kept unchanged, and only the parameters of the newly added convolutional layer C(·) and classification layer H(·) are updated. The loss function of the retraining is defined as a combination of the following two parts:

[0045] Segmentation loss Used to measure the performance of the model on the category segmentation task:

[0046]

[0047] Where N is the number of samples, M is the number of pixels in each sample, and the Loss() loss function is the improved cross entropy loss function proposed in Step 4. is the model's predicted output for the jth pixel in the ith sample, is the true label of the jth pixel of the i-th sample;

[0048] Classification Loss To ensure the accuracy of category classification:

[0049]

[0050] Among them, the CrossEntropy() function is the basic cross entropy loss function, and the total retraining loss function is defined as: λ is the balance coefficient, which is used to adaptively balance the relative importance of the two parts of the loss.

[0051] As a further improvement of the present invention, the specific steps of binarizing the image in step 3 are as follows:

[0052] Step 31: read the binary mask and depth map, specifically read the binary mask image and the corresponding depth map, wherein the binary mask image is obtained by converting the original image into a black and white image, the white area represents the main part of interest, and the black area represents the background or the part of no interest, the mask image can be generated from the image processing or segmentation algorithm, at the same time, the depth map records the actual distance from each pixel to the camera, usually expressed as a grayscale value, the higher the value, the farther the distance, the two images need to be read with the same resolution and alignment to ensure the correct matching of the mask and depth data;

[0053] Step 32: Use the binary mask image as a filtering tool to retain only the depth information of the main part. Specifically, for each pixel, check the value of its mask image. If the mask value is white, it means that the pixel belongs to the main part of interest, and the corresponding depth value in the depth map is retained; if the mask value is black, it means that the pixel belongs to the background or an unnecessary part, and its depth value is ignored.

[0054] Step 33, obtaining the camera intrinsic parameters, specifically reading or setting the camera intrinsic parameter matrix;

[0055] Step 34: Calculate the point cloud. Specifically, first obtain the depth map data, and use the camera intrinsic parameters obtained in step 33 to convert these depth information into three-dimensional point cloud coordinates, as shown in the following formula. For each pixel point (u, v) and its corresponding depth value Z, its three-dimensional space coordinates (X, Y, Z) can be calculated by the following formula:

[0056]

[0057] Z=depth

[0058] Where (u, v) is the pixel coordinate on the image plane, (c x ,c y ) is the optical center coordinate, (f x,f y ) is the focal length of the camera, and the depth value Z is the depth data of the corresponding pixel.

[0059] As a further improvement of the present invention, the step of filtering the point cloud in step 4 is as follows:

[0060] First, for each point p i First, we construct its neighborhood. We can use the K nearest neighbor algorithm to find the K nearest points in its neighborhood. Then, we calculate the average distance d from the point to all points in the neighborhood. i :

[0061]

[0062] Add this average distance to the average distance of all points For comparison, if d i If the value is greater than a preset threshold, the point is considered an outlier and is removed. The specific steps are:

[0063] 1. Construct neighborhood: For each point p in the point cloud i , find the K nearest points in its neighborhood {p j |j=1,2,...,K};

[0064] 2. Calculate the average distance: Calculate point p i The average distance d to all points in its neighborhood i ;

[0065] 3. Remove outliers: Compare d i With the preset threshold d i +ασ, where is the average distance of all points, σ ​​is the standard deviation, and α is a tuning parameter. If d i If it is greater than the threshold, the point is removed.

[0066] As a further improvement of the present invention, the specific steps of fitting the edge point cloud data into a straight line in step 6 are as follows:

[0067] Step 6.1: Perform data preprocessing to remove noise and filter outliers from edge point cloud data to ensure data accuracy and stability.

[0068] Step 62: Use the RANSAC algorithm to perform straight line fitting on the processed edge point cloud data. The algorithm continuously adjusts the plane parameters to find the straight line that best matches the point cloud data.

[0069] Step 63, through geometric calculation, measure the size parameters such as the side length, width and angle and distance between the planes formed by the straight lines at the fitting edge ends;

[0070] Step 64: Organize and output the calculated size parameters to provide basic data for further analysis and processing.

[0071] The beneficial effects of the present invention are as follows: the three-dimensional object size measurement method based on a depth vision camera of this embodiment is based on a depth map and camera internal parameters, extracts features in the RGB map through a deep neural network, and introduces a small sample learning strategy, so that the model has the ability to accurately extract subject information with a small amount of annotation, recognizes all possible objects in a given RGB map, and realizes subject segmentation. Finally, a retraining strategy is used to migrate the model task from subject segmentation to category segmentation, so as to adapt to the application scenarios of variable three-dimensional object size measurement. After that, the segmented object is binarized as a depth map mask, combined with the internal parameters to generate a three-dimensional point cloud of the subject, and then through corresponding data filtering and plane fitting operations, the size information of the subject is obtained, and the coordinates are mapped to a two-dimensional image display. The size analysis of the subject of a single RGB-D image is realized through subject segmentation, point cloud information extraction, data filtering, data mining and other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is an example diagram of the simulation data acquisition of the depth camera of the present invention;

[0073] Figure 2 Schematic diagram of the subject segmentation model architecture;

[0074] Figure 3 It is a schematic diagram of the feature extraction module structure;

[0075] Figure 4 Schematic diagram of the subject selection module;

[0076] Figure 5 is a schematic diagram of a feature decoding module;

[0077] Figure 6 It is the flow chart of 3D edge segmentation algorithm;

[0078] Figure 7 Schematic diagram of the 3D coordinate mapping result. DETAILED DESCRIPTION

[0079] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0080] Reference Figure 1 As shown, a three-dimensional object size measurement method based on a depth camera in this embodiment mainly includes the following steps:

[0081] Construct a dataset for image subject segmentation, which includes house walls and columns, as well as the corresponding three-dimensional size and position information.

[0082] Step 1: Equipment deployment. Deploy acquisition equipment in multiple rough rooms. The equipment must be able to read images and depth information while having sufficient transmission speed, that is, the ability to encode and decode.

[0083] Step 2: Camera calibration. The camera needs to be calibrated before the corresponding data is collected. Calibration is the basis for accurate measurement and obtaining accurate position information. Its main function is to establish the relationship between the image pixel size and the actual size, as well as the mapping relationship between the image coordinate system and the world coordinate system. The checkerboard calibration method can be used for calibration to obtain the camera intrinsic parameters and distortion coefficients. First, place the checkerboard pattern at different angles and positions for shooting, and then extract the checkerboard corner points in each image. Through the correspondence between these corner points and the actual three-dimensional coordinates, the least squares method or nonlinear optimization method is used to estimate the camera's intrinsic parameters (focal length f x ,f y , optical center c x ,c y ) and distortion coefficients, as well as external parameters (rotation matrix R and translation matrix t). The camera internal parameter matrix is ​​in the form of:

[0084]

[0085] The relationship between the three-dimensional coordinates (X, Y, Z) of the corner points of the chessboard and the image coordinates (u, v) can be expressed as:

[0086]

[0087] It should be noted that the intrinsic parameter coefficients of different cameras in the same scene are also different, and they need to be calibrated separately.

[0088] Step 3: Collect data. The data that needs to be collected include image and depth information. The collection angle should cover the whole house as much as possible. At the same time, it is necessary to pay attention to the fact that the subject's perspective needs to maintain a 3D shape, with a normal length, width and height ratio to avoid deformation caused by perspective problems, which will affect the measurement results.

[0089] Step 4: Data annotation. The information that needs to be annotated includes whether the column is truncated or blocked, the rotation angle of the object, the two-dimensional bounding box in the image, the length, width and height of the column, the position of the column relative to the camera, the angle of the column around the y-axis, and finally the confidence of the result. The final data sample is as follows Figure 1 As shown, it includes RGB images and depth images, as well as annotation data.

[0090] Step 5: Data cleaning. Before obtaining the final data, the data needs to be cleaned to filter out data that is not suitable for model training. This includes processing outliers, missing values, duplicate data, error data, noise data, and irrelevant data. The steps of data cleaning include statistical analysis, rule screening, missing value processing, duplicate data processing, noise filtering, and subject selection. Data cleaning can improve data quality, thereby improving the training effect and accuracy of the model. The cleaned data will be used as input for model training and verification to ensure that the final model performance reaches the expected goal.

[0091] Image subject segmentation model, the model architecture is as follows Figure 2 As shown in the figure, it is mainly divided into three parts, namely feature extraction module, subject selection module and decoding module. The feature extraction module is used to extract the overall features of the input image, the subject selection module is used to judge and select the subject features in the image, so as to highlight the key information, and the decoding module is mainly responsible for parsing and restoring the extracted feature information to generate a visual output result. The specific steps of building this structure are as follows:

[0092] Step 1: Build a feature extraction module. The present invention adopts a feature extraction module based on a two-dimensional convolution function. The model structure is as follows Figure 3 As shown in Figure 1, the structure includes 5 network blocks, each of which has a similar structure, consisting of two convolutional layers, a Leaky Relu function, and a MaxPooling layer. The parameter settings of each layer are as follows: Figure 3 shown.

[0093] Step 2: Build a subject selection module. The present invention adopts a strategy of randomly generating matrix frames and calculating deviations under different depths of vision. The model structure is as follows Figure 4 , which consists of multiple operation layers. First, the features extracted by the feature extraction module in Step 1 are input into the feature input terminal Conv2DBackpropInput to calculate the back propagation layer of the convolution operation. At the same time, the randomly generated matrix after processing is input. The matrix is ​​first generated by the Const operation to generate the original constant tensor, and then copied into three copies. Each matrix is ​​sliced ​​and processed. The slice object is the first 3 matrix elements. Then the first two elements are divided by 2 and spliced ​​together. The elements are generated by the pack operation. Then the spliced ​​elements and the previous features are put into the back propagation layer of the convolution operation, and the output value is put into the following set of standard batch normalization operation layers. However, it should be noted that the back propagation operation needs to be stopped before calculating the square difference to avoid the target value being updated. Then the standardized value is passed to the activation function to obtain the result of a subject selection. It should be noted that the same operation will appear multiple times in the Step 3 decoding module.

[0094] Step 3: Build a feature decoding module. Similar to the feature extraction module, the feature decoding module is also based on a two-dimensional convolution function. However, the difference is that the subject selection module and the Max pooling layer are replaced between each decoding layer as a manifestation of multi-view. The specific structure of the model is as follows Figure 5 ,The structure is similar to Step 1, except that the pooling layer is replaced by the Topic Select layer.

[0095] Step 4: Build a loss function module. The present invention adopts an improved cross-entropy loss function. Due to the different data distributions of subject segmentation, the imbalance of data and the importance of different categories should be considered when calculating the loss. The traditional cross-entropy loss function often leads to the neglect of minority classes when processing imbalanced data. In the subject segmentation task, the number of pixels in each category may vary greatly, which will affect the training effect of the model. Therefore, when calculating the loss, it is necessary to assign different weights to each category to enhance the attention to the minority class. At the same time, the Dice coefficient or IoU (Intersection over Union) and other indicators can be combined for joint loss calculation to further improve the segmentation accuracy of the model at boundaries and small areas. In addition, for different application scenarios, it may be necessary to dynamically adjust the weight coefficient of the loss function to meet the needs of specific tasks. Through these improvement measures, the data distribution problem in subject segmentation can be handled more effectively, and the accuracy and robustness of segmentation can be improved. The specific formula is as follows:

[0096]

[0097] where p i is the value of the i-th dimension of the predicted distribution, g i is the value of the i-th dimension of the corresponding true value distribution, and d is the dimension of the predicted distribution.

[0098] Step 5: Due to the high annotation cost of semantic segmentation, the present invention adopts a small sample learning strategy and introduces a meta-learning method, so that the model has the ability to accurately extract subject information with a small amount of annotated data. Meta-learning can train a meta-model to enable it to quickly adapt to new tasks, thereby achieving efficient subject information extraction with a small amount of annotated data. Specifically, the training in the present invention includes two main optimization processes: inner loop and outer loop.

[0099] First, in the inner loop, the model performs several gradient descents on a small amount of training data to quickly adjust the model parameters. The initial model parameters are set to θ. For each task T i , with a small amount of training data Calculating Losses And perform gradient descent to update the parameters. The update formula is as follows:

[0100]

[0101] Among them, α is the learning rate of the inner loop, For task T i The updated parameters.

[0102] Second, in the outer loop, the model updates the meta-model parameters using the validation losses of different tasks.

[0103] For example, using the updated parameters In verifying the data Calculate the loss And update the meta-model parameters θ through these losses. The update formula of the outer loop is:

[0104]

[0105] Among them, β is the learning rate of the outer loop.

[0106] The present invention simulates multiple tasks during the training phase, so that the model learns a good initialization parameter θ, so that it can achieve good performance on new tasks with only a small amount of gradient updates. This strategy significantly improves the performance of the model on small sample data sets and enhances its generalization ability on different tasks.

[0107] Step 6: Model retraining strategy. Since the requirements for segmenting objects under different tasks are different, the model needs to be retrained to meet the requirements of different tasks. The present invention adopts a transfer learning method to transfer the subject segmentation model to category segmentation to adapt to the situations of different segmentation objects.

[0108] The present invention realizes the task conversion of the model through the retraining strategy, so as to maintain efficient performance in the new task. Specifically, it includes the following steps:

[0109] 1. Subject segmentation output layer conversion, first retain the feature extraction layer of the pre-trained model and use it for the category segmentation task. The retained feature extraction layer contains the ability to efficiently identify the target area in the image. In the subject segmentation task, the pre-trained model has learned some common features, which are also useful for the category segmentation task.

[0110] 2. Add output layers for new tasks. Add new output layers for the category segmentation task. The parameters of these output layers are initialized according to the specific requirements of the category segmentation task. The new output layer usually includes a convolutional layer and a classification layer to map features to specific category labels.

[0111] 3. Retrain the model using the category segmentation dataset. During retraining, keep the parameters of the feature extraction layer F(·) unchanged and only update the parameters of the newly added convolutional layer C(·) and classification layer H(·). The loss function of retraining can be defined as a combination of the following two parts:

[0112] Segmentation loss Used to measure the performance of the model on the category segmentation task:

[0113]

[0114] Where N is the number of samples, M is the number of pixels in each sample, and the Loss() loss function is the improved cross entropy loss function proposed in Step 4. is the model's predicted output for the jth pixel in the ith sample, is the true label of the jth pixel of the i-th sample.

[0115] Classification Loss To ensure the accuracy of category classification:

[0116]

[0117] Among them, the CrossEntropy() function is the basic cross entropy loss function. The total retraining loss function is defined as: λ is the balance coefficient, which is used to adaptively balance the relative importance of the two parts of the loss.

[0118] Although the image has generated a corresponding mask after segmentation, the value of the main part has not yet been unified to 1, so the image needs to be binarized. The specific steps are as follows: first read the segmented mask image, then select an appropriate threshold, then apply the threshold to set the pixel values ​​above the threshold to 1, and the pixel values ​​below the threshold to 0, and finally save or display the processed binary image. After obtaining the binary mask information, first apply it to the depth map, segment and extract the depth information of the main part, and then convert the depth information into a point cloud. The specific steps are as follows:

[0119] Step 1: Read the binary mask and depth map. First, you need to read the binary mask image and the corresponding depth map. The binary mask image is obtained by converting the original image into a black and white image, where the white area (usually a value of 255) represents the subject of interest, and the black area (value 0) represents the background or uninteresting part. The mask image can be generated from image processing or segmentation algorithms. At the same time, the depth map records the actual distance of each pixel from the camera, usually expressed as a grayscale value, where the higher the value, the farther the distance. The two images need to be read at the same resolution and alignment to ensure correct matching of the mask and depth data.

[0120] Step 2: Apply mask. The step of applying mask is to use the binary mask image as a filtering tool to retain only the depth information of the main part. Specifically, for each pixel, check the value of its mask image. If the mask value is white (255), it means that the pixel belongs to the main part of interest, and the corresponding depth value in the depth map is retained; if the mask value is black (0), it means that the pixel belongs to the background or unnecessary part, and its depth value is ignored. In this way, the mask filters out the depth information of the non-main part in the depth map and generates a depth map that only contains the depth information of the main part. This processing step is very important for subsequent point cloud generation and subject analysis because it ensures that only the area of ​​interest is further processed.

[0121] Step 3: Get the camera intrinsic parameters. The camera intrinsic parameters are important parameters used to convert two-dimensional image coordinates into three-dimensional space coordinates, including focal length and optical center coordinates. First, you need to read or set the camera's intrinsic parameter matrix. The intrinsic parameter matrix usually includes the focal length f x ,f y , optical center c x ,c y The focal length represents the magnification of the camera lens, while the optical center coordinates represent the center point of the image plane. These parameters can be obtained through the camera calibration process, which requires taking a specific calibration plate and using these calibration images to calculate the accurate intrinsic parameter matrix through the algorithm. When reading these parameters, they can be extracted from the camera's calibration file, or if in the application, the user can set them through the settings file or using the SDK API.

[0122] Step 4: Calculate point cloud. The process of calculating point cloud depends on camera intrinsic parameters and depth map. First, obtain the depth map data, which records the distance from each pixel to the camera. Using camera intrinsic parameters, these depth information can be converted into three-dimensional point cloud coordinates. Specifically, for each pixel point (u, v) and its corresponding depth value Z, its three-dimensional space coordinates (X, Y, Z) can be calculated by the following formula:

[0123]

[0124] Z=depth

[0125] Where (u, v) is the pixel coordinate on the image plane, (c x ,c y ) is the optical center coordinate, (f x ,f y) is the focal length of the camera, and the depth value Z is the depth data of the corresponding pixel. These calculations convert the depth information of each pixel into 3D coordinates, thereby generating a complete 3D point cloud data set. These point cloud data can be used for further dimensional analysis, object recognition and other applications.

[0126] Since outliers are inevitable in the process of point cloud extraction, these outliers will affect the subsequent data processing, so it is necessary to filter the point cloud. First, for each point p i First, we construct its neighborhood. We can use the K nearest neighbor algorithm to find the K nearest points in its neighborhood. Then, we calculate the average distance d from the point to all points in the neighborhood. i :

[0127]

[0128] Add this average distance to the average distance of all points For comparison, if d i If the value is greater than a preset threshold, the point is considered an outlier and is removed. The specific steps are as follows:

[0129] 1. Construct neighborhood: For each point p in the point cloud i , find the K nearest points in its neighborhood {p j |j=1,2,...,K}.

[0130] 2. Calculate the average distance: Calculate point p i The average distance d to all points in its neighborhood i .

[0131] 3. Remove outliers: Compare d i With the preset threshold d i +ασ(where is the average distance of all points, σ ​​is the standard deviation, and α is a tuning parameter). If d i If it is greater than the threshold, the point is removed.

[0132] Through these steps, the noise and outliers in the point cloud can be significantly reduced, the quality of the point cloud data can be improved, and a more reliable data foundation can be provided for subsequent 3D reconstruction, feature extraction and analysis.

[0133] The filtered point cloud is subjected to data mining. The purpose of this step is to obtain the edge information of the subject. The specific algorithm flow chart is as follows Figure 6 As shown in the figure. First, after obtaining the point cloud data, the point cloud segmentation model is used to segment the point cloud components, mainly into the front and side faces, and the edge points of the two faces are recorded. At the same time, the edges of the intersection of the faces are also recorded. After recording, they are merged and output to obtain the edge information of the three-dimensional point cloud.

[0134] Next, straight line fitting is performed to calculate the edge size. The key to this step is to use a fitting algorithm to fit the extracted edge point cloud data into multiple straight lines, so as to calculate the actual size of each edge. Specifically, the edge point cloud data is first preprocessed to remove noise and outliers to ensure the accuracy of the data. Then, common straight line fitting algorithms, such as the RANSAC (Random Sampling Consensus) algorithm, are used to fit these edge point cloud data into straight lines. During the fitting process, the algorithm continuously adjusts the plane parameters to find the best fitting straight line so that the distance error between the point cloud data and the fitting straight line is minimized. After the straight line fitting is completed, the edge size of each plane is accurately measured through geometric calculations.

[0135] The detailed steps for calculating edge size by plane fitting are as follows:

[0136] Step 1: Data preprocessing. Remove noise and filter outliers from edge point cloud data to ensure data accuracy and stability.

[0137] Step 2: Plane fitting. The RANSAC algorithm is used to perform straight line fitting on the processed edge point cloud data. The algorithm continuously adjusts the plane parameters to find the straight line that best matches the point cloud data.

[0138] Step 3: Dimension calculation: Through geometric calculation, the length and width of the plane composed of the straight lines at the fitting edge ends, as well as the angles and distances between the planes and other dimension parameters are measured.

[0139] Step 4: Data output. The calculated size parameters are sorted and output to provide basic data for further analysis and processing.

[0140] 3D coordinates are mapped to 2D images. After calculating the size of the object in the image, the 3D coordinates of the edge points can be mapped to the 2D image through the camera intrinsic parameters, such as Figure 7 shown.

[0141] In summary, the present invention realizes data collection and model retraining through steps 1) to 2), steps 3) to 6) are the processing of initial data, including data filtering, 3D edge segmentation, straight line fitting and other operations to obtain specific object size parameters, and step 7) is a visualization operation to display the fitted plane in a two-dimensional image.

[0142] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A three-dimensional object size measurement method based on a depth vision camera, characterized in that: The steps include: Step 1: Build a dataset for image subject segmentation, which includes house walls and columns, as well as the corresponding three-dimensional size and position information; Step 2: construct an image subject segmentation model, and use the constructed image subject segmentation model to segment the image and generate a corresponding mask; Step 3: read the mask image segmented in step 2, then select an appropriate threshold, and then apply the threshold to set the pixel values ​​above the threshold to 1, and the pixel values ​​below the threshold to 0, and finally save or display the processed binary image. After obtaining the binary mask information, first apply it to the depth map, segment and extract the depth information of the main part, and then convert the depth information into a point cloud to binarize the image; Step 4: Filter the point cloud transformed in step 3; Step 5: Data mining is performed on the point cloud filtered in step 4. Specifically, after obtaining the point cloud data, the point cloud segmentation model is used to segment the point cloud components, mainly into the front and side faces, and the edge points of the two faces are recorded. At the same time, the edges of the intersection of the faces are also recorded. After recording, the records are merged and output to obtain the edge information of the three-dimensional point cloud; Step 6: Preprocess the edge point cloud data obtained in step 5 to remove noise and outliers to ensure the accuracy of the data. Then, use a common straight line fitting algorithm to fit these edge point cloud data into a straight line. During the fitting process, the algorithm continuously adjusts the plane parameters to find the best fitting straight line so that the distance error between the point cloud data and the fitting straight line is minimized. After the straight line fitting is completed, the edge size of each plane is accurately measured through geometric calculation; Step 7: Mapping 3D coordinates to the 2D image. After calculating the size of the object in the image, the 3D coordinates of the edge points are mapped to the 2D image through the camera intrinsic parameters.

2. The three-dimensional object size measurement method based on a depth vision camera according to claim 1, characterized in that: The specific steps of constructing a data set for image subject segmentation in step 1 are as follows: Step 1: Deploy acquisition equipment in multiple rough rooms. The deployed equipment must be able to read images and depth information and have sufficient transmission speed, that is, the ability to encode and decode; Step 1 and 2: Place the checkerboard pattern at different angles and positions for shooting, then extract the checkerboard corner points in each image, and estimate the camera's intrinsic parameters (focal length f) using the least squares method or nonlinear optimization method based on the correspondence between these corner points and the actual three-dimensional coordinates. x ,f y , optical center c x ,c y ) and distortion coefficients, as well as external parameters, the camera internal parameter matrix is ​​in the form of: The relationship between the three-dimensional coordinates (X, Y, Z) of the checkerboard corner points and the image coordinates (u, v) can be expressed as: Among them, the intrinsic parameter of the camera is the focal length f x and f y , optical center c x and c y ; Step 13: Collect data. The collected data includes image and depth information. The collected angle covers the whole house. The subject is photographed in 3D form with normal length, width and height ratio. Step 14: label the data collected in step 13; Step 15: Clean the data marked in step 14. The steps of data cleaning include statistical analysis, rule screening, missing value processing, duplicate data processing, noise filtering and subject selection.

3. The three-dimensional object size measurement method based on a depth vision camera according to claim 2, characterized in that: The image subject segmentation model constructed in step 2 includes a feature extraction module, a subject selection module and a decoding module, wherein the feature extraction module is used to extract the overall features of the input image, the subject selection module is used to judge and select the subject features in the image, thereby highlighting the key information, and the decoding module is mainly responsible for parsing and restoring the extracted feature information to generate a visual output result. The specific steps of constructing the image subject segmentation model are as follows: Step 21: Build a feature extraction module based on the two-dimensional convolution function. The feature extraction module includes 5 network blocks, each of which has a similar structure, consisting of two convolutional layers, a Leaky Relu function, and a MaxPooling layer. Step 22, first input the features extracted by the feature extraction module in step 21 into the feature input terminal Conv2DBackpropInput to calculate the back propagation layer of the convolution operation, and at the same time input the randomly generated matrix after processing. The matrix is ​​first generated by the Const operation to generate the original constant tensor, and then copied into three copies, each matrix is ​​sliced, and the slice object is the first 3 matrix elements. Then, the first two elements are divided by 2 and spliced ​​together, and the elements are generated by the pack operation. Then, the spliced ​​elements and the previous features are put into the back propagation layer of the convolution operation, and the output value is put into the following set of standard batch normalization operation layers to complete the construction of the subject selection module; Step 2 and 3: build a feature decoding module based on a two-dimensional convolution function. The subject selection module and the Max pooling layer will be replaced between each decoding layer of the feature decoding module as a manifestation of multi-view. Step 24: Build a loss function module. The loss function built is as follows: Among them, p i is the value of the i-th dimension of the predicted distribution, g i is the value of the i-th dimension of the corresponding true value distribution, and d is the dimension of the predicted distribution; Step 25, optimize the model by adding inner loop and outer loop; Step 26: Use transfer learning to transfer the model to category segmentation to retrain the model to adapt to different segmentation objects.

4. The three-dimensional object size measurement method based on a depth vision camera according to claim 3 is characterized in that: The specific steps of optimizing the model by means of inner loop plus outer loop in step 25 are as follows: First, in the inner loop, the model performs several gradient descents through a small amount of training data to quickly adjust the model parameters and set the initial model parameters to θ. For each task T i , with a small amount of training data Calculating Losses And perform gradient descent to update the parameters. The update formula is as follows: Among them, α is the learning rate of the inner loop, For task T i The updated parameters. Second, in the outer loop, the model updates the meta-model parameters using the validation loss of different tasks. Specifically, the updated parameters In verifying the data Calculate the loss And update the meta-model parameters θ through these losses. The update formula of the outer loop is: Among them, β is the learning rate of the outer loop.

5. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 4, characterized in that: The specific steps for retraining the model in step 26 are as follows: Step 261, perform subject segmentation output layer conversion, retain the feature extraction layer of the pre-trained model, and use it for the category classification task; Step 262, add an output layer for the new task. Add a new output layer for the category segmentation task. The new output layer includes a convolution layer and a classification layer, which are used to map features to specific category labels. Step 263, retrain the model using the category segmentation dataset.

6. The three-dimensional object size measurement method based on a depth vision camera according to claim 5, characterized in that: When the model is retrained using the category segmentation dataset in step 263, the parameters of the feature extraction layer F(·) are kept unchanged, and only the parameters of the newly added convolutional layer C(·) and classification layer H(·) are updated. The loss function of the retraining is defined as a combination of the following two parts: Segmentation loss Used to measure the performance of the model on the category segmentation task: Where N is the number of samples, M is the number of pixels in each sample, and the Loss() loss function is the improved cross entropy loss function proposed in Step 4. is the model's predicted output for the jth pixel in the ith sample, is the true label of the jth pixel of the i-th sample; Classification Loss To ensure the accuracy of category classification: Among them, the CrossEntropy() function is the basic cross entropy loss function, and the total retraining loss function is defined as: λ is the balance coefficient, which is used to adaptively balance the relative importance of the two parts of the loss.

7. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 6, characterized in that: The specific steps of binarizing the image in step 3 are as follows: Step 31: read the binary mask and depth map, specifically read the binary mask image and the corresponding depth map, wherein the binary mask image is obtained by converting the original image into a black and white image, the white area represents the main part of interest, and the black area represents the background or the part of no interest, the mask image can be generated from the image processing or segmentation algorithm, at the same time, the depth map records the actual distance from each pixel to the camera, usually expressed as a grayscale value, the higher the value, the farther the distance, the two images need to be read with the same resolution and alignment to ensure the correct matching of the mask and depth data; Step 32: Use the binary mask image as a filtering tool to retain only the depth information of the main part. Specifically, for each pixel, check the value of its mask image. If the mask value is white, it means that the pixel belongs to the main part of interest, and the corresponding depth value in the depth map is retained. If the mask value is black, it means that the pixel belongs to the background or an unnecessary part, and its depth value is ignored; Step 33, obtaining the camera intrinsic parameters, specifically reading or setting the camera intrinsic parameter matrix; Step 34: Calculate the point cloud. Specifically, first obtain the depth map data, and use the camera intrinsic parameters obtained in step 33 to convert these depth information into three-dimensional point cloud coordinates, as shown in the following formula. For each pixel point (u, v) and its corresponding depth value Z, its three-dimensional space coordinates (X, Y, Z) can be calculated by the following formula: Z=depth Where (u, v) is the pixel coordinate on the image plane, (c x ,c y ) is the optical center coordinate, (f x ,f y ) is the focal length of the camera, and the depth value Z is the depth data of the corresponding pixel.

8. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 7, characterized in that: The steps of filtering the point cloud in step 4 are as follows: First, for each point p i First, we construct its neighborhood. We can use the K nearest neighbor algorithm to find the K nearest points in its neighborhood. Then, we calculate the average distance d from the point to all points in the neighborhood. i : Add this average distance to the average distance of all points For comparison, if d i If the value is greater than a preset threshold, the point is considered an outlier and is removed. The specific steps are:

1. Construct neighborhood: For each point p in the point cloud i , find the K nearest points in its neighborhood {p j |j=1,2,...,K}; 2. Calculate the average distance: Calculate point p i The average distance d to all points in its neighborhood i ; 3. Remove outliers: Compare d i With the preset threshold d i +ασ, where is the average distance of all points, σ ​​is the standard deviation, and α is a tuning parameter. If d i If it is greater than the threshold, the point is removed.

9. The method for measuring the size of a three-dimensional object based on a depth vision camera according to claim 8, characterized in that: The specific steps of fitting the edge point cloud data into a straight line in step 6 are as follows: Step 6.1: Perform data preprocessing to remove noise and filter outliers from edge point cloud data to ensure data accuracy and stability. Step 62: Use the RANSAC algorithm to perform straight line fitting on the processed edge point cloud data. The algorithm continuously adjusts the plane parameters to find the straight line that best matches the point cloud data. Step 63, through geometric calculation, measure the size parameters such as the side length, width and angle and distance between the planes formed by the straight lines at the fitting edge ends; Step 64: Organize and output the calculated size parameters to provide basic data for further analysis and processing.

Images