A volume estimation method, device, electronic device and storage medium

Abstract

The application discloses a volume estimation method and device, electronic equipment and storage medium. The method comprises the following steps: acquiring a target image containing a to-be-measured object; determining a first image position of the to-be-measured object in the target image based on a first color feature and a first depth feature of the target image; obtaining target point cloud data of the to-be-measured object based on the first image position; and obtaining a volume of the to-be-measured object based on the target point cloud data. In this way, the application can obtain an accurate object volume.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a volume estimation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Measuring the volume of objects is common in people's daily lives. For example, when people mail packages, they need to use various courier companies, which can charge fees by measuring the volume of the packages. When e-commerce platforms sort and assemble items and make precise deliveries, accurate estimation of package volume can improve delivery efficiency and reduce delivery decision errors.

[0003] Currently, the estimation of object volume still relies on relatively traditional methods, such as direct measurement with instruments or integration of surface area. However, these traditional methods struggle to accurately measure the volume of complex and irregular objects. In other words, current industrial algorithms for measuring object volume cannot guarantee accuracy for complex and irregular objects. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a volume estimation method, apparatus, electronic device, and storage medium that can obtain an accurate volume of an object.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a volume estimation method, which includes acquiring a target image containing an object to be measured; determining a first image position of the object to be measured in the target image based on a first color feature and a first depth feature of the target image; obtaining target point cloud data of the object to be measured based on the first image position; and obtaining the volume of the object to be measured based on the target point cloud data.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a volume estimation apparatus, comprising: an acquisition module for acquiring a target image containing an object to be measured; a position determination module for determining a first image position of the object to be measured in the target image based on a first color feature and a first depth feature of the target image; a point cloud generation module for obtaining target point cloud data of the object to be measured based on the first image position; and a volume prediction module for obtaining the volume of the object to be measured based on the target point cloud data.

[0007] To address the aforementioned technical problems, a third aspect of this application provides a volume estimation device, which includes a memory and a processor coupled to each other; the memory stores program instructions; and the processor executes the program instructions stored in the memory to implement the method described in the first aspect.

[0008] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium for storing program instructions that can be executed to implement the method described in the first aspect.

[0009] The beneficial effects of this application are as follows: Unlike existing technologies, this application determines the first image position of the object to be measured in the target image based on the first color feature and the first depth feature of the target image. Based on the first image position, target point cloud data of the object to be measured is obtained, and based on the target point cloud data, the volume of the object to be measured is obtained. By combining the first color feature and the first depth feature, a more accurate first image position can be obtained, thereby obtaining more accurate target point cloud data and the volume of the object to be measured. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the first embodiment of the volume estimation method provided in this application;

[0011] Figure 2 This is a flowchart illustrating one implementation method of step S120;

[0012] Figure 3 This is a schematic diagram of the semantic segmentation model;

[0013] Figure 4 This is a flowchart illustrating one implementation method of step S130;

[0014] Figure 5 This is a flowchart illustrating one implementation method of step S140;

[0015] Figure 6 This is a flowchart illustrating another embodiment of step S140;

[0016] Figure 7 This is a flowchart illustrating the second embodiment of the volume estimation method provided in this application;

[0017] Figure 8 This is a schematic diagram of the framework of one embodiment of the volume estimation device provided in this application;

[0018] Figure 9 This is a schematic diagram of the framework structure of one embodiment of the electronic device provided in this application;

[0019] Figure 10 This is a schematic diagram of a framework of one embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] It should be noted that the embodiments of this application contain descriptions involving "first," "second," etc., which are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] Please see Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the volume estimation method provided in this application. The method includes:

[0024] S110: Acquire the target image containing the object to be tested.

[0025] In one implementation, a Time of Flight (TOF) camera can be used to capture images of the object under test from a bird's-eye view, obtaining an initial image containing the object. Specifically, a depth TOF lens, such as the Kinect V2, is used for imaging. This lens continuously sends light pulses to the object under test, and then a sensor receives the light reflected back from the object. The distance to the target object is obtained by detecting the time of flight (round trip) of the light pulses. Capturing the object under test from a bird's-eye view allows for the acquisition of information about the most facets of the object.

[0026] The initial image can include an initial color image and an initial depth image. Both the initial color image and the initial depth image can be obtained by capturing images of the object under test using the same acquisition parameters of a ToF camera. These same acquisition parameters include the same viewing angle, the same field of view, and the same acquisition time. The initial color image and the initial depth image have been aligned internally within the camera.

[0027] After obtaining the initial color image and initial depth image, an object detection network (such as a lightweight network like YOLOv4, YOLOv5, or Nanodet) can be used to perform object detection on the initial color image or the initial color image to obtain the detection box of the object to be tested in the initial color image. The detection box can be the bounding box of the object to be tested. The image portion of the initial color image that is located within the detection box is cropped to obtain the target color image, and the image portion of the initial depth image that is located within the detection box is cropped to obtain the target depth image.

[0028] In one specific implementation, the user can perform object detection on the initial color image to obtain the bounding box of the object to be tested in the initial color image. Since the initial depth image and the initial color image are obtained by capturing the object to be tested using the same acquisition parameters of a ToF camera, the bounding box of the object to be tested in the initial depth image can be obtained based on the position coordinates of the upper left corner or center point of the bounding box in the initial color image and the pixel width and height of the object to be tested. It is understood that in other implementations, the user can also perform object detection on the initial depth image and then use the position coordinates of the upper left corner or center point of the bounding box in the initial depth image and the pixel width and height of the object to be tested to obtain the bounding box of the object to be tested in the initial color image.

[0029] S120: Determine the first image position of the object to be tested in the target image based on the first color feature and the first depth feature of the target image.

[0030] In one embodiment, feature extraction is performed on the target depth image and the target color image to obtain a first color feature and a first depth feature. The first color feature and the first depth feature are then fused to obtain a target fused feature. Semantic segmentation is performed on the target fused feature to obtain the first image position of the object under test in the target depth image and the target color image. The target depth image and the target color image can be obtained by processing and cropping initial depth images and initial color images captured by a camera under the same acquisition parameters. Therefore, the first image position of the object under test in the target color image is the same as the first image position of the object under test in the target depth image.

[0031] S130: Based on the position of the first image, obtain the target point cloud data of the object to be tested.

[0032] In one embodiment, after obtaining the first image position, a first local image corresponding to the first image position is extracted from the target image. The first local image is then transformed using the camera's intrinsic parameters to obtain initial point cloud data, which is then used as the target point cloud data. In another embodiment, the initial point cloud data can be projected onto a second plane to obtain a second projected image. The second color feature and second depth feature of the second projected image are used to determine the second image position of the object to be measured in the second projected image. A second local image corresponding to the second image position is extracted from the second projected image. The second local image is then transformed using the camera's intrinsic parameters to obtain the target point cloud data.

[0033] The target image includes a target color image and a target depth image. Understandably, users can select at least one of the target color image and target depth image as needed, extract a first local image corresponding to the first image location, and obtain target point cloud data. The first local image represents the first projection image of the object under test on a first plane perpendicular to a preset viewing angle. The preset viewing angle can be a bird's-eye view. Assuming the object is placed on the ground, a three-dimensional Cartesian coordinate system can be established with one corner of the object as the origin. The first plane can be the XY plane perpendicular to the bird's-eye view, and the second plane can be the YZ plane parallel to the bird's-eye view.

[0034] S140: Based on the target point cloud data, obtain the volume of the object to be measured.

[0035] In one embodiment, a volume prediction model predicts the volume of the object to be measured based on target point cloud data. Specifically, the volume prediction model can be a point cloud deep network (such as PointNet or PointNet++). The target point cloud data is input into the volume prediction model, downsampled to obtain sampled point cloud data, divided into several blocks, and point cloud features corresponding to the several blocks are obtained. The volume of the object to be measured is obtained using the point cloud features of the several blocks.

[0036] The above method involves acquiring a target image containing the object to be measured, determining the first image position of the object in the target image based on the first color feature and the first depth feature of the target image, obtaining the target point cloud data of the object based on the first image position, and then obtaining the volume of the object based on the target point cloud data. By combining the first color feature and the first depth feature, a more accurate first image position can be obtained, thereby obtaining more accurate target point cloud data and the volume of the object to be measured.

[0037] Please refer to the following: Figure 2 and Figure 3 , Figure 2 This is a flowchart illustrating one embodiment of step S120. Figure 3This is a schematic diagram of the semantic segmentation model; step S120 may include:

[0038] S221: Extract the first color feature and the first depth feature from the target image.

[0039] S222: Fuse the first color feature and the first depth feature to obtain the target fused feature.

[0040] S223: Perform semantic segmentation on the target fusion features to obtain the first image position.

[0041] In one embodiment, a semantic segmentation model extracts features from the target image to obtain first color features and first depth features. The target color image and target depth image contained in the target image are input into the semantic segmentation model, which may include three branches: a color branch, a depth branch, and a fusion branch. Feature extraction is performed on the target color image in the color branch. Specifically, multiple convolutions with max pooling are performed in the color branch to extract and downsample the first color features. Similarly, multiple convolutions with max pooling are performed in the depth branch to extract and downsample the first depth features, resulting in several feature groups at different scales. Each feature group includes the first color feature and the first depth feature corresponding to one scale.

[0042] Furthermore, the semantic segmentation model can include a multi-layer perceptron (MLP) to obtain first color features and first depth features. The first color features and first depth features in the feature group are then fused in the fusion branch to obtain the initial fused features corresponding to each feature group. The fusion of the first color features and first depth features can be performed immediately after obtaining a feature group. For example, the color and depth branches extract a first feature group with a scale of 112*112 in the first extraction and a first feature group with a scale of 56*56 in the second extraction. After the first extraction, the first color features and first depth features contained in the first feature group can be fused using a concat method to obtain the first initial fused features. The second initial fused features can be obtained in the same way after the second extraction. Using the concat method for fusion can increase the number of features (channels) describing the image itself.

[0043] After obtaining the initial fusion features corresponding to each feature group, each initial fusion feature can be extracted through multiple convolutions to obtain new fusion features. These new fusion features can then be fused using an "add" method to obtain the target fusion feature. Using the "add" method for fusion can increase the amount of information describing the image features.

[0044] Finally, the target fusion features are decoded. Specifically, the target fusion features are deconvolved multiple times. The deconvolution can use methods such as nearest neighbor interpolation and bilinear interpolation to generate a probability map with the same size as the target image. Each pixel in the probability map has a target value. By judging the size of the target value and the preset value, it can be determined whether the pixel corresponding to the target value is the position of the object to be tested. In other words, the first projection image of the object to be tested on the first plane perpendicular to the preset viewing direction can be obtained. The first projection image represents the position of the object to be tested. In one embodiment, the first projection image can be represented by a mask.

[0045] In this embodiment, the accurate position of the object under test can be obtained by fusing the first depth feature and the first color feature.

[0046] Understandably, the semantic segmentation model can be trained before use. Specifically, a large number of training images can be input, including the labeled positions of the test object in the training images. The CE loss is calculated based on the training positions of the test object in the training images obtained by the semantic segmentation model from predicting them, and the parameters of the semantic segmentation model are adjusted according to the CE loss.

[0047] Please see Figure 4 , Figure 4 This is a flowchart illustrating one embodiment of step S130, which may include:

[0048] S431: Extract the first local image corresponding to the first image position from the target image.

[0049] In one embodiment, after determining the first image position of the object to be tested in the target image in step S120, a first local image corresponding to the first image position can be extracted from the target image. That is, a first local color image corresponding to the first image position is extracted from the target color image, and a first local depth image corresponding to the first image position is extracted from the target depth image. The first local image represents a first projection image of the object to be tested on a first plane perpendicular to a preset viewing angle. Specifically, the preset viewing angle can be a bird's-eye view. Assuming the object is placed on the ground, a three-dimensional rectangular coordinate system can be established with one corner of the object as the origin. The first plane can be an XY plane perpendicular to the bird's-eye view, and the second plane can be a YZ plane parallel to the bird's-eye view.

[0050] S432: Using the camera's intrinsic parameters, the first local image is transformed to obtain initial point cloud data.

[0051] In one embodiment, initial point cloud data is obtained by converting the first local color image and the first local depth image using the camera's intrinsic parameters. Each spatial point in the initial point cloud data contains both color and depth information. Specifically, Formula 1 can be used to convert the first local image to obtain the initial point cloud data, where x, y, and z are the coordinates of the point cloud data, u and v are the corresponding pixel coordinates, and f... x f y These refer to the camera's focal length on the x and y axes, respectively. x c y The center of the camera's aperture. This is the camera intrinsic parameter matrix.

[0052]

[0053] S433: Project the initial point cloud data onto the second plane to obtain the second projected image.

[0054] The second plane can be either the YZ plane or the XZ plane. In one embodiment, the initial point cloud data is projected onto the YZ plane to obtain a second projected image. Specifically, the color information of each spatial point in the initial point cloud data is projected onto the second plane to obtain a color projected image; and the depth information of each spatial point in the initial point cloud data is projected onto the second plane to obtain a depth projected image.

[0055] S434: Using the second color feature and the second depth feature of the second projection image, determine the second image position of the object to be measured in the second projection image.

[0056] In one embodiment, the second projected image may include a color projected image and a depth projected image. The color projected image and the depth projected image are input into a semantic segmentation model to obtain second color features and second depth features, thereby determining the second image position of the image to be tested within the second projected image. The step of obtaining the second image position of the image to be tested within the second projected image is similar to step S120. For specific implementation details, please refer to the detailed description of the above-described embodiment of step S120, which will not be repeated here.

[0057] In another embodiment, the user can perform target detection on the second projected image to obtain a detection box for the object to be tested in the second projected image. The image portion of the second projected image located within the detection box is then cropped to obtain a second target projected image. The second target projected image includes a second target depth projected image and a second target color projected image. The second target depth projected image and the second target color projected image are input into a semantic segmentation model to obtain the second image position of the object to be tested in the second projected image.

[0058] S435: Extract the second local image corresponding to the position of the second image from the second projected image.

[0059] Specifically, extracting a second local image corresponding to the second image position from the second projected image includes: extracting a second local color image corresponding to the second image position from the color projected image, and extracting a second local depth image corresponding to the second image position from the depth projected image.

[0060] S436: Using the camera's intrinsic parameters, the second local image is transformed to obtain the target point cloud data.

[0061] In one embodiment, the target point cloud data is obtained by converting the second local color image and the second local depth image using the camera's intrinsic parameters. Specifically, the second local image can be converted using the same steps as in step S432 to obtain the target point cloud data.

[0062] By performing the two point cloud transformations described above, the object to be tested in the target image can be separated from the background, and the shape of the object to be tested can be obtained.

[0063] Please see Figure 5 , Figure 5 This is a flowchart illustrating one embodiment of step S140, which may include:

[0064] S541: Downsample the target point cloud data to obtain sampled point cloud data.

[0065] In one embodiment, a downsampling layer of a volume prediction model can be used to downsample the target point cloud data. Specifically, the farthest point sampling can be used to make the sampled points as far apart as possible, thereby making the downsampling process more uniform. The original N(d+C) is reduced to N1(d+C), where N is the number of target point clouds; N1 is the number of target point clouds after the downsampling layer; d is the coordinate dimension, which is 3 in this embodiment; and C is other features such as normal vectors, which is 0 in this embodiment.

[0066] S542: Divide the sampled point cloud data into several blocks.

[0067] Furthermore, the sampled point cloud data is divided into several blocks using the combined layers of the volume prediction network. Each block contains a preset number of sampled point clouds, and the blocks are used as the unit to input the prediction layer of the volume prediction model.

[0068] S543: Determine the volume of the object to be measured by using the point cloud features corresponding to the point cloud data of each block.

[0069] In the prediction layer of the volume prediction model, feature extraction is performed on the point cloud data of each block to obtain the point cloud features corresponding to the point cloud data of each block. The point cloud features are then fed into the fully connected layer to obtain the volume of the object.

[0070] Please refer to the following: Figure 3 and Figure 6 , Figure 6 This is a flowchart illustrating another embodiment of step S140, which may include:

[0071] S641: Downsample the target point cloud data to obtain sampled point cloud data.

[0072] S642: Divide the sampled point cloud data into several blocks.

[0073] S643: For each block, find the key points of the block among the spatial points belonging to the block in the sampled point cloud data.

[0074] S644: Select no more than a preset threshold of spatial points from the preset range of key points to form point cloud data of blocks.

[0075] S645: Determine the volume of the object to be measured by using the point cloud features corresponding to the point cloud data of each block.

[0076] In one embodiment, target point cloud data is input into a volume prediction model. The downsampling layer of the volume prediction model downsamples the target point cloud data to obtain sampled point cloud data. Specifically, the farthest point sampling can be used to make the sampled points as far apart as possible, thereby making the downsampling process more uniform. The original N(d+C) is reduced to N1(d+C), where N is the number of target point clouds; N1 is the number of target point clouds after the downsampling layer; d is the coordinate dimension, which is 3 in this embodiment; and C is other features such as normal vectors, which is 0 in this embodiment. Further, the sampled point cloud data is divided into several blocks in the combination layer of the volume prediction model. For each block, the key point of the block is found among the spatial points belonging to the block in the sampled point cloud data. The key point can be the center point of each block. After finding the key point, a sphere can be drawn with the key point as the center and a sphere radius specified. Points within the range of the sphere with the key point as the center are defined as neighbors. Spatial points within the range of the sphere, not exceeding a preset threshold, are selected to form the point cloud data of the block. Understandably, the preset threshold can be set as needed and is not limited here. Finally, the point cloud data of the constituent blocks is input into the prediction layer of the volume prediction model to obtain the point cloud features corresponding to the point cloud data of each block. Using the point cloud features corresponding to the point cloud data of each block, the volume of the object to be measured is determined.

[0077] In this embodiment, the volume prediction model can be a PointNet++ model. Predicting the volume of the object under test using the above method can reduce the computational load while maintaining accuracy.

[0078] Furthermore, before predicting the volume of the object under test, the volume prediction model can be trained to improve its accuracy. The training process specifically includes: acquiring a large number of sample images containing the object to obtain sample target point cloud data; downsampling the sample target point cloud data using the downsampling layer of the volume prediction model to obtain sample sample point cloud data; dividing the sample sample point cloud data into several sample blocks using the combination layer of the volume prediction model; determining the predicted volume of the object using the prediction layer of the volume prediction model based on the point cloud features corresponding to the point cloud data of each sample block; decoding the data using the decoder based on the point cloud features corresponding to the point cloud data of each sample block to obtain reference point cloud data of the same size as the sample target point cloud data; and adjusting the network parameters of the volume prediction model based on the differences between the actual volume and the predicted volume of the object, and the differences between the sample target point cloud data and the reference point cloud data.

[0079] In one specific implementation, a large number of sample images containing sample objects can be obtained. These sample images are then fed into a semantic segmentation model to obtain the positions of the sample objects within the sample images. Based on these positions, sample target point cloud data of the sample objects is obtained. The specific implementation steps for obtaining the positions of the sample objects in the sample images and obtaining the sample target point cloud data are described in detail in steps S120 and S130 above, and will not be repeated here. The sample target point cloud data is then input into a volume prediction model. The downsampling layer of the volume prediction model is used to downsample the sample target point cloud data of the sample objects, resulting in sample sample point cloud data. Specifically, the same downsampling method as in step S141 can be used for downsampling, and will not be repeated here. The sample sample point cloud data is divided into several sample blocks using a combination layer of the volume prediction model. The volume prediction model uses a prediction layer to obtain point cloud features corresponding to the point cloud data of each sample block. Based on these features, the predicted volume of the sample object is determined. A decoder then decodes the point cloud data based on these features to obtain reference point cloud data of the same size as the target sample point cloud data. Specifically, the decoder decodes the point cloud data based on the features of each sample block and uses algorithms such as point cloud feature interpolation to obtain reference point cloud data of the same size as the target sample point cloud data. A first loss is calculated based on the actual volume and predicted volume of the sample object, and a second loss is calculated based on the target sample point cloud data and the reference point cloud data. The first and second losses are then weighted and summed to obtain the total loss. The network parameters of the volume prediction model are adjusted based on this total loss.

[0080] The above method restores the downsampled and discarded point cloud data through the decoder part (i.e., the decoder) to obtain reference point cloud data. Based on the difference between the actual volume and the predicted volume of the sample object, and the difference between the sample target point cloud data and the reference point cloud data, the network parameters of the volume prediction model are adjusted, which can enable the volume prediction model to better perceive the size of the object to be measured, thereby obtaining the volume of the object to be measured more accurately.

[0081] Please see Figure 7 , Figure 7 This is a flowchart illustrating the second embodiment of the volume estimation method provided in this application. The method includes:

[0082] S710: Acquire a target image containing the object to be tested.

[0083] S720: Determine the first image position of the object to be tested in the target image based on the first color feature and the first depth feature of the target image.

[0084] S730: Based on the position of the first image, obtain the target point cloud data of the object under test.

[0085] S740: Preprocesses the target point cloud data.

[0086] S750: Obtain the volume of the object under test based on the target point cloud data.

[0087] For a detailed description of steps S710-S730, please refer to steps S110-S130 of the first embodiment of the volume estimation method, which will not be repeated here.

[0088] Due to the influence of the data acquisition environment and the instability of data caused by the error of the acquisition lens itself, the target point cloud data of the object under test can be preprocessed. Preprocessing can include at least one of the following: downsampling, outlier removal, and hole filling. In one embodiment, if downsampling, outlier removal, and hole filling are required, voxel filtering can be used first for downsampling to reduce the number of points in the target point cloud data, thereby reducing computation and memory usage. Then, outlier removal can be performed using statistical filtering to remove points with abrupt outliers on the point cloud surface. Specifically, first, all points in the target point cloud are traversed, and the average distance between each point and its K nearest neighbors is calculated. Then, the mean μ and standard deviation σ of all average distances are calculated. The distance threshold is then μ + α × σ, where α is a constant, i.e., a scaling factor, which depends on the number of neighboring points. Finally, all points in the target point cloud are traversed again, and points whose average distance to their K nearest neighbors is greater than the distance threshold are removed. Finally, if holes are generated in the image due to the angle of light and the lens, a gradient method can be used to fill smaller holes. Specifically, we can first find the outline of the hole to be filled, traverse the points on the outline, and for each point, find the point with the smallest gradient other than the points on the hole outline. Use this gradient and the straight line fitted to the outline point to calculate the value to be filled in the hole that is closest to the outline point. After the traversal is completed, recalculate the new outline and repeat the above steps until the hole is filled.

[0089] The preprocessed target point cloud data is input into the volume prediction model to obtain the volume of the object to be measured.

[0090] In this embodiment, by preprocessing the target point cloud data and then using the preprocessed target point cloud data to obtain the volume of the object to be measured, a more accurate volume can be obtained, and the amount of computation can be reduced.

[0091] Please see Figure 8 , Figure 8 This is a schematic diagram of an embodiment of the volume estimation device provided in this application. The volume estimation device 80 includes an acquisition module 81, a position determination module 82, a point cloud generation module 83, and a volume prediction module 84. The acquisition module 81 is used to acquire a target image containing the object to be measured; the position determination module 82 is used to determine a first image position of the object to be measured in the target image based on a first color feature and a first depth feature of the target image; the point cloud generation module 83 is used to obtain target point cloud data of the object to be measured based on the first image position; and the volume prediction module 84 is used to obtain the volume of the object to be measured based on the target point cloud data.

[0092] The determination of the first image position of the object to be tested in the target image based on the first color feature and the first depth feature of the target image is performed by the semantic segmentation model; and / or, the determination of the first image position of the object to be tested in the target image based on the first color feature and the first depth feature of the target image includes: extracting the first color feature and the first depth feature from the target image; fusing the first color feature and the first depth feature to obtain the target fused feature; and performing semantic segmentation on the target fused feature to obtain the first image position.

[0093] The process of extracting the first color feature and the first depth feature from the target image includes: extracting several feature groups of different scales, each feature group including the first color feature and the first depth feature corresponding to a scale; fusing the first color feature and the first depth feature to obtain the target fusion feature includes: fusing the first color feature and the first depth feature in each feature group to obtain the initial fusion feature corresponding to each feature group; and fusing the initial fusion features corresponding to each feature group to obtain the target fusion feature.

[0094] The target image includes a target color image and a target depth image. The target color image and the target depth image are obtained by taking pictures of the object under test using the same acquisition parameters, including the same viewing angle, the same field of view, and the same acquisition time. The first color feature is extracted from the target color image, and the first depth feature is extracted from the target depth image. The first image position of the object under test in the target color image is the same as the first image position of the object under test in the target depth image.

[0095] The process of acquiring a target image containing the object under test includes: acquiring an initial color image and an initial depth image of the object under test captured using the same acquisition parameters; performing target detection on the initial color image to obtain a detection box for the object under test; cropping the portion of the initial color image located within the detection box to obtain a target color image; and cropping the portion of the initial depth image located within the detection box to obtain a target depth image.

[0096] The target image is obtained by taking a picture of the object under test with a camera at a preset viewing angle. Based on the position of the first image, the target point cloud data of the object under test is obtained, including: extracting a first local image corresponding to the position of the first image from the target image, wherein the first local image represents the first projection image of the object under test on a first plane perpendicular to the preset viewing angle; using the camera's intrinsic parameters to transform the first local image to obtain initial point cloud data; and obtaining target point cloud data based on the initial point cloud data.

[0097] The process of obtaining target point cloud data based on initial point cloud data includes: projecting the initial point cloud data onto a second plane to obtain a second projected image; using the second color feature and the second depth feature of the second projected image to determine the second image position of the object to be measured in the second projected image; extracting a second local image corresponding to the second image position from the second projected image; and using the camera's intrinsic parameters to transform the second local image to obtain target point cloud data.

[0098] The target image includes a target color image and a target depth image;

[0099] Extracting a first local image corresponding to the location of the first image from the target image includes: extracting a first local color image corresponding to the location of the first image from the target color image, and extracting a first local depth image corresponding to the location of the first image from the target depth image; transforming the first local image using camera intrinsics to obtain initial point cloud data includes: transforming the first local color image and the first local depth image using camera intrinsics to obtain initial point cloud data, wherein each spatial point in the initial point cloud data contains color information and depth information; projecting the initial point cloud data onto a second plane to obtain a second projected image includes: projecting each spatial point in the initial point cloud data onto a second plane to obtain a second projected image. The color information of spatial points is projected onto a second plane to obtain a color projection image; and the depth information of each spatial point in the initial point cloud data is projected onto the second plane to obtain a depth projection image; a second local image corresponding to the position of the second image is extracted from the second projection image, including: extracting a second local color image corresponding to the position of the second image from the color projection image, and extracting a second local depth image corresponding to the position of the second image from the depth projection image; the second local image is transformed using the camera's intrinsic parameters to obtain target point cloud data, including: transforming the second local color image and the second local depth image using the camera's intrinsic parameters to obtain target point cloud data.

[0100] The process of obtaining the volume of the object to be measured based on the target point cloud data includes: downsampling the target point cloud data to obtain sampled point cloud data; dividing the sampled point cloud data into several blocks; and using the point cloud features corresponding to each block of point cloud data to determine the volume of the object to be measured.

[0101] Before determining the volume of the object to be measured using the point cloud features corresponding to the point cloud data of each block, the method further includes: for each block, finding key points of the block among the spatial points belonging to the block in the sampled point cloud data; selecting no more than a preset threshold of spatial points from the preset range of the key points to form the point cloud data of the block; and / or, the method for obtaining the volume of the object to be measured based on the target point cloud data is performed by the volume prediction model, and further includes: using the downsampling layer of the volume prediction model to downsample the sample target point cloud data of the sample object to obtain sample sampled point cloud data; using the combination layer of the volume prediction model to divide the sample sampled point cloud data into several sample blocks; using the prediction layer of the volume prediction model to determine the predicted volume of the sample object based on the point cloud features corresponding to the point cloud data of each sample block; and using the decoder to decode based on the point cloud features corresponding to the point cloud data of each sample block to obtain reference point cloud data with the same size as the sample target point cloud data; and adjusting the network parameters of the volume prediction model based on the difference between the actual volume and the predicted volume of the sample object, and the difference between the sample target point cloud data and the reference point cloud data.

[0102] Before obtaining the volume of the object to be measured based on the target point cloud data, the method includes: preprocessing the target point cloud data, the preprocessing including at least one of the following: downsampling, outlier removal, and hole filling.

[0103] The volume estimation device 80 provided in this application includes an acquisition module 81, a location determination module 82, a point cloud generation module 83, and a volume prediction module 84, which are used to execute the corresponding steps in the above-described volume estimation method embodiments. The implementation of the specific steps can be referred to the relevant descriptions in the above-described volume estimation method embodiments, and will not be repeated here.

[0104] Please see Figure 9 , Figure 9 This is a schematic diagram of the framework structure of one embodiment of the electronic device provided in this application.

[0105] The electronic device 90 includes a memory 91 and a processor 92 coupled to each other. The memory 91 stores program instructions, and the processor 92 executes the program instructions stored in the memory 91 to implement the steps of any of the above-described method embodiments. In a specific implementation scenario, the electronic device 90 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 90 may also include mobile devices such as laptops and tablets, which are not limited here.

[0106] Specifically, processor 92 controls itself and memory 91 to implement the steps of any of the above-described organizational structure construction methods. Processor 92 can also be referred to as a CPU (Central Processing Unit). Processor 92 may be an integrated circuit chip with signal processing capabilities. Processor 92 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 92 can be implemented using integrated circuit chips.

[0107] Please see Figure 10 , Figure 10 This is a schematic diagram of a framework of one embodiment of the computer-readable storage medium provided in this application.

[0108] The computer-readable storage medium 100 stores program instructions 101, which, when executed by a processor, are used to implement the steps in any of the above method embodiments.

[0109] The computer-readable storage medium 100 may specifically be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a medium capable of storing computer programs. Alternatively, it may be a server storing the computer program, which can send the stored computer program to other devices for execution or can also execute the stored computer program itself.

[0110] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0112] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0113] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0114] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A volume estimation method, characterized in that, The method includes: Acquire a target image containing the object to be tested; Based on the first color feature and the first depth feature of the target image, the first image position of the object to be tested in the target image is determined; Extract a first local image corresponding to the position of the first image from the target image; wherein, the first local image represents a first projection image of the object under test on a first plane perpendicular to a preset viewing direction; Using the camera's intrinsic parameters, the first local image is transformed to obtain initial point cloud data; The initial point cloud data is projected onto the second plane to obtain a second projected image; The second image position of the object under test in the second projection image is determined by using the second color feature and the second depth feature of the second projection image; Extract a second local image corresponding to the location of the second image from the second projected image; Using the camera's intrinsic parameters, the second local image is transformed to obtain target point cloud data; Based on the target point cloud data, the volume of the object to be measured is obtained.

2. The method according to claim 1, characterized in that, The determination of the first image position of the object to be tested in the target image based on the first color feature and the first depth feature of the target image is performed by a semantic segmentation model; And / or, determining the first image position of the object to be tested in the target image based on the first color feature and the first depth feature of the target image includes: The first color feature and the first depth feature are extracted from the target image; The first color feature and the first depth feature are fused to obtain the target fused feature; The target fusion features are semantically segmented to obtain the first image location.

3. The method according to claim 2, characterized in that, The step of extracting the first color feature and the first depth feature from the target image includes: Several feature groups with different scales are extracted, and each feature group includes a first color feature and a first depth feature corresponding to a scale; The step of fusing the first color feature and the first depth feature to obtain the target fused feature includes: The first color feature and the first depth feature in each feature group are fused to obtain the initial fused feature corresponding to each feature group; The initial fusion features corresponding to each feature group are fused to obtain the target fusion feature.

4. The method according to claim 1, characterized in that, The target image includes a target color image and a target depth image. The target color image and the target depth image are obtained by taking pictures of the object under test using the same acquisition parameters, including the same viewing angle, the same field of view, and the same acquisition time. The first color feature is extracted from the target color image, and the first depth feature is extracted from the target depth image. The first image position of the object under test in the target color image is the same as the first image position of the object under test in the target depth image.

5. The method according to claim 4, characterized in that, The step of acquiring a target image containing the object to be tested includes: Acquire the initial color image and initial depth image of the object under test using the same acquisition parameters; Target detection is performed on the initial color image to obtain the detection box of the object to be tested; The image portion located within the detection box in the initial color image is cropped to obtain the target color image; and, The target depth image is obtained by cropping the portion of the initial depth image that is located within the detection box.

6. The method according to claim 1, characterized in that, The target image is obtained by taking a picture of the object under test using a camera at a preset viewing angle.

7. The method according to claim 1, characterized in that, The target image includes a target color image and a target depth image; Extracting the first local image corresponding to the location of the first image from the target image includes: Extract a first local color image corresponding to the position of the first image from the target color image, and extract a first local depth image corresponding to the position of the first image from the target depth image; The step of converting the first local image using the camera's intrinsic parameters to obtain initial point cloud data includes: Using the camera's intrinsic parameters, the first local color image and the first local depth image are combined and converted to obtain initial point cloud data, wherein each spatial point in the initial point cloud data contains color information and depth information; The step of projecting the initial point cloud data onto a second plane to obtain a second projected image includes: The color information of each spatial point in the initial point cloud data is projected onto the second plane to obtain a color projection image; and the depth information of each spatial point in the initial point cloud data is projected onto the second plane to obtain a depth projection image. Extracting the second local image corresponding to the position in the second image from the second projected image includes: Extract a second local color image corresponding to the position of the second image from the color projection image, and extract a second local depth image corresponding to the position of the second image from the depth projection image; The step of converting the second projected image using the camera's intrinsic parameters to obtain target point cloud data includes: The target point cloud data is obtained by converting the second local color image and the second local depth image using the camera's intrinsic parameters.

8. The method according to claim 1, characterized in that, The process of obtaining the volume of the object under test based on the target point cloud data includes: The target point cloud data is downsampled to obtain sampled point cloud data; The sampled point cloud data is divided into several blocks; The volume of the object under test is determined by using the point cloud features corresponding to the point cloud data of each block.

9. The method according to claim 8, characterized in that, Before determining the volume of the object to be measured using the point cloud features corresponding to the point cloud data of each of the blocks, the method further includes: For each block, among the spatial points belonging to the block in the sampled point cloud data, find the key points of the block; Select no more than a preset threshold of spatial points from the preset range of the key points to form the point cloud data of the block; And / or, the step of obtaining the volume of the object to be measured based on the target point cloud data is performed by a volume prediction model, and the method further includes: The sample target point cloud data of the sample object is downsampled using the downsampling layer of the volume prediction model to obtain sampled point cloud data; The sampled point cloud data is divided into several sample blocks using the combined layer of the volume prediction model. The predicted volume of the sample object is determined by the prediction layer of the volume prediction model based on the point cloud features corresponding to the point cloud data of each sample block; and a reference point cloud data with the same size as the sample target point cloud data is obtained by decoding based on the point cloud features corresponding to the point cloud data of each sample block. Based on the difference between the actual volume of the sample object and the predicted volume, and the difference between the sample target point cloud data and the reference point cloud data, the network parameters of the volume prediction model are adjusted.

10. The method according to claim 1, characterized in that, Before obtaining the volume of the object to be measured based on the target point cloud data, the method includes: The target point cloud data is preprocessed, and the preprocessing includes at least one of the following: downsampling, outlier removal, and hole filling.

11. A volume estimation device, characterized in that, The device includes: The acquisition module is used to acquire a target image containing the object to be tested; The position determination module is used to determine the first image position of the object to be tested in the target image based on the first color feature and the first depth feature of the target image; A point cloud generation module is used to extract a first local image corresponding to the first image position from the target image; transform the first local image using camera intrinsic parameters to obtain initial point cloud data; project the initial point cloud data onto a second plane to obtain a second projected image; determine the second image position of the object under test in the second projected image using the second color feature and the second depth feature of the second projected image; extract a second local image corresponding to the second image position from the second projected image; transform the second local image using the camera intrinsic parameters to obtain target point cloud data; wherein, the first local image represents the first projected image of the object under test on a first plane perpendicular to a preset viewing direction; The volume prediction module is used to obtain the volume of the object to be measured based on the target point cloud data.

12. A volume estimation device, characterized in that, The device includes a memory and a processor that are coupled to each other; The memory stores program instructions; The processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed by a processor to implement the method as described in any one of claims 1-10.

Images