Cargo volume estimation method and device, electronic equipment and storage medium

Abstract

The invention provides a cargo volume estimation method and device, electronic equipment and a storage medium, and relates to the technical field of logistics, and the method comprises the steps: obtaining a structured light 3D point cloud, a radar point cloud and an image of a to-be-estimated cargo; performing registration fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fusion data; determining a target type of the to-be-estimated cargo based on the fusion data; and inputting the fusion data into a target volume estimation model corresponding to the target type to obtain the volume of the to-be-estimated cargo. According to the invention, after the fusion data of the to-be-estimated cargo is obtained, the target type of the to-be-estimated cargo is determined based on the fusion data, and then the fusion data is input into the target volume estimation model corresponding to the target type for volume estimation; and the target volume estimation model corresponding to the target type is obtained by training the sample fusion data of the first sample cargo corresponding to the target type, so that the accuracy of cargo volume estimation can be improved.

Description

Technical Field

[0001] The present invention relates to the field of logistics technology, and in particular to a cargo volume estimation method, device, electronic equipment and storage medium. Background Art

[0002] In the field of logistics, accurate measurement of cargo volume is crucial for optimizing transportation planning, reasonable billing, and efficient warehouse management. With the rapid development of e-commerce and the increasing complexity of supply chains, accurate volume data has become a core link in improving logistics efficiency and reducing costs.

[0003] In the related art, cargo images are usually collected and input into a volume estimation model, and the volume estimation model is used to extract features such as edges from the cargo images to estimate the volume of the cargo. However, the volume estimation model in the related art is difficult to adapt to different types of cargo, thereby reducing the accuracy of cargo volume estimation. Summary of the invention

[0004] The present invention provides a cargo volume estimation method, device, electronic device and storage medium, which are used to solve the defect of reducing the accuracy of cargo volume estimation in the prior art.

[0005] The present invention provides a cargo volume estimation method, comprising the following steps.

[0006] Acquire multimodal data of the goods to be estimated, wherein the multimodal data includes a structured light 3D point cloud of the goods to be estimated, a radar point cloud of the goods to be estimated, and an image of the goods to be estimated; Performing registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fused data; Determining the target type of the cargo to be estimated based on the fused data; The fused data is input into a target volume estimation model corresponding to the target type to obtain the volume of the cargo to be estimated output by the target volume estimation model, wherein the target volume estimation model is trained based on sample fusion data of a first sample cargo corresponding to the target type.

[0007] According to a cargo volume estimation method provided by the present invention, the target volume estimation model is trained based on the following method: Determine a first model structure parameter and a first model training parameter corresponding to the target type based on the correspondence between the type, the model structure parameter and the model training parameter; Determining a first initial volume estimation model based on the first model structure parameters and the first model training parameters; Inputting the sample fusion data of the first sample cargo into the first initial volume estimation model to obtain a first predicted volume of the first sample cargo output by the first initial volume estimation model; Based on the first predicted volume and the volume label corresponding to the first sample cargo, the first model structure parameters and the first model training parameters are iteratively adjusted to obtain the target volume estimation model.

[0008] According to a cargo volume estimation method provided by the present invention, the step of inputting the fused data into a target volume estimation model corresponding to the target type to obtain the volume of the cargo to be estimated output by the target volume estimation model comprises: Determining a target logistics scenario where the goods to be estimated are located; The fused data is input into a target volume estimation model corresponding to the target type and the target logistics scenario, and the volume of the goods to be estimated output by the target volume estimation model is obtained.

[0009] According to a cargo volume estimation method provided by the present invention, the target volume estimation model is trained based on the following method: Determine the second model structure parameters and the second model training parameters corresponding to the target type and the target logistics scenario based on the correspondence between the logistics scenario, type, model structure parameters and model training parameters; Determining a second initial volume estimation model based on the second model structure parameters and the second model training parameters; Inputting the sample fusion data of the first sample cargo into the second initial volume estimation model to obtain a second predicted volume of the first sample cargo output by the second initial volume estimation model; Based on the second predicted volume and the volume label corresponding to the first sample cargo, the second model structure parameters and the second model training parameters are iteratively adjusted to obtain the target volume estimation model.

[0010] According to a cargo volume estimation method provided by the present invention, the registering and fusing the structured light 3D point cloud, the radar point cloud and the image to obtain fused data includes: Performing denoising on the structured light 3D point cloud and the radar point cloud based on a statistical filtering algorithm to obtain a denoised structured light 3D point cloud and a denoised radar point cloud; Performing denoising on the image based on a Gaussian filtering algorithm to obtain a denoised image; The denoised structured light 3D point cloud, the denoised radar point cloud and the denoised image are registered and fused to obtain the fused data.

[0011] According to a cargo volume estimation method provided by the present invention, the step of determining the target type of the cargo to be estimated based on the fused data includes: The fused data is input into a clustering model to obtain the target type of the goods to be estimated output by the clustering model, wherein the clustering model is trained in an unsupervised manner based on the sample fusion data of each of the different types of second sample goods.

[0012] According to a cargo volume estimation method provided by the present invention, the target volume estimation model includes a convolution module, a pooling layer and a fully connected layer connected in sequence, and the convolution module includes a plurality of 3D convolution layers connected in sequence.

[0013] The present invention also provides a cargo volume estimation device, comprising: an acquisition unit, configured to acquire multimodal data of the goods to be estimated, wherein the multimodal data includes a structured light 3D point cloud of the goods to be estimated, a radar point cloud of the goods to be estimated, and an image of the goods to be estimated; A registration and fusion unit, used for performing registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fused data; a determination unit, configured to determine the target type of the cargo to be estimated based on the fused data; an estimation unit, for inputting the fused data into a target volume estimation model corresponding to the target type, and obtaining the volume of the cargo to be estimated output by the target volume estimation model, wherein the target volume estimation model is trained based on the sample fusion data of the first sample cargo corresponding to the target type.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned methods for estimating the cargo volume is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the cargo volume estimation method as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned cargo volume estimation methods.

[0017] The cargo volume estimation method, device, electronic device and storage medium provided by the present invention obtain the structured light 3D point cloud of the cargo to be estimated, the radar point cloud of the cargo to be estimated and the image of the cargo to be estimated, perform registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fusion data, determine the target type of the cargo to be estimated based on the fusion data, input the fusion data into the target volume estimation model trained based on the sample fusion data of the first sample cargo corresponding to the target type, and obtain the volume of the cargo to be estimated output by the target volume estimation model. It can be seen that after obtaining the fusion data of the cargo to be estimated, the present invention first determines the target type of the cargo to be estimated based on the fusion data, and then inputs the fusion data into the target volume estimation model corresponding to the target type for volume estimation, and the target volume estimation model corresponding to the target type is specially trained based on the sample fusion data of the first sample cargo corresponding to the target type, so it can improve the accuracy of cargo volume estimation; in addition, the multimodal data collected by the present invention includes the structured light 3D point cloud, radar point cloud and image of the cargo to be evaluated, which improves the integrity of cargo information collection and further improves the accuracy of cargo volume estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of a cargo volume estimation method provided in an embodiment of the present invention.

[0020] Figure 2 This is one of the flowcharts of the training method of the target volume estimation model provided by an embodiment of the present invention.

[0021] Figure 3 This is the second flowchart of the target volume estimation model training method provided by the embodiment of the present invention.

[0022] Figure 4 It is an overall framework diagram of cargo volume estimation provided by an embodiment of the present invention.

[0023] Figure 5 It is a structural schematic diagram of a cargo volume estimation device provided by an embodiment of the present invention.

[0024] Figure 6 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Combine the following Figure 1-Figure 4 The cargo volume estimation method of the present invention is described. The execution subject of the cargo volume estimation method can be an electronic device such as a terminal, a tablet computer, a computer, a server, etc., or a cargo volume estimation device set in the electronic device, and the cargo volume estimation device can be implemented by software, hardware, or a combination of the two.

[0027] Figure 1 FIG. 1 is a flow chart of a cargo volume estimation method provided by an embodiment of the present invention. Figure 1 As shown, the cargo volume estimation method includes the following steps: Step 101: Acquire multimodal data of goods to be estimated, wherein the multimodal data includes a structured light 3D point cloud of the goods to be estimated, a radar point cloud of the goods to be estimated, and an image of the goods to be estimated.

[0028] For example, a multimodal data acquisition module integrating structured light 3D camera, laser radar and RGB camera is constructed. Structured light 3D camera, laser radar and RGB (Red Green Blue) camera are deployed at key nodes of logistics transportation, such as the entrance and exit of the warehouse, the cargo sorting area and the inside of the transport vehicle. The structured light 3D camera projects a specific pattern (e.g., stripes or grids) to the cargo to be estimated and analyzes the deformation of the reflected light to obtain the depth information of the surface of the cargo to be estimated. This technology can provide high-resolution surface details of objects at close range. The data collected by the structured light 3D camera is called structured light 3D point cloud. The laser radar uses laser pulses to measure the distance to the cargo to be estimated, and can quickly obtain the general outline information of the cargo to be estimated within a large range. It is suitable for long-distance and large-area scenes. The data collected by the laser radar is called radar point cloud. The RGB camera captures the image of the cargo to be estimated, which includes the color information and texture information of the cargo to be estimated. The color information and texture information are used to assist in identifying the boundaries and features of the cargo to be estimated. Structured light 3D cameras, lidar and RGB cameras work together to fully acquire multimodal data of the cargo to be estimated at different scales and angles, effectively making up for the shortcomings of a single sensor.

[0029] Step 102: perform registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fused data.

[0030] For example, when the structured light 3D point cloud, radar point cloud and image of the goods to be estimated are obtained, feature points of the structured light 3D point cloud and radar point cloud are extracted through a feature-based registration algorithm. For example, the extracted feature points include edge points and corner points, etc., and then the extracted feature points are matched and aligned using registration methods such as iterative closest point (ICP), so as to fuse multi-source data and generate more complete and accurate fused point cloud data of the goods to be estimated. The fused point cloud data is then registered with the image captured by the RGB camera device, and the texture features and color features in the image are used to assist in determining the boundary and shape of the goods to be estimated, so as to obtain the final fused data, which can more accurately characterize the goods to be estimated.

[0031] Step 103: Determine the target type of the cargo to be estimated based on the fused data.

[0032] For example, after obtaining the fused data of the goods to be estimated, the similarity between the fused data of the goods to be estimated and each reference fused data stored in the goods type database can be calculated, and the goods type to which the reference fused data corresponding to the highest similarity belongs is determined as the target type of the goods to be estimated. Among them, the corresponding relationship between the reference fused data and the goods type is stored in the goods type database, and the reference fused data is determined based on the reference structured light 3D point cloud, the reference radar point cloud and the reference image of the reference goods. In addition, the goods types mentioned here can be divided into small goods, medium goods and large goods based on the size of the goods. For example, letters and documents belong to small goods, clothes and shoes belong to medium goods, and televisions and washing machines belong to large goods; the goods types can also be divided into regular-shaped goods and irregular-shaped goods based on the shape of the goods. For example, goods packed in cartons are regular-shaped goods, and goods packed in sacks are irregular-shaped goods. The present invention does not limit this.

[0033] Step 104: input the fused data into a target volume estimation model corresponding to the target type to obtain the volume of the cargo to be estimated output by the target volume estimation model, wherein the target volume estimation model is trained based on the sample fusion data of the first sample cargo corresponding to the target type.

[0034] For example, after determining the target type of the cargo to be estimated, based on the correspondence between the type and the volume estimation model, the target volume estimation model corresponding to the target type is called, and the fused data of the cargo to be estimated is input into the target volume estimation model corresponding to the target type. The input fused data is subjected to feature analysis through the target volume estimation model to estimate the volume of the cargo to be estimated.

[0035] The cargo volume estimation method provided by the present invention obtains the structured light 3D point cloud of the cargo to be estimated, the radar point cloud of the cargo to be estimated and the image of the cargo to be estimated, performs registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fusion data, determines the target type of the cargo to be estimated based on the fusion data, inputs the fusion data into the target volume estimation model trained based on the sample fusion data of the first sample cargo corresponding to the target type, and obtains the volume of the cargo to be estimated output by the target volume estimation model. It can be seen that after obtaining the fusion data of the cargo to be estimated, the present invention first determines the target type of the cargo to be estimated based on the fusion data, and then inputs the fusion data into the target volume estimation model corresponding to the target type for volume estimation, and the target volume estimation model corresponding to the target type is specially trained based on the sample fusion data of the first sample cargo corresponding to the target type, so the accuracy of cargo volume estimation can be improved; in addition, the multimodal data collected by the present invention includes the structured light 3D point cloud, the radar point cloud and the image of the cargo to be evaluated, which improves the integrity of cargo information collection, can effectively deal with all cargo types, background complexity and occlusion problems in logistics scenes, and further improves the accuracy of cargo volume estimation.

[0036] In one embodiment, Figure 2 FIG. 1 is one of the flow charts of the training method of the target volume estimation model provided by the embodiment of the present invention. Figure 2 As shown, the target volume estimation model is trained based on the following method: Step 201: Determine first model structure parameters and first model training parameters corresponding to the target type based on the correspondence between the type, model structure parameters and model training parameters.

[0037] For example, model structure parameters and model training parameters are predefined for each type of goods. Model structure parameters may include convolution kernel size, number of convolution layers, network depth, etc. Model training parameters may include learning rate and regularization coefficient, etc. Taking the classification of goods type based on the size of goods as an example, the convolution kernel size of small goods can be predefined as 3x3, the number of convolution layers is 3 layers, the network depth is shallow, the learning rate is 0.001, and the regularization coefficient is 0.01; the convolution kernel size of medium-sized goods is predefined as 5x5, the number of convolution layers is 5 layers, the network depth is medium, the learning rate is 0.0005, and the regularization coefficient is 0.005; the convolution kernel size of large goods is predefined as 7x7, the number of convolution layers is 7 layers, the network depth is deep, the learning rate is 0.0001, and the regularization coefficient is 0.001. Based on the correspondence between the type, model structure parameters and model training parameters, the first model structure parameters and the first model training parameters corresponding to the target type can be found. For example, if the target type is small cargo, the first model structure parameters include a convolution kernel size of 3x3, a number of convolution layers of 3, a shallow network depth, and the first model training parameters include a learning rate of 0.001 and a regularization coefficient of 0.01.

[0038] Step 202: Determine a first initial volume estimation model based on the first model structure parameters and the first model training parameters.

[0039] For example, when the first model structure parameters and the first model training parameters are obtained, the model architecture can be defined according to the first model structure parameters, and the model training process can be configured according to the first model training parameters, and finally, the first initial volume estimation model can be constructed.

[0040] Step 203: Input the sample fusion data of the first sample cargo into the first initial volume estimation model to obtain a first predicted volume of the first sample cargo output by the first initial volume estimation model.

[0041] For example, for each first sample cargo of the target type, the structured light 3D camera device projects a specific pattern (e.g., stripes or grids) onto the first sample cargo and analyzes the deformation of the reflected light, thereby obtaining the depth information of the surface of the first sample cargo. The data collected by the structured light 3D camera device is referred to as a sample structured light 3D point cloud. The laser radar uses laser pulses to measure the distance to the first sample cargo, and can quickly obtain the general outline information of the first sample cargo within a large range. The data collected by the laser radar is referred to as a sample radar point cloud. The RGB camera device captures a sample image of the first sample cargo, and the sample image includes color information and texture information of the first sample cargo. The color information and texture information are used to assist in identifying the boundaries and features of the first sample cargo. The sample structured light 3D point cloud, the sample radar point cloud, and the sample image are fused and registered to obtain sample fusion data of the first sample cargo. The sample fusion data of the first sample cargo is input into the constructed first initial volume estimation model. The first initial volume estimation model is used to perform feature analysis on the input sample fusion data to estimate the first predicted volume of the first sample cargo.

[0042] Step 204: Based on the first predicted volume and the volume label corresponding to the first sample cargo, iteratively adjust the first model structure parameters and the first model training parameters to obtain the target volume estimation model.

[0043] For example, when the first predicted volume of the first sample cargo output by the first initial volume estimation model is obtained, the first loss function is determined based on the following formula (1): , the first loss function is the mean square error loss function, based on the first loss function The first model structure parameters and the first model training parameters are iteratively adjusted until a convergence condition is reached, and finally a target volume estimation model is obtained, so that the prediction result of the target volume estimation model is closer to the actual volume.

[0044]

[0045] in, Indicates The first predicted volume of the first sample cargo, Indicates Volume label for the first sample shipment, Indicates the quantity of the first sample goods.

[0046] In this embodiment, the correspondence between types, model structure parameters and model training parameters is pre-stored. For each type of goods, the model structure parameters and model training parameters corresponding to the type are determined based on the correspondence, and an initial volume estimation model of the type is constructed based on the model structure parameters and model training parameters corresponding to the type. The initial volume estimation model is then trained based on sample fusion data of sample goods of the type, and finally a target volume estimation model of the type is obtained, so that the target volume estimation model of the type can fully adapt to the goods of this type. The target volume estimation model corresponding to each type can be obtained by adopting the same method. Through this adaptive model optimization mechanism, the combination of target volume estimation models corresponding to each type can dynamically adapt to different types of goods, thereby improving the generalization ability and adaptability of the model.

[0047] In one embodiment, the above step 104 inputs the fused data into the target volume estimation model corresponding to the target type to obtain the volume of the cargo to be estimated output by the target volume estimation model, which can be specifically implemented in the following manner: Determine the target logistics scenario in which the goods to be estimated are located; input the fused data into a target volume estimation model corresponding to the target type and the target logistics scenario, and obtain the volume of the goods to be estimated output by the target volume estimation model.

[0048] For example, the environment where the goods to be estimated are located is photographed by a camera device to obtain an environmental image, and the target logistics scene where the goods to be estimated are located is determined based on the analysis of the environmental image. The logistics scene can be classified based on the means of transportation. For example, the logistics scene includes air transportation scene, road transportation scene, and sea transportation scene. For example, if the goods to be estimated are located on an airplane, the target logistics scene where the goods to be estimated are located is an air transportation scene. Based on the correspondence between the type of logistics scene and the volume estimation model, the target volume estimation model corresponding to the target type and the target logistics scene is called, and the fused data of the goods to be estimated is input into the target volume estimation model corresponding to the target type and the target logistics scene. The input fused data is subjected to feature analysis by the target volume estimation model to estimate the volume of the goods to be estimated.

[0049] In this embodiment, the target volume estimation model is determined based on the target type of the goods to be estimated and the target logistics scenario, so that the target volume estimation model can fully adapt to the needs of specific goods types and specific logistics scenarios, further improving the accuracy of cargo volume estimation.

[0050] In one embodiment, Figure 3 FIG. 2 is a flow chart of a training method for a target volume estimation model provided by an embodiment of the present invention. Figure 3 As shown, the target volume estimation model is trained based on the following method: Step 301: Based on the correspondence between logistics scenarios, types, model structure parameters and model training parameters, determine the second model structure parameters and second model training parameters corresponding to the target type and the target logistics scenario.

[0051] For example, model structure parameters and model training parameters are predefined for each logistics scenario and cargo type. Model structure parameters may include convolution kernel size, number of convolution layers, network depth, etc. Model training parameters may include learning rate and regularization coefficient, etc. Taking the logistics scenario based on the classification of transport tools as an example, the convolution kernel size of the air transport scenario can be predefined as 3x3, the number of convolution layers is 4 layers, the network depth is medium, the learning rate is 0.0008, and the regularization coefficient is 0.008; the convolution kernel size of the road transport scenario is predefined as 5x5, the number of convolution layers is 5 layers, the network depth is medium, the learning rate is 0.0005, and the regularization coefficient is 0.005; the convolution kernel size of the shipping scenario is predefined as 7x7, the number of convolution layers is 6 layers, the network depth is deep, the learning rate is 0.0002, and the regularization coefficient is 0.002. Based on the correspondence between type, logistics scenario, model structure parameters and model training parameters, the second model structure parameters and second model training parameters corresponding to the target type and target logistics scenario can be found. For example, if the target type is medium-sized cargo and the target logistics scenario is road transportation, the second model structure parameters include convolution kernel size of 5x5, number of convolution layers of 5 layers, medium network depth, and the second model training parameters include learning rate of 0.0005 and regularization coefficient of 0.005. This scenario-based and type-based parameter definition and matching method can ensure that the target volume estimation model better adapts to the needs of specific cargo types and logistics scenarios, thereby significantly improving the accuracy and practicality of volume estimation.

[0052] Step 302: Determine a second initial volume estimation model based on the second model structure parameters and the second model training parameters.

[0053] For example, when the second model structure parameters and the second model training parameters are obtained, the model architecture can be defined according to the second model structure parameters, and the model training process can be configured according to the second model training parameters, and finally, the second initial volume estimation model can be constructed.

[0054] Step 303: Input the sample fusion data of the first sample cargo into the second initial volume estimation model to obtain a second predicted volume of the first sample cargo output by the second initial volume estimation model.

[0055] For example, for each first sample cargo of the target type, the structured light 3D camera device projects a specific pattern (e.g., stripes or grids) onto the first sample cargo and analyzes the deformation of the reflected light, thereby obtaining the depth information of the surface of the first sample cargo. The data collected by the structured light 3D camera device is referred to as a sample structured light 3D point cloud. The laser radar uses laser pulses to measure the distance to the first sample cargo, and can quickly obtain the general outline information of the first sample cargo within a large range. The data collected by the laser radar is referred to as a sample radar point cloud. The RGB camera device captures a sample image of the first sample cargo, and the sample image includes color information and texture information of the first sample cargo. The color information and texture information are used to assist in identifying the boundaries and features of the first sample cargo. The sample structured light 3D point cloud, the sample radar point cloud, and the sample image are fused and registered to obtain sample fusion data of the first sample cargo. The sample fusion data of the first sample cargo is input into the constructed second initial volume estimation model. The input sample fusion data is feature analyzed by the second initial volume estimation model to estimate the second predicted volume of the first sample cargo.

[0056] Step 304: Based on the second predicted volume and the volume label corresponding to the first sample cargo, iteratively adjust the second model structure parameters and the second model training parameters to obtain the target volume estimation model.

[0057] For example, when the second predicted volume of the first sample cargo output by the second initial volume estimation model is obtained, the second loss function is determined based on a method similar to the above formula (1), and the second model structure parameters and the second model training parameters are iteratively adjusted based on the second loss function until the convergence condition is reached, and finally the target volume estimation model is obtained.

[0058] In this embodiment, the correspondence between logistics scenarios, types, model structure parameters and model training parameters is pre-stored. For each type of goods and logistics scenario, the model structure parameters and model training parameters corresponding to the type and logistics scenario are determined based on the correspondence, and an initial volume estimation model for the type and logistics scenario is constructed based on the model structure parameters and model training parameters corresponding to the type and logistics scenario. The initial volume estimation model is then trained based on sample fusion data of sample goods under the type and logistics scenario, and finally a target volume estimation model for the type and logistics scenario is obtained, so that the target volume estimation model for the type and logistics scenario can fully adapt to the goods of this type in the logistics scenario. By using the same method, the target volume estimation model corresponding to each type and logistics scenario can be obtained, so that the combination of target volume estimation models corresponding to each type and logistics scenario can adapt to different types of goods and different logistics scenarios, thereby improving the generalization ability and adaptability of the model.

[0059] In one embodiment, the step 102 performs registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fused data, which can be specifically implemented in the following manner: The structured light 3D point cloud and the radar point cloud are denoised based on a statistical filtering algorithm to obtain a denoised structured light 3D point cloud and a denoised radar point cloud; the image is denoised based on a Gaussian filtering algorithm to obtain a denoised image; the denoised structured light 3D point cloud, the denoised radar point cloud and the denoised image are registered and fused to obtain the fused data.

[0060] For example, the structured light 3D point cloud and the radar point cloud can be denoised based on the following formula (2), that is, a statistical filtering algorithm is used to remove outliers, and points that are significantly deviated from surrounding points are judged and removed according to the distribution of points in the neighborhood of the point in the point cloud, so as to finally obtain a denoised structured light 3D point cloud and a denoised radar point cloud.

[0061]

[0062] Among them, when denoising the structured light 3D point cloud, Represents the position of a point in the structured light 3D point cloud, Represents the denoised structured light 3D point cloud. When denoising the radar point cloud, represents the position of a point in the radar point cloud, represents the radar point cloud after denoising, express Neighborhood of express The mean of the midpoint locations, express The standard deviation of the positions of the interior points, Indicates the set threshold.

[0063] The image is denoised based on the following formula (3), that is, the Gaussian filter algorithm is used to remove noise to obtain the denoised image:

[0064] in, represents the denoised image, represents the pixel coordinates in the image, Represents the standard deviation, which is a parameter of the Gaussian filter. It is set according to the type and degree of noise to be filtered out. Controls the filtering strength.

[0065] In this embodiment, the statistical filtering algorithm can effectively remove outliers and noise points in the structured light 3D point cloud and the radar point cloud, retain the true geometric features of the goods to be estimated, and the Gaussian filtering algorithm can smooth the noise in the image of the goods to be estimated, while retaining the edge and texture information and improving the clarity of the image. The denoised structured light 3D point cloud, the denoised radar point cloud and the denoised image are then registered and fused, so that the final fused data can accurately characterize the goods to be estimated, further improving the accuracy of the cargo volume estimation.

[0066] In one embodiment, the above step 103 determines the target type of the cargo to be estimated based on the fusion data, which can be specifically implemented in the following manner: The fused data is input into a clustering model to obtain the target type of the goods to be estimated output by the clustering model, wherein the clustering model is trained in an unsupervised manner based on the sample fusion data of each of the different types of second sample goods.

[0067] For example, the sample fusion data of the second sample cargo is obtained, including the denoised sample structured light 3D point cloud, denoised sample radar point cloud and denoised image of the second sample cargo, and the feature vectors of all the sample fusion data of the second sample cargo are input into the initial clustering model. The initial clustering model can be K-Means, and the feature vectors of the sample fusion data are assigned to the nearest cluster by iteratively updating the cluster center, and whether the initial clustering model converges is determined according to the algorithm characteristics. For example, K-Means stops iterating when the cluster center no longer changes, and finally obtains the clustering model trained in an unsupervised manner. When the fusion data of the cargo to be estimated is obtained, the fusion data of the cargo to be estimated is input into the trained clustering model, and the fusion data of the cargo to be estimated is analyzed by the clustering model, and finally the target type of the cargo to be estimated is obtained.

[0068] In this embodiment, a clustering model is obtained by training in an unsupervised manner based on sample fusion data of different types of second sample goods. The type of goods to be estimated is identified through the clustering model, which reduces the dependence on large-scale labeled data, reduces the labeling cost, and also improves the accuracy of determining the type of goods.

[0069] In one embodiment, the target volume estimation model includes a convolution module, a pooling layer and a fully connected layer connected in sequence, and the convolution module includes a plurality of 3D convolution layers connected in sequence.

[0070] For example, feature extraction and regression prediction are performed based on the fused data of the goods to be estimated through multiple 3D convolutional layers, pooling layers and fully connected layers, and finally the volume of the goods to be estimated is obtained. The target volume estimation model of the present invention includes multiple 3D convolutional layers. For example, the convolution kernel size of the first 3D convolutional layer is 3 × 3 × 3, the step size is 1, and the ReLU activation function is used. The formula is f (x) = max (0, x), and preliminary feature extraction is performed on the input fused data. As the network layer advances, the feature map gradually shrinks, and the extracted features are more abstract and advanced. Among them, x in the formula f (x) = max (0, x) represents the output value after the convolution operation on the fused data, and f (x) represents the value after processing by the ReLU activation function.

[0071] It should be noted that the present invention can also use a variant of a recurrent neural network (RNN), such as a long short-term memory network (LSTM) or a gated recurrent unit (GRU) to process fused data to replace a 3D convolutional neural network (3D-CNN) including a convolution module, a pooling layer and a fully connected layer, and the present invention is not limited to this.

[0072] Figure 4 is an overall framework diagram of cargo volume estimation provided by an embodiment of the present invention, such as Figure 4As shown, it includes a multimodal data acquisition module, a data preprocessing and fusion module, a deep learning volume estimation module and an adaptive model optimization module, wherein the multimodal data acquisition module includes a structured light 3D camera device, a laser radar and an RGB camera device, and the structured light 3D point cloud of the goods to be estimated is obtained by the structured light 3D camera device, the radar point cloud of the goods to be estimated is obtained by the laser radar, and the image of the goods to be estimated is obtained by the RGB camera device. The data preprocessing and fusion module is used to denoise the structured light 3D point cloud and the radar point cloud based on the statistical filtering algorithm to obtain the denoised structured light 3D point cloud and the denoised radar point cloud, and denoise the image based on the Gaussian filtering algorithm to obtain the denoised image; and the denoised structured light 3D point cloud, the denoised radar point cloud and the denoised image are registered and fused to obtain fused data, and the fused data is used to characterize the 3D model of the goods to be estimated. The deep learning volume estimation module is used to input the fusion data into the volume estimation model, perform 3D-CNN estimation, feature extraction and regression prediction through the volume estimation model, and finally obtain the volume of the goods to be estimated output by the volume estimation model. In addition, an adaptive model optimization mechanism is introduced to determine the second model structure parameters and the second model training parameters corresponding to the target type and the target logistics scenario based on the corresponding relationship between the logistics scenario, type, model structure parameters and model training parameters. Based on the second model structure parameters and the second model training parameters, a second initial volume estimation model is determined, and the sample fusion data of the first sample goods is input into the second initial volume estimation model to obtain the second predicted volume of the first sample goods output by the second initial volume estimation model. Based on the second predicted volume and the volume label corresponding to the first sample goods, the second model structure parameters and the second model training parameters are iteratively adjusted to obtain the target volume estimation model corresponding to the target type and the target logistics scenario. Further, the fusion data is input into the target volume estimation model corresponding to the target type and the target logistics scenario to estimate the volume of the goods to be estimated, so as to improve the accuracy of the cargo volume estimation.

[0073] The cargo volume estimation method provided by the embodiment of the present invention can more accurately identify the shape and boundary of the cargo and effectively reduce the volume estimation error through multimodal data fusion and deep learning model. Compared with the traditional method and the machine learning method of a single sensor, the accuracy is significantly improved; and the multimodal data collected by the present invention can adapt to the complex environment, diversity of cargo types and occlusion conditions in the logistics scene, and can be suitable for volume measurement of different types of cargo and logistics links; in addition, a clustering model is obtained by training the sample fusion data of each different type of second sample cargo in an unsupervised manner, and the type of the cargo to be estimated is identified through the clustering model, which reduces the dependence on large-scale labeled data, reduces the labeling cost, and also improves the accuracy of cargo type determination.

[0074] The cargo volume estimation device provided by the present invention is described below. The cargo volume estimation device described below and the cargo volume estimation method described above can be referenced to each other.

[0075] Figure 5 is a schematic diagram of the structure of a cargo volume estimation device provided by an embodiment of the present invention. Figure 5 As shown, the cargo volume estimation device 500 includes an acquisition unit 501, a registration and fusion unit 502, a determination unit 503 and an estimation unit 504; wherein: An acquisition unit 501 is used to acquire multimodal data of the goods to be estimated, wherein the multimodal data includes a structured light 3D point cloud of the goods to be estimated, a radar point cloud of the goods to be estimated, and an image of the goods to be estimated; A registration and fusion unit 502 is used to perform registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fused data; A determination unit 503, configured to determine the target type of the cargo to be estimated based on the fused data; The estimation unit 504 is used to input the fused data into a target volume estimation model corresponding to the target type to obtain the volume of the cargo to be estimated output by the target volume estimation model, wherein the target volume estimation model is trained based on the sample fusion data of the first sample cargo corresponding to the target type.

[0076] The cargo volume estimation device provided by the present invention obtains the structured light 3D point cloud of the cargo to be estimated, the radar point cloud of the cargo to be estimated and the image of the cargo to be estimated, performs registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fusion data, determines the target type of the cargo to be estimated based on the fusion data, inputs the fusion data into the target volume estimation model trained based on the sample fusion data of the first sample cargo corresponding to the target type, and obtains the volume of the cargo to be estimated output by the target volume estimation model. It can be seen that after obtaining the fusion data of the cargo to be estimated, the present invention first determines the target type of the cargo to be estimated based on the fusion data, and then inputs the fusion data into the target volume estimation model corresponding to the target type for volume estimation, and the target volume estimation model corresponding to the target type is specially trained based on the sample fusion data of the first sample cargo corresponding to the target type, so it can improve the accuracy of cargo volume estimation; in addition, the multimodal data collected by the present invention includes the structured light 3D point cloud, the radar point cloud and the image of the cargo to be evaluated, which improves the integrity of cargo information collection and further improves the accuracy of cargo volume estimation.

[0077] Based on any of the above embodiments, the target volume estimation model is trained based on the following method: Determine a first model structure parameter and a first model training parameter corresponding to the target type based on the correspondence between the type, the model structure parameter and the model training parameter; Determining a first initial volume estimation model based on the first model structure parameters and the first model training parameters; Inputting the sample fusion data of the first sample cargo into the first initial volume estimation model to obtain a first predicted volume of the first sample cargo output by the first initial volume estimation model; Based on the first predicted volume and the volume label corresponding to the first sample cargo, the first model structure parameters and the first model training parameters are iteratively adjusted to obtain the target volume estimation model.

[0078] Based on any of the above embodiments, the estimating unit 504 is specifically configured to: Determining a target logistics scenario where the goods to be estimated are located; The fused data is input into a target volume estimation model corresponding to the target type and the target logistics scenario, and the volume of the goods to be estimated output by the target volume estimation model is obtained.

[0079] Based on any of the above embodiments, the target volume estimation model is trained based on the following method: Determine the second model structure parameters and the second model training parameters corresponding to the target type and the target logistics scenario based on the correspondence between the logistics scenario, type, model structure parameters and model training parameters; Determining a second initial volume estimation model based on the second model structure parameters and the second model training parameters; Inputting the sample fusion data of the first sample cargo into the second initial volume estimation model to obtain a second predicted volume of the first sample cargo output by the second initial volume estimation model; Based on the second predicted volume and the volume label corresponding to the first sample cargo, the second model structure parameters and the second model training parameters are iteratively adjusted to obtain the target volume estimation model.

[0080] Based on any of the above embodiments, the registration and fusion unit 502 is specifically used for: Performing denoising on the structured light 3D point cloud and the radar point cloud based on a statistical filtering algorithm to obtain a denoised structured light 3D point cloud and a denoised radar point cloud; Performing denoising on the image based on a Gaussian filtering algorithm to obtain a denoised image; The denoised structured light 3D point cloud, the denoised radar point cloud and the denoised image are registered and fused to obtain the fused data.

[0081] Based on any of the foregoing embodiments, the determining unit 503 is specifically configured to: The fused data is input into a clustering model to obtain the target type of the goods to be estimated output by the clustering model, wherein the clustering model is trained in an unsupervised manner based on the sample fusion data of each of the different types of second sample goods.

[0082] Based on any of the above embodiments, the target volume estimation model includes a convolution module, a pooling layer and a fully connected layer connected in sequence, and the convolution module includes a plurality of 3D convolution layers connected in sequence.

[0083] Figure 6 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the cargo volume estimation method, the method comprising: acquiring multimodal data of the cargo to be estimated, the multimodal data comprising a structured light 3D point cloud of the cargo to be estimated, a radar point cloud of the cargo to be estimated and an image of the cargo to be estimated; Performing registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fused data; Determining the target type of the cargo to be estimated based on the fused data; The fused data is input into a target volume estimation model corresponding to the target type to obtain the volume of the cargo to be estimated output by the target volume estimation model, wherein the target volume estimation model is trained based on sample fusion data of a first sample cargo corresponding to the target type.

[0084] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0085] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the cargo volume estimation method provided by the above methods, the method comprising: acquiring multimodal data of the cargo to be estimated, the multimodal data including a structured light 3D point cloud of the cargo to be estimated, a radar point cloud of the cargo to be estimated, and an image of the cargo to be estimated; Performing registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fused data; Determining the target type of the cargo to be estimated based on the fused data; The fused data is input into a target volume estimation model corresponding to the target type to obtain the volume of the cargo to be estimated output by the target volume estimation model, wherein the target volume estimation model is trained based on sample fusion data of a first sample cargo corresponding to the target type.

[0086] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the cargo volume estimation method provided by the above methods, the method comprising: acquiring multimodal data of the cargo to be estimated, the multimodal data comprising a structured light 3D point cloud of the cargo to be estimated, a radar point cloud of the cargo to be estimated, and an image of the cargo to be estimated; Performing registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fused data; Determining the target type of the cargo to be estimated based on the fused data; The fused data is input into a target volume estimation model corresponding to the target type to obtain the volume of the cargo to be estimated output by the target volume estimation model, wherein the target volume estimation model is trained based on sample fusion data of a first sample cargo corresponding to the target type.

[0087] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cargo volume estimation method, characterized in that: include: Acquire multimodal data of the goods to be estimated, wherein the multimodal data includes a structured light 3D point cloud of the goods to be estimated, a radar point cloud of the goods to be estimated, and an image of the goods to be estimated; Performing registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fused data; Determining the target type of the cargo to be estimated based on the fused data; The fused data is input into a target volume estimation model corresponding to the target type to obtain the volume of the cargo to be estimated output by the target volume estimation model, wherein the target volume estimation model is trained based on sample fusion data of a first sample cargo corresponding to the target type.

2. The cargo volume estimation method according to claim 1, characterized in that: The target volume estimation model is trained based on the following method: Determine a first model structure parameter and a first model training parameter corresponding to the target type based on the correspondence between the type, the model structure parameter and the model training parameter; Determining a first initial volume estimation model based on the first model structure parameters and the first model training parameters; Inputting the sample fusion data of the first sample cargo into the first initial volume estimation model to obtain a first predicted volume of the first sample cargo output by the first initial volume estimation model; Based on the first predicted volume and the volume label corresponding to the first sample cargo, the first model structure parameters and the first model training parameters are iteratively adjusted to obtain the target volume estimation model.

3. The cargo volume estimation method according to claim 1, characterized in that: The step of inputting the fused data into a target volume estimation model corresponding to the target type to obtain the volume of the cargo to be estimated output by the target volume estimation model includes: Determining a target logistics scenario where the goods to be estimated are located; The fused data is input into a target volume estimation model corresponding to the target type and the target logistics scenario, and the volume of the goods to be estimated output by the target volume estimation model is obtained.

4. The cargo volume estimation method according to claim 3, characterized in that: The target volume estimation model is trained based on the following method: Determine the second model structure parameters and the second model training parameters corresponding to the target type and the target logistics scenario based on the correspondence between the logistics scenario, type, model structure parameters and model training parameters; Determining a second initial volume estimation model based on the second model structure parameters and the second model training parameters; Inputting the sample fusion data of the first sample cargo into the second initial volume estimation model to obtain a second predicted volume of the first sample cargo output by the second initial volume estimation model; Based on the second predicted volume and the volume label corresponding to the first sample cargo, the second model structure parameters and the second model training parameters are iteratively adjusted to obtain the target volume estimation model.

5. The cargo volume estimation method according to any one of claims 1 to 4, characterized in that: The registering and fusing the structured light 3D point cloud, the radar point cloud and the image to obtain fused data includes: Performing denoising on the structured light 3D point cloud and the radar point cloud based on a statistical filtering algorithm to obtain a denoised structured light 3D point cloud and a denoised radar point cloud; Performing denoising on the image based on a Gaussian filtering algorithm to obtain a denoised image; The denoised structured light 3D point cloud, the denoised radar point cloud and the denoised image are registered and fused to obtain the fused data.

6. The cargo volume estimation method according to any one of claims 1 to 4, characterized in that: The determining the target type of the cargo to be estimated based on the fused data includes: The fused data is input into a clustering model to obtain the target type of the cargo to be estimated output by the clustering model, wherein the clustering model is trained in an unsupervised manner based on the sample fusion data of each of the different types of second sample cargoes.

7. The cargo volume estimation method according to any one of claims 1 to 4, characterized in that: The target volume estimation model includes a convolution module, a pooling layer and a fully connected layer connected in sequence, and the convolution module includes a plurality of 3D convolution layers connected in sequence.

8. A cargo volume estimation device, characterized in that: include: an acquisition unit, configured to acquire multimodal data of the goods to be estimated, wherein the multimodal data includes a structured light 3D point cloud of the goods to be estimated, a radar point cloud of the goods to be estimated, and an image of the goods to be estimated; A registration and fusion unit, used for performing registration and fusion processing on the structured light 3D point cloud, the radar point cloud and the image to obtain fused data; a determination unit, configured to determine the target type of the cargo to be estimated based on the fused data; an estimation unit, for inputting the fused data into a target volume estimation model corresponding to the target type, and obtaining the volume of the cargo to be estimated output by the target volume estimation model, wherein the target volume estimation model is trained based on the sample fusion data of the first sample cargo corresponding to the target type.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the cargo volume estimation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cargo volume estimation method according to any one of claims 1 to 7 is implemented.

Images