Cargo volume calculation method and system based on panoramic video

Abstract

The present invention discloses a cargo volume calculation method and system based on panoramic video, which belongs to the field of machine vision technology, and specifically includes: obtaining a panoramic video of a warehouse through a panoramic camera, and using deep learning and spatiotemporal context analysis technology to detect and segment the two-dimensional contour of cargo in real time; using a binocular camera and active projection structured light technology to extract cargo depth information, convert it into three-dimensional point cloud data, and construct a high-quality three-dimensional model after radius filtering and optimization processing; using adaptive camera calibration and point cloud registration technology to optimize the three-dimensional model, and accurately calculating the cargo volume through volume integration and physical simulation models; the system also integrates multi-dimensional time series analysis and machine learning models, automatically learns measurement error patterns, and introduces an intelligent error correction mechanism to achieve dynamic adjustment of volume measurement values, thereby improving measurement accuracy and reliability, and enhancing the automation and intelligence level of cargo volume measurement.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision, and in particular to a cargo volume calculation method and system based on panoramic video. Background Art

[0002] With the continuous development of machine vision technology, the method of measuring cargo volume has also been greatly improved. Traditional cargo volume measurement methods usually use manual measurement of length, width and height or plane two-dimensional vision technology measurement methods; among them, the manual measurement method mainly relies on the accuracy of the measuring ruler, there are large subjective differences, large workload and low efficiency; and the measurement method of plane two-dimensional vision technology mainly uses the conversion relationship between pixel length and physical length to calculate the actual size of the object under a fixed depth of field, but due to the limitation of its imaging principle, it is impossible to make a specific analysis based on the actual situation of near big and far small, and can only measure cargo of a single specification. Therefore, when this method is combined with logistics application scenarios, it is impossible to perform targeted measurements according to cargo of different specifications, and it is even more impossible to accurately measure the volume of stored cargo. Therefore, the traditional cargo volume measurement method is not only time-consuming and labor-intensive, but also inefficient in large-scale warehouse environments. Although there are image-based volume estimation technologies, their accuracy is often difficult to guarantee due to viewing angle limitations and occlusion problems.

[0003] For example, the patent with publication number CN116363192A discloses a method, device, computer equipment and storage medium for measuring the volume of stored goods, including: obtaining the three-dimensional point cloud data of the stored goods to be measured; performing deep filtering processing on the three-dimensional point cloud data based on the preset impurity influence order to obtain the filtered three-dimensional point cloud data; performing point cloud filling processing on the filtered three-dimensional point cloud data to obtain the target three-dimensional point cloud data; parsing the target three-dimensional point cloud data to obtain the cargo volume information of the stored goods to be measured. With this technical solution, the point cloud data of the stored goods to be measured can be accurately extracted and improved, thereby effectively improving the accuracy of the volume measurement of the stored goods.

[0004] For example, the patent with publication number CN115330854A discloses a cargo management system and a cargo management method, including: a depth detection device, a processor, and a cargo moving device. The depth detection device is used to collect a three-dimensional image of the cargo transported to a preset detection position; the processor is used to calculate the volume of the cargo based on the three-dimensional image, and calculate the quantity of the cargo based on the volume of the cargo and the preset unit volume of the cargo; the cargo moving device is used to transport the cargo to the preset warehouse after determining the quantity of the cargo; the processor is also used to update the inventory information based on the quantity of the cargo after the cargo is put into storage. The depth detection device automatically collects the three-dimensional image of the cargo, and then calculates the volume of the cargo through the three-dimensional image, and accurately calculates the quantity of the cargo based on the volume and the preset unit volume of the cargo. After the quantity of the cargo is calculated, the cargo can be put into storage and the inventory information can be updated, thereby automatically realizing the calculation of the quantity of the cargo and the update of the inventory information, which can improve the update speed and accuracy of the inventory information.

[0005] The above existing technologies all have the following problems: 1) lack of multi-view data fusion; 2) lack of specific methods for obtaining depth information; 3) lack of error correction and dynamic adjustment mechanism; 4) lack of adaptability to complex environments. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a cargo volume calculation method and system based on panoramic video. The panoramic video of the warehouse is obtained by a panoramic camera, and the two-dimensional contour of the cargo is detected and segmented in real time by using deep learning and spatiotemporal context analysis technology. A binocular camera and active projection structured light technology are used to extract cargo depth information, which is converted into three-dimensional point cloud data, and a high-quality three-dimensional model is constructed after radius filtering and optimization processing. The three-dimensional model is optimized by using adaptive camera calibration and point cloud registration technology, and the cargo volume is accurately calculated by volume integration and physical simulation model. The system also integrates multi-dimensional time series analysis and machine learning models to automatically learn measurement error patterns, and introduces an intelligent error correction mechanism to realize dynamic adjustment of volume measurement values, thereby improving measurement accuracy and reliability, and enhancing the automation and intelligence level of cargo volume measurement.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The cargo volume calculation method based on panoramic video includes:

[0009] Step S1: Obtain a panoramic video stream in the warehouse through a panoramic camera, and use a deep learning algorithm combined with a spatiotemporal context analysis method to perform real-time detection and segmentation of goods in the video frame to obtain a two-dimensional outline of the goods;

[0010] Step S2: Using a binocular camera and active projected structured light technology, extract the depth information of the cargo surface from the video frame, and convert the extracted depth information into three-dimensional point cloud data. Using radius filtering and optimization algorithms, remove noise and outliers in the three-dimensional point cloud data to generate new three-dimensional point cloud data.

[0011] Step S3: using an adaptive camera calibration algorithm to establish a three-dimensional coordinate system corresponding to the actual space, based on the new three-dimensional point cloud data, using a point cloud registration method to align point cloud data of different perspectives, and using a surface reconstruction algorithm to construct a preliminary three-dimensional model, and then optimize the preliminary three-dimensional model;

[0012] Step S4: Based on the optimized three-dimensional model, the actual volume of the cargo is calculated by using the grid division and volume integration method. At the same time, in the volume calculation process, a physical simulation model is introduced to simulate the actual cargo, and the cargo volume data verified by the physical simulation is output;

[0013] Step S5: Collect volume measurement results at multiple time points, combine them with environmental sensor data, perform multi-dimensional time series analysis, build a machine learning model, use historical data and real-time feedback to automatically learn the error pattern in the measurement process, and at the same time, introduce an intelligent error correction mechanism to dynamically adjust the volume measurement value according to the prediction results of the machine learning model.

[0014] Specifically, the specific steps of extracting the cargo surface depth information in step S2 include:

[0015] S2.1: Set the internal parameters of each binocular camera and the relative position and posture parameters between binocular cameras;

[0016] S2.2: Correct the original image captured by the binocular camera so that the corrected images are on the same plane and parallel to each other, obtain the right image and the left image, and match the pixels on the corrected images;

[0017] S2.3: Calculate the disparity of each pixel based on the matching point pairs obtained by stereo matching ,in, represents the horizontal pixel value of the i-th pixel in the right image, represents the horizontal pixel value of the i-th pixel in the left image, where i represents the coordinate index of the pixel in the image;

[0018] S2.4: Based on disparity information And camera parameters, use the triangulation principle to calculate the depth information of the object and optimize the depth map. The formula is:

[0019] ;

[0020] Where d represents the depth from the object to the camera. represents the focal length of the camera, represents the baseline length between the binocular cameras, represents the lens distortion correction factor, represents the ambient light compensation factor, represents the distance-dependent attenuation coefficient, Represents the angle between the surface normal of an object and the line of sight.

[0021] Specifically, the specific steps of step S3 include:

[0022] S3.1: Use a camera to shoot the calibration object from different angles to obtain the cargo calibration image, and extract the characteristic corner points of the calibration object from the cargo calibration image;

[0023] S3.2: According to each 3D point of the extracted calibration feature corner points and the corresponding observation points on the cargo calibration image , using the current camera parameters , predict the position of the 3D point on the cargo calibration image through the camera model , calculate the reprojection error , and perform iterative optimization, the formula is:

[0024] ;

[0025] in, represents the coordinate position of the jth 3D point in the characteristic corner point of the calibration object, Represents the coordinate index of the first dimension of the j-th three-dimensional point, Represents the coordinate index of the second dimension of the j-th three-dimensional point, Represents the coordinate index of the third dimension of the j-th three-dimensional point, represents the coordinate position of the observation point corresponding to the j-th three-dimensional point, Represents the horizontal coordinate index of the observation point corresponding to the j-th 3D point, represents the ordinate index of the observation point corresponding to the jth 3D point, K represents the intrinsic parameter matrix, R represents the extrinsic parameter matrix, represents the distortion parameter, b represents the number of distortion parameters, t represents the timestamp, represents the predicted coordinate position of the j-th 3D point, Represents the horizontal coordinate index of the predicted j-th 3D point, Represents the ordinate index of the predicted j-th 3D point, represents the scale factor of the j-th 3D point, and Both represent the distortion correction residual, and Both represent noise terms.

[0026] Specifically, the specific steps of step S3 also include:

[0027] S3.3: Use a depth camera to obtain depth images of the goods from different viewing angles or positions, and convert the grayscale value or code of each pixel in the depth image into a physical distance value to obtain a pixel depth value;

[0028] S3.4: According to the calibrated camera parameters and pixel depth values, each pixel point in the image coordinate system is converted into a three-dimensional point in the camera coordinate system, and the converted three-dimensional coordinate points are collected to form point cloud data;

[0029] S3.5: Using the improved iterative closest point algorithm, the point cloud data is aligned to the same coordinate system through coordinate transformation, preliminary corresponding point pairs are found based on feature matching and spatial position relationship, and incorrect matching pairs are eliminated through random sampling consistency.

[0030] Specifically, the specific steps of step S3 also include:

[0031] S3.6: Use the preliminary registration result obtained in step S3.5 as the initial value of iterative optimization, solve the optimal rotation matrix and translation vector through iterative optimization, use the least squares method to calculate the minimum error of the corresponding point pair, and obtain the registered point cloud data;

[0032] S3.7: Construct an octree based on the registered point cloud data, set the function space and vector field, and use the Laplace matrix to iteratively solve the Poisson equation. Based on the solution of the Poisson equation, use the isosurface extraction algorithm to extract the isosurface;

[0033] S3.8: Construct a three-dimensional surface model based on the isosurface and optimize the three-dimensional surface model using a mesh optimization algorithm.

[0034] Specifically, the specific steps of S3.5 include:

[0035] S3.51: pre-process the point cloud data and set the initial transformation matrix M;

[0036] S3.52: In each iteration, for each point in the point cloud data, the KD tree is used to find its nearest point in the target point cloud as the corresponding point;

[0037] S3.53: Based on the corresponding point pairs found, use the least squares method to calculate the new transformation matrix , and the calculated transformation matrix Applied to point cloud data, update the point cloud data position, and obtain a preliminary set of corresponding point pairs;

[0038] S3.54: Randomly select N point pairs from the preliminary corresponding point pair set, use the N point pairs to estimate the transformation model, and calculate the error of all remaining point pairs under the estimated transformation model , l represents the number of point pairs in the point pair set.

[0039] Specifically, the specific steps of S3.5 also include:

[0040] S3.55: Set the point pair threshold to h and the number of iterations to r;

[0041] like , it means the pairing is correct;

[0042] like , it means the pairing is wrong;

[0043] S3.56: If , then update the transformation model parameters and point pair set until the preset number of iterations is reached, where Indicates that the point pair The error under the transformation model at the iteration, Indicates that the point pair The error under the transformation model at the iteration, Indicates that the point pair The correct set is paired at the iteration. Indicates that the point pair The correct set is paired at the iteration. A counting function representing the number of sets;

[0044] S3.57: Use the final set of inliers and the corresponding transformation model as the preliminary registration result.

[0045] The cargo volume calculation system based on panoramic video includes: image analysis module, 3D reconstruction module, volume calculation module, error correction module and data management module;

[0046] The image analysis module is used to obtain a panoramic video stream of the goods, and perform preprocessing to extract the contours and feature points of the goods;

[0047] The three-dimensional reconstruction module uses the output data of the image analysis module to reconstruct the three-dimensional model of the goods through stereo matching and depth estimation methods;

[0048] The volume calculation module calculates the volume of the cargo based on the three-dimensional model of the cargo obtained by the three-dimensional reconstruction module;

[0049] The error correction module uses a statistical method to correct the calculated volume data;

[0050] The data management module is used to store, manage and retrieve the data generated during the entire measurement process.

[0051] Specifically, the 3D reconstruction module includes: a point cloud generation unit, a point cloud optimization unit, a calibration unit, a point cloud registration unit, and a 3D model construction unit;

[0052] The point cloud generation unit generates point cloud data of the surface of the cargo through stereo vision based on image feature matching;

[0053] The calibration unit is used to calibrate the internal and external parameters of the camera device used to ensure the accuracy of the conversion to the real world coordinate system;

[0054] The point cloud registration unit uses a point cloud registration method to align and merge point cloud data at different viewing angles or time points to form a point cloud model of the complete cargo;

[0055] The point cloud optimization unit is used to perform filtering, denoising, and hole filling operations on the initial point cloud data to improve the quality of the point cloud;

[0056] The three-dimensional model construction unit is used to convert the optimized point cloud data into a solid model and complete the three-dimensional reconstruction through a surface reconstruction algorithm.

[0057] Specifically, the volume calculation module includes a volume calculation unit, which is equipped with a grid division and volume integration algorithm, and the grid division and volume integration algorithm are used to calculate the actual volume of the goods; the error correction module includes an error model construction unit and an error correction unit, and the error model construction unit is used to construct an error model based on machine learning through multi-dimensional time series analysis, and the error correction unit is equipped with an intelligent error correction mechanism for dynamically adjusting the volume measurement value.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. The present invention proposes a cargo volume calculation system based on panoramic video, and optimizes and improves the architecture, operation steps and processes. The system has the advantages of simple process, low investment and operation costs, and low production work costs.

[0060] 2. The present invention proposes a cargo volume calculation method based on panoramic video. By combining panoramic cameras, binocular cameras and active projection structured light technology, as well as deep learning algorithms and spatiotemporal context analysis methods, it can capture the two-dimensional contours and three-dimensional information of cargo in the warehouse in real time and accurately, thereby improving the accuracy of cargo volume measurement and reducing human errors and systematic errors. By utilizing advanced point cloud processing technology and three-dimensional model construction algorithms, it can efficiently process large-scale three-dimensional point cloud data and construct an accurate three-dimensional model. In addition, the cargo volume is calculated through grid division and volume integration methods, further improving the accuracy and efficiency of data processing.

[0061] 3. The present invention proposes a cargo volume calculation method based on panoramic video. The entire calculation process is highly automated and intelligent. From cargo detection, segmentation, three-dimensional data acquisition, model construction to volume calculation, automated processing without human intervention is achieved. At the same time, by introducing machine learning models and intelligent error correction mechanisms, the system can automatically learn error patterns in the measurement process and dynamically adjust the measurement values ​​according to real-time feedback, thereby improving the intelligence level and response speed of the system. At the same time, the system can obtain the video stream of the cargo in the warehouse in real time, and process and analyze it immediately, which helps to grasp the dynamic changes of the cargo in a timely manner and provide real-time and accurate data support for inventory management. Through multi-dimensional time series analysis and the construction of machine learning models, measurement errors can be more accurately predicted and corrected, thereby improving the accuracy and stability of the measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of a cargo volume calculation method based on panoramic video according to the present invention;

[0063] Figure 2 The following is a flow chart showing the principle of the cargo volume calculation method based on panoramic video of the present invention;

[0064] Figure 3 A flowchart of constructing a three-dimensional surface model for a cargo volume calculation method based on panoramic video according to the present invention;

[0065] Figure 4 This is an architecture diagram of the cargo volume calculation system based on panoramic video of the present invention. DETAILED DESCRIPTION

[0066] Example 1

[0067] See also Figure 1-Figure 2 , an embodiment of the present invention provides: a method for calculating cargo volume based on panoramic video, comprising the following steps:

[0068] Step S1: Obtain a panoramic video stream in the warehouse through a panoramic camera, and use a deep learning algorithm combined with a spatiotemporal context analysis method to perform real-time detection and segmentation of goods in the video frame to obtain a two-dimensional outline of the goods;

[0069] Furthermore, the specific steps of step S1 include:

[0070] S1.1: Install Insta360 Pro panoramic cameras in the warehouse to ensure that all areas that need to be monitored are covered, and ensure that the network connection between the camera and the data processing center is stable and reliable to transmit the panoramic video stream in real time;

[0071] S1.2: Decode the received panoramic video stream and convert it into AVI format, and split the video stream into individual video frames for frame-by-frame processing;

[0072] S1.3: Use the semantic segmentation model to detect and segment goods. Use labeled goods images or video frames as training data to train the model. During the training process, attention should be paid to the accuracy and speed of the model to ensure real-time detection requirements.

[0073] S1.4: In the detection process, the timing information and context information of the video frame are combined to improve the accuracy and stability of the detection. In the present invention, the optical flow method is used to track the movement trajectory of the goods between different frames, thereby reducing false detection and missed detection;

[0074] S1.5: Apply the trained deep learning model to each frame of video to detect the goods, and obtain the position and size of the goods in the image. At the same time, apply the semantic segmentation model to the detected goods area to separate the goods from the background and obtain the two-dimensional outline of the goods;

[0075] S1.6: Post-process the segmentation results, such as removing noise and filling holes, to improve the accuracy and smoothness of the contour, and output the processed two-dimensional contour of the goods in image or data format.

[0076] Step S2: Using a binocular camera and active projected structured light technology, extract the depth information of the cargo surface from the video frame, and convert the extracted depth information into three-dimensional point cloud data. Using radius filtering and optimization algorithms, remove noise and outliers in the three-dimensional point cloud data to generate new three-dimensional point cloud data.

[0077] Furthermore, the specific steps of step S2 include:

[0078] (1) Calibrate the binocular camera to determine the positional relationship between the two cameras and internal parameters such as focal length and optical center;

[0079] (2) Use binocular cameras to simultaneously capture scenes in the warehouse, ensuring that the two cameras capture images at the same time;

[0080] (3) Actively projecting structured light, such as laser stripes and gratings, onto the surface of the cargo to form characteristic patterns. These characteristic patterns are deformed on the surface of the cargo and captured by the binocular camera.

[0081] (4) Using the parallax principle of binocular vision, depth information is calculated by matching corresponding feature points in the left and right images;

[0082] (5) Based on the calculated depth information, the pixels in the two-dimensional image are mapped into three-dimensional space to form three-dimensional point cloud data;

[0083] (6) Apply algorithms such as radius filtering to remove noise and outliers in the 3D point cloud, and use statistical filtering to further smooth the point cloud data;

[0084] (7) Output the processed 3D point cloud data for subsequent processing.

[0085] Step S3: using an adaptive camera calibration algorithm to establish a three-dimensional coordinate system corresponding to the actual space, based on the new three-dimensional point cloud data, using a point cloud registration method to align point cloud data of different perspectives, and using a surface reconstruction algorithm to construct a preliminary three-dimensional model, and then optimize the preliminary three-dimensional model;

[0086] Step S4: Based on the optimized three-dimensional model, the actual volume of the cargo is calculated by using a grid division and volume integration method. At the same time, in the volume calculation process, a physical simulation model is introduced to simulate the actual cargo, and the cargo volume data verified by the physical simulation is output. The grid division and volume integration methods are the prior art content in this field, and are not the inventive solution of the present application, and will not be described in detail here.

[0087] Furthermore, the specific steps of step S4 include:

[0088] S4.1: Clean and optimize the original 3D model, remove redundant data, repair holes, ensure that the model surface is closed and the normal vector direction is consistent, and use the 3D modeling software Blender to optimize the model;

[0089] S4.2: discretize the optimized three-dimensional model into a mesh consisting of simple triangles or quadrilaterals;

[0090] S4.3: According to the meshing result, use the volume calculation formula based on the basic voxel of the tetrahedron to calculate the volume of each basic voxel, and add up the volumes of all the basic voxels to obtain the total volume of the cargo;

[0091] S4.4: Build a physical simulation model to simulate the physical properties of the cargo and the external environment, and use the physical simulation software ANSYS to perform simulations to verify and optimize the physical properties of the three-dimensional model, where the physical properties include: mass, density, elasticity, and the external environment includes: gravity, friction;

[0092] S4.5: Input the total volume of the cargo into the physical simulation model, output the cargo volume data, and compare it with the actual measured data or known data to verify the accuracy of the calculated results.

[0093] Step S5: Collect volume measurement results at multiple time points, combine them with environmental sensor data, perform multi-dimensional time series analysis, build a machine learning model, use historical data and real-time feedback to automatically learn the error pattern in the measurement process, and at the same time, introduce an intelligent error correction mechanism to dynamically adjust the volume measurement value according to the prediction results of the machine learning model.

[0094] The specific steps of extracting cargo surface depth information in step S2 include:

[0095] S2.1: Set the internal parameters of each binocular camera and the relative position and posture parameters between binocular cameras;

[0096] S2.2: Correct the original image captured by the binocular camera so that the corrected images are on the same plane and parallel to each other, obtain the right image and the left image, and match the pixels on the corrected images;

[0097] Furthermore, the specific steps of correcting the original image captured by the binocular camera include:

[0098] S2.21: According to the step of S2.1, determine whether the internal parameters of each camera and the relative position and posture parameters between the binocular cameras are set, and the internal parameters include focal length, optical center, and distortion coefficient;

[0099] S2.22: If it has been set, the original images captured by the two cameras are corrected using the Bouguet epipolar correction method, wherein the Bouguet epipolar correction method is a prior art content in the art and is not an inventive solution of the present application, and is not described in detail here;

[0100] S2.23: Use the stereoRectify function in the OpenCV library to calculate the mapping matrix for correction based on the camera's intrinsic matrix, distortion coefficient, rotation matrix, and translation vector. The mapping matrix is ​​used to transform the original image into the corrected image.

[0101] S2.24: Use the initUndistortRectifyMap function of OpenCV to calculate the distortion map, and use the remap function to apply the obtained map to the image to obtain the corrected image;

[0102] S2.25: By observing the corrected images, check whether the images are on the same plane and parallel to each other. At the same time, check whether the epipolar lines of the images are horizontal to verify the accuracy of the correction;

[0103] S2.26: performing pixel matching on the corrected image;

[0104] S2.3: Calculate the disparity of each pixel based on the matching point pairs obtained by stereo matching ,in, represents the horizontal pixel value of the i-th pixel in the right image, represents the horizontal pixel value of the i-th pixel in the left image, where i represents the coordinate index of the pixel in the image;

[0105] S2.4: Based on disparity information And camera parameters, use the triangulation principle to calculate the depth information of the object and optimize the depth map. The formula is:

[0106] ;

[0107] Where d represents the depth from the object to the camera. represents the focal length of the camera, represents the baseline length between the binocular cameras, represents the lens distortion correction factor, represents the ambient light compensation factor, represents the distance-dependent attenuation coefficient, Represents the angle between the surface normal of an object and the line of sight.

[0108] Example 2

[0109] See also Figure 3 In this embodiment, the specific steps of step S3 include:

[0110] S3.1: Use a camera to shoot the calibration object from different angles to obtain the cargo calibration image, and extract the characteristic corner points of the calibration object from the cargo calibration image;

[0111] S3.2: According to each 3D point of the extracted calibration feature corner points and the corresponding observation points on the cargo calibration image , using the current camera parameters , predict the position of the 3D point on the cargo calibration image through the camera model , calculate the reprojection error , and perform iterative optimization, the formula is:

[0112] ;

[0113] in, represents the coordinate position of the jth 3D point in the characteristic corner point of the calibration object, Represents the coordinate index of the first dimension of the j-th three-dimensional point, Represents the coordinate index of the second dimension of the j-th three-dimensional point, Represents the coordinate index of the third dimension of the j-th three-dimensional point, represents the coordinate position of the observation point corresponding to the j-th three-dimensional point, Represents the horizontal coordinate index of the observation point corresponding to the j-th 3D point, represents the ordinate index of the observation point corresponding to the jth 3D point, K represents the intrinsic parameter matrix, R represents the extrinsic parameter matrix, represents the distortion parameter, b represents the number of distortion parameters, t represents the timestamp, represents the predicted coordinate position of the j-th 3D point, Represents the horizontal coordinate index of the predicted j-th 3D point, Represents the ordinate index of the predicted j-th 3D point, represents the scale factor of the j-th 3D point, and Both represent the distortion correction residual, and Both represent noise terms;

[0114] Furthermore, the camera intrinsic parameter matrix and extrinsic parameter matrix are: , ,in, represents the normalized focal length, represents the optical center coordinates, , , Represent the three-dimensional translation vector respectively, Represents the external parameter matrix element.

[0115] S3.3: Use a depth camera to obtain depth images of the goods from different viewing angles or positions, and convert the grayscale value or code of each pixel in the depth image into a physical distance value to obtain a pixel depth value;

[0116] S3.4: According to the calibrated camera parameters and pixel depth values, each pixel point in the image coordinate system is converted into a three-dimensional point in the camera coordinate system, and the converted three-dimensional coordinate points are collected to form point cloud data;

[0117] S3.5: Using the improved iterative closest point algorithm, the point cloud data is aligned to the same coordinate system through coordinate transformation, preliminary corresponding point pairs are found based on feature matching and spatial position relationship, and wrong matching pairs are eliminated through random sampling consistency;

[0118] S3.6: Use the preliminary registration result obtained in step S3.5 as the initial value of iterative optimization, solve the optimal rotation matrix and translation vector through iterative optimization, use the least squares method to calculate the minimum error of the corresponding point pair, and obtain the registered point cloud data;

[0119] Furthermore, the specific steps of S3.6 include:

[0120] S3.61: Initialization: Use the preliminary alignment result obtained in step S3.5 as the initial value of the iterative optimization, including the initial value of the rotation matrix and the initial value of the translation vector;

[0121] S3.62: Iterative process:

[0122] 1) Find corresponding points: Based on the current rotation matrix and translation vector, transform the point cloud data to the coordinate system of the target point cloud, and find corresponding point pairs whose distance between the transformed point cloud and the target point cloud is less than a certain threshold;

[0123] 2) Optimizing the rotation matrix and translation vector: Using the corresponding point pairs found, the new rotation matrix and translation vector are solved by the least squares method to minimize the error between the corresponding point pairs. In the present invention, the minimum error is solved by calculating the sum of squares of the Euclidean distances. The calculation formula of the least squares method and the solution formula of the sum of squares of the Euclidean distances are the prior art content in this field, which is not the inventive solution of the present application and will not be described in detail here.

[0124] 3) Iterative update: Repeat the above steps using the new rotation matrix and translation vector until the iteration termination condition is met, such as the change in the rotation matrix and translation vector is less than the preset threshold, or the error between the corresponding point pairs is less than the preset threshold;

[0125] S3.63: Output the final rotation matrix and translation vector, as well as the registered point cloud data.

[0126] S3.7: Construct an octree based on the registered point cloud data, set the function space and vector field, and use the Laplace matrix to iteratively solve the Poisson equation. Based on the solution of the Poisson equation, use the isosurface extraction algorithm to extract the isosurface;

[0127] Furthermore, the specific steps of S3.7 include:

[0128] S3.71: Poisson surface reconstruction:

[0129] (1) Constructing an octree: Construct an octree based on the registered point cloud data for space partitioning;

[0130] (2) Setting the function space: Setting a spatial function for each node of the octree. The linear sum of all node functions can represent a vector field.

[0131] (3) Create a vector field: approximate the gradient of the indicator function through the vector field;

[0132] (4) Solving the Poisson equation: using the Laplace matrix to iteratively solve the Poisson equation to obtain an approximate solution of the indicator function, wherein the Laplace matrix solution formula is the prior art content in this field, is not the inventive solution of the present application, and is not described in detail here;

[0133] (5) Extracting isosurfaces: selecting a threshold value q and using an isosurface extraction algorithm to extract isosurfaces from the solution of the indicator function as the initial mesh of the three-dimensional surface model, wherein the isosurface extraction algorithm is a prior art in the art and is not an inventive solution of the present application, and is not described in detail herein;

[0134] S3.72: Smooth, denoise, simplify and optimize the extracted three-dimensional surface model mesh to improve the quality of the mesh.

[0135] S3.8: Construct a three-dimensional surface model based on the isosurface, and use a mesh optimization algorithm to optimize the three-dimensional surface model, wherein the mesh optimization algorithm is the prior art content in this field, is not the creative solution of this application, and is not elaborated here.

[0136] The specific steps of S3.5 include:

[0137] S3.51: pre-process the point cloud data and set the initial transformation matrix M;

[0138] S3.52: In each iteration, for each point in the point cloud data, the KD tree is used to find its nearest point in the target point cloud as the corresponding point;

[0139] S3.53: Based on the corresponding point pairs found, use the least squares method to calculate the new transformation matrix , and the calculated transformation matrix Applied to point cloud data, update the point cloud data position, and obtain a preliminary set of corresponding point pairs;

[0140] S3.54: Randomly select N point pairs from the preliminary corresponding point pair set, use the N point pairs to estimate the transformation model, and calculate the error of all remaining point pairs under the estimated transformation model , l represents the number of point pairs in the point pair set;

[0141] S3.55: Set the point pair threshold to h and the number of iterations to r;

[0142] like , it means the pairing is correct;

[0143] like , it means the pairing is wrong;

[0144] S3.56: If , then update the transformation model parameters and point pair set until the preset number of iterations is reached, where Indicates that the point pair The error under the transformation model at the iteration, Indicates that the point pair The error under the transformation model at the iteration, Indicates that the point pair The correct set is paired at the iteration. Indicates that the point pair The correct set is paired at the iteration. A counting function representing the number of sets;

[0145] S3.57: Use the final set of inliers and the corresponding transformation model as the preliminary registration result.

[0146] Example 3

[0147] See also Figure 4 Another embodiment provided by the present invention is a cargo volume calculation system based on panoramic video, comprising:

[0148] Image analysis module, 3D reconstruction module, volume calculation module, error correction module, data management module;

[0149] Image analysis module, used to obtain panoramic video stream of goods, perform preprocessing, and extract the outline and feature points of goods;

[0150] The 3D reconstruction module uses the output data of the image analysis module to reconstruct the 3D model of the cargo through stereo matching and depth estimation methods. The purpose is to restore the 3D spatial structure of the cargo from the 2D image information and provide a direct basis for volume calculation;

[0151] A volume calculation module calculates the volume of the cargo based on the three-dimensional model of the cargo obtained by the three-dimensional reconstruction module;

[0152] The error correction module uses statistical methods to correct the calculated volume data to improve the accuracy of the final result;

[0153] The data management module is used to store, manage and retrieve the data generated during the entire measurement process, including raw video, intermediate processing results, final volume data and calibration records.

[0154] The 3D reconstruction module includes: point cloud generation unit, point cloud optimization unit, calibration unit, point cloud registration unit, and 3D model construction unit;

[0155] Point cloud generation unit, which generates point cloud data of the cargo surface through stereo vision based on image feature matching;

[0156] The calibration unit is used to calibrate the internal and external parameters of the camera device used to ensure the accuracy of the conversion to the real-world coordinate system;

[0157] Point cloud registration unit, which uses point cloud registration method to align and merge point cloud data from different perspectives or time points to form a point cloud model of the complete cargo;

[0158] Point cloud optimization unit, used to filter, denoise, and fill holes on the initial point cloud data to improve the quality of the point cloud;

[0159] The 3D model building unit is used to convert the optimized point cloud data into a solid model and complete the 3D reconstruction through the surface reconstruction algorithm.

[0160] The volume calculation module includes: a volume calculation unit and a simulation verification unit;

[0161] The volume calculation unit calculates the volume formula of the geometric body based on the constructed three-dimensional model through mesh division and volume integration method to obtain the actual volume of the cargo;

[0162] The simulation verification unit is used to verify the calculated volume through simulation methods, simulating the stability and accuracy of volume calculation under different conditions, such as when the cargo placement posture changes.

[0163] The error correction module includes: a data analysis unit, an error model building unit, and an error correction unit;

[0164] A data analysis unit is used to collect error data generated during reconstruction and calculation, perform statistical analysis, and identify the source of errors;

[0165] The error model building unit builds a mathematical model to describe the relationship between the error and the error factors based on the analysis results;

[0166] The error correction unit applies the error model to compensate and adjust the measurement results, reduce systematic or random errors, and improve the accuracy of the final volume measurement.

[0167] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in the field may also change, modify, replace and modify the above-mentioned embodiments without departing from the purpose and scope of protection of the present invention, and all of these are within the protection of the present invention.

Claims

1. A cargo volume calculation method based on panoramic video, characterized in that: include: Step S1: Obtain a panoramic video stream in the warehouse through a panoramic camera, and use a deep learning algorithm combined with a spatiotemporal context analysis method to perform real-time detection and segmentation of goods in the video frame to obtain a two-dimensional outline of the goods; Step S2: Using a binocular camera and active projected structured light technology, extract the depth information of the cargo surface from the video frame, and convert the extracted depth information into three-dimensional point cloud data. Using radius filtering and optimization algorithms, remove noise and outliers in the three-dimensional point cloud data to generate new three-dimensional point cloud data. Step S3: using an adaptive camera calibration algorithm to establish a three-dimensional coordinate system corresponding to the actual space, based on the new three-dimensional point cloud data, using a point cloud registration method to align point cloud data of different perspectives, and using a surface reconstruction algorithm to construct a preliminary three-dimensional model, and then optimize the preliminary three-dimensional model; Step S4: Based on the optimized three-dimensional model, the actual volume of the cargo is calculated by using the grid division and volume integration method. At the same time, in the volume calculation process, a physical simulation model is introduced to simulate the actual cargo, and the cargo volume data verified by the physical simulation is output; Step S5: Collect volume measurement results at multiple time points, combine them with environmental sensor data, perform multi-dimensional time series analysis, build a machine learning model, use historical data and real-time feedback to automatically learn the error pattern in the measurement process, and at the same time, introduce an intelligent error correction mechanism to dynamically adjust the volume measurement value according to the prediction results of the machine learning model.

2. The cargo volume calculation method based on panoramic video according to claim 1, characterized in that: The specific steps of extracting cargo surface depth information in step S2 include: S2.1: Set the internal parameters of each binocular camera and the relative position and posture parameters between binocular cameras; S2.2: Correct the original image captured by the binocular camera so that the corrected images are on the same plane and parallel to each other, obtain the right image and the left image, and match the pixels on the corrected images; S2.3: Calculate the disparity of each pixel based on the matching point pairs obtained by stereo matching ,in, represents the horizontal pixel value of the i-th pixel in the right image, represents the horizontal pixel value of the i-th pixel in the left image, where i represents the coordinate index of the pixel in the image; S2.4: Based on disparity information And camera parameters, use the triangulation principle to calculate the depth information of the object and optimize the depth map. The formula is: ； Where d represents the depth from the object to the camera. represents the focal length of the camera, represents the baseline length between the binocular cameras, represents the lens distortion correction factor, represents the ambient light compensation factor, represents the distance-dependent attenuation coefficient, Represents the angle between the surface normal of an object and the line of sight.

3. The cargo volume calculation method based on panoramic video according to claim 2, characterized in that: The specific steps of step S3 include: S3.1: Use a camera to shoot the calibration object from different angles to obtain the cargo calibration image, and extract the characteristic corner points of the calibration object from the cargo calibration image; S3.2: According to each 3D point of the extracted calibration feature corner points and the corresponding observation points on the cargo calibration image , using the current camera parameters , predict the position of the 3D point on the cargo calibration image through the camera model , calculate the reprojection error , and perform iterative optimization, the formula is: ； in, represents the coordinate position of the jth 3D point in the characteristic corner point of the calibration object, Represents the coordinate index of the first dimension of the j-th three-dimensional point, Represents the coordinate index of the second dimension of the j-th three-dimensional point, Represents the coordinate index of the third dimension of the j-th three-dimensional point, represents the coordinate position of the observation point corresponding to the j-th three-dimensional point, Represents the horizontal coordinate index of the observation point corresponding to the j-th 3D point, represents the ordinate index of the observation point corresponding to the jth 3D point, K represents the intrinsic parameter matrix, R represents the extrinsic parameter matrix, represents the distortion parameter, b represents the number of distortion parameters, t represents the timestamp, represents the predicted coordinate position of the j-th 3D point, Represents the horizontal coordinate index of the predicted j-th 3D point, Represents the ordinate index of the predicted j-th 3D point, represents the scale factor of the j-th 3D point, and Both represent the distortion correction residual, and Both represent noise terms.

4. The cargo volume calculation method based on panoramic video according to claim 3, characterized in that: The specific steps of step S3 also include: S3.3: Use a depth camera to obtain depth images of the goods from different viewing angles or positions, and convert the grayscale value or code of each pixel in the depth image into a physical distance value to obtain a pixel depth value; S3.4: According to the calibrated camera parameters and pixel depth values, each pixel point in the image coordinate system is converted into a three-dimensional point in the camera coordinate system, and the converted three-dimensional coordinate points are collected to form point cloud data; S3.5: Using the improved iterative closest point algorithm, the point cloud data is aligned to the same coordinate system through coordinate transformation, preliminary corresponding point pairs are found based on feature matching and spatial position relationship, and incorrect matching pairs are eliminated through random sampling consistency.

5. The cargo volume calculation method based on panoramic video according to claim 4, characterized in that: The specific steps of step S3 also include: S3.6: Use the preliminary registration result obtained in step S3.5 as the initial value of iterative optimization, solve the optimal rotation matrix and translation vector through iterative optimization, use the least squares method to calculate the minimum error of the corresponding point pair, and obtain the registered point cloud data; S3.7: Construct an octree based on the registered point cloud data, set the function space and vector field, and use the Laplace matrix to iteratively solve the Poisson equation. Based on the solution of the Poisson equation, use the isosurface extraction algorithm to extract the isosurface; S3.8: Construct a three-dimensional surface model based on the isosurface and optimize the three-dimensional surface model using a mesh optimization algorithm.

6. The cargo volume calculation method based on panoramic video according to claim 5, characterized in that: The specific steps of S3.5 include: S3.51: pre-process the point cloud data and set the initial transformation matrix M; S3.52: In each iteration, for each point in the point cloud data, the KD tree is used to find its nearest point in the target point cloud as the corresponding point; S3.53: Based on the corresponding point pairs found, use the least squares method to calculate the new transformation matrix , and the calculated transformation matrix Applied to point cloud data, update the point cloud data position, and obtain a preliminary set of corresponding point pairs; S3.54: Randomly select N point pairs from the preliminary corresponding point pair set, use the N point pairs to estimate the transformation model, and calculate the error of all remaining point pairs under the estimated transformation model , l represents the number of point pairs in the point pair set.

7. The cargo volume calculation method based on panoramic video according to claim 6, characterized in that: The specific steps of S3.5 also include: S3.55: Set the point pair threshold to h and the number of iterations to r. like , it means the pairing is correct; like , it means the pairing is wrong; S3.56: If , then update the transformation model parameters and point pair set until the preset number of iterations is reached, where Indicates that the point pair The error under the transformation model at the iteration, Indicates that the point pair The error under the transformation model at the iteration, Indicates that the point pair The correct set is paired at the iteration. Indicates that the point pair The correct set is paired at the iteration, A counting function representing the number of sets; S3.57: Use the final set of inliers and the corresponding transformation model as the preliminary registration result.

8. A cargo volume calculation system based on panoramic video, which is used to implement the cargo volume calculation method based on panoramic video according to any one of claims 1 to 7, characterized in that: include: Image analysis module, 3D reconstruction module, volume calculation module, error correction module, data management module; The image analysis module is used to obtain a panoramic video stream of the goods, and perform preprocessing to extract the contours and feature points of the goods; The three-dimensional reconstruction module uses the output data of the image analysis module to reconstruct the three-dimensional model of the goods through stereo matching and depth estimation methods; The volume calculation module calculates the volume of the cargo based on the three-dimensional model of the cargo obtained by the three-dimensional reconstruction module; The error correction module uses a statistical method to correct the calculated volume data; The data management module is used to store, manage and retrieve the data generated during the entire measurement process.

9. The cargo volume calculation system based on panoramic video according to claim 8, characterized in that: The 3D reconstruction module includes: a point cloud generation unit, a point cloud optimization unit, a calibration unit, a point cloud registration unit, and a 3D model construction unit; The point cloud generation unit generates point cloud data of the surface of the cargo through stereo vision based on image feature matching; The calibration unit is used to calibrate the internal and external parameters of the camera device used to ensure the accuracy of the conversion to the real world coordinate system; The point cloud registration unit uses a point cloud registration method to align and merge point cloud data at different viewing angles or time points to form a point cloud model of the complete cargo; The point cloud optimization unit is used to perform filtering, denoising, and hole filling operations on the initial point cloud data to improve the quality of the point cloud; The three-dimensional model construction unit is used to convert the optimized point cloud data into a solid model and complete the three-dimensional reconstruction through a surface reconstruction algorithm.

10. The cargo volume calculation system based on panoramic video according to claim 9, characterized in that: The volume calculation module includes a volume calculation unit, which is equipped with grid division and volume integration algorithms, and the grid division and volume integration algorithms are used to calculate the actual volume of the goods; the error correction module includes an error model construction unit and an error correction unit, and the error model construction unit is used to construct an error model based on machine learning through multi-dimensional time series analysis, and the error correction unit is equipped with an intelligent error correction mechanism for dynamically adjusting the volume measurement value.

Images