A space size measurement method and system based on multi-modal data fusion

Abstract

The application relates to the technical field of space size measurement, in particular to a space size measurement method and system based on multi-modal data fusion, which comprises the following steps: acquiring a two-dimensional image and a three-dimensional point cloud in the same space coordinate system, extracting a two-dimensional mask of a measurement target and carrying out inflation processing; extracting a candidate point cloud; carrying out voxelization processing on the candidate point cloud, determining boundary voxels, and stripping the candidate point cloud outside the inner space into a to-be-determined point cloud; comparing the depth difference of adjacent boundary voxels of the same light path with a dynamic threshold value, and removing deep noise points; constructing a background distance field matrix for a non-inflated two-dimensional mask; reversely projecting the to-be-determined point cloud, checking the distance field matrix according to the reprojected coordinates to remove background noise points, and merging the remaining point cloud into a pure point cloud; fitting a directional bounding box for the pure point cloud, and extracting the edge length of the bounding box as the physical size of the measurement target. The application can solve the size expansion error caused by the penetration of background point clouds, and realizes accurate measurement.

Description

Technical Field

[0001] This application relates to the technical field of spatial dimension measurement, and in particular to a spatial dimension measurement method and system based on multimodal data fusion. Background Technology

[0002] Automatic spatial dimension measurement systems are widely used in industrial manufacturing and logistics measurement. The system typically acquires two-dimensional images and three-dimensional raw point clouds of the target simultaneously. It extracts a two-dimensional mask of the target using the connected component boundaries of the two-dimensional image, and then maps the mask to three-dimensional space using perspective to extract the corresponding target point cloud.

[0003] Chinese patent application CN109544456A discloses a panoramic environment perception method based on the fusion of two-dimensional images and three-dimensional point cloud data. This method performs feature matching and stitching on grayscale images to generate panoramic images. Point cloud data is then sliced ​​into depth segments according to a preset cutting length. The segmented point cloud sets within a specific depth range are projected onto a two-dimensional plane to generate a grayscale image. The projected image is binarized and subjected to morphological dilation and erosion operations. A watershed segmentation method is used to obtain a connected component image, which is then binarized to form an environmental information mask. The system performs a bitwise AND operation between this mask and the panoramic image, outputting the panoramic extraction result of the environmental information within that depth range.

[0004] In real-world applications, due to the dense optical imaging and sparse laser scanning, the physical resolution of 3D point clouds is far lower than that of 2D images, and the spatial density of point clouds decreases significantly with increasing detection distance. Limited by the inherent physical resolution mismatch between high-resolution 2D pixels and sparse 3D coordinates at spatial edges, when the system extracts 3D point clouds based on 2D mask boundaries, the 3D point clouds at the target's geometric edges are sparsely discrete. The perspective rays corresponding to the mask boundary pixels lack spatial obstruction after passing through the target's actual physical edge, directly hitting the background surface at a considerable distance behind the same optical path. This projection mechanism incorrectly includes the detached background point cloud into the target candidate point cloud set. Subsequently, when fitting a 3D oriented bounding box based on the spatial extrema of the point cloud, the unremoved distant background noise severely stretches the axial geometric dimensions of the bounding box, resulting in a positive expansion error in the calculated length, width, and height values. Summary of the Invention

[0005] To address the dimensional expansion error caused by background point cloud penetration and achieve accurate measurement, this application provides a spatial dimensional measurement method and system based on multimodal data fusion.

[0006] Firstly, this application provides a spatial dimension measurement method based on multimodal data fusion, employing the following technical solution: A spatial dimension measurement method based on multimodal data fusion, comprising: A two-dimensional image and a three-dimensional point cloud in the same spatial coordinate system are acquired. A two-dimensional mask of the target to be measured in the two-dimensional image is extracted. The two-dimensional mask is then subjected to dilation processing to generate a dilated mask. Along multiple perspective projection light paths, a three-dimensional view frustum space is generated based on the dilated mask back projection, and an inner view frustum space and an inner view frustum boundary are generated based on the undilated two-dimensional mask back projection. Point clouds that fall into the three-dimensional view frustum space are extracted as candidate point clouds. The candidate point cloud is voxelized to generate multiple voxels. The voxels that intersect with the inner view frustum boundary and contain the candidate point cloud are identified as boundary voxels. The candidate point cloud located entirely outside the inner view frustum space is peeled off from the candidate point cloud to a point cloud to be determined. The depth difference and dynamic depth jump threshold of two adjacent boundary voxels ordered by depth are obtained respectively on the same perspective projection light path. If the depth difference is greater than the dynamic depth jump threshold, the candidate point cloud in the boundary voxel with the larger depth is removed. Perform mask inversion on the undilated two-dimensional mask to extract the inverted background region, perform a distance transformation on the inverted background region, and construct a background distance field matrix; The undetermined point cloud is reverse-projected to obtain reprojected pixel coordinates to check the background distance field matrix. Based on the table value corresponding to the reprojected pixel coordinates in the background distance field matrix, background noise in the undetermined point cloud is removed. The undetermined point cloud with the table value of 0 corresponding to the reprojected pixel coordinates in the background distance field matrix is ​​merged with the remaining candidate point cloud to form a clean point cloud. A bounding box is fitted to the clean point cloud, and the side length of the bounding box is extracted as the physical size of the measurement target.

[0007] Optionally, acquiring the two-dimensional image and the three-dimensional point cloud in the same spatial coordinate system includes: Acquire the original 2D image and the original 3D point cloud with synchronized timestamps; The preset extrinsic parameter matrix is ​​used to transform the original 3D point cloud to the target camera coordinate system corresponding to the original 2D image; The depth direction of the original 3D point cloud after constraint transformation is parallel to the optical axis of the target camera coordinate system, and the output is the 3D point cloud and the 2D image in the same spatial coordinate system.

[0008] Optionally, the step of voxelizing the candidate point cloud to generate multiple voxels, determining the voxels that intersect with the inner view frustum boundary and contain the candidate point cloud as boundary voxels, and peeling the candidate point cloud within the voxels completely outside the inner view frustum space from the candidate point cloud into a point cloud to be determined, includes: Calculate the average depth of the candidate point cloud along the depth direction of the three-dimensional frustum space; Based on the average depth, a basic edge length is generated, and the candidate point cloud is divided into a three-dimensional voxel mesh according to the basic edge length to obtain the multiple voxels; The voxels in the three-dimensional voxel mesh that intersect with the inner view frustum boundary and contain the candidate point cloud are identified as boundary voxels, and the candidate point cloud located entirely outside the inner view frustum space is stripped from the candidate point cloud into an undetermined point cloud.

[0009] Optionally, obtaining the depth difference and dynamic depth jump threshold of two adjacent boundary voxels ordered by depth on the same perspective projection light path includes: Calculate the local depth mean of the candidate point cloud within each of the two adjacent boundary voxels, and use the absolute difference between the two local depth means as the depth difference; The distance attenuation compensation is calculated based on the local depth mean of the larger of the two adjacent boundary voxels and a preset depth divergence coefficient. The distance attenuation compensation is then superimposed on a preset system base error to generate the dynamic depth jump threshold.

[0010] Optionally, after determining the boundary voxel, the method further includes: Calculate the average point cloud density of the candidate point cloud within the boundary voxel. When the average point cloud density is less than a preset lower density limit, the basic side length is iteratively increased according to a preset expansion step size, and the spatial range of the boundary voxel and the average point cloud density are updated so that the average point cloud density reaches the preset lower density limit.

[0011] Optional, also includes: When the base side length after iterative amplification reaches the preset tolerance upper limit and the average point cloud density is less than the preset density lower limit, the iterative amplification stops, the dynamic depth jump threshold is configured to the preset maximum value, and a preset accuracy degradation status code indicating that the average point cloud density is lower than the preset density lower limit is output.

[0012] Optionally, performing mask inversion on the undilated two-dimensional mask to extract the inverted background region, performing a distance transformation on the inverted background region, and constructing a background distance field matrix includes: performing mask inversion on the undilated two-dimensional mask to extract the inverted background region and performing a Euclidean distance transformation, wherein the table value of the background distance field matrix represents the absolute pixel distance to the physical edge of the measured target; The step of inversely projecting the undetermined point cloud to obtain reprojected pixel coordinates for verification against the background distance field matrix includes: using a preset camera intrinsic parameter matrix to inversely project the undetermined point cloud to calculate the reprojected pixel coordinates, and aligning them to the background distance field matrix through perspective mapping to obtain the corresponding absolute pixel distance.

[0013] Optionally, before verifying the background range field matrix, the method further includes: The original imaging resolution of the two-dimensional image is analyzed, and the pixel mapping scale is determined by combining the depth value of the point cloud to be determined. By combining the pixel mapping scale with the preset system basic error, a dynamic pixel tolerance threshold is generated, so that its dimension is aligned with the absolute pixel distance. The step of removing background noise from the undetermined point cloud based on the table value corresponding to the reprojected pixel coordinates in the background distance field matrix includes: verifying the reprojected pixel coordinates; determining the undetermined point cloud whose corresponding reprojected pixel coordinates have a table value in the background distance field matrix greater than 0 and not greater than the dynamic pixel tolerance threshold as penetration noise, and resetting the table value of the corresponding reprojected pixel coordinates in the background distance field matrix to 0; determining the undetermined point cloud whose corresponding reprojected pixel coordinates have a table value in the background distance field matrix greater than the dynamic pixel tolerance threshold as free noise, and removing the free noise as background noise.

[0014] Optionally, fitting a bounding box to the clean point cloud and extracting the side length of the bounding box as the physical size of the measurement target includes: Extract the orthogonal principal component feature vector of the pure point cloud as the local spatial coordinate axis to fit the oriented bounding box, obtain the flat detection signal, and when the flat detection signal is received, select the local spatial coordinate axis with the smallest angle with the normal direction of the preset horizontal reference plane and align it to the normal direction; Based on the normal direction, the remaining local spatial coordinate axes are reconstructed, the oriented bounding box is degenerated into an oriented bounding box with Z-axis constraints, and the pure point cloud is re-traversed and projected to extract the axial side length of the oriented bounding box with Z-axis constraints as the physical size of the measurement target.

[0015] Secondly, this application provides a spatial dimension measurement system based on multimodal data fusion, which adopts the following technical solution: A spatial dimension measurement system based on multimodal data fusion, comprising: The frustum extraction module is used to acquire a two-dimensional image and a three-dimensional point cloud in the same spatial coordinate system, extract a two-dimensional mask of the target in the two-dimensional image, perform dilation processing on the two-dimensional mask to generate a dilated mask, generate a three-dimensional frustum space by back-projection along multiple perspective projection light paths based on the dilated mask, and generate an inner frustum space and an inner frustum boundary based on the undilated two-dimensional mask by back-projection, and extract the point cloud in the three-dimensional point cloud that falls into the three-dimensional frustum space as candidate point clouds; A voxel scanning module is used to voxelize the candidate point cloud to generate multiple voxels, identify voxels that intersect with the inner view frustum boundary and contain the candidate point cloud as boundary voxels, and peel off the candidate point cloud within the voxel that is completely outside the inner view frustum space from the candidate point cloud as undetermined point cloud; obtain the depth difference and dynamic depth jump threshold of two adjacent boundary voxels in depth order on the same perspective projection light path, and if the depth difference is greater than the dynamic depth jump threshold, remove the candidate point cloud within the boundary voxel with the larger depth. The reverse verification module is used to perform mask inversion on the undilated two-dimensional mask to extract the inverted background region, perform distance transformation on the inverted background region, and construct a background distance field matrix; reverse project the undetermined point cloud to obtain reprojected pixel coordinates to check the background distance field matrix, remove background noise in the undetermined point cloud according to the table value corresponding to the reprojected pixel coordinates in the background distance field matrix, and merge the undetermined point cloud with the remaining candidate point cloud whose corresponding reprojected pixel coordinates have a table value of 0 in the background distance field matrix with the remaining candidate point cloud to form a clean point cloud; The size calculation module is used to fit an oriented bounding box to the clean point cloud and extract the side length of the oriented bounding box as the physical size of the measurement target.

[0016] In summary, this application includes the following beneficial technical effects: 1. This application constructs a two-layer frustum truncation mechanism based on dilated and undilated masks to progressively confine the 3D point cloud to the vicinity of the inner frustum boundary. By combining the depth difference of adjacent boundary voxels on the same optical path with the comparison of dynamic depth jump threshold, the distant background point cloud that penetrates the target edge is physically removed during the forward propagation stage. At the same time, in conjunction with the table value verification of the inverse reprojection and background distance field matrix, a secondary backtracking verification is performed on the residual background noise. Through the closed-loop denoising mechanism of forward truncation and inverse verification, the background point cloud is effectively prevented from penetrating and participating in the directional bounding box fitting, thus eliminating the positive dilation error caused by background noise in 3D spatial dimension measurement from the root.

[0017] 2. This application introduces a distance attenuation compensation based on the local depth mean and depth divergence coefficient in the calculation of the dynamic depth jump threshold, so that the threshold is adaptively amplified as the detection distance increases. This compensates for the physical broadening effect of the lidar point cloud as it propagates in space, and avoids the inherent sparse point cloud at a long distance being misjudged as depth jump noise due to ranging error. This effectively improves the discrimination ability and robustness of the threshold decision mechanism at different detection distances.

[0018] 3. This application determines the pixel mapping scale by analyzing the resolution of the two-dimensional image and the depth value of the point cloud to be determined, and generates a dynamic pixel tolerance threshold accordingly. The penetration noise points whose table values ​​in the background distance field matrix are within the tolerance threshold are reset to zero instead of being directly removed. While removing the real free background noise points, legitimate body edge points with small projection offsets caused by sensor calibration residuals or optical distortions are retained, preventing the irreversible loss of effective three-dimensional edge features and improving the fidelity of the final fitted pure point cloud to the geometric boundary of the measured target. Attached Figure Description

[0019] Figure 1 This is a logic flowchart of the spatial dimension measurement method according to an embodiment of this application. Detailed Implementation

[0020] The following combination Figure 1 This application will be described in further detail.

[0021] like Figure 1 As shown in the embodiments of this application, the spatial dimension measurement method based on multimodal data fusion runs on a supporting measurement system. The hardware architecture of this measurement system includes a synchronous acquisition unit, a computing and processing unit, and an output unit. The synchronous acquisition unit integrates an industrial camera, a LiDAR, and a time synchronization module. The time synchronization module uses the IEEE 1588PTP protocol to synchronize the clocks of the industrial camera and the LiDAR. The computing and processing unit includes a frustum capture module, a voxel scanning module, a reverse verification module, and a dimension calculation module. These modules all rely on physical hardware entities such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA) built into the computing and processing unit. Data transfer between modules is completed through physical memory and a cache bus; the four modules complete the entire process in the order of data transfer. The output unit outputs the measured physical dimensions and corresponding status codes.

[0022] In this embodiment, the industrial camera used is a Hikvision MV-CA013-20GC model with a resolution of 1280×960 and a frame rate of 30fps. The LiDAR used is a DJI Livox Mid-40 model with a ranging range of 0.1m to 20m and an angular resolution of [missing information]. The computing and processing unit uses an NVIDIA Jetson Xavier NX embedded industrial computer, running the Ubuntu 20.04 operating system and the ROSNoetic platform. The algorithm is written in C++ and implemented using the PCL point cloud library and the OpenCV image processing library. This embodiment defines a unified data structure for the entire process. The 3D point cloud uses the PointXYZ data structure from the PCL library, with each point containing three single-precision floating-point coordinate values ​​(X, Y, and Z), corresponding to the 3D spatial position in the target camera coordinate system. The 2D mask uses the CV_8UC1 single-channel Mat matrix format from the OpenCV library, with the matrix size completely consistent with the width and height of the original 2D image.

[0023] Before implementation, sensor calibration is performed, and the calibration process consists of two stages. The first stage uses Zhang Zhengyou's calibration method to perform single-target calibration of the industrial camera. A 12×9 grid calibration board with 20mm side squares is used, and 20 images of the calibration board at different angles and distances are acquired. The camera's intrinsic parameters and distortion coefficients are calculated, and the reprojection error after calibration must be less than 0.5 pixels. The second stage performs joint calibration of the industrial camera and LiDAR. Using the same grid calibration board, 15 sets of calibration board data at different poses are acquired. The corner points and point cloud planar features of the calibration board images are extracted, and the extrinsic parameter matrix is ​​calculated based on the 3D-2D correspondence. The extrinsic parameter matrix includes a rotation matrix R and a translation vector T, used to achieve spatial transformation between the LiDAR coordinate system and the target camera coordinate system. After calibration, the point cloud projection error must be less than 1 pixel. The pre-calibrated camera intrinsic parameter matrix, sensor extrinsic parameter matrix, and distortion coefficients are stored in the computing unit's storage medium.

[0024] This embodiment uses a unified status code encoding rule: 0x00 indicates normal measurement completion, 0x01 indicates data acquisition timeout, 0x02 indicates visual extraction failure, 0x03 indicates no valid target point cloud, 0x04 indicates no valid boundary voxels, and 0x05 indicates accuracy degradation.

[0025] After the host computer issues the measurement start command, the synchronous acquisition unit synchronously triggers the industrial camera and LiDAR to complete frame synchronous acquisition. This embodiment uses hardware triggering for synchronization. The trigger input terminals of both the industrial camera and LiDAR are connected to the synchronization signal output terminal of the time synchronization module. The time synchronization module outputs a 10Hz square wave trigger signal and simultaneously provides a 1PPS clock synchronization signal and a PTP timestamp to the industrial camera and LiDAR, ensuring that the hardware triggering time synchronization accuracy is better than 100μs. If the synchronous acquisition unit fails to acquire the 2D raw image or 3D raw point cloud within the preset 500ms timeout period, the measurement is terminated, and a 0x01 status code is output through the output unit.

[0026] The synchronous acquisition unit outputs a two-dimensional raw image through an industrial camera and a three-dimensional raw point cloud through a LiDAR. During the acquisition process, the time synchronization module controls the hardware trigger timestamp difference between the two-dimensional raw image and the three-dimensional raw point cloud in the same frame to not exceed 1ms, effectively reducing the spatial misalignment deviation of the measured target under moving conditions.

[0027] The view frustum extraction module calls a pre-calibrated extrinsic parameter matrix to perform a spatial transformation of the 3D raw point cloud from the LiDAR coordinate system to the target camera coordinate system corresponding to the 2D raw image. It constrains the depth direction of the transformed 3D raw point cloud to be parallel to the optical axis of the target camera coordinate system, outputting the 3D point cloud and its corresponding 2D image in the same spatial coordinate system. The point cloud depth value is taken as the Z-axis coordinate of the corresponding point in the target camera coordinate system.

[0028] The frustum extraction module extracts a 2D mask of the target object within the 2D pixel plane of the 2D image. Morphological dilation is then performed on the 2D mask to generate a dilated mask. The resolution of the 2D mask corresponds one-to-one with the original resolution of the 2D image. 2D mask generation can employ a traditional image processing pipeline, sequentially performing grayscale conversion, adaptive binarization, morphological opening and closing operations on the 2D image. After removing image noise, the connected components of the target object are extracted to generate a binarized 2D mask. Within the mask, the pixel value corresponding to the target object is 255, and the pixel value corresponding to the background is 0. Alternatively, a pre-trained MaskR-CNN or YOLOv8-seg lightweight instance segmentation network can be used. The input is the original RGB 2D image after pre-calibrated distortion coefficient correction, and the output is a single-channel binary mask with the same resolution as the original image. Morphological opening operations on 3×3 rectangular structuring elements are then performed on the output to remove isolated noise points from the mask. Adaptive binarization uses Gaussian weighted neighborhood mean to calculate the threshold, adapting to complex lighting conditions in industrial environments. Morphological operations use 3×3 rectangular structural elements to remove isolated noise points and small burrs at the mask edges, ensuring smooth and continuous mask boundaries.

[0029] In some embodiments, a GrabCut-based interactive foreground extraction algorithm can also be used to generate a two-dimensional mask, which is suitable for high-precision mask extraction in static measurement scenarios.

[0030] When the extracted 2D mask area is less than a preset pixel threshold, the system determines that visual extraction has failed. The preset pixel threshold is set to 0.1% of the total number of pixels in the 2D image. In this embodiment, the threshold corresponding to a 1280×960 resolution is 1229 pixels. After determining that visual extraction has failed, the system suspends the entire process of subsequent view frustum truncation, voxel scanning, and reverse verification, and directly performs a local normal curvature region growing algorithm on the 3D point cloud that has been transformed to the target camera coordinate system. The system selects the top 1% of points in the 3D point cloud with depth values ​​within the effective range of the lidar and the smallest curvature values ​​as initial seed points, with curvature changes less than... As a region growth constraint, Euclidean clustering is completed and independent target point clouds are segmented. These point clouds are defined as pure point clouds and sent to the size calculation module. At the same time, the output unit outputs the 0x02 status code.

[0031] The frustum extraction module calls the pre-calibrated camera intrinsic parameter matrix and back-projects along the perspective projection light path, generating a 3D frustum space, an inner frustum space, and an inner frustum boundary based on the dilated mask and the undilated 2D mask, respectively. Each pixel of the dilated mask, after back-projection, corresponds to a perspective projection light path originating from the camera's optical center and passing through that pixel on the 2D pixel plane. The closed space enclosed by all perspective projection light paths between the near clipping plane and the far clipping plane constitutes the 3D frustum space. The perspective projection light paths diverging from the camera's optical center form the 3D frustum space, which contains nested inner frustum boundaries and corresponding inner frustum spaces. The measured target body lies within these boundaries, and the extracted 3D point cloud is distributed on the surface of the measured target and the background surface behind it. The near clipping plane is set to the minimum effective detection range of the lidar (0.3m) to avoid interference from invalid point clouds in the near-range measurement blind zone of the equipment. The far clipping plane is set to the maximum working distance of 10m in industrial measurement scenarios to cut off invalid space beyond the effective measurement range, reducing the computational overhead of traversal calculations.

[0032] The view frustum extraction module traverses all 3D point clouds that have completed coordinate system alignment. For each point, it calculates its projected coordinates on the 2D pixel plane, verifies whether these coordinates fall within the target area of ​​the dilation mask, and simultaneously verifies whether the point's depth value is between the near and far clipping planes. Points that pass both verifications are considered as point clouds falling within the 3D view frustum space and extracted as candidate point clouds. When the number of candidate point clouds is 0, the measurement is terminated, and a 0x03 status code is output through the output unit.

[0033] The voxel scanning module calculates the average depth of the candidate point cloud along the depth direction of the 3D view frustum space, hereinafter referred to as the average depth. The result is obtained by dividing the sum of the depth values ​​of all points in the candidate point cloud by the total number of points. Based on the average depth, a basic side length is dynamically generated. The basic side length is obtained by multiplying the average depth by a resolution coefficient, which is determined according to the angular resolution of the LiDAR; in this embodiment, it is taken as 1. Tangent value. The basic side length is set with a lower limit of 0.001m and an upper limit of 0.01m. If the calculated value exceeds the range, the corresponding boundary value is directly used. Based on the finally determined basic side length, the spatial enclosure of the candidate point cloud is divided into a uniform three-dimensional voxel mesh.

[0034] In some embodiments, octree voxelization can be used instead of uniform voxel meshing. Fine voxel meshing is performed for the boundary region of the view frustum, while coarse voxel meshing is used for the internal region of the view frustum, thereby reducing computational overhead while ensuring boundary detection accuracy.

[0035] The voxel scanning module identifies 3D voxels in the 3D voxel mesh that intersect with the inner view frustum boundary and contain candidate point clouds. These are marked as boundary voxels. Point clouds within 3D voxels completely outside the inner view frustum space are separated from the candidate point clouds and designated as undetermined point clouds. The boundary voxel determination logic is as follows: of the voxel's eight vertices, some vertices are located within the inner view frustum space, and some vertices are located outside the view frustum space; and the voxel contains at least one candidate point cloud. This determination mechanism ensures that the voxels involved in the mean calculation are not completely empty, eliminating the risk of low-level errors when the program performs division operations. When the number of boundary voxels is 0, the candidate point cloud is directly confirmed as a pure point cloud, and a 0x04 status code is output through the output unit, skipping subsequent reverse verification steps and directly sent to the dimension calculation module. When the number of boundary voxels is less than 2, depth difference calculation cannot be completed, and the candidate point cloud is also directly confirmed as a pure point cloud and sent to the dimension calculation module.

[0036] The voxel scanning module calculates the average point cloud density of candidate point clouds within boundary voxels. The calculation result is obtained by dividing the total number of candidate point clouds within all boundary voxels by the total number of boundary voxels. In this embodiment, the preset lower limit of density is set to 3 to ensure that the number of point clouds within a single voxel can support stable local depth mean calculation. When the average point cloud density is less than the preset lower limit of density, the base side length is iteratively enlarged according to a preset expansion step size. The preset expansion step size is set to 0.2 times the base side length. After each iteration, the 3D voxel mesh is re-divided, the boundary voxels are determined, and the average point cloud density is calculated until the updated average point cloud density reaches the preset lower limit of density or the maximum number of iterations reaches 10.

[0037] When the base side length after iterative amplification reaches the preset tolerance upper limit, or the number of iterations reaches the maximum limit, and the average point cloud density is still less than the preset density lower limit, the iterative amplification of the base side length stops. The preset tolerance upper limit is determined based on the maximum allowable dimensional error in industrial measurement; in this embodiment, it is set to 0.01m. After stopping iteration, the dynamic depth jump threshold is configured as follows: The preset maximum limit is set, and a 0x05 status code is output through the output unit. This preset accuracy degradation status code indicates that the accuracy of the measurement result is degraded due to the sparse point cloud. The maximum permissible error of the measurement result is set to [value missing]. Downgraded to It synchronously outputs the corresponding precision prompt.

[0038] When the aforementioned accuracy degradation branch is not triggered, the voxel scanning module acquires the depth difference and dynamic depth jump threshold between two adjacent boundary voxels sorted by depth along the same perspective projection light path. For the depth difference, the local depth mean of the candidate point cloud within two adjacent boundary voxels along the same perspective projection light path is calculated, and the absolute difference between the two local depth mean values ​​is used as the depth difference.

[0039] The computational processing unit calculates the distance attenuation compensation based on the local depth mean of the larger depth value among two adjacent boundary voxels along the same perspective projection light path, and a preset depth divergence coefficient. Specifically, the system uses a linear product compensation model to calculate this distance attenuation compensation, and its mathematical derivation formula is as follows: The deductive logic clarifies that the measurement error caused by beam diffusion with distance is mathematically linearly amplified with the absolute detection distance. As an alternative implementation in this field, for complex working conditions with severe light scattering such as dust, rain, and fog, a quadratic exponential attenuation compensation model based on the air attenuation coefficient can also be used. In this case, the calculation formula is replaced with: Alternatively, a pre-calibrated segmented discrete mapping table lookup method can be used to obtain the compensation amount for the corresponding depth interval. Using the above calculation mechanism, the physical error effect of point cloud broadening caused by laser beam divergence with distance can be accurately quantified and offset, avoiding the misjudgment of inherent physical errors of distant sparse point clouds as jump noise, and maintaining a highly reliable threshold decision benchmark in long-distance scenarios. Furthermore, the generated distance attenuation compensation amount is superimposed on the preset system base error to calculate and generate a dynamic depth jump threshold. The preset system base error is taken as the inherent ranging error of the lidar, which is set to 0.01m in this embodiment. The preset depth divergence coefficient is determined based on the divergence angle of the lidar, and is taken as twice the angle in this embodiment. Tangent value.

[0040] The voxel scanning module compares the depth difference with the dynamic depth jump threshold. The internal view frustum boundary passes through the boundary voxels distributed along the perspective projection light path. If the depth difference between two adjacent boundary voxels on this light path is greater than the dynamic depth jump threshold, the candidate point cloud within the boundary voxel with the larger depth is discarded. This logical inversion correction eliminates the possibility of misjudging distant real backgrounds as undetermined point clouds and sending them into the reverse projection, cutting off the paradoxical deadlock path of noise points behind the same light path being erroneously recycled and protected by the system due to their constant table value of 0 under 2D reprojection.

[0041] If the accuracy degradation branch is not triggered, the reverse verification module performs subsequent processing in this step. The reverse verification module performs mask inversion on the undilated 2D mask to extract the inverted background region, and then performs a distance transformation on the inverted background region to construct a background distance field matrix. Specifically, it performs a Boolean inversion on the undilated 2D mask, setting the pixel values ​​corresponding to the measured target to 0 and the pixel values ​​corresponding to the background to 255. After extracting the binary image of the inverted background region, it performs a Euclidean distance transformation on it to generate the background distance field matrix. The values ​​in the background distance field matrix represent the absolute pixel distance from the corresponding pixel to the physical edge of the measured target. The distance transformation is implemented using the Saito algorithm, balancing computational accuracy and processing efficiency.

[0042] In some embodiments, the Manhattan distance transformation can be used instead of the Euclidean distance transformation, which is suitable for embedded scenarios with high computational speed requirements.

[0043] The reverse verification module analyzes the original imaging resolution of the 2D image. Based on the original imaging resolution and the depth value of the point cloud to be determined, it calculates the pixel mapping scale between the 2D pixel plane and the 3D physical space point by point. Combining the pixel mapping scale with the preset system baseline error, it performs discretized pixel conversion and dynamically generates a dynamic pixel tolerance threshold point by point. The pixel mapping scale is obtained by multiplying the depth value of the point cloud to be determined by the tangent of half the horizontal field of view, and then dividing by half the image width, representing the lateral dimension of a single pixel in the 3D physical space at the corresponding depth. The horizontal field of view is calculated from the pre-calibrated camera intrinsic parameters. ,in The x-axis normalized focal length is the camera's intrinsic parameter. The preset system baseline error is set to 0.01m in this embodiment. The dynamic pixel tolerance threshold is obtained by dividing the preset system baseline error by the corresponding pixel mapping scale, perfectly aligned with the dimensional reference of the absolute pixel distance represented by the background distance field matrix.

[0044] The reverse verification module calls a preset camera intrinsic parameter matrix to perform reverse perspective projection on the point cloud to be determined. It calculates floating-point reprojected pixel coordinates on a 2D pixel plane, rounds these coordinates to obtain integer pixel indices, and aligns them to the background distance field matrix using the perspective mapping relationship of the preset camera intrinsic parameter matrix. The u-axis value of the reprojected pixel coordinates is determined by... Multiply by the X-axis coordinate of the undetermined point cloud, divide by the depth value, and add... The v-axis value is obtained from Multiply by the Y-axis coordinate of the undetermined point cloud, divide by the depth value, and add... Received, among which These are the coordinates of the principal point, which are intrinsic parameters of the camera. When the reprojected pixel coordinates exceed the pixel range of the 2D image, the undetermined point cloud is directly identified as background noise and removed from the undetermined point cloud.

[0045] When the point cloud to be determined is empty, the candidate point cloud is directly confirmed as a pure point cloud and sent to the size calculation module. When the point cloud to be determined is not empty, the reverse verification module uses the aligned reprojected pixel coordinates to check the background distance field matrix, and removes background noise from the point cloud to be determined based on the corresponding table value of the reprojected pixel coordinates in the background distance field matrix. At this time, the point cloud to be determined participating in the verification is actually the outer point cloud excluded by the inner frustum. Specifically, the table value of the penetration noise whose corresponding reprojected pixel coordinates in the background distance field matrix are greater than 0 and not greater than the corresponding dynamic pixel tolerance threshold is reset to 0, and the free noise whose corresponding reprojected pixel coordinates in the background distance field matrix are greater than the corresponding dynamic pixel tolerance threshold is removed as background noise. The point cloud to be determined obtained with the corresponding reprojected pixel coordinates in the background distance field matrix being greater than 0 and not greater than the corresponding dynamic pixel tolerance threshold is essentially a legitimate body edge point that has a slight projection offset on the real physical edge of the target due to slight optical distortion of the camera or sensor joint calibration error. To prevent valid 3D edge features of the ontology from being mistakenly identified, the system validates them by resetting their values ​​to 0. After resetting, the undetermined point clouds whose corresponding reprojected pixel coordinates have a value of 0 in the background distance field matrix (including valid ontology points that fall precisely within the mask during the original reprojection and the aforementioned validated small offset points) are all retained as undetermined point clouds and merged with the remaining candidate point clouds that were not marked as undetermined point clouds to form a clean point cloud.

[0046] As an alternative implementation, for penetration noise points whose table values ​​are no greater than the tolerance threshold, a lower spatial weight reset confidence parameter (e.g., a value of 0.5) can be assigned. This parameter is then used for weighted downgrading when calculating the covariance matrix in subsequent fitting of the oriented bounding box, thereby achieving a balance between preserving edge ontological features and suppressing minor drift noise. This tolerance decision mechanism effectively prevents the irreversible loss of real physical edges, generating a high-fidelity, clean point cloud.

[0047] The dimension calculation module calculates the centroid of the clean point cloud. The centroid coordinates are taken as the average of the coordinates of all points within the clean point cloud. Based on the centroid, the covariance matrix of the clean point cloud is calculated. Eigenvalue decomposition is performed on the covariance matrix to obtain three eigenvalues ​​and three corresponding orthogonal eigenvectors. The eigenvectors are sorted in descending order of eigenvalues, and the three sorted orthogonal principal component eigenvectors are used as local space coordinate axes. Since Principal Component Analysis (PCA) has locked the 3D spatial pose at the analytical geometry level, the dimension calculation module directly adopts a purely analytical method, traversing and projecting the clean point cloud onto these three orthogonal local space coordinate axes. The maximum and minimum values ​​of the projected coordinates of the clean point cloud along each axis are calculated, and the absolute difference between the maximum and minimum values ​​is used to calculate the side length of each axis, thereby fitting the orientation bounding box. The side length is extracted as the physical dimension output of the measurement target, and a 0x00 status code is output simultaneously through the output unit. This purely analytical extremum retrieval algorithm utilizes the locked principal axis orientation, avoiding rotational degree of freedom conflicts caused by forcibly introducing optimization iteration algorithms, and completely eliminating deadlock errors in the underlying solution engine.

[0048] As an alternative implementation in this field, for targets with extremely irregular boundaries and where stable principal components are difficult to extract using PCA, PCA pose analysis can be abandoned. Instead, the clean point cloud can be directly projected onto a 2D plane along a specific normal to extract the 2D convex hull. This is then combined with a 2D rotating calipers algorithm to iteratively rotate and optimize along the convex hull boundary to find the bounding rectangle with the smallest area. Finally, the extreme heights of the normal projections are superimposed to reconstruct the complete 3D oriented bounding box. This dimensionality reduction optimization scheme avoids algorithmic deadlock at the underlying level and expands the system's computational robustness in measuring the dimensions of complex irregular parts.

[0049] In some embodiments, a least-squares-based directional bounding box fitting algorithm can be used to replace the caliper algorithm, which is suitable for scenarios where the point cloud has a small amount of noise.

[0050] If the number of points in a clean point cloud is less than 4, the directional bounding box fitting cannot be completed, the measurement is terminated, and a status code of 0x03 is output through the output unit.

[0051] The system acquires a flatness detection signal. Upon receiving this signal, which indicates that the target is flat on a preset horizontal reference plane, the dimension calculation module selects the local spatial coordinate axis with the smallest angle to the normal direction of the preset horizontal reference plane and aligns it to the normal direction. Based on this normal direction, the remaining local spatial coordinate axes are reconstructed. The reconstruction logic is as follows: the aligned normal direction is used as the first axis; the projection of one of the remaining principal component eigenvectors not aligned to the normal direction onto a plane perpendicular to the first axis is used as the second axis; and the cross product of the first and second axes is used as the third axis, ensuring strict orthogonality among the three axes. Since this operation only unidirectionally constrains the Z-axis to be parallel to the world coordinate system normal, its X and Y axes retain their yaw rotation properties, degenerating the bounding box into a bounding box constrained by the Z-axis. After the local coordinate system undergoes yaw reconstruction, the system performs a re-traversal projection of the clean point cloud onto the reconstructed local spatial coordinate axes to search for and obtain the spatial projection extreme boundary under the new attitude, and re-extracts the axial extreme side length of the bounding box constrained by the Z-axis as the physical dimension. This essential traversal and reprojection process strictly adheres to 3D geometric logic, correcting spatial data misalignment caused by the failure of original maxima and minima due to coordinate axis yaw and torsion. This ensures that the output dimensional parameters objectively reflect the physical position of the object on the real horizontal plane. The acquired flatness detection signal can come from a horizontal sensor or user-input configuration parameters, with the preset normal direction of the horizontal reference plane being the Z-axis direction of the world coordinate system.

[0052] This application also discloses a spatial dimension measurement system based on multimodal data fusion, including a synchronous acquisition unit, a computing and processing unit, and an output unit. The synchronous acquisition unit is communicatively coupled to the computing and processing unit, and the computing and processing unit is communicatively coupled to the output unit.

[0053] The synchronous acquisition unit includes an industrial camera, a LiDAR, and a time synchronization module. The trigger terminals of both the industrial camera and the LiDAR are electrically coupled to the synchronization signal output terminal of the time synchronization module, and the data stream output terminals of both the industrial camera and the LiDAR are communicatively coupled to the signal input terminal of the computing processing unit. The time synchronization module is configured to provide clock synchronization signals and hardware trigger signals to the industrial camera and LiDAR, controlling them to complete frame-synchronized data acquisition. The industrial camera is configured to acquire a 2D raw image containing the measurement target and output the 2D raw image to the computing processing unit. The LiDAR is configured to acquire a 3D raw point cloud containing the measurement target and output the 3D raw point cloud to the computing processing unit.

[0054] The computational processing unit includes a frustum extraction module, a voxel scanning module, a reverse verification module, and a dimension calculation module. The signal input terminal of the frustum extraction module is communicatively coupled to the data stream output terminal of the synchronous acquisition unit, the first signal output terminal of the frustum extraction module is communicatively coupled to the signal input terminal of the voxel scanning module, and the second signal output terminal of the frustum extraction module is communicatively coupled to the first signal input terminal of the reverse verification module. The frustum extraction module is configured to receive the 2D raw image and 3D raw point cloud output by the synchronous acquisition unit, call the preset extrinsic matrix to complete the spatial transformation of the 3D raw point cloud from the lidar coordinate system to the target camera coordinate system corresponding to the 2D raw image, constrain the depth direction of the transformed 3D raw point cloud to be parallel to the optical axis of the target camera coordinate system, and output the 3D point cloud and the corresponding 2D image in the same spatial coordinate system; it is configured to extract the 2D mask of the target in the 2D pixel plane where the 2D image is located, perform morphological dilation processing on the 2D mask to generate a dilated mask; along the perspective projection optical path, based on the dilated mask and the undilated 2D mask, respectively, backproject to generate the 3D frustum space, the inner frustum space and the inner frustum boundary, and extract the point cloud in the 3D point cloud that falls into the 3D frustum space as candidate point clouds, and output the candidate point clouds to the voxel scanning module; it is configured to output the 2D mask to the reverse verification module.

[0055] The first signal output terminal of the voxel scanning module is communicatively coupled to the second signal input terminal of the reverse verification module, and the second signal output terminal of the voxel scanning module is communicatively coupled to the signal input terminal of the size calculation module. The voxel scanning module is configured to receive the candidate point cloud output by the frustum extraction module, calculate the average depth of the candidate point cloud along the depth direction of the 3D frustum space, dynamically generate the base edge length based on the average depth, divide the candidate point cloud into a 3D voxel mesh according to the base edge length, determine the 3D voxels that intersect with the inner frustum boundary and contain the candidate point cloud as boundary voxels, and peel the point cloud within the 3D voxels completely outside the inner frustum space from the candidate point cloud into undetermined point clouds, outputting the undetermined point clouds to the reverse verification module; it is also configured to calculate the average point cloud density of the candidate point cloud within the boundary voxels, and when the average point cloud density is less than a preset lower density limit, iteratively enlarge the base edge according to a preset expansion step size. The function is configured to update the spatial range and average point cloud density of boundary voxels; it is configured to obtain the depth difference and dynamic depth jump threshold of two adjacent boundary voxels sorted by depth on the same perspective projection light path, compare the depth difference with the dynamic depth jump threshold, and if the depth difference is greater than the dynamic depth jump threshold, remove the candidate point cloud in the boundary voxel with the larger depth; it is configured to stop iteratively amplifying the basic side length when the basic side length after iterative amplification reaches the preset tolerance upper limit and the average point cloud density is less than the preset density lower limit, configure the dynamic depth jump threshold to the preset maximum value, and output the preset precision degradation status code representing that the average point cloud density is lower than the preset density lower limit to the output unit.

[0056] The signal output of the reverse verification module is communicatively coupled to the signal input of the size calculation module. The reverse verification module is configured to receive the two-dimensional mask output by the frustum capture module, perform mask inversion on the undilated two-dimensional mask to extract the inverted background region, and perform a distance transformation on the inverted background region to construct a background distance field matrix. The values ​​of the background distance field matrix represent the absolute pixel distance from the corresponding pixel to the physical edge of the measured target. It is also configured to receive the undetermined point cloud output by the voxel scanning module, call a preset camera intrinsic parameter matrix to perform reverse perspective projection on the undetermined point cloud, calculate the reprojected pixel coordinates on the two-dimensional pixel plane, and project the reprojected pixels using the perspective mapping relationship of the preset camera intrinsic parameter matrix. The coordinates are aligned to the background distance field matrix; configured to analyze the original imaging resolution of the 2D image, determine the pixel mapping scale between the 2D pixel plane and the 3D physical space based on the original imaging resolution, and perform discretized pixel conversion on the preset system fundamental error in combination with the pixel mapping scale, dynamically generating a dynamic pixel tolerance threshold, so that the dynamic pixel tolerance threshold is aligned with the dimensional benchmark of the absolute pixel distance represented by the background distance field matrix; configured to use the reprojected pixel coordinates to check the background distance field matrix, and remove background noise in the undetermined point cloud according to the table value corresponding to the reprojected pixel coordinates in the background distance field matrix. Specifically, the table value of the penetrating noise whose corresponding reprojected pixel coordinates in the background distance field matrix is ​​greater than 0 and not greater than the dynamic pixel tolerance threshold is reset to 0, the free noise whose corresponding reprojected pixel coordinates in the background distance field matrix is ​​greater than the dynamic pixel tolerance threshold is removed as background noise, and the undetermined point cloud whose corresponding reprojected pixel coordinates in the background distance field matrix are 0 is merged with the remaining candidate point cloud to generate a clean point cloud and output to the size calculation module.

[0057] The signal output terminal of the dimension calculation module is communicatively coupled to the signal input terminal of the output unit. The dimension calculation module is configured to receive a clean point cloud, extract the orthogonal principal component feature vectors of the clean point cloud as local spatial coordinate axes to fit an oriented bounding box, and extract the axial side length of the oriented bounding box as the physical dimension of the measured target and output it to the output unit; it is also configured to acquire a flatness detection signal, and when receiving a flatness detection signal representing that the measured target is flat on a preset horizontal reference plane, select the local spatial coordinate axis with the smallest angle with the normal direction of the preset horizontal reference plane and align it to the normal direction, reconstruct the remaining local spatial coordinate axes based on the normal direction, degenerate the oriented bounding box into a Z-axis constrained oriented bounding box, and re-traverse and project the clean point cloud to extract the axial side length of the Z-axis constrained oriented bounding box as the physical dimension and output it to the output unit.

[0058] In this embodiment, the frustum capture module, voxel scanning module, reverse verification module, and size calculation module are all computer program modules stored in the memory of the computing processing unit. The computing processing unit also includes a processor and a communication bus. The processor, memory, and output unit complete signal and data interaction with each other through the communication bus. The processor executes the computer program stored in the memory to realize the corresponding functions of each module.

[0059] The micro-details of the specific data processing and control logic executed by each module and hardware in this embodiment have been described in detail in the foregoing method embodiments, and will not be repeated here.

[0060] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A spatial dimension measurement method based on multimodal data fusion, characterized in that, include: A two-dimensional image and a three-dimensional point cloud in the same spatial coordinate system are acquired. A two-dimensional mask of the target to be measured in the two-dimensional image is extracted. The two-dimensional mask is then subjected to dilation processing to generate a dilated mask. Along multiple perspective projection light paths, a three-dimensional view frustum space is generated based on the dilated mask back projection, and an inner view frustum space and an inner view frustum boundary are generated based on the undilated two-dimensional mask back projection. Point clouds that fall into the three-dimensional view frustum space are extracted as candidate point clouds. The candidate point cloud is voxelized to generate multiple voxels. The voxels that intersect with the inner view frustum boundary and contain the candidate point cloud are identified as boundary voxels. The candidate point cloud located entirely outside the inner view frustum space is peeled off from the candidate point cloud to a point cloud to be determined. The depth difference and dynamic depth jump threshold of two adjacent boundary voxels ordered by depth are obtained respectively on the same perspective projection light path. If the depth difference is greater than the dynamic depth jump threshold, the candidate point cloud in the boundary voxel with the larger depth is removed. Perform mask inversion on the undilated two-dimensional mask to extract the inverted background region, perform a distance transformation on the inverted background region, and construct a background distance field matrix; The undetermined point cloud is reverse-projected to obtain reprojected pixel coordinates to check the background distance field matrix. Based on the table value corresponding to the reprojected pixel coordinates in the background distance field matrix, background noise in the undetermined point cloud is removed. The undetermined point cloud with the table value of 0 corresponding to the reprojected pixel coordinates in the background distance field matrix is ​​merged with the remaining candidate point cloud to form a clean point cloud. A bounding box is fitted to the clean point cloud, and the side length of the bounding box is extracted as the physical size of the measurement target.

2. The spatial dimension measurement method based on multimodal data fusion according to claim 1, characterized in that, The acquisition of a two-dimensional image and a three-dimensional point cloud in the same spatial coordinate system includes: Acquire the original 2D image and the original 3D point cloud with synchronized timestamps; The preset extrinsic parameter matrix is ​​used to transform the original 3D point cloud to the target camera coordinate system corresponding to the original 2D image; The depth direction of the original 3D point cloud after constraint transformation is parallel to the optical axis of the target camera coordinate system, and the output is the 3D point cloud and the 2D image in the same spatial coordinate system.

3. The spatial dimension measurement method based on multimodal data fusion according to claim 1, characterized in that, The process of voxelizing the candidate point cloud to generate multiple voxels, identifying the voxels that intersect with the inner view frustum boundary and contain the candidate point cloud as boundary voxels, and stripping the candidate point cloud within the voxels completely outside the inner view frustum space from the candidate point cloud to a point cloud to be determined includes: Calculate the average depth of the candidate point cloud along the depth direction of the three-dimensional frustum space; Based on the average depth, a basic edge length is generated, and the candidate point cloud is divided into a three-dimensional voxel mesh according to the basic edge length to obtain the multiple voxels; The voxels in the three-dimensional voxel mesh that intersect with the inner view frustum boundary and contain the candidate point cloud are identified as boundary voxels, and the candidate point cloud located entirely outside the inner view frustum space is stripped from the candidate point cloud into an undetermined point cloud.

4. The spatial dimension measurement method based on multimodal data fusion according to claim 3, characterized in that, The step of obtaining the depth difference and dynamic depth jump threshold of two adjacent boundary voxels ordered by depth on the same perspective projection light path includes: Calculate the local depth mean of the candidate point cloud within each of the two adjacent boundary voxels, and use the absolute difference between the two local depth means as the depth difference; The distance attenuation compensation is calculated based on the local depth mean of the larger of the two adjacent boundary voxels and a preset depth divergence coefficient. The distance attenuation compensation is then superimposed on a preset system base error to generate the dynamic depth jump threshold.

5. The spatial dimension measurement method based on multimodal data fusion according to claim 3, characterized in that, After determining the boundary voxel, the process further includes: Calculate the average point cloud density of the candidate point cloud within the boundary voxel. When the average point cloud density is less than a preset lower density limit, the basic side length is iteratively increased according to a preset expansion step size, and the spatial range of the boundary voxel and the average point cloud density are updated so that the average point cloud density reaches the preset lower density limit.

6. The spatial dimension measurement method based on multimodal data fusion according to claim 5, characterized in that, Also includes: When the base side length after iterative amplification reaches the preset tolerance upper limit and the average point cloud density is less than the preset density lower limit, the iterative amplification stops, the dynamic depth jump threshold is configured to the preset maximum value, and a preset accuracy degradation status code indicating that the average point cloud density is lower than the preset density lower limit is output.

7. The spatial dimension measurement method based on multimodal data fusion according to claim 1, characterized in that, The step of performing mask inversion on the undilated two-dimensional mask to extract the inverted background region, performing distance transformation on the inverted background region, and constructing a background distance field matrix includes: performing mask inversion on the undilated two-dimensional mask to extract the inverted background region and performing Euclidean distance transformation, wherein the table value of the background distance field matrix represents the absolute pixel distance to the physical edge of the measured target; The step of inversely projecting the undetermined point cloud to obtain reprojected pixel coordinates for verification against the background distance field matrix includes: using a preset camera intrinsic parameter matrix to inversely project the undetermined point cloud to calculate the reprojected pixel coordinates, and aligning them to the background distance field matrix through perspective mapping to obtain the corresponding absolute pixel distance.

8. A spatial dimension measurement method based on multimodal data fusion according to claim 7, characterized in that, Before verifying the background range field matrix, the following steps are also included: The original imaging resolution of the two-dimensional image is analyzed, and the pixel mapping scale is determined by combining the depth value of the point cloud to be determined. By combining the pixel mapping scale with the preset system basic error, a dynamic pixel tolerance threshold is generated, so that its dimension is aligned with the absolute pixel distance. The step of removing background noise from the undetermined point cloud based on the table value corresponding to the reprojected pixel coordinates in the background distance field matrix includes: verifying the reprojected pixel coordinates; determining the undetermined point cloud whose corresponding reprojected pixel coordinates have a table value in the background distance field matrix greater than 0 and not greater than the dynamic pixel tolerance threshold as penetration noise, and resetting the table value of the corresponding reprojected pixel coordinates in the background distance field matrix to 0; determining the undetermined point cloud whose corresponding reprojected pixel coordinates have a table value in the background distance field matrix greater than the dynamic pixel tolerance threshold as free noise, and removing the free noise as background noise.

9. A spatial dimension measurement method based on multimodal data fusion according to claim 1, characterized in that, The process of fitting a bounding box to the clean point cloud and extracting the side length of the bounding box as the physical size of the measurement target includes: Extract the orthogonal principal component feature vector of the pure point cloud as the local spatial coordinate axis to fit the oriented bounding box, obtain the flat detection signal, and when the flat detection signal is received, select the local spatial coordinate axis with the smallest angle with the normal direction of the preset horizontal reference plane and align it to the normal direction; Based on the normal direction, the remaining local spatial coordinate axes are reconstructed, the oriented bounding box is degenerated into an oriented bounding box with Z-axis constraints, and the pure point cloud is re-traversed and projected to extract the axial side length of the oriented bounding box with Z-axis constraints as the physical size of the measurement target.

10. A spatial dimension measurement system based on multimodal data fusion, used to perform the method as described in any one of claims 1-9, characterized in that, include: The frustum extraction module is used to acquire a two-dimensional image and a three-dimensional point cloud in the same spatial coordinate system, extract a two-dimensional mask of the target in the two-dimensional image, perform dilation processing on the two-dimensional mask to generate a dilated mask, generate a three-dimensional frustum space by back-projection along multiple perspective projection light paths based on the dilated mask, and generate an inner frustum space and an inner frustum boundary based on the undilated two-dimensional mask by back-projection, and extract the point cloud in the three-dimensional point cloud that falls into the three-dimensional frustum space as candidate point clouds; A voxel scanning module is used to voxelize the candidate point cloud to generate multiple voxels, identify voxels that intersect with the inner view frustum boundary and contain the candidate point cloud as boundary voxels, and peel off the candidate point cloud within the voxel that is completely outside the inner view frustum space from the candidate point cloud as undetermined point cloud; obtain the depth difference and dynamic depth jump threshold of two adjacent boundary voxels in depth order on the same perspective projection light path, and if the depth difference is greater than the dynamic depth jump threshold, remove the candidate point cloud within the boundary voxel with the larger depth. The reverse verification module is used to perform mask inversion on the undilated two-dimensional mask to extract the inverted background region, perform distance transformation on the inverted background region, and construct a background distance field matrix. The undetermined point cloud is reverse-projected to obtain reprojected pixel coordinates to check the background distance field matrix. Based on the table value corresponding to the reprojected pixel coordinates in the background distance field matrix, background noise in the undetermined point cloud is removed. The undetermined point cloud with the table value of 0 corresponding to the reprojected pixel coordinates in the background distance field matrix is ​​merged with the remaining candidate point cloud to form a clean point cloud. The size calculation module is used to fit an oriented bounding box to the clean point cloud and extract the side length of the oriented bounding box as the physical size of the measurement target.

Images