A multi-modal scene automatic semantic labeling method and system

Abstract

The application relates to the technical field of automatic semantic labeling, in particular to a multi-modal scene automatic semantic labeling method and system, which comprises the following steps: obtaining two-dimensional image frames and three-dimensional point cloud frames of a scene to be labeled with hardware timestamp alignment, extracting target pixel points carrying two-dimensional semantic labels in the two-dimensional image frames, and converting the three-dimensional point cloud frames into a three-dimensional voxel space; traversing candidate voxels in turn from near to far according to radial depth; if a candidate voxel is found within a preset maximum search level, binding the candidate voxel with the two-dimensional semantic label with the minimum perpendicular distance to the center ray of the back projection tracking domain, and terminating the binding operation of the target pixel points; if a candidate voxel is not found within the preset maximum search level, terminating the binding operation of the target pixel points. The application can solve the spatial dislocation caused by the error interception of the semantic label of a real target in the rear by a front high-transmittance material or sparse noise, and realize accurate three-dimensional labeling.

Description

Technical Field

[0001] This application relates to the technical field of automatic semantic annotation, and in particular to a method and system for automatic semantic annotation of multimodal scenes. Background Technology

[0002] In autonomous driving and robotics environmental perception engineering, LiDAR is typically used to acquire 3D point cloud data, which is then combined with 2D images captured by cameras for multimodal scene semantic annotation. Point cloud data provides the absolute coordinates and physical contours of targets in 3D space, while 2D images provide pixel-level semantic details of the target region. In engineering practice, it is necessary to map the semantic labels within the 2D images across modalities to the voxel space constructed from the 3D point cloud. This mapping process requires the system to accurately establish the data association between the pixel coordinate system and the 3D spatial coordinate system to ensure the positioning accuracy of multimodal features in real physical space.

[0003] Chinese patent application CN119323776A discloses a method for pre-annotating the subdivision attributes of 3D obstacles based on point clouds and images. This technique uses a point cloud 3D target detection algorithm to generate a 3D detection box for each obstacle. By calculating the eight vertices of the 3D detection box, it projects the bounding box onto a multi-view image using the intrinsic and extrinsic parameters of the camera and LiDAR. The system finds vertex boundaries in the image plane to extract the region of interest (ROI) and retrieves the corresponding subdivision prediction model based on the target category. After scaling and image normalization preprocessing of the ROI, the scheme stitches multi-view image data along the channel dimension and inputs it into a feature extraction network. The subdivision attribute category of the obstacle in 3D space is output through multi-frame result fusion.

[0004] The aforementioned scheme relies on a rigid geometric transformation matrix to establish the correspondence between 3D bounding boxes and 2D image features. Its logical foundation depends on the ability of LiDAR to acquire continuous and dense physical surface reflection points. When high-transmittance materials such as glass are present in the test scene, or when long-distance occlusion occurs, the laser pulse may penetrate or be lost due to echo, resulting in extremely sparse local point clouds within the target space. Limited by this mechanism's reliance on extrinsic parameters for forced cross-modal binding without considering the actual material bearing state at physical coordinates, the 2D semantic labels of the real target in the background are incorrectly intercepted by high-transmittance materials (such as glass) or sparse noise points in the foreground, leading to incorrect projection and attachment to non-target point clouds at a deeper depth. This lack of a fundamental mechanism results in uncontrollable semantic error interception and spatial misalignment, forcing the system to generate ghost semantic obstacles at incorrect 3D spatial coordinates, causing severe spatial misalignment of the final scene structured data. Summary of the Invention

[0005] To address the spatial misalignment caused by the erroneous interception of semantic labels of real targets in the background by high-transmittance materials or sparse noise in the foreground, and to achieve accurate 3D annotation, this application provides a method and system for automatic semantic annotation of multimodal scenes.

[0006] Firstly, this application provides an automatic semantic annotation method for multimodal scenes, employing the following technical solution: An automatic semantic annotation method for multimodal scenes, comprising: Acquire two-dimensional image frames and three-dimensional point cloud frames of the scene to be labeled, which are aligned with hardware timestamps; extract target pixels carrying two-dimensional semantic labels from the two-dimensional image frames; and convert the three-dimensional point cloud frames into three-dimensional voxel space. A back-projection tracing domain with a central ray corresponding to the target pixel is emitted into the three-dimensional voxel space. Non-empty voxels that are within the back-projection tracing domain and contain a three-dimensional point cloud are selected as candidate voxels, and the candidate voxels are traversed sequentially from near to far according to radial depth. Extract the radial depth, local point density, and intensity features of the candidate voxels currently being traversed, and calculate the lower limit threshold of density based on the radial depth and the preset attenuation relationship; If the local point density is not less than the lower density threshold and the intensity feature is within a preset range, the candidate voxel is determined to be valid, and the two-dimensional semantic label is bound to the candidate voxel, and the traversal and binding operation of the target pixel to subsequent deeper voxels is terminated. If the state of the candidate voxel is not determined to be valid, then continue to traverse the next candidate voxel; If all candidate voxels have been traversed and none of the states are deemed valid, then backtrack to calculate the ratio of the local point density of each candidate voxel to the corresponding density lower limit threshold, and extract the candidate voxel with the largest ratio as the center, and expand the bounding box range layer by layer according to the preset step size to perform neighborhood search. The voxels covered by the neighborhood search and whose status is determined to be valid are used as candidate voxels. If the candidate voxel is found within the preset maximum search level, the candidate voxel with the smallest vertical distance to the central ray of the back projection tracking domain is selected and bound to the two-dimensional semantic label, and the binding operation of the target pixel is terminated. If the preset maximum search level is reached and no candidate voxel is found, the binding operation of the target pixel is terminated.

[0007] Optionally, the step of emitting a back-projection tracking domain with a central ray corresponding to the target pixel in the three-dimensional voxel space includes: Obtain the intrinsic parameter matrix of the camera that acquires the two-dimensional image frames, and the spatial extrinsic parameter matrix between the coordinate system of the camera and the reference coordinate system of the three-dimensional voxel space; Based on the intrinsic parameter matrix and the spatial extrinsic parameter matrix, a coordinate mapping relationship from the two-dimensional pixel coordinate system to the reference coordinate system is constructed; Extract the optical center of the camera and use the coordinate mapping relationship to determine the three-dimensional physical coordinates of the optical center in the reference coordinate system, while calculating the three-dimensional direction vector of the target pixel point mapping; Starting from the three-dimensional physical coordinates, the vector is projected along the three-dimensional direction into the three-dimensional voxel space to generate the central ray of the back-projection tracing domain.

[0008] Optionally, the step of selecting non-empty voxels within the back-projection tracking domain and containing a 3D point cloud as candidate voxels, and sequentially traversing the candidate voxels from near to far according to radial depth, includes: Obtain the pixel footprint parameters of the camera, and calculate the spatial divergence angle of the central ray based on the pixel footprint parameters; Using the three-dimensional physical coordinates as the cone apex and the three-dimensional direction vector as the central axis, a spatial view cone projected into the three-dimensional voxel space is constructed in combination with the spatial divergence angle, which serves as the back-projection tracking domain. Non-empty voxels that have spatial geometric intersection with the back-projection tracking domain and contain the three-dimensional point cloud are extracted as candidate voxels, and the candidate voxels are traversed sequentially from near to far according to radial depth.

[0009] Optionally, the extraction of the radial depth, local point density, and intensity features of the currently traversed candidate voxels includes: Calculate the three-dimensional Euclidean distance from the three-dimensional geometric center of the candidate voxel to the origin of the reference coordinate system, and use it as the radial depth; The number of three-dimensional point clouds contained within the candidate voxel is counted and used as the local point density; The mean value of the reflection intensity of all the three-dimensional point clouds within the candidate voxel is calculated as the intensity feature.

[0010] Optionally, calculating the density lower limit threshold based on the radial depth and a preset attenuation relationship includes: Obtain the preset spatial attenuation constant, preset smooth minimum value, preset maximum cutoff threshold, and preset noise floor minimum value; The ratio of the spatial attenuation constant to the sum of the square of the radial depth and the smooth minimum value is calculated as the first attenuation density value. The smaller value between the first attenuation density value and the maximum cutoff threshold is selected and compared with the minimum noise floor value, and the larger value is taken as the lower density threshold.

[0011] Optionally, calculating the density lower limit threshold based on the radial depth and a preset attenuation relationship includes: Obtain a preset segmented lookup table, which includes multiple radial depth intervals and a reference density threshold corresponding to each interval, and each reference density threshold is not less than a preset minimum noise floor value. The radial depth is used as an index to query the segmented lookup table, and the matching baseline density threshold is extracted as the lower density threshold.

[0012] Optionally, binding the two-dimensional semantic label to the candidate voxel includes: Extract the three-dimensional geometric center of the candidate voxel whose status is determined to be valid; Associating the two-dimensional semantic tags with the three-dimensional geometric center, and encapsulating them to generate a structured semantic node data packet; The structured semantic node data packet is written into a preset scene semantic database to construct a mapping node between the two-dimensional semantic label and the candidate voxel in the three-dimensional voxel space.

[0013] Optionally, the step of extracting the candidate voxel with the largest ratio as the center and expanding the bounding box range layer by layer with a preset step size to perform a neighborhood search includes: Using the three-dimensional geometric center of the candidate voxel with the largest ratio as a reference, an initial three-dimensional bounding box is constructed; The three-dimensional boundary of the initial three-dimensional bounding box is expanded outward layer by layer according to the preset step size to generate the bounding box search domain of the current search level. The radial depth, local point density, and intensity features of the neighboring voxels within the bounding box search domain are extracted, and the density lower limit threshold is recalculated based on the extracted radial depth to determine whether the state of the neighboring voxels is valid.

[0014] Optionally, the process of selecting the candidate voxel with the smallest vertical distance to the central ray of the back-projection tracking domain and binding it to the two-dimensional semantic label includes: If there are at least two neighboring voxels in the bounding box search domain of the current search level that are determined to be valid, then each of the neighboring voxels is marked as a candidate voxel. Calculate the perpendicular spatial Euclidean distance from the three-dimensional geometric center of each candidate voxel to the central ray; The candidate voxel with the smallest vertical spatial Euclidean distance is selected and bound to the two-dimensional semantic label, and the neighborhood search is terminated.

[0015] Secondly, this application provides a multimodal scene automatic semantic annotation system, which adopts the following technical solution: A multimodal scene automatic semantic annotation system, comprising: The data acquisition module is used to acquire two-dimensional image frames and three-dimensional point cloud frames of the scene to be labeled, which are aligned with hardware timestamps, extract target pixels carrying two-dimensional semantic labels from the two-dimensional image frames, and convert the three-dimensional point cloud frames into three-dimensional voxel space. The mapping and tracking module is used to emit a back-projection tracking domain with a central ray corresponding to the target pixel in the three-dimensional voxel space, and to take non-empty voxels that are in the back-projection tracking domain and contain three-dimensional point clouds as candidate voxels, and to traverse the candidate voxels in order from near to far according to radial depth. The feature arbitration module is used to extract the radial depth, local point density, and intensity features of the candidate voxel being traversed, and to calculate the density lower limit threshold based on the radial depth and a preset attenuation relationship; if the local point density is not less than the density lower limit threshold and the intensity feature is located in a preset range, the candidate voxel is determined to be valid, and the two-dimensional semantic label is bound to the candidate voxel, and the traversal and binding operation of the target pixel to subsequent deeper voxels is terminated; The annotation compensation module is used to: if the state of the candidate voxel is not determined to be valid, continue traversing the next candidate voxel; if all candidate voxels have been traversed and none of their states are determined to be valid, backtrack to calculate the ratio of the local point density of each candidate voxel to the corresponding density lower limit threshold, and extract the candidate voxel with the largest ratio as the center, and expand the bounding box range layer by layer according to a preset step size for neighborhood search; voxels covered by the neighborhood search and whose states are determined to be valid are used as candidate voxels; if the candidate voxel is found within the preset maximum search level, select the candidate voxel with the smallest vertical distance to the center ray of the back-projection tracking domain and bind it to the two-dimensional semantic label, and terminate the binding operation of the target pixel; if the candidate voxel is not found even after reaching the preset maximum search level, terminate the binding operation of the target pixel.

[0016] In summary, this application includes the following beneficial technical effects: 1. This application traverses candidate voxels layer by layer from near to far along the back projection tracking domain, and introduces a dual validity judgment condition consisting of a density lower limit threshold and an intensity feature interval based on the dynamic change of radial depth. Only when the local point density and intensity feature of the voxel simultaneously meet the threshold requirements will the two-dimensional semantic label be bound to the current candidate voxel and the traversal of deeper layers be terminated. This objectively avoids the erroneous interception of two-dimensional semantic labels by high-transmittance materials or sparse noise points in front, effectively reduces the risk of the real target semantics in the back being incorrectly attached to non-target voxels in front, and improves the positional accuracy of semantic annotation in three-dimensional voxel space.

[0017] 2. This application performs neighborhood search by expanding the bounding box layer by layer around the candidate voxel with the largest ratio, and selects the binding target from the candidate voxels based on the minimum vertical distance to the center ray. When the main line traversal fails to determine the effective voxel, it provides a supplementary annotation path, which effectively improves the ability of 2D semantic labels to be anchored to the 3D physical surface in sparse point cloud or occluded scenes, and expands the working condition adaptability of the automatic semantic annotation method.

[0018] 3. This application extracts target pixels from two-dimensional image frames and three-dimensional point cloud frames after hardware timestamp alignment and transforms them into three-dimensional voxel space. It constructs a back-projection tracking domain by combining pre-calibrated camera intrinsic and spatial extrinsic matrices, and realizes the synchronous extraction of local point density and intensity features based on the statistical analysis of the number of points in the voxel cloud and the calculation of the average reflection intensity. This effectively reduces semantic drift caused by sensor spatiotemporal misalignment or insufficient feature extraction during cross-modal mapping and enhances the information fidelity of the multimodal data fusion process. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the automatic semantic annotation method for multimodal scenes according to an embodiment of this application. Figure 2 This is a schematic diagram illustrating the principle of alternative voxel neighborhood search and binding in an embodiment of this application. Detailed Implementation

[0020] The following combination Figures 1-2 This application will be described in further detail.

[0021] like Figure 1As shown in the figure, this application discloses an automatic semantic annotation method for multimodal scenes, which relates to automatic semantic annotation technology for multimodal scenes. It can be applied to engineering scenarios that require cross-modal semantic fusion of two-dimensional images and three-dimensional point clouds, such as autonomous driving environment perception and robot scene mapping. The entire method is implemented based on a corresponding automatic semantic annotation system for multimodal scenes. The system is deployed on an automotive-grade domain controller or robot edge computing platform and includes four functional units: a data acquisition module, a mapping and tracking module, a feature arbitration module, and an annotation compensation module. Each unit completes the interaction of data and control signals through an on-chip bus. The data acquisition module interfaces with sensor hardware to complete the acquisition and preprocessing of raw data. The mapping and tracking module completes the mapping from pixels to three-dimensional space and the selection of candidate voxels. The feature arbitration module completes the determination of the physical carrying validity of voxels and the binding of semantic labels. The annotation compensation module completes the neighborhood search of invalid voxels and the handling of abnormal scenes.

[0022] The sensor unit used in this embodiment includes an industrial camera and a 64-line LiDAR. The two are synchronized at the microsecond level via a hardware synchronization unit, ensuring that the acquired 2D image frames and 3D point cloud frames have strictly aligned hardware timestamps. The industrial camera outputs image data with a resolution of 1920×1080 pixels, a frame rate of 30 frames per second, a horizontal field of view of 90 degrees, and a vertical field of view of 50 degrees. The 64-line LiDAR outputs point cloud data containing 3D coordinates and reflection intensity information, at a frame rate of 10 frames per second, a ranging range of 0.5m to 200m, a horizontal angular resolution of 0.2 degrees, and a vertical angular resolution of 0.33 degrees. The hardware synchronization unit synchronously outputs PPS pulses and GPRMC timecodes to both the industrial camera and the 64-line LiDAR, with synchronization accuracy controlled within 1μs, eliminating spatiotemporal misalignment caused by sensor acquisition delays.

[0023] The computing platform employs an automotive-grade domain controller and runs the QNX real-time operating system. Its core computing power unit consists of a 128 TOPS AI acceleration core and an 8-core ARM processor. All system functional modules are embedded in the computing platform's storage unit and loaded into memory for real-time processing during runtime.

[0024] This embodiment pre-calibrates and configures the basic parameters, all of which are determined based on the sensor's physical characteristics and engineering scenario requirements. First, the camera's intrinsic and extrinsic parameters are calibrated. The camera's intrinsic parameters are obtained using the Zhang Zhengyou calibration method, including the x-axis equivalent focal length, y-axis equivalent focal length, image principal point coordinates, and distortion coefficients. The calibration reprojection error does not exceed 0.5 pixels. The extrinsic parameters are the transformation parameters between the camera coordinate system and the LiDAR reference coordinate system in 3D voxel space, including rotation matrices and translation vectors. Calibration is performed using a hand-eye calibration method to ensure the mapping accuracy from the 2D pixel coordinate system to the LiDAR reference coordinate system. If the calibration parameters fail, the system throws a calibration exception, terminates the data acquisition process, and waits for parameter recalibration.

[0025] Then, the core parameters of the algorithm were preset. The three-dimensional voxel space was divided into uniform cubic meshes, with each voxel having a side length of 0.1m. The spatial coverage range along the lidar reference coordinate system was -50m to 200m along the x-axis, -50m to 50m along the y-axis, and -5m to 10m along the z-axis. The preset spatial attenuation constant K was set to 400, with dimensions in meters. 2 • Points / voxels, matching the lower limit of standard point cloud density at a depth of 10m for a 64-line LiDAR; the preset smoothing minimum ε is set to 0.01m. 2 To avoid division by zero errors at close-range depths, a preset maximum truncation threshold of 20 points / voxel is used, matching the maximum point cloud capacity of a voxel at a close range of 0.5m. Specifically, a preset minimum noise floor is set to 2 points / voxel, serving as a mandatory lower threshold for long-range detection to prevent isolated single-point noise from penetrating the threshold. The preset interval for intensity features is set to a lower limit of 10 and an upper limit of 200, matching the reflection intensity distribution characteristics of targets in typical scenes. The preset step size for neighborhood search is set to 0.1m, consistent with the voxel side length, and the preset maximum search level is set to 5 layers.

[0026] The data acquisition module responds to the PPS pulse and GPRMC timecode trigger signal output by the hardware synchronization unit, latches the two-dimensional image frame and the three-dimensional point cloud frame under the same hardware timestamp, discards the current frame data when hardware synchronization fails, and waits for the next synchronization trigger signal; when the two-dimensional image frame or the three-dimensional point cloud frame is empty, the processing flow of the current frame is terminated.

[0027] For the acquired 2D image frames, radial and tangential distortion correction is performed based on pre-calibrated camera distortion coefficients, outputting distortion-free image data aligned to the pixel coordinate system. The distortion-corrected image data is then input into a pre-trained semantic segmentation network. This pre-trained semantic segmentation network employs an encoder-decoder architecture; specifically, the encoder contains consecutively stacked convolutional feature extraction layers to downsample the distortion-corrected image data and extract multi-scale spatial features; the decoder contains feature upsampling layers to progressively upsample the extracted multi-scale spatial features, outputting a single-channel semantic mask with the same resolution as the distortion-corrected image data. This encoder-decoder architecture extracts and preserves the multi-scale spatial features of the image, improving the accuracy of target pixel extraction and its 2D semantic label extraction. Each pixel value in the single-channel semantic mask corresponds to a pre-defined 2D semantic label, covering typical targets in various scenarios such as vehicles, pedestrians, roads, buildings, guardrails, and traffic signs. The data acquisition module traverses all pixels within a single-channel semantic mask, extracting pixels carrying non-background 2D semantic labels as target pixels. It simultaneously records the pixel coordinates and corresponding 2D semantic label for each target pixel. Background pixels are directly discarded and do not proceed to subsequent processing. If no target pixels are found after semantic segmentation, the processing flow for the current frame is terminated.

[0028] For the synchronously acquired 3D point cloud frames, statistical filtering is used to remove outliers. The filter selects 50 neighboring points, with a standard deviation factor of 2, filtering out discrete noise points whose average distance from their neighbors exceeds twice the standard deviation. The preprocessed point cloud data is mapped to a predefined 3D voxel space in the LiDAR reference coordinate system; point clouds outside the voxel space are directly discarded. After mapping from point cloud to voxels, three sets of basic data are synchronously recorded for each voxel: the absolute coordinates of the voxel's 3D geometric center in the LiDAR reference coordinate system, the total number of valid 3D point clouds contained within the voxel, and the arithmetic mean of the reflection intensities of all valid point clouds within the voxel, thus completing the construction of the 3D voxel space.

[0029] The mapping and tracking module constructs a coordinate mapping relationship from the 2D pixel coordinate system to the LiDAR reference coordinate system based on the pre-calibrated camera intrinsic and spatial extrinsic matrices. It extracts the camera's optical center and converts it into 3D physical coordinates in the LiDAR reference coordinate system using the coordinate mapping relationship. For each target pixel, its pixel coordinates are converted into a normalized direction vector in the camera coordinate system using the camera intrinsic matrix, and then converted into a 3D unit direction vector in the LiDAR reference coordinate system using the rotation component of the spatial extrinsic matrix. Starting from the 3D physical coordinates of the optical center, this 3D unit direction vector is projected into the 3D voxel space, generating a central ray of the back-projection tracking domain. The effective depth range of the central ray is limited to the effective ranging range of the LiDAR, from 0.5m to 200m. The processing flow for the current target pixel is terminated when the camera intrinsic matrix is ​​singular or the target pixel coordinates exceed the effective range of the image.

[0030] The mapping and tracking module acquires the camera's pixel footprint parameters, which correspond to the field-of-view angle increment of a single pixel on the imaging plane. Based on these parameters, the spatial divergence angle of the central ray is calculated. For an imaging system with a resolution of 1920×1080, a horizontal field-of-view angle of 90 degrees, and a vertical field-of-view angle of 50 degrees, the horizontal field-of-view angle of a single pixel is 0.046875 degrees, and the vertical field-of-view angle of a single pixel is 0.046296 degrees, which are used as the spatial divergence angle values ​​in the horizontal and vertical directions, respectively. The mapping and tracking module constructs a rectangular cross-section spatial cone projected into the three-dimensional voxel space, using the three-dimensional physical coordinates of the optical center as the apex and the three-dimensional unit direction vector as the central axis, combined with the horizontal and vertical spatial divergence angles. This cone serves as the back-projection tracking domain corresponding to the target pixel. The cross-sectional size of the cone increases linearly with the projection distance, corresponding one-to-one with the physical projection range of a single pixel at different depths. When the back-projection tracking domain completely exceeds the coverage of the voxel space, the processing flow for the current target pixel is terminated.

[0031] The mapping and tracing module employs a three-level traversal logic to filter candidate voxels: first, it calculates the axis-aligned bounding box of the back-projection tracing domain to limit the spatial range of voxel traversal; then, based on the effective depth range of the central ray, it eliminates invalid voxels outside the depth range; finally, it performs a precise hexahedral intersection test on the coarsely filtered voxels to extract non-empty voxels that are completely within the back-projection tracing domain and contain 3D point clouds, marking them as candidate voxels, and traversing these candidate voxels sequentially from near to far radial depth. If there are no valid candidate voxels within the back-projection tracing domain, the processing flow for the current target pixel is directly terminated.

[0032] In some embodiments, for large distortion imaging systems such as fisheye cameras, an asymmetric view frustum based on a pixel distortion model can be used to construct the back-projection tracking domain, adapting to the nonlinear variation characteristics of pixel footprints in wide-angle scenes. In some embodiments, for fast preview scenes with low voxelization accuracy requirements, a 3D DDA ray traversal algorithm can be used instead of axis-aligned bounding box intersection testing, traversing voxels along the central ray path to simplify the calculation process. In close-range indoor scenes, the spatial divergence of pixel footprints is negligible, and a cylindrical back-projection tracking domain with a fixed radius of 0.05m can be directly generated with the central ray as the axis, simplifying the calculation process of spatial intersection. In some embodiments, the point spread function of the camera can be used to calculate pixel footprint parameters, and the spatial divergence angle can be dynamically adjusted by combining the optical distortion characteristics of the lens and the focusing distance to improve the matching accuracy of the back-projection tracking domain.

[0033] After the mapping and tracking module completes the candidate voxel selection, it generates a structured output data package containing a candidate voxel list, central ray parameters, and the 2D semantic labels and pixel coordinates of the target pixels, which is then synchronously transmitted to the feature arbitration module. Each data entry in the candidate voxel list includes the voxel ID, 3D geometric center, total number of internal point clouds, and average reflection intensity; the central ray parameters include the starting point's 3D coordinates, unit direction vector, and effective depth range.

[0034] When the feature arbitration module traverses the candidate voxel sequentially from near to far according to radial depth, it reads the pre-stored 3D geometric center, total number of internal point clouds, and average point cloud reflection intensity for the currently traversed candidate voxel, thus extracting radial depth, local point density, and intensity features. Radial depth is the 3D Euclidean distance from the candidate voxel's 3D geometric center to the origin of the lidar reference coordinate system; local point density is the total number of effective 3D point clouds contained within the candidate voxel; and intensity feature is the arithmetic mean of the reflection intensities of all effective point clouds within the candidate voxel.

[0035] The feature arbitration module calculates the lower density threshold corresponding to the current candidate voxel based on the extracted radial depth and the preset attenuation relationship. In this embodiment, a controlled attenuation formula is used to complete the calculation. The preset spatial attenuation constant, preset smooth minimum value, preset maximum cutoff threshold, and preset noise floor minimum value are read. First, the ratio of the preset spatial attenuation constant to the sum of the square of the radial depth and the preset smooth minimum value is calculated to obtain the first attenuation density value. Then, the smaller value between the first attenuation density value and the preset maximum cutoff threshold is selected. Subsequently, it is compared with the preset noise floor minimum value, and the larger value is selected as the lower density threshold. This makes the lower density threshold exhibit a trend of first maintaining a horizontal cutoff, then smoothly decreasing, and finally being forcibly supported by the preset noise floor minimum value at a long distance as the radial depth increases.

[0036] As a supplementary implementation, the lower density threshold can be obtained through a preset segmented lookup table. The segmented lookup table pre-stores multiple sets of continuous radial depth intervals and corresponding baseline density thresholds for each interval, with all baseline density thresholds set to be no less than a preset minimum noise floor value. In this embodiment, the radial depth intervals 0m to 5m correspond to 20 points / voxel of baseline density thresholds, 5m to 20m to 8 points / voxel, 20m to 50m to 3 points / voxel, 50m to 100m to 2 points / voxel, and 100m to 200m to 2 points / voxel. The feature arbitration module uses the radial depth of the candidate voxels as an index to query the segmented lookup table and extracts the matching baseline density threshold as the lower density threshold. Using this segmented lookup table to replace the floating-point division operation of the controlled decay formula reduces the computational overhead and execution latency of the automotive-grade computing platform.

[0037] The feature arbitration module compares the local point density of candidate voxels with a density lower limit threshold and simultaneously compares the intensity features with a preset interval. A candidate voxel is deemed valid only when the local point density is not less than the density lower limit threshold and the intensity feature is within the preset interval; otherwise, it is deemed invalid. Because the density lower limit threshold introduces a constraint of maximizing the preset minimum noise value, even if the theoretical threshold calculated by the attenuation formula is lower than the preset minimum noise value, the system will still use the preset minimum noise value for judgment in long-distance conditions, thereby filtering out isolated penetrating noise points on deep rays caused by environmental interference. For candidate voxels deemed valid, their 3D geometric center is extracted, and the 2D semantic label corresponding to the target pixel is associated with the 3D geometric center. This is encapsulated to generate a structured semantic node data package, written to a preset scene semantic database, constructing a mapping node between the 2D semantic label and the candidate voxel in 3D physical space, and terminating the traversal and binding operation of the target pixel to subsequent deeper voxels. For candidate voxels whose status is determined to be invalid, they are simultaneously transmitted to the annotation compensation module, and the corresponding central ray parameters and two-dimensional semantic labels are simultaneously associated.

[0038] For candidate voxels whose states are not determined to be valid, the annotation compensation module continues the traversal and determination process for the next candidate voxel. If all candidate voxels corresponding to the target pixel have been traversed and none of them are determined to be valid, the annotation compensation module will backtrack to calculate the ratio of the local point density of all candidate voxels to their corresponding density lower limit threshold, and extract the candidate voxel with the largest ratio. Using its 3D geometric center as a reference, an initial 3D bounding box is constructed. The size of the initial 3D bounding box is consistent with the voxel side length. The 3D boundary of the initial 3D bounding box is expanded outward layer by layer according to a preset step size to generate the bounding box search domain of the current search level. The bounding box of each level expands by 2 steps in each of the x, y, and z axes, and the corresponding bounding box size increases by 0.2m for each level. The annotation compensation module extracts the pre-stored 3D geometric center, total number of internal point clouds, and average point cloud reflection intensity of all neighboring voxels within the bounding box search domain. It then extracts radial depth, local point density, and intensity features. Based on the extracted radial depth, it recalculates the corresponding density lower limit threshold and determines the validity status of neighboring voxels according to the same judgment rules as the candidate voxels.

[0039] The annotation compensation module marks voxels whose neighborhood search coverage is complete and whose status is determined to be valid as candidate voxels. When there are at least two candidate voxels within the bounding box search domain of the current search level, it combines... Figure 2 Draw a spatial perpendicular line from the three-dimensional geometric center of each candidate voxel to the central ray, calculate the corresponding vertical spatial Euclidean distance, select the candidate voxel with the smallest vertical spatial Euclidean distance and bind it to the two-dimensional semantic label of the target pixel, and simultaneously terminate the neighborhood search and binding operation of the target pixel.

[0040] If the iterative search reaches the preset maximum search level and still fails to find a candidate voxel, the binding operation of the target pixel is terminated, no structured semantic node data packet is generated, and the 3D mapping link of the target pixel is directly discarded.

[0041] This embodiment verifies the effectiveness through a typical urban scenario. In this scenario, a glass curtain wall is placed 10 meters in front of the autonomous vehicle, and a stationary vehicle is located 30 meters behind the glass curtain wall. Existing technologies incorrectly project the 2D semantic tags of the vehicle behind the glass onto a glass voxel at a depth of 10 meters using a rigid extrinsic matrix, generating a ghost semantic obstacle. In this embodiment, the number of point clouds within the glass voxel at a depth of 10 meters is extremely small, with local point density below the lower density threshold corresponding to that depth. Simultaneously, the glass reflection intensity is below a preset lower limit, thus it is judged as an invalid voxel. The system continues to traverse deeper candidate voxels until it reaches 30 meters behind the vehicle. Since the local point density and intensity characteristics of the real vehicle voxel meet the validity conditions, the system determines it as valid and binds it, thereby terminating the process early. Only when the real vehicle point cloud on the ray is extremely sparse, rendering the entire line invalid, will the labeling compensation module initiate a backtracking neighborhood search after traversal. This objectively avoids the problem of semantic labels of real targets in the back being incorrectly intercepted and misbound by high-transmittance materials (such as glass) or sparse noise points in front, thus achieving accurate anchoring of two-dimensional semantic information to the surface of three-dimensional physical targets.

[0042] In an alternative implementation, the preset interval of the intensity feature can be dynamically adjusted according to the category of the two-dimensional semantic label to match the material reflection characteristics of different categories of targets and improve the accuracy of validity determination. The preset step size and preset maximum search level of the neighborhood search can be dynamically adjusted according to the radial depth. The larger the radial depth, the larger the preset step size and the smaller the preset maximum search level, balancing the search range and computational efficiency of distant scenes.

[0043] This application also discloses a multimodal scene automatic semantic annotation system. The system is deployed on an automotive-grade domain controller or robot edge computing platform and includes a sensor unit, a hardware synchronization unit, and a computing processing unit. The sensor unit and the hardware synchronization unit are electrically connected, and the hardware synchronization unit and the computing processing unit are communicatively coupled through an automotive-grade high-speed serial bus. The various functional modules within the computing processing unit complete data interaction and control signal transmission through an on-chip bus.

[0044] The sensor unit comprises an industrial camera and a 64-line LiDAR. The industrial camera is configured to acquire 2D image frames of the scene to be labeled, outputting image data with a resolution of 1920×1080 pixels and a frame rate of 30 frames per second, a horizontal field of view of 90 degrees, and a vertical field of view of 50 degrees. The 64-line LiDAR is configured to acquire 3D point cloud frames of the scene to be labeled, outputting point cloud data containing 3D coordinates and reflection intensity information, with a frame rate of 10 frames per second, a ranging range of 0.5m to 200m, a horizontal angular resolution of 0.2 degrees, and a vertical angular resolution of 0.33 degrees.

[0045] The hardware synchronization unit is configured to synchronously output PPS pulses and GPRMC timecodes to the industrial camera and 64-line LiDAR, realizing hardware timestamp synchronization of two-dimensional image frames and three-dimensional point cloud frames, with synchronization accuracy controlled within 1μs.

[0046] The computing and processing unit is equipped with the QNX real-time operating system and integrates an automotive-grade multi-core ARM processor, a 128 TOPS AI acceleration core, and non-volatile memory. The processor, AI acceleration core, and memory interact with each other via an on-chip high-speed bus. The computing and processing unit comprises four functional units: a data acquisition module, a mapping and tracking module, a feature arbitration module, and a labeling and compensation module.

[0047] The data acquisition module is communicatively coupled to the sensor unit and the hardware synchronization unit. It is configured to acquire two-dimensional image frames and three-dimensional point cloud frames of the scene to be labeled, which are aligned with hardware timestamps. Based on the pre-calibrated camera distortion coefficients, the two-dimensional image frames are distorted. The rectified image frames are then subjected to pixel-level semantic segmentation through a pre-trained semantic segmentation network. This pre-trained semantic segmentation network adopts an encoder-decoder architecture consisting of a convolutional feature extraction layer and a feature upsampling layer. It extracts target pixels carrying two-dimensional semantic labels from the two-dimensional image frames and simultaneously records the pixel coordinates and corresponding two-dimensional semantic labels of the target pixels. The three-dimensional point cloud frames are then de-scrambled and noise is removed. The pre-processed three-dimensional point cloud frames are mapped to a three-dimensional voxel space under a predefined lidar reference coordinate system. For each voxel in the three-dimensional voxel space, the three-dimensional geometric center, the total number of effective three-dimensional point clouds inside, and the arithmetic mean of the reflection intensity of all effective point clouds inside are recorded simultaneously.

[0048] The mapping and tracking module is communicatively coupled to the data acquisition module and is configured to construct a coordinate mapping relationship from the two-dimensional pixel coordinate system to the lidar reference coordinate system based on the pre-calibrated camera intrinsic parameter matrix and spatial extrinsic parameter matrix; it emits a back-projection tracking domain corresponding to the target pixel with a central ray into the three-dimensional voxel space; it extracts non-empty voxels that are completely within the back-projection tracking domain and contain three-dimensional point clouds, marks them as candidate voxels, and traverses these candidate voxels sequentially from near to far in radial depth; it generates and outputs a structured data packet containing a list of candidate voxels, central ray parameters, and two-dimensional semantic tags of the target pixel.

[0049] The feature arbitration module is communicatively coupled to the mapping and tracking module and is configured to extract the radial depth, local point density, and intensity features of the currently traversed candidate voxels. It calculates a lower density threshold based on the radial depth and a preset attenuation relationship, and introduces a preset minimum noise floor as a constraint during the calculation process to ensure that the lower density threshold is not less than the preset minimum noise floor. It compares the local point density with the lower density threshold and the intensity features with a preset interval. If the local point density is not less than the lower density threshold and the intensity features are within the preset interval, the candidate voxel is deemed valid. A two-dimensional semantic label is then bound to the candidate voxel, encapsulated to generate a structured semantic node data package, and written to a preset scene semantic database. The traversal and binding operation of the target pixel to subsequent deeper voxels is terminated. If the candidate voxel is not deemed valid, the candidate voxel, its corresponding central ray parameters, and the two-dimensional semantic label are output synchronously.

[0050] The annotation compensation module is communicatively coupled to the feature arbitration module and the mapping tracking module. It is configured to receive candidate voxels whose status is not determined to be valid, their corresponding center ray parameters, and two-dimensional semantic labels, and continue to traverse the next candidate voxel. If all candidate voxels have been traversed and none of them are determined to be valid, the ratio of the local point density of each candidate voxel to its corresponding density lower limit threshold is calculated backtrackingly. The candidate voxel with the largest ratio is extracted as the center, and the bounding box range is expanded layer by layer according to a preset step size for neighborhood search. The radial depth, local point density, and intensity features of the neighboring voxels covered by the neighborhood search are extracted. Based on the extracted radial depth, the density lower limit threshold is recalculated to complete the validity status determination of the neighboring voxels. Voxels whose status is determined to be valid are marked as candidate voxels. When a candidate voxel is found within the preset maximum search level, the candidate voxel with the smallest vertical distance to the center ray of the back projection tracking domain is selected and bound to the two-dimensional semantic label, and the binding operation of the target pixel is terminated. If no candidate voxel is found after reaching the preset maximum search level, the binding operation of the target pixel is terminated.

[0051] The micro-level details of the specific data processing and control logic executed by each module in this embodiment have been described in detail in the foregoing method embodiments and will not be repeated here. In this embodiment, the data acquisition module, mapping and tracking module, feature arbitration module, and annotation compensation module are all logical function modules, and their functions are implemented by the processor or AI acceleration core in the computing unit executing the computer program stored in the storage medium.

[0052] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. All equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for automatic semantic annotation of multimodal scenes, characterized in that, include: Acquire two-dimensional image frames and three-dimensional point cloud frames of the scene to be labeled, which are aligned with hardware timestamps; extract target pixels carrying two-dimensional semantic labels from the two-dimensional image frames; and convert the three-dimensional point cloud frames into three-dimensional voxel space. A back-projection tracing domain with a central ray corresponding to the target pixel is emitted into the three-dimensional voxel space. Non-empty voxels that are within the back-projection tracing domain and contain a three-dimensional point cloud are selected as candidate voxels, and the candidate voxels are traversed sequentially from near to far according to radial depth. Extract the radial depth, local point density, and intensity features of the candidate voxels currently being traversed, and calculate the lower limit threshold of density based on the radial depth and the preset attenuation relationship; If the local point density is not less than the lower density threshold and the intensity feature is within a preset range, the candidate voxel is determined to be valid, and the two-dimensional semantic label is bound to the candidate voxel, and the traversal and binding operation of the target pixel to subsequent deeper voxels is terminated. If the state of the candidate voxel is not determined to be valid, then continue to traverse the next candidate voxel; If all candidate voxels have been traversed and none of the states are deemed valid, then backtrack to calculate the ratio of the local point density of each candidate voxel to the corresponding density lower limit threshold, and extract the candidate voxel with the largest ratio as the center, and expand the bounding box range layer by layer according to the preset step size to perform neighborhood search. The voxels covered by the neighborhood search and whose status is determined to be valid are used as candidate voxels. If the candidate voxel is found within the preset maximum search level, the candidate voxel with the smallest vertical distance to the central ray of the back projection tracking domain is selected and bound to the two-dimensional semantic label, and the binding operation of the target pixel is terminated. If the preset maximum search level is reached and no candidate voxel is found, the binding operation of the target pixel is terminated.

2. The automatic semantic annotation method for multimodal scenes according to claim 1, characterized in that, The step of emitting a back-projection tracking domain with a central ray corresponding to the target pixel in the three-dimensional voxel space includes: Obtain the intrinsic parameter matrix of the camera that acquires the two-dimensional image frames, and the spatial extrinsic parameter matrix between the coordinate system of the camera and the reference coordinate system of the three-dimensional voxel space; Based on the intrinsic parameter matrix and the spatial extrinsic parameter matrix, a coordinate mapping relationship from the two-dimensional pixel coordinate system to the reference coordinate system is constructed; Extract the optical center of the camera and use the coordinate mapping relationship to determine the three-dimensional physical coordinates of the optical center in the reference coordinate system, while calculating the three-dimensional direction vector of the target pixel point mapping; Starting from the three-dimensional physical coordinates, the vector is projected along the three-dimensional direction into the three-dimensional voxel space to generate the central ray of the back-projection tracing domain.

3. The automatic semantic annotation method for multimodal scenes according to claim 2, characterized in that, The step of selecting non-empty voxels within the back-projection tracking domain and containing a 3D point cloud as candidate voxels, and traversing the candidate voxels sequentially from near to far according to radial depth, includes: Obtain the pixel footprint parameters of the camera, and calculate the spatial divergence angle of the central ray based on the pixel footprint parameters; Using the three-dimensional physical coordinates as the cone apex and the three-dimensional direction vector as the central axis, a spatial view cone is constructed by combining the spatial divergence angle and projected into the three-dimensional voxel space, serving as the back-projection tracking domain. Non-empty voxels that have spatial geometric intersection with the back-projection tracking domain and contain the three-dimensional point cloud are extracted as candidate voxels, and the candidate voxels are traversed sequentially from near to far according to radial depth.

4. The automatic semantic annotation method for multimodal scenes according to claim 2, characterized in that, The extraction of the radial depth, local point density, and intensity features of the candidate voxels currently being traversed includes: Calculate the three-dimensional Euclidean distance from the three-dimensional geometric center of the candidate voxel to the origin of the reference coordinate system, and use it as the radial depth; The number of three-dimensional point clouds contained within the candidate voxel is counted and used as the local point density; The mean value of the reflection intensity of all the three-dimensional point clouds within the candidate voxel is calculated as the intensity feature.

5. The automatic semantic annotation method for multimodal scenes according to claim 4, characterized in that, The calculation of the density lower limit threshold based on the radial depth and the preset attenuation relationship includes: Obtain the preset spatial attenuation constant, preset smooth minimum value, preset maximum cutoff threshold, and preset noise floor minimum value; The ratio of the spatial attenuation constant to the sum of the square of the radial depth and the smooth minimum value is calculated as the first attenuation density value. The smaller value between the first attenuation density value and the maximum cutoff threshold is selected and compared with the minimum noise floor value, and the larger value is taken as the lower density threshold.

6. The automatic semantic annotation method for multimodal scenes according to claim 4, characterized in that, The calculation of the density lower limit threshold based on the radial depth and the preset attenuation relationship includes: Obtain a preset segmented lookup table, which includes multiple radial depth intervals and a reference density threshold corresponding to each interval, and each reference density threshold is not less than a preset minimum noise floor value. The radial depth is used as an index to query the segmented lookup table, and the matching baseline density threshold is extracted as the lower density threshold.

7. The automatic semantic annotation method for multimodal scenes according to claim 1, characterized in that, The step of binding the two-dimensional semantic label to the candidate voxel includes: Extract the three-dimensional geometric center of the candidate voxel whose status is determined to be valid; Associating the two-dimensional semantic tags with the three-dimensional geometric center, and encapsulating them to generate a structured semantic node data packet; The structured semantic node data packet is written into a preset scene semantic database to construct a mapping node between the two-dimensional semantic label and the candidate voxel in the three-dimensional voxel space.

8. The automatic semantic annotation method for multimodal scenes according to claim 1, characterized in that, The process of extracting the candidate voxel with the largest ratio as the center and expanding the bounding box range layer by layer according to a preset step size to perform a neighborhood search includes: Using the three-dimensional geometric center of the candidate voxel with the largest ratio as a reference, an initial three-dimensional bounding box is constructed; The three-dimensional boundary of the initial three-dimensional bounding box is expanded outward layer by layer according to the preset step size to generate the bounding box search domain of the current search level. The radial depth, local point density, and intensity features of the neighboring voxels within the bounding box search domain are extracted, and the density lower limit threshold is recalculated based on the extracted radial depth to determine whether the state of the neighboring voxels is valid.

9. The automatic semantic annotation method for multimodal scenes according to claim 8, characterized in that, The process of selecting the candidate voxel with the smallest vertical distance to the central ray of the back-projection tracing domain and binding it to the two-dimensional semantic label includes: If there are at least two neighboring voxels in the bounding box search domain of the current search level that are determined to be valid, then each of the neighboring voxels is marked as a candidate voxel. Calculate the perpendicular spatial Euclidean distance from the three-dimensional geometric center of each candidate voxel to the central ray; The candidate voxel with the smallest vertical spatial Euclidean distance is selected and bound to the two-dimensional semantic label, and the neighborhood search is terminated.

10. A multimodal scene automatic semantic annotation system, used to implement the multimodal scene automatic semantic annotation method as described in any one of claims 1-9, characterized in that, The system includes: The data acquisition module is used to acquire two-dimensional image frames and three-dimensional point cloud frames of the scene to be labeled, which are aligned with hardware timestamps, extract target pixels carrying two-dimensional semantic labels from the two-dimensional image frames, and convert the three-dimensional point cloud frames into three-dimensional voxel space. The mapping and tracking module is used to emit a back-projection tracking domain with a central ray corresponding to the target pixel in the three-dimensional voxel space, and to take non-empty voxels that are in the back-projection tracking domain and contain three-dimensional point clouds as candidate voxels, and to traverse the candidate voxels in order from near to far according to radial depth. The feature arbitration module is used to extract the radial depth, local point density, and intensity features of the candidate voxel being traversed, and to calculate the density lower limit threshold based on the radial depth and a preset attenuation relationship; if the local point density is not less than the density lower limit threshold and the intensity feature is located in a preset range, the candidate voxel is determined to be valid, and the two-dimensional semantic label is bound to the candidate voxel, and the traversal and binding operation of the target pixel to subsequent deeper voxels is terminated; The annotation compensation module is used to: if the state of the candidate voxel is not determined to be valid, continue traversing the next candidate voxel; if all candidate voxels have been traversed and none of their states are determined to be valid, backtrack to calculate the ratio of the local point density of each candidate voxel to the corresponding density lower limit threshold, and extract the candidate voxel with the largest ratio as the center, and expand the bounding box range layer by layer according to a preset step size for neighborhood search; voxels covered by the neighborhood search and whose states are determined to be valid are used as candidate voxels; if the candidate voxel is found within the preset maximum search level, select the candidate voxel with the smallest vertical distance to the center ray of the back-projection tracking domain and bind it to the two-dimensional semantic label, and terminate the binding operation of the target pixel; if the candidate voxel is not found even after reaching the preset maximum search level, terminate the binding operation of the target pixel.

Images