A vehicle-mounted laser radar point cloud semantic segmentation method based on time sequence visual supervision

Abstract

The present application relates to the technical field of intelligent vehicle environment perception and point cloud semantic segmentation, and provides a vehicle-mounted laser radar point cloud semantic segmentation method based on time sequence visual supervision. Current laser radar point cloud, current and adjacent time camera image, sensor calibration parameters and vehicle pose information are obtained; camera visibility of the current point is judged, and the current invisible point is projected to the adjacent camera across time, and the current visible point, time sequence supplementary visible point and continuous invisible point are divided; semantic teacher information is obtained from the real adjacent image corresponding to the time sequence supplementary visible point, time sequence visual supervision is applied to the pure laser radar student network, and prediction retention constraint is applied to the non-time sequence supplementary visible point; after training, only the current laser radar point cloud is input to output the point-by-point semantic category. The present application can expand the real visual supervision coverage of the points outside the current camera field of view, and maintain pure laser radar reasoning.

Description

Technical Field

[0001] This invention relates to the fields of intelligent vehicle environmental perception, 3D computer vision, and semantic understanding of LiDAR point clouds. Specifically, it relates to a semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision. Background Technology

[0002] Vehicle-mounted LiDAR can acquire 3D point clouds with accurate spatial geometry and depth information, but point clouds are discrete, sparse, and lack color and texture semantics. Vehicle-mounted cameras can provide dense visual appearance and semantic information. Therefore, in autonomous vehicle environmental perception, using camera visual information to assist LiDAR point cloud semantic segmentation is an important technical approach to improve the ability to recognize categories such as roads, vehicles, pedestrians, vegetation, and building structures.

[0003] Existing multimodal point cloud semantic segmentation methods typically utilize extrinsic parameters between the camera and LiDAR to project 3D points onto the current camera image, establishing spatial correspondences between points and pixels, and then performing feature fusion, semantic fusion, or knowledge distillation. However, the fields of view of the camera and LiDAR do not completely overlap, and factors such as vehicle motion, occlusion, target distance, and surface orientation can further lead to a large number of current LiDAR points failing to obtain effective correspondences in the current camera image.

[0004] For points currently invisible to the camera, one approach directly sets the camera features to zero, while another predicts pseudo-image features based on adjacent visible points or point cloud features. Neither of these methods fully utilizes real-world camera observations from adjacent moments during the vehicle's continuous movement. In fact, being invisible to the camera at the current moment does not equate to being invisible throughout the entire time window; some current LiDAR points can be re-observed effectively in real-world images from adjacent moments.

[0005] On the other hand, directly regressing the high-dimensional original visual features output by the visual base model to a low-dimensional sparse point cloud feature space can easily lead to mismatches in cross-modal feature dimensions, resolution, and representation structure. Especially when visual supervision is applied only to local points and gradients are fed into a shared point cloud backbone network, it may also alter the existing point cloud feature distribution and semantic decision boundaries in non-target regions.

[0006] Therefore, a point cloud semantic segmentation method is needed that can explicitly distinguish between currently visible points, currently invisible but actually visible points in adjacent time intervals, and persistently invisible points, and construct task-related semantic supervision for currently invisible points based on real adjacent images, while limiting the visual supervision from causing irrelevant perturbations to the prediction of non-target regions; it is also necessary to ensure that the camera, temporal matching, and visual teacher are used only during the training phase, so as to maintain single-frame pure LiDAR inference during actual deployment. Summary of the Invention

[0007] The purpose of this invention is to provide a semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision, in order to solve the problems in the prior art where LiDAR points outside the current camera field of view lack real visual supervision, and where local cross-modal supervision may disturb the existing representation of the pure LiDAR model.

[0008] To achieve the above objectives, the present invention adopts the following technical solution.

[0009] A semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision includes the following steps:

[0010] S1. Acquire the lidar point cloud, the camera image at the current time t, and the camera image at at least one time adjacent to the current time of the target vehicle; acquire camera intrinsic parameters, sensor extrinsic parameters, and vehicle pose information at the current time and the adjacent time; wherein, the sensor extrinsic parameters include the extrinsic parameters between the lidar and the vehicle body and the extrinsic parameters between the camera and the vehicle body.

[0011] S2. Project the three-dimensional points in the lidar point cloud onto the current camera image to obtain the visibility at the current time; for three-dimensional points that are not visible at the current time, transform and project the three-dimensional points onto the adjacent camera images according to the camera intrinsic parameters, the sensor extrinsic parameters and the vehicle pose information to determine the effective camera observations and obtain the visibility at the adjacent time.

[0012] S3. Generate a visibility status marker based on the current visibility and the visibility of adjacent times, and divide the three-dimensional point into currently visible points, temporally supplemented visible points, and continuously invisible points; wherein, the temporally supplemented visible points are three-dimensional points that are not visible at the current time but have the effective camera observation at at least one adjacent time, and the continuously invisible points are three-dimensional points that have no effective camera observation within a preset adjacent time window.

[0013] S4. For the temporal supplementary visible point, based on the effective camera observation corresponding to the temporal supplementary visible point, obtain visual semantic information from the camera image at the adjacent time, and map the visual semantic information to the point cloud semantic category space to obtain semantic teacher information for temporal visual supervision.

[0014] S5. Input the current LiDAR point cloud into the pure LiDAR semantic segmentation student network to obtain student semantic prediction; for the temporally supplemented visible points, apply temporal visual supervision to the student semantic prediction based on the semantic teacher information; for non-temporally supplemented visible points, apply prediction maintenance constraints to the student semantic prediction with reference to the initial semantic prediction output by the pure LiDAR semantic segmentation student network before temporal visual supervision training.

[0015] S6. Train the pure LiDAR semantic segmentation student network based on the point cloud semantic segmentation loss, the temporal visual supervision loss corresponding to the temporal visual supervision, and the prediction holding loss corresponding to the prediction holding constraint.

[0016] S7. After training is completed, in the inference phase, only the current time LiDAR point cloud is input, and the current time camera image and the adjacent time camera image are no longer input. The determination of the visibility of the adjacent time and the generation of the semantic teacher information are no longer performed. The semantic category of each 3D point in the current time LiDAR point cloud is output.

[0017] Furthermore, the cross-time coordinate transformation in S2 includes at least rigid body transformations from the current lidar coordinate system to the current vehicle coordinate system, from the current vehicle coordinate system to the global coordinate system, from the global coordinate system to the vehicle coordinate system at the adjacent time, and from the vehicle coordinate system at the adjacent time to the camera coordinate system at the adjacent time.

[0018] pC(t+Δ)=TCE(t+Δ)·TEG(t+Δ)·TGE(t)·TEL(t)·pL(t)

[0019] Where pL(t) represents a 3D point in the current lidar coordinate system, and pC(t+Δ) represents a 3D point in the camera coordinate system at an adjacent time; TEL(t), TGE(t), TEG(t+Δ), and TCE(t+Δ) represent the rigid body transformation matrices between the corresponding coordinate systems. Here, Δ represents the time offset between adjacent time points and the current time, and its value can be positive or negative.

[0020] Furthermore, pinhole imaging projection is performed on pC(t+Δ) using the camera intrinsic parameter matrix, requiring the point to be located in front of the camera and the projection coordinates to be within the effective image boundary. Depth consistency, occlusion detection, or semantic confidence can be further used to determine whether the camera observations at adjacent time points are valid.

[0021] Furthermore, when the same current 3D point has valid observations in multiple adjacent time points or multiple cameras, the highest quality observation can be selected based on one or more observation quality indicators among time interval, imaging depth, observation angle, distance from projection position to image edge, occlusion consistency, and teacher confidence, or the semantic teacher information of multiple valid observations can be weighted and aggregated.

[0022] Furthermore, for static scene points such as roads, sidewalks, terrain, vegetation, and building structures, cross-time projection can be directly performed based on the vehicle's self-movement pose; for dynamic target points such as vehicles, pedestrians, bicycles, and motorcycles, adjacent time-time visual supervision can be omitted, or adjacent time-time visual correspondence can be constructed after target motion compensation is completed using 3D target bounding boxes, target tracking, scene flow, or target motion models.

[0023] Furthermore, the semantic teacher information is a K-dimensional semantic category score, a category probability distribution, or a soft probability distribution scaled by temperature. The semantic teacher can be obtained by projecting image semantic segmentation networks, multimodal semantic networks, or visual basic model features onto a K-class point cloud semantic space via a semantic mapping module.

[0024] Preferably, instead of directly requiring pure LiDAR student features to regress high-dimensional original visual base model features, the observations of real adjacent images are converted into category semantic teacher information consistent with the final point cloud semantic segmentation task, in order to reduce the modality, dimension and resolution differences between the visual feature space and the sparse point cloud feature space.

[0025] Furthermore, the pure LiDAR semantic segmentation student network includes a base network that already possesses point cloud semantic segmentation capabilities and a trainable adaptive branch. In a preferred embodiment, when training with newly added temporal visual supervision, at least the backbone network of the base network is frozen, and a lightweight residual adaptive branch with zero or near-zero initialization is set to ensure that the student predictions in the initial training phase are consistent with the predictions of the base network.

[0026] For the i-th point, the basic network outputs the initial semantic prediction zᵢ 0 The residual adaptation branch outputs the semantic residual rᵢ, and the updated prediction can be expressed as: zᵢ=zᵢ 0 +gᵢ·rᵢ

[0027] Wherein, gᵢ is a gating coefficient determined by the temporal visibility state and / or semantic teacher confidence. For temporal supplementary visibility points, gᵢ takes a larger value; for non-temporal supplementary visibility points, gᵢ takes a smaller value or zero, and unnecessary prediction drift is suppressed by prediction preservation constraints.

[0028] Furthermore, the total training loss can be expressed as: L=Lseg+λtLtemp+λaLanchor+λrLres

[0029] Where Lseg is the segmentation loss based on the ground truth semantic labels of point clouds, Ltemp is the temporal visual supervision loss computed only on temporally supplemented visible points, Lanchor is the prediction preservation loss computed on non-temporally supplemented visible points, Lres is the residual magnitude or the regularization term of the residual branch parameter, and λt, λa and λr are the loss weights.

[0030] The Ltemp can be soft-label cross-entropy, KL divergence, or other class distribution consistency loss; the Lanchor can be KL divergence, L2 distance, or class consistency loss between student predictions and frozen baseline predictions.

[0031] Compared with the prior art, the beneficial effects of the present invention include at least the following: First, it distinguishes between the current camera being invisible and the entire adjacent time window being continuously invisible, so that a portion of the lidar points that currently have no image corresponding to can obtain visual supervision from real adjacent images, without having to generate pseudo image features out of thin air based on the point cloud neighborhood.

[0032] Second, real-world adjacent image information is mapped to the task-related point cloud semantic category space, reducing the difficulty of direct regression between high-dimensional visual basic model features and sparse point cloud representations.

[0033] Third, by adopting different supervision strategies for temporally supplemented visible points and non-temporally supplemented visible points, and using prediction-preservation constraints or residual update mechanisms, the irrelevant disturbance of local visual supervision to the semantic decision boundary of existing pure lidar is reduced.

[0034] Fourth, camera images, adjacent time-series matching, and visual teachers are only used during the training phase. After training is completed, inference only requires the current LiDAR point cloud, thereby reducing the reliance on multi-sensor synchronization and visual base model computing resources during the deployment phase. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision, according to the present invention.

[0036] Figure 2 This is a schematic diagram illustrating the temporal visibility layering based on camera observations at the current time and adjacent times according to the present invention.

[0037] Figure 3 This is a schematic diagram of the residual adaptation and temporal visual supervision structure based on visibility state of the present invention.

[0038] Figure 4 This is a schematic diagram of the pure lidar inference process after the training is completed according to the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention; equivalent substitutions made to the point cloud backbone network, visual teacher network, time window, loss form and dynamic target motion compensation method without departing from the core concept of the present invention can be applied to the present invention.

[0040] like Figure 1 As shown, during the training phase, the current point cloud of the vehicle-mounted LiDAR is first read and denoted as P. t Simultaneously, it reads the camera image at the current moment, as well as the camera image at the previous key moment and / or the next key moment. The training data also includes the extrinsic parameters of each sensor, the intrinsic parameters of the camera, and the pose of the vehicle in the global coordinate system at each moment.

[0041] For each 3D point in the current point cloud, first perform projection on the current camera. If the point satisfies the positive depth and image boundary conditions of any current camera, it is marked as a currently visible point; otherwise, it is treated as a candidate point for being currently invisible, and the cross-time projection continues.

[0042] For currently invisible candidate points, the three-dimensional point itself remains part of the current point cloud, and historical or future point clouds are not used as student input; the spatial position of the current three-dimensional point is transformed to the camera coordinate system of the adjacent time moment only based on the vehicle's self-motion pose and sensor calibration parameters, and then it is determined whether the point has real adjacent image observations.

[0043] like Figure 2 As shown, based on the above two-level visibility judgment, the current point is divided into three categories: currently visible point, temporally supplemented visible point, and continuously invisible point. A currently visible point indicates that the current camera has made a real observation; a temporally supplemented visible point indicates that the current camera has no effective observation, but the camera has made a real observation at at least one adjacent time; a continuously invisible point indicates that there is still no reliable camera corresponding to it within the preset time window.

[0044] In the first implementation, temporal visual supervision is prioritized for static scene categories. This is because the world position of static points remains unchanged within a short time window, and cross-time correspondence can be mainly achieved through vehicle self-motion transformation. For dynamic objects, temporal visual supervision can be initially withheld; once reliable target tracking or 3D motion estimation is available, object-level motion compensation is performed to establish visual correspondence between adjacent time moments.

[0045] For temporally supplemented visible points, visual semantic information is read from the real image projection positions at their adjacent time points, and semantic teacher information for temporal visual supervision is generated; the semantic teacher information corresponding to the i-th temporally supplemented visible point is denoted as qᵢ. If the visual teacher first outputs two-dimensional high-dimensional features, a semantic mapping module is set to convert the two-dimensional features into K-class semantic scores, the same as those for point cloud segmentation tasks; if the image or multimodal semantic segmentation teacher can directly output K-class results, the class probabilities of the corresponding pixels or local regions are directly read.

[0046] When multiple real-world adjacent image observations exist for the same point, one observation can be selected based on temporal distance and imaging geometric quality; alternatively, the weights of each observation can be calculated and the K-dimensional category probabilities can be normalized and weighted. Compared to performing point-by-point high-dimensional regression on the original visual features, the category semantic teacher directly constrains the final task-related variables.

[0047] A pure LiDAR student network can employ semantic segmentation structures based on points, voxels, sparse convolutions, or point cloud Transformers. Let's denote the basic semantic segmentation network, which already possesses point cloud semantic segmentation capabilities, as F0. Taking a point cloud Transformer as an example, the LiDAR point cloud P at the current time... t Input the basic semantic segmentation network F0, which, through encoder and decoder, yields the point-by-point feature hᵢ of the i-th point, and outputs the K-dimensional initial semantic prediction zᵢ from the basic segmentation head. 0 .

[0048] like Figure 3 As shown, in a preferred embodiment, basic weights with stable point cloud semantic segmentation capabilities are loaded and the basic backbone network is frozen. A lightweight residual adaptation branch is added at the back end of the decoder or before the segmentation head. This residual adaptation branch is denoted as Aθ, where θ represents the trainable parameters of the residual adaptation branch. The residual adaptation branch takes the point-wise feature hᵢ as input and outputs the semantic residual rᵢ corresponding to the i-th point. The residual adaptation branch can employ a two-layer multilayer perceptron, a lightweight attention module, or a small sparse network, with its last layer initialized to zero so that the initial output of the newly added branch is close to zero.

[0049] During training, for temporally supplemented visible points, the temporal visual supervision loss Ltemp is calculated between the semantic teacher distribution and the final semantic distribution of the students; for non-temporally supplemented visible points, the prediction preservation loss Lanchor is calculated between the updated student predictions and the initial semantic predictions. Lanchor does not replace standard point cloud ground truth supervision, but rather serves as an additional safeguard to limit anomalous drift in non-target regions caused by local temporal visual supervision.

[0050] In another implementation, the entire basic backbone may not be frozen, but only the decoder end or a small number of adaptation parameters may be updated; in this case, the temporal visual supervision for temporally supplemented visible points and the prediction preservation constraints for non-temporally supplemented visible points are still retained.

[0051] To verify the coverage potential of real-time visual supervision, statistics were performed on 200 verification samples from a set of publicly available vehicle-mounted multi-sensor data, totaling 6,944,128 LiDAR points. Points that are visible to the camera at least once at the current time account for approximately 58.684%, while points that are completely invisible to the camera at the current time account for approximately 41.316%.

[0052] After further observation of static points using real cameras at adjacent key moments, the current visibility rate of static points is approximately 77.3694%, and the previously visible rate within adjacent time windows increases to approximately 91.8251%, an increase of approximately 14.4557 percentage points. Static points that were currently invisible but regained visual correspondence through real camera observation at adjacent moments account for approximately 10.703% of all points.

[0053] The above statistics show that a considerable number of points that are currently invisible to the camera actually have usable real neighboring visual information. Therefore, this invention uses images from real neighboring moments to construct temporal visual supervision, rather than generating pseudo-visual features for these points.

[0054] For points that remain invisible throughout the entire time window, this invention does not require them to fit specific camera features that do not exist. These points continue to be supervised by ground truth semantics of the point cloud and can be trained using prediction-preserving constraints or other pure point cloud regularization terms.

[0055] The cross-time projection relationship can be generated and stored offline. To reduce the overhead of metadata access and computation during training, a temporal visibility auxiliary file can be created for each current sample, which at least records the neighboring camera identifiers, the projection coordinates from the current point to the neighboring images, the effective visibility mask, and the optional observation quality value.

[0056] When performing geometric augmentations such as rotation, scaling, translation, or cropping on the point cloud during the training phase, the corresponding projection relationships should be updated synchronously or the coordinate transformations should be kept consistent before and after augmentation to avoid incorrect correspondence between the 3D position of the point cloud and the projection of adjacent images.

[0057] After training, the semantic teacher, adjacent time-series images, and temporal visibility auxiliary information are no longer used in inference. For example... Figure 4 As shown, the inference end only receives the current LiDAR point cloud, and the trained pure LiDAR semantic segmentation network outputs point-by-point categories. If the LiDAR-side residual adaptation branch is used during training, this branch only depends on the internal features of the current point cloud and can be retained together with the base network.

[0058] This invention is applicable not only to a fixed time window of one key moment before and after, but can also be extended to multiple moments before and after. As the time window increases, observation attenuation weights can be set according to the time distance to reduce the impact of dynamic environmental changes and calibration error accumulation at excessively distant moments.

[0059] This invention can also be applied to unmanned vehicles, mobile robots, and other ground mobile platforms equipped with LiDAR and cameras; the visual base model, point cloud semantic segmentation network, and training framework used do not constitute a limitation on the scope of protection of this invention.

[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. All modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for point clouds of vehicle-mounted LiDAR based on temporal visual supervision, characterized in that, Includes the following steps: S1. Acquire the lidar point cloud, the camera image at the current time t, and the camera image at at least one time adjacent to the current time of the target vehicle; acquire camera intrinsic parameters, sensor extrinsic parameters, and vehicle pose information at the current time and the adjacent time; wherein, the sensor extrinsic parameters include the extrinsic parameters between the lidar and the vehicle body and the extrinsic parameters between the camera and the vehicle body. S2. Project the three-dimensional points in the lidar point cloud onto the current camera image to obtain the current visibility. For three-dimensional points that are not visible at the current time, transform and project the three-dimensional points onto the adjacent camera images according to the camera intrinsic parameters, the sensor extrinsic parameters and the vehicle pose information to determine the effective camera observations and obtain the adjacent visibility. S3. Generate a visibility status marker based on the current visibility and the visibility of adjacent times, and divide the three-dimensional point into currently visible points, temporally supplemented visible points, and continuously invisible points; wherein, the temporally supplemented visible points are three-dimensional points that are not visible at the current time but have the effective camera observation at at least one adjacent time, and the continuously invisible points are three-dimensional points that have no effective camera observation within a preset adjacent time window. S4. For the temporal supplementary visible point, based on the effective camera observation corresponding to the temporal supplementary visible point, obtain visual semantic information from the camera image at the adjacent time, and map the visual semantic information to the point cloud semantic category space to obtain semantic teacher information for temporal visual supervision. S5. Input the current lidar point cloud into the pure lidar semantic segmentation student network to obtain student semantic prediction; for the temporally supplemented visible points, apply temporal visual supervision to the student semantic prediction based on the semantic teacher information; for non-temporally supplemented visible points, apply prediction maintenance constraints to the student semantic prediction with reference to the initial semantic prediction output by the pure lidar semantic segmentation student network before temporal visual supervision training. S6. Train the pure LiDAR semantic segmentation student network based on the point cloud semantic segmentation loss, the temporal visual supervision loss corresponding to the temporal visual supervision, and the prediction preservation loss corresponding to the prediction preservation constraint. S7. After training is completed, in the inference phase, only the current time LiDAR point cloud is input, and the current time camera image and the adjacent time camera image are no longer input. The determination of the visibility of the adjacent time and the generation of the semantic teacher information are no longer performed. The semantic category of each 3D point in the current time LiDAR point cloud is output.

2. The semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision according to claim 1, characterized in that, In step S2, the specific method for determining the effective camera observation includes: S21, transforming the three-dimensional point in the current lidar coordinate system to the current vehicle coordinate system based on the sensor extrinsic parameters; S22, transforming the three-dimensional point from the current vehicle coordinate system to the global coordinate system based on the vehicle pose information at the current time and the adjacent time, and then transforming it from the global coordinate system to the vehicle coordinate system at the adjacent time; S23, transforming the three-dimensional point from the vehicle coordinate system at the adjacent time to the camera coordinate system at the adjacent time based on the sensor extrinsic parameters; S24, projecting the transformed three-dimensional point onto the camera image at the adjacent time based on the camera intrinsic parameters. When the transformed three-dimensional point satisfies the positive depth condition and the projected coordinates are within the effective image boundary, the corresponding projection is determined as the effective camera observation.

3. The semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision according to claim 1, characterized in that, In step S3, the specific method for generating the visibility status marker includes: S31, when the current time visibility indicates that the 3D point is visible at the current time, marking the 3D point as the currently visible point; S32, when the current time visibility indicates that the 3D point is not visible at the current time, and the adjacent time visibility indicates that the 3D point is visible at at least one adjacent time, marking the 3D point as the temporally supplemented visible point; S33, when the current time visibility indicates that the 3D point is not visible at the current time, and the adjacent time visibility indicates that the 3D point is not visible within the preset adjacent time window, marking the 3D point as the continuously invisible point.

4. The semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision according to claim 1, characterized in that, In step S4, the specific method for obtaining the semantic teacher information includes: S41, determining the projection position of the temporal supplementary visible point in the camera image at the corresponding adjacent time based on the effective camera observations; S42, extracting visual semantic features at the projection position, and using a semantic mapping module to map the visual semantic features into a K-dimensional semantic category score or category probability distribution consistent with the point cloud semantic segmentation task, thereby obtaining the semantic teacher information; S43, when the same temporal supplementary visible point corresponds to multiple effective camera observations, determining the observation weight of each effective camera observation based on at least one of the following: time interval, imaging depth, observation angle, distance between the projection position and the image boundary, occlusion consistency, and semantic confidence; S44, weighting and aggregating the semantic teacher information corresponding to multiple effective camera observations based on the observation weights, thereby obtaining the semantic teacher information corresponding to the temporal supplementary visible point.

5. A semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision according to claim 1 or 2, characterized in that, For static scene points, the vehicle self-motion transformation determined by the vehicle pose information is used to establish effective camera observations in the camera images at adjacent time points; for dynamic target points, at least one of the following processing methods is used: the temporal visual supervision is not applied to the dynamic target points; Alternatively, motion compensation can be performed on the dynamic target points based on the 3D target bounding box, target tracking results, scene flow, or target motion model before establishing the effective camera observation.

6. The semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision according to claim 1, characterized in that, In step S5, the pure LiDAR semantic segmentation student network includes a basic semantic segmentation network and a trainable residual adaptation branch. During training, at least the backbone network of the basic semantic segmentation network is frozen, and the initial semantic prediction is obtained from the basic semantic segmentation network. The residual adaptation branch generates semantic residuals based on the point cloud features output by the basic semantic segmentation network and uses the semantic residuals to update the student semantic predictions. The residual adaptation branch adopts zero initialization or near-zero initialization.

7. The semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision according to claim 6, characterized in that, The total training loss used in S6 satisfies: L=Lseg+λtLtemp+λaLanchor+λrLres; where Lseg is the point cloud semantic segmentation loss, Ltemp is the temporal visual supervision loss, Lanchor is the prediction preservation loss, Lres is the residual regularization loss that restricts the magnitude of the semantic residual or the residual adaptation branch parameters, and λt, λa and λr are the corresponding loss weights.

8. The semantic segmentation method for vehicle-mounted LiDAR point clouds based on temporal visual supervision according to claim 1, characterized in that, In step S7, the current lidar point cloud is the only sensor input for the inference stage, and the network structure retained in the inference stage depends only on the current lidar point cloud or the point cloud features generated from it.

Images