Random domain comparative learning-based severe weather point cloud tracking method

Abstract

The invention discloses a severe weather point cloud tracking method based on random field comparative learning, which aims at three-dimensional point cloud visual data acquired by a LiDAR sensor and aims at how to learn robust feature representation of target perception in a point cloud tracking task in an actual severe weather scene. And the method shows more accurate tracking performance under different weathers. According to the method, various adverse conditions are simulated through point cloud domain randomization, and an enhancement strategy from simulation data to real data generalization is given; secondly, introducing a dual-twinborn random domain contrast learning mode into a tracking task, and simultaneously performing feature tuning on neural networks of a main branch and an auxiliary branch by adopting a mutual learning mode between a normal sample and an adversarial sample; and finally, designing a constraint for performing comparative learning on local geometry of the point cloud so as to strengthen the consistency of geometric features, so that the tracker can keep effective on a normal branch and a weather interference branch.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional visual perception technology in artificial intelligence and computer vision, and specifically relates to a severe weather point cloud tracking method based on random field contrast learning. Background Art

[0002] Three-dimensional target tracking using LiDAR (Light Detection and Ranging) is one of the most important tasks in understanding and perceiving complex dynamic scenes in the intelligent process. By accurately acquiring three-dimensional spatial information, it gives intelligent terminals a new pair of machine eyes. This sensor is widely used in intelligent terminals or platforms such as rail transit, civil aviation, road traffic, and robotics. However, in actual application scenarios, due to the complex and changing road conditions and weather conditions, the current deep learning-based three-dimensional point cloud tracking method (3DSOT) still has shortcomings when dealing with such scenarios with changing environmental perception. Considering the important factor of safety and reliability in autonomous unmanned systems, research on more robust point cloud tracking methods can effectively meet practical needs.

[0003] Research on deep learning-based point cloud tracking has achieved remarkable results and can be categorized into two categories: appearance similarity modeling-based and motion estimation-based. Appearance similarity-based methods utilize Siamese networks to extract shape features from the target template and search region. They then design various relationship modeling methods to highlight the characteristics of the tracked object. Finally, candidate bounding boxes are predicted based on keypoint clustering or bird's-eye view representations. The SC3D method pioneered the use of PointNet networks to learn appearance shape matching. Subsequently, the P2B family of methods introduced the PointNet++ feature embedding framework and utilized deep Hough voting to predict candidate bounding boxes. Furthermore, researchers have further improved tracking performance by incorporating techniques such as Transformer networks, graph networks, geometric size priors, spatiotemporal information, and bird's-eye view assistance. Motion estimation-based methods leverage information from multiple or a single historical frame to directly predict the approximate target center in the current frame and use this center as motion prior knowledge to optimize the final bounding box prediction. MMTracker utilizes historical information from a single frame to first segment potential targets within the search region and then fuse this information with the template to estimate the motion offset between the two frames. DMT only uses multi-frame historical information in the motion prediction module to directly predict the potential target center, and then combines it with the deep Hough voting of P2B to output the target bounding box.

[0004] Current LiDAR point cloud tracking methods (3DSOT) focus on the design of network structures or modules, such as feature matching, motion estimation guidance, and bird's-eye view assistance. These methods are primarily targeted at scenarios where the point cloud distribution varies little between the training and test domains. While their tracking accuracy and success rate metrics are increasing, they lack consideration for the model's safety and reliability under adverse conditions. In the open world, LiDAR point clouds face the threat of out-of-distribution (OOD) data changes, which can lead to erroneous perception results. These OOD data threats can be summarized into three aspects: first, the threat of unobserved data, such as transitions from simulation to real data, from known to unknown classes, and from familiar to unfamiliar cities; second, the threat of adversarial attack samples, whose interference is difficult to perceive but can mislead the trained model into outputting erroneous results; and third, the threat of changes in the perceptual environment, such as inclement weather such as rain, snow, fog, and differences in device resolution. These three aspects are currently under urgent scrutiny in point cloud tracking research. Given the significant challenges of addressing these three issues with a unified approach, this paper focuses on the threats posed by changes in the perceptual environment.

[0005] The information disclosed in this background technology section is only intended to enhance understanding of the overall background of the invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the Invention

[0006] The purpose of the present invention is to provide a point cloud tracking method for severe weather based on random domain contrast learning. The current 3DSOT method is not robust enough to handle changes in the environmental domain. This invention focuses on the three-dimensional point cloud visual data obtained by the LiDAR sensor and focuses on actual severe weather scenarios. It studies how to learn robust feature representations of target perception in point cloud tracking tasks, so that it can show more accurate tracking performance under different weather conditions.

[0007] To address the lack of point cloud tracking data in inclement weather, this paper generates a simulated tracking dataset based on the physical properties of rain, snow, and fog. It also incorporates domain randomization technology for data augmentation, providing an enhancement strategy that generalizes from simulated data to real data. To address the lack of robustness to inclement weather conditions in tracking scenarios, this paper utilizes normal samples and generated adversarial samples to design a contrastive learning network structure and proposes a mutual learning mechanism at the level of local geometric features to improve the network's robustness in complex environments.

[0008] In order to achieve the above object, a specific embodiment of the present invention provides the following technical solutions:

[0009] A method for tracking point clouds in severe weather conditions based on random domain contrastive learning includes two modules: point cloud domain randomization and local geometric contrastive learning. The point cloud domain randomization simulates various adverse conditions to enhance the model's anti-interference ability, while the local geometric contrastive learning strengthens the consistency of local geometric features to improve the network's robustness in complex environments. The point cloud tracking method includes the following steps:

[0010] Step S1: Generate data using the physical properties of rain, snow and fog, and use domain randomization technology to enhance the data and generate adversarial samples;

[0011] Step S2: The main branch inputs the normal sample for forward propagation, and outputs the sampled point cloud and its corresponding features in the middle layer of the network;

[0012] Step S3: The auxiliary branch inputs random domain samples for forward propagation, and outputs the sampled point cloud and corresponding features in the middle layer of the network;

[0013] Step S4: Through local geometric contrast learning, the main branch and auxiliary branch learn each other’s representation capabilities under normal samples and adversarial samples;

[0014] Step S5: The dual-branch networks of steps S2 and S3 are used to embed the target template and search area information respectively to generate target perception features, and then regress the respective confidence scores, 3D bounding box center coordinates, and yaw angles.

[0015] In one or more embodiments of the present invention, the point cloud domain randomization module includes data construction, which is auxiliary training data construction for severe weather conditions. This data construction includes data generation and data augmentation. Given the high cost of 3D data acquisition and annotation, and the challenges of learning a relatively general feature representation network from scratch based solely on unordered point clouds, the present invention addresses both data generation and data augmentation.

[0016] In one or more embodiments of the present invention, the data generation is simulation data generation based on the physical properties of rain, snow, and fog. The data generation uses an existing tracking data set as a basis to simulate the reflection and refraction changes of the laser radar beam of the sensor under various weather conditions for each frame of point cloud. Specifically, the data generation method is:

[0017] Step 1: Select KITTI and nuScenes as the basis, where the resolution of KITTI is 64 beams and the resolution of nuScenes is 32 beams;

[0018] Step 2: Add fog, rain and snow weather simulation to the data in step 1.

[0019] For simulation data generation, we plan to use existing tracking datasets as a basis, and then simulate the changes in reflection and refraction of the sensor's lidar beam under various weather conditions for each frame of the point cloud. Specifically, considering the device resolution to cover a variety of data samples, we plan to use KITTI and nuScenes as the basis, with KITTI's resolution being 64 beams and nuScenes' being 32 beams. Furthermore, we will add fog, rain, and snow weather simulations to the data. For rainy and snowy weather, we simulate different degrees of rain and snow by setting parameters such as the rainfall rate and snowfall rate based on the LISA method. For foggy days, we control the parameters of the meteorological optical range to achieve generation from light fog to dense fog. In this way, the generated tracking samples are 15 times the number of the original dataset. This is because we simulated 15 weather conditions for each category in each dataset, i.e., 5 levels of 3 types of weather.

[0020] In one or more embodiments of the present invention, the data enhancement is performed in combination with domain randomization technology. The domain randomization technology is formed by combining multiple basic enhancement means. The multiple technical enhancement means include adding noise, deleting points, and point perturbations. The point cloud representation after the data enhancement is as follows:

[0021]

[0022] Where P represents the input point cloud; P noise represents the added noise point set; P dropout represents the set of points to be removed; and δ jittering represents the point-by-point perturbation.

[0023] In one or more embodiments of the present invention, the entire process of the point cloud representation after data enhancement is shown in Algorithm 1, which is a domain randomized data enhancement, specifically:

[0024] Input: Point cloud

[0025] Output: Enhanced point cloud

[0026]

[0027]

[0028] In one or more embodiments of the present invention, the local geometric contrast learning module includes a contrast learning framework for point cloud tracking, local geometric mutual learning and relational modeling, the local geometric contrast learning adopts MBPTrack as the benchmark tracking network, and the local geometric contrast learning adopts twin branches for contrast learning, the twin branches include a main branch and an auxiliary branch, the main branch takes point cloud data under normal weather as input and outputs the sampled point cloud and corresponding features in the middle layer of the network; the auxiliary branch randomly selects enhanced data or generated simulation data as input and outputs the sampled point cloud and corresponding features in the middle layer.

[0029] In one or more embodiments of the present invention, the contrastive learning framework for point cloud tracking is constructed based on a feature extraction network of spatial points. The contrastive learning framework for point cloud tracking is a dual twin network structure, that is, an MBPTrack feature extraction process containing two branches, each of which is a twin sub-network based on template and search area input. The twin sub-network is denoted as:

[0030] B=ψ (o) (χ (o) (f (o) (P T ),f (o) (P S ))),o∈{master,assist}

[0031] Among them, o takes the value of master or assist, representing the main branch and auxiliary branch respectively, f represents the feature extraction network, which is generally DGCNN or PointNet++; χ is the relationship model between the target to be tracked and the search area, which is generally intended to embed the target's discriminant information into the search area; ψ is the target perception positioning module, which outputs the final response map and the corresponding target center and yaw angle offset;

[0032] The trunk branches are represented as follows:

[0033]

[0034] The auxiliary branches are represented as follows:

[0035]

[0036] Among them, P T The weather-resistant point cloud of the target template, P T is the point cloud of the target template under normal weather conditions; P S and P S It represents the adversarial weather and normal weather point cloud data of the search area.

[0037] In one or more embodiments of the present invention, the process of local geometric mutual learning is as follows:

[0038] Input point cloud P T and After the feature encoding of the backbone network, the feature F is obtained T and And the downsampled point cloud and The points in are regarded as the core point set Q in local geometric mutual learning, and let Q i Indicates its i-th key point, through the ball query Find its K neighboring points Then the local geometric features are aggregated through the maximum pooling operation This process can be expressed as:

[0039]

[0040] For normal weather point cloud Extract features for trunk branches Considering both the target template and the search area, the training target loss of local geometric mutual learning can be expressed as:

[0041]

[0042] In one or more embodiments of the present invention, the relational modeling follows the feature propagation module in MBPTrack, which decouples geometric features and target mask features into two independent propagation branches. The relational modeling is supervised and trained using the following loss function:

[0043]

[0044] where o∈{master,assist},l mask 、l quality and l score They are respectively the foreground mask confidence, the center quality score and the 3D bounding box confidence, which are predicted with the help of the cross entropy function, l center The mean square error function is used to estimate the center offset, center coordinates and yaw angle (i.e. l box )Use Huber loss function;

[0045] Finally, consider the loss function of the two branches and The loss of the target estimate of the entire neural network is

[0046]

[0047] Among them, the parameters λ1=0.2, λ2=10, λ3=λ4=λ5=1, and β=1.

[0048] In one or more embodiments of the present invention, in the N frames tested, ρ represents the proportion of frames in which the center error c between the bounding box estimated by the tracking model and the true bounding box is less than a given threshold τ to the total number of frames, and for M different thresholds τ i Find the average:

[0049]

[0050] Where σ represents the proportion of frames in which the volume intersection ratio v between the bounding box estimated by the tracking model and the true bounding box is greater than a given threshold τ to the total number of frames, and for M different thresholds τ i Find the average:

[0051]

[0052] In order to reflect the performance degradation of the tracker under severe weather conditions, the decay rate d is used as the robustness indicator:

[0053]

[0054] Where S∈{ρ,σ}, which measures the degree of degradation of the accuracy and success rate of the algorithm under adverse conditions; S * Represents performance indicators under normal weather conditions.

[0055] Compared with the existing technology, the present invention simulates various adverse conditions through point cloud domain randomization, and provides an enhancement strategy for generalization from simulated data to real data. Secondly, a dual-twin random domain contrastive learning method is introduced in the tracking task, and the mutual learning between normal samples and adversarial samples is used to simultaneously tune the features of the neural networks of the main branch and the auxiliary branch. Finally, constraints are designed for contrastive learning on the local geometry of the point cloud to strengthen the consistency of geometric features, so that the tracker can maintain effectiveness on both the normal branch and the weather interference branch. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 This is a diagram summarizing a method for tracking severe weather point clouds based on random field contrast learning in one embodiment of the present invention;

[0058] Figure 2 Schematic diagram of LiDAR point cloud perception affected by environmental changes in one embodiment of the present invention;

[0059] Figure 3 Flowchart of a contrastive learning framework based on random fields in one embodiment of the present invention;

[0060] Figure 4 This is a diagram showing the tracking effect in severe weather conditions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0062] like Figure 1 As shown, in one embodiment of the present invention, a method for tracking point clouds in severe weather based on random domain contrastive learning is provided. Currently, most of the data samples for point cloud target tracking are collected under normal weather conditions, but data for time series tracking of point clouds in severe weather conditions is still lacking, and the relevant algorithms have not been fully studied. In the present invention, the impact of different weather conditions and different intensities on the algorithm is taken into account. First, we generate a simulated tracking dataset based on the physical properties of rain, snow and fog, and at the same time combine domain randomization technology to perform data enhancement. We use the two as adversarial samples and provide an enhancement strategy that generalizes from simulated data to real data. Secondly, the representation ability of features greatly affects the tracking performance of the algorithm in the face of severe weather. In order to address the problem of insufficient robustness caused by changes in point cloud distribution in tracking scenarios, the present invention utilizes normal samples and adversarial samples, and based on the network structure of contrastive learning, designs a method for adapting normal weather to severe weather without losing the accuracy in normal weather.

[0063] Specifically, a method for tracking point clouds in severe weather conditions based on random domain contrastive learning includes two modules: point cloud domain randomization and local geometric contrastive learning. Point cloud domain randomization simulates various adverse conditions to enhance the model's anti-interference ability, while local geometric contrastive learning strengthens the consistency of local geometric features to improve the network's robustness in complex environments. The point cloud tracking method includes the following steps:

[0064] Step S1: Generate data using the physical properties of rain, snow and fog, and use domain randomization technology to enhance the data and generate adversarial samples;

[0065] Step S2: The main branch inputs the normal sample for forward propagation, and outputs the sampled point cloud and its corresponding features in the middle layer of the network;

[0066] Step S3: The auxiliary branch inputs random domain samples for forward propagation, and outputs the sampled point cloud and corresponding features in the middle layer of the network;

[0067] Step S4: Through local geometric contrast learning, the main branch and auxiliary branch learn each other’s representation capabilities under normal samples and adversarial samples;

[0068] Step S5: The dual-branch networks of steps S2 and S3 are used to embed the target template and search area information respectively to generate target perception features, and then regress the respective confidence scores, 3D bounding box center coordinates, and yaw angles.

[0069] Example 1: Construction of auxiliary training data for severe weather

[0070] Inclement weather conditions are a common occurrence in the perceptual world. Currently, all 3D point cloud object tracking methods are evaluated on LiDAR datasets for normal weather conditions, such as KITTI and nuScenes. While existing datasets capture scenes in inclement weather, most lack object temporal information or have inconsistent target IDs across tracking sequences, making them unsuitable for object tracking tasks.

[0071] Specifically, the point cloud domain randomization module includes data construction, which is auxiliary training data for severe weather conditions. This data construction includes data generation and data augmentation. Given the high cost of 3D data acquisition and annotation, and the challenges of learning a more general feature representation network from scratch based solely on unordered point clouds, this paper addresses both data generation and data augmentation.

[0072] like Figure 2 As shown in the figure, data generation is based on the physical characteristics of rain, snow and fog. The data generation uses the existing tracking data set as the basis to simulate the reflection and refraction changes of the lidar beam of the sensor under various weather conditions for each frame of point cloud. Specifically, the data generation method is as follows:

[0073] Step 1: Select KITTI and nuScenes as the basis, where the resolution of KITTI is 64 beams and the resolution of nuScenes is 32 beams;

[0074] Step 2: Add fog, rain and snow weather simulation to the data in step 1.

[0075] For simulation data generation, we plan to use existing tracking datasets as a basis, and then simulate the changes in reflection and refraction of the sensor's lidar beam under various weather conditions for each frame of the point cloud. Specifically, considering the device resolution to cover a variety of data samples, we plan to use KITTI and nuScenes as the basis, with KITTI's resolution being 64 beams and nuScenes' being 32 beams. Furthermore, we will add fog, rain, and snow weather simulations to the data. For rainy and snowy weather, we simulate different degrees of rain and snow by setting parameters such as the rainfall rate and snowfall rate based on the LISA method. For foggy days, we control the parameters of the meteorological optical range to achieve generation from light fog to dense fog. In this way, the generated tracking samples are 15 times the number of the original dataset. This is because we simulated 15 weather conditions for each category in each dataset, i.e., 5 levels of 3 types of weather.

[0076] Domain randomization is an effective technique for training models to enhance robustness and generalization. Randomization can create complex and varied samples, which in turn improves the ability to represent complex samples. Data augmentation combines domain randomization with data augmentation. Domain randomization is a composite of multiple basic augmentation techniques, including adding noise, deleting points, and perturbing points. The point cloud representation after data augmentation is:

[0077]

[0078] Among them, P noise represents the added noise point set; P dropout represents the set of points to be removed; and δ jittering represents the point-wise perturbation amount.

[0079] Specifically, the entire process of point cloud representation after data enhancement is shown in Algorithm 1. Algorithm 1 is domain randomized data enhancement, specifically:

[0080] Input: Point cloud

[0081] Output: Enhanced point cloud

[0082]

[0083]

[0084] Example 2: Target State Estimation Based on Random Field Contrastive Learning

[0085] Point cloud tracking methods have made considerable progress in conventional scenarios. However, even for the same scene, different weather conditions can result in different point cloud features calculated by neural networks. Considering the challenges of universal feature representation for point clouds and the inconsistent features caused by distribution variations, this paper uses adversarial examples generated above to perform comparative learning with normal examples, thereby improving the robustness of tracking in adverse weather conditions.

[0086] In one or more embodiments of the present invention, the local geometric contrast learning module includes a contrast learning framework for point cloud tracking, local geometric mutual learning and relationship modeling. The local geometric contrast learning adopts MBPTrack as the benchmark tracking network, and the local geometric contrast learning adopts twin branches for contrast learning. The twin branches include a main branch and an auxiliary branch. The main branch takes point cloud data under normal weather as input and outputs the sampled point cloud and corresponding features in the middle layer of the network; the auxiliary branch randomly selects enhanced data or generated simulation data as input and outputs the sampled point cloud and corresponding features in the middle layer.

[0087] Specifically, given MBPTrack's excellent performance in normal weather conditions, we used it as a baseline tracking network and employed a dual-twin branch for comparative learning. The entire process consists of two branches: a main branch and an auxiliary branch. The main branch takes point cloud data from normal weather conditions as input and outputs a downsampled point cloud and corresponding features at the network's intermediate layer. The auxiliary branch randomly selects augmented data or generated simulated data as input and similarly outputs a downsampled point cloud and corresponding features at the intermediate layer. Through local geometric comparative learning, the main and auxiliary branches learn from each other at the feature level. Finally, the dual-branch network models the relationship between the target template and the search area, generating target-aware features, and then regresses the respective confidence scores, 3D bounding box center coordinates, and yaw angles.

[0088] Most contrastive learning methods focus on spatially aligned representations, typically using bird's-eye view (BEV) features or voxel features for contrastive learning. However, for tracking tasks, backbone networks based on native spatial points currently perform better. We consider building a contrastive learning framework using a feature extraction network based on spatial points.

[0089] Specifically, the contrastive learning framework for point cloud tracking is built based on the feature extraction network of spatial points. The contrastive learning framework for point cloud tracking is a twin network structure, that is, an MBPTrack feature extraction process with two branches. Each branch is a twin network based on the template and search area input. The twin network is denoted as:

[0090] B=ψ (o) (x (o) (f(o) (P T ),f (o) (P S ))),o∈{master,assist}

[0091] Among them, o takes the value of master or assist, representing the main branch and auxiliary branch respectively, f represents the feature extraction network, which is generally DGCNN or PointNet++; χ is the relationship model between the target to be tracked and the search area, which is generally intended to embed the target's discriminant information into the search area; ψ is the target perception positioning module, which outputs the final response map and the corresponding target center and yaw angle offset;

[0092] The trunk branches are represented as follows:

[0093]

[0094] The auxiliary branches are represented as follows:

[0095]

[0096] Among them, P T The weather-resistant point cloud of the target template, P T is the point cloud of the target template under normal weather conditions; P S and P S It represents the adversarial weather and normal weather point cloud data of the search area.

[0097] After constructing the twin contrastive learning framework, the main branch needs to be able to understand both normal and adversarial weather. Therefore, it needs to balance the feature differences between its own normal samples and the adversarial samples contained in the auxiliary branch. However, due to differences in point number and distribution, the point clouds of the two branches often lack a one-to-one correspondence, making point-by-point feature mutual learning challenging. Therefore, we propose a local geometric mutual learning module to address this issue. It aligns the neighborhood features of the two branch point clouds.

[0098] Specifically, such as Figure 3 As shown in Figure 2, the process of local geometric mutual learning is as follows:

[0099] Taking the target template as an example, input point cloud P T and After the feature encoding of the backbone network, we obtain the feature F T and And the downsampled point cloud and First, we will The points in are regarded as the core point set Q in local geometric mutual learning, and let Qi Indicates its i-th key point. We can query the ball in Find its K neighbors Then the local geometric features are aggregated through the maximum pooling operation This process can be expressed as:

[0100]

[0101] Similarly, for point clouds in normal weather We can also extract features for trunk branches Finally, considering both the target template and the search area, the training target loss of local geometric mutual learning can be expressed as:

[0102]

[0103] After learning the feature representation, we need to model the relationship between the target template and the search region to estimate the target state. This utilizes the feature propagation module from MBPTrack, which is based on the Transformer model. It extracts spatial and temporal context from historical frames and propagates it to the current frame to assist in target tracking.

[0104] The feature propagation module (i.e., χ) decouples the geometric features and the target mask features into two independent propagation branches to avoid the loss of geometric information during deep propagation. In addition, this module also designs a memory mechanism that stores historical frames and their target masks. During the propagation process, its input is the geometric features of the current frame, the geometric features from the memory, and the mask features. Then, through cross-frame attention operations and self-attention operations, the information in the historical frames is propagated to the current frame. For the F in the contrastive learning framework of this invention, S and Let pass through this module in turn to get and

[0105] Finally, the search region features embedded with target information are fed into the Region Proposal Network (RPN) to regress the foreground mask confidence, center offset, center quality, 3D bounding box confidence, center (x, y, z) and yaw angle θ, and supervised training is performed using the following loss function:

[0106]

[0107] where o∈{master,assist},l mask 、l quality and l scoreThey are respectively the foreground mask confidence, the center quality score and the 3D bounding box confidence, which are predicted with the help of the cross entropy function, l center The mean square error function is used to estimate the center offset, center coordinates and yaw angle (i.e. l box )Use Huber loss function;

[0108] Finally, consider the loss function of the two branches and The loss of the target estimate of the entire neural network is

[0109]

[0110] Among them, the parameters λ1=0.2, λ2=10, λ3=λ4=λ5=1, β=1.

[0111] Example 3: Method Evaluation

[0112] like Figure 4 As shown, there are two key indicators in target tracking: precision rate ρ and success rate σ. In the N frames tested, ρ represents the proportion of frames in which the center error c between the bounding box estimated by the tracking model and the true bounding box is less than a given threshold τ, and the success rate is σ for M different thresholds τ. i Find the average:

[0113]

[0114] Where σ represents the proportion of frames in which the volume intersection ratio v between the bounding box estimated by the tracking model and the true bounding box is greater than a given threshold τ to the total number of frames, and for M different thresholds τ i Find the average:

[0115]

[0116] In order to reflect the performance degradation of the tracker under severe weather conditions, the decay rate d is used as the robustness indicator:

[0117]

[0118] Where S∈{ρ,σ}, which measures the degree of degradation of the accuracy and success rate of the algorithm under adverse conditions; S * Represents performance indicators under normal weather conditions.

[0119] Table 1 Severe weather test results

[0120]

[0121]

[0122] Table 1 and Figure 4The indicator measurement results and visualization results are given respectively.

[0123] Example 4: Training and testing phases

[0124] The technical solution of the present invention will be further described below in conjunction with specific embodiments and drawings.

[0125] A method for tracking severe weather point clouds based on random field contrastive learning includes training and testing phases. The steps are as follows:

[0126] Phase P1: Training process

[0127] Steps P1-S1: Data generation and data enhancement to generate adversarial samples

[0128] 1) For simulation data generation, for rainy and snowy weather scenarios, we simulate varying degrees of rain and snow by setting parameters such as rainfall and snowfall rates based on the LISA method. For foggy days, we manipulate the parameters of the meteorological optical range to achieve fog generation from light to dense. This method generates 15 times more tracking samples than the original dataset.

[0129] 2) For data enhancement, we use domain randomization, which is a combination of three basic enhancement methods: adding noise, deleting points, and perturbation. The specific implementation is based on the process of Algorithm 1.

[0130] Steps P1-S2: Trunk-branch forward propagation

[0131] 1) Take the target template point cloud under normal weather conditions as input, and output the downsampled point cloud and its corresponding features in the middle layer of the DGCNN network:

[0132] 2) Take the search area point cloud under normal weather conditions as input, and output the sampled point cloud and its corresponding features in the middle layer of the DGCNN network:

[0133] Steps P1-S3: Auxiliary branch forward propagation

[0134] 1) Take the target template point cloud under the adversarial weather as input, and output the sampled point cloud and its corresponding features in the middle layer of the DGCNN network:

[0135] 2) Take the search area point cloud under the adversarial weather as input, and output the sampled point cloud and its corresponding features in the middle layer of the DGCNN network:

[0136] Steps P1-S4: Local geometric contrastive learning

[0137] 1) The points in are regarded as the core point set Q in local geometric mutual learning, and let Q i represents its i-th key point.

[0138] 2) Query by ball Find its K neighbors Then the local geometric features are aggregated through the maximum pooling operation

[0139] 3) Similarly, for steps 1) and 2) of steps P1-S4, and Perform the same operation as above and get and

[0140] 4) The main branch and auxiliary branch learn from each other at the feature level and calculate

[0141] Step P1-S5: The dual-branch networks of steps P1-S2 and P1-S3 embed the target template and search area information respectively to generate target perception features, and then regress the respective confidence scores, 3D bounding box center coordinates, and yaw angles.

[0142] 1) For the backbone network, the feature F generated by steps P1-S2 S Send it to the feature propagation module and calculate it

[0143] 2) The above features The region candidate network is fed into the regression to obtain the foreground mask confidence, center offset, center quality, 3D bounding box confidence, center (x, y, z) and yaw angle θ, and the following loss function is calculated:

[0144] 3) Similarly, for the auxiliary branch network, calculate and loss function

[0145] 4) Calculate the final loss And back propagate to update the network parameters.

[0146] Phase P2: Testing Process

[0147] Step P2-S1: Trunk-branch forward propagation

[0148] 1) Take the target template point cloud in any scene as input, and output the sampled point cloud and corresponding features in the middle layer of the DGCNN network:

[0149] 2) Take the search area point cloud in any scene as input, and output the sampled point cloud and corresponding features in the middle layer of the DGCNN network:

[0150] Step P2-S2: Based on the features calculated by the network in step P2-S1, the target template and search area information are embedded respectively to generate target perception features, and then the respective confidence scores, 3D bounding box center point coordinates and yaw angles are regressed.

[0151] 1) The feature F generated in step P2-S1 S Send it to the feature propagation module and calculate it

[0152] 2) The above features The region candidate network is fed into the regression to obtain the 3D bounding box confidence, the bounding box center (x, y, z) and the yaw angle θ.

[0153] According to the predicted 3D bounding box confidence, the center and yaw angle corresponding to the candidate point with the highest confidence are selected to track the target in the current frame.

[0154] Step P2-S3: Track the time sequence frames sequentially. For the next frame input, execute steps P2-S1 and P2-S2 until the last frame of the tracking sequence is reached.

[0155] Example 4 is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

[0156] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0157] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for tracking severe weather point clouds based on random field contrastive learning, characterized in that: The method comprises two modules: point cloud domain randomization and local geometric contrast learning. The point cloud domain randomization simulates various adverse conditions to enhance the anti-interference ability of the model. The local geometric contrast learning enhances the consistency of local geometric features to improve the robustness of the network in complex environments. The point cloud tracking method comprises the following steps: Step S1: Generate data using the physical properties of rain, snow and fog, and use domain randomization technology to enhance the data and generate adversarial samples; Step S2: The trunk branch inputs normal samples for forward propagation, and outputs the sampled point cloud and the corresponding features in the middle layer of the network; Step S3: The auxiliary branch inputs random domain samples for forward propagation, and outputs the sampled point cloud and the corresponding features in the middle layer of the network; Step S4: Through local geometric contrast learning, the main branch and the auxiliary branch learn each other's representation capabilities under normal samples and adversarial samples; Step S5: The dual-branch networks of steps S2 and S3 are used to embed the information of the target template and the search area respectively to generate target-aware features, and then regress the respective confidence scores, coordinates of the center point of the three-dimensional bounding box, and yaw angles.

2. The method for tracking bad weather point cloud based on random field contrastive learning according to claim 1, characterized in that: The point cloud domain randomization module includes data construction, which is auxiliary training data construction for severe weather, and the data construction includes data generation and data enhancement.

3. The method for tracking bad weather point cloud based on random field contrastive learning according to claim 2, characterized in that: The data generation is simulation data generation based on the physical characteristics of rain, snow and fog. The data generation uses the existing tracking data set as a basis to simulate the reflection and refraction changes of the laser radar beam of the sensor under various weather conditions for each frame of point cloud; specifically, the data generation method is: Step 1: Select KITTI and nuScenes as the basis, where the resolution of KITTI is 64 beams and the resolution of nuScenes is 32 beams; Step 2: Add fog, rain and snow weather simulation to the data in step 1.

4. The method for tracking bad weather point cloud based on random field contrastive learning according to claim 3, characterized in that: The data enhancement is performed by combining domain randomization technology. The domain randomization technology is compounded by multiple basic enhancement means. The multiple technical enhancement means include adding noise, deleting points and point perturbations. The point cloud representation after data enhancement is as follows: Where P represents the input point cloud; P noise represents the set of added noise points; P dropout represents the set of points to be removed; and δ jittering represents the point-wise perturbation amount.

5. The method for tracking bad weather point cloud based on random field contrastive learning according to claim 4, characterized in that: The whole process of the point cloud representation after data enhancement is shown in Algorithm 1, which is a domain randomized data enhancement, specifically:

6. The method for tracking bad weather point cloud based on random field contrastive learning according to claim 1, characterized in that: The local geometric contrast learning module includes a contrast learning framework for point cloud tracking, local geometric mutual learning and relationship modeling. The local geometric contrast learning adopts MBPTrack as a benchmark tracking network, and the local geometric contrast learning adopts double twin branches for contrast learning. The double twin branches include a trunk branch and an auxiliary branch. The trunk branch takes point cloud data under normal weather as input and outputs sampled point clouds and corresponding features in the middle layer of the network; the auxiliary branch randomly selects enhanced data or generated simulation data as input and outputs sampled point clouds and corresponding features in the middle layer.

7. The method for tracking bad weather point cloud based on random field contrastive learning according to claim 6, characterized in that: The contrastive learning framework for point cloud tracking is constructed based on the feature extraction network of spatial points. The contrastive learning framework for point cloud tracking is a dual twin network structure, that is, an MBPTrack feature extraction process containing two branches, each branch is a twin sub-network based on template and search area input, and the twin sub-network is recorded as: B=ψ (o) (x (o) (f (o) (P T ),f (o) (P S ))),o∈{master,assist} Among them, o is master or assist, representing the main branch and auxiliary branch respectively, f represents the feature extraction network, which is generally DGCNN or PointNet++; χ is the relationship model between the target to be tracked and the search area, which is generally intended to embed the target's discrimination information into the search area; ψ is the target perception positioning module, which outputs the final response map and the corresponding target center and yaw angle offset; The main branches are represented as follows: The auxiliary branches are represented as follows: Where P T is the weather-resistant point cloud of the target template, P T is the point cloud of the target template under normal weather conditions; P S and P S It represents the adversarial weather and normal weather point cloud data of the search area.

8. The method for tracking bad weather point cloud based on random field contrastive learning according to claim 7, characterized in that: The process of local geometric mutual learning is as follows: Input point cloud P T and After the feature encoding of the backbone network, the feature F is obtained. T and And the downsampled point cloud and The points in are regarded as the core point set Q in local geometric mutual learning, and Q i Represents its i-th key point, through the ball query in Find its K neighbors Then the local geometric features are aggregated through the maximum pooling operation. This process can be described as: Point cloud for normal weather Extract features for trunk branches Considering both the target template and the search area, the training target loss of local geometric mutual learning can be expressed as:

9. The method for tracking bad weather point cloud based on random field contrastive learning according to claim 8, characterized in that: The relationship modeling follows the feature propagation module in MBPTrack, which decouples geometric features and target mask features into two independent propagation branches. The relationship modeling is supervised and trained using the following loss function: where o∈{master,assist},l mask , l quality and l score They are foreground mask confidence, center quality score and 3D bounding box confidence, which are predicted with the help of cross entropy function, l center The center offset is estimated using the mean square error function, and the center coordinates and yaw angle use the Huber loss function; Finally, consider the loss function of the two branches and The loss of the target estimate of the entire neural network is Among them, the parameters λ1=0.2, λ2=10, λ3=λ4=λ5=1, β=1.

10. The method for tracking bad weather point cloud based on random field contrastive learning according to claim 1, characterized in that: In the N frames tested, ρ represents the proportion of frames in which the center error c between the bounding box estimated by the tracking model and the true bounding box is less than a given threshold τ to the total number of frames, and for M different thresholds τ i Find the average: Where σ represents the proportion of frames whose volume intersection ratio v between the bounding box estimated by the tracking model and the true bounding box is greater than a given threshold τ to the total number of frames, and i Find the average: In order to reflect the performance degradation of the tracker under severe weather conditions, the decay rate d is proposed to be used as the robustness indicator: where S∈{ρ,σ} measures the degradation of the accuracy and success rate of the algorithm under adverse conditions; S * Represents performance indicators under normal weather conditions.

Images