An abnormal environment perception method and system based on multi-source information fusion

Abstract

The application relates to the technical field of intelligent driving and discloses an abnormal environment sensing method and system based on multi-source information fusion, which comprises the following steps: collecting vehicle surrounding environment information and constructing a millimeter wave radar characteristic vector; constructing a convolution network model to obtain a target layer fusion characteristic vector and a fusion target position; splicing to obtain a multi-source fusion characteristic vector; designing a deep network architecture, performing data layer fusion through an attention mechanism to obtain a data layer fusion characteristic vector and an abnormal environment state category probability; constructing Bayesian inference observation data, calculating an abnormal environment state posterior probability, outputting an abnormal environment sensing result, and adjusting parameters of the convolution network model and the deep network architecture based on the abnormal environment state posterior probability. The application effectively breaks through the limitation of a single sensing source, significantly improves the sensing accuracy and reliability of an autonomous driving vehicle on an abnormal environment, and better meets the safe driving demand of autonomous driving in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to an abnormal environment perception method and system based on multi-source information fusion. Background Technology

[0002] With the rapid development of autonomous driving technology, ensuring accurate perception in complex environments has become a critical issue that urgently needs to be addressed. Currently, most autonomous vehicles use a single sensor for environmental perception, but this approach has many limitations.

[0003] On the one hand, visual sensors suffer from severe image quality degradation in poor lighting conditions (such as at night or under direct sunlight) and adverse weather conditions (such as heavy rain or fog), leading to deviations in the recognition and judgment of objects in the environment and affecting the vehicle's accurate perception of its surroundings. For example, in dim lighting at night, visual sensors may not be able to clearly distinguish obstacles on the road, thus posing a threat to driving safety.

[0004] On the other hand, while lidar can accurately measure distance and object outlines, it has a weaker ability to perceive information such as color and texture. Furthermore, when facing certain special materials (such as glass and mirrors), the reflected signal may be abnormal, leading to misjudgments. Although millimeter-wave radar has strong anti-interference capabilities and penetration, its resolution is relatively low, making it difficult to accurately identify small targets or accurately determine the specific shape and details of targets. In complex traffic scenarios, it may not provide sufficiently accurate information.

[0005] The limitations of these single sensors prevent autonomous vehicles from fully and accurately perceiving their surroundings in complex and ever-changing real-world road environments, severely hindering the development and application of autonomous driving technology. Especially in special scenarios, such as construction zones or accident scenes, the inadequacy of a single sensor may prevent vehicles from detecting potential hazards in a timely manner, potentially leading to safety accidents.

[0006] To overcome these challenges and enhance the perception capabilities and safety of autonomous vehicles in various environments, researching anomaly perception technology based on multi-source information fusion is crucial. By integrating information from multiple sensors, such as visual sensors, LiDAR, and millimeter-wave radar, and fully leveraging the strengths of each sensor to achieve information complementarity, autonomous vehicles can be provided with more comprehensive and accurate environmental perception, meeting their needs for safe driving in complex scenarios. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a method and system for abnormal environment perception based on multi-source information fusion.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: In a first aspect, the present invention provides an abnormal environment perception method based on multi-source information fusion, comprising: The system simultaneously collects information about the vehicle's surrounding environment using visual sensors, lidar, and millimeter-wave radar to acquire images, lidar point cloud data, and millimeter-wave radar data, and then uses a positioning system to acquire the vehicle's three-dimensional position. With heading angle Constructing millimeter-wave radar feature vectors based on millimeter-wave radar data ; Construct the first convolutional network model to process the image and output visual feature vectors. A second convolutional network model is constructed to process the laser point cloud data and output the lidar feature vector. Semi-supervised training is used to enhance the generalization ability of the first and second convolutional network models. and Perform target layer fusion to obtain the target layer fused feature vector. and the target location of fusion ;Will , and Concatenate into a multi-source fusion feature vector ; Design orientation The deep network architecture, through the attention mechanism, , and Perform data layer fusion to obtain the data layer fused feature vector. and the probability of abnormal environmental state categories ; Combination , , , , , In addition to autonomous driving map information, Bayesian inference observation data is constructed, the posterior probability of abnormal environment states is calculated, the abnormal environment perception results are output, and the parameters of the convolutional network model and deep network architecture are adjusted based on the posterior probability of abnormal environment states.

[0009] In one embodiment, the method further includes preprocessing the image, laser point cloud data, and millimeter-wave radar data, specifically including: Median filtering is used to remove image noise from the image, voxel filtering is used to reduce the point cloud density and remove outliers from the laser point cloud data, and target range, azimuth angle, radial velocity and reflection intensity are extracted from the millimeter-wave radar data. Time synchronization is performed based on timestamp interpolation, and the coordinate systems of the vision sensor, LiDAR, and millimeter-wave radar are transformed to the vehicle coordinate system according to joint calibration parameters to obtain the preprocessed image. Laser point cloud data and millimeter-wave radar data ; The construction of millimeter-wave radar feature vectors based on millimeter-wave radar data specifically includes: based on preprocessed millimeter-wave radar data... The target range, azimuth, radial velocity, and reflection intensity are used to form the millimeter-wave radar feature vector. .

[0010] In one embodiment, the construction of the first convolutional network model processes the image and outputs a visual feature vector. A second convolutional network model is constructed to process the laser point cloud data and output the lidar feature vector. Semi-supervised training is used to enhance the generalization ability of the first and second convolutional network models. and Perform target layer fusion to obtain the target layer fused feature vector. and the target location of fusion Specifically, it includes: A first convolutional network model is constructed for the image. The input of the first convolutional network model is the preprocessed image, which is processed sequentially through convolutional layers and pooling layers, flattened, and connected to a fully connected layer to output a visual feature vector. ; A second convolutional network model is constructed for laser point cloud data. This model uses the 3D coordinate range of the preprocessed laser point cloud data as the point cloud space, divides the point cloud space into voxels, and generates voxel feature vectors from the points within each voxel through feature aggregation, forming a voxel feature map. This map is then processed through multiple convolutional, pooling, and fully connected layers to output the LiDAR feature vector. ; A semi-supervised training method is adopted to train the first convolutional network model and the second convolutional network model using labeled and unlabeled data. The total loss function of each convolutional network model is composed of a weighted sum of supervised loss and unsupervised consistency loss. The supervised loss is calculated based on cross-entropy, and the unsupervised consistency loss is calculated based on the difference between the output of the convolutional network model of the original data and the augmented data. The parameters of the convolutional network model are iteratively optimized using the stochastic gradient descent algorithm. Visual feature vectors and lidar feature vector Mapping to the same dimension yields the mapped visual feature vectors. and the mapped lidar feature vector Then, target layer weighted fusion is performed to obtain the target layer fused feature vector. : ; in, and These are the target layer fusion weights for visual feature vectors and LiDAR feature vectors, respectively. The fused target position is obtained by matching the coordinates of the visual target position with the spatial position of the lidar. .

[0011] In one embodiment, the construction of a first convolutional network model for the image involves taking a preprocessed image as input, processing it sequentially through convolutional and pooling layers, flattening it, and connecting it to a fully connected layer to output a visual feature vector. Specifically, it includes: Let the first convolutional network model be... The number of convolutional kernels in each convolutional layer is The kernel size is Step size is Fill with The height and width of the feature map output by the previous convolutional layer are respectively and Then the first The height of the output feature map of each convolutional layer and width They are respectively: ; ; in The function represents rounding down; the pooling layer uses max pooling, and the pooling window size is [value missing]. Step size is ; Flatten the feature map output from the pooling layer and connect it to... Each fully connected layer outputs a visual feature vector that describes the visual representation of a target or anomalous region in an image. ;in, The number of neurons in each fully connected layer is respectively .

[0012] In one embodiment, a second convolutional network model is constructed for the laser point cloud data. This second convolutional network model uses the three-dimensional coordinate range of the preprocessed laser point cloud data as the point cloud space, divides the point cloud space into voxels, and generates voxel feature vectors from points within each voxel through feature aggregation, forming a voxel feature map. This map is then processed through multiple convolutional, pooling, and fully connected layers to output the LiDAR feature vector. Specifically, it includes: laser point cloud data The three-dimensional coordinate range is taken as the point cloud space, and the lengths of the point cloud space in the three coordinate directions are respectively... , , The side length of the voxel is The number of voxels in the three coordinate directions are as follows: ; ; ; in, The function represents rounding up. Each point within a voxel generates a voxel feature vector through feature aggregation. Let the dimension of each voxel feature vector be 1. The size of the voxel feature map obtained after voxelization is ; , , These represent the number of voxels in the three coordinate directions, respectively. The feature dimension of each voxel is represented; convolution operations are performed on the voxel feature maps, and after multiple convolution and pooling layers, they are connected to multiple fully connected layers to output a lidar feature vector describing the spatial structure features of the target in the lidar point cloud data. .

[0013] In one embodiment, a semi-supervised training method is employed, using labeled and unlabeled data to train the first and second convolutional network models. The total loss function of each convolutional network model is composed of a weighted sum of supervised loss and unsupervised consistency loss. The supervised loss is calculated based on cross-entropy, and the unsupervised consistency loss is calculated based on the difference between the outputs of the convolutional network models on the original data and the augmented data. The parameters of the convolutional network models are iteratively optimized using a stochastic gradient descent algorithm, specifically including: For the first convolutional network model, construct a labeled image dataset. and unlabeled image datasets : ; ; in, For the first Each image has labeled samples. for The corresponding target category label or abnormal environment state category label, The number of labeled samples in the image. For the first One image without labeled samples, This represents the number of unlabeled samples in the image. First convolutional network model Output visual feature vector Classification Head Output image sample target category or abnormal environment state category probability : ; It is the softmax function; Supervised loss of the first convolutional network model Unsupervised consistency loss Total loss They are respectively: ; ; ; An index representing the target category or abnormal environment state category. This indicates the total number of target categories or abnormal environmental state categories. This indicates that the i-th image has labeled samples belonging to the k-th class with true labels. This represents the probability that a labeled sample in the i-th image is predicted to be of the k-th class. It is the F2 norm. This indicates a data augmentation operation. This is the first balance coefficient; For the second convolutional network model, construct a labeled point cloud dataset. And unlabeled point cloud datasets : ; ; in, For the first Each point cloud has labeled samples. For the corresponding target category label or abnormal environment state category label, The number of labeled samples in the point cloud. For the first A point cloud sample without labeling. This represents the number of unlabeled point cloud samples. Second convolutional network model Output lidar feature vector Classification Head Output the probability of the target category or abnormal environment state category of the point cloud sample. : ; Supervised loss for the second convolutional network model Unsupervised consistency loss Total loss They are respectively: ; ; ; in, This indicates that the i-th point cloud has a labeled sample belonging to the k-th class with a true label. This represents the probability that a labeled sample in the i-th point cloud is predicted to be of the k-th class. This is the second balance coefficient.

[0014] In one embodiment, the design is oriented towards The deep network architecture, through the attention mechanism, , and Perform data layer fusion to obtain the data layer fused feature vector. and the probability of abnormal environmental state categories Specifically, it includes: Deep network architecture The input is a multi-source fusion feature vector. ; Visual feature vector LiDAR feature vector Millimeter-wave radar feature vectors Linear mapping to the same dimension yields the mapped visual feature vectors. Mapped lidar feature vectors and mapped millimeter-wave radar feature vector : ; ; ; in, , , Linear mapping layers for visual modalities, lidar modalities, and millimeter-wave radar modalities, respectively; The multimodal feature fusion module employing an attention mechanism calculates the attention scores for each of the three modalities: ; ; ; in, , , The attention scores are for the visual modality, lidar modality, and millimeter-wave radar modality, respectively. , , All are multilayer perceptrons; Calculate the normalized attention weights for each modality based on the attention scores: ; ; ; in, , , The attention weights are for the visual modality, the lidar modality, and the millimeter-wave radar modality, respectively. ; The data layer fusion feature vector is obtained based on the attention weights. : ; The data layer fusion feature vector After deep network architecture The fully connected layer and output layer are used to obtain the probability of abnormal environment state categories. : .

[0015] In one embodiment, the training method of the deep network architecture specifically includes: The deep network architecture is trained using a sparse constraint classifier, and the total loss function used includes cross-entropy loss and sparse constraint terms. Cross-entropy loss for: ; in, The number of feature vector samples fused at the data layer. This represents the number of abnormal environmental state categories. For the first The data layer fusion feature vector sample belongs to the first... The true label of an abnormal environmental state. Predicting the first for deep network architecture The data layer fusion feature vector belongs to the th . The probability of an abnormal environmental state; For the first Each data layer fuses feature vector samples. For deep network architecture Predict the k-th component of the output; Sparse constraint terms use Norm: ; in, For sparse constraint coefficients, The weight matrix of a deep network architecture is the first... Each weight element; total loss function for: .

[0016] In one embodiment, the combination , , , , , In addition to autonomous driving map information, Bayesian inference observation data is constructed, the posterior probability of abnormal environment states is calculated, the abnormal environment perception results are output, and the parameters of the convolutional network model and deep network architecture are adjusted based on the posterior probability of abnormal environment states. Specifically, this includes: S401: Determine the nearest road centerline point to the vehicle in the autonomous driving map. and the corresponding tangential heading angle Calculate the lateral deviation between the vehicle and the road centerline. and heading deviation : ; ; in, For the three-dimensional position of the vehicle plane coordinates, The centerline of the road is at point The normal vector at that point, This represents the angle normalization function. For transpose; S402: Fuse feature vectors into the data layer Target layer fusion feature vector Probability of Abnormal Environmental State Categories , merge target location Vehicle three-dimensional position Heading angle lateral deviation Heading deviation Map semantic information The Bayesian inference observation data consists of obstacle distribution probability maps. : ; Among them, map semantic information This includes the coordinates of the road centerline and the location information of traffic signs in the autonomous driving map. This is a probability map of obstacle distribution obtained through autonomous driving maps or historical perception statistics. S403: Let the set of abnormal environment states be... Prior probabilities are initialized based on autonomous driving map information, historical perception samples, vehicle traffic records, and manually labeled abnormal environment samples. : ; in, The first in historical data The statistical number of abnormal environmental states. The first in historical data The statistical number of abnormal environmental states. For smoothing coefficients; This represents the total number of abnormal environmental state categories. pass and Calculate the likelihood function The posterior probability is calculated using Bayes' theorem: ; in, For the first Candidate abnormal environment states. For Bayesian inference of observation data The following abnormal environment state is The posterior probability; S404: Determine the abnormal environment perception result based on the posterior probability: ; ; ; in, The category of abnormal environmental state with the highest posterior probability. This represents the corresponding posterior probability confidence level. For the location of the corresponding abnormal target or abnormal area; when Greater than the preset threshold At that time, output the abnormal environment perception results. ;when Less than or equal to the preset threshold When the time is right, output either a normal environment status or a low-confidence abnormality message; S405: Adjust parameters for convolutional network models and deep network architectures based on posterior probabilities. ; in, For the first The parameters of the convolutional network model or deep network architecture in each iteration. For learning rate, Indicates to The gradient operator for finding the gradient. This indicates that, under parameter θ, and conditioned on Bayesian inference observation data Z, the abnormal environment state is... The posterior probability; for convolutional network models, the parameters of the convolutional layers are fixed while the weights and biases of the fully connected layers are fine-tuned; for deep network architectures, the parameters of all layers are fine-tuned.

[0017] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any embodiment of the first aspect.

[0018] Compared with the prior art, the beneficial technical effects of the present invention are: 1. This invention innovatively integrates data from multiple sensors simultaneously, organically combining information collected by visual sensors, lidar, and millimeter-wave radar with data obtained from a high-precision positioning system to construct a comprehensive and high-precision environmental perception model. This enables autonomous vehicles to accurately capture details of their surrounding environment, providing reliable basis for driving decisions in complex city streets or highways with varying road conditions. This greatly improves the comprehensiveness and accuracy of environmental perception, far exceeding traditional single-sensor technology.

[0019] 2. This invention employs a unique multi-stage fusion training strategy, progressively deepening the fusion from the data layer and the target layer. In the data preprocessing stage, targeted filtering, synchronization algorithms, and normalization operations are used to ensure high data quality. A convolutional network model suitable for data from different sensors is constructed, and semi-supervised training is used to tap into the data's potential. A sparse constraint classifier is combined to strengthen the deep network architecture, accelerating model convergence, enhancing generalization ability, and efficiently addressing the challenges of processing multi-source heterogeneous data. Compared to traditional, generalized training methods, training efficiency and model performance are significantly improved.

[0020] 3. This invention comprehensively considers various performance requirements of autonomous driving and designs a sophisticated hierarchical reward function. It is no longer limited to a single indicator, but fully incorporates key elements such as safety, speed efficiency, and navigation matching accuracy. These elements are weighted and integrated according to their importance, making the learning objectives of the intelligent agent more aligned with actual driving expectations. While ensuring basic driving safety, it optimizes driving speed and accurately matches navigation routes, comprehensively improving the driving experience and overcoming the shortcomings of existing technologies that sometimes neglect certain aspects.

[0021] 4. This invention deeply integrates actual driving logic into the safety enhancement aspect. It fully considers the continuity and state dependence of driving behavior, anticipates potential dangers in advance, and preemptively punishes dangerous behaviors. It cleverly transforms the probability of disaster events into continuous reward and punishment signals, which are fed back to the driving strategy in real time. This reduces the risk of collisions at the source, which is completely different from the traditional post-event risk response mode. It safeguards the safety of autonomous driving and makes driving decisions more robust and reliable. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention.

[0023] Figure 2 This is a flowchart of the multi-source sensing data acquisition and preprocessing process of the present invention.

[0024] Figure 3 This is a flowchart of the multi-source feature fusion and abnormal environment perception of the present invention. Detailed Implementation

[0025] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] like Figure 1 As shown, an abnormal environment perception method based on multi-source information fusion in this invention includes the following steps: S1 utilizes visual sensors, LiDAR, and millimeter-wave radar to simultaneously collect information about the vehicle's surrounding environment, acquiring images, LiDAR point cloud data, and millimeter-wave radar data, and also uses a positioning system to collect the vehicle's three-dimensional position. With heading angle Constructing millimeter-wave radar feature vectors based on millimeter-wave radar data ; S2, Construct the first convolutional network model, process the image, and output visual feature vectors. A second convolutional network model is constructed to process the laser point cloud data and output the lidar feature vector. Semi-supervised training is used to enhance the generalization ability of the first and second convolutional network models. and Perform target layer fusion to obtain the target layer fused feature vector. and the target location of fusion ;Will , and Concatenate into a multi-source fusion feature vector ; S3, designed for The deep network architecture, through the attention mechanism, , and Perform data layer fusion to obtain the data layer fused feature vector. and the probability of abnormal environmental state categories ; S4, combined , , , , , In addition to autonomous driving map information, Bayesian inference observation data is constructed, the posterior probability of abnormal environment states is calculated, the abnormal environment perception results are output, and the parameters of the convolutional network model and deep network architecture are adjusted based on the posterior probability of abnormal environment states.

[0027] This invention provides an abnormal environment perception technology based on multi-source information fusion, which aims to improve the ability of autonomous vehicles to perceive abnormal situations in complex environments and ensure driving safety.

[0028] First, environmental information is collected using visual sensors, LiDAR, and millimeter-wave radar to construct a multi-view data source, laying the foundation for subsequent perception processing. Next, convolutional network models suitable for visual sensor data and LiDAR point cloud data are constructed separately, employing semi-supervised training to enhance model generalization ability and complete target layer fusion, thereby improving the accuracy of identifying different types of environmental information. Then, a deep network architecture oriented towards multi-source heterogeneous information input is designed, using a sparse constraint classifier for training to achieve data layer fusion, achieving accurate target detection and fully utilizing the complementarity of multi-source information. Finally, combining the data layer and target layer fusion results, high-precision positioning and autonomous driving map information are used, and parameters of the previously trained convolutional network model and deep network architecture are adjusted based on a Bayesian inference optimization algorithm, further improving the abnormal environment perception capability of autonomous vehicles based on vision and radar fusion.

[0029] The present invention will be described in detail below in several parts.

[0030] 1. Data acquisition and preprocessing.

[0031] Environmental information is acquired simultaneously using visual sensors, lidar, and millimeter-wave radar. The visual sensor acquires information at a rate of [data rate per second]. The frame frequency is obtained at a resolution of An image of pixels, where each pixel of the image has Each color channel records the color, texture, and shape of the environment. The LiDAR emits and receives signals every second. A laser beam measures the distance to surrounding objects, creating point cloud data. Each point in the point cloud data is represented by three-dimensional coordinates. and reflection intensity This indicates that millimeter-wave radar uses frequencies... The speed of work and target detection ,angle and distance Information such as...

[0032] Simultaneously, by combining a high-precision positioning system, the real-time three-dimensional position of the vehicle is obtained. and heading angle These data, along with environmental information collected by sensors, constitute the vehicle's raw observations of the surrounding world.

[0033] Data collected from different sensors is preprocessed to ensure data quality and consistency. First, for images acquired by the visual sensor, median filtering is used to remove salt-and-pepper noise. For point cloud data from the LiDAR, voxel filtering is used to reduce data density and remove outliers. Voxel filtering divides the point cloud space into segments with sides of length... The voxels are represented by points within each voxel, with their centroids used to represent the points within the voxels.

[0034] Regarding time synchronization, let the timestamps of the data collected by the visual sensor, lidar, and millimeter-wave radar be respectively... , and A synchronization algorithm based on timestamp interpolation is adopted for timestamps of 10 ... The data is used to obtain synchronous data from other sensors at the corresponding time using linear interpolation. For example, for images... To obtain time The image can be interpolated using the following formula, where... The next timestamp for the vision sensor: .

[0035] Next, the coordinate system of each sensor data point is transformed to the vehicle coordinate system. Let the transformation matrix from the visual sensor coordinate system to the vehicle coordinate system be... The transformation matrix from the lidar coordinate system to the vehicle coordinate system is: The transformation matrix from the millimeter-wave radar coordinate system to the vehicle coordinate system is: Taking lidar point cloud data as an example, the conversion formula is: ; for The corresponding point in the vehicle coordinate system.

[0036] Finally, the sensor data is normalized.

[0037] 2. Construct a convolutional network model.

[0038] A first convolutional network model is constructed for the image. The input of the first convolutional network model is the preprocessed image, which is processed sequentially through convolutional layers and pooling layers, flattened, and connected to a fully connected layer to output a visual feature vector. .

[0039] Specifically, let the first convolutional network model be... The number of convolutional kernels in each convolutional layer is The kernel size is Step size is Fill with The height and width of the feature map output by the previous convolutional layer are respectively and Then the first The height of the output feature map of each convolutional layer and width They are respectively: ; ; in The function represents rounding down; the pooling layer uses max pooling, and the pooling window size is [value missing]. Step size is .

[0040] Flatten the feature map output from the pooling layer and connect it to... Each fully connected layer outputs a visual feature vector that describes the visual representation of a target or anomalous region in an image. ;in, The number of neurons in each fully connected layer is respectively .

[0041] A second convolutional network model is constructed for laser point cloud data. This model uses the 3D coordinate range of the preprocessed laser point cloud data as the point cloud space, divides the point cloud space into voxels, and generates voxel feature vectors from the points within each voxel through feature aggregation, forming a voxel feature map. This map is then processed through multiple convolutional, pooling, and fully connected layers to output the LiDAR feature vector. .

[0042] Specifically, the second convolutional network model for laser point cloud data is constructed. This model uses the three-dimensional coordinate range of the preprocessed laser point cloud data as the point cloud space, divides the point cloud space into voxels, and generates voxel feature vectors from points within each voxel through feature aggregation, forming a voxel feature map. This map is then processed through multiple convolutional, pooling, and fully connected layers to output the LiDAR feature vector. Specifically, it includes: laser point cloud data The three-dimensional coordinate range is taken as the point cloud space, and the lengths of the point cloud space in the three coordinate directions are respectively... , , The side length of the voxel is The number of voxels in the three coordinate directions are as follows: ; ; ; in, The function represents rounding up. Each point within a voxel generates a voxel feature vector through feature aggregation. Let the dimension of each voxel feature vector be 1. The size of the voxel feature map obtained after voxelization is ; , , These represent the number of voxels in the three coordinate directions, respectively. The feature dimension of each voxel is represented; convolution operations are performed on the voxel feature maps, and after multiple convolution and pooling layers, they are connected to multiple fully connected layers to output a lidar feature vector describing the spatial structure features of the target in the lidar point cloud data. .

[0043] A semi-supervised training method is adopted, using labeled and unlabeled data to train the first and second convolutional network models. The total loss function of each convolutional network model is composed of a weighted sum of supervised loss and unsupervised consistency loss. The supervised loss is calculated based on cross-entropy, and the unsupervised consistency loss is calculated based on the difference between the output of the convolutional network model of the original data and the augmented data. The parameters of the convolutional network model are iteratively optimized using the stochastic gradient descent algorithm.

[0044] Specifically, for the first convolutional network model, a labeled image dataset is constructed. and unlabeled image datasets : ; ; in, For the first Each image has labeled samples. for The corresponding target category label or abnormal environment state category label, The number of labeled samples in the image. For the first One image without labeled samples, This represents the number of unlabeled samples in the image. First convolutional network model Output visual feature vector Classification Head Output image sample target category or abnormal environment state category probability : ; The softmax function is used. When the training samples are labeled with target categories such as vehicles, pedestrians, and obstacles, the classification head outputs the target category. When the training samples are labeled with abnormal environmental conditions such as construction areas, accident scenes, abnormal obstacles, and severe weather, the classification head outputs the abnormal environmental condition category.

[0045] Supervised loss of the first convolutional network model Unsupervised consistency loss Total loss They are respectively: ; ; ; It is the F2 norm. This indicates a data augmentation operation. This is the first balance coefficient; For the second convolutional network model, construct a labeled point cloud dataset. And unlabeled point cloud datasets : ; ; in, For the first Each point cloud has labeled samples. For the corresponding target category label or abnormal environment state category label, The number of labeled samples in the point cloud. For the first A point cloud sample without labeling. This represents the number of unlabeled point cloud samples. The second convolutional network model outputs the lidar feature vector. Classification Head Output the class probabilities of point cloud samples : ; Supervised loss for the second convolutional network model Unsupervised consistency loss Total loss They are respectively: ; ; ; in, This is the second balance coefficient.

[0046] Two convolutional network models were trained using the stochastic gradient descent (SGD) algorithm. During training, the model parameters were continuously adjusted to minimize the loss function.

[0047] Visual feature vectors and lidar feature vector Mapping to the same dimension yields the mapped visual feature vectors. and the mapped lidar feature vector Then, target layer weighted fusion is performed to obtain the target layer fused feature vector. : ; in, and These are the target layer fusion weights for visual feature vectors and LiDAR feature vectors, respectively. The fused target position is obtained by matching the coordinates of the visual target position with the spatial position of the lidar. .

[0048] Fusion target location This is the spatial location result obtained after target layer fusion. The first convolutional network model is used to obtain the two-dimensional image planar position of the target or anomaly region, and the second convolutional network model is used to obtain the three-dimensional point cloud spatial position of the corresponding target or anomaly region. Based on joint calibration parameters, the two-dimensional image planar position and the three-dimensional point cloud spatial position are matched, and the spatial position of the target or anomaly region in the vehicle coordinate system is determined by the matching result. It represents the center coordinates or spatial extent of the spatial location and is used as part of the Bayesian inference observation data to output the location of abnormal targets or abnormal areas in the abnormal environment perception results.

[0049] By conducting experiments on the validation set, the weights were adjusted. and This is to maximize the accuracy of target recognition by fusing feature vectors at the target layer.

[0050] 3. Build a deep network architecture.

[0051] Deep network architecture The input is a multi-source fusion feature vector. ; Visual feature vector LiDAR feature vector Millimeter-wave radar feature vectors Linear mapping to the same dimension yields the following results: , and : ; ; ; in, , , Linear mapping layers for visual modalities, lidar modalities, and millimeter-wave radar modalities, respectively; The multimodal feature fusion module employing an attention mechanism calculates the attention scores for each of the three modalities: ; ; ; in, , , The attention scores are for the visual modality, lidar modality, and millimeter-wave radar modality, respectively. , , All are multilayer perceptrons; Calculate the normalized attention weights for each modality based on the attention scores: ; ; ; in, , , The attention weights are for the visual modality, the lidar modality, and the millimeter-wave radar modality, respectively. ; The data layer fusion feature vector is obtained based on the attention weights. : ; The data layer fusion feature vector After deep network architecture The fully connected layer and output layer are used to obtain the probability of abnormal environment state categories. : .

[0052] The training methods for deep network architectures specifically include: The deep network architecture is trained using a sparse constraint classifier, and the total loss function used includes cross-entropy loss and sparse constraint terms. Cross-entropy loss for: ; in, The number of feature vector samples fused at the data layer. This represents the number of abnormal environmental state categories. For the first Each data layer fuses feature vector samples. For the first The data layer fusion feature vector sample belongs to the first... The true label of an abnormal environmental state. Predicting the first for deep network architecture The data layer fusion feature vector belongs to the th . The probability of an abnormal environmental state; Sparse constraint terms use Norm: ; in, For sparse constraint coefficients, The weight matrix of the deep network architecture is the first one. Each weight element; total loss function for: .

[0053] 4. Abnormal environment perception.

[0054] Determine the road centerline point closest to the vehicle in the autonomous driving map. and the corresponding tangential heading angle Calculate the lateral deviation between the vehicle and the road centerline. and heading deviation : ; ; in, For the three-dimensional position of the vehicle plane coordinates, The centerline of the road is at point The normal vector at that point, This represents the angle normalization function. This is a transpose.

[0055] Fuse feature vectors at the data layer Target layer fusion feature vector Probability of Abnormal Environmental State Categories , merge target location Vehicle three-dimensional position Heading angle lateral deviation Heading deviation Map semantic information The Bayesian inference observation data consists of obstacle distribution probability maps. : ; Among them, map semantic information This includes the coordinates of the road centerline and the location information of traffic signs in the autonomous driving map. This is a probability map of obstacle distribution obtained through autonomous driving maps or historical perception statistics.

[0056] Let the set of abnormal environment states be Prior probabilities are initialized based on autonomous driving map information, historical perception samples, vehicle traffic records, and manually labeled abnormal environment samples. : ; in, The first in historical data The statistical number of abnormal environmental states. For smoothing coefficients; This represents the total number of abnormal environmental state categories. pass and Calculate the likelihood function The posterior probability is calculated using Bayes' theorem: ; in, For the first Candidate abnormal environment states. For Bayesian inference of observation data The following abnormal environment state is The posterior probability.

[0057] Determine the abnormal environment perception result based on the posterior probability: ; ; ; in, The category of abnormal environmental state with the highest posterior probability. This represents the corresponding posterior probability confidence level. For the corresponding abnormal target or abnormal area location; when Greater than the preset threshold At that time, output the abnormal environment perception results. ;when Less than or equal to the preset threshold When the time is right, output either a normal environment status or a low-confidence abnormality message.

[0058] Parameters of convolutional network models and deep network architectures are adjusted based on posterior probabilities: ; in, For the first The parameters of the convolutional network model or deep network architecture in each iteration. For learning rate, Indicates to The gradient operator is used to calculate the gradient; for convolutional network models, the parameters of the convolutional layers are fixed while the weights and biases of the fully connected layers are fine-tuned; for deep network architectures, the parameters of all layers are fine-tuned.

[0059] The model is continuously iterated and optimized. In each iteration, data acquisition, data preprocessing, model prediction, and parameter tuning are performed again. Performance metrics, such as accuracy, are evaluated by testing the model in various abnormal environmental scenarios. Recall rate and The model is considered converged when its performance metrics no longer improve after multiple iterations or reach a preset performance threshold. At this point, the model can accurately perceive abnormal environments around the autonomous vehicle, providing reliable environmental information to the autonomous driving system and ensuring safe driving in complex scenarios.

[0060] This invention aims to improve the environmental perception capabilities of autonomous vehicles in complex scenarios. The method utilizes multiple devices such as visual sensors, LiDAR, and millimeter-wave radar to collect environmental information, and combines this with a high-precision positioning system and autonomous driving maps to achieve comprehensive perception of the surrounding environment. A convolutional network is designed to process visual sensor data and LiDAR point cloud data separately, extracting key features. A fully connected network works in conjunction with a specific temporal analysis module to deeply fuse and dynamically model multi-source information, thereby accurately capturing real-time changes in the environment. Simultaneously, a deep network architecture oriented towards multi-source heterogeneous information is constructed and trained using a sparse constraint classifier, effectively removing noise and redundancy from the data and enhancing the model's ability to handle complex data and its generalization performance. A semi-supervised learning strategy is introduced to optimize the model, using a small amount of labeled data and a large amount of unlabeled data for training, mining the potential value of the data, reducing dependence on large-scale labeled data, and improving training efficiency. An optimization mechanism based on Bayesian inference, combined with map prior knowledge, dynamically adjusts model parameters according to the fused multi-source information, improving the accuracy and reliability of abnormal environment perception. Ultimately, this method enables autonomous vehicles to accurately identify various abnormal environments, providing a reliable basis for safe driving decisions and promoting the application and development of autonomous driving technology in complex scenarios.

[0061] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0062] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0063] In one embodiment, the present invention provides a computer device, which may be a server. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data used in the methods described above. The network interface communicates with external terminals via a network connection. The computer program is executed by the processor to implement the methods described above.

[0064] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0065] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0066] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An abnormal environment perception method based on multi-source information fusion, characterized in that, include: The system simultaneously collects information about the vehicle's surrounding environment using visual sensors, lidar, and millimeter-wave radar to acquire images, lidar point cloud data, and millimeter-wave radar data, and then uses a positioning system to acquire the vehicle's three-dimensional position. With heading angle Constructing millimeter-wave radar feature vectors based on millimeter-wave radar data ; Construct the first convolutional network model to process the image and output visual feature vectors. A second convolutional network model is constructed to process the laser point cloud data and output the lidar feature vector. Semi-supervised training is used to enhance the generalization ability of the first and second convolutional network models. and Perform target layer fusion to obtain the target layer fused feature vector. and the target location of fusion ;Will , and Concatenate into a multi-source fusion feature vector ; Design orientation The deep network architecture, through the attention mechanism, , and Perform data layer fusion to obtain the data layer fused feature vector. and the probability of abnormal environmental state categories ; Combination , , , , , In addition to autonomous driving map information, Bayesian inference observation data is constructed, the posterior probability of abnormal environment states is calculated, the abnormal environment perception results are output, and the parameters of the convolutional network model and deep network architecture are adjusted based on the posterior probability of abnormal environment states.

2. The abnormal environment perception method based on multi-source information fusion according to claim 1, characterized in that, This also includes preprocessing of image, laser point cloud data, and millimeter-wave radar data, specifically including: Median filtering is used to remove image noise from the image, voxel filtering is used to reduce the point cloud density and remove outliers from the laser point cloud data, and target range, azimuth angle, radial velocity and reflection intensity are extracted from the millimeter-wave radar data. Time synchronization is performed based on timestamp interpolation, and the coordinate systems of the vision sensor, LiDAR, and millimeter-wave radar are transformed to the vehicle coordinate system according to joint calibration parameters to obtain the preprocessed image. Laser point cloud data and millimeter-wave radar data ; The construction of millimeter-wave radar feature vectors based on millimeter-wave radar data specifically includes: based on preprocessed millimeter-wave radar data... The target range, azimuth, radial velocity, and reflection intensity are used to form the millimeter-wave radar feature vector. .

3. The abnormal environment perception method based on multi-source information fusion according to claim 1, characterized in that, The first convolutional network model is constructed to process images and output visual feature vectors. A second convolutional network model is constructed to process the laser point cloud data and output the lidar feature vector. Semi-supervised training is used to enhance the generalization ability of the first and second convolutional network models. and Perform target layer fusion to obtain the target layer fused feature vector. and the target location of fusion Specifically, it includes: A first convolutional network model is constructed for the image. The input of the first convolutional network model is the preprocessed image, which is processed sequentially through convolutional layers and pooling layers, flattened, and connected to a fully connected layer to output a visual feature vector. ; A second convolutional network model is constructed for laser point cloud data. This model uses the 3D coordinate range of the preprocessed laser point cloud data as the point cloud space, divides the point cloud space into voxels, and generates voxel feature vectors from the points within each voxel through feature aggregation, forming a voxel feature map. This map is then processed through multiple convolutional, pooling, and fully connected layers to output the LiDAR feature vector. ; A semi-supervised training method is adopted to train the first convolutional network model and the second convolutional network model using labeled and unlabeled data. The total loss function of each convolutional network model is composed of a weighted sum of supervised loss and unsupervised consistency loss. The supervised loss is calculated based on cross-entropy, and the unsupervised consistency loss is calculated based on the difference between the output of the convolutional network model of the original data and the augmented data. The parameters of the convolutional network model are iteratively optimized using the stochastic gradient descent algorithm. Visual feature vectors and lidar feature vector Mapping to the same dimension yields the mapped visual feature vectors. and the mapped lidar feature vector Then, target layer weighted fusion is performed to obtain the target layer fused feature vector. : ； in, and These are the target layer fusion weights for visual feature vectors and LiDAR feature vectors, respectively. The fused target position is obtained by matching the coordinates of the visual target position with the spatial position of the lidar. .

4. The abnormal environment perception method based on multi-source information fusion according to claim 3, characterized in that, The first convolutional network model for the image is constructed. The input of the first convolutional network model is the preprocessed image, which is processed sequentially through convolutional layers and pooling layers, flattened, and connected to a fully connected layer to output a visual feature vector. Specifically, it includes: Let the first convolutional network model be... The number of convolutional kernels in each convolutional layer is The kernel size is Step size is Fill with The height and width of the feature map output by the previous convolutional layer are respectively and Then the first The height of the output feature map of each convolutional layer and width They are respectively: ； ； in The function represents rounding down; the pooling layer uses max pooling, and the pooling window size is [value missing]. Step size is ; Flatten the feature map output from the pooling layer and connect it to... Each fully connected layer outputs a visual feature vector that describes the visual representation of a target or anomalous region in an image. ;in, The number of neurons in each fully connected layer is respectively .

5. The abnormal environment perception method based on multi-source information fusion according to claim 3, characterized in that, The second convolutional network model is constructed for laser point cloud data. This model uses the three-dimensional coordinate range of the preprocessed laser point cloud data as the point cloud space, divides the point cloud space into voxels, and generates voxel feature vectors from points within each voxel through feature aggregation, forming a voxel feature map. This map is then processed through multiple convolutional, pooling, and fully connected layers to output the LiDAR feature vector. Specifically, it includes: laser point cloud data The three-dimensional coordinate range is taken as the point cloud space, and the lengths of the point cloud space in the three coordinate directions are respectively... , , The side length of the voxel is The number of voxels in the three coordinate directions are as follows: ； ； ； in, The function represents rounding up. Each point within a voxel generates a voxel feature vector through feature aggregation. Let the dimension of each voxel feature vector be 1. The size of the voxel feature map obtained after voxelization is ; , , These represent the number of voxels in the three coordinate directions, respectively. The feature dimension of each voxel is represented; convolution operations are performed on the voxel feature maps, and after multiple convolution and pooling layers, they are connected to multiple fully connected layers to output a lidar feature vector describing the spatial structure features of the target in the lidar point cloud data. .

6. The abnormal environment perception method based on multi-source information fusion according to claim 3, characterized in that, The method employs semi-supervised training, using labeled and unlabeled data to train the first and second convolutional network models. The total loss function of each convolutional network model is a weighted sum of supervised loss and unsupervised consistency loss. The supervised loss is calculated based on cross-entropy, and the unsupervised consistency loss is calculated based on the difference between the convolutional network model outputs of the original data and the augmented data. The parameters of the convolutional network models are iteratively optimized using a stochastic gradient descent algorithm. Specifically, this includes: For the first convolutional network model, construct a labeled image dataset. and unlabeled image datasets : ； ； in, For the first Each image has labeled samples. for The corresponding target category label or abnormal environment state category label, The number of labeled samples in the image. For the first One image without labeled samples, This represents the number of unlabeled samples in the image. First convolutional network model Output visual feature vector Classification Head Output image sample target category or abnormal environment state category probability : ； It is the softmax function; Supervised loss of the first convolutional network model Unsupervised consistency loss Total loss They are respectively: ； ； ； An index representing the target category or abnormal environment state category. This indicates the total number of target categories or abnormal environmental state categories. This indicates that the i-th image has labeled samples belonging to the k-th class with true labels. This represents the probability that a labeled sample in the i-th image is predicted to be of the k-th class. It is the F2 norm. This indicates a data augmentation operation. This is the first balance coefficient; For the second convolutional network model, construct a labeled point cloud dataset. And unlabeled point cloud datasets : ； ； in, For the first Each point cloud has labeled samples. For the corresponding target category label or abnormal environment state category label, The number of labeled samples in the point cloud. For the first A point cloud sample without labeling. This represents the number of unlabeled point cloud samples. Second convolutional network model Output lidar feature vector Classification Head Output the probability of the target category or abnormal environment state category of the point cloud sample. : ； Supervised loss for the second convolutional network model Unsupervised consistency loss Total loss They are respectively: ； ； ； in, This indicates that the i-th point cloud has a labeled sample belonging to the k-th class with a true label. This represents the probability that a labeled sample in the i-th point cloud is predicted to be of the k-th class. This is the second balance coefficient.

7. The abnormal environment perception method based on multi-source information fusion according to claim 1, characterized in that, The design orientation The deep network architecture, through the attention mechanism, , and Perform data layer fusion to obtain the data layer fused feature vector. and the probability of abnormal environmental state categories Specifically, it includes: Deep network architecture The input is a multi-source fusion feature vector. ; Visual feature vector LiDAR feature vectors Millimeter-wave radar feature vectors Linear mapping to the same dimension yields the mapped visual feature vectors. Mapped lidar feature vectors and mapped millimeter-wave radar feature vector : ； ； ； in, , , Linear mapping layers for visual modalities, lidar modalities, and millimeter-wave radar modalities, respectively; The multimodal feature fusion module employing an attention mechanism calculates the attention scores for each of the three modalities: ； ； ； in, , , The attention scores are for the visual modality, lidar modality, and millimeter-wave radar modality, respectively. , , All are multilayer perceptrons; Calculate the normalized attention weights for each modality based on the attention scores: ； ； ； in, , , The attention weights are for the visual modality, the lidar modality, and the millimeter-wave radar modality, respectively. ; The data layer fusion feature vector is obtained based on the attention weights. : ； The data layer fusion feature vector After deep network architecture The fully connected layer and output layer are used to obtain the probability of abnormal environment state categories. : 。 8. The abnormal environment perception method based on multi-source information fusion according to claim 1, characterized in that, The training methods for the deep network architecture specifically include: The deep network architecture is trained using a sparse constraint classifier, and the total loss function used includes cross-entropy loss and sparse constraint terms. Cross-entropy loss for: ； in, The number of feature vector samples fused at the data layer. This represents the number of abnormal environmental state categories. For the first The data layer fusion feature vector sample belongs to the first... The true label of an abnormal environmental state. Predicting the first for deep network architecture The data layer fusion feature vector belongs to the th . The probability of an abnormal environmental state; For the first Each data layer fuses feature vector samples. For deep network architecture Predict the k-th component of the output; Sparse constraint terms use Norm: ； in, For sparse constraint coefficients, The weight matrix of a deep network architecture is the first... Each weight element; total loss function for: 。 9. The abnormal environment perception method based on multi-source information fusion according to claim 1, characterized in that, The combination , , , , , In addition to autonomous driving map information, Bayesian inference observation data is constructed, the posterior probability of abnormal environment states is calculated, the abnormal environment perception results are output, and the parameters of the convolutional network model and deep network architecture are adjusted based on the posterior probability of abnormal environment states. Specifically, this includes: S401: Determine the nearest road centerline point to the vehicle in the autonomous driving map. and the corresponding tangential heading angle Calculate the lateral deviation between the vehicle and the road centerline. and heading deviation : ； ； in, For the three-dimensional position of the vehicle plane coordinates, The centerline of the road is at point The normal vector at that point, This represents the angle normalization function. For transpose; S402: Fuse feature vectors into the data layer Target layer fusion feature vector Probability of Abnormal Environmental State Categories , merge target location Vehicle three-dimensional position Heading angle lateral deviation Heading deviation Map semantic information The Bayesian inference observation data consists of obstacle distribution probability maps. : ； Among them, map semantic information This includes the coordinates of the road centerline and the location information of traffic signs in the autonomous driving map. This is a probability map of obstacle distribution obtained through autonomous driving maps or historical perception statistics. S403: Let the set of abnormal environment states be... Prior probabilities are initialized based on autonomous driving map information, historical perception samples, vehicle traffic records, and manually labeled abnormal environment samples. : ； in, The first in historical data The statistical number of abnormal environmental states. The first in historical data The statistical number of abnormal environmental states. For smoothing coefficients; This represents the total number of abnormal environmental state categories. pass and Calculate the likelihood function The posterior probability is calculated using Bayes' theorem: ； in, For the first Candidate abnormal environment states. For Bayesian inference of observation data The following abnormal environment state is The posterior probability; S404: Determine the abnormal environment perception result based on the posterior probability: ； ； ； in, The category of abnormal environmental state with the highest posterior probability. This represents the corresponding posterior probability confidence level. For the corresponding abnormal target or abnormal area location; when Greater than the preset threshold At that time, output the abnormal environment perception results. ;when Less than or equal to the preset threshold When the time is right, output either a normal environment status or a low-confidence abnormality message; S405: Adjust parameters for convolutional network models and deep network architectures based on posterior probabilities. ； in, For the first The parameters of the convolutional network model or deep network architecture in each iteration. For learning rate, Indicates to The gradient operator for finding the gradient. This indicates that, under parameter θ, and conditioned on Bayesian inference observation data Z, the abnormal environment state is... The posterior probability; for convolutional network models, the parameters of the convolutional layers are fixed while the weights and biases of the fully connected layers are fine-tuned; for deep network architectures, the parameters of all layers are fine-tuned.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images