Automatic driving multi-sensor fusion method and system facing adverse weather

Abstract

The invention provides an unfavorable weather-oriented automatic driving multi-sensor fusion method, relates to the field of unmanned driving assistance in a closed scene, and provides an unmanned vehicle auxiliary sensing system depending on 5G communication, and the auxiliary sensing system comprises an unmanned vehicle, a millimeter wave radar, a camera sensing sensor, a laser radar sensor and a CAN communication system. According to the method, the communication transmission speed is high, the calculated amount is accurate, the obstacle information and distance in front of the autonomous vehicle in adverse weather can be accurately predicted, and efficient and accurate data fusion between the unmanned vehicle and sensor equipment is achieved. According to the invention, stable and efficient operation of the unmanned vehicle can be effectively guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving assistance in closed scenes, and in particular to an autonomous driving multi-sensor fusion method for adverse weather. Background Art

[0002] With the continuous progress and innovation of artificial intelligence technology, the field of unmanned driving has also ushered in rapid development. The integration of artificial intelligence makes autonomous driving technology more intelligent. In a closed environment, since traffic participants are relatively fixed, the application of autonomous driving vehicles becomes more practical. At present, more and more autonomous driving vehicles use perception sensors to assist driving.

[0003] As an indispensable 3D sensor for autonomous driving, the point cloud data provided by LiDAR has significant advantages in terms of geometry, scale information, precise distance and fine semantic description, which is extremely beneficial for understanding three-dimensional scenes. However, the sparsity, disorder and uneven distribution of point cloud data limit its application. At the same time, although camera images contain rich semantic information such as color, they lack depth and scale information. Although millimeter-wave radar can provide accurate position information, it is not as good as LiDAR in terms of geometry and scale information of objects. Therefore, the complementary information of LiDAR, camera and millimeter-wave radar provides the possibility for multi-sensor fusion. However, sensors are also easily affected by external environmental factors such as weather and lighting, which may interfere with the judgment of autonomous driving. Traditional fusion algorithms, whether combining camera and LiDAR data or obtaining obstacle positions through monocular depth estimation methods, have certain limitations. If multi-sensor fusion can be realized on autonomous driving vehicles, it can not only improve the operating stability of the vehicle without significantly increasing costs, but also have important significance for the update and iteration of perception algorithm models.

[0004] The information disclosed in the background technology section of this application is only intended to deepen the understanding of the general background technology of this application, and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0005] The present invention provides an autonomous driving multi-sensor fusion method for adverse weather conditions, which can solve the technical problem of high deployment and operation costs of assisted autonomous driving vehicles.

[0006] According to a first aspect of the present invention, there is provided a multi-sensor fusion method for autonomous driving in adverse weather, comprising: Under good weather conditions, autonomous vehicle A calibrates and time-aligns the camera sensor, lidar sensor, and millimeter-wave radar sensor; During operation, the autonomous driving vehicle A determines whether the current scene weather exceeds the single sensor perception calculation threshold based on the information obtained by the perception sensor. If it exceeds the single sensor perception calculation threshold, the autonomous driving vehicle A will send a multi-sensor fusion signal; After receiving the multi-sensor fusion signal transmitted by the autonomous driving vehicle A, the vehicle-mounted system B starts a multi-sensor fusion instruction for the autonomous driving vehicle A; The autonomous driving vehicle A starts synchronously collecting sensor perception data, wherein the sensor perception data includes: camera sensor information, lidar sensor information, and millimeter wave radar sensor information; After receiving the sensor perception data sent by the autonomous driving vehicle A, the on-board control program starts segmented multiple sensor fusion. The fusion process includes: obtaining accurate object information of obstacles and point cloud depth features through 2D and 3D feature extraction fusion and obtaining accurate position information of obstacles through millimeter wave radar distance fusion, and sending the fused high-definition sensor perception data to the autonomous driving vehicle A through the CAN communication module; After receiving the fused high-definition sensor perception data from the vehicle control system, the autonomous driving vehicle A continues to upgrade to the normal operating mode; The control program module of the vehicle system determines whether the fused high-definition sensor perception data meets the requirement that the unmanned vehicle is operating in a normal mode. If the threshold is exceeded, a stop instruction is issued to the autonomous driving vehicle A to instruct the autonomous driving vehicle A to pull over. The autonomous driving vehicle A receives the stop command issued by the control program and enters the maintenance mode, and the process ends.

[0007] According to a second aspect of the present invention, there is provided an autonomous driving multi-sensor fusion system for adverse weather, comprising: The software and hardware modules installed on the unmanned vehicle include: a millimeter wave module for high-precision distance equipment; a camera for collecting 2D image information during the operation of the unmanned vehicle, and the perceived information includes lane lines, surrounding environment images, and traffic lights; a lidar for obtaining 3D semantic information of objects in the unmanned driving environment through 3D point cloud data; a bus control module, including a CAN communication module, for communicating with the vehicle, and a control processor for processing sensor information sent by the sensor and fusing it with subsequent sensors.

[0008] According to a third aspect of the present invention, there is provided an autonomous driving multi-sensor fusion device for adverse weather, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the autonomous driving multi-sensor fusion method for adverse weather.

[0009] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the multi-sensor fusion method for autonomous driving in adverse weather.

[0010] Technical effect: According to the present invention, it can be used as an effective supplement to the auxiliary perception of the unmanned driving system in a closed scene. When the weather becomes bad when the vehicle is running, the obstacle information in front of the vehicle can be accurately perceived by fusing multi-sensor information, which can improve the environmental perception ability of the unmanned vehicle. In addition, the continuous data fusion of a large number of unmanned vehicles can also help improve the performance of the perception algorithm, and help the computing platform to handle various environmental perceptions. The star topology structure also greatly reduces the cost of deployment and operation. When determining the image relationship function between the global atmospheric light, transmission map, clean image and foggy image, the relationship between the global atmospheric light, transmission map, clean image and foggy image can be accurately analyzed, which improves the accuracy, scientificity and objectivity of the image relationship function. When determining the approximate solution of the transmission map, it can provide a data basis for the subsequent restoration of the clean image. When determining the specific spatial filtering processing steps, the high-frequency components of the image can be enhanced, and the contours and details of the objects in the image can be made clearer. When determining the point cloud depth features, the point cloud depth features can be determined through the multi-head attention mechanism, and the detailed information of the object can be captured, thereby improving the accuracy of target obstacle detection and recognition. When determining the specific implementation of enhancing the point cloud depth feature, the point cloud depth feature can be enhanced through the enhanced multi-head attention module, multi-head self-attention module, point-by-point convolution layer, multi-layer perceptron and connection function, respectively, to improve the accuracy of the point cloud depth feature. When determining the fused obstacle feature, the foggy image feature map and the point cloud depth feature map can be fused through the fusion function to determine the fused obstacle feature, thereby improving the comprehensiveness of the obstacle feature.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present invention. Other features and aspects of the present invention will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative work. Figure 1 A schematic diagram of a flow chart of a multi-sensor fusion method for autonomous driving in adverse weather according to an embodiment of the present invention is exemplarily shown; Figure 2 An on-board flow chart of an autonomous driving multi-sensor fusion system without adverse weather conditions according to an embodiment of the present invention is exemplarily shown; Figure 3 The logical structure diagram of the multi-sensor fusion method for autonomous driving in adverse weather according to an embodiment of the present invention is exemplarily shown. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0015] Figure 1 A flowchart of a multi-sensor fusion method for autonomous driving in adverse weather according to an embodiment of the present invention is exemplarily shown, and the method includes: Step S101, under good weather conditions, the autonomous driving vehicle A calibrates and time-aligns the camera sensor, the lidar sensor, and the millimeter-wave radar sensor; Step S102, during operation, the autonomous driving vehicle A determines whether the current scene weather exceeds the single sensor perception calculation threshold based on the information obtained by the perception sensor. If it exceeds the single sensor perception calculation threshold, the autonomous driving vehicle A will send a multi-sensor fusion signal; Step S103, after receiving the multi-sensor fusion signal transmitted by the autonomous driving vehicle A, the vehicle-mounted system B starts a multi-sensor fusion instruction for the autonomous driving vehicle A; Step S104, the autonomous driving vehicle A starts synchronously collecting sensor perception data, wherein the sensor perception data includes: camera sensor information, lidar sensor information and millimeter wave radar sensor information; Step S105, after receiving the sensor perception data sent by the autonomous driving vehicle A, the on-board control program starts segmented multiple sensor fusion, and the fusion process includes: obtaining accurate obstacle object information and point cloud depth features through 2D and 3D feature extraction fusion and obtaining accurate obstacle position information through millimeter wave radar distance fusion, and sending the fused high-definition sensor perception data to the autonomous driving vehicle A through the CAN communication module; Step S106, the autonomous driving vehicle A continues to upgrade to the normal operating mode after receiving the fused high-definition sensor perception data sent by the vehicle control system; Step S107, the control program module of the vehicle system determines whether the fused high-definition sensor perception data meets the requirement that the unmanned vehicle is operating in normal mode. If the threshold is exceeded, a stop instruction is issued to the autonomous driving vehicle A to instruct the autonomous driving vehicle A to pull over. Step S108, the automatic driving vehicle A receives the stop command issued by the control program and enters the maintenance mode, and the process ends.

[0016] The multi-sensor fusion method for autonomous driving in adverse weather according to an embodiment of the present invention can be used as an effective supplement to the auxiliary perception of the unmanned driving system in closed scenes. When the weather during vehicle operation becomes bad, the obstacle information in front of the vehicle can be accurately perceived through the fusion of multi-sensor information, which can improve the environmental perception ability of the unmanned vehicle. In addition, the continuous data fusion of a large number of unmanned vehicles can also help improve the performance of the perception algorithm, and help the computing platform to handle a variety of environmental perception improvements. The star topology structure also greatly reduces the cost of deployment and operation.

[0017] According to one embodiment of the present invention, in step S101, under good weather conditions, the autonomous driving vehicle A calibrates and time-aligns the camera sensor, the lidar sensor, and the millimeter-wave radar sensor.

[0018] For example, in clear weather conditions, the autonomous vehicle calibrates and time-aligns the camera sensors, lidar sensors, and millimeter-wave radar sensors.

[0019] According to one embodiment of the present invention, in step S102, during operation, the autonomous driving vehicle A determines whether the current scene weather exceeds the single sensor perception calculation threshold based on the information obtained by the perception sensor. If it exceeds the single sensor perception calculation threshold, the autonomous driving vehicle A will send a multi-sensor fusion signal.

[0020] For example, during operation, autonomous driving vehicle A determines whether the current scene weather exceeds the single sensor perception calculation threshold based on information obtained by the camera sensor, lidar sensor and millimeter wave radar sensor. If it exceeds the single sensor perception calculation threshold, autonomous driving vehicle A will send a multi-sensor fusion signal.

[0021] According to one embodiment of the present invention, in step S103, after receiving the multi-sensor fusion signal transmitted by the autonomous driving vehicle A, the on-board system B starts a multi-sensor fusion instruction to the autonomous driving vehicle A.

[0022] According to one embodiment of the present invention, in step S104, the autonomous driving vehicle A starts synchronously collecting sensor perception data, wherein the sensor perception data includes: camera sensor information, lidar sensor information and millimeter wave radar sensor information.

[0023] For example, after receiving the multi-sensor fusion instruction, autonomous driving vehicle A simultaneously turns on the camera sensor, lidar sensor and millimeter-wave radar sensor to collect sensor perception data.

[0024] According to one embodiment of the present invention, in step S105, after receiving the sensor perception data sent by the autonomous driving vehicle A, the on-board control program starts segmented multiple sensor fusion, and the fusion process includes: obtaining the precise object information of the obstacle and the point cloud depth features through 2D and 3D feature extraction and fusion, and obtaining the precise position information of the obstacle through millimeter wave radar distance fusion, and sending the fused high-definition sensor perception data to the autonomous driving vehicle A through the CAN communication module.

[0025] According to one embodiment of the present invention, step S105 includes: Determine a foggy image through the camera sensor information; Extracting features of the foggy image by CNN and Transformer respectively to determine the features of the foggy image, wherein the features of the foggy image include: 2D features and 3D features; determining an image relationship function between a global atmospheric light, a transmission map, a clean image, and the foggy image; Get the approximate solution of the transmission diagram; Restoring the clean image according to the foggy image, the relationship function and the approximate solution value of the transmission map; Through the designed image filter module, the foggy image is subjected to defogging, white balance, gamma, contrast enhancement and spatial filtering; Determine the deep features of point cloud through multi-head attention mechanism; Enhancing the depth features of the point cloud; Obtaining fused obstacle features according to the foggy image features and the point cloud depth features; Determine obstacle shape estimation based on point cloud computing; Determining accurate object information of the obstacle based on the fused obstacle features and the obstacle shape estimation; Obtain accurate obstacle location information by fusing laser radar information and millimeter wave radar information; Determining fused high-definition sensor perception data according to the precise object information of the obstacle and the precise position information of the obstacle; The fused high-definition sensor perception data is sent to autonomous driving vehicle A through the CAN communication module.

[0026] For example, a foggy image is determined through an image collected by a camera sensor; two BatchNormalization (BN) layers, a downsampling layer, an upsampling layer, two 1x1 convolutional layers, and finally a Sigmoid layer are used for normalization to obtain the features required by the Transformer and CNN layers; the image relationship function between the global atmospheric light, the transmission map, the clean image, and the foggy image is determined, and the approximate solution of the transmission map is determined, and the approximate solution of the transmission map is substituted into the image relationship function to restore the clean image; the foggy image is defogged, white balanced, gamma, contrast enhanced, and spatially filtered through the designed image filter module. The local white balance method is used for white balance. The brightest area in the image is searched as a white area. The statistical averages of the R, GB three-color components in the area are the same. At the same time, the mean of the R and B components in the area is used as the basis for white balance calibration. Spatial filtering is used as the local white balance method for the image. For image processing of neighborhood pixels, filtering can be used to enhance the parts of interest in the image and the high-frequency components of the image, which can make the outlines and details of objects in the image clearer; the efficient multi-head attention mechanism is first used on the point cloud branch to determine the depth features of the point cloud; the global features of the point cloud depth are enhanced; the foggy image features and the point cloud depth features are fused to obtain the fused obstacle features, and the obstacle depth distance and obstacle shape estimation calculated by the point cloud are obtained at the same time. At the same time, the precise location information of the obstacle is obtained through the millimeter-wave radar, and the distance and shape of the obstacle are obtained by fusing the lidar information and the millimeter-wave radar information. The fused information can better visualize the shape and precise location information of the obstacle; based on the precise object information and precise location information of the obstacle, the fused high-definition sensor perception data is determined, and the fused high-definition sensor perception data is sent to the autonomous driving vehicle A through the CAN communication module.

[0027] According to an embodiment of the present invention, determining an image relationship function between global atmospheric light, a transmission map, a clean image and the foggy image includes: determining the image relationship function according to formula (1), (1) in, For foggy images, is a clean image, A is the global atmospheric light, For the transmission diagram.

[0028] According to one embodiment of the present invention, is a foggy image, which means the foggy image collected by the camera sensor. is a clean image, which represents the target reflection image in the absence of fog, that is, the target image that needs to be restored. It is a transmission diagram, which represents the transfer function. It describes the remaining energy ratio of light after being scattered by the atmosphere in the process of reaching the detection system from the target. A is the global atmospheric light, which represents the background light formed after the ambient light is scattered in the atmosphere.

[0029] In this way, the relationship between global atmospheric light, transmission diagrams, clean images and foggy images can be accurately analyzed, which improves the accuracy, scientificity and objectivity of the image relationship function.

[0030] According to an embodiment of the present invention, obtaining an approximate solution value of a transmission graph includes: determining an approximate solution value of the transmission graph according to formula (2): , (2) Among them, min is the minimum value function, C is the color channel index, , is a learnable parameter, , is the neighborhood of x, is the color channel at pixel y The strength value of is the intensity of atmospheric light in the color channel.

[0031] According to one embodiment of the present invention, is a learnable parameter, a small constant, used to retain a certain fog effect and avoid unnatural images caused by complete defogging. Represents a neighborhood domain at position x The operation of finding the minimum value in It is usually a window or region centered on x, whose size and shape depends on the specific implementation of the algorithm. Indicates taking the minimum value on the color channel. For each pixel y, calculate the minimum value on the red, green and blue color channels respectively. , and determine the The minimum value of It represents the intensity of pixel y on color channel c and the intensity of atmospheric light on the color channel. This ratio reflects the degree of influence on light when passing through the atmosphere.

[0032] In this way, the approximate solution value of the transmission map can be determined, providing a data basis for the subsequent restoration of the clean image.

[0033] According to an embodiment of the present invention, the foggy image is subjected to defogging, white balance, gamma, contrast enhancement and spatial filtering by a designed image filter module, including: determining the spatial filtering processing step according to formulas (3), (4), (5) and (6) , , and , (3) (4) (5) (6) in, is the output pixel value, is the mapping function, is the input pixel, , and are the channels corresponding to the three primary colors, , and is the white balance parameter, is the gamma parameter, is the contrast enhancement parameter, , and is the spatial filter parameter.

[0034] According to one embodiment of the present invention, is a mapping function used to adjust the nonlinear relationship of image brightness. is the contrast enhancement parameter, which is used to adjust the contrast of the image. It means to adjust each color channel in the image independently to eliminate the color cast of the image caused by different light source colors, so that the colors in the image are closer to the colors in the actual scene. By applying a gamma correction to the input pixels, It means mapping the input pixels from the current range (non-standard or extended range) to the standard range (for example, 0-255 for 8-bit images) and adjusting the contrast of the image to enhance the differences between different objects and regions in the image. It means that during the filtering process, the three primary color channels are processed differently according to specific filtering parameters.

[0035] In this way, the specific spatial filtering processing steps can be determined, the high-frequency components of the image can be enhanced, and the contours and details of the objects in the image can be made clearer.

[0036] According to an embodiment of the present invention, the point cloud depth feature is determined by a multi-head attention mechanism, including: determining the output point cloud feature value according to formulas (7), (8) and (9): , (7) (8) (9) in, is multi-head self-attention, is the discard function, is the layer norm, To apply a layer norm operation to the input X, is the connection function, SA is the self-attention operator, is the first self-attention operator, is the second self-attention operator, is the h-th self-attention operator, is the original input data, For standard attention, , , and is the linear layer for context encoding, s is the stride, Represents an average pooling operation with stride s.

[0037] According to one embodiment of the present invention, in formula (7), It is a regularization method for data with spatial structure (such as images and point clouds). In point cloud processing, spatial Dropout can randomly discard spatially adjacent points or point clusters to simulate missing or occluded points in point cloud data. Represents the original input data Perform layer normalization. It means that after the layer-normalized input X is enhanced, a multi-head self-attention mechanism is applied to capture complex relationships. It means that the dropout operation is used to prevent overfitting, thereby obtaining a model output with stronger generalization ability.

[0038] According to one embodiment of the present invention, in formula (8), Indicates that , arrive The output is through This connection operation is performed on the last dimension, that is, the output sequences of all heads are concatenated along this dimension. The concatenated tensor contains information from different representation subspaces, providing a richer contextual representation for the model. Represents applying a linear transformation (usually a fully connected layer) to the concatenated tensor to transform it back to the same or desired dimension as the input. This linear transformation helps to integrate the information from different heads and produce the final output representation.

[0039] According to one embodiment of the present invention, in formula (9), represents the query matrix, represents the key matrix, represents the value matrix, ,in, is the scale factor, It means that each element of the input vector is transformed by an exponential function and normalized to a probability distribution.

[0040] In this way, the deep features of the point cloud can be determined through the multi-head attention mechanism, and the detailed information of the object can be captured, thereby improving the accuracy of target obstacle detection and recognition.

[0041] According to one embodiment of the present invention, enhancing the point-to-cloud depth feature includes: According to formulas (10), (11), (12), (13) and (14) (10) (11) (12) (13) (14) Determine the specific implementation of enhancing the point cloud depth feature, where: is the output of multi-head self-attention, is the output of the enhanced multi-head attention module, is the feature output of the previous layer, and is the output of the point-by-point convolutional layer, is a point-wise convolutional layer for channel projection, is the output of the connection function, is a multi-layer perceptron, is the connection function.

[0042] According to one embodiment of the present invention, in formula (10), It means that the feature output of the previous layer is passed through the point-by-point convolution layer to transform the feature and adjust the dimension of the output of the previous layer. In formula (11), Indicates the feature data of the previous layer that will be convolved through the point-by-point convolution layer, using a multi-head self-attention mechanism. In formula (12), It means that the output of the previous layer is transformed twice and the dimension is adjusted through the point-by-point convolution layer. In formula (13), Indicates that the output of the point-by-point convolutional layer is passed through The operations are connected. In formula (14), It is a neural network connected by multiple layers of neurons. It can perform nonlinear transformations on input data, thereby learning complex patterns and features in the data. Represents the output of the enhanced multi-head attention module determined by the output of the multi-layer perceptron and the point-by-point convolutional layer.

[0043] In this way, the specific implementation of enhancing the point cloud deep features is determined, and the point cloud deep features are enhanced through enhanced multi-head attention module, multi-head self-attention module, point-by-point convolution layer, multi-layer perceptron and connection function, thereby improving the accuracy of point cloud deep features.

[0044] According to an embodiment of the present invention, obtaining fused obstacle features according to the foggy image features and the point cloud depth features includes: According to formula (15), the obstacle characteristics after fusion are determined , (15) in, is the feature map of foggy image, is the point cloud depth feature map, is the input foggy image feature, is the input point cloud depth feature, is the fusion function, and is used to generate feature maps of different modalities.

[0045] According to one embodiment of the present invention, It means that the foggy image feature map and the point cloud depth feature map are fused through the fusion function to determine the obstacle features after fusion. To determine the foggy image feature map based on the input foggy image features and feature mapping, To determine the point cloud depth feature map based on the input point cloud depth features.

[0046] In this way, the foggy image feature map and the point cloud depth feature map can be fused through the fusion function to determine the fused obstacle features, thereby improving the comprehensiveness of the obstacle features.

[0047] According to one embodiment of the present invention, in step S106, the autonomous driving vehicle A continues to upgrade to the normal operating mode after receiving the fused high-definition sensor perception data sent by the vehicle control system.

[0048] According to one embodiment of the present invention, in step S107, the control program module of the vehicle-mounted system determines whether the fused high-definition sensor perception data is consistent with the unmanned vehicle operating in normal mode. If it exceeds the threshold, a stop operation instruction is issued to the autonomous driving vehicle A, instructing the autonomous driving vehicle A to pull over.

[0049] For example, when the fused high-definition sensor perception data does not conform to the normal operation of the unmanned vehicle, the on-board system sends a stop command to the autonomous driving vehicle A, instructing the autonomous driving vehicle A to pull over.

[0050] According to one embodiment of the present invention, in step S108, the autonomous driving vehicle A receives the stop command issued by the control program and enters the maintenance mode, and the process ends.

[0051] According to the multi-sensor fusion method for unfavorable weather of the present invention, it can be used as an effective supplement to the auxiliary perception of the unmanned driving system in a closed scene. When the weather for vehicle operation becomes bad, the obstacle information in front of the vehicle can be accurately perceived by fusing multi-sensor information, which can improve the environmental perception ability of the unmanned vehicle. In addition, the continuous data fusion of a large number of unmanned vehicles also helps to improve the performance of the perception algorithm, and helps the computing platform to process various environmental perceptions. The star topology structure also greatly reduces the cost of deployment and operation. When determining the image relationship function between the global atmospheric light, the transmission map, the clean image and the foggy image, the relationship between the global atmospheric light, the transmission map, the clean image and the foggy image can be accurately analyzed, and the accuracy, scientificity and objectivity of the image relationship function are improved. When determining the approximate solution of the transmission map, a data basis can be provided for the subsequent restoration of the clean image. When determining the specific spatial filtering processing steps, the high-frequency components of the image can be enhanced, and the contours and details of the objects in the image can be made clearer. When determining the point cloud depth features, the point cloud depth features can be determined through the multi-head attention mechanism, and the detailed information of the object can be captured, thereby improving the accuracy of target obstacle detection and recognition. When determining the specific implementation of enhancing the point cloud depth feature, the point cloud depth feature can be enhanced through the enhanced multi-head attention module, multi-head self-attention module, point-by-point convolution layer, multi-layer perceptron and connection function, respectively, to improve the accuracy of the point cloud depth feature. When determining the fused obstacle feature, the foggy image feature map and the point cloud depth feature map can be fused through the fusion function to determine the fused obstacle feature, thereby improving the comprehensiveness of the obstacle feature.

[0052] Figure 2 The on-board flow chart of the multi-sensor fusion system for autonomous driving in adverse weather according to an embodiment of the present invention is exemplarily shown, and the system includes: The software and hardware modules installed on the unmanned vehicle include: a millimeter wave module for high-precision distance equipment; a camera for collecting 2D image information during the operation of the unmanned vehicle, and the perceived information includes lane lines, surrounding environment images, and traffic lights; a lidar for obtaining 3D semantic information of objects in the unmanned driving environment through 3D point cloud data; a bus control module, including a CAN communication module, for communicating with the vehicle, and a control processor for processing sensor information sent by the sensor and fusing it with subsequent sensors.

[0053] Figure 3 The following is a logical structure diagram of a multi-sensor fusion method for autonomous driving in adverse weather according to an embodiment of the present invention, wherein the method includes: Obtain foggy image features based on camera sensors; According to the radar sensor, the depth point cloud features are obtained; Obtain obstacle distance characteristics based on millimeter wave radar; Determine the target obstacle features based on the depth point cloud features and foggy image features; Determine the target obstacle distance based on the depth point cloud features and obstacle distance features; According to the target obstacle characteristics and the target obstacle distance, the obstacle characteristics and the distance are output.

[0054] According to one embodiment of the present invention, there is provided an autonomous driving multi-sensor fusion device for adverse weather, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the autonomous driving multi-sensor fusion method for adverse weather.

[0055] According to one embodiment of the present invention, there is provided a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the multi-sensor fusion method for autonomous driving in adverse weather.

[0056] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0057] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.

Claims

1. A multi-sensor fusion method for autonomous driving in adverse weather conditions. It is characterized in that include: Under good weather conditions, autonomous vehicle A calibrates and time-aligns the camera sensor, lidar sensor, and millimeter-wave radar sensor; During operation, the autonomous driving vehicle A determines whether the current scene weather exceeds the single sensor perception calculation threshold based on the information obtained by the perception sensor. If it exceeds the single sensor perception calculation threshold, the autonomous driving vehicle A will send a multi-sensor fusion signal; After receiving the multi-sensor fusion signal transmitted by the autonomous driving vehicle A, the vehicle-mounted system B starts a multi-sensor fusion instruction for the autonomous driving vehicle A; The autonomous driving vehicle A starts synchronously collecting sensor perception data, wherein the sensor perception data includes: camera sensor information, lidar sensor information, and millimeter wave radar sensor information; After receiving the sensor perception data sent by the autonomous driving vehicle A, the on-board control program starts segmented multiple sensor fusion. The fusion process includes: obtaining accurate object information of obstacles and point cloud depth features through 2D and 3D feature extraction fusion and obtaining accurate position information of obstacles through millimeter wave radar distance fusion, and sending the fused high-definition sensor perception data to the autonomous driving vehicle A through the CAN communication module; After receiving the fused high-definition sensor perception data from the vehicle control system, the autonomous driving vehicle A continues to upgrade to the normal operating mode; The control program module of the vehicle system determines whether the fused high-definition sensor perception data meets the requirement that the unmanned vehicle is operating in a normal mode. If the threshold is exceeded, a stop instruction is issued to the autonomous driving vehicle A to instruct the autonomous driving vehicle A to pull over. The autonomous driving vehicle A receives the stop command issued by the control program and enters the maintenance mode, and the process ends.

2. The multi-sensor fusion method for autonomous driving in adverse weather according to claim 1, It is characterized in that After receiving the sensor perception data sent by the autonomous driving vehicle A, the vehicle control program starts segmented multiple sensor fusion, including: Determine a foggy image through the camera sensor information; Extracting features of the foggy image by CNN and Transformer respectively to determine the features of the foggy image, wherein the features of the foggy image include: 2D features and 3D features; determining an image relationship function between a global atmospheric light, a transmission map, a clean image, and the foggy image; Get the approximate solution of the transmission diagram; Restoring the clean image according to the foggy image, the relationship function and the approximate solution value of the transmission map; Through the designed image filter module, the foggy image is subjected to defogging, white balance, gamma, contrast enhancement and spatial filtering; Determine the deep features of point cloud through multi-head attention mechanism; Enhancing the depth features of the point cloud; Obtaining fused obstacle features according to the foggy image features and the point cloud depth features; Determine obstacle shape estimation based on point cloud computing; Determining accurate object information of the obstacle based on the fused obstacle features and the obstacle shape estimation; Obtain accurate obstacle location information by fusing laser radar information and millimeter wave radar information; Determining fused high-definition sensor perception data according to the precise object information of the obstacle and the precise position information of the obstacle; The fused high-definition sensor perception data is sent to autonomous driving vehicle A through the CAN communication module.

3. The multi-sensor fusion method for autonomous driving in adverse weather according to claim 2, It is characterized in that Determining an image relationship function between a global atmospheric light, a transmission map, a clean image, and the foggy image, including: According to the formula Determine the image relationship function, where For foggy images, is a clean image, A is the global atmospheric light, For the transmission diagram.

4. The multi-sensor fusion method for autonomous driving in adverse weather according to claim 3, It is characterized in that Get approximate solutions to transmission diagrams, including: According to the formula Determine the approximate solution to the transmission diagram , where min is the minimum function, C is the color channel index, , is a learnable parameter, , is the neighborhood of x, is the color channel at pixel y The strength value of is the intensity of atmospheric light in the color channel.

5. The multi-sensor fusion method for autonomous driving in adverse weather according to claim 2, It is characterized in that Through the designed image filter module, the foggy image is subjected to defogging, white balance, contrast enhancement and spatial filtering, including: According to the formula Determine the spatial filtering process steps , , and ,in, is the output pixel value, is the mapping function, is the input pixel, , and are the channels corresponding to the three primary colors, , and is the white balance parameter, is the gamma parameter, is the contrast enhancement parameter, , and is the spatial filter parameter.

6. The multi-sensor fusion method for autonomous driving in adverse weather according to claim 2, It is characterized in that Through the multi-head attention mechanism, the deep features of the point cloud are determined, including: According to the formula Determine the output point cloud feature values ,in, is multi-head self-attention, is the discard function, is the layer norm, To apply a layer norm operation to the input X, is the connection function, SA is the self-attention operator, is the first self-attention operator, is the second self-attention operator, is the h-th self-attention operator, is the original input data, For standard attention, , , and is the linear layer for context encoding, s is the stride, Represents an average pooling operation with stride s.

7. The multi-sensor fusion method for autonomous driving in adverse weather according to claim 6, It is characterized in that The point cloud depth feature is enhanced, including: According to the formula Determine the specific implementation of enhancing the point cloud depth feature, where: is the output of multi-head self-attention, is the output of the enhanced multi-head attention module, is the feature output of the previous layer, and is the output of the point-by-point convolutional layer, is a point-wise convolutional layer for channel projection, is the output of the connection function, is a multi-layer perceptron, is the connection function.

8. The multi-sensor fusion method for autonomous driving in adverse weather according to claim 7, It is characterized in that Obtaining fused obstacle features according to the foggy image features and the point cloud depth features, including: According to the formula Determine the characteristics of the fused obstacle ,in, is the feature map of foggy image, is the point cloud depth feature map, is the input foggy image feature, is the input point cloud depth feature, is the fusion function, and is used to generate feature maps of different modalities.

9. An autonomous driving multi-sensor fusion system for adverse weather conditions for executing the method according to any one of claims 1 to 8, It is characterized in that include: The hardware and software modules installed on the unmanned vehicle include: millimeter wave modules for high-precision distance equipment; The camera is used to collect 2D image information during the operation of the unmanned vehicle. The perceived information includes lane lines, surrounding environment images, and traffic lights. The lidar obtains 3D semantic information of objects in the unmanned driving environment through 3D point cloud data. The bus control module includes a CAN communication module, which is used to communicate with the vehicle. The control processor is used to process the sensor information sent by the sensor and fuse it with subsequent sensors.

Images