An all-weather multi-modal fusion method and device based on information entropy

Abstract

The application provides an all-day multi-modal fusion method and device based on information entropy, and relates to the technical field of automatic driving. The method comprises the following steps: acquiring an RGB image and three-dimensional point cloud data of a same target scene; performing spatial transformation on the three-dimensional point cloud data to obtain a depth projection image with the same size as the RGB image; processing the RGB image and the depth projection image respectively to obtain a first feature map and a second feature map; calculating an entropy map of the RGB image; and processing the entropy map, the first feature map and the second feature map by using a self-attention mechanism to obtain a fused feature map. The application optimizes the entropy of the RGB image, so that the entropy can more accurately reflect the light distribution in the image. The multi-modal fusion is realized through the entropy and the self-attention, the robustness of the scene is improved, and the multi-modal fusion is suitable for all-day.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an all-day multimodal fusion method and apparatus based on information entropy. Background Technology

[0002] Lane detection is a crucial perception task in autonomous driving, playing a vital role in lane keeping and lane departure warning. Segmentation-based lane detection methods use semantic segmentation models to make pixel-level predictions from images. Currently, in autonomous driving perception systems, lane detection can be categorized into three types based on modality: image-based methods, radar point cloud data-based methods, and multimodal fusion-based methods. Images contain rich texture information, which is highly beneficial for lane feature learning. However, images are very susceptible to lighting conditions, especially in poor environments such as nighttime, where many details are lost. In recent years, lane detection using radar-acquired point cloud data has become a hot topic. Point clouds are unaffected by lighting conditions and possess scene robustness. However, point clouds are essentially just collections of sparse points, which poses a significant constraint for lanes requiring dense prediction. Multimodal fusion-based methods combine camera and LiDAR data, leveraging the complementarity between modalities to obtain a more robust lane detection model.

[0003] Existing lane line detection methods based on multimodal fusion have the following problems: (1) They are usually applicable to normal and suitable environments. However, due to the complexity and variety of scenarios in actual applications, especially in adverse scenarios such as night and low light, the acquired lane line images may be blurred, lose details, and have reduced contrast, resulting in a significant decrease in detection accuracy and poor perception effect. (2) Information complementarity is achieved through RGB images and point clouds, but there are significant differences between the modalities, and direct fusion for lane line detection is not effective. Summary of the Invention

[0004] In view of this, this application provides an all-day multimodal fusion method and apparatus based on information entropy to solve the above-mentioned technical problems.

[0005] This application provides an all-day, all-modal fusion method based on information entropy, including:

[0006] Acquire RGB images and 3D point cloud data of the same target scene;

[0007] Perform spatial transformation on the 3D point cloud data to obtain a depth projection map of the same size as the RGB image;

[0008] The RGB image and the depth projection map are processed separately to obtain the first feature map and the second feature map;

[0009] The entropy map of the RGB image is calculated, and the entropy map, the first feature map, and the second feature map are processed using a self-attention mechanism to obtain the fused feature map.

[0010] Furthermore, a spatial transformation is performed on the 3D point cloud data to obtain a depth projection map of the same size as the RGB image; including:

[0011] The 3D point cloud data is projected from the 3D coordinate system onto the plane of the 2D grayscale image, and the value of each pixel is either the depth value of the point or 0, thus obtaining the depth projection map of the 3D point cloud data.

[0012] Further, the RGB image and depth projection map are processed separately to obtain a first feature map and a second feature map; including:

[0013] The RGB image is processed using the pre-trained first convolutional layer to obtain a 64-channel first feature map with a height of H and a width of W.

[0014] The depth projection map is processed using a pre-trained second convolutional layer to obtain a 64-channel second feature map with height H and width W.

[0015] Further, the entropy map of the RGB image is calculated; including:

[0016] Convert the RGB image to a grayscale image. gray ;

[0017] Statistical Grayscale Image I gray The histogram is then normalized to obtain the normalized histogram hist(s):

[0018]

[0019] Where s is the pixel value, which ranges from 0 to 255;

[0020] Function I(I gray (i,j)=s) is:

[0021]

[0022] The normalized histograms are summed to obtain the histogram summation result. sum (s):

[0023] hist sum (s)=hist sum (s-1)+hist(s)

[0024] Calculate the entropy(s):

[0025] entropy(s)=entropy(s-1)-hist sum (s)·log(hist sum (s))

[0026] Binarizing the entropy yields an entropy mask.

[0027] The entropy map is obtained by removing noise points from the entropy mask using morphological opening operations.

[0028] Furthermore, the entropy map, the first feature map, and the second feature map are processed using a self-attention mechanism to obtain a fused feature map, including:

[0029] Set the entropy map as the key in the triplet of the self-attention mechanism; set the first feature map as the value in the triplet of the self-attention mechanism; and set the second feature map as the query in the triplet of the self-attention mechanism.

[0030] Convert the query, key, and value into N-dimensional vectors, where N = H × W;

[0031] The pixel (n, m) value of the attention map is calculated using the vector query and the vector key. nm :

[0032]

[0033] Among them, query n The key is the nth element of the vector query. m Let be the m-th element of the vector key; 1 ≤ m ≤ N, 1 ≤ n ≤ N;

[0034] After performing a softmax operation on the attention map, a normalized attention map is obtained.

[0035] The normalized attention map is multiplied by the vector value to obtain an N-dimensional vector G. The N-dimensional vector G is then reshaped to obtain the feature map E, with H and W as the height and width of the feature map E.

[0036] Calculate the fused feature map F fusion :

[0037] F fusion =F1+E+F2

[0038] Wherein, F1 is the first feature map and F2 is the second feature map.

[0039] Furthermore, the semantic segmentation network has a U-shaped symmetrical structure, comprising an encoder and a decoder; the encoder comprises a first ResNet layer, a second ResNet layer, a third convolutional layer, and a fourth convolutional layer connected in sequence; the decoder comprises a first deconvolutional layer, a second deconvolutional layer, a third deconvolutional layer, and a fourth deconvolutional layer connected in sequence; wherein, the first ResNet layer and the fourth deconvolutional layer are connected; the second ResNet layer and the third deconvolutional layer are connected; the third convolutional layer and the second deconvolutional layer are connected; and the fourth convolutional layer and the first deconvolutional layer are connected.

[0040] The method further includes: using a semantic segmentation network to process the fused feature map to obtain pixel-level prediction results for lane lines.

[0041] Furthermore, the method also includes the step of jointly training the first convolutional layer, the second convolutional layer, and the semantic segmentation network.

[0042] Secondly, embodiments of this application provide an all-day, all-modal fusion device based on information entropy, comprising:

[0043] The acquisition unit is used to acquire RGB images and 3D point cloud data of the same target scene;

[0044] The first processing unit is used to perform spatial transformation on the three-dimensional point cloud data to obtain a depth projection map of the same size as the RGB image.

[0045] The second processing unit is used to process the RGB image and the depth projection map respectively to obtain the first feature map and the second feature map;

[0046] The fusion unit is used to calculate the entropy map of the RGB image. It uses a self-attention mechanism to process the entropy map, the first feature map, and the second feature map to obtain the fused feature map.

[0047] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0048] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the methods of embodiments of this application.

[0049] This application optimizes the entropy of RGB images to more accurately reflect the illumination distribution in the image; it achieves multimodal fusion through entropy and self-attention, improving the robustness of the scene and making it suitable for multimodal fusion throughout the day. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 A flowchart of the all-day multimodal fusion method based on information entropy provided in the embodiments of this application;

[0052] Figure 2 This is a structural diagram of the semantic segmentation network provided in an embodiment of this application;

[0053] Figure 3 Functional structure diagram of the all-day multimodal fusion device based on information entropy provided in the embodiments of this application;

[0054] Figure 4 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0057] First, the relevant terms involved in the embodiments of this invention are defined as follows:

[0058] RGB image: A color image acquired by a monocular camera, which is a three-channel image.

[0059] 3D point cloud: Point data about the surface of an object obtained through measuring instruments such as LiDAR. Each point data includes x, y, z coordinates, depth, and reflection intensity information.

[0060] Two-dimensional point cloud depth projection map: This is a two-dimensional projection map obtained by projecting the point cloud collected by the lidar onto the plane of the grayscale image. The value is the depth value of the corresponding point in the point cloud.

[0061] Labels: These represent the category labels used for supervised training of the semantic segmentation neural network, indicating the category of each pixel.

[0062] Image entropy: A statistical method for representing the amount of information in an image; the higher the entropy, the richer the pixel grayscale of the image.

[0063] Self-attention: The self-attention mechanism distinguishes the attention levels of different information through weight allocation. The attention mechanism can be viewed as a multilayer perceptron consisting of a query matrix, keys, and a weighted average. Given a query, the relevance between the query and different keys is calculated in the source, i.e., the weight coefficients of different values ​​in the source are calculated. The weighted average of the values ​​can be used as the attention value.

[0064] The design concept of the embodiments of this application will be briefly introduced below.

[0065] Existing lane line detection methods based on multimodal fusion have the following problems: (1) They are usually applicable to normal and suitable environments. However, due to the complexity and variety of scenarios in actual applications, especially in adverse scenarios such as night and low light, the acquired lane line images may be blurred, lose details, and have reduced contrast, resulting in a significant decrease in detection accuracy and poor perception effect. (2) Information complementarity is achieved through RGB images and point clouds, but there are significant differences between the modalities, and direct fusion for lane line detection is not effective.

[0066] To address the aforementioned technical challenges, this application provides an all-day multimodal fusion method based on information entropy. The method's approach is as follows: image entropy reflects the amount of information in an image, specifically the distribution of areas under normal and poor lighting. This is crucial for fusion because RGB images contain more information in normally lit areas, while point cloud depth maps contain even more information in poorly lit areas. By capturing the spatial information between entropy and modality using an attention-based approach based on image entropy, multimodal data can be adaptively fused according to the lighting distribution in the scene, thus achieving robustness across different scenarios. This adaptive modality fusion also provides more semantic information for subsequent semantic segmentation, making lane line segmentation more robust in complex scenes. This method is applicable to various lighting scenarios and exhibits higher generalization ability.

[0067] Compared with existing technologies, the advantages of this application are:

[0068] 1. This application optimizes the entropy of RGB images to more accurately reflect the illumination distribution in the image; it achieves multimodal fusion through entropy and self-attention, improving the robustness of the scene and making it suitable for multimodal fusion throughout the day;

[0069] 2. This application uses multimodal fusion data for lane line detection, that is, it uses pixel-level mask images to detect lane lines, thereby improving the accuracy of lane line detection.

[0070] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0071] like Figure 1 As shown, this application provides an all-day, all-modal fusion method based on information entropy, including:

[0072] Step 101: Acquire three-channel RGB images and point cloud data of the same target scene;

[0073] In this embodiment, RGB images of the road conditions are acquired using an onboard camera; a forward-facing monocular camera or a forward-facing monocular camera mounted on the vehicle captures images of the area ahead. The forward-facing monocular camera captures images of the area directly in front of the vehicle's direction of travel and above the road surface. Three-dimensional point cloud data of the road conditions is acquired using an onboard LiDAR. Scene image information and scene point cloud information are acquired simultaneously.

[0074] For ease of calculation, the point clouds involved in the embodiments of this application are all portions of the 360° point cloud facing the front of the vehicle, i.e., the direction in which the image is located. Since the camera and LiDAR have been calibrated, the transformation matrix of the point cloud projection onto the pixel plane can be determined to facilitate the subsequent alignment of modal features.

[0075] Step 102: Perform spatial transformation on the 3D point cloud data to obtain a depth projection map of the same size as the RGB image;

[0076] The 3D point cloud data is projected from the 3D coordinate system onto the plane of the 2D grayscale image. The value of each pixel is the depth value of the point (there is a point at the pixel) or 0 (there is no point at the pixel), thus obtaining the depth projection map of the 3D point cloud data.

[0077] Step 103: Process the RGB image and the depth projection map respectively to obtain the first feature map and the second feature map;

[0078] In this embodiment, the step includes:

[0079] The RGB image is processed using the pre-trained first convolutional layer to obtain a 64-channel first feature map F. i ;

[0080] The depth projection map is processed using a pre-trained second convolutional layer to obtain a 64-channel second feature map F. d .

[0081] Step 104: Calculate the entropy map of the RGB image, and use the self-attention mechanism to process the entropy map, the first feature map, and the second feature map to obtain the fused feature map.

[0082] The calculation of the entropy map of the RGB image includes:

[0083] Convert the RGB image to a grayscale image. gray ;

[0084] Statistical Grayscale Image I gray The histogram is then normalized to obtain the normalized histogram hist(s):

[0085]

[0086] Where s is the pixel value, ranging from 0 to 255; H and W represent the height and width of the grayscale image;

[0087] Function I(I gray (i,j)=s) is:

[0088]

[0089] The normalized histograms are summed to obtain the histogram summation result. sum (s):

[0090] hist sum (s)=hist sum (s-1)+hist(s)

[0091] Calculate the entropy(s):

[0092] entropy(s)=entropy(s-1)-hist sum (s)·log(hist sum (s))

[0093] Binarizing the entropy yields an entropy mask.

[0094] The entropy map is obtained by removing noise points from the entropy mask using morphological opening operations.

[0095] Specifically, a self-attention mechanism is used to process the entropy map, the first feature map, and the second feature map to obtain a fused feature map; including:

[0096] Set the entropy map as the key in the triplet of the self-attention mechanism; set the first feature map as the value in the triplet of the self-attention mechanism; and set the second feature map as the query in the triplet of the self-attention mechanism.

[0097] Convert the query, key, and value into N-dimensional vectors, where N = H × W;

[0098] The pixel (n, m) value of the attention map is calculated using the vector query and the vector key. nm :

[0099]

[0100] Among them, query n The key is the nth element of the vector query. m Let be the m-th element of the vector key; 1 ≤ m ≤ N, 1 ≤ n ≤ N;

[0101] After performing a softmax operation on the attention map, a normalized attention map is obtained.

[0102] The normalized attention map is multiplied by the vector value to obtain an N-dimensional vector G. The N-dimensional vector G is then reshaped to obtain the feature map E, with H and W as the height and width of the feature map E.

[0103] Calculate the fused feature map F fusion :

[0104] F fusion =F1+E+F2

[0105] Wherein, F1 is the first feature map and F2 is the second feature map.

[0106] In addition, the method also includes: processing the fused feature map using a semantic segmentation network to obtain pixel-level prediction results of lane lines.

[0107] The semantic segmentation network employs a U-shaped symmetrical structure. The encoder downsamples through convolutional layers, while the decoder gradually recovers feature resolution through deconvolutional layers. The network used is a U-shaped symmetrical structure, consisting of an encoder and a decoder. The encoder's first two layers utilize the first two layers of ResNet, followed by two convolutional layers to further extract semantic information. The decoder uses four deconvolutional layers to gradually recover details. In the U-shaped structure, skip connections bridge the gap between the encoder and decoder, facilitating the recovery of feature details during upsampling. Simulations demonstrate its effectiveness in recovering target details.

[0108] like Figure 2As shown, the semantic segmentation network includes an encoder and a decoder; the encoder includes a first ResNet layer, a second ResNet layer, a third convolutional layer, and a fourth convolutional layer connected in sequence; the decoder includes a first deconvolutional layer, a second deconvolutional layer, a third deconvolutional layer, and a fourth deconvolutional layer connected in sequence; wherein, the first ResNet layer and the fourth deconvolutional layer are connected; the second ResNet layer and the third deconvolutional layer are connected; the third convolutional layer and the second deconvolutional layer are connected; and the fourth convolutional layer and the first deconvolutional layer are connected.

[0109] Semantic segmentation can also be any predictive neural network, which can be pre-trained or trained on local data, using image semantic segmentation labels as supervision labels. The input to the semantic segmentation network is an RGB image and depth map of size (512, 512, 3), which becomes (512, 512, 64) after initial modality extraction. The resulting entropy is a binary image of size (512, 512), and the attention result obtained from the entropy is (512, 512). Finally, these are fused to obtain a fused feature of size (512, 512, 64).

[0110] The method further includes: using a semantic segmentation network to process the fused feature map to obtain pixel-level prediction results for lane lines.

[0111] The method further includes the step of jointly training the first convolutional layer, the second convolutional layer, and the semantic segmentation network.

[0112] Before training the neural network, the three-channel RGB image samples and depth map samples need to be preprocessed to crop the size of both images to (512, 512, 3). The target output of the network is (512, 512, 2).

[0113] The segmentation network is implemented using the PyTorch tool. After setting the batch size, number of training epochs, learning rate (lr) for each epoch, and other hyperparameters that need to be defined manually, training begins. The difference value is calculated based on the loss function, and the network parameters are optimized using the backpropagation algorithm until all epochs of training are completed.

[0114] The loss function is the common cross-entropy loss function, Loss:

[0115]

[0116] Where: M is the number of categories to be segmented; c is the currently calculated category; y c These are real tags; p c It is the prediction result of the network, and its value represents the probability.

[0117] Based on the above embodiments, this application provides an all-day multimodal fusion device based on information entropy, see reference. Figure 3 As shown, the all-day multimodal fusion device 200 based on information entropy provided in this application embodiment includes at least:

[0118] The acquisition unit 201 is used to acquire RGB images and 3D point cloud data of the same target scene;

[0119] The first processing unit 202 is used to perform spatial transformation on the three-dimensional point cloud data to obtain a depth projection map with the same size as the RGB image.

[0120] The second processing unit 203 is used to process the RGB image and the depth projection map respectively to obtain the first feature map and the second feature map;

[0121] The fusion unit 204 is used to calculate the entropy map of the RGB image. It uses a self-attention mechanism to process the entropy map, the first feature map, and the second feature map to obtain the fused feature map.

[0122] It should be noted that the principle of the all-day multimodal fusion device 200 based on information entropy provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the all-day multimodal fusion device 200 based on information entropy provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0123] Based on the above embodiments, this application also provides an electronic device, see below. Figure 4 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, it implements the all-day multimodal fusion method based on information entropy provided in this application embodiment.

[0124] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0125] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0126] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0127] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 4 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 4 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0128] It should be noted that, Figure 4 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0129] This application also provides a computer-readable storage medium storing computer instructions. When executed by a processor, these instructions implement the all-day, all-weather multimodal fusion method based on information entropy provided in this application. Specifically, the executable program can be built into or installed in an electronic device 300, allowing the electronic device 300 to implement the all-day, all-weather multimodal fusion method based on information entropy provided in this application by executing the built-in or installed executable program.

[0130] The all-day multimodal fusion method based on information entropy provided in this application embodiment can also be implemented as a program product. The program product includes program code. When the program product can run on the electronic device 300, the program code is used to make the electronic device 300 execute the all-day multimodal fusion method based on information entropy provided in this application embodiment.

[0131] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0132] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0133] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0134] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. An all-weather multi-modal fusion method based on information entropy, characterized in that, The method comprises the following steps: obtaining an RGB image and three-dimensional point cloud data of the same target scene; performing spatial transformation on the three-dimensional point cloud data to obtain a depth projection map with the same size as the RGB image; processing the RGB image and the depth projection map respectively to obtain a first feature map and a second feature map; calculating an entropy map of the RGB image, and processing the entropy map, the first feature map and the second feature map by using a self-attention mechanism to obtain a fused feature map; processing the entropy map, the first feature map and the second feature map by using a self-attention mechanism to obtain a fused feature map; comprising: setting the entropy map as key in a three-tuple of the self-attention mechanism, setting the first feature map as value in the three-tuple of the self-attention mechanism, and setting the second feature map as query in the three-tuple of the self-attention mechanism; Convert query, key, and value to a vector, ; height of the first feature map, width; Computing the values of the pixels of the attention map by vector query and vector key :​ wherein, is the n-th element of the vector query; is the m-th element of the vector key; , ; performing softmax operation on the attention map to obtain a normalized attention map; The normalized attention map is dotted with the vector value to obtain The vector G is reshaped to obtain a feature map The vector G is reshaped to obtain a feature map The height and width of the feature map are and ; computing the fused feature map : wherein is a first feature map, is a second feature map.

2. The method of claim 1, wherein, performing spatial transformation on the three-dimensional point cloud data to obtain a depth projection map with the same size as the RGB image; comprising: projecting the three-dimensional point cloud data from a three-dimensional coordinate system to a two-dimensional gray image plane, and taking the depth value or 0 of the point as the value of each pixel point to obtain a depth projection map of the three-dimensional point cloud data.

3. The method of claim 1, wherein, processing the RGB image and the depth projection map respectively to obtain a first feature map and a second feature map; comprising: The RGB image is processed by using the first convolutional layer which is pre-trained to obtain a first feature map with 64 channels, a height of , and a width of ; The second convolutional layer is used to process the depth projection map, and a 64-channel second feature map is obtained, with a height of and a width of .

4. The method of claim 3, wherein, calculating an entropy map of the RGB image; comprising: converting the RGB image to a grayscale image ; statistical gray scale map histogram and normalizing to obtain a normalized histogram : wherein, is a pixel value, taking a value in the range of 0 to 255; Function is: accumulating the normalized histograms to obtain a histogram accumulation result : Computing entropy : performing binaryzation on the entropy to obtain an entropy mask map; adopting morphological opening operation to remove noise points of the entropy mask map to obtain the entropy map.

5. The method of claim 1, wherein, The method further comprises: processing the fused feature map by using a semantic segmentation network to obtain a pixel-level prediction result of the lane line. The semantic segmentation network is a symmetrical structure of U type, and the semantic segmentation network comprises: an encoder and a decoder; the encoder comprises a first ResNet layer, a second ResNet layer, a third convolution layer and a fourth convolution layer connected in sequence; the decoder comprises a first deconvolution layer, a second deconvolution layer, a third deconvolution layer and a fourth deconvolution layer connected in sequence; wherein the first ResNet layer is connected with the fourth deconvolution layer; the second ResNet layer is connected with the third deconvolution layer; the third convolution layer is connected with the second deconvolution layer; and the fourth convolution layer is connected with the first deconvolution layer.

6. The method of claim 5, wherein, The method further comprises the step of jointly training the first convolution layer, the second convolution layer and the semantic segmentation network.

7. An all-weather multi-modal fusion device based on information entropy, characterized in that, comprising: an acquisition unit configured to acquire an RGB image and three-dimensional point cloud data of the same target scene; a first processing unit configured to perform spatial transformation on the three-dimensional point cloud data to obtain a depth projection map with the same size as the RGB image; a second processing unit configured to process the RGB image and the depth projection map respectively to obtain a first feature map and a second feature map; a fusion unit configured to calculate an entropy map of the RGB image, and process the entropy map, the first feature map and the second feature map by using a self-attention mechanism to obtain a fused feature map; processing the entropy map, the first feature map and the second feature map by using a self-attention mechanism to obtain a fused feature map; comprising: Set the entropy graph as the key in the triplets of the self-attention mechanism; set the first feature graph as the value in the triplets of the self-attention mechanism, and set the second feature graph as the query in the triplets of the self-attention mechanism; convert the query, the key, and the value to a vector, ; a height of the first feature map, a width of the first feature map; calculating values of pixels of an attention map from a vector query and a vector key :​ wherein, is the n-th element of the vector query; is the m-th element of the vector key; , ; After the attention graph is subjected to the softmax operation, a normalized attention graph is obtained. The normalized attention map is dotted with the vector value to obtain The vector G is reshaped to obtain a feature map The vector G is reshaped to obtain a feature map , the height and width of the feature map are and ; computing the fused feature map : wherein is a first feature map, is a second feature map.

8. An electronic device, comprising: The method comprises the steps of: A memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the method according to any one of claims 1-6.

Images