Multi-modal bird's eye view fusion method and device based on location encoding

Abstract

The application provides a multi-modal bird's eye view fusion method and device based on position coding, wherein the method comprises the following steps: acquiring point cloud data and a plurality of RGB images of a target scene at an intersection; processing the point cloud data by using a three-dimensional backbone network to obtain a first feature map, and performing height dimension feature compression on the first feature map to obtain a first BEV feature map; processing each RGB image to obtain a BEV feature map, and superimposing the BEV feature maps of all the RGB images to obtain a second BEV feature map; processing the cascaded first BEV feature map and the second BEV feature map by using a feature query model to obtain a first feature query result; and processing the first feature query result, the first BEV feature map and the second BEV feature map by using a modulation position coding attention network to obtain a second feature query result. The application effectively fuses multi-source information, makes the advantages of the two kinds of data complementary, and improves the accuracy of a subsequent 3D target detection task.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method and apparatus for multimodal bird's-eye view fusion based on location coding. Background Technology

[0002] Roadside lighthouses are typically equipped with visual cameras and LiDAR to collect images and point cloud data of the intersection scene. If only images are used for target detection, serious errors will occur when changes in lighting conditions or other factors cause image information to be lost. Furthermore, the lack of spatial distance information in images also limits their detection capabilities. While point cloud data can overcome the problem of lighting changes, the sparsity and disorder of point clouds mean that LiDAR alone is insufficient for robust 3D target detection.

[0003] To address the insufficient detection capabilities when using only cameras or LiDAR, a combination of multiple sensors and data fusion is considered to improve environmental perception. However, LiDAR point clouds are three-dimensional data, while camera image data is two-dimensional data; they exist in different data spaces and cannot be easily fused. Currently, a common method is to simultaneously project point cloud data and visual images onto a bird's-eye view space for data fusion, but this method loses spatial height information and the scale information of objects themselves. Summary of the Invention

[0004] In view of this, this application provides a method and apparatus for multimodal bird's-eye view fusion based on location coding to solve the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this application provide a multimodal bird's-eye view fusion method based on location encoding, including:

[0006] Acquire point cloud data and multiple RGB images of the target scene at the intersection;

[0007] The point cloud data is processed using a pre-trained 3D backbone network to obtain the first feature map. The first feature map is then compressed to obtain the first BEV feature map.

[0008] Each RGB image is processed to obtain a BEV feature map. The BEV feature maps of all RGB images are superimposed to obtain a second BEV feature map.

[0009] The first BEV feature map and the second BEV feature map of the cascade are processed to obtain the first feature query result, which includes the first feature vector of multiple target center points.

[0010] The first feature query result, the first BEV feature map, and the second BEV feature map are processed using a pre-trained modulation position coding attention network to obtain the second feature query result. The second feature query result includes second feature vectors of multiple target center points, and the dimension of the second feature vector is greater than the dimension of the first feature vector.

[0011] Furthermore, each RGB image is processed to obtain a BEV feature map; including:

[0012] Each RGB image is processed using a pre-trained two-dimensional backbone network to obtain a two-dimensional feature vector for each pixel of the RGB image;

[0013] Add a predicted depth value to the two-dimensional feature vector of each pixel to obtain the two-dimensional feature vector of each pixel;

[0014] The second feature map is obtained by projecting the three-dimensional feature vectors of all pixels onto the coordinate system of the point cloud data.

[0015] The second feature map is subjected to high-dimensional feature compression to obtain the BEV feature map.

[0016] Furthermore, the cascaded first BEV feature map and second BEV feature map are processed to obtain the first feature query result; including:

[0017] The first and second BEV feature maps of the cascade are processed using pre-trained convolutional layers to obtain the BEV feature map F1.

[0018] The BEV feature map F1 is processed using a pre-trained SE channel attention mechanism network to obtain the fused BEV feature map F. fused ;

[0019] The first feedforward neural network, which has been pre-trained, is used to fuse the BEV feature map F. fused The process yields multiple target center points and their classification results.

[0020] Obtain the probability values ​​of generating prediction boxes from multiple target center points, sort the probability values ​​in descending order, and obtain the target center points corresponding to the top N probability values;

[0021] Extract the feature vector of the target center point from the first BEV feature map as the first feature vector of the target center point, and combine the first feature vectors of N target center points to form the first feature query result Q. LiDAR .

[0022] Furthermore, the modulation position coding attention network includes: a first attention mechanism network, a modulation position coding unit, a cascaded unit, and a second attention mechanism network;

[0023] The first feature query result, the first BEV feature map, and the second BEV feature map are processed using a pre-trained modulation position coding attention network to obtain a fused feature map based on the feature query; including:

[0024] Using a first attention mechanism network to query the first feature result Q LiDAR and the first BEV feature map F LiDAR The third feature query result Q is obtained by performing fusion processing. L The third feature query result Q L The third feature vector includes N target center points;

[0025] Using the second feedforward neural network to query the third feature result Q L The process yields N first target detection boxes;

[0026] The N first target detection boxes are processed by the modulation position coding unit to obtain the position coding vector of the center point of the first target detection box, wherein the position coding vector includes height information and scale information;

[0027] By using concatenated units, the position encoding vector of the center point of each first target detection box and the corresponding third feature vector are concatenated to obtain the fourth feature vector of each target center point;

[0028] The second attention mechanism network is used to analyze the fourth feature vector and the second BEV feature map F of N target center points. Camera The fusion process is performed to obtain the second feature query result Q. LC .

[0029] Furthermore, the modulation position coding unit processes the N first target detection boxes to obtain the position coding vector of the center point of the first target detection box; including:

[0030] The first target detection box A q Transform to BEV space to obtain the target representation PE(A) in BEV space. q A q =(x q ,y q ,z q ,l q ,w q ,h q ); where, (x q ,y q ,z q ) represents the first target detection box A q The three-dimensional coordinates of the center point, l q ,w q and h qThe first object detection box A q The length, width, and height;

[0031] For PE(A) q ) Perform length, width, and height modulation to obtain the target modulation representation in BEV space Modulate(PE(A) q )):

[0032]

[0033] Among them, l q,ref ,w q,ref and h q,ref These are the reference length, reference width, and reference height; PE(x) q ) represents x q Values ​​converted to BEV space; PE(y) q ) represents y q Values ​​converted to BEV space; PE(z) q ) represents z q Values ​​converted to BEV space;

[0034] Modulate(PE(A) target modulation representation in BEV space using a fully connected network q )) Dimension expansion is performed to obtain the first object detection box A. q The center point location encoding vector P q .

[0035] Furthermore, the method also includes:

[0036] Using a third feedforward neural network to query the second feature result Q LC The process yields N second target detection boxes.

[0037] Furthermore, the method also includes the step of jointly training the three-dimensional backbone network, the two-dimensional backbone network, the convolutional layer, the SE channel attention mechanism network, the first feedforward neural network, the modulation position coding attention network, and the third feedforward neural network.

[0038] Secondly, embodiments of this application provide a multimodal bird's-eye view fusion device based on location encoding, including:

[0039] The acquisition unit is used to acquire point cloud data collected by LiDAR and multiple RGB images collected by multiple surround-view cameras of the target scene at the intersection.

[0040] The first feature extraction unit is used to process point cloud data using a pre-trained 3D backbone network to obtain a first feature map, and to perform high-dimensional feature compression on the first feature map to obtain a first BEV feature map.

[0041] The second feature extraction unit is used to process each RGB image to obtain a BEV feature map, and to superimpose the BEV feature maps of all RGB images to obtain a second BEV feature map.

[0042] The first processing unit is used to process the cascaded first BEV feature map and second BEV feature map to obtain a first feature query result, wherein the first feature query result includes the first feature vectors of multiple target center points.

[0043] The second processing unit is used to process the first feature query result, the first BEV feature map, and the second BEV feature map using a pre-trained modulation position coding attention network to obtain a second feature query result. The second feature query result includes second feature vectors of multiple target center points, and the dimension of the second feature vector is greater than the dimension of the first feature vector.

[0044] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the methods of embodiments of this application.

[0046] This application effectively integrates information from multiple sources, allowing the two types of data to complement each other and improving the accuracy of subsequent 3D target detection tasks. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 A technical roadmap for the location-encoded multimodal bird's-eye view fusion method provided in the embodiments of this application;

[0049] Figure 2 A flowchart illustrating the location-encoded multimodal bird's-eye view fusion method provided in this application embodiment;

[0050] Figure 3 A functional structure diagram of the location-encoded multimodal bird's-eye view fusion device provided in the embodiments of this application;

[0051] Figure 4 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0053] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0054] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0055] Roadside beacon perception technology in intelligent transportation refers to the real-time detection and tracking of traffic participants in complex scenarios such as intersections using technologies such as computers and sensors. This provides autonomous vehicles with more complete perception information, improving their driving capabilities. Roadside beacons utilize various sensors (such as LiDAR, cameras, and ultrasonic sensors) to acquire information about the surrounding environment, including road conditions and obstacle locations. With the continuous development and improvement of roadside beacon technology, it is expected to be widely used in urban transportation, logistics and distribution, and other fields, improving the safety level of autonomous vehicles.

[0056] As a fundamental function of roadside lighthouse systems, 3D target detection technology is a crucial component of environmental perception. Its main function is to acquire three-dimensional information about the surrounding environment through sensors and accurately detect and identify objects within it. Compared to traditional two-dimensional target detection technology, 3D target detection technology can more accurately locate and track targets, improving safety and stability.

[0057] From a hardware perspective, the data input for 3D object detection tasks mainly comes from two types of sensors: cameras and LiDAR. Cameras are sensors based on optical imaging principles. They capture image information from a scene to identify and locate objects, providing high-resolution, color-rich images. However, their performance may be limited at long distances or in low-light environments. LiDAR, on the other hand, is an active sensor that emits a laser beam and receives the reflected laser signal to obtain the position and shape information of objects in a scene. It has advantages such as high precision, high resolution, and strong anti-interference capabilities, making it suitable for target detection at long distances or in complex environments. However, because it requires emitting a laser beam, it may be limited in some special situations.

[0058] Both types of sensors are widely used in 3D object detection. However, both have significant drawbacks. Cameras typically can only capture objects within a certain distance, limiting their effectiveness at long distances or in complex environments. Furthermore, cameras usually have a fixed field of view, unlike LiDAR which can perform omnidirectional scanning, potentially missing coverage areas. In addition, their performance in low light or strong light can be affected, leading to image quality degradation or malfunction. For LiDAR, the primary issue is cost; compared to cameras, LiDAR hardware is more expensive and requires more energy and computing resources to operate. Secondly, because LiDAR needs to emit a laser beam and receive the reflected signal, it is typically larger than a camera, making it less portable and easier to install. Finally, LiDAR is sensitive to environmental interference; weather conditions such as wind and rain can affect its performance.

[0059] To address the insufficient detection capabilities when using only cameras or LiDAR, a combination of multiple sensors and data fusion is considered to improve environmental perception. However, LiDAR point clouds are three-dimensional data, while camera image data is two-dimensional data; they exist in different data spaces and cannot be easily fused. Currently, a common method is to simultaneously project point cloud data and visual images onto a bird's-eye view space for data fusion, but this method loses spatial height information and the scale information of objects themselves.

[0060] To address the aforementioned technical problems, this application provides a multimodal bird's-eye view fusion method based on location encoding, which can effectively provide height and scale information for fusion methods based on bird's-eye view features. This method first initializes feature queries, then fuses bird's-eye view features from LiDAR, and finally fuses bird's-eye view features from cameras, incorporating location encoding information. The resulting feature query is then fed into a detection head for 3D target detection. Verification has shown that this method can effectively improve the accuracy of 3D target detection on the roadside. The specific technical route is as follows: Figure 1 As shown.

[0061] Compared with existing technologies, this application effectively fuses multi-source information using an attention mechanism, enabling the two types of data to complement each other and improving the accuracy of 3D target detection. Since laser point cloud data and RGB images reside in different feature spaces, direct fusion is difficult. Therefore, they are first converted to a unified BEV space before feature fusion, and a position encoding module is used to add height and scale information. This method significantly improves the accuracy of detecting small targets such as pedestrians and cyclists.

[0062] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0063] like Figure 2 As shown in the figure, an embodiment of this application provides a multimodal bird's-eye view fusion method based on location encoding, which includes the following steps:

[0064] Step 101: Acquire point cloud data and multiple RGB images of the target scene at the intersection;

[0065] Point cloud data is acquired by LiDAR on roadside lighthouses, and multiple RGB images are acquired by multiple surround-view cameras on roadside lighthouses, typically six surround-view cameras are set up; the LiDAR on the roadside lighthouses and the six surround-view cameras have been calibrated and aligned.

[0066] Step 102: Process the point cloud data using the pre-trained 3D backbone network to obtain the first feature map, and perform height dimension feature compression on the first feature map to obtain the first BEV feature map.

[0067] In this embodiment, the point cloud data is segmented into 3D voxels, and then the points within each voxel are aggregated and their features extracted. Subsequently, the voxels are projected according to their height, and the voxel features at the same location in the Bird's Eye View (BEV) space are stitched together.

[0068] The raw point cloud data of LiDAR can be represented as a matrix of [N,4], where N is the number of point clouds in a frame of the scene, 4 represents the four-dimensional features of each point: [x,y,z,i], (x,y,z) represents the spatial coordinates of the point, and i represents the laser reflection intensity.

[0069] The first method involves using a 3D Backbone network to extract features from the point cloud data, and then compressing the height dimension features using a BEV encoder to obtain the bird's-eye view (BEV) feature map F of the point cloud data. LiDAR .

[0070] The second approach utilizes the VoxelNet network to obtain the BEV features of point cloud data. X, Yh and C LiDAR This represents the length, width, and number of channels for the LiDAR point cloud features in the BEV space.

[0071] Step 103: Process each RGB image to obtain a BEV feature map, and superimpose the BEV feature maps of all RGB images to obtain a second BEV feature map;

[0072] Each RGB image is processed to obtain a BEV feature map; including:

[0073] Each RGB image is processed using a pre-trained two-dimensional backbone network (ResNet50) to obtain a two-dimensional feature vector for each pixel of the RGB image;

[0074] Add a predicted depth value to the two-dimensional feature vector of each pixel to obtain the two-dimensional feature vector of each pixel;

[0075] The second feature map is obtained by projecting the three-dimensional feature vectors of all pixels onto the coordinate system of the point cloud data. Where X, Y and C Image The number of channels representing the length, width, and panoramic image features of the BEV space;

[0076] The second feature map is subjected to high-dimensional feature compression to obtain a BEV feature map of an RGB image from a surround-view camera.

[0077] The BEV feature maps of all RGB images are superimposed to obtain the second BEV feature map. C Camera This represents the number of channels in the second BEV feature map.

[0078] Step 104: Process the cascaded first BEV feature map and second BEV feature map to obtain the first feature query result, which includes the first feature vectors of multiple target center points.

[0079] This step specifically includes:

[0080] The first and second BEV feature maps of the cascade are processed using pre-trained convolutional layers to obtain the BEV feature map F1.

[0081] F1 = conv(concat(F) LiDAR ,F Camera ))

[0082] Among them, concat(F Camera ,F LiDARF is the first BEV feature map in the cascade. LiDAR Second BEV feature map F Camera ;conv() represents a convolutional layer;

[0083] The BEV feature map F1 is processed using a pre-trained SE channel attention mechanism network (Squeeze-and-Excitation Block) to obtain the fused BEV feature map F. fused :

[0084] F fused =σ(conv(pool(F1)))·F1

[0085] Where pool(F1) represents the result of F1 after passing through the pooling layer; σ represents the normalized softmax function;

[0086] The first feedforward neural network, which has been pre-trained, is used to fuse the BEV feature map F. fused The process yields multiple target center points and their classification results.

[0087] Obtain the probability values ​​of generating prediction boxes from multiple target center points, sort the probability values ​​in descending order, and obtain the target center points corresponding to the top N probability values;

[0088] Extract the feature vector of the target center point from the first BEV feature map as the first feature vector of the target center point, and combine the first feature vectors of N target center points to form the first feature query result Q. LiDAR .

[0089] In this embodiment of the application, a ResNet+feature pyramid network can be designed, with multiple target center points as input and the output being the probability value of generating a prediction box for each target center point, which is used to obtain the probability value of generating prediction boxes for multiple target center points.

[0090] Step 105: Use the pre-trained Modulation Position Encoding Attention Network to process the first feature query result, the first BEV feature map, and the second BEV feature map to obtain the second feature query result. The second feature query result includes the second feature vectors of multiple target center points, and the dimension of the second feature vector is greater than the dimension of the first feature vector.

[0091] In this embodiment of the application, the Modulated Position Encoder Transformer (MPET) includes: a first attention mechanism network, a modulation position coding unit, a cascaded unit, and a second attention mechanism network;

[0092] The first feature query result, the first BEV feature map, and the second BEV feature map are processed using a pre-trained modulation position coding attention network to obtain a fused feature map based on the feature query; including:

[0093] Using a first attention mechanism network to query the first feature result Q LiDAR and the first BEV feature map F LiDAR The third feature query result Q is obtained by performing fusion processing. L The third feature query result Q L The third feature vector includes N target center points;

[0094] Using the second feedforward neural network to query the third feature result Q L The process yields N first target detection boxes;

[0095] The modulation position coding unit is used to process N first target detection boxes to obtain the position coding vector of the center point of the first target detection box. The position coding vector includes height information and scale information.

[0096] The cascaded unit concatenates the position encoding vector of the center point of each first target detection box and the corresponding third feature vector to obtain the fourth feature vector of each target center point.

[0097] The second attention mechanism network is used to analyze the fourth feature vector and the second BEV feature map F of N target center points. Camera The fusion process is performed to obtain the second feature query result Q. LC .

[0098] It should be noted that the hardware and software platform for building, training, and testing the MPET network is Ubuntu 16.04 + RTX 3090 + PyTorch 1.7 + CUDA 11.1 + Python 3.7.

[0099] Both the first and second attention mechanism networks are transformer networks, employing a Spatially Modulated Co-Attention (SMCA) module. The Object Query Modulated Position Encoder unit adds height and scale information to the N first object detection boxes generated by the first transformer network.

[0100] Specifically, the modulation position coding unit processes N first target detection boxes to obtain the position coding vector of the center point of the first target detection box; including:

[0101] The first target detection box A q Transform to BEV space to obtain the target representation PE(A) in BEV space. q A q =(x q ,y q ,z q ,l q ,w q ,h q ); where, (x q ,y q ,z q ) represents the first target detection box A q The three-dimensional coordinates of the center point, l q ,w q and h q The first object detection box A q The length, width, and height;

[0102] PE(A q For the first target detection box A q The center point coordinates, length, width, and height are scaled to fit the proportions of the BEV space. This is because the BEV space is typically a regular plane, and a unit distance in the BEV space may correspond to different scales in the real world. Therefore, the original 3D coordinates need to be adjusted accordingly to match the proportions of the BEV features.

[0103] In this application, different target categories may have different scales, and even objects of the same category may have different scales. In this case, for PE(A)... q The purpose of length, width, and height modulation is to obtain scale information between the actual length, width, and height and the reference length, width, and height; this will help feature queries better adapt to objects of different sizes, thereby improving the robustness of object detection.

[0104] For PE(A) q ) Perform length, width, and height modulation to obtain the target modulation representation in BEV space Modulate(PE(A) q )):

[0105]

[0106] Among them, l q,ref ,w q,ref and h q,ref The reference length, width, and height are calculated using the following formula:

[0107] l q,ref ,w q,ref ,h q,ref =σ(MLP(Q) L ))

[0108] Among them, MLP(Q L ) represents Q L The output obtained through a fully connected network; σ represents the normalized softmax function;

[0109] PE(x q ) represents x q Values ​​converted to BEV space; PE(y) q ) represents y q Values ​​converted to BEV space; PE(z) q ) represents z q Values ​​converted to BEV space;

[0110] Expanding the target modulation representation in the BEV space using fully connected networks: Modulate(PE(A)) q The first object detection box A is obtained by considering the dimensions of the bounding box. q The center point location encoding vector P q .

[0111] In some embodiments, the method further includes: using a pre-trained third feedforward neural network to process the second feature query result Q. LC The process yields N second target detection boxes.

[0112] Compared to the first object detection box, due to the second feature query result Q LC The second object detection box contains richer and more accurate target information, therefore, it is more accurate than the first object detection box.

[0113] In some embodiments, the method further includes the step of jointly training a three-dimensional backbone network, a two-dimensional backbone network, a convolutional layer, an SE channel attention mechanism network, a first feedforward neural network, a modulation position coding attention network, and a third feedforward neural network.

[0114] Based on the same inventive concept, this application provides a multimodal bird's-eye view fusion device based on location encoding, see reference. Figure 3 As shown, the location-encoded multimodal bird's-eye view fusion device 200 provided in this application embodiment includes at least:

[0115] The acquisition unit 201 is used to acquire point cloud data collected by the LiDAR and multiple RGB images collected by multiple surround-view cameras of the target scene at the intersection.

[0116] The first feature extraction unit 202 is used to process point cloud data using a pre-trained three-dimensional backbone network to obtain a first feature map, and to perform high-dimensional feature compression on the first feature map to obtain a first BEV feature map.

[0117] The second feature extraction unit 203 is used to process each RGB image to obtain a BEV feature map, and to superimpose the BEV feature maps of all RGB images to obtain a second BEV feature map.

[0118] The first processing unit 204 is used to process the cascaded first BEV feature map and second BEV feature map to obtain a first feature query result, wherein the first feature query result includes the first feature vectors of multiple target center points.

[0119] The second processing unit 205 is used to process the first feature query result, the first BEV feature map, and the second BEV feature map using a pre-trained modulation position coding attention network to obtain a second feature query result. The second feature query result includes second feature vectors of multiple target center points, and the dimension of the second feature vector is greater than the dimension of the first feature vector.

[0120] It should be noted that the principle of the multimodal bird's-eye view fusion device 200 based on location coding provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the multimodal bird's-eye view fusion device 200 based on location coding for roadside lighthouses provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0121] Based on the above embodiments, this application also provides an electronic device, see below. Figure 4 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, it implements the location-encoded multimodal bird's-eye view fusion method provided in this application embodiment.

[0122] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0123] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0124] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0125] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 4 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 4 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0126] It should be noted that, Figure 4 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0127] This application also provides a computer-readable storage medium storing computer instructions. When executed by a processor, these computer instructions implement the location-encoded multimodal bird's-eye view fusion method provided in this application. Specifically, the executable program can be built into or installed in an electronic device 300, so that the electronic device 300 can implement the location-encoded multimodal bird's-eye view fusion method provided in this application by executing the built-in or installed executable program.

[0128] The vehicle re-identification method provided in this application embodiment can also be implemented as a program product, which includes program code. When the program product can run on the electronic device 300, the program code is used to enable the electronic device 300 to execute the location-encoded multimodal bird's-eye view fusion method provided in this application embodiment.

[0129] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0130] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0131] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0132] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A multimodal bird's-eye view fusion method based on location encoding, characterized in that, include: Acquire point cloud data and multiple RGB images of the target scene at the intersection; The point cloud data is processed using a pre-trained 3D backbone network to obtain the first feature map. The first feature map is then compressed to obtain the first BEV feature map. Each RGB image is processed to obtain a BEV feature map. The BEV feature maps of all RGB images are superimposed to obtain a second BEV feature map. The first BEV feature map and the second BEV feature map of the cascade are processed to obtain the first feature query result, which includes the first feature vector of multiple target center points. The first feature query result, the first BEV feature map, and the second BEV feature map are processed using a pre-trained modulation position coding attention network to obtain the second feature query result. The second feature query result includes the second feature vector of multiple target center points, and the dimension of the second feature vector is greater than the dimension of the first feature vector. The modulation position coding attention network includes: a first attention mechanism network, a modulation position coding unit, a cascaded unit, and a second attention mechanism network; The first feature query result, the first BEV feature map, and the second BEV feature map are processed using a pre-trained modulation position coding attention network to obtain the second feature query result, which includes: Using the first attention mechanism network to query the first feature and the first BEV feature map The third feature query result is obtained by performing a fusion process. Third feature query results The third feature vector includes N target center points; Using the second feedforward neural network to query the third feature results The process yields N first target detection boxes; The N first target detection boxes are processed by the modulation position coding unit to obtain the position coding vector of the center point of the first target detection box, wherein the position coding vector includes height information and scale information; By using concatenated units, the position encoding vector of the center point of each first target detection box and the corresponding third feature vector are concatenated to obtain the fourth feature vector of each target center point; The second attention mechanism network is used to analyze the fourth feature vector and the second BEV feature map of N target center points. The fusion process is performed to obtain the second feature query result. .

2. The multimodal bird's-eye view fusion method based on location encoding according to claim 1, characterized in that, Each RGB image is processed to obtain a BEV feature map; including: Each RGB image is processed using a pre-trained two-dimensional backbone network to obtain a two-dimensional feature vector for each pixel of the RGB image; Add a predicted depth value to the two-dimensional feature vector of each pixel to obtain the two-dimensional feature vector of each pixel; The second feature map is obtained by projecting the three-dimensional feature vectors of all pixels onto the coordinate system of the point cloud data. The second feature map is subjected to high-dimensional feature compression to obtain the BEV feature map.

3. The multimodal bird's-eye view fusion method based on location encoding according to claim 2, characterized in that, The first and second BEV feature maps of the cascade are processed to obtain the first feature query result, including: The first and second BEV feature maps are processed using pre-trained convolutional layers to obtain the BEV feature map. ; Utilizing a pre-trained SE channel attention mechanism network to analyze BEV feature maps The process is performed to obtain the fused BEV feature map. ; The fused BEV feature map is obtained by using a pre-trained first feedforward neural network. The process yields multiple target center points and their classification results. Obtain the probability values ​​of generating prediction boxes from multiple target center points, sort the probability values ​​in descending order, and obtain the target center points corresponding to the top N probability values; Extract the feature vector of the target center point from the first BEV feature map as the first feature vector of the target center point, and combine the first feature vectors of N target center points to form the first feature query result. .

4. The multimodal bird's-eye view fusion method based on location encoding according to claim 3, characterized in that, The modulation position coding unit processes N first target detection boxes to obtain the position coding vector of the center point of the first target detection box; including: The first target detection box Transform to BEV space to obtain the target representation in BEV space. ; ;in, The first target detection box The three-dimensional coordinates of the center point and It is the first object detection box The length, width, and height; right Length, width, and height modulation are performed to obtain the target modulation representation in the BEV space. : in, and These are the reference length, reference width, and reference height; Indicate Values ​​converted to BEV space; Indicate Values ​​converted to BEV space; express Values ​​converted to BEV space; Target modulation representation in BEV space using fully connected networks Dimension expansion is performed to obtain the first object detection box. The position encoding vector of the center point .

5. The multimodal bird's-eye view fusion method based on location encoding according to claim 4, characterized in that, The method further includes: The second feature query result is obtained by using a pre-trained third feedforward neural network. The process yields N second target detection boxes.

6. The method according to claim 5, characterized in that, The method further includes the step of jointly training a three-dimensional backbone network, a two-dimensional backbone network, a convolutional layer, an SE channel attention mechanism network, a first feedforward neural network, a modulation position coding attention network, and a third feedforward neural network.

7. A multimodal bird's-eye view fusion device based on location encoding, characterized in that, include: The acquisition unit is used to acquire point cloud data collected by LiDAR and multiple RGB images collected by multiple surround-view cameras of the target scene at the intersection. The first feature extraction unit is used to process point cloud data using a pre-trained 3D backbone network to obtain a first feature map, and to perform high-dimensional feature compression on the first feature map to obtain a first BEV feature map. The second feature extraction unit is used to process each RGB image to obtain a BEV feature map, and to superimpose the BEV feature maps of all RGB images to obtain a second BEV feature map. The first processing unit is used to process the cascaded first BEV feature map and second BEV feature map to obtain a first feature query result, wherein the first feature query result includes the first feature vectors of multiple target center points. The second processing unit is used to process the first feature query result, the first BEV feature map, and the second BEV feature map using a pre-trained modulation position coding attention network to obtain the second feature query result. The second feature query result includes second feature vectors of multiple target center points, and the dimension of the second feature vector is greater than the dimension of the first feature vector. The modulation position coding attention network includes: a first attention mechanism network, a modulation position coding unit, a cascaded unit, and a second attention mechanism network; The first feature query result, the first BEV feature map, and the second BEV feature map are processed using a pre-trained modulation position coding attention network to obtain the second feature query result, which includes: Using the first attention mechanism network to query the first feature and the first BEV feature map The third feature query result is obtained by performing a fusion process. Third feature query results The third feature vector includes N target center points; Using the second feedforward neural network to query the third feature results The process yields N first target detection boxes; The N first target detection boxes are processed by the modulation position coding unit to obtain the position coding vector of the center point of the first target detection box, wherein the position coding vector includes height information and scale information; By using concatenated units, the position encoding vector of the center point of each first target detection box and the corresponding third feature vector are concatenated to obtain the fourth feature vector of each target center point; The second attention mechanism network is used to analyze the fourth feature vector and the second BEV feature map of N target center points. The fusion process is performed to obtain the second feature query result. .

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-6.

Images