Multi-modal aerial view fusion method and device based on position coding

Abstract

The invention provides a multi-modal aerial view fusion method and device based on position coding. The method comprises the following steps: acquiring point cloud data and a plurality of RGB images of an intersection target scene; processing the point cloud data by using a three-dimensional backbone network to obtain a first feature map, and performing height dimension feature compression on the first feature map to obtain a first BEV feature map; processing each RGB image to obtain a BEV feature map, and superposing the BEV feature maps of all the RGB images to obtain a second BEV feature map; processing the cascaded first BEV feature map and second BEV feature map by using a feature query model to obtain a first feature query result; and processing the first feature query result, the first BEV feature map and the second BEV feature map by using a modulation position coding attention network to obtain a second feature query result. According to the method, multi-source information is effectively fused, the advantages of two kinds of data are complementary, and the precision of a subsequent 3D target detection task is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a multimodal bird's-eye view fusion method and device based on position coding. Background Art

[0002] Roadside lighthouses are usually equipped with visual cameras and lidars to collect images and point cloud data of intersection scenes. If only images are used for target detection, when conditions such as lighting change and image information is lost, the detection results will be seriously wrong. In addition, the lack of spatial distance information in the image also limits its detection capabilities. Although point cloud data can overcome the problem of lighting changes, due to the sparsity and disorder of point clouds, lidar alone is not enough for robust 3D target detection.

[0003] In order to solve the problem of insufficient detection capability when using only cameras or lidar, it is considered to combine multiple sensors and perform data fusion to improve the perception of the environment. However, the lidar point cloud is three-dimensional data, and the camera image data is two-dimensional data. The two are in different data spaces and cannot be simply fused. The commonly used method is to project the point cloud data and the visual image into the bird's-eye view space at the same time for data fusion, but this method loses the height information in the space and the scale information of the object itself. Summary of the invention

[0004] In view of this, the present application provides a multimodal bird's-eye view fusion method and device based on position coding to solve the above technical problems.

[0005] In a first aspect, an embodiment of the present application provides a multimodal bird's-eye view fusion method based on position coding, comprising:

[0006] Obtain point cloud data and multiple RGB images of the intersection target scene;

[0007] The point cloud data is processed using a pre-trained three-dimensional backbone network to obtain a first feature map, and the first feature map is highly dimensionally compressed to obtain a first BEV feature map;

[0008] Process each RGB image to obtain a BEV feature map, and superimpose the BEV feature maps of all RGB images to obtain a second BEV feature map;

[0009] Processing the cascaded first BEV feature map and the second BEV feature map to obtain a first feature query result, wherein the first feature query result includes first feature vectors of multiple target center points;

[0010] The first feature query result, the first BEV feature map and the second BEV feature map are processed using a pre-trained modulation position coding attention network to obtain a second feature query result, wherein the second feature query result includes a second feature vector of multiple target center points, and the dimension of the second feature vector is greater than that of the first feature vector.

[0011] Furthermore, each RGB image is processed to obtain a BEV feature map; including:

[0012] Each RGB image is processed using a pre-trained two-dimensional backbone network to obtain a two-dimensional feature vector for each pixel of the RGB image;

[0013] Add the predicted depth value to the two-dimensional feature vector of each pixel to obtain the two-dimensional feature vector of each pixel;

[0014] Project the three-dimensional feature vectors of all pixels to the coordinate system where the point cloud data is located to obtain a second feature map;

[0015] The second feature map is subjected to high-dimensional feature compression to obtain a BEV feature map.

[0016] Further, the cascaded first BEV feature graph and the second BEV feature graph are processed to obtain a first feature query result; including:

[0017] The concatenated first BEV feature map and the second BEV feature map are processed using the pre-trained convolutional layer to obtain a BEV feature map F1;

[0018] The BEV feature map F1 is processed using the pre-trained SE channel attention mechanism network to obtain the fused BEV feature map F fused ;

[0019] The first feedforward neural network trained in advance is used to fusion BEV feature map F fused Processing is performed to obtain multiple target center points and their classification results;

[0020] Obtain the probability values ​​of generating prediction boxes for multiple target center points, sort the probability values ​​in descending order, and obtain the target center points corresponding to the first N probability values;

[0021] Extract the feature vector of the target center point in the first BEV feature map as the first feature vector of the target center point, and combine the first feature vectors of the N target center points into a first feature query result Q LiDAR .

[0022] Furthermore, the modulation position coding attention network includes: a first attention mechanism network, a modulation position coding unit, a cascade unit and a second attention mechanism network;

[0023] The first feature query result, the first BEV feature map and the second BEV feature map are processed by using the pre-trained modulation position coding attention network to obtain a fusion feature map based on the feature query; including:

[0024] Use the first attention mechanism network to query the first feature result Q LiDAR and the first BEV characteristic diagram F LiDAR Perform fusion processing to obtain the third feature query result Q L , the third feature query result Q L A third eigenvector including N target center points;

[0025] The second feedforward neural network is used to query the third feature result Q L Processing is performed to obtain N first target detection frames;

[0026] Processing the N first target detection frames using a modulation position coding unit to obtain a position coding vector of a center point of the first target detection frame, wherein the position coding vector includes height information and scale information;

[0027] Using a cascade unit, cascade the position encoding vector of the center point of each first target detection frame and the corresponding third feature vector to obtain a fourth feature vector of the center point of each target;

[0028] The fourth feature vector and the second BEV feature map F of the N target center points are analyzed using the second attention mechanism network Camera Perform fusion processing to obtain the second feature query result Q LC .

[0029] Further, the N first target detection frames are processed by a modulation position coding unit to obtain a position coding vector of a center point of the first target detection frame; including:

[0030] The first target detection box A q Convert to BEV space and get the target representation PE(A) in BEV space q );A q =(x q ,y q ,z q ,l q ,w q ,h q ); where (x q ,y q ,z q ) is the first target detection box A q The three-dimensional coordinates of the center point, l q ,w q and h qis the first target detection box A q length, width and height;

[0031] For PE(A q ) to modulate the length, width and height, and obtain the target modulation representation Modulate(PE(A q )):

[0032]

[0033] Among them, l q,ref ,w q,ref and h q,ref is the reference length, reference width and reference height; PE(x q ) represents x q Converted to the value in BEV space; PE(y q ) represents y q Converted to the value in BEV space; PE(z q ) represents z q Convert to the value in BEV space;

[0034] The fully connected network is used to modulate the target in the BEV space. q )) Dimension expansion is performed to obtain the first target detection frame A q The position encoding vector P of the center point q .

[0035] Furthermore, the method further comprises:

[0036] The third feedforward neural network is used to query the second feature result Q LC Processing is performed to obtain N second target detection frames.

[0037] Furthermore, the method also includes: a step of jointly training a three-dimensional backbone network, a two-dimensional backbone network, a convolutional layer, an SE channel attention mechanism network, a first feedforward neural network, a modulation position coding attention network and a third feedforward neural network.

[0038] In a second aspect, an embodiment of the present application provides a multimodal bird's-eye view fusion device based on position coding, comprising:

[0039] An acquisition unit, used to acquire point cloud data collected by a laser radar of a target scene at an intersection and multiple RGB images collected by multiple surround view cameras;

[0040] A first feature extraction unit is used to process the point cloud data using a pre-trained three-dimensional backbone network to obtain a first feature map, and perform high-dimensional feature compression on the first feature map to obtain a first BEV feature map;

[0041] A second feature extraction unit is used to process each RGB image to obtain a BEV feature map, and superimpose the BEV feature maps of all RGB images to obtain a second BEV feature map;

[0042] A first processing unit is used to process the cascaded first BEV feature map and the second BEV feature map to obtain a first feature query result, wherein the first feature query result includes first feature vectors of multiple target center points;

[0043] The second processing unit is used to use the pre-trained modulation position coding attention network to process the first feature query result, the first BEV feature map and the second BEV feature map to obtain a second feature query result, wherein the second feature query result includes a second feature vector of multiple target center points, and the dimension of the second feature vector is greater than the dimension of the first feature vector.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions implement the method of the embodiment of the present application when executed by a processor.

[0046] This application effectively integrates multi-source information, making the two types of data complement each other and improving the accuracy of subsequent 3D object detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A technical roadmap for a multimodal bird's-eye view fusion method based on position coding provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of a multimodal bird's-eye view fusion method based on position coding provided in an embodiment of the present application;

[0050] Figure 3 A functional structure diagram of a multi-modal bird's-eye view fusion device based on position coding provided in an embodiment of the present application;

[0051] Figure 4 A functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0053] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0054] First, the design concept of the embodiments of the present application is briefly introduced.

[0055] Roadside lighthouse perception technology for intelligent transportation refers to the real-time detection and tracking of traffic participants in complex scenarios such as intersections through computers, sensors and other technical means, providing more complete perception information for autonomous vehicles and improving the driving ability of vehicles. Roadside lighthouses use various sensors (such as lidar, cameras, ultrasonic sensors, etc.) to obtain information about the surrounding environment, including road conditions, obstacle locations, etc. With the continuous development and improvement of roadside lighthouse technology, roadside lighthouses are expected to be widely used in urban transportation, logistics and distribution, and improve the safety level of autonomous vehicles.

[0056] As one of the basic functions of the roadside lighthouse system, 3D target detection technology is an extremely important part of environmental perception. Its main function is to obtain three-dimensional information of the surrounding environment through sensors and accurately detect and identify objects in it. Compared with traditional two-dimensional target detection technology, 3D target detection technology can locate and track targets more accurately, improving safety and stability.

[0057] From a hardware perspective, data input for 3D target detection tasks mainly comes from two types of sensors: cameras and lidar. A camera is a sensor based on the principle of optical imaging. It recognizes and locates objects by capturing image information in the scene. It can provide high-resolution, colorful image information, but its performance may be limited at long distances or in low-light environments. LiDAR is an active sensor that obtains the position and shape information of objects in the scene by emitting laser beams and receiving reflected laser signals. It has the advantages of high precision, high resolution, and strong anti-interference, and is suitable for target detection at long distances or in complex environments. However, since it needs to emit laser beams, it may be limited in some special occasions.

[0058] Both sensors are widely used in 3D object detection. However, both sensors have obvious defects. For cameras, they can usually only capture objects within a certain distance range, so the detection effect may be limited at long distances or in complex environments. At the same time, the camera's viewing angle is usually fixed and cannot perform all-round scanning like LiDAR, so it may not cover certain areas. In addition, the performance under low light or strong light may be affected, resulting in degraded image quality or malfunction. For LiDAR, the first is the cost issue. Compared with cameras, LiDAR hardware costs more and requires more energy and computing resources to run. Secondly, because LiDAR needs to emit laser beams and receive reflected signals, its size is usually larger than that of cameras, which is not easy to carry and install. Moreover, LiDAR is sensitive to environmental interference. For example, weather conditions such as wind and rain may affect its performance.

[0059] In order to solve the problem of insufficient detection capability when using only cameras or lidar, it is considered to combine multiple sensors and perform data fusion to improve the perception of the environment. However, the lidar point cloud is three-dimensional data, and the camera image data is two-dimensional data. The two are in different data spaces and cannot be simply fused. The commonly used method is to project the point cloud data and the visual image into the bird's-eye view space at the same time for data fusion, but this method loses the height information in the space and the scale information of the object itself.

[0060] In order to solve the above technical problems, the present application provides a multimodal bird's-eye view fusion method based on position coding, which can effectively provide height information and scale information for the fusion method based on bird's-eye view features; the method first initializes the feature query, then fuses the bird's-eye view features of the lidar, and finally fuses the bird's-eye view features of the camera, and adds the position coding information, and finally sends the obtained feature query to the detection head for 3D target detection. It has been verified that the method of the present application can effectively improve the 3D target detection accuracy of the roadside. The specific technical route is as follows: Figure 1 shown.

[0061] Compared with the existing technology, this application uses the attention mechanism to effectively fuse multi-source information, making the two types of data complement each other and improving the accuracy of 3D target detection. Since the laser point cloud data and RGB images are in different feature spaces and difficult to fuse directly, they are first converted to a unified BEV space and then feature fusion is performed. At the same time, a position encoding module is used to increase height information and scale information. This method has significantly improved the accuracy of detecting small targets such as pedestrians and cyclists.

[0062] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0063] like Figure 2 As shown, the embodiment of the present application is a multimodal bird's-eye view fusion method based on position coding, comprising the following steps:

[0064] Step 101: Obtaining point cloud data and multiple RGB images of the intersection target scene;

[0065] Among them, the point cloud data is collected by the laser radar on the roadside lighthouse, and multiple RGB images are collected by multiple surround-view cameras on the roadside lighthouse. Usually six surround-view cameras are set; the laser radar on the roadside lighthouse and the six surround-view cameras have been calibrated and aligned.

[0066] Step 102: Process the point cloud data using a pre-trained three-dimensional backbone network to obtain a first feature map, and perform high-dimensional feature compression on the first feature map to obtain a first BEV feature map;

[0067] In the embodiment of the present application, the point cloud data is segmented into 3D voxels, and then the points in each voxel are aggregated and feature extracted. Afterwards, the voxels are projected according to the height, and the voxel features at the same position in the Bird's Eye View (BEV) space are spliced.

[0068] The raw point cloud data of the lidar can be represented as a [N,4] matrix, where N is the number of point clouds in a frame of the scene, and 4 represents the four-dimensional features of each point: [x, y, z, i], (x, y, z) represents the spatial coordinates of the point, and i represents the laser reflection intensity.

[0069] The first method is to use the 3D Backbone network to extract the features of the point cloud data, and then use the BEV encoder to compress the height dimension features to obtain the bird's-eye view (BEV) feature map F of the point cloud data. LiDAR .

[0070] The second method is to use the voxelnet network to obtain the BEV features of point cloud data X, Yh and C LiDAR Represents the length, width of the BEV space and the number of channels of the lidar point cloud features.

[0071] Step 103: Process each RGB image to obtain a BEV feature map, and superimpose the BEV feature maps of all RGB images to obtain a second BEV feature map;

[0072] Among them, each RGB image is processed to obtain a BEV feature map; including:

[0073] Each RGB image is processed using a pre-trained two-dimensional backbone network (ResNet50) to obtain a two-dimensional feature vector for each pixel of the RGB image;

[0074] Add the predicted depth value to the two-dimensional feature vector of each pixel to obtain the two-dimensional feature vector of each pixel;

[0075] Project the three-dimensional feature vectors of all pixels to the coordinate system where the point cloud data is located to obtain the second feature map Among them, X, Y and C Image Represents the length, width and number of channels of surround image features in the BEV space;

[0076] The second feature map is subjected to high-dimensional feature compression to obtain a BEV feature map of the RGB image of the surround camera.

[0077] Superimpose the BEV feature maps of all RGB images to obtain the second BEV feature map C Camera is the number of channels of the second BEV feature map.

[0078] Step 104: Processing the cascaded first BEV feature map and the second BEV feature map to obtain a first feature query result, wherein the first feature query result includes first feature vectors of multiple target center points;

[0079] This step specifically includes:

[0080] The concatenated first BEV feature map and the second BEV feature map are processed using the pre-trained convolutional layer to obtain a BEV feature map F1;

[0081] F1 = conv(concat(F LiDAR ,F Camera ))

[0082] Among them, concat(F Camera ,F LiDAR) is the first BEV characteristic diagram F of the cascade LiDAR and the second BEV characteristic diagram F Camera ;conv() represents the convolutional layer;

[0083] The BEV feature map F1 is processed using the pre-trained SE channel attention mechanism network (Squeeze-and-Excitation Block) to obtain the fused BEV feature map F fused :

[0084] F fused =σ(conv(pool(F1)))·F1

[0085] Where pool(F1) represents the result after F1 passes through the pooling layer; σ represents the normalized softmax function;

[0086] The first feedforward neural network trained in advance is used to fusion BEV feature map F fused Processing is performed to obtain multiple target center points and their classification results;

[0087] Obtain the probability values ​​of generating prediction boxes for multiple target center points, sort the probability values ​​in descending order, and obtain the target center points corresponding to the first N probability values;

[0088] Extract the feature vector of the target center point in the first BEV feature map as the first feature vector of the target center point, and combine the first feature vectors of the N target center points into a first feature query result Q LiDAR .

[0089] In an embodiment of the present application, a ResNet+feature pyramid network can be designed, with multiple target center points as input and a probability value of generating a prediction box for each target center point as output, which is used to obtain the probability values ​​of generating prediction boxes for multiple target center points.

[0090] Step 105: Use the pre-trained modulation position coding attention network to process the first feature query result, the first BEV feature map and the second BEV feature map to obtain a second feature query result, wherein the second feature query result includes a second feature vector of multiple target center points, and the dimension of the second feature vector is greater than the dimension of the first feature vector.

[0091] In an embodiment of the present application, the modulated position encoding attention network (Modulated PositionEncoder Transformer, MPET) includes: a first attention mechanism network, a modulated position encoding unit, a cascade unit and a second attention mechanism network;

[0092] The first feature query result, the first BEV feature map and the second BEV feature map are processed by using the pre-trained modulation position coding attention network to obtain a fusion feature map based on the feature query; including:

[0093] Use the first attention mechanism network to query the first feature result Q LiDAR and the first BEV characteristic diagram F LiDAR Perform fusion processing to obtain the third feature query result Q L , the third feature query result Q L A third eigenvector including N target center points;

[0094] The second feedforward neural network is used to query the third feature result Q L Processing is performed to obtain N first target detection frames;

[0095] Processing the N first target detection frames using a modulation position coding unit to obtain a position coding vector of a center point of the first target detection frame, where the position coding vector includes height information and scale information;

[0096] The cascade unit cascades the position encoding vector of the center point of each first target detection frame and the corresponding third feature vector to obtain a fourth feature vector of the center point of each target;

[0097] The fourth feature vector and the second BEV feature map F of the N target center points are analyzed using the second attention mechanism network Camera Perform fusion processing to obtain the second feature query result Q LC .

[0098] It should be noted that the hardware and software platform for building, training and testing the MPET network is Ubuntu16.04+RTX3090+pytorch1.7+cuda11.1+python3.7.

[0099] The first attention mechanism network and the second attention mechanism network are both transformer networks, using the Spatially Modulated Co-Attention (SMCA) module. The object query-based modulated position encoder (ObjectQuery Modulated Position Encoder) unit adds height information and scale information to the N first target detection boxes generated by the first transformer network.

[0100] Specifically, the N first target detection frames are processed by a modulation position coding unit to obtain a position coding vector of a center point of the first target detection frame; including:

[0101] The first target detection box A q Convert to BEV space and get the target representation PE(A) in BEV space q );A q =(x q ,y q ,z q ,l q ,w q ,h q ); where (x q ,y q ,z q ) is the first target detection box A q The three-dimensional coordinates of the center point, l q ,w q and h q is the first target detection box A q length, width and height;

[0102] PE(A q ) for the first target detection box A q The center point coordinates and length, width and height are scaled to fit the scale of the BEV space. This is because the BEV space is usually a regular plane, and a unit of distance in the BEV space may correspond to different scales in the real world, so the original 3D coordinates need to be adjusted accordingly to match the scale of the BEV features.

[0103] In the implementation of this application, different target categories may have different scales, and objects of the same category may also have different scales. q ) is used to modulate the length, width and height in order to obtain the scale information between the actual length, width and height and the reference length, width and height; this will help feature queries better adapt to objects of different sizes, thereby improving the robustness of target detection.

[0104] PE(A q ) to modulate the length, width and height, and obtain the target modulation representation Modulate(PE(A q )):

[0105]

[0106] Among them, l q,ref ,w q,ref and h q,ref are the length, width and height of the reference, calculated as:

[0107] l q,ref ,w q,ref ,h q,ref =σ(MLP(Q L ))

[0108] Among them, MLP (Q L ) indicates Q L The output result obtained by the fully connected network; σ represents the normalized softmax function;

[0109] PE(x q ) represents x q Converted to the value in BEV space; PE(y q ) represents y q Converted to the value in BEV space; PE(z q ) represents z q Convert to the value in BEV space;

[0110] Using fully connected networks to expand the target modulation representation Modulate(PE(A) in BEV space q )) dimension, and obtain the first target detection frame A q The position encoding vector P of the center point q .

[0111] In some embodiments, the method further comprises: using a pre-trained third feedforward neural network to query the second feature result Q LC Processing is performed to obtain N second target detection frames.

[0112] Compared with the first target detection box, due to the second feature query result Q LC It contains richer and more precise target information. Therefore, the second target detection frame is more accurate than the first target detection frame.

[0113] In some embodiments, the method further includes: a step of jointly training a three-dimensional backbone network, a two-dimensional backbone network, a convolutional layer, an SE channel attention mechanism network, a first feedforward neural network, a modulated position coding attention network, and a third feedforward neural network.

[0114] Based on the same inventive idea, the embodiment of the present application provides a multi-modal bird's-eye view fusion device based on position coding, see Figure 3 As shown, the multimodal bird's-eye view fusion device 200 based on position coding provided in the embodiment of the present application at least includes:

[0115] An acquisition unit 201 is used to acquire point cloud data collected by a laser radar and multiple RGB images collected by multiple surround view cameras of a target scene at an intersection;

[0116] A first feature extraction unit 202 is used to process the point cloud data using a pre-trained three-dimensional backbone network to obtain a first feature map, and perform high-dimensional feature compression on the first feature map to obtain a first BEV feature map;

[0117] The second feature extraction unit 203 is used to process each RGB image to obtain a BEV feature map, and to superimpose the BEV feature maps of all RGB images to obtain a second BEV feature map;

[0118] A first processing unit 204 is used to process the cascaded first BEV feature map and the second BEV feature map to obtain a first feature query result, wherein the first feature query result includes first feature vectors of multiple target center points;

[0119] The second processing unit 205 is used to use the pre-trained modulation position coding attention network to process the first feature query result, the first BEV feature map and the second BEV feature map to obtain a second feature query result, wherein the second feature query result includes a second feature vector of multiple target center points, and the dimension of the second feature vector is greater than the dimension of the first feature vector.

[0120] It should be noted that the principle of solving the technical problem by the multimodal bird's-eye view fusion device 200 based on position coding provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the multimodal bird's-eye view fusion device 200 based on position coding of roadside lighthouses provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0121] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 4 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the multimodal bird's-eye view fusion method based on position coding provided in the embodiment of the present application is implemented.

[0122] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0123] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0124] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0125] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 4 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0126] It should be noted that Figure 4 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0127] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the multimodal bird's-eye view fusion method based on position coding provided in the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the multimodal bird's-eye view fusion method based on position coding provided in the embodiment of the present application by executing the built-in or installed executable program.

[0128] The vehicle re-identification method provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on the electronic device 300, the program code is used to enable the electronic device 300 to execute the multimodal bird's-eye view fusion method based on position coding provided in the embodiment of the present application.

[0129] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0130] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0131] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0132] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A multimodal bird's-eye view fusion method based on position coding, characterized in that: include: Obtain point cloud data and multiple RGB images of the intersection target scene; The point cloud data is processed using a pre-trained three-dimensional backbone network to obtain a first feature map, and the first feature map is highly dimensionally compressed to obtain a first BEV feature map; Process each RGB image to obtain a BEV feature map, and superimpose the BEV feature maps of all RGB images to obtain a second BEV feature map; Processing the cascaded first BEV feature map and the second BEV feature map to obtain a first feature query result, wherein the first feature query result includes first feature vectors of multiple target center points; The first feature query result, the first BEV feature map and the second BEV feature map are processed using a pre-trained modulation position coding attention network to obtain a second feature query result, wherein the second feature query result includes a second feature vector of multiple target center points, and the dimension of the second feature vector is greater than that of the first feature vector.

2. The multimodal bird's-eye view fusion method based on position coding according to claim 1 is characterized in that: Each RGB image is processed to obtain a BEV feature map; including: Each RGB image is processed using a pre-trained two-dimensional backbone network to obtain a two-dimensional feature vector for each pixel of the RGB image; Add the predicted depth value to the two-dimensional feature vector of each pixel to obtain the two-dimensional feature vector of each pixel; Project the three-dimensional feature vectors of all pixels to the coordinate system where the point cloud data is located to obtain a second feature map; The second feature map is subjected to high-dimensional feature compression to obtain a BEV feature map.

3. The multimodal bird's-eye view fusion method based on position coding according to claim 2 is characterized in that: Processing the cascaded first BEV feature graph and the second BEV feature graph to obtain a first feature query result; including: The concatenated first BEV feature map and the second BEV feature map are processed using the pre-trained convolutional layer to obtain a BEV feature map F1; The BEV feature map F1 is processed using the pre-trained SE channel attention mechanism network to obtain the fused BEV feature map F fused ; The first feedforward neural network trained in advance is used to fusion BEV feature map F fused Processing is performed to obtain multiple target center points and their classification results; Obtain the probability values ​​of generating prediction boxes for multiple target center points, sort the probability values ​​in descending order, and obtain the target center points corresponding to the first N probability values; Extract the feature vector of the target center point in the first BEV feature map as the first feature vector of the target center point, and combine the first feature vectors of the N target center points into a first feature query result Q LiDAR .

4. The multimodal bird's-eye view fusion method based on position coding according to claim 3 is characterized in that: The modulation position coding attention network includes: a first attention mechanism network, a modulation position coding unit, a cascade unit, and a second attention mechanism network; The first feature query result, the first BEV feature map and the second BEV feature map are processed by using the pre-trained modulation position coding attention network to obtain a fusion feature map based on the feature query; including: Use the first attention mechanism network to query the first feature result Q LiDAR and the first BEV characteristic diagram F LiDAR Perform fusion processing to obtain the third feature query result Q L , the third feature query result Q L A third eigenvector including N target center points; The second feedforward neural network is used to query the third feature result Q L Processing is performed to obtain N first target detection frames; Processing the N first target detection frames using a modulation position coding unit to obtain a position coding vector of a center point of the first target detection frame, wherein the position coding vector includes height information and scale information; Using a cascade unit, cascade the position encoding vector of the center point of each first target detection frame and the corresponding third feature vector to obtain a fourth feature vector of the center point of each target; The fourth feature vector and the second BEV feature map F of the N target center points are analyzed using the second attention mechanism network C[mer[ Perform fusion processing to obtain the second feature query result Q LC .

5. The multimodal bird's-eye view fusion method based on position coding according to claim 4 is characterized in that: Using a modulation position coding unit to process N first target detection frames to obtain a position coding vector of a center point of the first target detection frame; including: The first target detection box A q Convert to BEV space and get the target representation PE(A) in BEV space q );A q =(x q ,y q ,z q ,l q ,w q ,h q ); where (x q ,y q ,z q ) is the first target detection box A q The three-dimensional coordinates of the center point, l q ,w q and h q is the first target detection box A q length, width and height; PE(A q ) to modulate the length, width and height, and obtain the target modulation representation in the BEV space Modulate(PE(A q )): Among them, l q,ref ,w q,ref and h q,ref is the reference length, reference width and reference height; PE(x q ) represents x q Converted to the value in BEV space; PE(y q ) represents y q Converted to the value in BEV space; PE(z q ) represents z q Convert to the value in BEV space; The fully connected network is used to modulate the target in the BEV space. q )) Dimension expansion is performed to obtain the first target detection frame A q The position encoding vector P of the center point q .

6. The multimodal bird's-eye view fusion method based on position coding according to claim 5 is characterized in that: The method further comprises: The third feedforward neural network trained in advance is used to query the second feature result Q LC Processing is performed to obtain N second target detection frames.

7. The method according to claim 6, characterized in that The method also includes: a step of jointly training a three-dimensional backbone network, a two-dimensional backbone network, a convolutional layer, an SE channel attention mechanism network, a first feedforward neural network, a modulation position coding attention network and a third feedforward neural network.

8. A multi-modal bird's-eye view fusion device based on position coding, characterized in that: include: An acquisition unit, used to acquire point cloud data collected by a laser radar of a target scene at an intersection and multiple RGB images collected by multiple surround view cameras; A first feature extraction unit is used to process the point cloud data using a pre-trained three-dimensional backbone network to obtain a first feature map, and perform high-dimensional feature compression on the first feature map to obtain a first BEV feature map; A second feature extraction unit is used to process each RGB image to obtain a BEV feature map, and superimpose the BEV feature maps of all RGB images to obtain a second BEV feature map; A first processing unit is used to process the cascaded first BEV feature map and the second BEV feature map to obtain a first feature query result, wherein the first feature query result includes first feature vectors of multiple target center points; The second processing unit is used to use the pre-trained modulation position coding attention network to process the first feature query result, the first BEV feature map and the second BEV feature map to obtain a second feature query result, wherein the second feature query result includes a second feature vector of multiple target center points, and the dimension of the second feature vector is greater than the dimension of the first feature vector.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images