BEV perception fusion method, target detection method, equipment, medium and vehicle

By adopting BEV perception fusion method based on visible light and infrared images in autonomous driving technology, the dual-light BEV feature space is generated using Transformer and VGG models, which solves the problem of viewing angle difference and robustness during multi-sensor fusion, and improves the detection Strong and adaptable, achieving efficient target recognition.

CN118999561APending Publication Date: 2024-11-22WU HAN XUAN YUAN ZHI JIA KE JI YOU XIAN GONG SI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411061754.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In autonomous driving technology, there are poor coordination of viewing angles, low camera-to-BEV conversion efficiency, poor detection robustness when fusion of multiple sensors, especially in harsh environments, low adaptability, and traditional image detection feature conversion takes a long time and unreasonable depth distribution estimates. , resulting in low recognition accuracy.

Method used

Using BEV-perceptual fusion method based on visible light and infrared images, feature extraction and processing are performed through Transformer and VGG models to generate dual-light BEV feature space, and robustness and adaptability are enhanced through weighted fusion and self-attention mechanisms.

Benefits of technology

It realizes efficient fusion across cameras and multimodals, overcomes the defects that radar sensors cannot detect in real time, improves the robustness and adaptability of target detection, isolates errors and noise, and strengthens key information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118999561A_ABST
    Figure CN118999561A_ABST
Patent Text Reader

Abstract

The invention discloses a BEV perception fusion method, and the method comprises the steps: constructing a visible light BEV space based on obtained visible light image mapping, and constructing an infrared BEV space based on obtained infrared image mapping; and carrying out splicing fusion processing on the visible light BEV space and the infrared BEV space to obtain a fused dual-light BEV feature space. Compared with an existing laser-radar sensor recognition scheme, the unified fused dual-light BEV feature space is easier to carry out cross-camera and multi-mode fusion, and the technical defect that a radar sensor cannot carry out real-time detection can be overcome.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to a BEV perception fusion method, target detection method, equipment, medium and vehicle. Background Art

[0002] At present, in the field of autonomous driving technology, multi-sensor perspective fusion and other methods are usually used for target recognition. However, when multiple sensors (such as lidar sensors) are fused, there are problems such as poor coordination of perspective differences, low conversion efficiency from camera to BEV, poor detection robustness, and low adaptability in harsh environments and weather. In addition, in traditional image detection schemes, the conversion of extracted features to BEV (bird's eye view) features takes too much time, and the depth distribution estimation is unreasonable, resulting in the inability to provide high-accuracy recognition. For example: the existing lidar fusion sensor has a limited perception range, and the information available in autonomous driving is limited, resulting in poor perception effects. Summary of the invention

[0003] In response to at least one defect or improvement need in the prior art, the present application provides a BEV perception fusion method, target detection method, equipment, medium and vehicle, which are used to at least solve the technical problems of poor coordination of perspective differences and inability of radar sensors to detect in real time during existing multi-sensor fusion.

[0004] To achieve the above objectives, in a first aspect, the present application provides a BEV perception fusion method, comprising:

[0005] Based on the acquired visible light image mapping, a visible light BEV space is constructed; based on the acquired infrared image mapping, an infrared BEV space is constructed;

[0006] The visible light BEV space and the infrared BEV space are spliced ​​and fused to obtain a fused dual-light BEV feature space.

[0007] Further, the visible light BEV space is constructed based on the acquired visible light image mapping, and the infrared BEV space is constructed based on the acquired infrared image mapping; the visible light BEV space and the infrared BEV space are spliced ​​and fused to obtain a fused dual-light BEV feature space, including:

[0008] Extracting features from the acquired visible light image and infrared image, and obtaining visible light feature data and infrared feature data respectively;

[0009] The visible light feature data and the infrared feature data are processed by a Transformer method respectively, so as to obtain the visible light BEV space and the infrared BEV space respectively;

[0010] The visible light BEV space and the infrared BEV space are spliced ​​according to the channel dimension, and then the fused dual-light BEV feature space is obtained through a convolution layer.

[0011] Furthermore, the extracting features of the acquired visible light image and infrared image and obtaining visible light feature data and infrared feature data respectively includes:

[0012] Performing feature extraction on the acquired visible light image and infrared image respectively through the VGG model to obtain a visible light feature map and an infrared feature map respectively;

[0013] The visible light feature map and the infrared feature map are respectively input into the FPN to obtain a visible light fine feature map and an infrared fine feature map that retain more detail information.

[0014] Further, the processing of the visible light feature data and the infrared feature data by a Transformer method to obtain the visible light BEV space and the infrared BEV space respectively includes:

[0015] Calculate Q, K and V for each first feature point of various light fine feature maps respectively, and obtain matrices related to Q, K and V respectively;

[0016] The matrices are respectively input into the multi-head self-attention mechanism modules, so that each first feature point is updated to a second feature point containing all feature point information, thereby obtaining corresponding image global information respectively;

[0017] The second feature points are respectively input into the Add&Norm layer for residual connection and normalization to obtain the third feature points;

[0018] Input the third feature points processed by the Add&Norm layer into the FFN respectively to obtain the encoded first BEV feature and the second BEV feature;

[0019] Based on the first BEV feature and the second BEV feature, respectively acquiring the visible light BEV space and the infrared BEV space by weighted fusion;

[0020] Among them, the light fine feature map includes the visible light fine feature map and the infrared fine feature map, and the Transformer method processing logic of the two is consistent, and the first BEV feature and the second BEV feature are obtained respectively.

[0021] Furthermore, the obtaining of the visible light BEV space and the infrared BEV space respectively by weighted fusion based on the first BEV feature and the second BEV feature includes:

[0022] Assigning a fusion weight to each of the first BEV feature and the second BEV feature respectively, and adding the two together according to the fusion weight to obtain a weighted fusion result;

[0023] Generate a blank BEV space grid, perform a deformable attention mechanism calculation with the BEV space of the previous frame, query the BEV space of the previous frame based on each point in the blank BEV space grid, and obtain an initial BEV space;

[0024] The initial BEV space is respectively subjected to spatial cross self-attention calculation with the corresponding BEV features output by the Transformer encoder, and each point in the initial BEV space is respectively queried for the corresponding light fine feature map, so as to obtain the corresponding final light BEV space;

[0025] The corresponding BEV features include the first BEV features and the second BEV features; and the corresponding final light BEV space includes the visible light BEV space and the infrared BEV space.

[0026] Furthermore, the VGG model is a VGG model that supports 16 layers.

[0027] In a second aspect, the present application provides a target detection method, which performs target recognition on the fused dual-light BEV feature space obtained by any of the above-mentioned BEV perception fusion methods based on a trained target detection and recognition model to obtain a target recognition result.

[0028] In a third aspect, the present application provides an electronic device comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit is enabled to execute the steps of the BEV perception fusion method described in any one of the above items, or to execute the steps of the above-mentioned target detection method.

[0029] In a fourth aspect, the present application provides a storage medium storing a computer program executable by an access authentication device. When the computer program runs on the access authentication device, the access authentication device can execute the steps of the BEV perception fusion method described in any one of the above items, or can execute the steps of the above-mentioned target detection method.

[0030] In a fifth aspect, the present application provides a vehicle storing a set of instruction sets, which are executed by the vehicle to implement the steps of the BEV perception fusion method described in any one of the above items, or to implement the steps of the above target detection method.

[0031] In general, the above technical solutions conceived by the present application can achieve the following beneficial effects compared with the prior art:

[0032] (1) Compared with the existing laser-radar sensor recognition scheme, the unified fused dual-light BEV feature space of the present application is easier to integrate across cameras and multiple modalities, and can overcome the technical defect that radar sensors cannot detect in real time.

[0033] (2) Compared with the existing feature-level fusion solutions, the post-fusion solution of the present application has higher efficiency and robustness due to its parallel computing. The failure of a sensor will not cause the entire system to fail. It can also isolate errors and noise and enhance key information. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A flowchart of a BEV perception fusion method provided in an embodiment of the present application;

[0036] Figure 2 A flowchart of processing a visible light fine feature map provided in an embodiment of the present application through a Transformer encoder;

[0037] Figure 3 A block diagram of an electronic device suitable for implementing the method described above provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0039] The terms "first", "second" or "third" etc. in the specification, claims or the above-mentioned drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0040] At present, in the field of autonomous driving technology, multi-sensor perspective fusion and other methods are usually used for target recognition. However, when multiple sensors (such as laser-radar sensors) are fused, there are problems such as poor coordination of perspective differences, low conversion efficiency from camera to BEV, poor detection robustness, and low adaptability in harsh environments and weather; and in traditional image detection schemes, the conversion of extracted features to BEV (bird's eye view) features takes too much time, and the depth distribution estimation is unreasonable, resulting in the inability to provide high-accuracy recognition. In view of this technical status, the present application provides a BEV perception fusion method, target detection method, device, medium and vehicle, which are used to at least solve the existing technical problems such as poor coordination of perspective differences when laser-radar multi-sensor fusion occurs and radar sensors cannot detect in real time.

[0041] refer to Figure 1 An embodiment of the present application provides a BEV perception fusion method, including:

[0042] Visible light images are acquired through a camera, and infrared images are acquired through an infrared camera.

[0043] The visible light BEV space is constructed based on the acquired visible light image mapping, and the infrared BEV space is constructed based on the acquired infrared image mapping.

[0044] The visible light BEV space and the infrared BEV space are spliced ​​and fused to obtain a fused dual-light BEV feature space.

[0045] The implementation method of BEV (Birds-Eyes-View) is to correct and change the original 2D camera perspective through an algorithm to form a bird's-eye view. It converts the perception method of the traditional autonomous driving 2D image perspective (Image View) plus ranging into 3D perception under the bird's-eye view. BEV is actually a technology that can convert multiple information inputs into the same three-dimensional space of the same reference system, so all subsequent perception, prediction and planning can be carried out in the same space, thereby reducing the trouble of the upper layer to make judgments and decisions on different spatial results. Even end-to-end intelligent driving has made a leap forward. Going further, BEV unifies different inputs in the same space, making it easier to do some things. In the past, visual perception through image frames, it was still difficult to process the depth, volume and posture of objects, and in the recognition of 2D images, for distant objects, an error of a few pixels may lead to a real error of tens of meters. However, 3D information in the BEV space can be more easily obtained, is more sensitive to errors, and the training results will be more accurate.

[0046] The existing technology mostly uses a target detection method that combines laser and radar sensors, which has the problem of poor coordination of perspective differences during fusion, low camera-to-BEV conversion efficiency, poor detection robustness, and low adaptability in harsh environments and weather. Compared with the existing laser-radar sensor recognition solution, this unified dual-light (visible light and infrared light) BEV space is easier to fuse across cameras and multiple modalities, which can overcome the technical defect that radar sensors cannot detect in real time, thereby improving the robustness and adaptability of target detection.

[0047] refer to Figure 1 In some embodiments, a BEV perception fusion method may further include the following steps.

[0048] Step 1: Acquire a visible light image, perform feature extraction on the visible light image, and obtain visible light feature data; acquire an infrared image, perform feature extraction on the infrared image, and obtain infrared feature data.

[0049] In some embodiments, the visible light image is acquired by a camera, and the infrared image is acquired by an infrared camera; the visible light image and the infrared image are respectively subjected to feature extraction through the same network. More specifically, preferably, the visible light image feature map and the infrared image feature map are extracted through the VGG-16 network of the same class in the VGG model, and the extracted features are input into the FPN for multi-scale feature fusion. The first few convolutional layers of VGG-16 are used as feature extractors, that is, the input image is passed through the convolution and pooling layers of VGG-16 to obtain a series of feature maps. Then, the feature maps extracted by VGG-16 are used as input and input into the FPN respectively. The top-down path of FPN gradually upsamples the high-resolution feature map extracted from VGG-16 to obtain a coarser but lower resolution feature map; the lateral connection of FPN fuses the low-level features with the high-level features to obtain a visible light feature map (visible light feature data) and an infrared feature map (infrared feature data) that retain more detail information.

[0050] The 16-layer VGG model, referred to as VGG-16, is a convolutional neural network model that uses filters with a 3×3 kernel size. VGG-16 can classify images into 1,000 object categories, including keyboards, animals, pencils, and mice. In addition, the image input size of this model is 224×224.

[0051] FPN (Feature Pyramid Network) is a neural network structure commonly used in target detection and semantic segmentation tasks, which aims to solve the detection and segmentation problems of targets of different scales. The core idea of ​​FPN is to obtain multi-scale feature representation by establishing a feature pyramid, and to achieve step-by-step upsampling and fusion from low-resolution, high-semantic information features to high-resolution, low-semantic information features by establishing a direct connection between the bottom feature map and the top feature map.

[0052] Step 2: The visible light feature data is subjected to the Transformer method to obtain the BEV feature of visible light (visible light BEV space); the infrared feature data is subjected to the Transformer method to obtain the BEV feature of infrared (infrared BEV space).

[0053] The Transformer method is a sequence-to-sequence model based on the self-attention mechanism, which is mainly used for tasks such as natural language processing. Compared with traditional models such as RNN and CNN, the Transformer model abandons the traditional RNN and CNN structures. First, it uses the Attention mechanism to reduce the distance between any two positions in the sequence to a constant, thereby solving the thorny long-term dependency problem in NLP. It is not a sequential structure similar to RNN, so it has better parallelism and conforms to the existing GPU framework. The Transformer model adopts an encoder-decoder structure, which is formed by stacking several encoders and decoders, in which the encoder is responsible for converting the input corpus into a feature vector, and the decoder is responsible for outputting the conditional probability of the final result.

[0054] refer to Figure 2 In some embodiments, the visible light feature map obtained in step 1 is put into the Transformer network to obtain the visible light BEV space. Specifically, the visible light feature map obtained in step 1 is put into the Transformer encoder module, and the visible light features containing global features are calculated through the self-attention mechanism. Step 2 can also include the following steps.

[0055] Step 21: Calculate Q, K and V for each first feature point (vector) of the feature graph to obtain the matrices of the three. Q, K and V in the attention mechanism are to calculate the similarity between the current query and all the keys. This similarity value is passed through the Softmax layer to obtain a set of weights. The value under attention is obtained by summing the product of this set of weights and the corresponding value.

[0056] Step 22: Input the matrix into the multi-head self-attention mechanism module, and update each first feature point to a second feature point containing all feature point information, so as to obtain the global information of the image.

[0057] The basic formula of the self-attention mechanism is:

[0058]

[0059] Step 23, input the updated second feature point vectors into the Add&Norm layer, perform residual connection and normalization, and obtain the third feature point. While solving the difficulty of multi-layer neural network training, normalizing the activation values ​​of each layer can accelerate the training process of the model and make it converge faster. Add: Residual connection can increase the depth without losing the initial features. Norm: Normalization, because Add adds the original features back, in order to avoid gradient disappearance and reduce operations, normalization is performed.

[0060] Step 24: Input each third feature point processed by the Add&Norm layer into the FFN network to obtain the final encoded first BEV feature and second BEV feature.

[0061] In the Transformer model, FFN (Feed-Forward Network) refers to a feed-forward neural network, which is an important component of the Transformer. FFN is mainly used to perform nonlinear transformation and enhancement on the input features. The core of the Transformer model is the self-attention mechanism, which allows the model to capture the dependencies between different positions when processing sequence data. However, the self-attention mechanism itself is position-independent, that is, it only considers the mutual relationship between the positions in the input sequence without explicit order information. In order to introduce position awareness and nonlinear transformation, FFN was introduced into the Transformer model. FFN consists of two linear transformation layers with a nonlinear activation function (usually ReLU) in the middle. Specifically, FFN takes the feature vector of each position of the input sequence as input and maps it to a higher-dimensional space through two linear transformations, and then performs a nonlinear transformation through the activation function. Finally, it is mapped back to the original feature dimension through another linear transformation.

[0062] Step 25, perform fusion by weighted fusion, assign a fusion weight to each feature of the first BEV feature obtained by visible light and the second BEV feature obtained by infrared, and add the two together according to the fusion weight to obtain the weighted fusion result; then, first generate a blank BEV space grid, perform deformable attention mechanism calculation with the BEV space of the previous frame, query the BEV space of the previous frame with each point in the blank BEV grid, and obtain the initial BEV space; then perform spatial cross self-attention calculation on the initial BEV space and the visible light feature output by the encoder, and query the visible light feature map with each point in the initial BEV space, so as to obtain the final visible light BEV space.

[0063] The infrared feature image processing method is the same as step 21 to step 25, and will not be repeated here.

[0064] Step 3: Concatenate the visible light BEV space and the infrared BEV space according to the channel dimension, and then obtain the fused dual-light BEV feature space through the convolution layer.

[0065] Compared with the existing feature-level fusion solutions, the post-fusion solution of the present application has higher efficiency and robustness due to its parallel computing. The failure of one sensor will not cause the entire system to fail. It can also isolate errors and noise and enhance key information.

[0066] An embodiment of the present application also provides a target detection method, which performs target recognition on the fused dual-light BEV feature space obtained by any of the above-mentioned BEV perception fusion methods based on a trained target detection and recognition model to obtain a target recognition result. The method for obtaining the fused dual-light BEV feature space is described in the above-mentioned BEV perception fusion method section and will not be repeated here.

[0067] The input of the detection and recognition model is an image from a bird's-eye view, and the output is the target recognition result corresponding to the input image. In some embodiments, the detection and recognition model can realize multi-target recognition at the same time, such as recognizing lane lines and obstacles at the same time, or recognizing lane lines, obstacles, pedestrians and vehicles at the same time; or recognizing lane lines, obstacles, pedestrians, the number of pedestrians, vehicle types (such as electric vehicles, cars, trucks, etc.) and the number of vehicles at the same time. With the continuous recognition of the detection and recognition model in time sequence, single target tracking or multi-target tracking can be performed.

[0068] Figure 3 A block diagram of an electronic device suitable for implementing the above-described method (including the BEV perception fusion method and the target detection method) according to an embodiment of the present application is schematically shown. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0069] like Figure 3 As shown, the electronic device 1000 described in this embodiment includes: a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), and the like. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0070] In RAM 1003, various programs and data required for the operation of system 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present application by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present application by executing the program stored in the one or more memories.

[0071] According to an embodiment of the present application, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The system 1000 may further include one or more of the following components connected to the I / O interface 1005: an input part 1006 including a keyboard, a mouse, etc.; an output part 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 1008 including a hard disk, etc.; and a communication part 1009 including a network interface card such as a LAN card, a modem, etc. The communication part 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that the computer program read therefrom is installed into the storage part 1008 as needed.

[0072] According to the method flow of the embodiment of the present application, it can be implemented as a computer software program. For example, the embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present application are executed. According to the embodiment of the present application, the system, equipment, device, module and / or unit described above, etc. can be implemented by a computer program module.

[0073] The embodiments of the present application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the steps of the method according to the embodiment of the present application (including the BEV perception fusion method and the target detection method) can be implemented.

[0074] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present application, the computer-readable storage medium may include one or more memories other than the ROM 1002 and / or RAM 1003 described above.

[0075] An embodiment of the present application also provides a vehicle storing a set of instruction sets, which are executed by the vehicle to implement the steps of any of the above-mentioned BEV perception fusion methods, or to implement the steps of the target detection method.

[0076] The vehicle provided in this embodiment may be a fuel vehicle, a hybrid vehicle, an electric vehicle, a fuel cell vehicle or other types of vehicles.

[0077] The vehicle may include multiple subsystems, such as a drive system, a control system, a perception system, a communication system, an information display system, and a computing system. The vehicle may also include more or fewer subsystems, and each subsystem may also include multiple components, which are not described in detail here.

[0078] The drive system includes components that provide power movement for the vehicle, such as the engine, energy source, transmission, etc.

[0079] The control system includes components that provide control for the vehicle, such as vehicle control, cockpit equipment control, driver assistance control, etc.

[0080] The perception system includes components that provide the vehicle with surrounding environment perception, such as vehicle positioning system, laser sensor, voice sensor, ultrasonic sensor, camera equipment, etc.

[0081] The communication system includes components that provide communication connections for the vehicle, such as mobile communication networks (such as 3G, 4G, 5G networks, etc.), WiFi, Bluetooth, and Internet of Vehicles.

[0082] The information display system includes components that provide various information displays for the vehicle, such as vehicle information display, navigation information display, entertainment information display, etc.

[0083] The computing and processing system includes components that provide data computing and processing capabilities for the vehicle. The computing and processing system may include at least one processor and a memory. The processor may execute instructions stored in the memory. The processor may be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a disk or an optical disk.

[0084] In an embodiment of the present application, a set of instruction sets is stored in the memory, and the processor can execute the instruction set to implement all or part of the steps of the BEV perception fusion method described in any of the above exemplary embodiments, or to implement all or part of the steps of the target detection method in the above exemplary embodiments.

[0085] It should be noted that the functional modules in the various embodiments of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product.

[0086] The flowchart and / or block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart and / or block diagram can represent a part of a module, program segment or code, and a part of the above-mentioned module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the technical features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope of the present application.

[0088] Although the present application has been shown and described with reference to specific exemplary embodiments of the present application, it should be understood by those skilled in the art that various changes in form and details may be made to the present application without departing from the spirit and scope of the present application as defined by the appended claims and their equivalents. Therefore, the scope of the present application should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A BEV perception fusion method, characterized in that: include: Based on the acquired visible light image mapping, a visible light BEV space is constructed; based on the acquired infrared image mapping, an infrared BEV space is constructed; The visible light BEV space and the infrared BEV space are spliced ​​and fused to obtain a fused dual-light BEV feature space.

2. The BEV perception fusion method according to claim 1, characterized in that: The visible light BEV space is constructed based on the acquired visible light image mapping, and the infrared BEV space is constructed based on the acquired infrared image mapping; The visible light BEV space and the infrared BEV space are spliced ​​and fused to obtain a fused dual-light BEV feature space, including: Extracting features from the acquired visible light image and infrared image, and obtaining visible light feature data and infrared feature data respectively; The visible light feature data and the infrared feature data are processed by a Transformer method respectively, so as to obtain the visible light BEV space and the infrared BEV space respectively; The visible light BEV space and the infrared BEV space are spliced ​​according to the channel dimension, and then the fused dual-light BEV feature space is obtained through a convolution layer.

3. The BEV perception fusion method according to claim 2, characterized in that: The step of extracting features from the acquired visible light image and infrared image and obtaining visible light feature data and infrared feature data respectively comprises: Performing feature extraction on the acquired visible light image and infrared image respectively through the VGG model to obtain a visible light feature map and an infrared feature map respectively; The visible light feature map and the infrared feature map are respectively input into the FPN to obtain a visible light fine feature map and an infrared fine feature map that retain more detail information.

4. The BEV perception fusion method according to claim 3, characterized in that: The processing of the visible light feature data and the infrared feature data by a Transformer method to obtain the visible light BEV space and the infrared BEV space respectively includes: Calculate Q, K and V for each first feature point of various light fine feature maps respectively, and obtain matrices related to Q, K and V respectively; The matrices are respectively input into the multi-head self-attention mechanism modules, so that each first feature point is updated to a second feature point containing all feature point information, thereby obtaining corresponding image global information respectively; The second feature points are respectively input into the Add&Norm layer for residual connection and normalization to obtain the third feature points; Input the third feature points processed by the Add&Norm layer into the FFN respectively to obtain the encoded first BEV feature and the second BEV feature; Based on the first BEV feature and the second BEV feature, respectively acquiring the visible light BEV space and the infrared BEV space by weighted fusion; Among them, the light fine feature map includes the visible light fine feature map and the infrared fine feature map, and the Transformer method processing logic of the two is consistent, and the first BEV feature and the second BEV feature are obtained respectively.

5. The BEV perception fusion method according to claim 4, characterized in that: The obtaining of the visible light BEV space and the infrared BEV space respectively by weighted fusion based on the first BEV feature and the second BEV feature comprises: Assigning a fusion weight to each of the first BEV feature and the second BEV feature respectively, and adding the two together according to the fusion weight to obtain a weighted fusion result; Generate a blank BEV space grid, perform a deformable attention mechanism calculation with the BEV space of the previous frame, query the BEV space of the previous frame based on each point in the blank BEV space grid, and obtain an initial BEV space; The initial BEV space is respectively subjected to spatial cross self-attention calculation with the corresponding BEV features output by the Transformer encoder, and each point in the initial BEV space is respectively queried for the corresponding light fine feature map, so as to obtain the corresponding final light BEV space; The corresponding BEV features include the first BEV features and the second BEV features; and the corresponding final light BEV space includes the visible light BEV space and the infrared BEV space.

6. The BEV perception fusion method according to any one of claims 3 to 5, characterized in that: The VGG model supports 16 layers.

7. A target detection method, characterized in that: Based on the trained target detection and recognition model, target recognition is performed on the fused dual-light BEV feature space obtained by the BEV perception fusion method described in any one of claims 1 to 6 to obtain a target recognition result.

8. An electronic device, characterized in that: It includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit can execute the steps of the BEV perception fusion method described in any one of claims 1 to 6, or can execute the steps of the target detection method described in claim 7.

9. A storage medium, characterized in that: It stores a computer program that can be executed by an access authentication device. When the computer program runs on the access authentication device, the access authentication device can execute the steps of the BEV perception fusion method described in any one of claims 1 to 6, or can execute the steps of the target detection method described in claim 7.

10. A vehicle, characterized in that: A set of instruction sets is stored, and the instruction sets are executed by the vehicle to implement the steps of the BEV perception fusion method described in any one of claims 1 to 6, or to implement the steps of the target detection method described in claim 7.