Multimodal dynamic fusion method and device for autonomous driving driven by environmental factor information

Abstract

The present application provides a multimodal dynamic fusion method and device for autonomous driving driven by environmental factor information, wherein the method includes: obtaining multimodal data collected by multiple sensors on the vehicle at the current moment; extracting vehicle features, environmental features, image features and radar features from the approved multimodal data, processing the vehicle features, environmental features, image features and radar features to obtain a scene state parameter vector; using the scene state parameter vector to fuse the vehicle features, environmental features, image features and radar features to obtain a first fused feature; using a decoder to decode the first fused feature to obtain a second fused feature; using an occupancy grid network head to process the second fused feature to obtain a first perception result; using a cross-attention network to process the first perception result to obtain a second perception result. The present application can provide higher perception accuracy and reliability in complex and changeable actual driving environments.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a multi-modal dynamic fusion method and device for autonomous driving driven by environmental factor information. Background Art

[0002] Current autonomous driving perception systems mainly rely on sensors such as cameras, lidar, and millimeter-wave radar to perceive the surrounding environment. These systems use a model paradigm of dynamic training and static reasoning. During the reasoning process, the architecture and parameters of the network model are fixed. This approach performs well in specific, single scene perception tasks, but does not work well when faced with complex and changing scenes in real driving environments.

[0003] The actual road environment is a diverse and complex scene, which not only contains dynamic target information such as vehicles and pedestrians, but also includes multiple scene state elements such as road conditions, weather changes, and light intensity. Under different scene conditions, the advantages and effectiveness of various sensors also vary significantly. For example, in good weather conditions, LiDAR can accurately measure distance and shape information, but in rainy and snowy weather, the effectiveness of LiDAR will be greatly reduced.

[0004] Adding environmental factors to multimodal fusion will improve the accuracy of scene perception, but there is currently no solution. Summary of the invention

[0005] In view of this, the present application provides an environmental factor information-driven multi-modal dynamic fusion method and device for autonomous driving to solve the above-mentioned technical problems.

[0006] In a first aspect, an embodiment of the present application provides an environmental factor information-driven multi-modal dynamic fusion method for autonomous driving, comprising:

[0007] Obtain multimodal data collected by multiple sensors on the vehicle at the current moment;

[0008] Performing data redundancy review on the multimodal data to obtain approved multimodal data;

[0009] Extract vehicle features, environment features, image features, and radar features from the approved multimodal data, process the vehicle features, environment features, image features, and radar features, and obtain a scene state parameter vector;

[0010] The scene state parameter vector is used to fuse the vehicle feature, the environment feature, the image feature and the radar feature to obtain a first fused feature;

[0011] Decoding the first fused feature using a decoder to obtain a second fused feature;

[0012] Processing the second fusion feature by using an occupancy grid network head to obtain a first perception result;

[0013] The first perception result is processed using a cross-attention network to obtain a second perception result.

[0014] Furthermore, the first modal data includes at least: steering angle, speed, acceleration, three-dimensional position, meteorological data, illumination, vibration displacement in the Z direction, audio, RGB image and point cloud data.

[0015] Furthermore, the multimodal data is subjected to a data redundancy review to obtain the approved multimodal data, including:

[0016] Step S1: determining whether each component in the multimodal data has a zero value, if yes, determining that the multimodal data has not been reviewed; otherwise, proceeding to step S2;

[0017] Step S2: dividing the multimodal data into primary modal data and non-primary modal data, wherein the primary modal data includes RGB images and point cloud data; the non-primary modal data includes steering angle, speed, acceleration, three-dimensional position, meteorological data, illumination, vibration displacement in the Z direction, and audio;

[0018] Step S3: determining whether each component in the main modal data has a duplicate value with the corresponding component at the previous moment, if yes, determining that the multimodal data has not been reviewed; otherwise, proceeding to step S4;

[0019] Step S4: determining in turn whether each component of the non-main modal data is associated with the main modal data; if so, determining that the multimodal data has been reviewed and approved; otherwise, determining that the multimodal data has not been reviewed and approved.

[0020] Furthermore, the vehicle features, environment features, image features and radar features are processed to obtain a scene state parameter vector, including:

[0021] The spliced ​​vehicle features and environment features are converted into spectral features using Fourier transform;

[0022] The concatenated image features and radar features are processed using a 3*3 convolutional layer to obtain the first feature;

[0023] The spectrum feature and the first feature are concatenated to obtain the second feature;

[0024] Perform a global pooling operation on the second feature to obtain the third feature;

[0025] The pre-trained weight matrix is ​​used to perform single-point convolution with the third feature to obtain the scene state parameter vector.

[0026] Further, the vehicle feature, the environment feature, the image feature and the radar feature are fused using the scene state parameter vector to obtain a first fused feature; including:

[0027] The vehicle feature, the environment feature, the image feature and the radar feature are combined to obtain a fourth feature;

[0028] The scene state parameter vector is multiplied by the fourth feature to obtain the first fused feature.

[0029] Furthermore, the decoder adopts a dynamic neural network structure, including: N layers of path space.

[0030] Further, the first fusion feature is decoded by a decoder to obtain a second fusion feature, including:

[0031] The first fusion feature is fused using the first layer path space Processing to obtain the first intermediate feature

[0032]

[0033] Among them, DW(·) is the depth-wise separable convolution, R(·) is the Relu activation function, Gate(·) is the gating function, For the first fusion feature The feature map with a resolution of S / 2 obtained by pooling, is the vector product, U(·) is the upsampling;

[0034] The first intermediate feature is transformed into Processing to obtain the second intermediate feature

[0035]

[0036] in, For the first intermediate feature The feature map with a resolution of S / 2 obtained by pooling;

[0037] Use the n-th layer path space to the n-1th intermediate feature Processing to obtain the second intermediate feature

[0038]

[0039] Among them, 3≤n≤N; is the n-1th intermediate feature The feature map with a resolution of S / 2 obtained by pooling;

[0040] Determine the Nth intermediate feature It is the second fusion feature.

[0041] In a second aspect, an embodiment of the present application provides an autonomous driving multi-modal dynamic fusion device driven by environmental factor information, comprising:

[0042] An acquisition unit, used to acquire multimodal data collected by multiple sensors on the vehicle at the current moment;

[0043] An audit unit, used to perform a data redundancy review on the multimodal data to obtain the multimodal data that has passed the review;

[0044] A first processing unit is used to extract vehicle features, environment features, image features and radar features from the approved multimodal data, and process the vehicle features, environment features, image features and radar features to obtain a scene state parameter vector;

[0045] A second processing unit is used to fuse the vehicle feature, the environment feature, the image feature and the radar feature using the scene state parameter vector to obtain a first fused feature;

[0046] A decoding unit, configured to decode the first fused feature using a decoder to obtain a second fused feature;

[0047] A first perception unit, configured to process the second fusion feature by using an occupancy grid network head to obtain a first perception result;

[0048] The second perception unit is used to process the first perception result by using a cross-attention network to obtain a second perception result.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, a method of the embodiment of the present application is implemented.

[0051] This application can provide higher perception accuracy and reliability in complex and changeable actual driving environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A schematic diagram of a multi-modal dynamic fusion perception architecture for autonomous driving driven by environmental factor information provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of scene state parameter extraction provided in an embodiment of the present application;

[0055] Figure 3 A flowchart of a multi-modal dynamic fusion method for autonomous driving driven by environmental factor information provided in an embodiment of the present application;

[0056] Figure 4 A functional structure diagram of an autonomous driving multi-modal dynamic fusion device driven by environmental factor information provided in an embodiment of the present application;

[0057] Figure 5 A functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0059] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0060] First, the design concept of the embodiments of the present application is briefly introduced.

[0061] Adding environmental factors to multimodal fusion will improve the accuracy of scene perception, but there is currently no solution.

[0062] To this end, this application proposes a new autonomous driving fusion perception architecture, which adds environmental element information to the fusion and introduces dynamic fusion perception technology based on the existing perception algorithm. This technology mainly relies on dynamic neural networks, which adaptively adjust their own structure or parameters according to input samples during the reasoning stage to achieve higher reasoning efficiency and stronger expression capabilities. Dynamic neural networks can dynamically adjust their own structure and parameters during the reasoning process to adapt to different environments and needs. This flexibility greatly improves the performance of the perception system in complex scenarios.

[0063] The theory of embodied intelligence and embodied perception provides a new perspective to more deeply interpret and realize the dynamic fusion perception process. Embodied Intelligence is a theory that creates an intelligent agent that combines software and hardware and can autonomously learn and evolve based on the interaction between machines and the physical world. Embodied agents can learn and improve their intelligent behaviors by constantly adapting to the environment, so as to better solve various problems in the real world. The theory of embodied perception focuses on providing embodied agents with the ability to explore the world. Its core idea is that the perception of embodied agents is deeply rooted in the interaction between their bodies and the external environment, and is the result of the dynamic fusion of multimodal information such as multi-sensory information and environmental state information.

[0064] Humans are the most perfect embodied intelligent entities. In the human perception system, the perception mode will be dynamically adjusted as the needs and scenarios change. For example, humans will dynamically adjust the perception direction, perception distance, perception confidence and perception granularity according to the actual situation. This dynamic adjustment ability provides new ideas and references for autonomous driving vehicles to achieve more accurate and flexible perception. At the same time, the comprehensiveness of human perception information also provides a direction for the introduction of our environmental factor information. Therefore, by introducing dynamic neural networks, combining environmental state information such as lighting and vibration with vehicle state information such as vehicle speed and steering angle into a multimodal fusion perception framework, and dynamically adjusting the perception strategy as the surrounding scene changes, a more comprehensive and accurate multimodal fusion perception method for autonomous driving can be achieved.

[0065] This application uses dynamic neural networks to introduce environmental state information such as lighting and vibration and vehicle state information such as vehicle speed and steering angle into a multimodal fusion perception framework, realizing a more comprehensive and accurate multimodal fusion perception method for autonomous driving. The specific solution architecture is as follows: Figure 1 shown.

[0066] First, data from a variety of sensors are collected, including three visual sensors: cameras, lidar, and millimeter-wave radar; three environmental status information sensors: light sensors, vibration sensors, and sound sensors; and three vehicle status information sensors: GPS, IMU, and inertial navigation. These data cover multiple information dimensions of the vehicle's surroundings, providing a rich data foundation for the perception system. However, due to the richness of the data modality, the amount of data is too large, which greatly affects the computational efficiency of the perception algorithm. Therefore, a data redundancy review module is used to screen the data, thereby reducing the algorithm's demand for computation while ensuring data validity.

[0067] Data redundancy review includes intra-modal data validity review and inter-modal data correlation review. Intra-modal review preprocesses by removing invalid values ​​and duplicate values ​​within a single modality to improve the validity of single modality data. The input is image, lidar and millimeter-wave radar data. Different modalities are input in sequence. Internal data review is performed on each modality through zero value review and parallel comparison to reduce computational complexity and improve overall performance. The intra-modal review formula is as follows:

[0068]

[0069] Among them, M i is the input data at the i-th moment under a single mode, is the data after zero value check, is the input data after unique value comparison, sim(x,y) is the similarity function between data segments x and y, and the similarity judgment is performed through the unique value comparator, M′ i It is the valid data after being selected by the MUX module.

[0070] The inter-modal data review unit constructs a hyper-edge graph structure with correlation through a graph convolutional neural network, and automatically learns the correlation between multi-modal scene state data such as illumination, vibration, and vehicle speed and images, lidar, and millimeter-wave radar. The hyper-edge graph structure is grouped through grouped hyper-graph convolution to extract inter-modal correlations to solve the problem of inter-modal correlation differences and eliminate invalid modal scene state data. Finally, through intra-modal and inter-modal data review, redundant multi-modal data and the correlation weights of each modality are obtained. The formula is as follows:

[0071] G'=IMA(G)

[0072] Where G is the hyperedge graph to which data of different modalities are mapped, G' is the graph structure after inter-modal review, IMA(·) is the inter-modal review function, and the correlation between modalities is extracted by performing grouped graph convolution on the nodes. The node update formula is as follows:

[0073] Hl+1 =GCN(H l ,A,W)

[0074] Among them, H l is the node representation of the lth layer, A is the adjacency matrix of the graph, and W is the convolution weight. Finally, the nodes in the graph are mapped to the association weight W rel ∈R 1×m as follows:

[0075] W rel =REF(G')

[0076] where REF(·) represents the hyperedge graph mapping function and m is the number of input modes.

[0077] When humans perceive the surrounding environment, they will perceive the surrounding information in an all-round way and dynamically adjust their perception mode according to this information. Based on this, this application proposes a scene state parameter extraction method, which converts environmental element information into scene state parameters to assist the dynamic perception algorithm in perceiving the current scene state, such as Figure 2 shown.

[0078] By using different feature extraction networks to simultaneously extract environmental state information features F 环境 ∈R 1×1×C环境 , vehicle status information feature F 自车 ∈R 1×1×C自车 , camera image features F 相机 ∈R X×Y×C相机 With the lidar feature F 雷达 ∈R X×Y×C雷达 , thus obtaining the feature vectors of different modes and recombining them into a high-dimensional feature vector, where

[0079]

[0080] The high-dimensional feature vector is mapped to a one-dimensional state space through global pooling and single-point convolution projection, thereby obtaining the environment adaptation parameter W e And pass it to the dynamic convolution module. The specific formula is as follows:

[0081]

[0082] Where f(·) represents feature extraction implemented by a 3*3 convolutional layer. To transform the one-dimensional features into the spectrum features of X×Y×C latitude through Fourier transform, Indicates feature concatenation through concat, Φ(·) is as follows:

[0083] Φ(F)=σ(Wf avg (F)

[0084] where f avg (·) is global pooling, W is the weight matrix of single-point convolution, and σ(·) is the sigmoid function.

[0085] The multimodal dynamic fusion model consists of two units: a dynamic convolution unit and an occupancy grid grid output unit. After obtaining the multimodal data and scene state parameters that have been reviewed and cleaned, the dynamic convolution network needs to adaptively adjust the structure and parameters of the network according to the characteristics of the input data. For example, under good lighting conditions, the system may rely more on camera data; while under poor lighting conditions or bad weather, the system may rely more on lidar or millimeter-wave radar data. At the same time, the system will also dynamically adjust the perception algorithm based on the current state information of the vehicle, such as vehicle speed and steering angle, to better adapt to the current driving needs and environment. In the selection of the output head, since the occupancy grid network can mark objects of unknown categories in the form of occupied space, the present invention chooses to use the occupancy grid network as the final output head.

[0086] Specifically, in the multimodal dynamic fusion module, the data that has been reviewed will be extracted by the feature extractor to obtain the corresponding features and mapped to the same space. The features will be fused by multiplying them with the corresponding scene state parameters and using the concat function. The fusion ratio of each mode will be adaptively determined according to the scene state parameters. The specific formula is as follows:

[0087]

[0088] Then the fusion feature F 融合 In the dynamic decoding path space, there are N layers of path space inside the dynamic decoding module, each layer contains up to 4 units, and each unit has 2 scale conversion paths, namely upsampling and resolution preservation. Each unit calculates the weighted sum of the input from the previous layer and uses the soft conditional gate to generate the activation weight. The larger the corresponding weight of the path, the more important it is.

[0089] The decoded features are passed to the occupancy grid network output head to obtain a rough preliminary perception result, which is then passed to the cross-attention network for optimization, and finally an accurate and comprehensive dynamic fusion perception result for autonomous driving is obtained.

[0090] The advantages of this application are:

[0091] First, data from multiple sensors is collected, including traditional sensors such as cameras, lidar, millimeter-wave radar, and new sensors such as ambient light sensors and body vibration sensors. These data cover multiple information dimensions of the vehicle's surroundings, providing a rich data foundation for the perception system.

[0092] Secondly, after receiving these multimodal data, the dynamic neural network will adaptively adjust the structure and parameters of the network according to the characteristics of the input data. For example, in good lighting conditions, the system may rely more on camera data; in poor lighting conditions or bad weather, the system may rely more on lidar or millimeter-wave radar data. At the same time, the system will also dynamically adjust the perception algorithm based on the vehicle's current status information, such as speed and steering angle, to better adapt to the current driving needs and environment.

[0093] Thirdly, the application of embodied perception theory in the present invention enables the system to better understand and utilize multimodal information from the environment and its own state. In the interaction with the environment, the embodied intelligent agent not only passively receives information, but actively detects and perceives the environment through movement and behavior. For example, when the vehicle is turning, the system can analyze the vibration data of the vehicle body to predict the possible risks in the turning process and adjust the driving strategy.

[0094] Through this dynamic, multi-modal fusion perception method, the present application can provide higher perception accuracy and reliability in complex and changeable actual driving environments. This not only improves the safety and efficiency of the autonomous driving system, but also provides an important theoretical and practical basis for the development of future intelligent driving technology.

[0095] In summary, this application introduces dynamic neural networks and embodied intelligence theory on the basis of traditional perception algorithms, and constructs a more comprehensive and accurate multimodal fusion perception method and architecture for autonomous driving by combining environmental state information such as lighting and vibration with vehicle state information. This method has shown significant advantages in practical applications and is expected to become an important development direction for autonomous driving perception technology in the future.

[0096] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0097] like Figure 3 As shown, the embodiment of the present application provides an environmental factor information-driven multi-modal dynamic fusion method for autonomous driving, including:

[0098] Step 101: Acquire multimodal data collected by multiple sensors on the vehicle at the current moment;

[0099] Step 102: performing data redundancy review on the multimodal data to obtain multimodal data that has passed the review;

[0100] Step 103: extracting vehicle features, environment features, image features, and radar features from the approved multimodal data, and processing the vehicle features, environment features, image features, and radar features to obtain a scene state parameter vector;

[0101] Step 104: using the scene state parameter vector to fuse the vehicle feature, the environment feature, the image feature and the radar feature to obtain a first fused feature;

[0102] Step 105: Decode the first fused feature using a decoder to obtain a second fused feature;

[0103] Step 106: Process the second fusion feature using an occupancy grid network head to obtain a first perception result;

[0104] Step 107: Use the cross-attention network to process the first perception result to obtain a second perception result.

[0105] In some embodiments, the first modal data includes at least: steering angle, speed, acceleration, three-dimensional position, meteorological data, lighting, vibration displacement in the Z direction, audio, RGB image and point cloud data.

[0106] In some embodiments, performing a data redundancy review on the multimodal data to obtain approved multimodal data includes:

[0107] Step S1: determining whether each component in the multimodal data has a zero value, if yes, determining that the multimodal data has not been reviewed; otherwise, proceeding to step S2;

[0108] Step S2: dividing the multimodal data into primary modal data and non-primary modal data, wherein the primary modal data includes RGB images and point cloud data; the non-primary modal data includes steering angle, speed, acceleration, three-dimensional position, meteorological data, illumination, vibration displacement in the Z direction, and audio;

[0109] Step S3: determining whether each component in the main modal data has a duplicate value with the corresponding component at the previous moment, if yes, determining that the multimodal data has not been reviewed; otherwise, proceeding to step S4;

[0110] Step S4: determining in turn whether each component of the non-main modal data is associated with the main modal data; if so, determining that the multimodal data has been reviewed and approved; otherwise, determining that the multimodal data has not been reviewed and approved.

[0111] In some embodiments, the vehicle features, environment features, image features, and radar features are processed to obtain a scene state parameter vector, including:

[0112] The spliced ​​vehicle features and environment features are converted into spectral features using Fourier transform;

[0113] The concatenated image features and radar features are processed using a 3*3 convolutional layer to obtain the first feature;

[0114] The spectrum feature and the first feature are concatenated to obtain the second feature;

[0115] Perform a global pooling operation on the second feature to obtain the third feature;

[0116] The pre-trained weight matrix is ​​used to perform single-point convolution with the third feature to obtain the scene state parameter vector.

[0117] In some embodiments, the vehicle feature, the environment feature, the image feature and the radar feature are fused using the scene state parameter vector to obtain a first fused feature; including:

[0118] The vehicle feature, the environment feature, the image feature and the radar feature are combined to obtain a fourth feature;

[0119] The scene state parameter vector is multiplied by the fourth feature to obtain the first fused feature.

[0120] In some embodiments, the decoder adopts a dynamic neural network structure, including: N layers of path space.

[0121] In some embodiments, decoding the first fused feature using a decoder to obtain a second fused feature includes:

[0122] The first fusion feature is fused using the first layer path space Processing to obtain the first intermediate feature

[0123]

[0124] Among them, DW(·) is the depth-wise separable convolution, R(·) is the Relu activation function, Gate(·) is the gating function, For the first fusion feature The feature map with a resolution of S / 2 obtained by pooling, is the vector product, U(·) is the upsampling;

[0125] The first intermediate feature is transformed into Processing to obtain the second intermediate feature

[0126]

[0127] in, For the first intermediate feature The feature map with a resolution of S / 2 obtained by pooling;

[0128] Use the n-th layer path space to the n-1th intermediate feature Processing to obtain the second intermediate feature

[0129]

[0130] Among them, 3≤n≤N; is the n-1th intermediate feature The feature map with a resolution of S / 2 obtained by pooling;

[0131] Determine the Nth intermediate feature It is the second fusion feature.

[0132] Based on the above embodiments, the present application provides an automatic driving multi-modal dynamic fusion device driven by environmental factor information, see Figure 4 As shown, the environmental factor information-driven autonomous driving multi-modal dynamic fusion device 200 provided in the embodiment of the present application at least includes:

[0133] An acquisition unit 201 is used to acquire multimodal data collected by multiple sensors on the vehicle at the current moment;

[0134] An audit unit 202 is used to perform a data redundancy review on the multimodal data to obtain multimodal data that has passed the review;

[0135] The first processing unit 203 is used to extract vehicle features, environment features, image features and radar features from the approved multimodal data, and process the vehicle features, environment features, image features and radar features to obtain a scene state parameter vector;

[0136] The second processing unit 204 is used to fuse the vehicle feature, the environment feature, the image feature and the radar feature using the scene state parameter vector to obtain a first fused feature;

[0137] A decoding unit 205, configured to decode the first fused feature using a decoder to obtain a second fused feature;

[0138] A first perception unit 206, configured to process the second fusion feature using an occupancy grid network head to obtain a first perception result;

[0139] The second perception unit 207 is used to process the first perception result by using a cross-attention network to obtain a second perception result.

[0140] It should be noted that the principle of solving the technical problem of the multimodal dynamic fusion device 200 for autonomous driving driven by environmental factor information provided in the embodiment of the present application is similar to the basic method provided in the embodiment of the present application. Therefore, the implementation of the multimodal dynamic fusion device 200 for autonomous driving driven by environmental factor information provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0141] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 5 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the multi-modal dynamic fusion method for autonomous driving driven by environmental factor information provided in the embodiment of the present application is implemented.

[0142] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0143] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0144] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0145] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 5 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0146] It should be noted that Figure 5 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0147] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the multi-modal dynamic fusion method of autonomous driving driven by environmental factor information provided in the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the multi-modal dynamic fusion method of autonomous driving driven by environmental factor information provided in the embodiment of the present application by executing the built-in or installed executable program.

[0148] The multi-modal dynamic fusion method for autonomous driving driven by environmental factor information provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on an electronic device 300, the program code is used to enable the electronic device 300 to execute the multi-modal dynamic fusion method for autonomous driving driven by environmental factor information provided in the embodiment of the present application.

[0149] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0150] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0151] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0152] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A multi-modal dynamic fusion method for autonomous driving driven by environmental factor information, characterized in that: include: Obtain multimodal data collected by multiple sensors on the vehicle at the current moment; Performing data redundancy review on the multimodal data to obtain approved multimodal data; Extracting vehicle features, environmental features, image features and radar features from the approved multimodal data, and processing the vehicle features, environmental features, image features and radar features to obtain a scene state parameter vector; the vehicle features include steering angle, speed, acceleration and three-dimensional position; the environmental features include meteorological data, illumination, vibration displacement in the Z direction and audio; The scene state parameter vector is used to fuse the vehicle feature, the environment feature, the image feature and the radar feature to obtain a first fused feature; Decoding the first fused feature using a decoder to obtain a second fused feature; Processing the second fusion feature by using an occupancy grid network head to obtain a first perception result; Processing the first perception result using a cross attention network to obtain a second perception result; The vehicle features, environment features, image features and radar features are processed to obtain a scene state parameter vector, including: The spliced ​​vehicle features and environment features are converted into spectral features using Fourier transform; The concatenated image features and radar features are processed using a 3*3 convolutional layer to obtain the first feature; The spectrum feature and the first feature are concatenated to obtain the second feature; Perform a global pooling operation on the second feature to obtain the third feature; Use the pre-trained weight matrix and the third feature to perform single-point convolution to obtain the scene state parameter vector; The vehicle feature, the environment feature, the image feature and the radar feature are fused using the scene state parameter vector to obtain a first fused feature; including: The vehicle feature, the environment feature, the image feature and the radar feature are combined to obtain a fourth feature; The scene state parameter vector is multiplied by the fourth feature to obtain the first fused feature.

2. The method according to claim 1, characterized in that The multimodal data includes at least: steering angle, speed, acceleration, three-dimensional position, meteorological data, illumination, vibration displacement in the Z direction, audio, RGB image and point cloud data.

3. The method according to claim 2, characterized in that The multimodal data is subjected to a data redundancy review to obtain the approved multimodal data, including: Step S1: determining whether each component in the multimodal data has a zero value, if yes, determining that the multimodal data has not been reviewed; otherwise, proceeding to step S2; Step S2: dividing the multimodal data into primary modal data and non-primary modal data, wherein the primary modal data includes RGB images and point cloud data; the non-primary modal data includes steering angle, speed, acceleration, three-dimensional position, meteorological data, illumination, vibration displacement in the Z direction, and audio; Step S3: determining whether each component in the main modal data has a duplicate value with the corresponding component at the previous moment, if yes, determining that the multimodal data has not been reviewed; otherwise, proceeding to step S4; Step S4: determining in turn whether each component of the non-main modal data is associated with the main modal data; if so, determining that the multimodal data has been reviewed and approved; otherwise, determining that the multimodal data has not been reviewed and approved.

4. The method according to claim 1, characterized in that: The decoder adopts a dynamic neural network structure, including: N layers of path space.

5. The method according to claim 4, characterized in that Decoding the first fused feature by using a decoder to obtain a second fused feature includes: The first fusion feature is fused using the first layer path space Processing to obtain the first intermediate feature Among them, DW(·) is the depth-wise separable convolution, R(·) is the Relu activation function, Gate(·) is the gating function, For the first fusion feature The feature map with a resolution of S / 2 obtained by pooling, is the vector product, U(·) is the upsampling; The first intermediate feature is transformed into Processing to obtain the second intermediate feature in, For the first intermediate feature The feature map with a resolution of S / 2 obtained by pooling; Use the n-th layer path space to the n-1th intermediate feature Process and obtain the nth intermediate feature Among them, 3≤n≤N; is the n-1th intermediate feature The feature map with a resolution of S / 2 obtained by pooling; Determine the Nth intermediate feature It is the second fusion feature.

6. An autonomous driving multi-modal dynamic fusion device driven by environmental factor information, characterized in that: include: An acquisition unit, used to acquire multimodal data collected by multiple sensors on the vehicle at the current moment; An audit unit, used to perform a data redundancy review on the multimodal data to obtain the multimodal data that has passed the review; A first processing unit is used to extract vehicle features, environmental features, image features and radar features from the approved multimodal data, and process the vehicle features, environmental features, image features and radar features to obtain a scene state parameter vector; the vehicle features include steering angle, speed, acceleration and three-dimensional position; the environmental features include meteorological data, illumination, vibration displacement in the Z direction and audio; A second processing unit is used to fuse the vehicle feature, the environment feature, the image feature and the radar feature using the scene state parameter vector to obtain a first fused feature; A decoding unit, configured to decode the first fused feature using a decoder to obtain a second fused feature; A first perception unit, configured to process the second fusion feature by using an occupancy grid network head to obtain a first perception result; A second perception unit, configured to process the first perception result by using a cross attention network to obtain a second perception result; The vehicle features, environment features, image features and radar features are processed to obtain a scene state parameter vector, including: The spliced ​​vehicle features and environment features are converted into spectral features using Fourier transform; The concatenated image features and radar features are processed using a 3*3 convolutional layer to obtain the first feature; The spectrum feature and the first feature are concatenated to obtain the second feature; Perform a global pooling operation on the second feature to obtain the third feature; Use the pre-trained weight matrix and the third feature to perform single-point convolution to obtain the scene state parameter vector; The second processing unit is specifically used for: The vehicle feature, the environment feature, the image feature and the radar feature are combined to obtain a fourth feature; The scene state parameter vector is multiplied by the fourth feature to obtain the first fused feature.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images