An unmanned vehicle adaptive fusion multi-modal information all-weather perception method and system

Abstract

The application discloses an unmanned vehicle adaptive fusion multi-modal information all-weather perception method and system, the method comprises the following steps: acquiring multi-source time sequence perception data of the surrounding environment of the unmanned vehicle, and performing time sequence alignment and preprocessing on the multi-source time sequence perception data; based on the physical characteristics of the multi-source time sequence perception data, using a preset deep neural network backbone to perform initial feature coding on the multi-source time sequence perception data, extracting independent features, changing the features, obtaining multi-modal features, and mapping the multi-modal features to a unified dimension shared embedding space; respectively performing time attention enhancement and space attention enhancement on the multi-modal features, adaptively adjusting the weights of the modal features based on the current environment condition, and fusing the multi-modal features after the time and space enhancement into unified space-time joint features. The application has good universality and scalability, and is suitable for different unmanned vehicle platforms and various environment perception tasks.

Description

Technical Field

[0001] This invention relates to the field of intelligent unmanned systems and autonomous driving technology, specifically to an adaptive fusion of multimodal information all-weather perception method and system for unmanned vehicles. Background Technology

[0002] With the rapid development of autonomous driving and unmanned system technologies, the demand for unmanned vehicles in logistics delivery, urban inspection, park security, and mining transportation is increasing. Environmental perception, as the foundation for unmanned vehicles to achieve safe and autonomous decision-making and control, directly determines the overall safety and reliability of the system.

[0003] Current autonomous vehicle perception systems largely rely on single or a small number of sensors, such as visible light cameras or lidar. However, in complex environments such as rain, fog, snow, nighttime, or strong light, single-modal sensors are susceptible to severe interference, leading to a significant decrease in perception accuracy or even failure. Although multimodal sensor fusion technology has been proposed, existing methods mostly employ simple feature stitching or fixed-weight fusion strategies, making it difficult to dynamically adapt to environmental changes and fully exploit the complementary information of multimodal data in the temporal and spatial dimensions. This results in insufficient robustness of the system in complex all-weather scenarios. Therefore, there is an urgent need for a highly robust perception system capable of adaptively fusing multimodal information in the temporal and spatial dimensions to meet the practical application needs of autonomous vehicles in complex environments. Summary of the Invention

[0004] Based on this, the purpose of this invention is to provide an all-weather perception method and system for autonomous vehicles that adaptively fuses multimodal information. This aims to solve the problems that current autonomous vehicle perception systems rely on a single or a small number of sensors under complex environmental conditions, and the perception results are easily affected by factors such as rain, fog, snow, night, and strong light. Furthermore, existing multimodal fusion methods mostly adopt static fusion strategies, which are difficult to adaptively adjust the fusion weights in the time and space dimensions, resulting in insufficient system robustness.

[0005] To achieve the above objectives, this invention proposes an adaptive fusion of multimodal information for all-weather perception of unmanned vehicles. The adaptive fusion of multimodal information for all-weather perception of unmanned vehicles includes:

[0006] Acquire multi-source temporal sensing data of the environment surrounding the unmanned vehicle, and perform temporal alignment and preprocessing on the multi-source temporal sensing data;

[0007] Based on the physical characteristics of multi-source time-series sensing data, the multi-source time-series sensing data is initially encoded using a preset deep neural network backbone, independent features are extracted, the independent features are transformed to obtain multimodal features, and the multimodal features are mapped to a shared embedding space of a unified dimension.

[0008] The multimodal features are enhanced with temporal attention and spatial attention respectively. Based on the current environmental conditions, the weights of each modal feature are adaptively adjusted, and the spatiotemporally enhanced multimodal features are fused into a unified spatiotemporal joint feature.

[0009] According to one aspect of the above technical solution, the steps of acquiring multi-source temporal sensing data of the environment surrounding the unmanned vehicle, and performing temporal alignment and preprocessing on the multi-source temporal sensing data include:

[0010] Multi-source temporal perception data of the environment surrounding the autonomous vehicle are acquired using visible light cameras, millimeter-wave radar, and lidar.

[0011]

[0012]

[0013] in, For multimodal joint input vector, For visual modalities, it represents RGB image data captured by a visible light camera, defined as follows: ,in These are the height and width of the image, respectively; For radar modes, it represents the target trace or range-Doppler image returned by millimeter-wave radar, which includes the target's range, azimuth, and relative radial velocity information; For laser modes, it represents the feature tensor of 3D point cloud data acquired by lidar or after voxelization, defined as follows: ,in, For points, Number of feature channels; Let be the time series input set, with a length of The spatiotemporal observation window;

[0014] After obtaining the timing input set, timestamps are marked for each modal data based on a unified clock source, and interpolation alignment is performed at the feature level. The raw data from different sensors are uniformly converted to the vehicle coordinate system with the center of mass of the autonomous vehicle as the origin, and the coordinate data of the raw data in the vehicle coordinate system are obtained.

[0015] According to one aspect of the above technical solution, the step of initially encoding the physical features of the multi-source time-series sensing data using a preset deep neural network backbone includes:

[0016] Based on the physical characteristics of each modality of data, a pre-defined deep neural network backbone is used to perform initial feature encoding on the preprocessed multi-source time-series sensing data to adapt to the data structures of different sensors:

[0017]

[0018] Among them: These represent feature extraction networks for vision, radar, and lidar, respectively. Visual modalities are extracted using a convolutional neural network encoder. Radar modes are characterized by features extracted through a radar feature extraction network. Laser modes are characterized by feature extraction using a point cloud feature encoder; These are the learnable weight parameters for each network; This is the intermediate feature tensor output by the encoder, where the dimensions of each modal feature are... Not yet unified (i.e.) );

[0019] The intermediate feature tensors are mapped to a shared embedding space of uniform dimension using a feature projection layer:

[0020]

[0021] in, For modality The corresponding linear projection matrix; It is the bias vector; To have the same channel dimension after mapping Alignment features.

[0022] According to one aspect of the above technical solution, the steps of performing temporal attention enhancement and spatial attention enhancement on the multimodal features are as follows:

[0023] Temporal attention enhancement is applied to multimodal features, and a construction length of [length missing] is constructed. A sliding window that displays the same mode in consecutive Alignment features at each time step are stacked along the time dimension to extract the same spatial location in the past. Intra-frame feature sequences: ,in, Indicates the first The feature tensor after spatial alignment at each time step;

[0024] Using linear transformation matrix Map features to different subspaces:

[0025]

[0026] Calculate the current time With historical moments Attention weights between :

[0027]

[0028] in This is the scaling factor;

[0029] Historical eigenvalues Perform weighted aggregation to obtain temporal augmentation features that include contextual information. :

[0030] .

[0031] According to one aspect of the above technical solution, temporal enhancement features are obtained. Subsequently, average pooling and max pooling are performed on the temporal enhancement features along the channel dimension, and the two pooling results are concatenated. Spatial correlation is then extracted through a convolutional neural network layer to generate spatial attention. picture:

[0032]

[0033] in, This indicates a splicing operation along the channel dimension; Indicates the kernel size as Convolution operations; The generated two-dimensional spatial weight matrix is ​​shown, where a value close to 1 indicates that the location is a key target, and a value close to 0 indicates that the location is noise or background.

[0034] Temporal enhancement features are applied using the generated weight matrix. Perform element-wise dot product under broadcast mechanism:

[0035]

[0036] in, It is also a weight matrix.

[0037] According to one aspect of the above technical solution, after obtaining the feature tensors of each modality data after temporal and spatial attention enhancement, the feature response intensity of each modality is extracted using a global descriptor generation function, and the adaptive confidence weight of each modality is calculated through exponential normalization.

[0038] Define a global feature scoring function The global feature scoring function is composed of global average pooling and a multilayer perceptron:

[0039]

[0040] The normalized weights are computed in the modality dimension using the Softmax function. :

[0041]

[0042] in Representing different sensor modes; This indicates the proportion of importance of each modality in the final decision;

[0043] Based on the calculated dynamic weights, the features of each modality are weighted and summed to output the final spatiotemporal joint features. :

[0044] .

[0045] According to one aspect of the above technical solution, after obtaining the spatiotemporal joint features, the spatial resolution of the spatiotemporal joint features is restored, the number of feature channels is compressed into the number of categories through a convolutional layer, and then the output is mapped to the [0,1] interval through the Sigmoid activation function to obtain a pixel-level semantic probability map;

[0046] The semantic probability map is subjected to an execution threshold determination to generate a passable area binary mask. Based on the value of the area binary mask, a safe area is defined, and the vehicle is guided to drive within the safe area.

[0047] This invention also proposes an adaptive fusion of multimodal information all-weather perception system for unmanned vehicles. This system is used to implement the aforementioned adaptive fusion of multimodal information all-weather perception method for unmanned vehicles. The system includes:

[0048] The acquisition and processing module is used to acquire multi-source temporal sensing data of the environment surrounding the unmanned vehicle, and to perform temporal alignment and preprocessing on the multi-source temporal sensing data.

[0049] The feature encoding module is used to encode the physical features of the multi-source time-series sensing data using a preset deep neural network backbone, extract independent features, perform feature transformation on the independent features to obtain multimodal features, and map the multimodal features to a shared embedding space of a unified dimension.

[0050] The feature fusion module is used to perform temporal attention enhancement and spatial attention enhancement on the multimodal features respectively, and adaptively adjust the weights of each modal feature based on the current environmental conditions, and fuse the spatiotemporally enhanced multimodal features into a unified spatiotemporal joint feature.

[0051] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described adaptive fusion of multimodal information all-weather perception method for unmanned vehicles.

[0052] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the above-described adaptive fusion of multimodal information all-weather perception method for unmanned vehicles.

[0053] In summary, the adaptive fusion of multimodal information for all-weather perception of unmanned vehicles provided by this invention significantly improves the perception robustness of unmanned vehicles under complex environmental conditions such as rain, fog, night, backlight, and snow by using multimodal sensor collaborative perception and spatiotemporal attention fusion mechanisms. By dynamically modeling the importance of multimodal features in the temporal and spatial dimensions, it achieves adaptive response to environmental changes, enhancing the perception capability of dynamic targets and key areas. Through a unified feature embedding space, it achieves cross-modal feature alignment, fully mining the complementary information of multimodal data and improving the completeness and consistency of perception feature expression. When the performance of a certain perception modality degrades under specific environmental conditions, the system can automatically suppress its weight and enhance the contribution of other modalities, thereby improving the stability and reliability of the overall perception system. This invention has good versatility and scalability, and is applicable to different unmanned vehicle platforms and various environmental perception tasks.

[0054] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0055] Figure 1 This is a flowchart of the all-weather perception method for adaptive fusion of multimodal information for unmanned vehicles in Embodiment 1 of the present invention;

[0056] Figure 2 This is a schematic diagram of the structure of the unmanned vehicle adaptive fusion multimodal information all-weather perception system in Embodiment 2 of the present invention;

[0057] Figure 3 This is a structural block diagram of the electronic device in Embodiment 4 of the present invention. Detailed Implementation

[0058] To make the objectives, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the present invention will be more thorough and complete.

[0059] It should be noted that when an element is referred to as being "fixed to" another element, it can be directly on the other element or there may be an intervening element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used herein are for illustrative purposes only and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0060] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances. The term "and / or" as used herein includes any and all combinations of one or more of the related listed items.

[0061] Example 1

[0062] The figure shows a flowchart of an all-weather perception method for autonomous vehicles that adaptively fuses multimodal information according to Embodiment 1 of the present invention. The all-weather perception method for autonomous vehicles that adaptively fuses multimodal information includes the following steps S01-S03, wherein:

[0063] S01. Acquire multi-source temporal perception data of the environment surrounding the unmanned vehicle, and perform temporal alignment and preprocessing on the multi-source temporal perception data;

[0064] S02. Based on the physical characteristics of multi-source time-series sensing data, the multi-source time-series sensing data is initially encoded using a preset deep neural network backbone, independent features are extracted, the independent features are transformed to obtain multimodal features, and the multimodal features are mapped to a shared embedding space of a unified dimension.

[0065] S03. Perform temporal attention enhancement and spatial attention enhancement on the multimodal features respectively, and adaptively adjust the weights of each modal feature based on the current environmental conditions, and fuse the spatiotemporally enhanced multimodal features into a unified spatiotemporal joint feature.

[0066] This embodiment uses multiple types of sensors to collaboratively collect time-series data of the environment surrounding the autonomous vehicle, providing raw input for subsequent fusion processing. Taking three types of sensors—visible light camera, millimeter-wave radar, and lidar—as examples, multi-source time-series perception data of the environment surrounding the autonomous vehicle are acquired respectively. Since the fields of view of the visible light camera, millimeter-wave radar, and lidar partially overlap in space, and the acquisition time sequence is aligned through hardware or software synchronization mechanisms, the set of multimodal perception data collected by the autonomous vehicle at any given time is formally defined as follows for ease of computation and processing:

[0067]

[0068] in, For multimodal joint input vector, For visual modalities, it represents RGB image data captured by a visible light camera, defined as follows: ,in These are the height and width of the image, respectively; For radar modes, it represents the target trace or range-Doppler image returned by millimeter-wave radar, which includes the target's range, azimuth, and relative radial velocity information; For laser modes, it represents the feature tensor of 3D point cloud data acquired by lidar or after voxelization, defined as follows: ,in, For points, This represents the number of feature channels.

[0069] Furthermore, in order to capture the temporal characteristics of dynamic traffic flow, the system sets the length to be... The spatiotemporal observation window, whose temporal input set is represented as:

[0070]

[0071] This time series set will be used as input to a subsequent deep neural network to extract spatiotemporal correlation features.

[0072] After acquiring the time-series input set, timestamps are added to each modal data based on a unified clock source, and interpolation and alignment are performed at the feature level. The raw data from different sensors are then uniformly converted to a vehicle coordinate system with the autonomous vehicle's center of gravity as the origin, and the coordinate data of the raw data within this system is obtained. Furthermore, physical model compensation is performed to address data distortion caused by sensor characteristics or environmental interference, and outliers in each modal data are identified and filtered out based on statistical features or deep learning models.

[0073] To encode preprocessed data from different modalities into feature representations with uniform dimensions and semantic alignment, and to address the problem of direct fusion of heterogeneous features, a set of modality-specific encoders is used to perform initial feature encoding on multi-source temporal sensing data. This set of modality-specific encoders includes a convolutional neural network encoder for processing visible light image data, a point cloud feature encoder for processing LiDAR data, and a radar feature extraction network for processing millimeter-wave radar data. Each encoder operates independently and in parallel, adapting to the data structure of each sensor.

[0074] Based on the physical characteristics of each modality of data, a pre-defined deep neural network backbone is used to perform initial feature encoding on the preprocessed multi-source time-series sensing data to adapt to the data structures of different sensors:

[0075]

[0076] Among them: These represent feature extraction networks for vision, radar, and lidar, respectively. Visual modalities are extracted using a convolutional neural network encoder. Radar modes are characterized by features extracted through a radar feature extraction network. Laser modes are characterized by feature extraction using a point cloud feature encoder; These are the learnable weight parameters for each network; This is the intermediate feature tensor output by the encoder, where the dimensions of each modal feature are... Not yet unified (i.e.) );

[0077] Since cross-modal semantic alignment cannot be directly fused due to differences in the number of channels and semantic space of features extracted by different sensors, this embodiment introduces a feature projection layer to map the intermediate feature tensor to a shared embedding space of a unified dimension.

[0078]

[0079] in, For modality The corresponding linear projection matrix; It is the bias vector; To have the same channel dimension after mapping The alignment features provide a unified input format for the subsequent attention fusion module.

[0080] Temporal attention enhancement is applied to multimodal features, and a construction length of [length missing] is constructed. A sliding window that displays the same mode in consecutive Alignment features at each time step are stacked along the time dimension to extract the same spatial location in the past. Intra-frame feature sequences: ,in, Indicates the first The feature tensor after spatial alignment at each time step;

[0081] To determine "how important a particular past frame is to the current moment", a self-attention mechanism is employed, using three learnable linear transformation matrices. Map features to different subspaces:

[0082]

[0083] Calculate the current time With historical moments Attention weights between This weight reflects the contribution of historical information to the present:

[0084]

[0085] in This is a scaling factor used to prevent the gradient from vanishing due to an excessively large dot product.

[0086] Historical eigenvalues Perform weighted aggregation to obtain temporal augmentation features containing rich contextual information. :

[0087]

[0088] If a certain frame There is severe noise, and its semantic relevance to the current frame is questionable. This will be automatically reduced, thus being ignored by the network. For moving vehicles or pedestrians, the attention mechanism can capture their continuous features in the time dimension, effectively solving the problem of missed detection caused by the target being suddenly occluded in a certain frame.

[0089] Spatial weight generation: To calculate the importance of each pixel in space, it is first necessary to compress the information in the channel dimension.

[0090] Acquiring Temporal Enhancement Features back, The temporal enhancement features are subjected to average pooling and max pooling along the channel dimension to compress feature depth and retain spatial distribution information of background and significant texture. Average pooling extracts background and overall feedback information; max pooling extracts the most significant texture features (such as lane line edges and vehicle outlines).

[0091] The two pooling results are concatenated and passed through a standard convolutional layer. Using the sigmoid activation function, a spatial attention weight matrix is ​​generated with values ​​ranging from 0 to 1. This constitutes spatial attention. picture:

[0092]

[0093] in, This indicates a splicing operation along the channel dimension; Indicates the kernel size as Convolution operations; The generated two-dimensional spatial weight matrix is ​​shown, where a value close to 1 indicates that the location is a key target, and a value close to 0 indicates that the location is noise or background.

[0094] The input features are multiplied element-wise using the weight matrix, thereby enhancing the feature response of key targets at the pixel level while suppressing background noise in non-semantic regions (such as sky and road surface water reflections).

[0095] The specific implementation of the spatial attention subnetwork, after processing by the temporal attention module, still results in features that include temporal context. However, in the spatial dimension, raindrop noise and real obstacles may still have similar feature values. The generated weight matrix is ​​then used to enhance the temporal features. Perform element-wise dot product under broadcast mechanism:

[0096]

[0097] in, It is also a weight matrix.

[0098] After obtaining the feature tensors of each modality data after temporal and spatial attention enhancement, the feature response intensity of each modality is extracted using a global descriptor generation function. Adaptive confidence weights for each modality are then calculated using exponential normalization. A weighted aggregation layer is used to linearly weight and sum the multimodal features based on these adaptive confidence weights, generating spatiotemporal joint features. ;

[0099] The aggregation layer is configured to automatically reduce the weight of the visible light mode and increase the weight of the radar or lidar mode when low light or precipitation interference features are detected.

[0100] The specific implementation of adaptive multimodal fusion, after processing by the aforementioned spatiotemporal sub-network, obtains enhanced features for three modalities: (Visual) (laser), (Radar). At this point, the system needs to make dynamic decisions based on environmental conditions.

[0101] Specifically, adaptive weight generation quantifies the reliability of each modality in the current scene. First, a global feature scoring function is defined. This function, composed of global average pooling and a multilayer perceptron, is used to map a high-dimensional feature map to a scalar score.

[0102]

[0103] Next, the normalized weights are calculated in the modality dimension using the Softmax function. This ensures that the sum of the weights of all modes is 1, forming a probability distribution:

[0104]

[0105] in Representing different sensor modes; This indicates the proportion of importance of each modality in the final decision;

[0106] Based on the calculated dynamic weights, the features of each modality are weighted and summed to output the final spatiotemporal joint features. :

[0107] .

[0108] Based on spatiotemporal joint characteristics As a public input base;

[0109] The spatial resolution of features is restored through deconvolution or upsampling operations, and pixel-level semantic probability maps are output using convolutional layers.

[0110] A threshold determination is performed on the semantic probability map to generate a binarized passable region mask, which is used to indicate the legal driving boundary of the vehicle at the current moment.

[0111] Semantic decoding and probabilistic graph generation are affected by the previous feature extraction process, which includes downsampling and spatiotemporal joint features. The resolution of the image is typically smaller than that of the original image. To obtain detailed road regions, the resolution must first be restored using an upsampling module, and then... Convolutional layers compress the number of channels into the number of classes. Let... The pixel coordinates on the output feature map are assigned a probability score indicating that they belong to the "passable region" category. :

[0112]

[0113] in: Use bilinear interpolation or transposed convolution operations; The Sigmoid activation function maps the output to... Interval.

[0114] Binarized mask generation is based on a probability map. The system sets a confidence threshold. Generate the final passable region binary mask. This corresponds to the indicator function logic in your formula, whose specific pixel-level operation is defined as follows:

[0115]

[0116] The region represents the physical space where vehicles can safely travel. Visual segmentation may fail at night or on unmarked dirt roads, but the geometrical information from the LiDAR and the non-obstructive properties of the millimeter-wave radar, contained in the fused features, can help the network still output correctly. The area is designated as a no-go zone to prevent vehicles from driving into ditches or bushes.

[0117] In summary, the adaptive fusion of multimodal information for all-weather perception of unmanned vehicles provided by this invention significantly improves the perception robustness of unmanned vehicles under complex environmental conditions such as rain, fog, night, backlight, and snow by using multimodal sensor collaborative perception and spatiotemporal attention fusion mechanisms. By dynamically modeling the importance of multimodal features in the temporal and spatial dimensions, it achieves adaptive response to environmental changes, enhancing the perception capability of dynamic targets and key areas. Through a unified feature embedding space, it achieves cross-modal feature alignment, fully mining the complementary information of multimodal data and improving the completeness and consistency of perception feature expression. When the performance of a certain perception modality degrades under specific environmental conditions, the system can automatically suppress its weight and enhance the contribution of other modalities, thereby improving the stability and reliability of the overall perception system. This invention has good versatility and scalability, and is applicable to different unmanned vehicle platforms and various environmental perception tasks.

[0118] Example 2

[0119] Another aspect of this invention provides an adaptive fusion of multimodal information all-weather perception system for unmanned vehicles. Please refer to [link / reference]. Figure 2 The diagram shown is a structural schematic of the autonomous vehicle adaptive fusion multimodal information all-weather perception system in Embodiment 2 of the present invention. The autonomous vehicle adaptive fusion multimodal information all-weather perception system includes:

[0120] The acquisition and processing module 11 is used to acquire multi-source temporal sensing data of the environment around the unmanned vehicle, and to perform temporal alignment and preprocessing on the multi-source temporal sensing data.

[0121] The feature encoding module 12 is used to encode the physical features of the multi-source time-series sensing data using a preset deep neural network backbone, extract independent features, perform feature transformation on the independent features to obtain multimodal features, and map the multimodal features to a shared embedding space of a unified dimension.

[0122] The feature fusion module 13 is used to perform temporal attention enhancement and spatial attention enhancement on the multimodal features respectively, and adaptively adjust the weights of each modal feature based on the current environmental conditions, and fuse the spatiotemporally enhanced multimodal features into a unified spatiotemporal joint feature.

[0123] Example 3

[0124] In another aspect, the present invention also proposes a computer-readable storage medium having stored thereon one or more computer programs that, when executed by a processor, implement the aforementioned adaptive fusion of multimodal information all-weather perception method for unmanned vehicles.

[0125] Those skilled in the art will understand that the logic or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0126] More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable storage media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0127] Example 4

[0128] Figure 3 This is a structural block diagram of an electronic device provided in Embodiment 4. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the adaptive fusion of multimodal information all-weather perception method for unmanned vehicles described in the above embodiments. Figure 3The electronic device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0129] like Figure 3 As shown, the electronic device 30 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 32 and processor 31).

[0130] Bus 33 includes a data bus, an address bus, and a control bus.

[0131] The memory 32 may include volatile memory, such as RAM 321 (random access memory), and / or cache memory 322, and may further include ROM 323 (read-only memory).

[0132] The memory 32 may also include a program tool 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0133] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the adaptive fusion of multimodal information all-weather perception method for unmanned vehicles as described above.

[0134] Electronic device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed via I / O interface 35 (input / output interface). Furthermore, electronic device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 36. Figure 3 As shown, network adapter 36 communicates with other modules of the model-generated electronic device 30 via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated electronic device 30, including but not limited to: microcode, device drivers, redundant processors, disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0135] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0136] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0137] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A method for adaptive fusion of multimodal information for all-weather perception in unmanned vehicles, characterized in that, The adaptive fusion of multimodal information for all-weather perception in unmanned vehicles includes: Acquire multi-source temporal perception data of the environment surrounding the unmanned vehicle, and perform temporal alignment and preprocessing on the multi-source temporal perception data, wherein the multi-source temporal perception data includes visual modality, radar modality, and laser modality; Based on the physical characteristics of multi-source time-series sensing data, the multi-source time-series sensing data is initially encoded using a preset deep neural network backbone, independent features are extracted, the independent features are transformed to obtain multimodal features, and the multimodal features are mapped to a shared embedding space of a unified dimension. Temporal attention enhancement is applied to the multimodal features to obtain temporal enhancement features. Subsequently, average pooling and max pooling are performed on the temporal enhancement features along the channel dimension, and the two pooling results are concatenated. Spatial correlation is then extracted through a convolutional neural network layer to generate spatial attention. picture: in, Indicates the activation function; This indicates a splicing operation along the channel dimension; Indicates the kernel size as Convolution operations; The generated two-dimensional spatial weight matrix is ​​shown, where values ​​approaching 1 represent key targets and values ​​approaching 0 represent noise or background. AvgPool is average pooling and MaxPool is max pooling. Temporal enhancement features are applied using the generated weight matrix. Perform element-wise dot product under broadcast mechanism: in, The feature tensor after spatial attention enhancement; As the weight matrix, after obtaining the feature tensors of each modality's data after temporal and spatial attention enhancement, the feature response intensity of each modality is extracted using the global descriptor generation function, and the adaptive confidence weight of each modality is calculated through exponential normalization. Define a global feature scoring function The global feature scoring function is composed of global average pooling and a multilayer perceptron: The normalized weights are computed in the modality dimension using the Softmax function. : in Representing different sensor modes; This indicates the proportion of importance of each modality in the final decision; Based on the calculated dynamic weights, the features of each modality are weighted and summed to output the final spatiotemporal joint features. : 。 2. The adaptive fusion of multimodal information for all-weather perception of unmanned vehicles according to claim 1, characterized in that, The steps of acquiring multi-source temporal sensing data of the environment surrounding the unmanned vehicle, and performing temporal alignment and preprocessing on the multi-source temporal sensing data include: Multi-source temporal perception data of the environment surrounding the autonomous vehicle are acquired using visible light cameras, millimeter-wave radar, and lidar. in, For multimodal joint input vector, For visual modalities, it represents RGB image data captured by a visible light camera, defined as follows: ,in These are the height and width of the image, respectively; For radar modes, it represents the target trace or range-Doppler image returned by millimeter-wave radar, which includes the target's range, azimuth, and relative radial velocity information; For laser modes, it represents the feature tensor of 3D point cloud data acquired by lidar or after voxelization, defined as follows: ,in, For points, Number of feature channels; Let be the time series input set, with a length of The spatiotemporal observation window; After obtaining the timing input set, timestamps are marked for each modal data based on a unified clock source, and interpolation alignment is performed at the feature level. The raw data from different sensors are uniformly converted to the vehicle coordinate system with the center of mass of the autonomous vehicle as the origin, and the coordinate data of the raw data in the vehicle coordinate system are obtained.

3. The adaptive fusion of multimodal information for all-weather perception of unmanned vehicles according to claim 2, characterized in that, The step of encoding the physical features of the multi-source time-series sensing data using a preset deep neural network backbone includes: Based on the physical characteristics of each modality of data, a pre-defined deep neural network backbone is used to perform initial feature encoding on the preprocessed multi-source time-series sensing data to adapt to the data structures of different sensors: Among them: These represent feature extraction networks for vision, radar, and lidar, respectively. Visual modalities are extracted using a convolutional neural network encoder. Radar modes are characterized by features extracted through a radar feature extraction network. Laser modes are characterized by feature extraction using a point cloud feature encoder; These are the learnable weight parameters for each network; This is the intermediate feature tensor output by the encoder, where the dimensions of each modal feature are... Not yet unified; The intermediate feature tensors are mapped to a shared embedding space of uniform dimension using a feature projection layer: in, For modality The corresponding linear projection matrix; It is the bias vector; To have the same channel dimension after mapping Alignment features.

4. The adaptive fusion of multimodal information for all-weather perception of unmanned vehicles according to claim 1, characterized in that, The step of performing temporal attention enhancement on the multimodal features is as follows: Temporal attention enhancement is applied to multimodal features, and a construction length of [length missing] is constructed. A sliding window that displays the same mode in consecutive Alignment features at each time step are stacked along the time dimension to extract the same spatial location in the past. Intra-frame feature sequences: ,in, Indicates the first The feature tensor after spatial alignment at each time step; Using linear transformation matrix Map features to different subspaces: Calculate the current time With historical moments Attention weights between : in Scaling factor For the current moment The query feature vector, For a historic moment The key feature vector, For a historic moment The eigenvectors of the values; Historical eigenvalues Perform weighted aggregation to obtain temporal augmentation features that include contextual information. : 。 5. The adaptive fusion of multimodal information for all-weather perception of unmanned vehicles according to claim 1, characterized in that, After obtaining the spatiotemporal joint features, the spatial resolution of the spatiotemporal joint features is restored. The number of feature channels is compressed into the number of categories through a convolutional layer, and then the output is mapped to the [0,1] interval through the Sigmoid activation function to obtain a pixel-level semantic probability map. The semantic probability map is subjected to an execution threshold determination to generate a passable area binary mask. Based on the value of the area binary mask, a safe area is defined, and the vehicle is guided to drive within the safe area.

6. An adaptive fusion multimodal information all-weather perception system for unmanned vehicles, characterized in that, The autonomous vehicle adaptive fusion multimodal information all-weather perception system is used to implement the autonomous vehicle adaptive fusion multimodal information all-weather perception method according to any one of claims 1-5, and the system includes: The acquisition and processing module is used to acquire multi-source temporal perception data of the environment around the unmanned vehicle, and to perform temporal alignment and preprocessing on the multi-source temporal perception data. The multi-source temporal perception data includes visual modal, radar modal, and laser modal. The feature encoding module is used to encode the physical features of the multi-source time-series sensing data using a preset deep neural network backbone, extract independent features, perform feature transformation on the independent features to obtain multimodal features, and map the multimodal features to a shared embedding space of a unified dimension. The feature fusion module is used to perform temporal attention enhancement on the multimodal features respectively to obtain temporally enhanced features. Subsequently, average pooling and max pooling are performed on the temporal enhancement features along the channel dimension, and the two pooling results are concatenated. Spatial correlation is then extracted through a convolutional neural network layer to generate spatial attention. picture: in, Indicates the activation function; This indicates a splicing operation along the channel dimension; Indicates the kernel size as Convolution operations; The generated two-dimensional spatial weight matrix is ​​shown, where values ​​approaching 1 represent key targets and values ​​approaching 0 represent noise or background. AvgPool is average pooling and MaxPool is max pooling. Temporal enhancement features are applied using the generated weight matrix. Perform element-wise dot product under broadcast mechanism: in, The feature tensor after spatial attention enhancement; As the weight matrix, after obtaining the feature tensors of each modality's data after temporal and spatial attention enhancement, the feature response intensity of each modality is extracted using the global descriptor generation function, and the adaptive confidence weight of each modality is calculated through exponential normalization. Define a global feature scoring function The global feature scoring function is composed of global average pooling and a multilayer perceptron: The normalized weights are computed in the modality dimension using the Softmax function. : in Representing different sensor modes; This indicates the proportion of importance of each modality in the final decision; Based on the calculated dynamic weights, the features of each modality are weighted and summed to output the final spatiotemporal joint features. : 。 7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the adaptive fusion of multimodal information all-weather perception method for unmanned vehicles as described in any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the all-weather perception method for adaptive fusion of multimodal information for unmanned vehicles as described in any one of claims 1-5.

Images