Multimodal scene perception method, device, electronic device and storage medium

Abstract

The present disclosure provides a multimodal scene perception method, device, electronic device and storage medium. In the method provided in the embodiment of the present disclosure, the image features of the multi-view image sequence and the radar features obtained from the 4D radar data sequence are first extracted, and the historical radar features and the current radar features are used to model the spatiotemporal evolution of the dynamic scene and the static scene in the BEV space and the voxel space to obtain the dynamic scene features and the static scene features. Then, the image features, the dynamic scene features and the static scene features are subjected to cross-modal interactive fusion to obtain the multimodal fusion features. The multimodal fusion features can be directly used for 3D target detection, semantic occupancy prediction and / or motion state estimation. The present disclosure can achieve high-precision and high-efficiency scene understanding in complex environments.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to a multimodal scene perception method, device, electronic device, and storage medium. Background Art

[0002] With the rapid development of autonomous driving technology, the fusion of multimodal sensor data has become a core technical route to improve the reliability of environmental perception.

[0003] Currently, multimodal fusion solutions based on cameras, LiDAR, and millimeter-wave radars mainly fuse camera images, 4D radar data, and point cloud data collected by LiDAR. They use a fixed-resolution voxel encoder to extract features and rely on the attention mechanism to achieve cross-modal feature fusion. Finally, they synchronously output 3D target detection boxes and scene occupancy prediction results through a multi-task head.

[0004] Although the above scheme can achieve target detection and occupancy prediction through multimodal data fusion, it still has problems such as low efficiency and large errors in multimodal data fusion in complex scenarios such as night and rainy days, and high consumption of computing resources. Therefore, a new multimodal data fusion scheme is urgently needed to improve the efficiency and accuracy of data fusion in complex scenarios, while reducing computing resource consumption, so as to meet the real-time requirements of autonomous driving scenarios. Summary of the Invention

[0005] In view of this, the present disclosure provides a multimodal scene perception method, device, electronic device and storage medium.

[0006] According to a first aspect of the present disclosure, a multimodal scene perception method is provided, the method comprising:

[0007] Acquire a current multi-view image sequence and a current 4D radar data sequence from a multi-view camera array and a 4D imaging radar group mounted on the object;

[0008] Obtaining current image features using the current multi-view image sequence;

[0009] Acquire current radar features using the current 4D radar data sequence, where the radar features include radar voxel features and radar BEV features;

[0010] Based on the previously cached historical radar features and the current radar features, the spatiotemporal evolution of the dynamic scene and the static scene is modeled in the BEV space and the voxel space to obtain the current dynamic scene features and the current static scene features;

[0011] Cross-modal interactive fusion is performed on the current image features, the current dynamic scene features, and the current static scene features to obtain multimodal fusion features, and the multimodal fusion features are used to perform 3D target detection, semantic occupancy prediction, and / or motion state estimation on the environment surrounding the object.

[0012] In some embodiments of the first aspect of the present disclosure, the current image features are obtained by using the current multi-view image sequence, including: using an image encoder to obtain a multi-view feature map sequence based on the current multi-view image sequence, the multi-view feature map sequence contains feature maps of each view image in the current multi-view image sequence, the image encoder includes a lightweight convolutional neural network that removes the last two convolutional layers and inserts a channel attention mechanism SE module in the penultimate layer; using a feature pyramid network FPN to obtain serialized pyramid features based on the multi-view feature map sequence, the serialized pyramid features are the current image features, and the upsampling of the FPN adopts a content-aware feature reconstruction CARAFE operator.

[0013] In some embodiments of the first aspect of the present disclosure, the obtaining of the current radar features using the current 4D radar data sequence includes: performing dynamic voxel compression on the current 4D radar data sequence to obtain the current radar voxel features; obtaining the current radar BEV features based on the current radar voxel features through a radar encoder, the radar encoder including a plurality of consecutive feature extraction modules, each of the feature extraction modules including a sparse convolution layer, the convolution kernels of the sparse convolution layers in the plurality of consecutive feature extraction modules having the same size and the number of channels increasing step by step.

[0014] In some embodiments of the first aspect of the present disclosure, modeling the spatiotemporal evolution of dynamic scenes and static scenes in BEV space and voxel space based on previously cached historical radar features and current radar features to obtain current dynamic scene features and current static scene features includes:

[0015] Reading previously cached historical radar features from a buffer, wherein the historical radar features include historical radar voxel features and historical radar BEV features;

[0016] A multi-scale atrous temporal fusion network is used to obtain dynamic scene features and static scene features based on the historical radar features and the current radar features, wherein the multi-scale atrous temporal fusion network includes a atrous temporal convolutional network, and the atrous temporal convolutional network includes a plurality of consecutive atrous convolutional layers, wherein atrous rates of the plurality of consecutive atrous convolutional layers increase exponentially in layers and convolution kernels have the same size;

[0017] Dynamic target compensation is performed on the dynamic scene features using a multi-target Kalman filter tracking method.

[0018] In some implementations of the first aspect of the present disclosure, performing cross-modal interactive fusion on the current image feature, the current dynamic scene feature, and the current static scene feature to obtain a multimodal fusion feature includes:

[0019] fusing the current image feature and the current static scene feature to obtain an image voxel feature;

[0020] Dynamically fusing the dynamic scene features and the image voxel features using a gated cross-attention module to obtain cross-modal interaction features;

[0021] fusing the dynamic scene feature with the image voxel feature under a geometric consistency constraint to obtain a geometric alignment feature;

[0022] The geometric alignment feature and the cross-modal interaction feature are integrated through multi-scale pyramid pooling to obtain the multimodal fusion feature.

[0023] In some implementations of the first aspect of the present disclosure, dynamically fusing the dynamic scene features and the image voxel features through a gated cross-attention module to obtain cross-modal interaction features includes:

[0024] Calculating a gating matrix of the image voxel feature and the dynamic scene feature, wherein the gating matrix represents a fusion weight of the dynamic scene feature;

[0025] dynamically adjusting the gating matrix according to current weather conditions;

[0026] The image voxel features and the dynamic scene features are fused based on the dynamically adjusted gating matrix to obtain the cross-modal interaction features.

[0027] In some implementations of the first aspect of the present disclosure, the method further includes: obtaining a 3D target detection result, a semantic occupancy prediction result and / or a motion estimation result of the surrounding environment of the object based on the multimodal fusion feature.

[0028] According to a second aspect of the present disclosure, a multimodal scene perception device is provided, comprising:

[0029] a data acquisition unit, configured to acquire a current multi-view image sequence and a current 4D radar data sequence from a multi-view camera array and a 4D imaging radar group mounted on the object;

[0030] An image feature extraction unit, configured to obtain current image features using the current multi-view image sequence;

[0031] a radar feature extraction unit, configured to obtain current radar features using the current 4D radar data sequence, wherein the radar features include radar voxel features and radar BEV features;

[0032] A temporal fusion unit is used to model the spatiotemporal evolution of dynamic scenes and static scenes in BEV space and voxel space based on previously cached historical radar features and current radar features to obtain current dynamic scene features and current static scene features;

[0033] A cross-modal interactive fusion unit is used to perform cross-modal interactive fusion on the current image features, the current dynamic scene features and the current static scene features to obtain multimodal fusion features, and the multimodal fusion features are used to perform 3D target detection, semantic occupancy prediction and / or motion state estimation on the environment around the object.

[0034] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0035] at least one processor; and

[0036] a memory communicatively coupled to the at least one processor;

[0037] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0038] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0039] As can be seen from the above, the embodiments of the present disclosure provide a new multimodal fusion method for multi-view image sequences and 4D radar data sequences, which can achieve high-precision and high-efficiency 3D target detection, semantic occupancy prediction and scene understanding in complex environments, and can meet the high real-time requirements, low computing resource requirements, high robustness and reliability requirements of scenarios such as autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 A schematic diagram of the system architecture to which the embodiments of the present disclosure are applicable;

[0042] Figure 2 A schematic diagram of a flow chart of a multimodal scene perception method provided in an embodiment of the present disclosure;

[0043] Figure 3 A schematic diagram of the structure of a scene perception model involved in an embodiment of the present disclosure;

[0044] Figure 4 Schematic diagram of a specific implementation process of modeling the spatiotemporal evolution of dynamic scenes and static scenes in BEV space and voxel space according to an embodiment of the present disclosure;

[0045] Figure 5 This is a schematic diagram of a specific implementation process of cross-modal interactive fusion involved in an embodiment of the present disclosure;

[0046] Figure 6 A schematic diagram of the structure of a multimodal scene perception device is provided for an embodiment of the present disclosure;

[0047] Figure 7 A schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0049] The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0050] As used herein, the words "if," "if," and the like may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0051] As previously mentioned, related technologies still suffer from low fusion efficiency, low resource utilization, a sharp drop in accuracy and reliability, difficulty in reconstructing complex-shaped objects, and a high rate of missed detection in static scenes when using multimodal data fusion in extreme scenarios such as thunderstorms and at night. In view of this, the present disclosure provides the following multimodal scene perception methods, devices, electronic devices, and storage media. These methods, using cameras and 4D radars as core sensors, achieve efficient multimodal scene perception through lightweight feature extraction, temporal-spatial joint modeling, and cross-modal adaptive fusion, thereby reliably achieving various tasks such as 3D object detection, semantic occupancy prediction, and motion state estimation.

[0052] To facilitate understanding, a brief description is first given of the system structure applicable to the embodiments of the present disclosure.

[0053] Figure 1 Schematic diagram of the structure of the system to which the embodiment of the present disclosure is applicable. Figure 1 The system to which the embodiments of the present disclosure apply may include: an electronic device and a peripheral sensor component connected to the electronic device, wherein the peripheral sensor component includes a multi-view camera array and a 4D imaging radar group.

[0054] The multi-view camera array can be used to collect multi-view images of the environment in which the vehicle is located, and the multi-view images can cover the surroundings of the environment in which the vehicle is located.

[0055] For example, the multi-view camera array can be implemented as, but is not limited to, a six-view camera array, including a front-view camera, a rear-view camera, a left-view camera, a right-view camera, an upward-view camera, and a downward-view camera. Taking the Traffic Jam Assistant (TJA) system as an example, the cameras in the multi-view camera array can use, but are not limited to, 1920×1080@30FPS and a 120° wide-angle lens.

[0056] In specific applications, cameras can be evenly deployed around the vehicle to ensure full coverage of the vehicle's surroundings. Each camera captures an image of the vehicle's surroundings that includes a portion of the scene surrounding the vehicle. The present disclosure does not limit the specific implementation of the multi-view camera array.

[0057] The 4D imaging radar array can be used to collect 4D radar data of the vehicle's environment. This 4D radar data can cover all four sides of the vehicle's environment. For example, a 4D imaging radar array can include four 4D imaging radars: a forward 4D imaging radar array, a left and right 4D imaging radar array, and a rearward 4D imaging radar array. Another example is a 4D imaging radar array can include six 4D imaging radars: a dual forward 4D imaging radar array, a left and right 4D imaging radar array, and a dual rearward 4D imaging radar array. The number of 4D imaging radars in a 4D imaging radar array and their deployment locations must be considered comprehensively, taking into account functional coverage, computing power allocation, cost, and reliability. Flexible deployment can be achieved based on actual needs in specific applications.

[0058] 4D radar data may include, but is not limited to, the target's spatial position, velocity, reflection intensity, and signal-to-noise ratio (SNR). Here, "target" refers to the physical entities detected by the 4D imaging radar array. These physical entities can be, but are not limited to, dynamic or static objects. Moving physical entities such as cars, trucks, motorcycles, bicycles, and pedestrians are considered dynamic objects, while stationary physical entities such as roadblocks, curbs, stationary vehicles, traffic signs, streetlights, bridges, tunnels, fixed ground structures, manhole covers, potholes, road cracks, cables, and tree branches are considered static objects.

[0059] The spatial position of a target includes its distance, azimuth, and elevation in three-dimensional space. Distance is the straight-line distance between the radar and the target, determined by calculating the time difference between signal transmission and reception. Azimuth represents the horizontal angle (left-right) of the target relative to the radar, while elevation represents the vertical angle (up-down) of the target relative to the radar.

[0060] Velocity refers to the radial velocity of an object relative to the radar (i.e., the velocity component along the radar beam), which can be measured using the Doppler effect. Velocity can be used to distinguish between stationary and moving objects (such as vehicles and pedestrians).

[0061] Reflection intensity refers to the strength of electromagnetic wave energy reflected back to the radar by a target. It is related to the target's material, surface roughness, geometry, and radar cross-section (RCS). Metallic objects, such as vehicles, have high reflection intensity, while non-metallic objects, such as pedestrians, have low reflection intensity. Reflection intensity can assist in target classification and enhance the accuracy of environmental perception.

[0062] The signal-to-noise ratio (SNR) is the ratio of the useful signal to the background noise in a radar signal, reflecting signal quality. A high SNR indicates reliable target detection and high data confidence; a low SNR indicates possible false detection or missed detection due to noise interference. SNR can be used for data filtering to improve system robustness.

[0063] In specific applications, the systems applicable to the embodiments of the present disclosure can be applied to, but are not limited to, intelligent control of multiple wheeled mobile robots, wheeled mobile robots, mobile robots, vehicles, aircraft, ships, Autonomous Rail Rapid Transit (ART) systems, industrial automation equipment, and the like. Vehicles can include, but are not limited to, passenger cars, commercial vehicles (e.g., trucks, buses, vans, etc.), special-purpose vehicles (e.g., ambulances, fire trucks, engineering vehicles, rescue vehicles, etc.), agricultural and industrial vehicles (e.g., harvesters, forklifts, etc.), transportation and logistics vehicles (e.g., container trucks, refrigerated trucks, etc.), new energy vehicles (e.g., electric vehicles, hybrid vehicles), and special vehicles (e.g., garbage trucks, water trucks, etc.). In other words, the "vehicle" in the embodiments of the present disclosure is equivalent to the various devices mentioned above.

[0064] The embodiments of the present disclosure can be applied to scenarios such as urban transportation, highways, ports, mines, farms, closed parks, and industrial production. They can be applied to many aspects, such as travel by car, public transportation, logistics distribution, unmanned transportation, terminal delivery, automated agricultural operations, and automated sanitation. Of course, the embodiments of the present disclosure can also be applied to any other intelligent control scenarios involving equipment such as vehicles. The present disclosure does not limit the application scenarios and applicable fields of the embodiments of the present disclosure.

[0065] The system to which the embodiments of the present disclosure are applicable may be, but is not limited to, any system requiring multimodal data fusion. For example, the system may be, but is not limited to, an autonomous driving system, an intelligent assisted driving system, a traffic congestion assistance system, etc. Figure 1 The system provided by the embodiments of the present disclosure can be installed in a vehicle and used as, but not limited to, an intelligent assisted driving system, a traffic congestion assistance system, an automatic driving system, etc. of the vehicle.

[0066] In addition, those skilled in the art should understand that the system to which the embodiments of the present disclosure are applicable is not limited to Figure 1 The application scenarios of the architecture shown are not limited to the above ones.

[0067] The specific implementation of the embodiment of the present disclosure is described in detail below.

[0068] Figure 2 The flowchart of the multimodal scene perception method provided by the embodiment of the present disclosure is shown. The multimodal scene perception method can be Figure 1 The electronics in the system shown perform. Figure 2 The multimodal scene perception method of the embodiment of the present disclosure includes the following steps:

[0069] Step 201 , obtaining a current multi-view image sequence and a current 4D radar data sequence from a multi-view camera array and a 4D imaging radar group mounted on an object;

[0070] Step 202, obtaining current image features using the current multi-view image sequence;

[0071] Step 203 , obtaining current radar features using the current 4D radar data sequence, where the radar features include radar voxel features and radar bird's eye view (BEV) features;

[0072] Step 204 , based on the previously cached historical radar features and the current radar features, model the spatiotemporal evolution of the dynamic scene and the static scene in the BEV space and the voxel space to obtain the current dynamic scene features and the current static scene features;

[0073] Among them, dynamic scene features can be used to characterize the spatiotemporal evolution of dynamic objects in the environment around the object, such as vehicles and pedestrians, and static scene features can be used to characterize the spatiotemporal evolution of static objects in the environment around the vehicle, such as roads, traffic signs, and signal lights.

[0074] For example, dynamic features can be in various formats, such as BEV format and trajectory sequences, while static features can be in various formats, such as voxel format, BEV format, or point cloud. Taking a vehicle as an example, dynamic features can be implemented in the vehicle coordinate system for real-time local perception, while static features can be in the world coordinate system to ensure global consistency.

[0075] In step 205, cross-modal interactive fusion is performed on the current image features, the current dynamic scene features, and the current static scene features to obtain multimodal fusion features. The multimodal fusion features can be used to obtain 3D target detection, semantic occupancy prediction, and / or motion state estimation about the environment surrounding the object.

[0076] Here, the object may be, but is not limited to, the aforementioned objects such as multiple wheeled mobile robots, wheeled mobile robots, mobile robots, vehicles, aircraft, ships, Autonomous Rail Rapid Transit (ART) systems, industrial automation equipment, etc. For details, please refer to the above description and will not be repeated here.

[0077] The disclosed embodiment collects multimodal data through a multi-view camera array and a 4D imaging radar group, and successively extracts image features, extracts radar features, models the spatiotemporal evolution of dynamic and static scenes in BEV space and voxel space, and performs cross-modal interactive fusion to obtain multimodal fusion features obtained by fusing multi-view images and 4D radar data. The multimodal fusion features can be used to obtain 3D target detection, semantic occupancy prediction, and / or motion about the surrounding environment of the object. Therefore, the disclosed embodiment provides a new multimodal fusion method that can reduce the impact of fusion bias while reducing computing resource consumption. While reducing computing resource requirements, it improves the efficiency and accuracy of data fusion in complex scenarios. It can achieve high-precision and high-efficiency scene perception and understanding in complex environments while taking into account system robustness and computing efficiency, thereby meeting the real-time requirements, low computing resource requirements, high robustness and reliability requirements of various scenarios such as autonomous driving.

[0078] The multimodal scene perception method of the embodiment of the present disclosure can be implemented through a pre-built scene perception model. Figure 3 The schematic diagram of the scene perception model is shown in FIG. Figure 3 The scene perception model may include a sensor input layer, an image feature extraction layer, a radar feature extraction layer, a time series fusion layer, a cross-modal interaction layer and a multi-task output layer. The sensor input layer is used to implement the aforementioned step 201, the image feature extraction layer is used to implement step 202, the radar feature extraction layer can be used to implement step 203, the time series fusion layer is used to implement the aforementioned step 204, the cross-modal interaction layer is used to implement the aforementioned step 205, and the multi-task output layer is used to implement the following step 206.

[0079] In step 201, the multi-view image sequence and the 4D radar data sequence are time-synchronized. Specifically, the time synchronization of the multi-view image sequence and the 4D radar data sequence can be achieved through mechanisms such as synchronous triggering and time alignment of the multi-view camera array and the 4D imaging radar group.

[0080] Before step 202 , the multi-view image sequence may be pre-processed. The pre-processing may include, but is not limited to, image signal processor (ISP) processing, de-distortion / white balancing, etc. for the original image (eg, RAW image).

[0081] Before step 203 , the 4D radar data sequence may also be preprocessed. The preprocessing may include but is not limited to multi-frame accumulation, Doppler compensation, and the like.

[0082] There are many specific implementations for obtaining the image features of the multi-view image sequence in step 202. In some implementations, the image features of the multi-view image sequence can be obtained using a pre-built image encoder and a feature pyramid network (FPN).

[0083] Furthermore, step 202 may include the following steps a1 and a2:

[0084] Step a1: Use an image encoder to process the current multi-view image sequence to obtain a multi-view feature map sequence, where the multi-view feature map sequence includes feature maps of each view image in the current multi-view image sequence.

[0085] In step a2, a Feature Pyramid Network (FPN) is used to process the multi-view feature map sequence to obtain serialized pyramid features. The serialized pyramid features are the image features of the multi-view image. The upsampling of the FPN can use the Content-Aware Reassembly of Features (CARAFE) operator.

[0086] As described above, feature extraction from multi-view image sequences is achieved through a lightweight image editor, SE module, and content-aware multi-scale fusion (CARAFE-FPN). This reduces model parameters by approximately 9.3%, improves inference speed by 29%, and CARAFE dynamic upsampling improves small object AP by 3.6%. CARAFE-FPN's independent processing and enhancement strategy ensures spatial alignment accuracy (+5.5%) for cross-view features, balancing computational efficiency with multi-scale feature representation while preserving multi-view detail and spatial consistency. Furthermore, this approach is particularly suitable for scenarios requiring real-time processing of multi-view image data on mobile devices, achieving an optimal balance between accuracy and efficiency.

[0087] In step a1, the image encoder can include a lightweight convolutional neural network with the last two convolutional layers removed and a channel attention mechanism (Squeeze-and-Excitation, SE) module inserted in the penultimate layer.

[0088] For example, the lightweight convolutional neural network can be, but is not limited to, MobileNetV3-Large. That is, the image encoder's backbone network can be a MobileNetV3-Large with the last two convolutional layers removed and an SE module inserted in the penultimate layer. For example, the image encoder's backbone network can be configured as follows: after removing the last two convolutional layers from the MobileNetV3-Large model, an SE module with a compression ratio of 16 can be inserted in the penultimate layer. In practical applications, the compression ratio of the SE module can be adjusted as needed to further optimize performance.

[0089] The SE module is a lightweight channel attention mechanism that dynamically adjusts the weights of the channel dimensions of a two-dimensional feature map to enhance the feature response of key channels. For example, the penultimate layer of lightweight convolutional neural networks such as MobileNetV3-Large is a key location for extracting high-level semantic features. Inserting the SE module into this penultimate layer of a lightweight convolutional neural network can enhance key information within high-level semantic features, prevent information loss, and optimize computational efficiency. This significantly improves accuracy and robustness while maintaining low computational overhead. This makes it more suitable for tasks requiring fine-grained spatial information while retaining its lightweight advantages.

[0090] Specifically, the image encoder may include: an initial convolutional layer, multiple inverted residual blocks, and an SE module connected in sequence. Specifically, in step a1, the following steps a11 to a13 may be performed on each view in the multi-view image sequence to obtain a feature map of each view:

[0091] Step a11: extracting a first feature map of the image using an initial convolutional layer, where the height and width of the first feature map are half the size of the corresponding dimensions of the image;

[0092] Step a12: Processing the first feature map using multiple inverted residual blocks to obtain a second feature map, where the height and width of the second feature map are 1 / 16 of the corresponding dimensions of the image, and the high-resolution spatial information of the image is retained in the second feature map;

[0093] In step a13, the SE module is used to dynamically adjust the weights of each channel in the second feature map to capture key information, thereby obtaining a third feature map. The height and width of the third feature map are 1 / 16 of the corresponding dimensions of the image, and the channel dimension can be fixed to a predetermined value (e.g., 1). For example, the channel dimension of the third feature map can be 256.

[0094] For example, if the size C×H×W of the image of each view in the multi-view image sequence is 1×544×960, the size of the third feature map can be: 256×34×60, where C represents the channel size, H represents the height size, and W represents the width size.

[0095] Among them, the specific process of using the SE module to dynamically adjust the weights of each channel in the second feature map to capture key information can include: first, Squeeze: perform global average pooling on each channel of the second feature map, compress the two-dimensional spatial information of each channel into a scalar, and obtain a channel description vector; second, Excitation: learn the dependency between channels through the fully connected layer to generate channel weights; finally, multiply the weight vector and the second feature map channel by channel to obtain the third feature map to enhance the response of important channels.

[0096] As can be seen from the above, by combining the SE module and the lightweight convolutional neural network to form an image encoder, it is possible to extract high-resolution spatial information of the image while retaining the global context, strengthen the capture of key features through channel attention, optimize computational efficiency, and achieve an effective balance between efficiency and accuracy.

[0097] The checkerboard effect is caused by uneven overlap in deconvolution. When the convolution kernel size is not divisible by the stride, the deconvolution output will overlap unevenly, resulting in checkerboard-like structures or jagged edges in the feature map. In step a2, when using FPN to fuse a multi-view feature map sequence, the CARAFE operator is used for FPN upsampling to reduce this checkerboard effect, thereby further improving accuracy and robustness while maintaining low computational overhead.

[0098] Specifically, CARAFE can be used as an upsampling method in the top-down path of FPN to upsample high-level features and fuse them with low-level features. CARAFE is an efficient and lightweight feature upsampling module. It dynamically adjusts the upsampling process through a content-aware mechanism, avoiding the uneven overlap that can occur in traditional upsampling methods. This significantly improves upsampling accuracy and efficiency. It also aggregates contextual information within a large receptive field, thereby better utilizing surrounding information and reducing the checkerboard effect. Furthermore, CARAFE has minimal computational overhead, enabling lightweight design.

[0099] In step a2, after FPN feature enhancement, the feature maps for each view are multi-scale enhanced to generate a set of feature pyramid features, thereby enhancing the perception of objects of different sizes. That is, FPN processing of the multi-view feature map sequence yields serialized pyramid features, with each view corresponding to a set of feature pyramids from the multiple sets of serialized pyramid features. These serialized pyramid features are the image features of the multi-view image sequence obtained in step 202.

[0100] In step a2, CARAFE restores details and improves spatial resolution, while FPN optimizes feature expression and enhances semantic information. The combination of CARAFE and FPN can significantly improve the representation ability of multi-scale features, enhance the accuracy of small target detection and edge segmentation, and achieve significant performance gains at a minimal computational cost without the need for complex structural adjustments. It excels in detail preservation, small target detection, and spatial consistency, while addressing the bottlenecks of traditional methods in detail loss and spatial misalignment.

[0101] Taking a six-view camera array as an example, the multi-view image sequence is a six-view image sequence [6, 1, 544, 960], where “6” represents the number of views. The processing of step 202 may include: the six-view image sequence [6, 1, 544, 960] is processed by an image encoder to obtain a six-view feature map sequence [6, 256, 34, 60], where “6” represents the number of views. The six-view feature map sequence can also be expressed as {F1, F2, ..., F6}, Fi∈R 256×34×60 , i = 1, 2, 3, …, 6; each view's feature map sequence is independently processed with FPN to produce a sequence of six feature pyramids, each corresponding to a view. A feature pyramid is a hierarchical structure consisting of multiple feature maps of different resolutions. For example, the feature pyramids corresponding to view 1 can be represented as {P3_1, P4_1, P5_1}, the feature pyramids corresponding to view 2 can be represented as {P3_2, P4_2, P5_2}, and the feature pyramids corresponding to view 3 can be represented as {P3_3, P4_3, P5_3}. The other views are similar and will not be further described. Among them, P3 represents a low-level feature map, which has high resolution, weak semantic information, but rich detail information, and is suitable for detecting small objects; P4 represents a mid-level feature map, which has low resolution and strong semantic information, and is suitable for detecting medium-sized objects; P5 represents a high-level feature map, which has low resolution and strong semantic information, and is suitable for detecting large objects.

[0102] Furthermore, step 203 may include: performing dynamic voxel compression on the current 4D radar data sequence to obtain current radar voxel features; and processing the radar voxel features through a radar encoder to obtain a radar BEV feature map, the radar encoder including a plurality of consecutive feature extraction modules, each feature extraction module including a sparse convolution layer, and the convolution kernels of the sparse convolution layers in the plurality of consecutive feature extraction modules have the same size and the number of channels increases step by step.

[0103] Specifically, the dynamic voxel compression process can include: dividing the three-dimensional space into a non-uniform voxel grid by adaptively adjusting the voxel resolution, mapping the point cloud coordinates corresponding to the 4D radar data sequence to the corresponding voxel position in the voxel grid, extracting statistical features from the points within each voxel, retaining only non-empty voxels, and using an octree to hierarchically encode the spatial positional relationships of these voxels to achieve efficient neighborhood queries and dynamically update and store non-empty voxels. Dynamic voxel compression can achieve efficient and accurate voxelization of 4D radar data sequences, providing an efficient and accurate voxelized data foundation for 3D perception tasks such as target detection and semantic occupancy prediction, reducing memory usage by 30%, and improving the detection AP of dynamic targets such as pedestrians by 3.5%.

[0104] The point cloud coordinates corresponding to the 4D radar data sequence can be obtained by performing coordinate system transformation on the spatial position of the target in the 4D radar data sequence. The point cloud coordinates can be expressed as, but not limited to, three-dimensional rectangular coordinate system coordinates.

[0105] For example, the point cloud coordinates corresponding to the 4D radar data sequence can be mapped to corresponding voxel positions in the voxel grid by rounding operations or other methods. The number of points within each voxel does not exceed a second preset threshold (e.g., 32 points). If the number of points within a voxel exceeds the second preset threshold, some points can be randomly retained to avoid computational overload.

[0106] Statistical feature extraction for points within each voxel involves calculating the position mean (x, y, z), maximum reflection intensity, and velocity variance for each point within the voxel. The position mean is calculated by averaging the spatial positions (i.e., x, y, z coordinates) of all points within the voxel, while the position mean represents the voxel center position. The maximum reflection intensity is obtained by taking the maximum reflection intensity of all points within the voxel, which is used to distinguish target materials. The velocity variance can be calculated based on the velocity of all points within the voxel and can be used to identify dynamic targets such as vehicles and pedestrians.

[0107] Furthermore, in dynamic voxel compression, the voxel resolution (0.1m³~0.4m³) of each area in the point cloud space of the 4D radar data sequence can be adaptively adjusted through confidence-driven subdivision rules so that target-dense areas use fine granularity and open areas use coarse granularity, thereby achieving adaptive adjustment of voxel resolution.

[0108] Specifically, the confidence-driven segmentation rule can be configured as follows: when the detection confidence of a certain area is greater than a first preset threshold (for example, 0.5), the probability of the target existence in the area is determined to be high, and it belongs to a high-confidence area. The voxels in the high-confidence area adopt a fine-grained first predetermined resolution (for example, 0.1m×0.1m×0.2m) to capture target details such as vehicle edges and pedestrian postures with higher accuracy.

[0109] Assuming that the initial voxel size of dynamic voxel compression is set to 0.4m×0.4m×0.8m, the subdivision rules of dynamic voxel compression may further include: when the detection confidence of a certain area is less than or equal to a first preset threshold (for example, 0.5), determining that the probability of the existence of the target in the area is high and that the area belongs to a low-confidence area, and maintaining a coarse-grained second predetermined resolution (for example, an initial voxel size of 0.4m×0.4m×0.8m) to reduce computational redundancy.

[0110] Specifically, the process of adaptively adjusting voxel resolution using confidence-driven subdivision rules can include traversing the point cloud space of the 4D radar data sequence, determining the voxel size of each region based on the confidence heat map, applying a first predetermined resolution to regions with confidence greater than a preset threshold, and retaining the voxels of other regions at a default size, or second predetermined resolution. This allows for adaptive dynamic voxel compression with high precision in critical areas and high efficiency in non-critical areas.

[0111] Furthermore, in dynamic voxel compression, a block-wise computation method can be used to extract statistical features from points within each voxel. Specifically, all points within a voxel can be divided into multiple sub-blocks of a predetermined size (e.g., 8×8). The attention weight is calculated for each sub-block separately, and a predetermined number (e.g., 10 points) of key points within the sub-block are selected based on the attention weight and retained as sampling points (e.g., the top 10% of key points are retained after sorting by attention weight from high to low). Statistical feature extraction is then performed on the sampling points to reduce computational complexity and improve efficiency.

[0112] Furthermore, the Compute Unified Device Architecture (CUDA) can be used in parallel to build Octrees in dynamic voxel compression to update the voxel index table in real time. That is, using CUDA to build the voxel index table in parallel can quickly map each point in the 4D radar data sequence to the corresponding voxel position, recording the voxel ID, number of points, statistical features, etc. Using CUDA to build the voxel index table in parallel can accelerate the optimization of dynamic voxel compression of 4D radar data sequences, compressing the dynamic voxel compression time to 10ms, meeting real-time requirements, and further reducing memory usage by 30%.

[0113] In step 203, the radar encoder can use a fixed-size convolution kernel and a channel-increasing strategy to strike a balance between computational efficiency and feature expression capability. Furthermore, the feature extraction modules in the radar encoder can adopt the same hierarchical structure. Each feature extraction module can include a sparse convolution layer with a predetermined convolution kernel size (e.g., 3×3×3), a batch normalization (BN) layer, and a ReLU activation function layer, all connected in sequence.

[0114] In some examples, a radar encoder may include multiple consecutive feature extraction modules, each of which includes a sparse convolutional layer (SparseConv). The convolution kernels of the sparse convolutional layers in these multiple consecutive feature extraction modules have the same size and the number of channels increases step by step to gradually extract higher-dimensional abstract features, forming an encoding process from local details to global semantics.

[0115] For example, the radar encoder can include multiple consecutive feature extraction modules. The convolution kernels of the sparse convolution layers in these three consecutive feature extraction modules have the same size and the number of channels doubles step by step, specifically 64→128→256. That is, the number of output data channels of the sparse convolution layer of the first feature extraction module is 64, the input data of the sparse convolution layer of the second feature extraction module is 64 channels and the output data is 128 channels, and the input data of the sparse convolution layer of the third feature extraction module is 128 channels and the output data is 256 channels. The size of the radar BEV feature obtained by this radar encoder is 256×160×240.

[0116] Furthermore, after obtaining the radar BEV features through the radar encoder, step 203 may also include the step of correcting the spatiotemporal misalignment in the radar BEV features through DBSCAN clustering and Hungarian matching combined with Q-matrix adaptive Kalman filtering.

[0117] In some embodiments, the process of correcting the spatiotemporal misalignment in the radar BEV signature may include the following steps b1 to b3:

[0118] Step b1, segment the point cloud in the radar BEV feature into independent targets through DBSCAN clustering;

[0119] Step b2: Hungarian matching is used to associate the clustering results of the current frame with the historical trajectory to maintain the consistency of the target ID.

[0120] In step b3, the state of each target is predicted and updated through Kalman filtering to obtain a prediction result and the position of the target in the radar BEV feature is compensated according to the prediction result of the Kalman filter. The state vector of the Kalman filter tracks the position and velocity of the target and the Q matrix is ​​adaptively adjusted.

[0121] Among them, the neighborhood radius eps of DBSCAN clustering can be set to 0.5m.

[0122] High-precision data association is achieved through DBSCAN clustering and Hungarian matching, and the robustness of complex scenarios is improved by combining Q-matrix adaptive Kalman filtering, which can effectively cope with the motion compensation and multi-target tracking challenges in radar BEV characteristics.

[0123] Figure 4 FIG2 shows a schematic diagram of a specific implementation process of modeling the spatiotemporal evolution of dynamic scenes and static scenes in BEV space and voxel space in step 204. Figure 4 , step 204 may include the following steps 401 to 403:

[0124] Step 401 , reading previously cached historical radar features from a buffer, the historical radar features including historical radar voxel features and historical radar BEV features;

[0125] The buffer can have a storage depth of 4 frames (the current frame + 3 historical frames), and the data structure can be a ring buffer covering a 1.2-second timing window. A ring buffer is a first-in-first-out data structure that efficiently manages continuous data. It has a fixed capacity and supports a cyclic overwrite mechanism.

[0126] Step 402: Using a multi-scale atrous temporal fusion network to obtain dynamic scene features and static scene features based on historical radar features and current radar features;

[0127] The multi-scale dilated temporal fusion network includes a dilated temporal convolutional network (Dilated TCN). This network consists of multiple consecutive dilated convolutional layers, each with an exponentially increasing dilation rate and uniform kernel size. This exponentially expands the receptive field, enabling the capture of global evolution patterns in both dynamic and static scenes.

[0128] The multi-scale atrous temporal fusion network can also include a dual-branch decoder to explicitly separate dynamic scene features from static scene features.

[0129] For example, a dilated temporal convolutional network can include three consecutive dilated convolutional layers, where the dilation rates of the three consecutive dilated convolutional layers are 1, 2, and 4, respectively. The convolution kernel size can be set to 3×3, and the number of channels of the output data can be set to 256.

[0130] In the dual-modal modeling of BEV space and voxel space, the multi-scale void time series fusion network adopts an exponentially growing void ratio and a multi-layer stacking structure that can cover multiple frames of data to capture the long-term evolution law when modeling the global scene evolution of static and dynamic features. The long-term stability of static elements such as road structures and guardrails is captured through low void ratio convolution (void ratio = 1) to extract static scene features. The acceleration changes of the target and the trajectory inflection points are associated through void convolution stacking with exponential growth of void ratio (void ratio greater than 1) to predict future motion trends to model dynamic scene features.

[0131] Furthermore, the multi-scale atrous temporal fusion network can use multi-layer stacking with low atrous rates to obtain static scene features, adopt mixed atrous rates to obtain dynamic scene features, and improve the global scene modeling capability through modal fusion.

[0132] The disclosed embodiment adopts a multi-scale void temporal fusion network to model the spatiotemporal evolution of dynamic and static scenes in BEV and voxel space. Compared with the related technology using LSTM, the number of parameters is reduced from 3.8M to about 1.2M, and the inference speed can be increased by about 30%, which can significantly improve efficiency while reducing computing resource consumption.

[0133] Step 403: Perform dynamic target compensation on the dynamic scene features using a multi-target Kalman filter tracking method.

[0134] Specifically, the process of using multi-target Kalman filter tracking to perform dynamic target compensation on dynamic scene features may include the following steps c1 to c5:

[0135] Step c1: convert the point cloud corresponding to the dynamic scene features into the global coordinate system to eliminate the observation distortion caused by the vehicle motion;

[0136] Specifically, the dynamic scene feature point cloud is converted to a global coordinate system (such as UTM or world coordinate system) based on the vehicle positioning system (such as GNSS / IMU fusion posture), and the residual distortion caused by positioning delay or error is corrected through the motion compensation algorithm (such as eliminating rotation distortion based on IMU angular velocity integration).

[0137] Step c2, segment the point cloud of dynamic scene features into independent targets through DBSCAN clustering;

[0138] Specifically, we segment point clouds of dynamic scene features based on ‌Adaptive DBSCAN clustering‌.

[0139] Step c3: associate the detection cluster of the current frame with the historical trajectory through Hungarian matching to maintain ID consistency;

[0140] Specifically, the current detection cluster is associated with the historical trajectory through ‌Multimodal Hungarian Matching‌.

[0141] Step c4, performing state estimation on each target through independent Kalman filter tracking to obtain a prediction result of each target, such as the next frame position;

[0142] Specifically, each target is assigned an independent Kalman filter (KF). The Kalman filter is initialized for each target and then updated to predict the next frame position of each target. For example, an Interacting Multi-Model Kalman Filter (IMM-KF) can be assigned to each target to support adaptive switching between multiple motion models (e.g., uniform velocity (CV) and uniform angular velocity (CTRV)).

[0143] In step c5, the prediction results obtained by independent Kalman filter tracking are matched with the measured point cloud. For the successfully matched dynamic targets, their motion offset relative to the vehicle is calculated to correct the coordinates of the dynamic scene features.

[0144] Specifically, after matching the predicted results with the measured point cloud, the coordinates of the dynamic scene features can be corrected based on the associated confidence.

[0145] Considering that multi-scale atrous temporal fusion networks may suffer from insufficient compensation for dynamic target motion and global pose transformations ignoring independent target motion, these issues are likely to lead to large velocity estimation errors. Therefore, the disclosed embodiments use multi-target independent Kalman filtering to track and compensate for the coordinated motion of the ego vehicle and targets, performing dynamic target compensation for dynamic scene features. This approach has been shown to reduce velocity error by approximately 18%.

[0146] Figure 5 FIG2 shows a schematic diagram of a specific implementation process of cross-modal interaction fusion in step 205. Figure 5 , step 205 may include the following steps 501 to 504:

[0147] Step 501: fusing image features of a multi-view image sequence and static scene features to obtain image voxel features;

[0148] Step 502: Dynamically fuse dynamic scene features and image voxel features using a gated cross-attention module to obtain cross-modal interaction features;

[0149] Step 503: fusing dynamic scene features and image voxel features under geometric consistency constraints to obtain geometric alignment features;

[0150] In step 504 , the geometric alignment features and the cross-modal interaction features are integrated through multi-scale pyramid pooling to obtain multi-modal fusion features.

[0151] From the above, we can fuse heterogeneous features through the geometric-semantic dual-driven strategy to obtain multimodal fusion features.

[0152] Furthermore, an exemplary process of using the gated cross-attention module to obtain cross-modal interaction features in step 502 may include the following steps d1 to d2:

[0153] Step d1, calculating the gating matrix of the image voxel features and the dynamic scene features and dynamically adjusting the gating matrix according to the current weather conditions, where the gating matrix represents the fusion weight of the dynamic scene features;

[0154] In step d2, the image voxel features and dynamic scene features are fused based on the gating matrix to obtain cross-modal interaction features.

[0155] As a result, a gating mechanism can be used to adaptively fuse radar and camera features.

[0156] Furthermore, in step d1, the weight generation of the gated cross-attention is achieved through the following steps:

[0157] Step d11, feature stitching, that is, stitching the radar BEV features and the image voxel features along the channel dimension to obtain stitching features;

[0158] Feature splicing can preserve the complete information of both modalities, avoid early information loss, and allow the model to autonomously learn the interaction between different modalities.

[0159] In step d12, a 1×1 convolution is used to compress the number of channels of the concatenated features obtained in step b21 to the target dimension to obtain compressed features.

[0160] Among them, the target dimension can be a single channel or aligned with the original feature. Through 1×1 convolution compression, the dimension can be reduced, the amount of calculation can be reduced, the key interaction features can be focused, and the cross-modal weight distribution can be learned through the convolution kernel parameters.

[0161] Step d13, apply the Sigmoid function to map the eigenvalues ​​of the compressed features to the [0,1] interval to generate the basic gating matrix G base ;

[0162] Specifically, the basic gating matrix can be calculated by the following formula (1): G base :

[0163] (1)

[0164] Among them, σ represents the Sigmoid function, ConV([F R ;F I ]) represents the compression feature obtained in step b22, F R represents the radar BEV feature, F I Represents image voxel features.

[0165] Basic gating matrix G base∈[0, 1], where each gating value represents the feature importance weight. A gating value of 0 indicates suppression, and a gating value of 1 indicates retention.

[0166] Step d14: Dynamically adjust the gating matrix G according to weather conditions so that the gating value under complex weather conditions is greater than 0.5, favoring the 4D radar.

[0167] Specifically, under complex weather conditions such as thunderstorms and nights, the G value favors the radar (G ∈ [0.6, 0.8]), and under weather conditions such as clear and daytime, G favors the camera (G ∈ [0.3, 0.5]).

[0168] To achieve the dynamic adjustment of the output interval of the gating matrix G under different weather conditions for scene adaptability, it can be achieved by introducing a weather condition encoding and dynamic range mapping module.

[0169] First, obtain the weather label W indicating the current weather condition, and encode the weather label W into a weather vector w by means such as One-Hot encoding, learnable embedding (Embedding), etc.

[0170] Second, map the weather vector w to weather condition parameters L and H through a small neural network such as MLP, where 0 ≤ L < H ≤ 1. For example, the Sigmoid function can be used to ensure that L, H ∈ [0, 1], and L is guaranteed to be less than H by sorting.

[0171] Then, according to the weather condition parameters L and H, perform a linear transformation on G base to obtain the dynamically adjusted gating matrix.

[0172] (2)

[0173] Thus, under complex weather conditions such as thunderstorms and nights, the G value can be adjusted to the first preset interval, i.e., G ∈ [0.6, 0.8], favoring the radar; under weather conditions such as clear and daytime, the G value can be adjusted to the second preset interval G ∈ [0.3, 0.5], favoring the camera.

[0174] As above, by introducing weather condition encoding and dynamic range mapping, the gating cross-attention can adaptively adjust the weight interval, significantly improving the model robustness in complex environments.

[0175] Step d15: Use the dynamically adjusted gating matrix to dynamically fuse the dynamic scene features and image voxel features to obtain cross-modal interaction features.

[0176] Specifically, perform dynamic fusion through the following formula (3) to obtain cross-modal interaction features.

[0177] (3)

[0178] Among them, F fusion represents the cross-modal interaction feature, F R represents the radar BEV feature, F I Represents image voxel features.

[0179] As described above, gated cross-attention achieves dynamic fusion of dynamic scene features and image voxel features through feature concatenation, 1×1 convolution compression, sigmoid activation, and dynamic allocation of radar and camera fusion weights based on weather conditions. This adaptively enhances key information and suppresses noise through a learnable gating mechanism, addressing issues such as poor camera imaging quality in inclement weather and the accuracy degradation caused by fixed multimodal fusion weights. The method improves mean average precision (mAP) by 14.4% and reduces speed error by 15% in rainy scenes. This lightweight approach enables cross-modal feature optimization, balancing computational efficiency and model performance, effectively reducing missed object detection rates.

[0180] Furthermore, an exemplary implementation process of step 503 of fusing dynamic scene features and image voxel features under geometric consistency constraints may include the following steps e1 to e4:

[0181] Step e1, key point sampling of dynamic scene features and image voxel features;

[0182] Specifically, a predetermined number (eg, 1000) of key points are randomly sampled from the dynamic scene features and image voxel features in areas such as the target center and intersection signs.

[0183] Step e2: Perform BEV to voxel projection calculation on the key points in the dynamic scene features to obtain the voxel space coordinates of the key points, and use the voxel space projection coordinates of the key points and the coordinates of the corresponding key points in the image voxel features to calculate the key point alignment loss L align and feature IoU loss L IoU , for key point alignment loss L align and feature IoU loss L IoU Add the same weights to get the joint optimization target L total , by minimizing the joint optimization objective L total To adjust the geometric parameters p of the projection calculation from BEV to voxel, so as to obtain the optimized geometric parameters p;

[0184] The projection calculation of BEV to voxel can be expressed as the following formula (4).

[0185] (4)

[0186] Formula (4) represents the projection process from the bird's-eye view space to the voxel space, p BEV =(x bev ,y bev ) represents a point in the bird's-eye view (BEV), which usually corresponds to the two-dimensional plane coordinates (X, Y) of the real world; p voxel =(v x ,v y ,v z ) is the three-dimensional coordinate in voxel space, indicating the position of the point in the three-dimensional voxel grid.

[0187] Keypoint alignment loss L align It is the position alignment loss, which is used to measure the deviation between the projection position of the key point and the actual coordinate, and can be calculated by Euclidean distance.

[0188] Feature IoU loss L IoU It is the feature similarity loss, which is used to measure the difference between BEV features and voxel features in the vector space.

[0189] Joint optimization objective L total It can be calculated by the following formula (5).

[0190] (5)

[0191] By assigning λ=0.5 to the feature loss and position loss, the position and feature similarity can be balanced, and the geometric consistency loss function can evenly optimize the feature alignment of cross-modal data and the position alignment of physical space.

[0192] In step e3, the optimized geometric parameters p are applied to project the dynamic scene features into the voxel space to obtain the aligned dynamic scene features.

[0193] Step e4, according to the feature IoU loss L IoU Dynamically weighted fusion of aligned dynamic scene features and image voxel features to obtain geometric semantic multimodal fusion features;

[0194] Here, dynamic weighted fusion can be expressed as the following formula (6).

[0195] (6)

[0196] Among them, F BEV represents the aligned dynamic scene features, F voxel represents the image voxel feature, F fused Represents geometric and semantic multimodal fusion features.

[0197] In specific applications, the joint optimization target Ltotal can be fed back into the subsequent training of the scene perception model as the geometric consistency loss (GeometricConsistency Loss) to update the parameters of the scene perception model and improve the geometric consistency of subsequent feature extraction.

[0198] The joint optimization of keypoint alignment loss and feature intersection over union (IoU) enables high-precision geometric consistency in complex scenarios, ensuring the consistency of multimodal data in geometric space. Feature alignment ensures that the feature representations of the same object in different modalities are as similar as possible, while position alignment ensures that the projected coordinates of keypoints accurately correspond in the target space. Combined with a dynamic weighting mechanism, this method can also effectively improve model robustness in complex scenarios.

[0199] Through fusion under geometric consistency constraints, we can solve problems such as spatial misalignment between BEV features and voxel features, and high missed detection rate of small targets, reduce the key point alignment error by about 31.3%, and improve the small target IoU by about 2.3%.

[0200] Furthermore, in step 404 , the multi-scale pyramid pooling may adopt an adaptive pooling strategy to sequentially perform multi-scale pooling operations, concatenation operations, and 1×1 convolution fusion on the geometric alignment features and the cross-modal interaction features, thereby obtaining multi-modal fusion features.

[0201] Adaptive pooling strategies can dynamically adjust the stride or padding of the pooling window based on the resolution of input features (i.e., geometrically aligned features or cross-modal interaction features) to ensure alignment of output feature sizes. Specifically, adaptive pooling strategies can include: using a first predetermined pooling size (e.g., 1×1) for high-resolution areas, such as densely populated target areas, to preserve details; and using a second predetermined pooling size (e.g., 4×4) for low-resolution areas, such as open areas, to reduce computational complexity, with the first predetermined pooling size being smaller than the second predetermined pooling size. High-resolution and low-resolution areas can be dynamically distinguished by presetting a resolution threshold. Consequently, fine-grained pooling is used in high-resolution areas, while coarse-grained pooling is used in low-resolution areas. This can improve the average precision (AP) of small dynamic targets, such as pedestrians, by 3.5%, while reducing redundant computation in open areas and lowering computational resource consumption.

[0202] Multi-scale pooling operations can be performed in parallel on geometric alignment features and cross-modal interaction features. Specifically, the multi-scale pooling operation of multi-scale pyramid pooling can be implemented through multi-level pooling, and the pooling size of this multi-level pooling can grow exponentially with the level. For example, the multi-level pooling of multi-scale pyramid pooling can be 3-level pooling, where the first level pooling size is 1×1, which is equivalent to global average pooling or global maximum pooling and can extract global statistical features; the second level pooling size is 2×2, which can capture medium-scale regional features; and the third level pooling size is 4×4, which can retain finer-grained local details.

[0203] The concatenate operation involves concatenating pooling results at different scales along the channel dimension to form a multi-scale, multi-modal fusion feature. This preserves independent information at each scale while avoiding feature overlap or loss.

[0204] A 1×1 convolution is used to reduce the dimension of the concatenated multi-scale multi-modal fusion features, compress redundant channels, and enhance the interaction of cross-scale features to obtain multi-modal fusion features.

[0205] Multi-scale pyramid pooling can solve problems such as insufficient multi-scale feature expression capabilities and low classification accuracy in complex scenes. The AP of small targets such as pedestrians can be improved by 3.5%, while memory usage can be reduced by 15%.

[0206] Further, see Figure 2 The method of the embodiment of the present disclosure may further include: step 206, obtaining a 3D target detection result, a semantic occupancy prediction result and / or a motion estimation result of the surrounding environment of the object based on the multimodal fusion feature.

[0207] In step 206 , a 3D object detection result of the vehicle's surrounding environment may be obtained by an end-to-end object detection Transformer (DETR) detection head based on multimodal fusion features.

[0208] The DETR detection head may include a multi-layer Transformer decoder, which uses a sparse query mechanism (for example, initializing 900 learnable query vectors, each query is associated with a 3D reference point (x, y, z), etc.) to process multimodal fusion features to obtain 3D target detection results. The 3D target detection results may include but are not limited to the category, spatial position and speed of each target in the environment around the object. The targets here may include dynamic targets and static targets. The spatial position can be expressed as but not limited to, for example, three-dimensional coordinates in the object coordinate system (for example, the vehicle body coordinate system) and heading angles, pitch angles, roll angles, etc. used to describe the direction or posture of the object.

[0209] For example, the 3D object detection result may include information such as the category, coordinates, and current speed of multiple 3D bounding boxes, each of which represents an object and has a confidence level higher than a preset confidence threshold.

[0210] The DETR detection head can include a 6-layer Transformer decoder, and its loss function includes classification loss and regression loss.

[0211] In step 206 , a semantic occupancy prediction result may be obtained based on the multimodal fusion features through a multilayer perceptron (MLP) decoder. The semantic occupancy prediction result is represented as a semantic occupancy probability map of the surrounding environment of the object.

[0212] An MLP decoder can include multiple consecutive decoding units, each of which consists of multiple consecutive 3D convolutional layers, normalization layers, and activation function layers. The number of output data channels of these consecutive 3D convolutional layers decreases layer by layer. For example, an MLP decoder can be configured as: 3×[Conv3D(3×3×3) + Batch Normalization + ReLU], meaning that the decoding network includes three consecutive decoding units, each of which consists of three consecutive 3D convolutional layers with a convolution kernel size of 3×3×3, a batch normalization layer (BN), and a ReLU activation function layer. The number of output data channels of these three 3D convolutional layers is 128, 64, and 11, respectively.

[0213] Semantic Occupancy Prediction (SEP) predicts the semantic category and occupancy state of each voxel in three-dimensional space. The occupancy state indicates whether the voxel is occupied by an object. Compared to 3D bounding boxes, SEP provides a more fine-grained understanding of the environment, capable of identifying objects of arbitrary shapes such as vegetation and building debris, as well as continuously changing dynamic scenes.

[0214] In some examples disclosed herein, semantic occupancy prediction results can be represented as a semantic occupancy probability map, which supports multi-class semantic prediction. The semantic occupancy probability map represents the semantic category probability distribution for each voxel. The semantic occupancy probability map generated by the MLP decoder has been verified to support 11 semantic category predictions on a specific dataset, achieving an overall mean Intersection Over Union (MIoU) of 23.2%.

[0215] In step 206, a speed regression model may be used to obtain a motion estimation result based on the 3D target detection result and the multimodal fusion feature. The motion estimation result may include but is not limited to the speed of dynamic targets in the environment around the object within a predetermined time period in the future.

[0216] Specifically, the 3D target detection results and multimodal fusion features can be input into the velocity regression model and processed. The velocity regression model outputs a two-dimensional matrix, which contains the two-dimensional components of each dynamic target (v x , v y ), each two-dimensional component represents the velocity component of a dynamic target in the horizontal direction.

[0217] Exemplarily, the speed regression module can be implemented as but not limited to a two-layer fully connected neural network, each fully connected neural network includes an input layer, a hidden layer and an output layer, and the structure is 256→64→2. The input layer is used to receive 3D target detection prediction results and multimodal fusion features, and its input data channel number can be 256. The hidden layer can use the ReLU activation function or other nonlinear activation functions such as Sigmoid, and its output data channel number is 64. The output layer is used to output the speed component, and its output data channel number is 2.

[0218] In some examples, a velocity regression model can be trained based on a 4D radar data sequence collected by a 4D imaging radar array. Specifically, the velocity regression model can be trained by filtering the noise data in the 4D radar data sequence using a preset signal-to-noise ratio threshold (e.g., SNR ≥ 10dB) and using the velocity in the filtered 4D radar data sequence as a supervisory signal to train the velocity regression model. The supervisory signal for the velocity regression model is derived from the Doppler velocity measurements of the 4D radar, effectively suppressing the effects of noise such as rain and fog reflections and sidelobe interference on velocity estimation, further improving the accuracy and reliability of velocity estimation in complex weather conditions.

[0219] In some examples, step 206 may include obtaining 3D object detection results, semantic occupancy prediction results, and motion estimation results based on multimodal fusion features through a multi-task processing layer including a DETR detection head, an MLP decoder, and a velocity regression model. Thus, collaborative optimization of 3D object detection, 3D semantic understanding, and motion estimation can be achieved through shared encoder features, independent task branch design, and dynamic loss weighting.

[0220] Figure 6 The structure diagram of the multimodal scene perception device provided by the embodiment of the present disclosure is shown. Figure 6 , the multimodal scene perception device 600 of the embodiment of the present disclosure may include:

[0221] A data acquisition unit 601 is configured to acquire a current multi-view image sequence and a current 4D radar data sequence from a multi-view camera array and a 4D imaging radar group mounted on an object;

[0222] An image feature extraction unit 602 is configured to obtain current image features using the current multi-view image sequence;

[0223] A radar feature extraction unit 603 is configured to obtain current radar features using the current 4D radar data sequence. The radar features include radar voxel features and radar BEV features.

[0224] A temporal fusion unit 604 is configured to model the spatiotemporal evolution of the dynamic scene and the static scene in the BEV space and the voxel space based on the previously cached historical radar features and the current radar features to obtain the current dynamic scene features and the current static scene features;

[0225] The cross-modal interactive fusion unit 605 is used to perform cross-modal interactive fusion on the current image features, the current dynamic scene features and the current static scene features to obtain multimodal fusion features, and the multimodal fusion features are used to perform 3D target detection, semantic occupancy prediction and / or motion state estimation on the surrounding environment of the object.

[0226] Furthermore, the image feature extraction unit 602 can be specifically used to: use an image encoder to obtain a multi-view feature map sequence based on the current multi-view image sequence, the multi-view feature map sequence includes a feature map of each view image in the current multi-view image sequence, the image encoder includes a lightweight convolutional neural network that removes the last two convolutional layers and inserts an SE module in the penultimate layer; and use FPN to obtain serialized pyramid features based on the multi-view feature map sequence, the serialized pyramid features are the current image features, and the upsampling of FPN adopts the CARAFE operator.

[0227] Furthermore, the radar feature extraction unit 603 can be specifically used to: perform dynamic voxel compression on the current 4D radar data sequence to obtain the current radar voxel features; and obtain the current radar BEV features based on the current radar voxel features through a radar encoder. The radar encoder includes multiple consecutive feature extraction modules, each feature extraction module includes a sparse convolution layer, and the convolution kernels of the sparse convolution layers in the multiple consecutive feature extraction modules are the same size and the number of channels increases step by step.

[0228] Furthermore, the time series fusion unit 604 can be specifically used to:

[0229] Read the previously cached historical radar features from the buffer, where the historical radar features include historical radar voxel features and historical radar BEV features;

[0230] A multi-scale atrous temporal fusion network is used to obtain dynamic and static scene features based on historical and current radar features. The multi-scale atrous temporal fusion network includes an atrous temporal convolutional network, which includes multiple consecutive atrous convolutional layers. The atrous ratios of the multiple consecutive atrous convolutional layers increase exponentially at each layer, and the convolution kernels have the same size.

[0231] Dynamic target compensation is performed on dynamic scene features using multi-target Kalman filter tracking.

[0232] Furthermore, the cross-modal interaction fusion unit 605 can be specifically used to:

[0233] Fusing the current image features and the current static scene features to obtain image voxel features;

[0234] Use gated cross-attention modules to dynamically fuse dynamic scene features and image voxel features to obtain cross-modal interaction features;

[0235] Under the constraint of geometric consistency, dynamic scene features and image voxel features are fused to obtain geometric alignment features;

[0236] Multi-scale pyramid pooling is used to integrate geometric alignment features and cross-modal interaction features to obtain multimodal fusion features.

[0237] Furthermore, the multimodal scene perception device 600 may also include: a multi-tasking unit 606, which is used to obtain 3D target detection results, semantic occupancy prediction results and / or motion estimation results of the environment around the object based on multimodal fusion features.

[0238] In specific applications, the multimodal scene perception device 600 can be implemented by software, hardware, or a combination of both. For example, the multimodal scene perception device 600 can be implemented as software running in the electronic device 700 described below.

[0239] In addition, an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. The program includes instructions, and when the instructions are executed by one or more processors of the computer, the steps of the aforementioned multimodal-based scene perception method are executed.

[0240] Figure 7 Schematic diagram of the structure of the electronic device provided by the embodiment of the present disclosure is shown. Figure 7 The electronic device 700 may include: one or more processors 701, and a memory 702 storing one or more programs, which are executed by the one or more processors 701 to implement the method flow shown in the above embodiments of the present disclosure and / or the program units corresponding to each unit in the device.

[0241] The various components are interconnected using various buses and may be mounted on a common motherboard or in other ways as needed. Processor 701 may process instructions for execution within the electronic device, including instructions stored in or on memory for displaying graphical information of a user interface on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses may be used with multiple memories and multiple storage devices, if desired.

[0242] The processor 701 may include one or more single-core processors or multi-core processors. The processor 701 may include any combination of general-purpose processors or dedicated processors (such as an image processor, an application processor, a baseband processor, etc.).

[0243] The memory 702 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the embodiment of the present disclosure. Figure 2 The processor 701 executes the non-transient software programs, instructions and units stored in the memory 702, thereby executing the above-mentioned method embodiment. Figure 2 The programs, instructions and units corresponding to the multimodal scene perception method are shown.

[0244] The electronic device 700 may further include an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0245] The programs (also referred to as software, software applications, or code) described above include machine instructions for a programmable processor and may be implemented using an object-oriented programming language, assembly, or machine language.

[0246] Over time and with the advancement of technology, the meaning of "medium" has become increasingly broad. The dissemination of computer programs is no longer limited to tangible media and can also be directly downloaded from the Internet. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component.

[0247] In specific applications, the electronic device 700 can be implemented as, but not limited to, a domain controller, an in-vehicle device, or other similar devices.

[0248] An embodiment of the present disclosure further provides a vehicle equipped with a multi-view camera array and a 4D imaging radar group, and the vehicle includes the aforementioned multi-modal scene perception device 600, an electronic device 700 and / or a computer-readable storage medium.

[0249] The technical solutions provided by the present disclosure are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The descriptions of the above embodiments are intended only to help understand the methods and core concepts of the present disclosure. Furthermore, those skilled in the art will appreciate that variations in the specific implementation methods and scope of application may occur based on the concepts of the present disclosure. In summary, the contents of this specification should not be construed as limiting the present disclosure.

[0250] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A multimodal scene perception method, characterized in that: The method comprises: Acquiring a current multi-view image sequence and a current 4D radar data sequence from a multi-view camera array and a 4D imaging radar system mounted on the vehicle. The 4D radar data in the 4D radar data sequence includes the spatial position, velocity, reflection intensity, and signal-to-noise ratio of the target. The velocity refers to the radial velocity of the target relative to the radar, which can be used to distinguish between stationary and moving objects. The target refers to a physical entity detected by the 4D imaging radar system. Obtaining current image features using the current multi-view image sequence; Acquire current radar features using the current 4D radar data sequence, where the radar features include radar voxel features and radar BEV features; Based on previously cached historical radar signatures and current radar signatures, the spatiotemporal evolution of dynamic and static scenes is modeled in BEV space and voxel space to obtain current dynamic scene signatures and current static scene signatures. The dynamic scene signatures are used to characterize the spatiotemporal evolution of dynamic objects in the vehicle's surrounding environment, and the static scene signatures are used to characterize the spatiotemporal evolution of static objects in the vehicle's surrounding environment. Performing cross-modal interactive fusion on the current image feature, the current dynamic scene feature, and the current static scene feature to obtain a multimodal fusion feature, wherein the multimodal fusion feature is used to perform 3D object detection, semantic occupancy prediction, and / or motion state estimation on the vehicle's surrounding environment; The method includes modeling the spatiotemporal evolution of dynamic scenes and static scenes in BEV space and voxel space based on previously cached historical radar features and current radar features to obtain current dynamic scene features and current static scene features, including: reading previously cached historical radar features from a buffer, wherein the historical radar features include historical radar voxel features and historical radar BEV features; obtaining dynamic scene features and static scene features based on the historical radar features and the current radar features using a multi-scale void temporal fusion network, wherein the multi-scale void temporal fusion network includes a void temporal convolution network, and the void temporal convolution network includes multiple consecutive void convolution layers, wherein the void rates of the multiple consecutive void convolution layers increase exponentially according to the layers and the convolution kernel sizes are the same; and performing dynamic target compensation on the dynamic scene features using a multi-target Kalman filter tracking method. Among them, the cross-modal interactive fusion of the current image features, the current dynamic scene features and the current static scene features to obtain multimodal fusion features includes: fusing the current image features and the current static scene features to obtain image voxel features; using a gated cross-attention module to dynamically fuse the dynamic scene features and the image voxel features to obtain cross-modal interaction features; fusing the dynamic scene features and the image voxel features under geometric consistency constraints to obtain geometric alignment features; and integrating the geometric alignment features and the cross-modal interaction features through multi-scale pyramid pooling to obtain the multimodal fusion features.

2. The method according to claim 1, characterized in that The obtaining of current image features by using the current multi-view image sequence includes: Using an image encoder to obtain a multi-view feature map sequence based on the current multi-view image sequence, the multi-view feature map sequence includes a feature map of each view image in the current multi-view image sequence, the image encoder includes a lightweight convolutional neural network with the last two convolutional layers removed and a channel attention mechanism SE module inserted in the penultimate layer; A feature pyramid network (FPN) is used to obtain serialized pyramid features based on the multi-view feature map sequence, where the serialized pyramid features are the current image features. The upsampling of the FPN uses a content-aware feature reconstruction CARAFE operator.

3. The method according to claim 1, characterized in that The obtaining of current radar features by using the current 4D radar data sequence includes: Perform dynamic voxel compression on the current 4D radar data sequence to obtain the current radar voxel features; The current radar BEV feature is obtained based on the current radar voxel feature through a radar encoder, wherein the radar encoder includes a plurality of consecutive feature extraction modules, each of which includes a sparse convolution layer, and the convolution kernels of the sparse convolution layers in the plurality of consecutive feature extraction modules have the same size and the number of channels increases step by step.

4. The method according to claim 1, wherein The dynamically fusing the dynamic scene features and the image voxel features through a gated cross attention module to obtain a cross-modal interaction feature comprises: Calculating a gating matrix of the image voxel feature and the dynamic scene feature, wherein the gating matrix represents a fusion weight of the dynamic scene feature; dynamically adjusting the gating matrix according to current weather conditions; The image voxel features and the dynamic scene features are fused based on the dynamically adjusted gating matrix to obtain the cross-modal interaction features.

5. The method according to claim 1, wherein The method further includes: obtaining a 3D target detection result, a semantic occupancy prediction result and / or a motion estimation result of the vehicle's surrounding environment based on the multimodal fusion features.

6. A multimodal scene perception device, characterized in that: include: a data acquisition unit configured to acquire a current multi-view image sequence and a current 4D radar data sequence from the multi-view camera array and 4D imaging radar array mounted on the vehicle, wherein the 4D radar data in the 4D radar data sequence includes the spatial position, velocity, reflection intensity, and signal-to-noise ratio of the target. The velocity refers to the radial velocity of the target relative to the radar, which can be used to distinguish between stationary and moving objects. The target refers to a physical entity detected by the 4D imaging radar array; An image feature extraction unit, configured to obtain current image features using the current multi-view image sequence; a radar feature extraction unit, configured to obtain current radar features using the current 4D radar data sequence, wherein the radar features include radar voxel features and radar BEV features; A temporal fusion unit is used to model the spatiotemporal evolution of dynamic scenes and static scenes in BEV space and voxel space based on previously cached historical radar features and current radar features to obtain current dynamic scene features and current static scene features, wherein the dynamic scene features are used to characterize the spatiotemporal evolution of dynamic objects in the vehicle's surrounding environment, and the static scene features are used to characterize the spatiotemporal evolution of static objects in the vehicle's surrounding environment; wherein, based on the previously cached historical radar features and current radar features, the spatiotemporal evolution of dynamic scenes and static scenes is modeled in BEV space and voxel space to obtain current dynamic scene features and current static scene features. Features, including: reading previously cached historical radar features from a buffer, the historical radar features including historical radar voxel features and historical radar BEV features; using a multi-scale void temporal fusion network to obtain dynamic scene features and static scene features based on the historical radar features and the current radar features, the multi-scale void temporal fusion network including a void temporal convolution network, the void temporal convolution network including a plurality of consecutive void convolution layers, the void rates of the plurality of consecutive void convolution layers increasing exponentially by level and the convolution kernels having the same size; performing dynamic target compensation on the dynamic scene features using a multi-target Kalman filter tracking method; A cross-modal interactive fusion unit is used to perform cross-modal interactive fusion on the current image features, current dynamic scene features and current static scene features to obtain multimodal fusion features, and the multimodal fusion features are used to perform 3D target detection, semantic occupancy prediction and / or motion state estimation on the vehicle's surrounding environment; wherein, the cross-modal interactive fusion on the current image features, current dynamic scene features and current static scene features to obtain multimodal fusion features includes: fusing the current image features and the current static scene features to obtain image voxel features; using a gated cross-attention module to dynamically fuse the dynamic scene features and the image voxel features to obtain cross-modal interactive features; fusing the dynamic scene features and the image voxel features under geometric consistency constraints to obtain geometric alignment features; integrating the geometric alignment features and the cross-modal interactive features through multi-scale pyramid pooling to obtain the multimodal fusion features.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 5.

Images