Environment-adaptive multimodal data fusion method, device, and vehicle

Abstract

The present disclosure provides an environment-adaptive multimodal data fusion method, device and vehicle. The environment-adaptive multimodal data fusion method of the embodiment of the present disclosure first adjusts the resolution of the perspective image according to the lighting conditions of each perspective image in the multi-perspective image, and then extracts the image features of the multi-perspective image, first downsamples the point cloud according to the vehicle speed and performs sparse area interpolation and completion on the point cloud, further downsamples the dense area, and then extracts the point cloud features, and finally fuses the image features and point cloud features to obtain multimodal fusion features, and then performs tasks such as 3D target detection and vehicle trajectory prediction based on the multimodal fusion features. The present disclosure can reduce the impact of fusion deviation while reducing the demand for computing resources, and effectively improve the accuracy and reliability of 3D target detection and / or vehicle trajectory prediction in extreme scenarios.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to an environment-adaptive multimodal data fusion method, device, and vehicle. Background Art

[0002] Currently, multimodal sensor data fusion technology has been widely used in fields such as autonomous driving and robotic environmental perception. However, existing fusion algorithms still face significant technical bottlenecks in terms of robustness and computational efficiency in extreme scenarios. For example, detection performance for sparse and high-reflection problems can plummet, requiring high computing resources, and facing issues such as information loss in sparse areas and computational redundancy in dense areas. Summary of the Invention

[0003] In view of this, the present disclosure provides an environment-adaptive multimodal data fusion method, device, and vehicle.

[0004] According to a first aspect of the present disclosure, an environment-adaptive multimodal data fusion method is provided. The method is applied to a vehicle equipped with a multi-view camera array and a laser radar. The method comprises:

[0005] Acquire multi-view images captured by the multi-view camera array and point clouds captured by the laser radar;

[0006] Preprocessing the multi-view images, the preprocessing comprising: adjusting the resolution of each view image in the multi-view images according to the illumination conditions of the view image;

[0007] Obtain image features using preprocessed multi-view images;

[0008] Preprocessing the point cloud, the preprocessing comprising: downsampling the point cloud according to the vehicle speed, and performing sparse area interpolation and dense area downsampling on the downsampled point cloud;

[0009] Use the preprocessed point cloud to obtain point cloud features;

[0010] fusing the image features and the point cloud features to obtain a multimodal fusion feature;

[0011] The multimodal fusion features are used to perform 3D object detection and / or vehicle trajectory prediction.

[0012] In some embodiments of the first aspect of the present disclosure, the resolution of the perspective image is adjusted in the following manner: determining the illumination level of the perspective image through a lightweight convolutional neural network classifier; when the illumination level of the perspective image is lower than a predetermined level, increasing the resolution of the perspective image to an image resolution corresponding to the predetermined level; when the illumination level of the perspective image is higher than the predetermined level, reducing the resolution of the perspective image to an image resolution corresponding to the predetermined level.

[0013] In some embodiments of the first aspect of the present disclosure, the obtaining of image features using preprocessed multi-view images includes: performing the following processing on each view image in the preprocessed multi-view images to extract features of each view image: dynamically adjusting the RGB channel weights of the basic convolution kernel according to the illumination level of the view image through an image feature extraction module including conditional convolution to generate a dynamic convolution kernel adapted to the illumination conditions of the view image and extracting features of the view image using the dynamic convolution kernel; aligning and fusing the features of each view image to obtain multi-scale features.

[0014] In some embodiments of the first aspect of the present disclosure, the method of obtaining image features using the adjusted multi-view images further includes: using a lightweight semantic segmentation network to generate a semantic segmentation mask based on the view image, and applying the semantic segmentation mask as an attention weight to the process of extracting features of the view image using the dynamic convolution kernel.

[0015] In some embodiments of the first aspect of the present disclosure, the use of the preprocessed point cloud to obtain point cloud features includes: adjusting the voxel grid size according to the local point cloud density to adaptively voxelize the preprocessed point cloud to obtain voxel data; and using separable 3D convolution to process the voxel data to obtain the point cloud features.

[0016] In some embodiments of the first aspect of the present disclosure, the method further includes: evaluating whether the cross-modal consistency of the image features and the point cloud features meets the requirements; when the cross-modal consistency of the image features and the point cloud features does not meet the requirements, using a generative adversarial network GAN to generate a pseudo multi-view image based on the point cloud, obtaining features of the pseudo multi-view image, and merging the features of the pseudo multi-view image into the image features.

[0017] In some implementations of the first aspect of the present disclosure, the evaluating whether the cross-modal consistency of the image features and the point cloud features meets the requirements includes one or more of the following:

[0018] Calculating the similarity between the point cloud feature and the image feature at the same spatial position through BEV spatial contrast learning, and determining that the cross-modal consistency of the point cloud feature and the image feature does not meet the requirement when the similarity is less than a first predetermined similarity threshold;

[0019] Calculate the cross-modal feature similarity between the forward view image and the point cloud in the multi-view image, and when the cross-modal feature similarity is less than a second predetermined similarity threshold, determine that the cross-modal consistency of the point cloud and the forward view image does not meet the requirements.

[0020] In some embodiments of the first aspect of the present disclosure, the method further includes: dynamically adjusting the modal weight of the image feature according to the exposure abnormality of each perspective image in the multi-perspective image; and / or dynamically adjusting the modal weight of the point cloud feature according to the local point cloud density of the point cloud; the fusing of the image feature and the point cloud feature to obtain a multimodal fusion feature includes: fusing the image feature and the point cloud feature based on the modal weight of the image feature and / or the modal weight of the point cloud feature to obtain the multimodal fusion feature.

[0021] According to a second aspect of the present disclosure, an environment-adaptive multimodal data fusion device is provided, wherein the device is applied to a vehicle equipped with a multi-view camera array and a laser radar;

[0022] The environment-adaptive multimodal data fusion device comprises:

[0023] A data acquisition unit, configured to acquire the multi-view images acquired by the multi-view camera array and the point cloud acquired by the laser radar;

[0024] An image preprocessing unit, configured to preprocess the multi-view images, wherein the preprocessing comprises: adjusting the resolution of each view image in the multi-view images according to the illumination conditions of the view image;

[0025] An image feature extraction unit, configured to obtain image features using the preprocessed multi-view images;

[0026] a point cloud preprocessing unit, configured to preprocess the point cloud, wherein the preprocessing comprises: downsampling the point cloud according to the vehicle speed, and performing sparse region interpolation and dense region downsampling on the downsampled point cloud;

[0027] A point cloud feature extraction unit, used to obtain point cloud features using the preprocessed point cloud;

[0028] a fusion unit, configured to fuse the image features and the point cloud features to obtain a multimodal fusion feature;

[0029] A task execution unit is used to perform 3D target detection and / or vehicle trajectory prediction using the multimodal fusion features.

[0030] According to a third aspect of the present disclosure, a vehicle is provided, wherein the vehicle is equipped with a multi-view camera sequence and a laser radar, and the vehicle includes the aforementioned environment-adaptive multimodal data fusion device.

[0031] The disclosed embodiments can achieve environment-adaptive multimodal data fusion while avoiding problems such as information loss in sparse areas and computational redundancy in dense areas. This reduces the impact of fusion bias while lowering computing resource requirements, improves the reliability and robustness of modal data fusion, and effectively improves the accuracy and reliability of 3D target detection and / or vehicle trajectory prediction in extreme scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] Figure 1 A schematic diagram of the system architecture to which the embodiments of the present disclosure are applicable;

[0034] Figure 2 A schematic diagram of a flow chart of an environment-adaptive multimodal data fusion method provided by an embodiment of the present disclosure;

[0035] Figure 3 Schematic diagram of a process for adjusting the resolution of a perspective image according to the lighting conditions of the perspective image according to an embodiment of the present disclosure;

[0036] Figure 4 A schematic diagram of a process for obtaining image features using pre-processed multi-view images according to an embodiment of the present disclosure;

[0037] Figure 5 Another schematic diagram of the process of the environment-adaptive multimodal data fusion method provided by the embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram of the structure of an environment-adaptive multimodal data fusion device provided in an embodiment of the present disclosure;

[0039] Figure 7 A schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0041] The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0042] As used herein, the words "if," "if," and the like may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0043] First, a detailed description of the relevant technologies is given.

[0044] Currently, there are several main solutions for multimodal data fusion in extreme scenarios:

[0045] 1) Fixed-weight multimodal fusion: This approach directly adds or concatenates radar and camera features using fixed weights, relying on a neural network model to automatically learn cross-modal associations. This approach cannot dynamically adjust modal weights based on lighting and point cloud density, leading to a sharp drop in detection performance in extreme scenarios (e.g., camera failure in low light). Furthermore, projection errors accumulate. The rigid projection of the radar's BEV space onto the camera's front view results in positioning offsets for small objects such as pedestrians and cones (average error >0.5m).

[0046] 2) Multimodal data fusion based on static voxelized point cloud processing: The radar-collected point cloud is evenly divided into a fixed-size voxel grid (e.g., 0.1 m³) and features are extracted using a 3D convolutional neural network. This solution suffers from information loss in sparse areas. Key points are filtered out in distant or low-density areas due to the overly large voxel grid, increasing the missed detection rate by 12% or more. It also suffers from redundant computation in dense areas. The same features are repeatedly calculated in close, high-density areas, reducing computing resource utilization by approximately 40% or more.

[0047] As can be seen from the above, the robustness and computational efficiency of multimodal data fusion in extreme scenarios still face significant technical bottlenecks. In view of this, the embodiments of the present disclosure provide the following environment-adaptive multimodal data fusion method, device, and vehicle.

[0048] To facilitate understanding, the system structure applicable to the embodiments of the present disclosure is briefly described below.

[0049] Figure 1 Schematic diagram of the structure of the system to which the embodiment of the present disclosure is applicable. Figure 1 The system to which the embodiments of the present disclosure are applicable may include: an electronic device and a peripheral sensor component connected to the electronic device, the peripheral sensor component including but not limited to a multi-view camera array and a lidar.

[0050] A multi-view camera array can be mounted on a vehicle and used to capture multi-view images covering the vehicle's surroundings.

[0051] The multi-view camera array can be implemented as, but is not limited to, a six-view camera group, including a front-view camera, a rear-view camera, a left-view camera, a right-view camera, an upward-view camera, and a downward-view camera. In specific applications, the cameras can be evenly distributed around the vehicle to provide comprehensive coverage. Each camera captures a portion of the surrounding scene in an image of the vehicle's surroundings.

[0052] LiDAR is mounted on a vehicle and can be used to collect point clouds in real time. In specific applications, the LiDAR's installation position can be flexibly adjusted to suit the specific application, ensuring that the point cloud collected by the LiDAR covers the vehicle's forward, rearward, or surrounding environment.

[0053] Each point in a point cloud collected by a laser radar can contain 3D coordinates, reflection intensity, and a timestamp. The 3D coordinates can be determined by calculating the distance using the laser's time-of-flight (TOF) and combining it with the radar's rotation angles (azimuth and elevation). The 3D coordinates of each point in the point cloud are defined in the LiDAR coordinate system, which can be converted to the vehicle coordinate system, the world coordinate system, and so on. Reflection intensity (also known as reflectivity) depends on the surface material of an object and can be used to distinguish material types (for example, identifying road markings). The timestamp records the time difference between laser emission and reception, which can be used for dynamic scene analysis, such as tracking the trajectory of moving objects.

[0054] Furthermore, the peripheral sensor components may also include an inertial measurement unit (IMU). The IMU is installed on the vehicle and connected to the electronic device, and can be used to measure and provide IMU data to the electronic device. IMU data may include, but is not limited to, the vehicle's three-axis acceleration and three-axis angular velocity. IMU data can be used to determine the vehicle's instantaneous motion state, assist in positioning, and estimate attitude. For example, IMU data can be used to determine the vehicle's roll, pitch, and yaw angles, among other attitude information.

[0055] Furthermore, the above system may also include an onboard communication module (e.g., 4G / 5G, Wi-Fi, or Internet of Vehicles) or an onboard environmental sensor. The onboard communication module can be used to connect to the cloud to obtain weather data. For example, it can receive real-time route-specific weather data from apps like AutoNavi and Google Maps. Another example is the ability to receive weather data from devices around the vehicle (e.g., other vehicles, roadside units, etc.). Onboard environmental sensors can be used to monitor the vehicle's surrounding weather conditions in real time and generate corresponding weather data. These sensors may include, but are not limited to, ambient light sensors, humidity sensors, and temperature sensors.

[0056] The system to which the embodiments of the present disclosure are applicable may be, but is not limited to, any system requiring multimodal data fusion. For example, the system may be, but is not limited to, an autonomous driving system, an intelligent assisted driving system, a traffic congestion assistance system, etc. Figure 1 The system provided by the embodiments of the present disclosure can be installed in a vehicle and used as, but not limited to, an intelligent assisted driving system, a traffic congestion assistance system, an automatic driving system, etc. of the vehicle.

[0057] It should be noted that the “vehicle” described in the embodiments of the present disclosure may be implemented as, but not limited to, multiple wheeled mobile robots, wheeled mobile robots, mobile robots, general-purpose vehicles, aircraft, ships, Autonomous Rail Rapid Transit (ART) systems, industrial automation equipment, etc. General-purpose vehicles may include, but are not limited to, passenger cars, commercial vehicles (e.g., trucks, buses, vans, etc.), special-purpose vehicles (e.g., ambulances, fire trucks, engineering vehicles, rescue vehicles, etc.), agricultural and industrial vehicles (e.g., harvesters, forklifts, etc.), transportation and logistics vehicles (e.g., container trucks, refrigerated trucks, etc.), new energy vehicles (e.g., electric vehicles, hybrid electric vehicles), and special vehicles (e.g., garbage trucks, water trucks, etc.).

[0058] The embodiments of the present disclosure can be applied to various scenarios such as urban transportation, highways, ports, mines, farms, closed parks, and industrial production. They can be applied to many aspects such as travel by car, public transportation, logistics distribution, unmanned transportation, terminal delivery, automated agricultural operations, and automated sanitation. Of course, the embodiments of the present disclosure can also be applied to any other intelligent control scenarios involving, for example, multimodal data fusion. The present disclosure does not limit the application scenarios and applicable fields of the embodiments of the present disclosure.

[0059] The specific implementation of the embodiment of the present disclosure is described in detail below.

[0060] Figure 2 The flowchart of the environment-adaptive multimodal data fusion method provided by the embodiment of the present disclosure is shown. The environment-adaptive multimodal data fusion method can be executed by the electronic device below, which can be implemented as but not limited to a domain controller or other similar devices. Figure 2 The environment-adaptive multimodal data fusion method of the embodiment of the present disclosure includes the following steps:

[0061] Step 201: Acquire multi-view images captured by a multi-view camera array and point clouds captured by a laser radar;

[0062] Step 202: pre-processing the multi-view images. The pre-processing includes: adjusting the resolution of each view image in the multi-view images according to the lighting conditions of the view image;

[0063] Step 203, obtaining image features using the pre-processed multi-view images;

[0064] Step 204 , preprocessing the point cloud, the preprocessing including: downsampling the point cloud according to the vehicle speed, performing sparse area interpolation and dense area downsampling on the downsampled point cloud;

[0065] Step 205, obtaining point cloud features using the pre-processed point cloud;

[0066] Step 206: fusing the image features and the point cloud features to obtain a multimodal fusion feature;

[0067] Step 207: Utilize the multimodal fusion features to perform 3D object detection and / or vehicle trajectory prediction.

[0068] The environment-adaptive multimodal data fusion method of the embodiment of the present disclosure adjusts the resolution of each perspective image according to the lighting conditions and then extracts the image features, downsamples the point cloud according to the vehicle speed and performs sparse area interpolation and dense area downsampling on the downsampled point cloud before extracting the point cloud features. Therefore, the present disclosure can realize environment-adaptive multimodal data fusion while avoiding problems such as information loss in sparse areas and computational redundancy in dense areas, thereby reducing the impact of fusion bias while reducing computing resource requirements, improving the reliability and robustness of modal data fusion, and effectively improving the accuracy and reliability of 3D target detection and / or vehicle trajectory prediction in extreme scenarios.

[0069] In step 201, the multi-view images and point cloud are synchronized. Specifically, the synchronization of the multi-view images and point cloud can be achieved through mechanisms such as synchronous triggering and time alignment of the multi-view camera array and the lidar.

[0070] In step 202, for each perspective image in the multi-perspective image, the resolution of the perspective image is adjusted according to the lighting conditions of the perspective image. The resolution of the perspective image can be improved in a low-light environment to retain details, and the resolution of the perspective image can be reduced in a strong-light environment to reduce redundant calculations.

[0071] Figure 3 FIG2 shows a schematic diagram of a specific implementation process of adjusting the resolution of the perspective image according to the illumination conditions of the perspective image in step 202. Figure 3 The process may include the following steps 301 to 302:

[0072] Step 301: Determine the illumination level of each view image in the multi-view image using a lightweight convolutional neural network (CNN) classifier.

[0073] Multiple light levels and their image resolutions can be pre-configured, with a high light level indicating high light intensity and a low light level indicating low light intensity.

[0074] The light level may be, but is not limited to, categorized information such as low light, normal light, and strong light, or may be a continuous value such as a brightness value. For example, three light levels (high, medium, and low) may be configured, and light intensity ranges may be set for each light level. The high light level has the highest light intensity, the medium light level has a moderate light intensity, and the low light level has the lowest light intensity. The medium level corresponds to a normal lighting environment and may be used as the preset level in step 302.

[0075] A frame of image can be captured from the camera corresponding to each viewpoint image, and processed using a lightweight CNN classifier to obtain the illumination level of the corresponding viewpoint image. Specifically, for each viewpoint image, a lightweight CNN classifier can be used to extract global illumination features to obtain the illumination level. Lightweight CNN classifiers can be used, but are not limited to, MobileNetV3. Estimating illumination levels using a lightweight CNN classifier can more accurately capture features under complex lighting conditions, such as shadows and mixed light sources, and thus accurately assess the illumination level of each viewpoint image.

[0076] In specific applications, other methods may be used to determine the illumination level of images at each viewing angle as needed.

[0077] For example, the light intensity of the area covered by each perspective image can be directly obtained through an ambient light sensor (ALS), and the light level of each perspective image can be determined based on the light intensity and pre-configured light intensity ranges for each light level.

[0078] For another example, a frame of RGB format image can be captured from the camera corresponding to each perspective image, and the image can be converted into a grayscale image or YUV color space to extract the brightness channel, and the brightness mean or brightness median reflecting the overall light intensity can be calculated. The brightness mean or brightness median is the light intensity, and the light level of each perspective image is determined by the light intensity and the pre-configured light intensity ranges for each light level.

[0079] Step 302: Adjust the resolution of the perspective image according to the illumination level of the perspective image.

[0080] Specifically, when the light level of the perspective image is below a predetermined level, the perspective image is determined to have been captured in a low-light environment, and the resolution of the perspective image is increased to the image resolution corresponding to the predetermined level. When the light level of the perspective image is above a predetermined level, the perspective image is determined to have been captured in a high-light environment, and the resolution of the perspective image is reduced to the image resolution corresponding to the predetermined light level. This allows the resolution of multi-perspective images to be increased in low-light environments to preserve more details, while the resolution of multi-perspective images can be reduced in high-light environments to reduce data volume and redundant computation.

[0081] Dynamically adjusting the resolution of each view in a multi-view image according to the light intensity can better balance image quality, processing efficiency, and energy consumption, thereby dynamically adapting to the ever-changing external environment.

[0082] Figure 4 FIG2 shows a flow chart of obtaining image features using pre-processed multi-view images in step 203. Figure 4 , the process may include steps 401 and 402:

[0083] In step 401, the following processing is performed on each view image in the multi-view image to extract its features: an image feature extraction module including conditional convolution is used to dynamically adjust the RGB channel weights of the basic convolution kernel according to the illumination level of the view image to generate a dynamic convolution kernel that adapts to the illumination conditions of the view image and the features of the view image are extracted using the dynamic convolution kernel.

[0084] Specifically, the process of extracting features of a certain perspective image using an image feature extraction module including conditional convolution may include: generating an RGB channel weight scaling factor according to the illumination level of the current perspective image, adjusting the RGB channel weights of the basic convolution kernel using the RGB channel weight scaling factor to generate a dynamic convolution kernel, and using the dynamic convolution kernel to perform feature extraction on the current perspective image to obtain the features of the current perspective image.

[0085] As shown above, in low-light areas, the dynamic convolution kernel can enhance the ability to extract high-frequency details and restore dark information. In overexposed areas, the dynamic convolution kernel can suppress noise and balance brightness. This allows the feature extraction process of each view to be adaptively optimized based on the lighting conditions of the image, enhancing color invariance in low light and suppressing noise in overexposed areas.

[0086] The features of a single-view image can include one or more of the following: color, texture, shape, spatial relationships, and semantics. Spatial relationships include object position, which describes the absolute or relative position of segmented objects in the image. Semantic information includes object category information, which can be used to describe the object category and its confidence at the pixel level. Semantic information can also include scene semantic information, which can be used to describe the overall scene of the image. The features of each view image can be pixel-level features.

[0087] Step 402: align and fuse the features of the images from each perspective to obtain multi-scale features.

[0088] Here, multi-scale features are image features from multi-view images. They can include one or more of the following information: color, texture, shape, semantics, and spatial constraints. Semantic information includes object type information. Similarly, multi-scale features can be pixel-level features.

[0089] Specifically, multi-scale features can be obtained by geometric alignment and attention weighting. The embodiments of this disclosure do not limit the specific implementation method of obtaining multi-scale features.

[0090] Furthermore, in step 401, a lightweight semantic segmentation network can be used to generate a semantic segmentation mask based on the view image, and the semantic segmentation mask can be used as an attention weight in the process of extracting features from the view image using a dynamic convolution kernel. In this way, the lightweight semantic segmentation network can dynamically guide the image feature extraction model to focus on key areas such as roads and obstacles, while suppressing irrelevant background interference.

[0091] For example, the lightweight semantic segmentation network can be, but is not limited to, DeepLabv3+ Mobile. DeepLabv3+ Mobile combines a lightweight architecture with multi-scale feature enhancement technology to efficiently generate semantic segmentation masks. The semantic segmentation masks generated based on the viewport image can be probability maps, binary masks, or other applicable forms.

[0092] Through the above method, in multi-view scenes, the convolution kernel can be dynamically adjusted to adapt to lighting changes while accurately focusing on key areas, thereby achieving efficient feature extraction of multi-view images.

[0093] In step 204, the point cloud is adaptively downsampled according to the vehicle speed. The resolution of the point cloud can be dynamically adjusted according to the change of vehicle speed, thereby achieving a balance between high precision and efficient processing, so that the point cloud is sparse in high-speed scenes and maintains a high resolution in low-speed scenes.

[0094] The vehicle speed can be obtained in various applicable ways. Here, the vehicle speed can be the instantaneous speed of the vehicle or the average speed of the vehicle within the time period corresponding to the point cloud.

[0095] In some examples, vehicle speed can be provided in real time by an IMU mounted on the vehicle.

[0096] In some examples, the vehicle speed can be determined based on an extended Kalman filter (EKF) using IMU data provided by an IMU installed on the vehicle. This can compensate for the vehicle's own motion and eliminate the impact of offsets caused by vehicle bumps, vehicle steering, and other conditions on the accuracy of the vehicle speed, thereby improving the stability and reliability of vehicle speed detection.

[0097] In other examples, the vehicle speed can be read from the vehicle CAN bus or estimated by the change in the pose of two consecutive frames of point cloud. The specific detection method of the vehicle speed is not limited in the embodiment of the present disclosure.

[0098] In step 204, the downsampling ratio of the point cloud can be dynamically adjusted based on piecewise linear interpolation of vehicle speed to achieve adaptive downsampling of the point cloud. Specifically, multiple speed intervals can be preconfigured, and a predetermined linear interpolation formula is used within the speed interval to calculate the downsampling retention ratio. The point cloud is then downsampled according to the downsampling retention ratio. Through piecewise linear interpolation, the resolution of the point cloud can be dynamically adjusted based on vehicle speed, achieving a balance between data processing efficiency and perception accuracy.

[0099] The starting speed and ending speed of each speed interval are preset values. For example, the speed intervals can be configured as follows: low speed interval (0-30 km / h), medium speed interval (30-60 km / h), high speed interval (60-90 km / h) and ultra-high speed interval (>90 km / h). Among them, the low speed interval corresponds to high resolution, retaining 100% of the point cloud; the medium speed interval corresponds to medium resolution, retaining 70%-100% of the point cloud; the high speed interval corresponds to low resolution, retaining 50%-70% of the point cloud; the ultra-high speed interval corresponds to extremely low resolution, retaining 50% of the point cloud.

[0100] When downsampling (i.e., reducing the resolution), voxel filtering or random sampling can be used to reduce the number of points in the cloud. For example, downsampling a point cloud according to the retention ratio of downsampling can be achieved by one of the following methods: 1) assuming the retention ratio is r, randomly discard (1-r) points in the point cloud; 2) dividing the point cloud into a voxel grid, retaining one representative point per voxel, and dynamically adjusting the voxel size based on the retention ratio r of downsampling. The lower the retention ratio r, the larger the voxel.

[0101] In step 204, sparse regions of the point cloud are interpolated and completed, and dense regions are further downsampled. This prevents information loss in sparse regions and avoids computational redundancy in dense regions. In some examples, sparse and dense regions can be dynamically demarcated based on the local density of the point cloud. For example, the local density of the point cloud can be estimated using K-Nearest Neighbors (KNN) and dynamically distinguished based on this local density.

[0102] Specifically, the process of performing sparse area interpolation and completion and dense area downsampling on the downsampled point cloud in step 204 may include: calculating the average K-nearest neighbor distance of each point through the KNN algorithm, dividing the point cloud into sub-blocks, and performing the following processing on each sub-block: independently calculating a threshold based on the average K-nearest neighbor distance of all points and separating it into dense areas and sparse areas through the threshold, further downsampling the point cloud in the dense area, performing interpolation and completion on the point cloud in the sparse area, and finally merging the point cloud in the dense area and the point cloud in the sparse area to obtain the preprocessed point cloud.

[0103] Among them, K (for example, K = 10~50) is selected according to the density characteristics of the point cloud to balance noise sensitivity and local detail preservation. The median or specific percentile (such as 25%) can be selected as the threshold by analyzing the average distance distribution of the K nearest neighbors of all points.

[0104] Based on KNN, the local density of point clouds is estimated and sparse and dense areas are dynamically distinguished. By dynamically responding to the local characteristics of point clouds, efficient and flexible compression of point clouds in complex scenes can be achieved. It has strong adaptability, retains details in key areas, and reduces information loss caused by uniform downsampling.

[0105] In step 204, adaptive downsampling based on vehicle speed and adaptive processing based on local point cloud density (i.e., interpolation and completion in sparse areas and downsampling in dense areas) work together. The vehicle speed determines the "overall resolution tone" of the point cloud, and the local point cloud density determines the "local detail enhancement or compression" of the point cloud. This can achieve "global lightweighting and local refinement" in a dynamic environment, while responding to both dynamic environments (i.e., speed changes) and static characteristics (i.e., local density distribution), and can optimize the processing efficiency and accuracy of the point cloud in multiple dimensions.

[0106] In step 205, obtaining point cloud features using the preprocessed point cloud may include: adjusting the voxel grid size according to the local point cloud density to adaptively voxelize the point cloud to obtain voxel data, and using separable 3D convolution to process the voxel data to obtain point cloud features.

[0107] In some examples, point clouds can be voxelized through dynamic meshing (Density-based Voxelization) (also known as density-based voxelization) to obtain voxel data of the point cloud. Specifically, the point cloud is discretized into a non-uniform voxel grid and the voxel resolution of the non-uniform voxel grid is dynamically adjusted according to the point cloud density. Dynamic meshing voxelization adjusts the voxel resolution in real time according to point cloud density, geometric features, or physical field changes. While retaining details, it can optimize computational efficiency, reduce unnecessary resource consumption, avoid information loss in sparse areas, and suppress computational redundancy in dense areas. It generates a non-uniform voxel grid and significantly reduces the computational complexity of subsequent convolutions (reducing ineffective computation by 30-50%).

[0108] Specifically, the process of voxelizing a point cloud through dynamic meshing can include: first, generating an initial voxel grid based on geometric or data features, typically using uniform meshing or multi-resolution initialization; then, calculating the local point density of the point cloud in real time. When the local point density exceeds a preset threshold, the voxels in the corresponding region (i.e., high-density areas such as object edges and complex structures) are updated to smaller voxels (e.g., 0.1 m³) to preserve details such as vehicle outlines; and when the local point density falls below the threshold, adjacent voxels are merged to make the corresponding region (e.g., low-density areas such as open space) use larger voxels (e.g., increasing from an initial 0.4 m³ to a current 0.5 m³) to compress the background point cloud and reduce computational effort. After the voxel grid is adjusted, voxel features (e.g., mean coordinates and surface normals) can be further calculated using bilinear interpolation or covariance matrices to ensure data continuity. Furthermore, efficient multi-resolution voxel management methods such as adaptive octrees, sparse voxel structures (SVOs), or hash tables can be used.

[0109] In step 205, the separable 3D convolution can be decomposed into 2D spatial convolution and 1D channel convolution. Specifically, the process of using separable 3D convolution to process the voxel data to obtain point cloud features can include: independently performing 2D convolution on each depth slice of the three-dimensional voxel grid to extract spatial features within each channel, and performing 1D convolution along the channel dimension to fuse the spatial features of different channels to obtain point cloud features. Among them, spatial features can include spatial structural features such as object shape, object position, object edge, etc. Spatial features are the basis of high-level semantics and can be gradually extracted through multi-level network layers.

[0110] In other implementations, the separable 3D convolution operation can also be performed according to the needs in the manner of “first 1D depth convolution and then 2D spatial convolution, mixed splitting”.

[0111] As can be seen from the above, the separable 3D convolution decomposes the three-dimensional convolution into two independent low-dimensional operations, which can reduce redundant calculations, avoid repeated calculations in three-dimensional convolutions, and reduce the waste of calculations on empty voxels, significantly improving computational efficiency without losing accuracy.

[0112] The extraction of point cloud features is achieved by combining density-adaptive voxelization and separable 3D convolution, which can effectively improve the overall efficiency and accuracy.

[0113] In step 206, various applicable methods can be used to achieve the fusion of point cloud features and image features. The present disclosure does not limit the specific fusion method. For example, a cross-modal shared attention mechanism can be used to map point cloud features and image features to a shared space, calculate attention weights, and perform weighted fusion to obtain multimodal fusion features.

[0114] Figure 5 Another flow chart of the environment-adaptive multimodal data fusion method provided by the embodiment of the present disclosure is shown. Figure 5 , further, before step 206, it may also include:

[0115] Step 208: evaluating whether the cross-modal consistency of the image features and the point cloud features meets the requirements;

[0116] In step 209 , when the cross-modal consistency between the image features and the point cloud features does not meet the requirements, a generative adversarial network (GAN) is used to generate a pseudo multi-view image based on the point cloud, obtain features of the pseudo multi-view image, and merge the features of the pseudo multi-view image into the image features.

[0117] In some embodiments, in step 208, whether the cross-modal consistency of the image features and the point cloud features meets the requirements can be evaluated in one or both of the following ways: 1) calculating the similarity between the point cloud features and the image features at the same spatial position through BEV spatial contrast learning, and when the similarity is less than a first predetermined similarity threshold, determining that the cross-modal consistency of the point cloud features and the image features does not meet the requirements; 2) calculating the cross-modal feature similarity between the forward perspective image and the point cloud in the multi-perspective image, and when the cross-modal feature similarity is less than a second predetermined similarity threshold, determining that the cross-modal consistency of the point cloud and the forward perspective image does not meet the requirements.

[0118] Specifically, the average cosine similarity between the point cloud features and image features at the same spatial location can be calculated. If the average cosine similarity is greater than or equal to a first predetermined similarity threshold, the evaluation is considered passed, and the image features obtained in step 203 and the point cloud features obtained in step 205 can be directly fused. If the average cosine similarity is less than the first predetermined similarity threshold, new image features can be obtained by combining the image features of the pseudo multi-view image generated by the generative adversarial network with the image features obtained in step 203. This new image feature is then fused with the point cloud features obtained in step 205. Thus, BEV spatial contrast learning can be used to dynamically evaluate the consistency of the feature distributions of the lidar and camera in BEV space, resolving projection bias issues, further reducing fusion bias caused by the external environment in extreme scenarios (e.g., rainy days, nighttime, etc.), and improving the reliability and robustness of multimodal data fusion.

[0119] Furthermore, during the model training phase, the parameters of the image feature-related model and the point cloud feature-related model can be further optimized by adding the contrast loss (InfoNCE Loss) function in BEV spatial contrast learning to the loss function, forcing cross-modal features to be aligned in the BEV space, thereby solving the projection deviation problem and further reducing the fusion deviation caused by the external environment in extreme scenarios (such as thunderstorms, nighttime, etc.), thereby improving the reliability and robustness of multimodal data fusion.

[0120] Specifically, the average cosine similarity between the forward-view image features (i.e., the features obtained in step 301) and the point cloud features at the same spatial location can be calculated, and this average cosine similarity can be used as the cross-modal feature similarity between the forward-view image and the point cloud in the multi-view image. If the average cosine similarity is greater than or equal to a second predetermined similarity threshold, the evaluation is considered passed, and the image features obtained in step 203 and the point cloud features obtained in step 205 can be directly fused. If the average cosine similarity is less than the second predetermined similarity threshold, new image features can be generated by combining the image features of the pseudo multi-view image generated by a generative adversarial network with the image features obtained in step 203. This new image feature can then be fused with the point cloud features obtained in step 205. This allows the consistency of the feature distributions of the lidar and camera to be assessed through forward graph feature matching, resolving projection bias issues, further reducing fusion bias caused by external environments in extreme scenarios (e.g., rainy days, nighttime), and improving the reliability and robustness of multimodal data fusion.

[0121] In specific applications, in the BEV space coordinate system, the point cloud and image features can be projected onto a unified BEV grid through pre-calibrated parameters or a learnable transformation matrix (such as MLP), and then the grid-level cosine similarity can be calculated.

[0122] Furthermore, during the model training phase, by adding the mutual information (MI) related to forward graph feature matching to the loss function, the parameters of the image feature-related model and the point cloud feature-related model can be further optimized, thereby enhancing the representation consistency of the same object in different modalities, solving the projection deviation problem, and further reducing the fusion deviation caused by the external environment (such as occlusion, lighting changes, etc.) in extreme scenarios (such as thunderstorms, nighttime, etc.), thereby improving the reliability and robustness of multimodal data fusion.

[0123] In step 209, the GAN can be, but is not limited to, a pixel-level generative adversarial network. Specifically, the GAN can be, but is not limited to, CycleGAN. A GAN consists of a generator and a discriminator. The generator is responsible for generating pseudo data in one modality based on data in another modality, while the discriminator distinguishes true from false, driving the generator to produce more realistic pseudo data.

[0124] In step 209, the GAN loss function includes: adversarial loss, cycle consistency loss, and cross-modal alignment loss. The adversarial loss (GAN loss) is used to train the generator and discriminator to perform adversarial training, making the features generated by the generator as realistic as possible. The cycle consistency loss ensures that the generated features are consistent with the original features after conversion back to the original modality, and the cross-modal alignment loss is used to measure the degree of alignment between the generated features and the target modality features. This allows the GAN to maintain cross-modal alignment, and the generated features should be semantically and structurally consistent with the target modality features, thereby improving the robustness of feature alignment of images and point clouds in extreme scenarios such as camera failure.

[0125] In specific applications, GAN can be jointly optimized through adversarial training and cross-modal contrast loss (such as InfoNCE Loss) to ensure that the generated features and point cloud features are distributed consistently in the BEV space.

[0126] In certain extreme scenarios, a particular modality may fail. For example, at night or in fog, a camera may not function properly due to insufficient lighting or low visibility. Another example is rain or snow, where the camera image may be interfered with by raindrops or snowflakes. Another example is night or fog. In such situations, the camera may fail completely or partially. Relying on the camera's visual features, as on clear days, would result in performance degradation or even failure. Similarly, in extreme scenarios, point clouds collected by lidar can be used to generate visual features. Given this, step 209 may also include obtaining meteorological data. When the meteorological data indicates that the current scene is a pre-defined extreme scenario (e.g., night, fog, rain, snow, etc.), the generator in the pixel-level adversarial network can be used to generate a pseudo-multi-view image based on the point cloud, obtain features of the pseudo-multi-view image, and merge the features of the pseudo-multi-view image into the image features. This can further improve the robustness of feature alignment between multi-view images and point clouds in extreme scenarios (e.g., single-modality failure).

[0127] See also Figure 5 Furthermore, before step 206, the method may further include: step 210, dynamically adjusting the modal weights of the image features based on the exposure anomalies of each view in the multi-view image; and / or, step 211, dynamically adjusting the modal weights of the point cloud features based on the local point cloud density of the point cloud. In step 206, the image features and point cloud features may be fused based on the modal weights of the image features and / or the modal weights of the point cloud features to obtain a multimodal fused feature.

[0128] In step 210, overexposed and / or underexposed areas can be detected for each of the multi-view images, and a two-dimensional mask can be generated to mark the overexposed and / or underexposed areas. The modal weights of the image features can then be dynamically adjusted based on the two-dimensional mask. This can suppress the impact of low-quality image areas on fusion.

[0129] In some examples, overexposed / underexposed areas can be detected and masks generated by histogram analysis and threshold judgment. Specifically, the following processing can be performed on each perspective image to generate its two-dimensional mask: extract the brightness information of the perspective image to generate a brightness histogram of the perspective image, normalize the pixel values ​​of the brightness histogram to [0, 255], use a fixed threshold method (i.e., the overexposure threshold and the underexposure threshold are both preset fixed values) or an adaptive threshold method (i.e., dynamically adjust the overexposure threshold and the underexposure threshold according to the brightness histogram, use the position where the brightness histogram suddenly drops on the right side as the overexposure threshold, and use the position where the brightness histogram suddenly rises on the left side as the underexposure threshold) to generate an overexposure mask and an underexposure mask based on the brightness histogram, and perform a dot product of the overexposure mask and the underexposure mask to obtain a joint mask, which is the two-dimensional mask of the perspective image.

[0130] In specific applications, adaptive threshold segmentation such as dynamically setting thresholds based on the mean and variance of image blocks or a lightweight U-Net model that dynamically adjusts the modal weights of image features according to the two-dimensional mask can be used, but is not limited to.

[0131] In some examples, dynamically adjusting the modal weights of image features based on the two-dimensional mask can be achieved as follows: mapping the two-dimensional masks of each view image to a unified coordinate system (e.g., BEV or panoramic plane) to obtain the two-dimensional mask of the multi-view image, converting the two-dimensional mask of the multi-view image into a confidence score C∈[0,1], and adjusting the modal weights of the image features according to the confidence score C.

[0132] By dynamically adjusting the modal weights of image features through overexposure / underexposure masks, the impact of low-quality image areas on fusion can be effectively suppressed in strong light, backlight, and low-light environments at night, reducing fusion deviation and improving fusion accuracy.

[0133] In step 211 , the local point cloud density of the point cloud may be calculated by kernel density estimation (KDE), the local point cloud density of the point cloud may be mapped to a reliability score, and the modal weight of the point cloud feature may be adjusted according to the reliability score.

[0134] Since the point spacing in the sparse area is large, the density value after KDE superposition is low, and the reliability score of the sparse area is low, the weight is reduced during fusion. That is, the sparse area can be automatically downgraded to avoid introducing unreliable point cloud features in multimodal data fusion, thereby effectively improving the robustness of multimodal fusion in scenarios with sparse point clouds or occlusion.

[0135] In specific applications, the fusion features of image feature modality weights obtained by dynamically adjusting the 2D mask of overexposed / underexposed areas and point cloud features obtained by dynamically adjusting the local point cloud density can be used simultaneously in the fusion of image and point cloud features. The synergy between the two can significantly improve the robustness of multimodal systems, especially in scenes with complex lighting and sparse point clouds.

[0136] In step 207, an anchor-free detection head can be used to perform 3D object detection based on multimodal fusion features, obtaining 3D object detection results. These 3D object detection results include the parameters of each 3D bounding box and its object category. The 3D bounding box parameters include center coordinates, dimensions, and heading angle. The heading angle represents the orientation angle of the target object, the center point represents the target object's location center, and the dimensions represent the target object's width, height, and length. This allows for high-precision 3D object detection results that meet the decision-making requirements of application scenarios such as autonomous driving.

[0137] In step 207, the sequence of multimodal fusion features can be modeled using models such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), gated recurrent units (GRUs), or transformers to predict the vehicle's trajectory over a predetermined time period (e.g., 5 seconds). The vehicle's trajectory can be represented as a sequence of trajectory points and may also include a sequence of motion parameters such as speed and acceleration. Predicting vehicle trajectory using multimodal fusion features fully leverages the advantages of both point cloud and image modalities, improving the accuracy and robustness of trajectory prediction.

[0138] Furthermore, after step 207, it can also include: performing smooth interpolation correction on the vehicle motion trajectory obtained in step 207 by combining Kalman filtering with polynomial fitting, thereby eliminating the influence of single-frame detection jitter on the vehicle motion trajectory and improving the prediction accuracy of the vehicle motion trajectory.

[0139] Figure 6 FIG2 shows a schematic diagram of the structure of the environment-adaptive multimodal data fusion device provided by the embodiment of the present disclosure. Figure 6 The environment-adaptive multimodal data fusion device 600 of the embodiment of the present disclosure may include:

[0140] A data acquisition unit 601 is configured to acquire multi-view images acquired by a multi-view camera array and point clouds acquired by a laser radar;

[0141] An image preprocessing unit 602 is configured to preprocess the multi-view images, wherein the preprocessing includes adjusting the resolution of each view image in the multi-view images according to the illumination conditions of the view image;

[0142] An image feature extraction unit 603 is configured to obtain image features using the pre-processed multi-view images;

[0143] The point cloud preprocessing unit 604 is used to preprocess the point cloud, wherein the preprocessing includes: downsampling the point cloud according to the vehicle speed, and performing sparse area interpolation and dense area downsampling on the downsampled point cloud;

[0144] The point cloud feature extraction unit 605 is used to obtain point cloud features using the pre-processed point cloud;

[0145] A fusion unit 606 is used to fuse image features and point cloud features to obtain multimodal fusion features;

[0146] The task execution unit 607 is configured to perform 3D target detection and / or vehicle trajectory prediction using multimodal fusion features.

[0147] Furthermore, the image preprocessing unit 602 can be specifically used to adjust the resolution of the perspective image in the following manner: determining the illumination level of the perspective image through a lightweight convolutional neural network classifier; when the illumination level of the perspective image is lower than a predetermined level, increasing the resolution of the perspective image to the image resolution corresponding to the predetermined level; when the illumination level of the perspective image is higher than the predetermined level, reducing the resolution of the perspective image to the image resolution corresponding to the predetermined level.

[0148] Furthermore, the image feature extraction unit 603 can be specifically used to: perform the following processing on each perspective image in the preprocessed multi-perspective image to extract the features of each perspective image: dynamically adjust the RGB channel weights of the basic convolution kernel according to the illumination level of the perspective image through an image feature extraction module including conditional convolution to generate a dynamic convolution kernel that adapts to the illumination conditions of the perspective image and use the dynamic convolution kernel to extract the features of the perspective image; and align and fuse the features of each perspective image to obtain multi-scale features.

[0149] Furthermore, the image feature extraction unit 603 can also be used to: use a lightweight semantic segmentation network to generate a semantic segmentation mask based on the view image, and apply the semantic segmentation mask as an attention weight in the process of extracting features of the view image using a dynamic convolution kernel.

[0150] Furthermore, the point cloud preprocessing unit 604 can be specifically used to: adjust the voxel grid size according to the local point cloud density to adaptively voxelize the preprocessed point cloud to obtain voxel data; and use separable 3D convolution to process the voxel data to obtain point cloud features.

[0151] Furthermore, the environment-adaptive multimodal data fusion device 600 may also include: a consistency evaluation unit 608 and a forgery unit 609; wherein, the consistency evaluation unit 608 can be used to evaluate whether the cross-modal consistency of the image features and the point cloud features meets the requirements; the forgery unit 609 can be used to use the generative adversarial network GAN to generate a pseudo multi-view image based on the point cloud when the cross-modal consistency of the image features and the point cloud features does not meet the requirements, obtain the features of the pseudo multi-view image, and merge the features of the pseudo multi-view image into the image features.

[0152] Furthermore, the consistency evaluation unit 608 can be specifically used to evaluate whether the cross-modal consistency of image features and point cloud features meets the requirements through one or two of the following: 1) calculating the similarity between point cloud features and image features at the same spatial position through BEV spatial contrast learning, and when the similarity is less than a first predetermined similarity threshold, determining that the cross-modal consistency of point cloud features and image features does not meet the requirements; 2) calculating the cross-modal feature similarity between the forward perspective image and the point cloud in the multi-perspective image, and when the cross-modal feature similarity is less than a second predetermined similarity threshold, determining that the cross-modal consistency of the point cloud and the forward perspective image does not meet the requirements.

[0153] Furthermore, the environment-adaptive multimodal data fusion device 600 may also include: an image feature modal weight adjustment unit 610 and / or a point cloud feature modal weight adjustment unit 611. The image feature modal weight adjustment unit 610 may be used to dynamically adjust the modal weight of the image feature based on the exposure anomaly of each view image in the multi-view image, and the point cloud feature modal weight adjustment unit 611 may be used to dynamically adjust the modal weight of the point cloud feature based on the local point cloud density of the point cloud. The fusion unit 606 may specifically be used to fuse the image feature and the point cloud feature based on the modal weight of the image feature and / or the modal weight of the point cloud feature to obtain a multimodal fusion feature.

[0154] In specific applications, the environment-adaptive multimodal data fusion device 600 can be implemented by software, hardware, or a combination of both. For example, the environment-adaptive multimodal data fusion device 600 can be implemented as the electronic device 700 described below or can be implemented as software in the electronic device 700 described below.

[0155] In addition, an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. The program includes instructions, and when the instructions are executed by one or more processors of a computing device, the steps of the aforementioned environment-adaptive multimodal data fusion method are executed.

[0156] Figure 7 Schematic diagram of the structure of the electronic device provided by the embodiment of the present disclosure is shown. Figure 7 The electronic device 700 may include: one or more processors 701, and a memory 702 storing one or more programs, which are executed by the one or more processors 701 to implement the method flow shown in the above embodiments of the present disclosure and / or the program units corresponding to each unit in the device.

[0157] The processor 701 may include one or more single-core processors or multi-core processors. The processor 701 may include any combination of general-purpose processors or special-purpose processors (such as CPUs, GPUs, etc.).

[0158] The memory 702 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the embodiment of the present disclosure. Figure 2 The program instructions / units corresponding to the environment-adaptive multimodal data fusion method shown. The processor 701 executes the non-transient software programs, instructions and units stored in the memory 702, thereby executing the above-mentioned method embodiment. Figure 2 The program, instructions and units corresponding to the environment-adaptive multimodal data fusion method shown.

[0159] The electronic device 700 may further include: an input device 703, an output device 704, a communication component 705, etc. The processor 701, the memory 702, the input device 703, the output device 704 and the communication component 705 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0160] The programs (also referred to as software, software applications, or code) described above include machine instructions for a programmable processor and may be implemented using a vehicle-specific programming language, assembly, or machine language.

[0161] Over time and with the advancement of technology, the meaning of "medium" has become increasingly broad. The dissemination of computer programs is no longer limited to tangible media and can also be directly downloaded from the Internet. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component.

[0162] In specific applications, the electronic device 700 can be implemented as, but not limited to, a domain controller or other similar devices.

[0163] The embodiment of the present disclosure further provides a vehicle equipped with a multi-perspective image sequence and a laser radar. The vehicle may include the aforementioned environment-adaptive multimodal data fusion device 600 and / or electronic device 700.

[0164] The technical solutions provided by the present disclosure are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The descriptions of the above embodiments are intended only to help understand the methods and core concepts of the present disclosure. Furthermore, those skilled in the art will appreciate that variations in the specific implementation methods and scope of application may occur based on the concepts of the present disclosure. In summary, the contents of this specification should not be construed as limiting the present disclosure.

[0165] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An environment-adaptive multimodal data fusion method, characterized in that: The method is applied to a vehicle equipped with a multi-view camera array and a laser radar; the method comprises: Acquire multi-view images captured by the multi-view camera array and point clouds captured by the laser radar; Preprocessing the multi-view images, the preprocessing comprising: adjusting the resolution of each view image in the multi-view images according to the illumination conditions of the view image; Obtain image features using preprocessed multi-view images; Preprocessing the point cloud, the preprocessing comprising: downsampling the point cloud according to the vehicle speed, and performing sparse area interpolation and dense area downsampling on the downsampled point cloud; Use the preprocessed point cloud to obtain point cloud features; evaluating whether cross-modal consistency between the image features and the point cloud features meets requirements; if the cross-modal consistency between the image features and the point cloud features does not meet requirements, generating a pseudo multi-view image based on the point cloud using a generative adversarial network (GAN), obtaining image features of the pseudo multi-view image, and obtaining new image features by mixing the image features of the pseudo multi-view image with image features obtained using the preprocessed multi-view image; fusing the new image features and the point cloud features to obtain a multimodal fusion feature; The multimodal fusion features are used to perform 3D object detection and / or vehicle trajectory prediction.

2. The method according to claim 1, characterized in that Adjust the resolution of the view image as follows: determining the illumination level of the view image using a lightweight convolutional neural network classifier; When the illumination level of the perspective image is lower than a predetermined level, increasing the resolution of the perspective image to an image resolution corresponding to the predetermined level; When the illumination level of the perspective image is higher than the predetermined level, the resolution of the perspective image is reduced to an image resolution corresponding to the predetermined level.

3. The method according to claim 2, characterized in that The method of obtaining image features by using the pre-processed multi-view images includes: The following processing is performed on each view image in the preprocessed multi-view image to extract features of each view image: dynamically adjusting the RGB channel weights of a basic convolution kernel according to the illumination level of the view image by an image feature extraction module including conditional convolution to generate a dynamic convolution kernel adapted to the illumination condition of the view image, and extracting features of the view image using the dynamic convolution kernel; The features of images from different viewpoints are aligned and fused to obtain multi-scale features.

4. The method according to claim 3, characterized in that The method of obtaining image features using the adjusted multi-view images also includes: using a lightweight semantic segmentation network to generate a semantic segmentation mask based on the view image, and applying the semantic segmentation mask as an attention weight in the process of extracting features of the view image using the dynamic convolution kernel.

5. The method according to claim 1, wherein The method of obtaining point cloud features by using the pre-processed point cloud includes: Adjusting the voxel grid size according to the local point cloud density to adaptively voxelize the pre-processed point cloud to obtain voxel data; The voxel data is processed using separable 3D convolution to obtain the point cloud features.

6. The method according to claim 1, characterized in that The evaluating whether the cross-modal consistency of the image features and the point cloud features meets the requirements includes one or more of the following: Calculating the similarity between the point cloud feature and the image feature at the same spatial position through BEV spatial contrast learning, and determining that the cross-modal consistency of the point cloud feature and the image feature does not meet the requirement when the similarity is less than a first predetermined similarity threshold; Calculate the cross-modal feature similarity between the forward view image and the point cloud in the multi-view image, and when the cross-modal feature similarity is less than a second predetermined similarity threshold, determine that the cross-modal consistency of the point cloud and the forward view image does not meet the requirements.

7. The method according to claim 1, characterized in that The method further includes: dynamically adjusting the modal weight of the image feature according to an exposure anomaly of each view image in the multi-view image; and / or dynamically adjusting the modal weight of the point cloud feature according to a local point cloud density of the point cloud; The fusing of the image features and the point cloud features to obtain a multimodal fusion feature includes: fusing the image features and the point cloud features based on the modal weights of the image features and / or the modal weights of the point cloud features to obtain the multimodal fusion feature.

8. An environment-adaptive multimodal data fusion device, characterized in that: The device is applied to a vehicle, which is equipped with a multi-view camera array and a laser radar; The environment-adaptive multimodal data fusion device comprises: A data acquisition unit, configured to acquire the multi-view images acquired by the multi-view camera array and the point cloud acquired by the laser radar; An image preprocessing unit, configured to preprocess the multi-view images, wherein the preprocessing comprises: adjusting the resolution of each view image in the multi-view images according to the illumination conditions of the view image; An image feature extraction unit, configured to obtain image features using the preprocessed multi-view images; a point cloud preprocessing unit, configured to preprocess the point cloud, wherein the preprocessing comprises: downsampling the point cloud according to the vehicle speed, and performing sparse region interpolation and dense region downsampling on the downsampled point cloud; A point cloud feature extraction unit, used to obtain point cloud features using the preprocessed point cloud; A consistency evaluation unit is used to evaluate whether the cross-modal consistency of image features and point cloud features meets the requirements; a forging unit, configured to generate a pseudo multi-view image based on the point cloud using a generative adversarial network (GAN) when cross-modal consistency between image features and point cloud features does not meet requirements, obtain image features of the pseudo multi-view image, and obtain new image features by mixing the image features of the pseudo multi-view image with the image features obtained by the image feature extraction unit; a fusion unit, configured to fuse the new image features and the point cloud features to obtain a multimodal fusion feature; A task execution unit is used to perform 3D target detection and / or vehicle trajectory prediction using the multimodal fusion features.

9. A vehicle equipped with a multi-view camera sequence and a laser radar, characterized in that: The vehicle comprises the device of claim 8.

Images