An intelligent networked vehicle multi-modal fusion automatic driving decision platform
By employing multimodal sensor collaboration and deep learning technology, the problems of perception instability and decision-making delay in high-level autonomous driving systems under complex environments have been solved, achieving stable and reliable perception and efficient decision-making, complying with traffic regulations and possessing good scalability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHONGXIONG ZHITU (XIONGAN) TECHNOLOGY CO LTD
- Filing Date
- 2026-02-28
- Publication Date
- 2026-05-26
Smart Images
Figure CN122090418A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent connected vehicles and autonomous driving technology, specifically to an autonomous driving decision-making platform that integrates multimodal technologies for intelligent connected vehicles. Background Technology
[0002] With technological advancements, advanced autonomous driving systems are facing increasingly stringent standards for comprehensive environmental perception, precise understanding, and real-time, safe decision-making capabilities. Currently, mainstream solutions in the industry face several common challenges: for example, reliance on single sensors (such as pure vision solutions) leads to unstable performance in adverse weather or backlight conditions; and simple multi-sensor pre-fusion or post-fusion strategies, with their fixed weights, struggle to adapt to dynamically changing and complex road conditions, resulting in fluctuating perception reliability. Furthermore, perception, prediction, and planning modules are often executed sequentially as independent tasks, increasing system latency and potentially leading to suboptimal or even contradictory decisions due to information asynchrony between tasks.
[0003] With technological advancements, advanced autonomous driving systems are facing increasingly stringent standards for comprehensive environmental perception, precise understanding, and real-time, safe decision-making capabilities. Currently, mainstream solutions in the industry face several common challenges: for example, reliance on single sensors (such as pure vision solutions) leads to unstable performance in adverse weather or backlight conditions; and simple multi-sensor pre-fusion or post-fusion strategies, with their fixed weights, struggle to adapt to dynamically changing and complex road conditions, resulting in fluctuating perception reliability. Furthermore, perception, prediction, and planning modules are often executed sequentially as independent tasks, increasing system latency and potentially leading to suboptimal or even contradictory decisions due to information asynchrony between tasks. Summary of the Invention
[0004] The purpose of this invention is to provide an autonomous driving decision-making platform and system for intelligent connected vehicles with multimodal fusion, aiming to systematically solve the aforementioned limitations in perception, rigid fusion, insufficient coordination, and safety challenges, thereby comprehensively improving the overall performance of autonomous driving systems in complex traffic scenarios.
[0005] To address the aforementioned technical problems, this invention provides an autonomous driving decision-making platform for intelligent connected vehicles based on multimodal fusion, characterized by the following steps:
[0006] 1. Multimodal raw data acquisition: High-definition cameras, multi-line lidar and millimeter-wave radar are deployed in a coordinated manner to acquire high-resolution images, 3D point clouds and target velocity and distance information, respectively, and construct a redundant and complementary environmental information input stream.
[0007] 2. Multimodal Raw Data Preprocessing: Targeted processing is performed on the raw data from each sensor. Image filtering and noise reduction are applied; point cloud filtering is applied to remove outlier noise points; and radar data is calibrated. Crucially, high-precision hardware timestamps and a pre-set spatial calibration matrix are used to achieve strict alignment of multi-source data in the spatiotemporal coordinate system, laying the foundation for subsequent fusion.
[0008] 3. Feature Extraction and Unified Representation: The preprocessed image and point cloud data are converted into a regular voxel grid. Subsequently, a deep neural network (such as CNN) is used to extract the semantic and texture features of the image, and a point cloud processing network is used to extract geometric and spatial features. Through a feature mapping layer, the two types of heterogeneous features are projected onto a unified latent space to form an isomorphic intermediate representation.
[0009] 4. Scene-Adaptive Feature Fusion: Design a dynamic weighted fusion module (such as a gating network based on an attention mechanism). This module can automatically adjust the contribution of each modality feature in the fusion process based on the real-time perceived environmental state (such as weather, lighting, and visibility). For example, in rainy or foggy weather, it increases the reliance on radar features; in good lighting conditions, it places more emphasis on high-precision visual and lidar features. This effectively improves the system's environmental robustness.
[0010] 5. Multi-task Collaborative Learning and Decision-Making: The fused unified features are fed into a multi-task learning model. This model adopts an architecture of "shared backbone network + independent task heads," where the shared layer extracts common features, and the task layers execute in parallel.
[0011] Object detection: Utilizing optimized object detection algorithms, it accurately identifies and locates key objects such as vehicles, pedestrians, and traffic signs.
[0012] Trajectory prediction: Based on recurrent neural networks (such as LSTM) or temporal Transformers, combined with the target's historical state, predict its motion trajectory in the short term.
[0013] Route planning: Combining vehicle status, global route, perception and prediction results, it plans safe, comfortable and traffic-compliant local driving routes in real time.
[0014] This design reduces computational redundancy through low-level feature sharing and enhances consistency between tasks through joint training.
[0015] 6. Compliance Verification and Scenario Understanding: Construct a traffic rule knowledge base to logically verify preliminary decision results (e.g., whether lane changing regulations are violated). Simultaneously, integrate a dedicated scenario understanding module, which combines image recognition (e.g., traffic light status) with multimodal information to perform deep reasoning on pedestrian intentions and special traffic scenarios (e.g., construction zones), providing richer context for decision-making.
[0016] 7. Hierarchical Decision Making and Action Execution: A hierarchical decision-making framework is adopted to decouple high-level navigation intent, mid-level behavioral decisions (following, lane changing, overtaking), and low-level trajectory control. Finally, smooth, feasible, and safe control commands are generated through optimization algorithms (such as Model Predictive Control, MPC) or search algorithms (such as Monte Carlo Tree Search) and then executed by the vehicle's drive-by-wire system.
[0017] This invention also provides an autonomous driving decision-making system, the hardware and software of which are as follows:
[0018] 1. Multi-sensor interface unit: integrates multiple interface protocols such as CANFD and automotive Ethernet to ensure high-speed and reliable data communication with various automotive sensors.
[0019] 2. High-performance data processing unit: Typically composed of a high-performance SoC (such as NVIDIA Orin) or combined with an FPGA. It is responsible for running computationally intensive algorithms such as sensor data preprocessing, feature extraction, and fusion, meeting stringent real-time requirements.
[0020] 3. Decision and Control Unit: Runs mature middleware frameworks such as ROS2, and is responsible for managing multi-task decision models, executing hierarchical decision logic, and translating decisions into specific vehicle control commands.
[0021] 4. Vehicle-Cloud Collaboration Unit: Connects to the cloud via a 5G / V2X communication module. While strictly protecting data privacy (e.g., using federated learning technology), it enables continuous model iteration and traffic situation sharing across vehicle fleets, improving the overall intelligence level of the system.
[0022] In summary, due to the adoption of the above-mentioned technologies, the beneficial effects of this invention are:
[0023] 1. The dynamic adaptive fusion strategy of this invention enables the system to maintain stable and reliable environmental perception capabilities under various complex environmental conditions without relying on a single sensor.
[0024] 2. The multi-task parallel learning framework of this invention reduces the information transmission delay and inconsistency between modules, and improves end-to-end efficiency and overall performance from perception to decision-making.
[0025] 3. By introducing rule verification and deep scenario understanding, this invention enables autonomous driving decisions to not only meet the technical optimality but also comply with traffic regulations and human driving habits, and has the highest level of functional safety backing.
[0026] 4. The modular design of hardware and software decoupling in this invention, as well as the continuous learning capability of vehicle-cloud collaboration, enable the system to be continuously optimized with the advancement of algorithms and the accumulation of data, and have good scalability and life cycle value. Attached Figure Description
[0027] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention, making other features, objects, and advantages of the invention more apparent. The illustrative embodiments of the invention illustrated in the drawings and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:
[0028] Figure 1 This is a schematic diagram of the process structure of the present invention. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0030] In the description of this invention, it should be understood that the terms indicating orientation or positional relationship are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing the invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention.
[0031] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific context of the specification.
[0032] This invention provides a multimodal fusion autonomous driving decision-making platform for intelligent connected vehicles. Taking online mathematics exam grading in K-12 education as an example, the specific steps are as follows:
[0033] Step 1: Acquisition of Multimodal Raw Data
[0034] Select high-definition cameras, LiDAR, and millimeter-wave radar with matching performance and meeting automotive-grade requirements as the core sensor combination to provide redundant and complementary environmental perception information. For example, a specific type of camera can be used to cover a wide field of view in the main driving directions of the vehicle, a specific beamwidth of LiDAR can be selected to obtain spatial point cloud density that meets the accuracy requirements, and multiple millimeter-wave radars can be deployed to cover blind spots around the vehicle. All sensors should be triggered through a unified synchronization mechanism to reduce time errors in data acquisition and provide a basic data stream for subsequent processing.
[0035] Step 2: Multimodal Data Preprocessing
[0036] 1. The collected modal data need to be preprocessed in a targeted manner to improve data quality and unify the reference system.
[0037] 2. Image Processing: Common image enhancement and filtering algorithms are used to process camera image data in order to suppress noise and improve image quality.
[0038] 3. Point cloud processing: For lidar point cloud data, ground segmentation, removal of non-obstacle point clouds (such as raindrops and dust) and necessary noise reduction processing are required.
[0039] 4. Radar data calibration: Calibrate the output of the millimeter-wave radar to correct measurement biases introduced by system installation and device characteristics.
[0040] 5. Spatiotemporal alignment: The core of achieving this step lies in precise positioning and precise time synchronization.
[0041] 6. Time synchronization: Strict timestamp alignment of multi-sensor data is achieved through high-precision clock signals (such as GPS-based time synchronization).
[0042] 7. Spatial Calibration: Through a pre-defined calibration process, the transformation relationship between each sensor coordinate system and the vehicle coordinate system is accurately determined, thereby unifying all sensing data into the same spatial coordinate system.
[0043] Step 3: Multimodal Feature Extraction and Unified Representation
[0044] To achieve deep integration of heterogeneous data, it is usually converted into a unified data structure for processing.
[0045] 1. Convert the preprocessed 2D image and 3D point cloud data into voxel representation, discretizing the continuous space to facilitate subsequent calculations.
[0046] 2. Utilize deep neural networks to extract deep features from images and point clouds respectively. For deep feature extraction from images, mature convolutional neural networks (CNNs) can be used; for feature extraction from point clouds, specialized point cloud processing networks can be used.
[0047] 3. By designing a mapping layer, the above image features and point cloud features are projected into a shared, high-dimensional feature space, enabling them to be directly subjected to mathematical operations, thus preparing for fusion.
[0048] Step 4: Context-based Adaptive Multimodal Fusion
[0049] To overcome the limitations of fixed-weight fusion, the key lies in introducing a fusion mechanism that can be dynamically adjusted according to the environment. For example, an attention-based module can be designed. This module can receive the environmental state context (such as the weather type determined by the system) and various modal features, and dynamically calculate the confidence level or fusion weight of each modal feature in the current scene. In this way, when perception conditions deteriorate (such as in heavy fog), the system can adaptively increase its reliance on less disturbed sensors (such as millimeter-wave radar), thereby maintaining the overall robustness of the perception system.
[0050] Step 5: Multi-task collaborative learning and decision generation
[0051] The Multi-Task Learning (MTL) framework is adopted to collaboratively complete perception, prediction and planning in a unified model.
[0052] 1. Model Architecture: Design a shared feature extraction backbone network, which then branches into multiple subnetworks for specific tasks.
[0053] 2. Task Branches:
[0054] Perception and Detection Branch: Responsible for identifying and locating key targets in the environment such as vehicles, pedestrians, and obstacles.
[0055] Trajectory prediction branch: Based on historical time series information, predict the future short-term movement trajectory of surrounding dynamic targets.
[0056] Path and Behavior Planning Branch: Combining vehicle status, global objectives, perception results, and prediction information, it generates a safe, comfortable, and executable local driving path or behavior command.
[0057] 3. Training strategy: Through joint training, the shared network layers learn general feature representations that are valuable for different tasks, thereby improving the overall efficiency of the system and the consistency of decisions across different tasks.
[0058] Step 6: Compliance Verification and Enhanced Scenario Understanding
[0059] To ensure the legality and humanistic rationality of the decision-making process, an understanding and verification layer is added after the initial decision is generated.
[0060] 1. Embedded traffic rule knowledge: Establish a configurable traffic rule knowledge base and logically compare the decision results with it to promptly detect possible violations (such as speeding and illegal lane changes) and trigger corrections.
[0061] 2. Scene Semantic Understanding: Integrating specialized computer vision algorithms to identify traffic light states and interpret specific traffic signs. Simultaneously, by analyzing pedestrian postures and movement trends, the system infers their intentions. This high-level understanding serves as a crucial basis for decision-making, making vehicle behavior more intelligent and in line with societal expectations.
[0062] Step 7: Hierarchical Decision Making and Final Action Generation
[0063] The final driving strategy is typically generated and determined within a hierarchical and progressive framework.
[0064] 1. Behavioral Decision-Making Layer: Makes macro-level behavioral choices based on task planning and scenario understanding.
[0065] 2. Motion Planning Layer: Transforms the above behavioral goals into a specific future trajectory of the vehicle that takes into account dynamic constraints and time dimensions.
[0066] 3. Action selection and evaluation: Before final execution, candidate actions can be simulated in a virtual environment using simulation- or search-based algorithms (such as fast sampling or Monte Carlo tree search) to evaluate their safety risks and execution effects, thereby selecting the action with the lowest overall risk for execution.
[0067] (ii) Implementing the autonomous driving decision-making system on this platform
[0068] The implementation of this system requires coordination of hardware platforms and software architecture to meet the requirements of high performance and high security.
[0069] 1. Hardware platform components:
[0070] Sensor access unit: It needs to have an interface that supports multiple communication protocols (such as CANFD, vehicle Ethernet) to adapt to and quickly access different types of sensors.
[0071] Data Processing Unit: The core computing unit typically employs a heterogeneous architecture. The central processing unit (CPU) is responsible for control flow and general computation; a high-performance graphics processing unit (GPU) or a dedicated AI accelerator (such as an NPU) is responsible for inference computation of deep learning models and is key to achieving real-time performance; for tasks such as low-level signal processing and sensor data fusion, hardware acceleration using a field-programmable gate array (FPGA) can be considered.
[0072] Execution and control unit: Typically runs a real-time operating system (RTOS) or a middleware framework with real-time capabilities, responsible for translating upper-level decisions into precise vehicle drive-by-wire commands (steering, braking, driving) and achieving reliable closed-loop control.
[0073] 2. Cloud-based collaboration and optimization:
[0074] By connecting to the cloud platform via 5G or C-V2X communication modules, and under the premise of strictly adhering to data security and privacy regulations, distributed machine learning technologies such as federated learning can be used to achieve continuous model optimization and experience sharing across the fleet, thereby improving the system's generalization ability in diverse scenarios.
[0075] 3. Functional Safety and System Design:
[0076] To achieve the high level of functional safety (such as ASILD level) required for autonomous driving, the system must be fully considered at the design and verification levels:
[0077] Security architecture: Employs a primary / backup redundancy or lockstep operation mechanism for computing units to ensure that a single failure will not lead to the loss of system functionality.
[0078] Security Mechanism: Implement multi-level software security monitoring measures, including program monitoring, memory protection, and data validity verification.
[0079] Safety State: Define a clear safety degradation mode (such as a minimum risk strategy) to ensure that the vehicle can autonomously enter a controllable safety state in the event of a system failure.
Claims
1. An autonomous driving decision-making platform for intelligent connected vehicles with multimodal fusion, characterized in that: Includes the following steps: First, visible light image data is collected using an onboard camera, three-dimensional point cloud information is obtained using a lidar, and target distance and velocity are detected using a millimeter-wave radar, forming a multimodal raw data stream; Secondly, the raw data is preprocessed, including image denoising, point cloud filtering, radar data calibration, and multi-sensor clock synchronization is achieved through hardware timestamps, and spatial alignment is completed using a calibration matrix. Next, the processed data is converted into a voxel grid representation, and then fed into a convolutional neural network to extract image features and a point cloud feature extraction network to process the 3D data, and finally mapped to a unified feature space. Then, an adaptive fusion module based on an attention mechanism is introduced to dynamically adjust the weights of each modality according to ambient light, weather conditions, etc., such as increasing the contribution of radar data in rainy and foggy weather. Then, the fused features are input into the multi-task learning model to simultaneously complete object detection, trajectory prediction and path planning. The object detection adopts the improved YOLOv5 algorithm, and the trajectory prediction is based on the LSTM network. Furthermore, the decision-making results are validated for compliance by combining a traffic rules knowledge base, and the traffic light status and pedestrian intentions are identified through a scenario understanding module. Finally, a hierarchical decision architecture is used to generate driving strategies, including lane changing, following, and emergency braking, and the feasibility of the actions is evaluated using a Monte Carlo tree search algorithm.
2. The intelligent connected vehicle multimodal fusion autonomous driving decision-making platform according to claim 1, characterized in that, The adaptive fusion module uses a gating mechanism to calculate the weights of each modality.
3. The intelligent connected vehicle multimodal fusion autonomous driving decision-making platform according to claim 1, characterized in that, The multi-task learning model adopts a structure of shared feature extraction layer and task-specific layer, and the loss function includes classification loss, regression loss and consistency loss.
4. The intelligent connected vehicle multimodal fusion autonomous driving decision-making platform according to claim 1, characterized in that, The scene understanding module integrates the OpenCV image processing library and a custom rule engine.
5. An autonomous driving decision-making system implementing the method of claim 1, characterized in that, The system includes: Multi-sensor interface unit, supporting multiple communication protocols such as CAN bus and Ethernet; The data preprocessing unit is equipped with an FPGA acceleration chip to achieve real-time filtering. The feature extraction unit is deployed on a GPU computing platform; The decision-making and execution unit integrates the ROS operating system and custom middleware. The cloud-based collaborative unit interacts with the vehicle-to-everything (V2X) platform via a 5G network.
6. The intelligent connected vehicle multimodal fusion autonomous driving decision-making platform according to claim 1, characterized in that, The cloud-based collaborative unit employs a federated learning framework to achieve continuous model optimization while ensuring data privacy.
7. The intelligent connected vehicle multimodal fusion autonomous driving decision-making platform according to claim 1, characterized in that, The system has passed ISO26262ASIL-D functional safety certification and has a dual-redundant computing architecture and a fault injection testing mechanism.