Intelligent Electric Multi-functional Vehicle Control System and Method Based on Multimodal Perception Fusion

By using multimodal perception fusion and adaptive cooperative control, the problems of perception reliability and robustness of electric multi-functional vehicles in harsh environments have been solved, enabling efficient decision-making and safe operation in complex scenarios.

CN123113309APending Publication Date: 2026-11-17SHANXI FORESIGHT STAR MEDIA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611033609.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-13
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing electric multi-functional vehicle control systems suffer from poor perception reliability in harsh environments such as low light and rain/fog. Their multi-sensor data fusion methods are simplistic, lacking dynamic consideration of the data quality and task relevance of different modalities. Their fixed control strategies result in insufficient adaptability and robustness.

Method used

It employs a multimodal sensing unit, an AI decision engine, an adaptive collaborative control unit, and an intelligent power management unit, and achieves adaptive adjustment of control strategies through multimodal sensing fusion and dynamic weighting mechanisms.

Benefits of technology

It improves perception integrity and decision continuity in harsh environments, reduces the risk of misjudgment and missed detection, and enhances the environmental adaptability and operational safety of electric multi-functional vehicles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123113309A_ABST
    Figure CN123113309A_ABST
Patent Text Reader

Abstract

This invention discloses a control system and method for an intelligent electric multi-functional vehicle based on multimodal perception fusion, relating to the field of intelligent vehicle control technology. The system includes a multimodal perception unit, an AI decision engine, an adaptive collaborative control unit, an execution unit, an intelligent power management unit, and an adaptive environmental adaptability module. The system adopts a multi-source heterogeneous perception data fusion scheme, and dynamically adjusts the fusion ratio of each sensor in real time through a dynamic weight formula. Under harsh working conditions, it automatically reduces the weight of disturbed modes and increases the weight of reliable sensors. The AI ​​decision engine is equipped with multiple deep learning and filtering algorithms, and adaptive collaborative control realizes the linkage of the walking mechanism, the on-board robotic arm, and the liftable roof. It is equipped with a dedicated signal conditioning and hierarchical power recovery circuit, which solves the defects of existing vehicle models such as single sensing, fixed fusion logic, and poor environmental robustness. It is suitable for electric wheelchairs, service mobility vehicles, and indoor multi-functional work vehicles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle control technology, and in particular to an intelligent electric multi-functional vehicle control system and method based on multimodal perception fusion. Background Technology

[0002] With the rapid development of artificial intelligence and sensor technology, electric multi-functional vehicles (such as electric wheelchairs, service robots, and automated guided vehicles) are increasingly being used in fields such as healthcare, logistics, and home services. Existing vehicle control systems typically rely on a single sensor (such as vision or lidar alone) for environmental perception, and their decision-making logic uses fixed threshold control rules, making it difficult to adapt to complex and changing environments.

[0003] The existing technology has the following shortcomings: In harsh environments such as low light, rain, and fog, the reliability of a single sensing mode is poor. Multi-sensor data fusion methods are simple (such as direct overlay) and lack dynamic consideration of the quality of data from different modalities and their relevance to the task. The control strategy is fixed and cannot adjust the perception weights and execution strategies according to the complexity of the environment, resulting in insufficient adaptability and robustness.

[0004] Therefore, there is an urgent need for an intelligent control system that can efficiently integrate multimodal sensing information and adaptively adjust the control strategy. Summary of the Invention

[0005] The purpose of this invention is to propose a control system and method for intelligent electric multi-functional vehicles based on multimodal perception fusion in order to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A control system for intelligent electric multi-functional vehicles based on multimodal perception fusion. It includes a multimodal perception unit, an AI decision engine, an adaptive collaborative control unit, an execution unit, and an intelligent power management unit; in, The multimodal perception unit includes a visual detection module, a speech recognition module, an environmental sensor module, and a user behavior recognition module; The AI ​​decision engine is based on a deep learning model and generates the optimal control strategy based on multimodal perception data. The adaptive collaborative control unit is based on a dynamic weighting mechanism to achieve efficient collaboration between the roof structure, the walking structure, and the robotic arm.

[0007] Preferably, the AI ​​decision engine includes a visual feature extraction network, a speech semantic analysis network, an environmental data processing network, and a user behavior analysis network; The visual feature extraction network uses an improved YOLOv5 model, the speech semantic analysis network uses a Transformer model combined with voiceprint recognition technology, the environmental data processing network uses an adaptive Kalman filter algorithm, and the user behavior analysis network uses an LSTM time series model.

[0008] Preferably, the adaptive collaborative control unit is based on a dynamic weighting mechanism, and the weighting calculation formula is: W_i=α*S_i+β*C_i+γ*E_i, where S_i is the quality of the perceived data, C_i is the relevance of the task, E_i is the environmental complexity, and α, β, and γ are weighting coefficients that are dynamically adjusted according to the environment.

[0009] Preferably, the visual detection module in the multimodal perception unit uses a panoramic camera, the voice recognition module uses a four-microphone array, and the environmental sensor module includes a height detection module, a center of gravity detection module, and a distance detection module.

[0010] Preferably, the signal processing circuit includes a differential amplifier, an adaptive low-pass filter, a programmable gain amplifier, and an ADC converter; The differential amplifier uses the AD8628, and the adaptive low-pass filter is designed based on the FIR filter.

[0011] Preferably, the power management circuit includes a multi-stage DC-DC converter, an energy recovery circuit, and a battery management module; wherein, the first-stage DC-DC converter adopts TI LM5164, and the energy recovery circuit adopts TI TPS54331.

[0012] Preferably, the system is also equipped with an adaptive environment adjustment module.

[0013] The control method for intelligent electric multi-functional vehicles based on multimodal perception fusion includes the following steps: a. Acquire multi-source environmental and user behavior data through a multimodal sensing unit; b. Generate the optimal overall machine control strategy through an AI decision engine; c. Achieve coordinated scheduling of all mechanisms in the machine through an adaptive and coordinated control unit; d. The mechanical movement and walking motion are achieved through the execution unit; e. Dynamically optimize overall power consumption through the intelligent power management unit.

[0014] In step a, the multimodal perception unit fuses data from various modalities through an adaptive attention mechanism.

[0015] In step c, the adaptive collaborative control unit dynamically adjusts the weights of each functional module according to environmental and task requirements.

[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention integrates visual, speech, environmental, and user behavior data through a multimodal perception unit and introduces an adaptive collaborative control mechanism based on dynamic weights. This mechanism can adjust the fusion weights of each modality in real time according to environmental complexity, perception data quality, and task relevance. Under adverse conditions such as low light, rain, fog, strong noise, or sudden road surface changes, the system automatically reduces the contribution of the interfered modalities while enhancing the weights of high-reliability modalities (such as infrared ranging and ultrasonic sensors) to prevent the failure of a single sensor from causing the entire vehicle to lose control.

[0017] 2. This invention dynamically optimizes process noise and observation noise parameters using an adaptive Kalman filter algorithm, ensuring stable and reliable height, center of gravity, and distance measurement data. Environmental adaptability control continuously adjusts the chassis height adjustment frequency, speed limits, and obstacle avoidance strategies through closed-loop feedback, enabling the vehicle to smoothly handle slope changes, slippery road surfaces, and dynamic obstacles. Compared to existing fixed fusion strategies, this invention significantly improves the perception integrity and decision-making continuity of electric multi-functional vehicles in complex indoor and outdoor scenarios, reducing the risk of misjudgment and missed detection, thereby greatly enhancing the vehicle's environmental adaptability and operational safety. Attached Figure Description

[0018] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a schematic diagram of the control system architecture of the present invention; Figure 2 This is a flowchart of the multimodal sensor fusion algorithm of the present invention; Figure 3 This is a schematic diagram of the adaptive cooperative control mechanism of the present invention; Figure 4 This is a schematic diagram of the signal processing circuit structure of the present invention; Figure 5 This is a schematic diagram of the power management circuit structure of the present invention; Figure 6 This is a schematic diagram of the electromagnetic compatibility layout of the entire PCB of the present invention; Figure 7 This is a flowchart illustrating the automatic adjustment of adaptive environmental parameters according to the present invention. Detailed Implementation

[0019] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0020] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0021] Example 1

[0022] Its specific implementation method is combined with the appendix Figure 1 To be continued Figure 7 Please provide a detailed explanation.

[0023] In this embodiment, it includes: Multimodal sensing unit implementation Visual inspection module: Employs a 2-megapixel panoramic camera (model: OV2710), supporting 360° environmental monitoring and equipped with an infrared fill light (850nm wavelength) to ensure normal operation in low-light conditions. The camera connects to the main control module via a USB 3.0 interface, achieving a frame rate of up to 60fps.

[0024] Voice recognition module: Employs a high-sensitivity microphone array (4 microphones, model: SPH0645LM4H), supporting voice command recognition and emotion analysis. The microphone array is arranged in a ring, with an array diameter of 10cm and a single microphone spacing of 5cm, achieving 360° sound source localization. Built-in echo cancellation and adaptive noise reduction algorithms (based on adaptive filtering algorithms) improve the overall signal-to-noise ratio by 15dB.

[0025] The environmental sensor module includes a height detection module (based on ultrasonic + infrared dual-mode), a center of gravity detection module (based on an MPU6050 six-axis sensor), a distance measurement module (based on lidar + ultrasonic fusion), and an ambient light sensor (BH1750). All sensors are connected to the main control module via an I2C interface.

[0026] User behavior recognition module: Analyzes user intent and emotions through camera and sensor data; video sampling frame rate of 30fps, relying on OpenPose algorithm to complete human posture analysis; equipped with IMU (MPU6050) and seat pressure sensor to collect human sitting posture data.

[0027] AI decision engine implementation The AI ​​decision engine employs deep learning models, including: 1. Visual Feature Extraction Network: An improved YOLOv5 model is used, including a Feature Pyramid Network (FPN). Adaptive Channel Attention (ACAM) is added to enhance small object detection capabilities. ACAM learns the correlation between channels and adaptively adjusts the weights of each channel, thereby improving feature representation.

[0028] 2. Speech Semantic Analysis Network: Utilizing the Transformer model, this network combines voiceprint recognition technology with sentiment analysis algorithms. Voiceprint recognition employs a ResNet34 network to extract voiceprint features; sentiment analysis uses a Multilayer Perceptron (MLP) to combine speech features (MFCC, fundamental frequency, etc.) and facial expression features.

[0029] 3. Environmental Data Processing Network: An adaptive Kalman filter algorithm is used to dynamically filter altitude, center of gravity, and distance measurement data. The Kalman filter parameters (process noise Q, observation noise R) are dynamically adjusted according to the environment to improve data reliability.

[0030] 4. User Behavior Analysis Network: This network uses an LSTM time-series model to analyze user behavior data and predict user intent. The LSTM network takes time-series behavior data as input and outputs an intent probability distribution.

[0031] Implementation of Adaptive Cooperative Control Mechanism like Figure 3 As shown, the adaptive cooperative control mechanism of this invention is based on dynamic weights, dynamically adjusting the weights of each functional module according to environmental and task requirements. The weight calculation formula is: W_i = α*S_i + β*C_i + γ*E_i; Where W_i is the weight of the i-th functional module, S_i is the quality of the perceived data, C_i is the task relevance, E_i is the environmental complexity, and α, β, and γ are weight coefficients that are dynamically adjusted according to the environment.

[0032] Weighting coefficient adjustment rules: α = 0.3 + 0.2 * (1E_i) / / Environmental complexity decreases, visual weight increases; β=0.4+0.3*C_i / / Task relevance increases, task weight increases; γ=0.3*E_i / / As environmental complexity increases, environmental weights also increase; Implementation of Multimodal Perception Fusion Algorithm like Figure 2 As shown, the multimodal sensor fusion algorithm of the present invention includes the following steps: 1. Extract features from each modality of data; Visual features: extracted using an improved YOLOv5 model; Speech features: extracted using the Transformer model; Environmental characteristics: extracted using Kalman filtering; User behavior features: extracted using an LSTM model; 2. Calculate the weights of each modality using an adaptive attention mechanism. Calculate S_i (perceived data quality), C_i (task relevance), and E_i (environmental complexity); Calculate the weight W_i=α*S_i+β*C_i+γ*E_i; 3. Fuse the weighted features Weighted average fusion is used: F = Σ(W_i * F_i); 4. Generate environmental perception results Control strategies are generated using an AI decision engine; Implementation methods for signal processing circuits; like Figure 4 As shown, the signal processing circuit of the present invention includes: 1. Differential amplifier (AD8628): 10MΩ input impedance, adjustable gain (1~100 times), improving signal anti-interference capability; 2. Adaptive low-pass filter (based on FIR filter): adjustable cutoff frequency (1Hz~100Hz), filtering out high-frequency noise; 3. Programmable gain amplifier (PGA): Gain range 0~60dB, optimizing signal amplitude; 4. ADC converter (ADS1115): 16-bit resolution, 860Hz sampling rate (ADS1115 standard parameters), converts analog signals into digital signals; Power management circuit implementation like Figure 5 As shown, the power management circuit of the present invention includes: 1.220VAC to 24VDC (TI LM5164), conversion efficiency 96.5%; 2.12VDC / 5VDC converter (TI LM2596): Input 12V, output 5V, conversion efficiency 94.3%; 3.5VDC / 3.3VDC converter (TI LM1117): Input 5V, output 3.3V, conversion efficiency 92.8%; 4. Braking energy recovery chip (TI TPS54331), with an average recovery efficiency of 78.5% during heavy-load braking; 5. Battery management module (BQ40Z80), compatible with 24V, 5000mAh lithium iron phosphate batteries, with overcharge and over-discharge protection.

[0033] Adaptive Environment Adaptive Implementation like Figure 7 As shown, the adaptive environment adaptability of the present invention includes the following steps: 1. Collect environmental data such as illumination, rain and snow, and road slope through a multimodal sensing unit; 2. Analyze environmental data and classify and determine the complexity of the environment; Lighting conditions: determined by an ambient light sensor; Weather conditions: determined through visual and environmental sensor data; Road surface conditions: determined by distance and height detection modules; 3. Automatically adjust sensing parameters and control strategies based on environmental complexity; For example, in rainy or foggy weather, increase the weight of the infrared ranging module / ultrasonic module and decrease the weight of the vision module; For example, when the road surface slope is greater than 5°, increase the frequency of chassis lifting and lowering adjustments; 4. Implement the adjusted control strategy; 5. Continuously optimize the control strategy through closed-loop feedback.

[0034] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0035] It should be noted that, in this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0036] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0037] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0038] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A control system for an intelligent electric multi-functional vehicle based on multimodal perception fusion, characterized in that, It includes a multimodal perception unit, an AI decision engine, an adaptive collaborative control unit, an execution unit, and an intelligent power management unit; in, The multimodal perception unit includes a visual detection module, a speech recognition module, an environmental sensor module, and a user behavior recognition module; The AI ​​decision engine is based on a deep learning model and generates the optimal control strategy based on multimodal perception data. The adaptive collaborative control unit is based on a dynamic weighting mechanism to achieve efficient collaboration between the roof structure, the walking structure, and the robotic arm.

2. The intelligent electric multi-functional vehicle control system based on multi-modal perception fusion according to claim 1, characterized in that, The AI ​​decision engine includes a visual feature extraction network, a speech and semantic analysis network, an environmental data processing network, and a user behavior analysis network. The visual feature extraction network uses an improved YOLOv5 model, the speech semantic analysis network uses a Transformer model combined with voiceprint recognition technology, the environmental data processing network uses an adaptive Kalman filter algorithm, and the user behavior analysis network uses an LSTM time series model.

3. The intelligent electric multi-functional vehicle control system based on multi-modal perception fusion according to claim 1, characterized in that, The adaptive collaborative control unit is based on a dynamic weighting mechanism. The weighting calculation formula is: W_i=α*S_i+β*C_i+γ*E_i, where S_i is the quality of the perceived data, C_i is the relevance of the task, E_i is the environmental complexity, and α, β, and γ are weighting coefficients that are dynamically adjusted according to the environment.

4. The intelligent electric multi-functional vehicle control system based on multi-modal perception fusion according to claim 1, characterized in that, The visual detection module in the multimodal perception unit uses a panoramic camera, the voice recognition module uses a four-microphone array, and the environmental sensor module includes a height detection module, a center of gravity detection module, and a distance measurement detection module.

5. The intelligent electric multi-functional vehicle control system based on multi-modal perception fusion according to claim 1, characterized in that, The signal processing circuit includes a differential amplifier, an adaptive low-pass filter, a programmable gain amplifier, and an ADC converter; The differential amplifier uses the AD8628, and the adaptive low-pass filter is designed based on the FIR filter.

6. The intelligent electric multi-functional vehicle control system based on multi-modal perception fusion according to claim 1, characterized in that, The power management circuit includes a multi-stage DC-DC converter, an energy recovery circuit, and a battery management module; the first-stage DC-DC converter uses the TI LM5164, and the energy recovery circuit uses the TI TPS54331.

7. The intelligent electric multi-functional vehicle control system based on multi-modal perception fusion according to claim 1, characterized in that, It also features an adaptive environment adjustment module, which can automatically adjust sensing parameters and control strategies according to environmental conditions.

8. The intelligent electric multi-functional vehicle control method based on multi-modal perception fusion, according to any one of claims 1-7, wherein Includes the following steps: a. Acquire multi-source environmental and user behavior data through a multimodal sensing unit; b. Generate the optimal overall machine control strategy through an AI decision engine; c. Achieve coordinated scheduling of all mechanisms in the machine through an adaptive and coordinated control unit; d. The mechanical movement and walking motion are achieved through the execution unit; e. Dynamically optimize overall power consumption through the intelligent power management unit. 9.The intelligent electric multi-functional vehicle control method based on multi-modal perception fusion according to claim 8, characterized in that, In step a, the multimodal perception unit fuses data from various modalities through an adaptive attention mechanism. 10.The intelligent electric multi-functional vehicle control method based on multi-modal perception fusion of claim 8, wherein, In step c, the adaptive collaborative control unit dynamically adjusts the weights of each functional module according to environmental and task requirements.