Intelligent driving multi-modal information dynamic adaptation system and method

Abstract

The application discloses a kind of intelligent driving multi-modal information dynamic adaptation system and method.The system includes perception hardware layer, decision controller, execution interaction layer, localization adaptation memory.Perception hardware layer is through mixed 3D convolution network to the space-time alignment and semantic fusion of multi-source sensor data, output 256-dimensional context feature vector.Decision controller is equipped with Stackelberg equilibrium solver and deep reinforcement learning inference chip, and the automatic driving system, driver and environment are modeled as three-party game, and the optimal strategy benchmark is solved by parallel alternating direction multiplier method, and visual, auditory, tactile weight distribution instruction is dynamically generated by PPO algorithm.Execution interaction layer drives AR-HUD, 3D surround sound and seat vibration by priority arbitration circuit, and suppresses multi-modal signal conflict.Localization adaptation memory realizes the adaptation of cross-area traffic rules and driving habits using federated learning and differential privacy.The application realizes the collaborative optimization of safety, comfort and localization adaptation.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving and in-vehicle human-machine interaction technology. Specifically, it relates to an intelligent driving multimodal information dynamic adaptation system and method, which is used to dynamically adjust the priority and weight of visual, auditory and tactile feedback in complex driving environments to balance system safety requirements, driver cognitive load and environmental complexity. Background Technology

[0002] With the development of Advanced Driver Assistance Systems (ADAS) and autonomous driving technologies, modern vehicles are generally equipped with multi-source sensors and environmental perception systems, which can generate a large amount of driving situation information in real time. At the same time, the in-vehicle human-machine interaction system provides feedback to the driver through various modalities such as the instrument panel, central control screen, head-up display, voice broadcast, and tactile vibration.

[0003] However, existing intelligent driving information interaction systems still have common problems in practical applications. Patent publication number (CN121572902A) discloses an intelligent cockpit interaction priority dynamic allocation system, method, and electronic device. This solution includes: a multimodal signal input layer for continuously receiving raw interaction command signals from a voice recognition module, gesture recognition module, eye-tracking module, and touchscreen module; a context awareness layer for parallel acquisition of vehicle status data, driver status data, and environmental status data; a dynamic priority orchestration engine, including a multi-factor fusion and weight calculation module and an arbitration and scheduling module; the multi-factor fusion and weight calculation module normalizes context data into weighted influence factors and calculates the real-time dynamic priority score for each interaction modality; the arbitration and scheduling module compares priority scores and selects the optimal execution modality while sending suppression signals to other modalities; and an instruction execution layer for receiving the final instruction from the arbitration and scheduling module and issuing it to the in-vehicle application for execution. This patent, by introducing a dynamic priority calculation and modality suppression mechanism, alleviates information conflict and driver cognitive burden in multimodal interaction scenarios to some extent.

[0004] However, the aforementioned existing technologies still have the following shortcomings: First, information overload, the system tends to push all sensory information to the driver without filtering, leading to distraction; second, rigid priorities, the system lacks the ability to dynamically adjust the information feedback priority according to the real-time driving situation, and critical safety information may be overwhelmed by low-priority information; third, multimodal signal conflict, when multiple feedback channels such as vision, hearing and touch are activated at the same time, there is a lack of effective conflict suppression mechanism, which may cause sensory overload or signal confusion for the driver; fourth, insufficient cross-scenario adaptability, the ability to adapt to special traffic rules and differences in driving habits across regions is limited. Summary of the Invention

[0005] This invention aims to provide a dynamic adaptation system and method for multimodal information in intelligent driving. It achieves high-precision cross-modal context perception through a hybrid 3D convolutional network, realizes dynamic priority decision-making through a Stackelberg equilibrium model, and achieves real-time optimization and conflict suppression of multimodal feedback weights through deep reinforcement learning.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A dynamic adaptation system for multimodal information in intelligent driving includes a perception hardware layer 1, a decision controller 2, an execution interaction layer 3, and a localized adaptation memory 4. The signal output terminal of the perception hardware layer 1 is connected to the data input terminal of the decision controller 2 via an in-vehicle Ethernet bus. The output terminal of the decision controller 2 is connected to the input terminal of the execution interaction layer 3. The localized adaptation memory 4 is connected to the decision controller 2 and the perception hardware layer 1 via a PCIe bus.

[0007] Furthermore, the output of the perception hardware layer 1 transmits the context feature vector signal to the data input of the decision controller 2; the output of the decision controller 2 transmits the dynamic feedback allocation instruction signal to the input of the execution interaction layer 3; the localized adaptation memory 4 transmits the game weight bias signal to the decision controller 2 through the PCIe bus, and transmits the model fine-tuning signal to the processing unit parameter update interface of the perception hardware layer 1.

[0008] Furthermore, the perception hardware layer 1 includes: a multi-source sensor group, a spatiotemporal alignment unit, and a hybrid 3D convolutional network processor; the multi-source sensor group includes a camera, LiDAR, a V2X communication module, a driver biosensor, and a steering wheel / pedal operation sensor; the input of the spatiotemporal alignment unit is connected to the multi-source sensor group, performing timestamp synchronization and spatial coordinate system mapping; the input of the hybrid 3D convolutional network processor is connected to the output of the spatiotemporal alignment unit, and the processor internally includes parallel residual network operation circuits, point transformer network operation circuits, long short-term memory network operation circuits, and cross-attention logic circuits for fusing the output features of the three, and the output context feature vector signal is a 256-dimensional feature vector, which includes obstacle type encoding, road friction coefficient parameters, and driver distraction probability values.

[0009] Furthermore, the decision controller 2 includes: a Stackelberg equilibrium solver and a deep reinforcement learning inference chip; the Stackelberg equilibrium solver uses a dedicated integrated circuit with firmware programmed with parallelized alternating direction multiplier method to solve the equilibrium point of the three-party game, taking the autonomous driving system, driver and environment as game participants, and outputting the optimal strategy benchmark signal that balances the system safety requirements, driver cognitive load and environmental complexity.

[0010] Furthermore, the deep reinforcement learning inference chip is a neural network processor based on the proximal policy optimization algorithm. Its internal registers define a normalized action space for visual weights, auditory weights, and tactile weights. The reward function operation unit of the chip is equipped with a weighted summation circuit based on safety score, comfort score, context fit score, and interaction efficiency score.

[0011] Furthermore, the execution interaction layer 3 includes: a multimodal signal distribution circuit, an AR-HUD display, a 3D surround sound speaker array, and a seat haptic feedback motor array; the input of the multimodal signal distribution circuit serves as the control port of the execution interaction layer 3, and the output of the multimodal signal distribution circuit is connected to the AR-HUD display, the 3D surround sound speaker array, and the seat haptic feedback motor respectively; the multimodal signal distribution circuit also includes a priority arbitration logic circuit, which includes four priority buffer queues (urgent, high, medium, and low) and corresponding timing scheduling logic gates to queue and suppress concurrent feedback instructions.

[0012] Furthermore, the priority arbitration logic circuit also includes a resource exclusive logic unit. When the priority arbitration logic circuit receives a control signal representing an emergency braking event, the logic unit sends an exclusive enable level signal to the output channel of the 3D surround sound speaker array or the output channel of the seat haptic feedback motor, and blocks the control signal corresponding to non-emergency events.

[0013] Furthermore, the dynamic feedback allocation command signal includes visual weight components, auditory weight components, and tactile weight components; the multimodal signal allocation circuit responds to the dynamic feedback allocation command signal by outputting a visual drive level signal to the AR-HUD display, an auditory drive level signal to the 3D surround sound speaker array, and a tactile drive level signal to the seat tactile feedback motor; when the priority corresponding to the dynamic feedback allocation command signal is an emergency level, the priority arbitration logic circuit activates the resource exclusive logic unit to shield non-emergency feedback output.

[0014] Furthermore, the localized adaptation memory 4 includes: a regional traffic regulations database, a transfer learning engine, and a policy transfer weight configuration table; the transfer learning engine uses a federated learning aggregation algorithm and a differential privacy noise addition mechanism to generate model fine-tuning signals sent to the perception hardware layer 1; the policy transfer weight configuration table stores environmental game weight bias values ​​corresponding to different traffic rule labels and generates game weight bias signals sent to the decision controller 2.

[0015] This invention also provides a method for dynamic adaptation of multimodal information in intelligent driving, comprising the following steps: S1: Obtain raw sensing data through sensing hardware layer 1, and generate context feature vector signals after spatiotemporal alignment and hybrid 3D convolution processing; S2: Input the context feature vector signal into the decision controller 2, use the Stackelberg equilibrium solver to calculate the policy guidance signal, and use the deep reinforcement learning inference chip to generate a multimodal feedback allocation instruction containing visual, auditory and tactile weights in combination with the composite reward function. S3: The multimodal signal distribution circuit of the interaction layer 3 drives the AR-HUD display, 3D surround sound speaker array and seat haptic feedback motor to perform corresponding actions according to the multimodal feedback distribution instructions, and performs conflict suppression through the priority arbitration logic circuit when an emergency event is detected. S4: The localized adaptation memory 4 updates the model parameters of the perception hardware layer 1 according to the real-time geographical location of the vehicle through the transfer learning engine, and adjusts the weights of the game participants in the decision controller 2 through the policy transfer weight configuration table.

[0016] Compared with traditional solutions, the present invention has the following advantages: (1) This invention achieves semantic-level fusion of multi-source heterogeneous data through spatiotemporal alignment technology and hybrid 3D convolutional network, and the output 256-dimensional feature vector provides a robust and comprehensive contextual expression for decision-making.

[0017] (2) This invention introduces the Stackelberg equilibrium model into vehicle human-machine interaction decision-making, dynamically balances the game relationship between the autonomous driving system, the driver and the environment, solves the defect of fixed priority in traditional systems, and can automatically adjust information feedback strategies in complex scenarios such as driver distraction and bad weather.

[0018] (3) This invention achieves millisecond-level dynamic allocation of visual, auditory and tactile weights through the hardware and software synergy of deep reinforcement learning algorithm and priority arbitration circuit, thereby reducing the cognitive burden on the driver.

[0019] (4) This invention combines federated learning, differential privacy and policy transfer mechanisms to enable vehicles to quickly adapt to the special traffic rules and driving habits of different countries / regions without disclosing user privacy, which greatly improves the global universality and practicality of the system. Attached Figure Description

[0020] This manual includes the following figures, which illustrate the following: Figure 1 This is a hardware block diagram of the intelligent driving multimodal information dynamic adaptation system of the present invention; Figure 2 This is an overall flowchart of the method provided by the present invention; The components are: 1. Perception hardware layer; 2. Decision controller; 3. Execution interaction layer; 4. Localized adaptation memory. Detailed Implementation

[0021] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, in order to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention, and to facilitate its implementation.

[0022] like Figure 1 As shown, the present invention provides an intelligent driving multimodal information dynamic adaptation system, which consists of a perception hardware layer 1, a decision controller 2, an execution interaction layer 3, and a localized adaptation memory 4.

[0023] The signal output of the perception hardware layer 1 is connected to the data input of the decision controller 2 via the vehicle Ethernet bus. In this embodiment, the vehicle Ethernet uses the 1000BASE-T1 standard supporting time-sensitive networking to ensure high bandwidth, low latency, and high deterministic data transmission performance. The output of the decision controller 2 is electrically connected to the input of the execution interaction layer 3 via the CAN bus. The localized adaptation memory 4 uses automotive-grade UFS 3.1 flash memory chips and is connected to the parameter configuration interface of the decision controller 2 and the parameter update interface of the processing unit inside the perception hardware layer 1 via the PCIe bus.

[0024] like Figure 1 As shown, the sensing hardware layer 1 includes a multi-source sensor group, a spatiotemporal alignment unit, and a hybrid 3D convolutional neural network processor.

[0025] The multi-source sensor group includes the following sensors: an RGB camera for capturing color images and depth information of the road scene in front of the vehicle; a LiDAR for acquiring 3D point cloud data of a 360° range around the vehicle; a V2X communication module for receiving vehicle-to-everything (V2X) information such as traffic light phases and remote vehicle motion status broadcast by roadside units; a set of driver biosensors, including an infrared pupil-tracking eye tracker mounted above the dashboard and a capacitive heart rate monitoring sensor integrated into the steering wheel, for real-time detection of the driver's gaze coordinates, pupil diameter changes, and heart rate variability; and steering wheel angle / grip sensor, accelerator pedal position sensor, and brake pedal travel sensor for monitoring the degree of driver intervention.

[0026] The spatiotemporal alignment unit is implemented using an FPGA chip. Its input is connected to the output of each sensor in the multi-source sensor group. The internal logic circuit performs two key processes: first, time synchronization, which marks each frame of sensor data with a unified master clock timestamp. After hardware-level synchronization, the time deviation between the multi-source data streams is controlled within 5 milliseconds; second, spatial fusion, which uses the extrinsic parameter rotation and translation matrix obtained in advance through joint calibration to map the three-dimensional coordinates of the LiDAR point cloud to the camera image coordinate system, generating pixel-by-pixel aligned fused data frames.

[0027] The hybrid 3D convolutional neural network processor is implemented using automotive-grade neural network processing units. Its input is connected to the output of the spatiotemporal alignment unit, and internally contains three parallel feature extraction paths and a top-level fusion module. The first path is a residual network operation circuit, which extracts visual image data based on the ResNet-50 structure, outputting a 2048-dimensional image semantic feature vector. The second path is a point transformer network operation circuit, which directly processes the original point cloud through a vector self-attention mechanism, outputting a 1024-dimensional point cloud geometric feature vector. The third path is a long short-term memory network operation circuit, which processes time-series data output from the V2X communication module and the driver's biosensor, with a hidden state dimension of 512. These three feature paths are input in parallel to a cross-attention logic circuit, which adaptively weights and fuses the features by calculating the attention weight matrix between different modalities. The fused comprehensive features are then compressed and dimensionality-reduced through a fully connected layer, finally outputting a 256-dimensional contextual feature vector signal. This feature vector includes obstacle type encoding, road friction coefficient parameters, and driver distraction probability values.

[0028] The decision controller 2 includes a Stackelberg equilibrium solver and a deep reinforcement learning inference chip.

[0029] The Stackelberg equilibrium solver is implemented using an FPGA-based application-specific integrated circuit with firmware programmed for parallelized alternating direction multiplier method (ADMM). Its core function is to establish a game model that treats the autonomous driving system, driver, and environment as players in the current driving scenario. The autonomous driving system is designated as the leader, initiating information feedback and control strategies; the driver and environment, as followers, make their optimal responses after observing the leader's strategy. The objective function of the game model comprehensively considers system safety requirements, driver cognitive load, and environmental complexity. Internally, the solver decomposes the original game problem into three sub-problems: AV strategy update, HD response, and E state transition. These are solved iteratively in parallel using the ADMM algorithm, outputting a baseline signal for the optimal control strategy of the autonomous driving system. This baseline signal includes the proposed braking deceleration value, lane keeping curvature, and the initial priority ranking of each modal feedback.

[0030] The deep reinforcement learning inference chip is a neural network processor based on the proximal policy optimization algorithm. Its input interface simultaneously receives a real-time updated 256-dimensional context feature vector signal and the optimal policy baseline signal output by the Stackelberg equalization solver. Internal registers define visual feedback weights Wv, auditory feedback weights Wa, and tactile feedback weights Wt, each ranging from [0, 1] and satisfying the constraint Wv + Wa + Wt = 1. The chip's built-in reward function hardware operation unit is equipped with a weighted summation circuit for the composite reward function, mathematically expressed as R = 0.5 × Safety + 0.3 × Comfort + 0.1 × Adapt + 0.1 × Efficiency. Within each decision cycle, the deep reinforcement learning inference chip calculates the current optimal weight allocation ratio in real-time using the PPO algorithm and generates a dynamic feedback allocation instruction signal containing the corresponding weight components. Simultaneously, when the Safety score falls below a preset threshold, such as 0.3, the chip's hardware logic automatically generates a conflict suppression control word, which is sent along with the instruction signal.

[0031] The interaction layer 3 includes a multimodal signal distribution circuit, an AR-HUD display, a 3D surround sound speaker array, and a seat haptic feedback motor.

[0032] The input of the multimodal signal distribution circuit is connected to the output of the decision controller 2 via a CAN bus. Its outputs are connected to an AR-HUD display via an LVDS video interface, a 3D surround sound speaker array via an A2B audio bus, and a seat haptic feedback motor via a PWM drive line. The multimodal signal distribution circuit integrates a priority arbitration logic circuit. When the dynamic feedback distribution command signal issued by the decision controller 2 arrives, the priority arbitration logic circuit first classifies the current event into one of four buffer queues: urgent, high, medium, and low, based on the priority field carried in the command. The timing scheduling logic gate polls the head of each queue in a fixed order: urgent first, high second, medium second, and low last. When outputting a priority command, it allocates a dedicated time window for that event. During this window, the output enable signal of low-priority commands is hardware-masked and temporarily stored in the queue awaiting subsequent scheduling. If the current command is at the emergency level and carries an emergency braking event control word, the embedded resource-exclusive logic unit will immediately trigger a hardware-level channel preemption action. It will send a mute enable signal to the audio channel of the 3D surround sound speaker array to forcibly interrupt any non-emergency voice prompts currently playing, send an exclusive enable level signal to the PWM drive channel of the seat haptic feedback motor to switch to emergency vibration mode, and simultaneously send a full-screen coverage command to the AR-HUD display to show a high-priority warning image. This ensures that the emergency event exclusively controls the driver's visual, auditory, and tactile sensory channels without interference at the first moment. After the emergency event is resolved, the resource-exclusive logic unit releases channel control, and the interrupted low-priority events resume execution sequentially according to their original queue order. This achieves orderly scheduling and deterministic conflict suppression of concurrent multimodal feedback signals.

[0033] During feedback execution, the multimodal signal distribution circuit analyzes the visual, auditory, and tactile weight components in the dynamic feedback distribution command signal and generates drive level signals accordingly. Specifically, when the visual weight component is higher than 0.7, a high-brightness red flashing visual drive level signal is output to the AR-HUD display; when the auditory weight component is high, the 3D surround sound speaker array is driven to output a directional warning tone of appropriate volume in the direction of the danger source; when the tactile weight component is high, the seat haptic feedback motor is driven to vibrate with a PWM waveform of a specific frequency and duty cycle.

[0034] The localized adaptation storage 4 is an automotive-grade solid-state storage medium, which includes a regional traffic regulations database, a transfer learning engine, and a policy transfer weight configuration table.

[0035] The regional traffic regulations database pre-stores specific traffic rule labels and corresponding numerical descriptions of rule parameters from multiple countries and regions worldwide, including but not limited to China's tidal flow lane rules, European bicycle priority rules, and Nordic winter driving rules. The transfer learning engine employs a federated learning aggregation algorithm and a differential privacy noise addition mechanism. When a vehicle detects via GPS that it has entered a new geofenced area, the system locally fine-tunes the parameter model of the hybrid 3D convolutional neural network processor using recently collected driving data. During gradient updates, differential privacy noise conforming to a Laplace distribution is added to the gradient, completing the localization adaptation of the perception model while protecting user privacy. The policy transfer weight configuration table stores the mapping relationship between different traffic rule labels and environmental game weight bias values. When a specific traffic rule label is retrieved, this table outputs the corresponding game weight bias value, which is sent to the parameter configuration interface of decision controller 2 via the PCIe bus to dynamically adjust the weight coefficients of environmental participants in the Stackelberg equilibrium solver, thereby achieving adaptive switching of cross-regional driving strategies.

[0036] like Figure 2 As shown, the intelligent driving multimodal information dynamic adaptation method based on the above system specifically includes the following steps S1 to S4.

[0037] S1: After the system powers on, the perception hardware layer 1 begins collecting data from multiple sensors at fixed intervals. For example, the camera outputs raw data streams continuously at 30Hz, the LiDAR at 20Hz, the V2X module at 10Hz, and the biosensor at 100Hz. The spatiotemporal alignment unit performs time synchronization and spatial registration on each frame of data, generating a fused data frame and sending it to the hybrid 3D convolutional neural network processor. The processor's residual network, point transformer network, and LSTM network extract features in parallel. After being fused by a cross-attention logic circuit, it outputs a 256-dimensional contextual feature vector signal containing obstacle type encoding, road friction coefficient parameters, and driver distraction probability values. This signal is transmitted in real time to the decision controller 2 via the vehicle Ethernet bus.

[0038] S2: Decision controller 2 receives the 256-dimensional situational feature vector signal. First, the Stackelberg equilibrium solver models the current scenario as a three-way non-cooperative game between the autonomous driving system, the driver, and the environment. It iteratively solves for the Stackelberg equilibrium point using a parallelized ADMM algorithm, outputting a baseline signal for the system's optimal control strategy. Subsequently, the deep reinforcement learning inference chip takes this baseline signal and the real-time situational feature vector as input. Based on the PPO algorithm and combined with the composite reward function R = 0.5 × Safety + 0.3 × Comfort + 0.1 × Adapt + 0.1 × Efficiency, it calculates the visual, auditory, and tactile weight components in real time, generating a dynamic feedback allocation instruction signal containing the corresponding weight components.

[0039] S3: The multimodal signal distribution circuit of the interaction layer receives the dynamic feedback distribution command signal via the CAN bus. The circuit analyzes the visual, auditory, and tactile weight components in the signal and outputs visual drive level signals to the AR-HUD display, auditory drive level signals to the 3D surround sound speaker array, and tactile drive level signals to the seat haptic feedback motor, respectively, to execute corresponding intensity of prompting actions. Simultaneously, the priority arbitration logic circuit monitors the command queue in real time. When an urgent priority event is detected, the resource exclusive logic unit is immediately activated, blocking non-urgent feedback outputs and exclusively controlling the sensory channel corresponding to the urgent event; otherwise, execution proceeds according to normal timing.

[0040] S4: During system operation, the localized adaptation memory 4 continuously monitors GPS geofence update events. When a vehicle is detected entering a new geographical area, the regional traffic rule database retrieves and outputs the currently valid traffic rule labels; the transfer learning engine performs federated fine-tuning of the model parameters of the perception hardware layer 1 based on local driving data, and updates the model parameters via the PCIe bus; the policy transfer weight configuration table reads the corresponding environmental game weight bias value according to the traffic rule label, and sends it to the parameter configuration interface of the decision controller 2 via the PCIe bus to adjust the weights of the game participants in the Stackelberg equilibrium solver, completing the adaptive policy adjustment of the system in cross-regional scenarios.

[0041] The intelligent driving multimodal information dynamic adaptation system and method provided in this embodiment have fully verified their technical effectiveness in real vehicle testing. Based on the 256-dimensional contextual feature vector output by the hybrid 3D convolutional neural network processor in the perception hardware layer, the system can accurately identify obstacle types, road friction coefficients, and driver distraction probabilities, providing high-precision, low-latency contextual perception input for upper-level decision-making.

[0042] At the decision-making level, the Stackelberg equilibrium solver completes the equilibrium solution of the three-party game through a parallelized alternating direction multiplier method, providing mathematically interpretable safety constraints and policy guidance for the deep reinforcement learning inference chip. Based on the proximal policy optimization algorithm and a composite reward function, the deep reinforcement learning inference chip achieves continuous dynamic allocation of visual, auditory, and tactile feedback weights, enabling multimodal interaction strategies to be smoothly adjusted according to real-time driving scenarios.

[0043] At the execution level, the priority arbitration logic circuit effectively solves the conflict problem when multimodal signals are concurrent through a four-level buffer queue and resource exclusivity mechanism. In emergency braking scenarios, it ensures that the tactile and auditory channels are exclusively occupied by the emergency event, which significantly reduces the driver's cognitive burden and reaction delay.

[0044] Furthermore, the localized adaptation memory, through a transfer learning engine and a policy transfer weight configuration table, enables vehicles to automatically increase environmental game weights and enhance auditory warning intensity when entering European bicycle priority zones, achieving seamless adaptation to cross-regional traffic rules. In summary, this embodiment fully realizes the organic integration of game theory, deep reinforcement learning, and transfer learning, ensuring driving safety while also considering driving comfort and the system's global applicability.

[0045] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution; or the direct application of the inventive concept and technical solution to other situations without modification, are all within the protection scope of the present invention.

Claims

1. A dynamic adaptation system for multimodal information in intelligent driving, characterized in that, It includes a perception hardware layer (1), a decision controller (2), an execution interaction layer (3), and a localized adaptation memory (4); the signal output terminal of the perception hardware layer (1) is connected to the data input terminal of the decision controller (2) through an in-vehicle Ethernet bus; the output terminal of the decision controller (2) is connected to the input terminal of the execution interaction layer (3); the localized adaptation memory (4) is connected to the decision controller (2) and the perception hardware layer (1) through a PCIe bus.

2. The intelligent driving multimodal information dynamic adaptation system according to claim 1, characterized in that, The output of the perception hardware layer (1) transmits the context feature vector signal to the data input of the decision controller (2); The output of the decision controller (2) transmits a dynamic feedback allocation instruction signal to the input of the execution interaction layer (3); the localized adaptation memory (4) transmits a game weight bias signal to the decision controller (2) via the PCIe bus, and transmits a model fine-tuning signal to the processing unit parameter update interface of the perception hardware layer (1).

3. The intelligent driving multimodal information dynamic adaptation system according to claim 2, characterized in that, The perception hardware layer (1) includes: a multi-source sensor group, a spatiotemporal alignment unit, and a hybrid 3D convolutional network processor; the multi-source sensor group includes a camera, a lidar, a V2X communication module, a driver biosensor, and a steering wheel / pedal operation sensor; the input end of the spatiotemporal alignment unit is connected to the multi-source sensor group to perform timestamp synchronization and spatial coordinate system mapping; the input end of the hybrid 3D convolutional network processor is connected to the output end of the spatiotemporal alignment unit, and the processor is equipped with parallel residual network operation circuits, point transformer network operation circuits, long short-term memory network operation circuits, and cross-attention logic circuits for fusing the output features of the three, and the output context feature vector signal is a 256-dimensional feature vector, which includes obstacle type encoding, road friction coefficient parameters, and driver distraction probability values.

4. The intelligent driving multimodal information dynamic adaptation system according to claim 1, characterized in that, The decision controller (2) includes: a Stackelberg equilibrium solver and a deep reinforcement learning inference chip; the Stackelberg equilibrium solver uses a dedicated integrated circuit with firmware for parallelized alternating direction multiplier method, takes the autonomous driving system, driver and environment as game participants, solves the equilibrium point of the three-party game, and outputs the optimal strategy benchmark signal that balances the system safety requirements, driver cognitive load and environmental complexity.

5. The intelligent driving multimodal information dynamic adaptation system according to claim 4, characterized in that, The deep reinforcement learning inference chip is a neural network processor based on the near-end policy optimization algorithm. Its internal registers define a normalized action space for visual weights, auditory weights, and tactile weights. The reward function operation unit of the chip is configured with a weighted summation circuit based on safety score, comfort score, context fit score, and interaction efficiency score.

6. The intelligent driving multimodal information dynamic adaptation system according to claim 1, characterized in that, The execution interaction layer (3) includes: a multimodal signal distribution circuit, an AR-HUD display, a 3D surround sound speaker array, and a seat haptic feedback motor array; the input terminal of the multimodal signal distribution circuit serves as the control port of the execution interaction layer (3), and the output terminal of the multimodal signal distribution circuit is connected to the AR-HUD display, the 3D surround sound speaker array, and the seat haptic feedback motor respectively; the multimodal signal distribution circuit also includes a priority arbitration logic circuit, which includes four priority buffer queues of emergency, high, medium, and low and corresponding timing scheduling logic gates to queue and suppress concurrent feedback instructions.

7. The intelligent driving multimodal information dynamic adaptation system according to claim 6, characterized in that, The priority arbitration logic circuit also includes a resource exclusive logic unit. When the logic unit receives a control signal representing an emergency braking event, it sends an exclusive enable level signal to the output channel of the 3D surround sound speaker array or the output channel of the seat haptic feedback motor, and blocks the control signal corresponding to non-emergency events.

8. The intelligent driving multimodal information dynamic adaptation system according to claims 6-7, characterized in that, The dynamic feedback allocation command signal includes visual weight components, auditory weight components, and tactile weight components; the multimodal signal allocation circuit responds to the dynamic feedback allocation command signal by outputting a visual drive level signal to the AR-HUD display, an auditory drive level signal to the 3D surround sound speaker array, and a tactile drive level signal to the seat tactile feedback motor. When the priority arbitration logic circuit corresponds to the urgent level of the dynamic feedback allocation instruction signal, it activates the resource exclusive logic unit to shield non-urgent feedback output.

9. The intelligent driving multimodal information dynamic adaptation system according to claim 2, characterized in that, The localized adaptation memory (4) includes: a regional traffic regulations database, a transfer learning engine, and a policy transfer weight configuration table; the transfer learning engine firmware uses a federated learning aggregation algorithm and a differential privacy noise addition mechanism to generate a model fine-tuning signal sent to the perception hardware layer (1); the policy transfer weight configuration table stores the environmental game weight bias values ​​corresponding to different traffic rule labels and generates a game weight bias signal sent to the decision controller (2).

10. An adaptation method for a dynamic adaptation system for multimodal information in intelligent driving according to any one of claims 1-9, characterized in that, Includes the following steps: S1: Obtain the original sensing data through the sensing hardware layer (1), and generate the context feature vector signal after spatiotemporal alignment and hybrid 3D convolution processing; S2: Input the situation feature vector signal into the decision controller (2), use the Stackelberg equilibrium solver to calculate the strategy guidance signal, and use the deep reinforcement learning inference chip to generate a multimodal feedback allocation instruction containing visual, auditory and tactile weights in combination with the composite reward function; S3: The multimodal signal distribution circuit of the execution interaction layer (3) drives the AR-HUD display, 3D surround sound speaker array and seat haptic feedback motor to perform corresponding actions according to the multimodal feedback distribution instruction, and performs conflict suppression through the priority arbitration logic circuit when an emergency event is detected; S4: The localized adaptation memory (4) updates the model parameters of the perception hardware layer (1) through the transfer learning engine according to the real-time geographical location of the vehicle, and adjusts the weights of the game participants in the decision controller (2) through the policy transfer weight configuration table.

Images