Simulation driving platform man-machine interaction system in intelligent network connection environment

Abstract

The invention discloses a man-machine interaction system for simulating a driving platform in an intelligent network connection environment. The man-machine interaction system comprises a network connection information perception and fusion module; a driver state and intention identification module; a multi-modal interaction decision and generation module; according to the man-machine interaction system for simulating the driving platform in the intelligent network connection environment, by constructing the network connection information perception and fusion module, real-time receiving, semantic analysis and space-time alignment of multi-source heterogeneous network connection information such as vehicle-road cooperation and inter-vehicle communication are achieved; and a dynamic environment cognition map rich in network connection environment characteristics is generated. Therefore, the information dimension of the simulation driving platform is expanded fundamentally, the man-machine interaction system can present complex and dynamic information flow existing in a real intelligent network connection environment, the authenticity and fidelity of a simulation scene are greatly improved, and a highly-simulated information environment is provided for research and development testing and personnel training of intelligent network connection automobiles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, in particular to a human-computer interaction system of a simulation driving platform in an intelligent networked environment. BACKGROUND

[0002] In the field of intelligent transportation and automotive engineering, intelligent networked vehicles, as the frontier of industrial technological innovation, highly rely on high-fidelity simulation driving platforms that can accurately reproduce the complex interaction of "human-vehicle-road-cloud" for their research and development, testing, and talent training. Among them, the human-computer interaction system is the core hub connecting the driver and the simulation environment, and its performance directly determines the authenticity of simulation training, the effectiveness of research and development testing, and the pros and cons of user experience. The human-computer interaction of the simulation driving platform aims to provide an efficient and natural information exchange interface for the driver with virtual vehicles and traffic environments through multi-modal channels. The existing human-computer interaction systems are mostly based on traditional driving simulation needs and face significant limitations when adapting to the intelligent networked environment. The existing technology has a single information interaction dimension, making it difficult to present multi-source dynamic information in networked scenarios such as vehicle-road cooperation and inter-vehicle communication, resulting in a significant deviation between simulation experience and real intelligent networked environment. At the same time, the interaction response mechanism lacks intelligence and real-time, with high data refresh delay, and cannot adaptively adjust according to the driving scene and driver state, which may easily lead to operation misjudgment in emergency conditions. In addition, the existing system lacks multi-modal interaction fusion, with isolated voice, touch, and visual interaction methods that cannot cooperatively handle complex networked information queries and presentations, and the system architecture is closed with low data interface standardization, making it difficult to realize real-time and efficient scene collaboration with cloud traffic big data and diversified networked devices. SUMMARY

[0003] The present application aims to provide a human-computer interaction system of a simulation driving platform in an intelligent networked environment to solve the problems raised in the background technology.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical solution: a human-computer interaction system of a simulation driving platform in an intelligent networked environment, characterized by comprising: The networked information perception and fusion module is used for receiving and processing multi-source heterogeneous data streams from the cloud traffic data center and the networked device simulation module in real time to construct a dynamic environment cognition map. The networked information perception and fusion module comprises a data analysis unit and a space-time alignment unit. The data analysis unit decodes and extracts semantics of the received vehicle-road cooperation messages, inter-vehicle communication messages, roadside unit broadcast information and cloud-issued traffic situation data according to a preset communication protocol stack. The space-time alignment unit synchronizes and correlates the information of various types after semantic extraction based on a unified world coordinate system and time stamp to construct a dynamic environment cognition map centered on the vehicle. The dynamic environment cognition map contains road geometry and traffic participant information in traditional driving simulation and integrates information layers specific to the networked environment, including front road event warning, intersection signal light phase and timing, surrounding vehicle intention, recommended speed interval and high-precision map service data. The driver state and intention recognition module is used for monitoring and analyzing multi-dimensional physiological and behavioral signals of the driver in real time to quantify the driving load and operation intention. The driver state and intention recognition module integrates a physiological signal acquisition unit, a visual tracking unit and an operation behavior analysis unit. The physiological signal acquisition unit continuously acquires heart rate variability, skin conductance level and specific frequency band energy value of the driver through a wearable sensor. The visual tracking unit captures eye movement trajectory, gaze point coordinates and pupil diameter change of the driver through an infrared camera installed in the driver cabin. The operation behavior analysis unit records the trigger sequence and frequency of steering wheel angle, accelerator and brake pedal opening, gear operation and self-defined function key in real time. The driver state and intention recognition module further comprises a fusion analysis model based on a deep neural network. The fusion analysis model receives synchronous output data from the physiological signal acquisition unit, visual tracking unit and operation behavior analysis unit and maps to generate a driver state vector and a driver intention vector. The dimensions of the driver state vector include cognitive load level, situational awareness level, fatigue index and stress level. The dimensions of the driver intention vector include lane keeping, following, lane changing, overtaking, braking and avoiding and networked information query intention probability distribution. A multimodal interaction decision-making and generation module is used to dynamically plan and generate multimodal interaction commands based on the dynamic environment cognitive map, the driver's state vector, and the driver's intention vector. The multimodal interaction decision-making and generation module includes an interaction strategy decision-maker, a modal conflict arbitrator, and a content generator. The interaction strategy decision-maker receives the dynamic environment cognitive map, the driver's state vector, and the driver's intention vector as input, and internally runs a decision algorithm based on a hierarchical reinforcement learning framework to select basic interaction modes from a preset strategy library. The basic interaction modes include information enhancement modes, risk prediction modes, and other modes. The system includes three interaction modes: alarm mode, decision support mode, and information query response mode. The content generator generates raw instruction data for the visual, auditory, and tactile channels based on the selected interaction mode and specific interactive content. For the visual channel, it generates visual meta-instructions containing information layers, layout, color coding, and dynamic effect parameters. For the auditory channel, it generates auditory meta-instructions containing voice text content, tone, speech rate, priority, and non-voice prompt tone type and rhythm. For the tactile channel, it generates tactile meta-instructions containing vibration intensity, frequency, waveform, and the specific location of the vibration on the steering wheel, seat, or pedals. A real-time rendering and feedback execution module is used to receive and execute the multimodal interaction commands, providing synchronous sensory feedback to the driver through high-fidelity hardware. The real-time rendering and feedback execution module consists of a visual rendering engine, a 3D audio synthesizer, and a distributed haptic driver. The visual rendering engine is deeply integrated with the main visual system of the simulated driving platform. After receiving the visual meta-commands, it uses overlay rendering technology to overlay information layers generated by the dynamic environment cognition map and interactive content on the basic 3D scene in real time. The 3D audio synthesizer synthesizes voice prompts and warning sounds with spatial orientation based on the auditory meta-commands using a head-related transfer function algorithm, and plays them through a surround sound system. The distributed haptic driver generates a position-specific haptic prompt sequence based on the haptic meta-commands through an array of linear resonant actuators arranged under the steering wheel rim, seat back and cushion, and accelerator and brake pedals.

[0005] Preferably, the hierarchical reinforcement learning decision-making algorithm in the interaction strategy decision-maker includes a meta-controller and multiple sub-executors; the meta-controller operates on a longer time scale, responsible for evaluating the overall driving situation including road type, traffic density, network information richness, and driver state vector, and selects the currently activated sub-executor accordingly; each sub-executor is specifically responsible for a specific interaction decision under a basic interaction mode, and operates on a finer time scale; the sub-executor outputs the specific interaction action to be taken at the current moment based on the real-time observed environmental cognitive map fragments, driver intention vector, and its own internal value network; the specific interaction action corresponds to the specific parameters required by the content generator.

[0006] Preferably, during the offline training phase, the hierarchical reinforcement learning decision algorithm optimizes the neural network parameters of the meta-controller and all sub-actuators end-to-end using a large amount of simulated data covering various intelligent connected scenarios and driver behaviors. The optimization objective is to maximize a composite reward function. The composite reward function consists of a weighted sum of task completion reward, load penalty, and conflict penalty. The task completion reward is issued when the driver successfully perceives key connected information and makes a correct response. The load penalty is positively correlated with the cognitive load level in the driver's state vector. The conflict penalty is triggered when the multimodal output information contradicts each other semantically or temporally.

[0007] Preferably, the modal conflict arbitrator monitors in real time the visual, auditory, and tactile meta-instructions prepared for output by the content generator; the modal conflict arbitrator internally maintains a modal priority mapping table and a conflict rule base; the modal priority mapping table defines the default priority order of each sensory channel under different interaction modes; the conflict rule base predefines a variety of potential inter-modal conflict modes and their arbitration logic; when the modal conflict arbitrator detects that the multi-channel instruction to be output has redundancy in information content, overlap and interference in timing, or potential ambiguity in semantics, it triggers the conflict arbitration process; the arbitration process first queries the modal priority mapping table according to the current interaction mode, and then adjusts the conflicting instruction in combination with the specific rules in the conflict rule base; the adjustment strategy includes simplifying the content of the instruction of the low-priority channel, delaying the output, or directly suppressing it.

[0008] Preferably, it also includes an open collaborative interface layer; the open collaborative interface layer is located between the connected information perception and fusion module and the external data source, providing standardized data access and protocol conversion services; the open collaborative interface layer defines a set of unified data description specifications based on an abstract semantic model, used to convert data from external data sources that conform to the specifications into a standard format that can be recognized internally by the system; at the same time, the open collaborative interface layer also provides a reverse control channel, used to encapsulate and transmit back the interactive effect data or driver behavior data generated by the system according to the format required by the external system.

[0009] Preferably, the fusion analysis model employs a neural network architecture based on a multi-head attention mechanism combined with gated recurrent units. This neural network architecture includes a shared feature extraction layer, three parallel modality-specific encoding branches, a cross-modal attention fusion layer, and two parallel fully connected output layers. The shared feature extraction layer performs preliminary standardization and dimensionality reduction on the original input signal. The three modality-specific encoding branches process physiological signal sequences, visual tracking sequences, and operational behavior sequences, respectively, with each branch consisting of a gated recurrent unit. The cross-modal attention fusion layer receives high-level feature vectors output from the three branches and dynamically fuses complementary information from different modalities by calculating attention weights between the feature vectors, generating a unified context-aware feature vector. This unified context-aware feature vector is then fed to the two parallel fully connected output layers to generate the driver state vector and the driver intention vector, respectively.

[0010] Preferably, the information layer overlay of the visual rendering engine employs a dynamic importance mapping algorithm; the dynamic importance mapping algorithm calculates a dynamic importance score in real time for each piece of connected information in the dynamic environment cognition map; the calculation basis includes the preset urgency level of the information itself, the spatiotemporal correlation between the information and the current vehicle position, the historical frequency of the information, and the situational awareness level in the current driver state vector; the visual rendering engine determines the visual attributes of the corresponding information layer based on the dynamic importance score; the visual attributes include the layer's display size on the screen, transparency, whether to add pulse highlight animation, and its visual depth relative to the base scene.

[0011] Preferably, the data parsing unit performs semantic parsing and context association operations on the data objects after decoding and semantic extraction based on a continuously updated ontology knowledge base, and adds richer context labels to each data object.

[0012] Preferably, the spatiotemporal alignment unit adopts a sliding window synchronization algorithm to calibrate the timestamps of external data objects to the unified timeline of the system and mark their timeliness; at the same time, the spatiotemporal alignment unit calculates an initial correlation score for each information entity by calculating the relative distance, relative speed, estimated arrival time and path correlation index between each external information entity and the current vehicle.

[0013] Preferably, the operation behavior analysis unit maintains a short-term history buffer to analyze the event sequence patterns of steering wheel angle, pedal opening and button triggering, in order to identify the intent features of lane keeping, emergency braking or network information query.

[0014] Compared with existing technologies, the beneficial effects of this invention are as follows: By constructing a connected information perception and fusion module, real-time reception, semantic parsing, and spatiotemporal alignment of multi-source heterogeneous connected information such as vehicle-road cooperation and inter-vehicle communication are achieved, generating a dynamic environmental cognitive map rich in connected environment characteristics. This fundamentally expands the information dimension of the simulated driving platform, enabling the human-machine interaction system to present the complex and dynamic information flow that exists only in real intelligent connected environments, greatly improving the realism and fidelity of the simulated scenario, and providing a highly realistic information environment for the research and development testing of intelligent connected vehicles and personnel training.

[0015] By integrating a driver state and intent recognition module with a multimodal interaction decision-making and generation module based on hierarchical reinforcement learning, a closed-loop interaction system capable of sensing the driver, understanding the environment, and making intelligent decisions has been constructed. The system can adaptively select the optimal interaction strategy and modal combination based on the driver's real-time cognitive load and operational intentions, as well as the urgency and complexity of the driving scenario, and ensure the consistency and efficiency of the interaction information through a modal conflict arbitrator. This transforms human-computer interaction from passive, fixed-pattern information pushing to proactive, context-aware intelligent services, significantly reducing the driver's cognitive burden in complex connected information environments, improving interaction efficiency and safety, and effectively assisting the driver in making rapid and accurate decisions, especially in emergency situations.

[0016] By designing an open collaborative interface layer and a high-fidelity real-time rendering and feedback execution module, the system architecture achieves openness and immersive interactive feedback. The open interface layer adopts standardized data description specifications, breaking the limitations of closed data interfaces in traditional simulation systems. It can flexibly access diverse cloud data services and connected device simulation resources, achieving cross-platform and cross-system scene collaboration capabilities. Simultaneously, the real-time rendering and feedback execution module, through overlay rendering, 3D audio synthesis, and distributed haptic drive technology, ensures that multimodal interactive commands are delivered to the driver's visual, auditory, and tactile channels simultaneously in a high-fidelity and highly immersive manner. This achieves a high degree of integration and spatial consistency between information presentation and physical feedback, providing users with a natural, realistic, and efficient immersive simulated driving experience. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the interactive system structure framework of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 This invention provides a technical solution: a human-machine interaction system for a simulated driving platform in an intelligent connected environment. This system is deployed on a complete simulated driving platform, which includes a steering wheel with force feedback, an accelerator and brake pedal combination, a gear shifting mechanism, a surround view display system, a multi-channel surround sound system, and a driver's seat with integrated haptic feedback. Specifically, it includes: The connected information perception and fusion module is used to receive and process multi-source heterogeneous data streams from the cloud-based traffic data center and the connected device simulation module in real time to construct a dynamic environment cognitive map. The connected information perception and fusion module includes a data parsing unit and a spatiotemporal alignment unit. The data parsing unit decodes and extracts semantics from received vehicle-road cooperative messages, inter-vehicle communication messages, roadside unit broadcast information, and traffic situation data sent from the cloud, based on a preset communication protocol stack. The spatiotemporal alignment unit synchronizes and correlates various types of information after semantic extraction based on a unified world coordinate system and timestamps, constructing a dynamic environment cognitive map centered on the vehicle. The dynamic environment cognitive map includes road geometry and traffic participant information from traditional driving simulations, and integrates information layers unique to the connected environment, including forward road event warnings, intersection traffic light phases and timings, surrounding vehicle intentions, recommended speed ranges, and high-precision map service data. Specifically, it also includes an open collaborative interface layer; the open collaborative interface layer is located between the connected information perception and fusion module and the external data source, providing standardized data access and protocol conversion services; the open collaborative interface layer defines a set of unified data description specifications based on an abstract semantic model, used to convert data from external data sources that conform to the specifications into a standard format that can be recognized internally by the system; at the same time, the open collaborative interface layer also provides a reverse control channel, used to encapsulate and transmit back the interactive effect data or driver behavior data generated by the system according to the format required by the external system; The data parsing unit uses a continuously updated ontology knowledge base to perform semantic parsing and context association operations on the data objects after decoding and semantic extraction, and adds richer context labels to each data object. The spatiotemporal alignment unit uses a sliding window synchronization algorithm to calibrate the timestamps of external data objects to the unified timeline of the system and mark their timeliness. At the same time, the spatiotemporal alignment unit calculates an initial correlation score for each information entity by calculating the relative distance, relative speed, estimated arrival time and path correlation index between each external information entity and the current vehicle.

[0020] The driver state and intention recognition module is used to monitor and analyze the driver's multi-dimensional physiological and behavioral signals in real time to quantify their driving load and operational intentions. This module integrates a physiological signal acquisition unit, a visual tracking unit, and an operational behavior analysis unit. The physiological signal acquisition unit continuously collects the driver's heart rate variability, skin conductance level, and specific frequency band energy values ​​of electroencephalograms using wearable sensors. The visual tracking unit captures the driver's eye movement trajectory, fixation point coordinates, and pupil diameter changes using an infrared camera installed in the cockpit. The operational behavior analysis unit records the steering wheel angle, accelerator, and brake pedal movements in real time. The system includes the trigger sequence and frequency of pedal opening, gear shifting, and custom function buttons; the driver state and intention recognition module also includes a fusion analysis model built on a deep neural network; the fusion analysis model receives synchronous output data from the physiological signal acquisition unit, visual tracking unit, and operation behavior analysis unit, and maps and generates driver state vectors and driver intention vectors; the dimensions of the driver state vector include cognitive load level, situational awareness level, fatigue index, and stress level; the dimensions of the driver intention vector include the probability distribution of intentions for lane keeping, following, lane changing, overtaking, braking and avoidance, and online information query; Specifically, the fusion analysis model employs a neural network architecture based on a multi-head attention mechanism combined with gated recurrent units. This neural network architecture includes a shared feature extraction layer, three parallel modality-specific encoding branches, a cross-modal attention fusion layer, and two parallel fully connected output layers. The shared feature extraction layer performs preliminary standardization and dimensionality reduction on the original input signal. The three modality-specific encoding branches process physiological signal sequences, visual tracking sequences, and operational behavior sequences, respectively, with each branch consisting of a gated recurrent unit. The cross-modal attention fusion layer receives high-level feature vectors output from the three branches and dynamically fuses complementary information from different modalities by calculating attention weights between the feature vectors, generating a unified context-aware feature vector. This unified context-aware feature vector is then fed into the two parallel fully connected output layers to generate the driver state vector and the driver intention vector, respectively.

[0021] The operation behavior analysis unit maintains a short-term history buffer to analyze the event sequence patterns of steering wheel angle, pedal opening, and button triggering in order to identify the intent features of lane keeping, emergency braking, or network information query.

[0022] A multimodal interaction decision-making and generation module is used to dynamically plan and generate multimodal interaction commands based on the dynamic environment cognitive map, the driver's state vector, and the driver's intention vector. The multimodal interaction decision-making and generation module includes an interaction strategy decision-maker, a modal conflict arbitrator, and a content generator. The interaction strategy decision-maker receives the dynamic environment cognitive map, the driver's state vector, and the driver's intention vector as input, and internally runs a decision algorithm based on a hierarchical reinforcement learning framework to select basic interaction modes from a preset strategy library. The basic interaction modes include information enhancement modes, risk prediction modes, and other modes. The system includes three interaction modes: alarm mode, decision support mode, and information query response mode. The content generator generates raw instruction data for the visual, auditory, and tactile channels based on the selected interaction mode and specific interactive content. For the visual channel, it generates visual meta-instructions containing information layers, layout, color coding, and dynamic effect parameters. For the auditory channel, it generates auditory meta-instructions containing voice text content, tone, speech rate, priority, and non-voice prompt tone type and rhythm. For the tactile channel, it generates tactile meta-instructions containing vibration intensity, frequency, waveform, and the specific location of the vibration on the steering wheel, seat, or pedals. Specifically, the hierarchical reinforcement learning decision-making algorithm in the interaction strategy decision-maker includes a meta-controller and multiple sub-executors. The meta-controller operates on a longer timescale, responsible for evaluating the overall driving situation, including road type, traffic density, network information richness, and driver state vector, and selecting the currently activated sub-executor accordingly. Each sub-executor is specifically responsible for a particular interaction decision under a basic interaction mode, operating on a finer timescale. Based on the real-time observed environmental cognitive map fragments, driver intention vector, and its own internal value network, the sub-executor outputs the specific interaction action to be taken at the current moment. The specific interaction action corresponds to the specific parameters required by the content generator.

[0023] The modal conflict arbitrator monitors in real time the visual, auditory, and tactile meta-instructions prepared for output by the content generator. Internally, the arbitrator maintains a modal priority mapping table and a conflict rule base. The modal priority mapping table defines the default priority order of each sensory channel under different interaction modes. The conflict rule base predefines various potential inter-modal conflict modes and their arbitration logic. When the arbitrator detects that the multi-channel instruction to be output has redundancy in information content, overlap in timing, or potential semantic ambiguity, it triggers the conflict arbitration process. The arbitration process first queries the modal priority mapping table based on the current interaction mode, and then adjusts the conflicting instruction by combining the specific rules in the conflict rule base. The adjustment strategies include simplifying the content of instructions from low-priority channels, delaying output, or directly suppressing them.

[0024] It should be further explained that during the offline training phase, the hierarchical reinforcement learning decision-making algorithm performs end-to-end optimization of the neural network parameters of the meta-controller and all sub-actuators using a large amount of simulated data covering various intelligent connected scenarios and driver behaviors. The optimization objective is to maximize a composite reward function, which is composed of a weighted sum of task completion reward, load penalty, and conflict penalty. The task completion reward is issued when the driver successfully perceives key connected information and makes a correct response. The load penalty is positively correlated with the cognitive load level in the driver's state vector. The conflict penalty is triggered when the multimodal output information contradicts each other semantically or temporally. A real-time rendering and feedback execution module is used to receive and execute the multimodal interaction commands, providing synchronous sensory feedback to the driver through high-fidelity hardware. The real-time rendering and feedback execution module consists of a visual rendering engine, a 3D audio synthesizer, and a distributed haptic driver. The visual rendering engine is deeply integrated with the main visual system of the simulated driving platform. After receiving the visual meta-commands, it uses overlay rendering technology to overlay information layers generated by the dynamic environment cognition map and interactive content on the basic 3D scene in real time. The 3D audio synthesizer synthesizes voice prompts and warning sounds with spatial orientation based on the auditory meta-commands using a head-related transfer function algorithm, and plays them through a surround sound system. The distributed haptic driver generates a position-specific haptic prompt sequence based on the haptic meta-commands through an array of linear resonant actuators arranged under the steering wheel rim, seat back and cushion, and accelerator and brake pedals.

[0025] Specifically, the visual rendering engine uses a dynamic importance mapping algorithm for overlaying information layers. This algorithm calculates a dynamic importance score for each piece of connected information in the dynamic environmental cognition map in real time. The calculation is based on factors including the information's preset urgency level, the spatiotemporal correlation between the information and the current vehicle's location, the information's historical frequency of occurrence, and the current driver's state vector's contextual awareness level. The visual rendering engine determines the visual attributes of the corresponding information layer based on this dynamic importance score. These visual attributes include the layer's display size on the screen, transparency, whether a pulse highlight animation is added, and its visual depth relative to the base scene.

[0026] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A human-computer interaction system of a simulation driving platform in an intelligent network environment, characterized in that, Comprise: The networked information perception and fusion module is used for receiving and processing multi-source heterogeneous data streams from the cloud traffic data center and the networked equipment simulation module in real time to construct a dynamic environment cognition map; the networked information perception and fusion module comprises a data analysis unit and a space-time alignment unit; the data analysis unit is based on a preset communication protocol stack, and the space-time alignment unit synchronizes and correlates various types of information after semantic extraction based on a unified world coordinate system and a timestamp to construct a dynamic environment cognition map centered on the vehicle; the dynamic environment cognition map comprises road geometry and traffic participant information in traditional driving simulation and integrates information layers specific to the networked environment; The driver state and intention recognition module is used for monitoring and analyzing multi-dimensional physiological and behavioral signals of the driver in real time; the driver state and intention recognition module integrates a physiological signal acquisition unit, a visual tracking unit and an operation behavior analysis unit, and further comprises a fusion analysis model based on a deep neural network; the fusion analysis model receives synchronous output data from the physiological signal acquisition unit, the visual tracking unit and the operation behavior analysis unit and maps to generate a driver state vector and a driver intention vector; The multi-modal interactive decision and generation module is used for dynamically planning and generating multi-modal interactive instructions according to the dynamic environment cognition map and the driver state vector and the driver intention vector; the multi-modal interactive decision and generation module comprises an interactive strategy decision maker, a modal conflict arbitrator and a content generator; the interactive strategy decision maker receives the dynamic environment cognition map, the driver state vector and the driver intention vector as input, internally runs a decision algorithm based on a hierarchical reinforcement learning framework to select a basic interactive mode from a preset strategy library; The real-time rendering and feedback execution module is used for receiving and executing the multi-modal interactive instructions to provide synchronous sensory feedback to the driver through high-fidelity hardware devices; The real-time rendering and feedback execution module is composed of a visual rendering engine, a three-dimensional audio synthesizer and a distributed haptic driver; the visual rendering engine is deeply integrated with a main visual system of a simulation driving platform; after receiving the visual meta-instruction, the visual rendering engine superimposes information layers generated by the dynamic environment cognition map and the interactive content on the basic three-dimensional scene in real time through overlay rendering technology.

2. The human-machine interaction system of a simulation driving platform in an intelligent network environment according to claim 1, wherein, The hierarchical reinforcement learning decision algorithm in the interactive strategy decision maker comprises a meta-controller and multiple sub-executors; the meta-controller works on a long time scale, is responsible for evaluating the overall driving situation including road type, traffic density, networked information richness and driver state vector, and selecting the currently activated sub-executor accordingly; each sub-executor is responsible for specific interactive decision in a basic interactive mode and works on a finer time scale; the sub-executor outputs specific interactive actions to be taken at the current time according to the environment cognition map fragment observed in real time, the driver intention vector and the internal value network; the specific interactive actions correspond to specific parameters required by the content generator.

3. The human-machine interaction system of a simulation driving platform in an intelligent network environment according to claim 2, characterized in that, The layered reinforcement learning decision algorithm optimizes the neural network parameters of the meta-controller and all sub-actuators end-to-end through a large amount of simulation data covering various intelligent network connection scenarios and driver behaviors in an offline training phase; the optimization goal is to maximize a composite reward function; the composite reward function is composed of a task completion reward, a load penalty and a conflict penalty; the task completion reward is issued when the key network connection information is successfully perceived by the driver and the correct response is made; the load penalty is positively correlated with the cognitive load level in the driver state vector; the conflict penalty is triggered when the multi-modal output information is contradictory in semantics or timing.

4. The human-machine interaction system of a simulation driving platform in an intelligent network environment according to claim 1, characterized in that, The modal conflict arbitrator monitors the visual, auditory and tactile meta-instructions prepared for output by the content generator in real time; the modal conflict arbitrator internally maintains a modal priority mapping table and a conflict rule library; the modal priority mapping table defines the default priority order of each perception channel in different interaction modes; the conflict rule library predefines various potential inter-modal conflict modes and their arbitration logic; when the modal conflict arbitrator detects that the multi-channel instructions to be output have redundancy in information content, overlap interference in timing or potential ambiguity in semantics, conflict arbitration process is triggered; The arbitration process first queries the modal priority mapping table according to the current interaction mode, and then adjusts the conflicting instructions in combination with the specific rules in the conflict rule library; the adjustment strategy includes content simplification, output delay or direct suppression of instructions in low-priority channels.

5. The human-machine interaction system of a simulation driving platform in an intelligent network environment according to claim 1, characterized in that, An open collaborative interface layer is also included; the open collaborative interface layer is located between the network connection information perception and fusion module and external data sources, providing standardized data access and protocol conversion services; the open collaborative interface layer defines a set of unified data description specifications based on abstract semantic models, which are used to convert data from external data sources conforming to the specifications into a standard format recognizable by the system; at the same time, the open collaborative interface layer also provides a reverse control channel for encapsulating and returning the interaction effect data or driver behavior data generated by the system in the format required by external systems.

6. The human-machine interaction system of a simulation driving platform in an intelligent network environment according to claim 1, characterized in that, The fusion analysis model adopts a neural network architecture based on a multi-head attention mechanism combined with a gated recurrent unit; the neural network architecture includes a shared feature extraction layer, three parallel modal-specific encoding branches, a cross-modal attention fusion layer, and two parallel fully connected output layers; the shared feature extraction layer performs preliminary standardization and dimensionality reduction on the original input signal; the three modal-specific encoding branches process physiological signal sequences, visual tracking sequences, and operation behavior sequences, respectively, and each branch is composed of a gated recurrent unit; the cross-modal attention fusion layer receives high-level feature vectors output by the three branches, dynamically fuses complementary information from different modalities by calculating attention weights between the feature vectors, and generates a unified context-aware feature vector; the unified context-aware feature vector is then fed into two parallel fully connected output layers to generate the driver state vector and the driver intent vector, respectively.

7. The human-machine interaction system of a simulation driving platform in an intelligent network environment according to claim 1, characterized in that, The information layer superposition of the visual rendering engine adopts a dynamic importance mapping algorithm; the dynamic importance mapping algorithm calculates a dynamic importance score for each networked information in the dynamic environment awareness graph in real time; The calculation depends on the preset emergency level of the information itself, the spatio-temporal correlation of the information with the current vehicle position, the historical occurrence frequency of the information, and the situational awareness level in the current driver state vector; the visual rendering engine determines the visual attributes of the corresponding information layer according to the dynamic importance score; the visual attributes include the display size, transparency, whether to add a pulse highlight animation, and the visual depth relative to the base scene of the layer on the screen. 8.The human-machine interaction system of a simulation driving platform in an intelligent network environment according to claim 1, wherein, The data analysis unit performs semantic analysis and context association operations on the data objects after decoding and semantic extraction based on a constantly updated ontology knowledge base, and adds more rich context labels to each data object. 9.The human-machine interaction system of a simulation driving platform in an intelligent network environment according to claim 1, wherein, The space-time alignment unit adopts a sliding window synchronization algorithm to calibrate the timestamps of external data objects to the system unified timeline and mark their time effectiveness; At the same time, the space-time alignment unit calculates an initial correlation score for each information entity by calculating the relative distance, relative speed, expected arrival time, and path relevance index of each external information entity and the current vehicle.

10. The human-machine interaction system of a simulation driving platform in an intelligent network environment according to claim 1, wherein, The operation behavior analysis unit maintains a short-term history buffer for analyzing the event sequence pattern of steering wheel angle, pedal opening, and key trigger to identify the intent features of lane keeping, emergency braking, or networked information query.

Images