An immersive remote driving simulation system and method based on home-vehicle interconnection

Abstract

This invention discloses an immersive remote driving simulation system and method based on home-vehicle interconnection. The system includes: a home simulation terminal, which comprises a high-dynamic haptic simulation platform, a high-fidelity simulation cockpit, an immersive display system, a local digital twin engine, and a biometric monitoring unit; a vehicle execution terminal, which comprises an onboard perception and acquisition kit, a drive-by-wire execution unit, and an edge computing gateway; and a converged communication network connecting the two. This invention solves the technical problems of lack of haptic feedback, poor network fluctuation tolerance, and stiff human-computer interaction in existing remote driving technologies by constructing a complete physical perception closed loop of "real vehicle dynamics → simulator haptic reproduction → driver haptic prediction → operation correction → real vehicle response." This invention can decouple vehicle ownership from usage rights, improve operational safety in hazardous environments, enhance the travel experience for special groups, and provide a reliable human backup for autonomous driving.

Description

Technical Field

[0001] This invention relates to the field of driving simulation technology, and in particular to an immersive remote driving simulation system and method based on home-vehicle interconnection. Background Technology

[0002] With the development of 5G / 6G / V2X communication technologies, remote driving technology has been applied in scenarios such as mining, logistics, and special operations. However, existing remote driving solutions (such as cloud-based driver control and remote-controlled driving) typically suffer from the following technical shortcomings:

[0003] First, the lack of haptic feedback leads to distorted operation. Existing remote driving systems mainly rely on video transmission and simple steering wheel feedback, with remote drivers only able to observe road conditions through the screen. Because they cannot perceive the acceleration, roll, pitch, and road bumps of the real vehicle, the driver's judgment of speed and vehicle posture is severely distorted, especially in complex road conditions (such as bumpy roads and slippery curves), which can easily lead to misoperation.

[0004] Secondly, insufficient immersion leads to driver fatigue and safety risks. Current remote cockpits mostly use a combination of fixed seats and flat screens, which cannot simulate the environment of a real cockpit. Drivers need to maintain a high level of concentration for extended periods, and the lack of sensory feedback forces the brain to rely on visual compensation, leading to increased cognitive load, accelerated fatigue, and ultimately affecting driving safety.

[0005] Third, there is a lack of effective compensation mechanisms for network fluctuations. Existing solutions mostly rely on a single network link. Once latency, jitter, or packet loss occurs, control commands and video feedback become out of sync, directly leading to a deterioration in the driving experience and even safety accidents. Although some research has proposed multi-link redundant communication schemes, these mainly focus on the data aggregation level and have not addressed the issue of network fluctuation compensation at the sensory level.

[0006] Fourth, remote takeover lacks a smooth transition in human-machine collaboration. In scenarios where the autonomous driving system fails and requires remote human intervention, existing solutions suffer from abrupt and delayed takeover processes due to the driver's lack of tactile awareness of the vehicle's current motion, making it difficult to achieve a safe and seamless switchover.

[0007] Fifth, the application scenarios are limited, and the pain points of ordinary users' travel are not addressed. Existing technologies are mainly geared towards special operations or commercial vehicles, without taking into account the pain points of ordinary private car users. The daily idle rate of private cars exceeds 95%, and users have to bear high holding costs such as parking, insurance, and maintenance, but existing technologies cannot effectively separate car ownership from usage rights.

[0008] Furthermore, while existing research has combined digital twins with virtual reality for human-computer interaction or enhanced driver intent recognition through brain-computer interfaces, none of these studies have constructed a complete physical perception loop from "the dynamics of the real vehicle body" to "the haptic reproduction of the home simulator," failing to achieve a deep integration of haptic sensation, vision, and control. Summary of the Invention

[0009] The main objective of this invention is to provide an immersive remote driving simulation system based on home-vehicle connectivity.

[0010] Another objective of this invention is to propose an immersive remote driving simulation method based on home-vehicle connectivity.

[0011] A third objective of this invention is to provide a computer-readable storage medium.

[0012] To achieve the above objectives, a first aspect of the present invention provides an immersive remote driving simulation system based on home-vehicle connectivity, comprising: The home simulation terminal includes a motion-sensing simulation platform for providing multi-degree-of-freedom motion, and a human-computer interaction cockpit set on the motion-sensing simulation platform; The vehicle actuator, which is installed on a remote real vehicle, includes a sensing unit for collecting information on the vehicle's own motion posture and the surrounding environment, and a drive-by-wire actuator for executing vehicle driving actions. A converged communication network aggregates multiple communication links to connect the home simulation terminal and the vehicle execution terminal; The vehicle actuator is used to send the collected vehicle motion posture data to the home simulation terminal in real time. The motion simulation platform reproduces the dynamic motion sensation consistent with the real vehicle in real time based on the received vehicle motion posture data.

[0013] Optionally, the home simulation terminal also includes a local digital twin controller, which pre-stores a dynamic model corresponding to the real vehicle. This model is used to predict the driving commands of the motion simulation platform based on the last received vehicle state data when the communication link is delayed or interrupted, so as to achieve motion compensation.

[0014] Optionally, the vehicle execution terminal also includes an edge computing gateway, which uses multi-link aggregation transmission technology to aggregate at least two heterogeneous communication links to transmit video streams, audio streams, and vehicle status data to the home simulation terminal in real time.

[0015] Optionally, the home simulation terminal also includes a biometric monitoring unit, which includes one or more of an electrocardiogram monitoring electrode, a pressure distribution sensor, and a brain-computer interface electrode, for collecting the driver's physiological signals to identify driving intentions and fatigue status.

[0016] Optionally, the home simulation terminal also includes a fuzzy sharing controller, which is used to fuse driver control commands, biometric recognition results, and vehicle active safety boundary information to generate optimized control commands and send them to the vehicle execution terminal.

[0017] Optionally, the motion simulation platform is a six-degree-of-freedom electric servo motion platform with a dynamic response delay of no more than 20 ms and a maximum dynamic overload capacity of no less than 0.8 G.

[0018] Optionally, the system supports the following two driving modes: Online mode: The home simulator and the vehicle actuator are connected in real time to achieve remote driving control; Training mode: Disconnect from the vehicle execution terminal, and the home simulation terminal runs independently in a virtual environment based on high-precision map data to conduct route rehearsals or driving training.

[0019] To achieve the above objectives, a second aspect of the present invention proposes an immersive remote driving simulation method based on home-vehicle interconnection, comprising: Downlink transmission steps: The remote real vehicle collects video streams, audio streams, and vehicle motion status data, and sends them to the home simulation terminal through the communication link; the home simulation terminal parses the data, outputs the video stream to the display system, and inputs the vehicle motion status data as driving commands into the motion simulation platform to control the human-machine interaction cockpit to reproduce the motion posture consistent with the real vehicle. Uplink transmission steps: The driver operates the control components in the human-machine interface cockpit to generate control commands, which are then sent to the remote real vehicle via the communication link, where the drive-by-wire execution unit performs the corresponding operations; Network fluctuation compensation steps: When a delay or packet loss occurs in the communication link, the vehicle motion trend is predicted using a dynamic model based on the last received vehicle status data, and a compensation command is generated to drive the motion simulation platform to achieve a smooth transition.

[0020] Optionally, the uplink transmission step further includes: While collecting driver control commands, the biometric monitoring unit collects the driver's physiological signals, identifies driving intentions, filters out physiological vibrations in the control commands, and generates optimized control commands by combining the vehicle's active safety boundary information.

[0021] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the method described in the second aspect above.

[0022] The embodiments of the present invention have the following beneficial effects: First, it decouples car ownership from usage rights. Users can deploy expensive vehicle assets in low-cost areas (such as suburban parking lots or shared parking spaces) and access the vehicles at any time via a simulator at home, without the burden of parking and moving cars in the city center. Vehicle utilization can be increased by 5-10 times, and user travel costs are significantly reduced.

[0023] Secondly, it breaks down the physiological and spatial limitations of drivers. The elderly, disabled, or those with a fear of driving can complete driving tasks in the familiar and comfortable environment of their homes. Professional drivers can work together from home, using simulators to take turns driving multiple commercial vehicles on the road, improving their working environment and expanding the human resource pool for transportation services.

[0024] Third, it enhances operational safety in extreme scenarios. In hazardous environments such as mines, disaster areas, chemical industrial parks, and epidemic zones, operators can complete driving tasks without being physically present, achieving "inherent safety" for personnel. Simultaneously, the introduction of haptic feedback improves the accuracy of remote operation and reduces the risk of misoperation due to a lack of haptic feedback.

[0025] Fourth, it provides a reliable human backup for autonomous driving. When a Level 4 autonomous driving system encounters unforeseen and challenging scenarios, the cloud-based driver can seamlessly take over through a highly immersive simulator. Because the simulator can reproduce the vehicle's movement trends in advance, the driver can form a tactile anticipation, making the takeover process more natural and safer.

[0026] Fifth, it overcomes the degradation of experience caused by network fluctuations. Through the predictive compensation mechanism of the digital twin engine, even in the event of network latency or packet loss, the driver can still obtain smooth and continuous tactile feedback, avoiding the dizziness and operational jerks commonly found in traditional long-distance driving.

[0027] Sixth, expand the depth and breadth of human-computer interaction. Through biometric monitoring and intent recognition, the system can dynamically adjust control assistance strategies according to the driver's physiological state, achieving a truly immersive "human-vehicle integration" interaction. Attached Figure Description

[0028] The above-described and additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which: Figure 1 This is a structural diagram of an immersive remote driving simulation system based on home-vehicle interconnection, provided as an embodiment of the present invention. Detailed Implementation

[0029] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] The following description, with reference to the accompanying drawings, describes an immersive remote driving simulation system and method based on home-vehicle interconnection according to an embodiment of the present invention.

[0032] Example 1 This embodiment provides an immersive remote driving simulation system based on home-vehicle connectivity. For example... Figure 1 As shown, the system includes a home simulation terminal, a vehicle execution terminal, and a converged communication network.

[0033] The home simulation terminal includes a motion-sensing simulation platform for providing multi-degree-of-freedom motion, and a human-computer interaction cockpit set on the motion-sensing simulation platform; the vehicle execution terminal is set on a remote real vehicle, including a sensing unit for collecting information on the vehicle's own motion posture and surrounding environment, and a drive-by-wire execution unit for executing vehicle driving actions; and a converged communication network that aggregates multiple communication links to connect the home simulation terminal and the vehicle execution terminal; wherein, the vehicle execution terminal is used to send the collected vehicle motion posture data to the home simulation terminal in real time, and the motion-sensing simulation platform reproduces the dynamic motion sensation consistent with the real vehicle in real time based on the received vehicle motion posture data.

[0034] The following is a detailed explanation of each part.

[0035] (1) Home simulation terminal.

[0036] In one embodiment of the present invention, the home simulator constitutes the core of the human-machine interaction in the remote driving system. Through multi-dimensional perception and feedback fusion, it enables the driver to immerse themselves in controlling a real vehicle in a non-vehicle environment. The home simulator is preferably installed in a separate space within the driver's residence to reduce external interference and improve operational safety and stability. It consists of a high-dynamic motion-sensing simulation platform, a high-fidelity simulated cockpit, an immersive display system, a local digital twin engine, and a biometric monitoring unit.

[0037] Specifically, the high-dynamic motion simulation platform serves as the motion execution foundation of the entire system, employing a six-degree-of-freedom parallel mechanism or a multi-degree-of-freedom motion architecture improved from a serial industrial robot. By precisely controlling the displacement and attitude changes of each execution unit, the platform can accurately reproduce the motion state of a real vehicle in yaw, pitch, roll, and three-axis translation directions. In a preferred embodiment, the platform's control system combines a high-speed real-time control bus with a servo drive system, keeping the overall dynamic response latency within 20 milliseconds, thereby avoiding motion sickness or control errors caused by perception lag. Simultaneously, the platform possesses a dynamic overload capacity of at least 0.8G, realistically reproducing the changes in motion under extreme conditions such as rapid acceleration, sudden braking, and sharp steering. Furthermore, the platform can modulate high-frequency vibrations through preset filtering and amplitude limiting algorithms to ensure realism while avoiding discomfort to the human body.

[0038] In terms of the high-fidelity simulated cockpit, its overall structure is strictly replicated one-to-one with the target vehicle, including key components such as the steering wheel assembly, pedal system, shift mechanism, and driver's seat. The mechanical characteristic parameters of each control component, such as damping curves, rebound characteristics, and travel range, are calibrated to be consistent with the real vehicle. In a preferred embodiment, a high-response torque motor is integrated into the steering wheel, which can simulate steering resistance, changes in road surface adhesion, and self-centering torque based on the vehicle's real-time status. The pedal system simulates the non-linear feedback characteristics of the accelerator and brake pedals through an adjustable damping mechanism. Multi-point linear vibration actuators are arranged inside the seat and pedals to reproduce low-frequency vibrations caused by engine operation and high-frequency vibrations caused by uneven road surfaces, thereby enhancing the overall immersive experience.

[0039] Regarding immersive display systems, this invention provides two main configurations that can be combined depending on the application scenario. One is a virtual reality head-mounted display device, providing a 360-degree field of view and updating the viewing angle in real time via a high-precision head tracking module. The tracking latency is preferably controlled within 15 milliseconds to ensure consistency between visual and tactile sensations. The other is a multi-screen curved display system, composed of three or more high-refresh-rate curved displays, forming a horizontal field of view exceeding 180 degrees. Seamless display between display units is achieved through image stitching and distortion correction algorithms. In specific applications, the display system presents panoramic video data collected and transmitted from the real vehicle, and overlays an augmented reality information layer on top of this, including navigation path guidance, potential hazard target markings, lane line recognition results, and environmental semantic information, thereby assisting the driver in decision-making.

[0040] The local digital twin engine, serving as the core computing unit of the system, is deployed on a high-performance computing platform. Internally, it integrates a multiphysics dynamics model consistent with the real vehicle, including suspension system models, tire contact models, and powertrain models. This engine first analyzes and synchronizes the motion state data from the real vehicle, converting it into control commands to drive the motion-sensing simulation platform. In the event of network communication delays or short-term interruptions, the digital twin engine can perform short-term predictions based on the most recent vehicle state using the dynamics model, generating continuous motion trend data to drive the motion-sensing platform through smooth transitions, avoiding discomfort or misoperation caused by sudden changes. Furthermore, the engine can construct a virtual reference scene and fuse it with real vehicle video data to achieve a virtual-real overlay display, improving the completeness of environmental understanding.

[0041] In terms of the biometric monitoring unit, multimodal sensors are integrated into contact interfaces such as the seat, seat belt, and steering wheel to continuously monitor the driver's physiological state. Specifically, this includes an electrode module for collecting electrocardiogram (ECG) signals to obtain heart rate and heart rate variability indicators; a distributed pressure sensor array for analyzing the driver's posture and center of gravity changes; and, in some embodiments, non-invasive electroencephalogram (EEG) acquisition electrodes to acquire EEG signals and further identify driving intentions and fatigue levels. After analysis by the local processing unit, the monitoring data can output the driver's fatigue level, attention level, and psychological load status in real time, serving as an important input for the shared control strategy in the remote driving system. This allows for dynamic adjustments to vehicle control when necessary, improving overall driving safety.

[0042] Through the coordinated operation of the above subsystems, the home simulation terminal of the present invention can highly reproduce the real driving experience in a non-vehicle environment, and achieve safe and stable control of remote vehicles while ensuring immersion.

[0043] (2) Vehicle execution end.

[0044] In one embodiment of the present invention, the vehicle actuator, as the actual execution entity of the remote driving system, is integrated into the controlled real vehicle to perform functions such as environmental information collection, data transmission, and control command execution. This actuator establishes a real-time connection with the home simulation terminal via a highly reliable communication link, thereby realizing closed-loop control of the vehicle by the driver remotely. It includes an onboard perception and acquisition kit, an onboard execution and control unit, and an edge computing gateway.

[0045] Specifically, the vehicle-mounted perception and acquisition suite forms the basis for the vehicle to acquire multi-dimensional information about its external environment and its own status. The surround-view camera array is preferably positioned at the front, rear, left, right, and necessary oblique angles of the vehicle to achieve omnidirectional coverage of the surrounding environment. In a preferred embodiment, each camera has a resolution of at least 1080P and a sampling rate of at least 60 frames per second. Through image stitching and calibration algorithms, a continuous 360-degree panoramic video stream is generated and transmitted in real-time to a remote simulation system for immersive visual reconstruction. Simultaneously, a high-sensitivity microphone array is distributed at different locations on the vehicle body to collect environmental sound field information and sound signals generated by the powertrain system. After processing, these signals are synchronously transmitted to a home simulation terminal, thereby achieving realistic sound reproduction and enhancing the sense of presence during remote driving.

[0046] In terms of positioning and attitude perception, the high-precision positioning module preferably employs GPS-RTK or BeiDou differential positioning technology, and is tightly coupled and fused with the inertial measurement unit to provide centimeter-level position accuracy and high-frequency attitude data output. This module can output the vehicle's three-axis acceleration, angular velocity, and attitude angle information in real time, providing crucial input for the motion drive and digital twin model calculations of the remote motion sensing platform. Simultaneously, through connection to the vehicle's CAN bus interface, the system can read multiple operating parameters in real time, including vehicle speed, wheel speed, throttle opening, brake pressure, steering angle, suspension displacement, and engine speed, and perform unified encoding and time synchronization processing to form a complete vehicle status data stream.

[0047] The onboard execution and control unit is the core actuator for remote control. Built on drive-by-wire technology, it electronically controls the drive, steering, and braking systems. In a preferred embodiment, each execution subsystem has independent redundancy and a self-checking mechanism to enhance system reliability. The control unit receives control commands from the home simulation terminal, converts them into specific execution signals, and updates them with a control cycle of no more than 10 milliseconds, ensuring the real-time and continuous response of the vehicle. Furthermore, to prevent dangerous behavior caused by abnormal commands, the control unit can also incorporate amplitude limiting and consistency verification mechanisms to constrain the reasonableness of input commands.

[0048] As the core of data processing and security control in vehicles, the edge computing gateway preferably employs an embedded computing platform with high computing power and low power consumption. This gateway first preprocesses data from various sensors, including noise reduction, compression encoding, and unified timestamp calibration, to ensure consistency and alignment of multi-source data during transmission. Simultaneously, its built-in multi-link aggregation communication module can dynamically schedule and allocate bandwidth for cellular networks, dedicated short-range communications, or other wireless links, thereby improving the overall stability and anti-interference capability of data transmission.

[0049] In terms of security strategies, the edge computing gateway operates multi-level security control logic in real time, including communication status monitoring, latency assessment, and anomaly detection. When a communication link interruption or severe delay is detected, the system can immediately trigger an emergency braking strategy or enter a safety mode, such as gradually decelerating and maintaining lane keeping. Furthermore, the gateway can also perform boundary protection based on the vehicle's current status and preset safety boundaries, such as restricting the vehicle from entering dangerous areas or preventing it from exceeding a set speed range. In a further implementation, the edge computing gateway also possesses in-vehicle edge computing takeover capability, meaning that in the event of a failure of all remote communication links, it can make local decisions and controls based on pre-set autonomous driving or assisted driving strategies to achieve short-term autonomous driving or safe parking, thereby minimizing the risks associated with remote control failures.

[0050] Through the coordinated work of the above modules, the vehicle actuator can not only transmit vehicle operating status and environmental information back to the remote end with high fidelity, but also accurately execute remote control commands while ensuring safety, thereby achieving stable operation of the remote driving system.

[0051] (3) Converged communication network.

[0052] In one embodiment of the present invention, a converged communication network serves as a key infrastructure connecting the home simulation terminal and the vehicle execution terminal, enabling highly reliable, low-latency transmission of various types of data. This communication network constructs a transmission system with high redundancy and adaptive capabilities by uniformly scheduling and collaboratively managing multiple heterogeneous communication links, thereby meeting the stringent real-time and stability requirements of remote driving.

[0053] Specifically, the converged communication network employs multi-link redundancy aggregation transmission technology to enable parallel access and dynamic fusion of multiple communication methods. 5G or 6G cellular networks serve as the primary communication link, leveraging their high bandwidth and low latency to handle the main data transmission tasks. At the home analog end, a fixed link is established via a fiber optic broadband network, providing stable uplink and downlink channels, particularly enhancing the continuity and reliability of data transmission in indoor environments. On the vehicle side, the system can further utilize C-V2X direct communication technology to interact with roadside units in the road infrastructure, enabling low-latency data supplementation and path optimization in specific scenarios. Furthermore, in remote areas or where cellular network coverage is insufficient, a low-Earth orbit satellite communication link can be optionally added as a supplementary channel to ensure system availability in complex geographical environments.

[0054] In terms of link management, the converged communication network monitors key indicators such as bandwidth, latency, and packet loss rate of each communication link in real time, and performs dynamic scheduling based on preset strategies or adaptive algorithms. Specifically, load balancing, primary / backup switching, or multi-path concurrent transmission can be used to distribute data to different links for transmission, so that the system can still maintain the continuity and stability of overall communication even when the performance of a single link degrades or is interrupted. At the same time, forward error correction and retransmission mechanisms are introduced at the data layer to further reduce the impact of data loss on system operation.

[0055] In terms of data transmission mechanisms, the converged communication network adopts a real-time audio and video transmission protocol, uniformly encapsulating and multiplexing video streams, control command streams, and vehicle status data streams. By establishing a priority scheduling strategy, the control command stream is given the highest priority to ensure real-time response to control signals; the vehicle status stream is next, used to maintain closed-loop system control; the video stream undergoes adaptive encoding and resolution adjustment based on bandwidth availability to ensure image continuity while avoiding excessive bandwidth consumption. During transmission, all data streams are aligned using timestamps and synchronization mechanisms to ensure consistency during remote reconstruction.

[0056] In terms of performance metrics, through the aforementioned multi-link fusion and scheduling mechanism, the system can control end-to-end communication latency to within 100 milliseconds, thereby meeting the real-time feedback requirements of remote driving. Simultaneously, by combining link redundancy and data error correction strategies, the overall data packet loss rate can be controlled to within 1%, ensuring the integrity of critical control and perception information. Furthermore, in some implementations, the system can dynamically adjust the encoding strategy and data transmission frequency according to the current network status to maintain stable service quality under different network conditions.

[0057] Through the above design, the converged communication network can provide stable and efficient data transmission support in complex and ever-changing communication environments, providing a reliable guarantee for the safe operation of remote driving systems.

[0058] The workflow of this system includes data flow and energy flow in two directions: downlink (vehicle → home) and uplink (home → vehicle), as detailed in Example 2.

[0059] Example 2 This invention also provides an immersive remote driving simulation method based on home-vehicle connectivity, which is applied to the system proposed in Embodiment 1. It includes the following steps: S1, Downlink transmission steps: The remote real vehicle collects video streams, audio streams, and vehicle motion status data, and sends them to the home simulation terminal through the communication link; the home simulation terminal parses the data, outputs the video stream to the display system, and inputs the vehicle motion status data as driving commands into the motion simulation platform to control the human-machine interaction cockpit to reproduce the motion posture consistent with the real vehicle.

[0060] In one embodiment of the present invention, the downlink transmission step is used to realize the synchronous transmission and immersive reconstruction of multimodal data from a remote real vehicle to a home simulation terminal. Its core is to map the vehicle's operating status and environmental perception information to the simulated cockpit in real time, thereby ensuring that the driver obtains the same perception experience and motion feedback as the real vehicle.

[0061] Specifically, it is accomplished through steps S101, S102, S103 and S104.

[0062] In step S101, the remote real vehicle continuously collects data on its surrounding environment and its own operating status through an onboard sensing and acquisition kit. Specifically, a surround-view camera array outputs multiple video data streams at a high frame rate, which are then stitched together to form a panoramic video stream covering the vehicle's surroundings. A microphone array simultaneously acquires ambient sound and powertrain sound signals. Meanwhile, a high-precision positioning module and an inertial measurement unit output real-time information on the vehicle's three-axis acceleration, three-axis angular velocity, and attitude angles. Furthermore, vehicle status data, including suspension travel, wheel speed, steering angle, and powertrain parameters, are acquired via the CAN bus interface. This data is continuously updated at a high frequency, forming a complete multi-source sensing dataset.

[0063] In step S102, the edge computing gateway at the vehicle end performs unified processing on multi-source data. First, a unified timestamp is added to various types of data through a time synchronization mechanism to eliminate sampling time differences between different sensors. Then, the video stream is compressed and encoded, the audio stream is denoised and encoded, and the status data is structured and encapsulated. Based on this, the gateway dynamically schedules multiple links through a converged communication network, transmitting the processed data in parallel or redundantly through cellular networks, fiber optic links, or other communication methods, thereby improving transmission stability while ensuring real-time performance.

[0064] In step S103, the home simulation device receives the data stream from the remote vehicle, which is then uniformly parsed and decoded by the local digital twin engine. The decoded video stream is output to the immersive display system, presenting the vehicle's surroundings in a panoramic view; the audio stream is processed through spatial audio and output to the sound system to recreate the realistic sound field distribution. This synchronized presentation of visual and auditory information allows the driver to establish a perceptual foundation consistent with the real vehicle environment.

[0065] In step S104, the digital twin engine further processes the received vehicle motion state data and inputs it into the motion calculation module of the haptic simulation platform. This module, based on kinematic and dynamic mapping algorithms, converts the vehicle's acceleration, angular velocity, and attitude changes into displacement and attitude commands for each degree of freedom of the platform, thereby driving the high-dynamic haptic simulation platform to move synchronously. Through this process, the simulated cockpit can accurately reproduce the dynamic response of a real vehicle under acceleration, braking, steering, and complex road conditions, allowing the driver to obtain a feedback experience consistent with real driving at the physical perception level, including the push-back feeling during acceleration, the forward tilt feeling during braking, the lateral tilt during cornering, and vibrations caused by uneven road surfaces.

[0066] Through the coordinated execution of the above steps, this invention achieves low-latency, high-consistency information transmission and perception reconstruction from a real vehicle to a home simulation terminal, providing a stable and highly immersive perception foundation for remote driving.

[0067] S2, Uplink transmission steps: The driver operates the control components in the human-machine interface cockpit to generate control commands. The control commands are sent to the remote real vehicle via the communication link, and the drive-by-wire execution unit performs the corresponding operations.

[0068] In one embodiment of the present invention, the uplink transmission step is used to realize the real-time transmission of the driver's operating intention from the home simulator to the remote real vehicle, and to complete vehicle control through the drive-by-wire system. This step, together with the downlink transmission step, constitutes a remote driving closed loop, enabling the driver to continuously, stably and safely control the real vehicle from within the home simulator cockpit.

[0069] Specifically, it is completed through steps S201, S202, S203, S204, and S205.

[0070] In step S201, the driver is located in a home-based simulated cockpit and inputs driving operations through control components such as the steering wheel, accelerator pedal, brake pedal, and gear lever. Each control component has built-in high-precision position sensors, torque sensors, and travel detection sensors to collect real-time operation signals such as steering wheel angle, steering wheel input torque, accelerator pedal opening, brake pedal travel, analog brake pressure, and gear status. The operation signals are converted into digital control data by a local acquisition module and timestamped to ensure the timing consistency between subsequent control commands and vehicle status data.

[0071] In step S202, the biometric monitoring unit synchronously collects the driver's physiological signals, including heart rate, heart rate variability, electromyography (EMG) signals, electroencephalography (EEG) signals, and postural stress distribution information. By analyzing these physiological signals in real time, the system can determine the driver's current level of attention, fatigue, tension, and operational stability, further assisting in identifying the driver's true operational intentions. For example, when the system detects that the driver is in a state of tension accompanied by slight hand tremors, it can identify this type of high-frequency abnormal input as a non-active driving intention.

[0072] In step S203, the local controller runs a fuzzy shared control algorithm based on the collected control signals and the driver's physiological state to optimize the original control commands. The algorithm first determines the credibility of the driver's intention based on the driver's physiological state, operation amplitude, and rate of change of operation. Then, it filters, limits, and smooths abnormal inputs that may be caused by physiological jitter, fatigue hysteresis, or accidental touches. Finally, it fuses the optimized driver commands with the vehicle's active safety boundaries. These active safety boundaries include, but are not limited to, anti-skid constraints, anti-lock braking constraints, yaw stability constraints, lane keeping constraints, and safe following distance constraints. When there is a conflict between the driver's input and the safety boundaries, the system can correct the control commands according to preset weights to avoid the vehicle entering a dangerous state while respecting the driver's control intentions.

[0073] In step S204, the optimized control data is encapsulated into standard control commands and transmitted to the remote real vehicle via a converged communication network. The standard control commands may include information such as target steering angle, target steering angular velocity, target driving torque, target braking force, target gear position, and auxiliary control status. During transmission, the system sets the control command stream to the highest priority and employs a low-latency transmission strategy to ensure that commands are transmitted before high-bandwidth data such as video and audio streams, thereby improving the real-time performance of remote driving control.

[0074] In step S205, after receiving the standard control command, the on-board execution and control unit of the remote real vehicle performs integrity verification, timeliness verification, and safety and rationality verification on it. After passing the verification, the on-board execution unit converts the control command into execution signals for the drive-by-wire, steering-by-wire, and braking-by-wire systems, and drives the vehicle to complete the corresponding operation. During the execution process, the vehicle continuously collects the execution results and vehicle status data, and transmits them back to the home simulation terminal via downlink, thus forming a closed-loop control process of driver input, vehicle execution, status feedback, and further correction.

[0075] Through the above steps, the present invention can stably convert the driver's operations in the home simulator into actual actions of a remote real vehicle, and reduce the impact of misoperation, fatigue driving and network latency on vehicle control safety by integrating biometric recognition, fuzzy shared control and active safety constraints.

[0076] S3, Network Fluctuation Compensation Step: When a delay or packet loss occurs in the communication link, based on the last received vehicle status data, the dynamic model is used to predict the vehicle's motion trend and generate compensation commands to drive the motion simulation platform to achieve a smooth transition.

[0077] In one embodiment of the present invention, the network fluctuation compensation step is used to solve the problems of unavoidable delay, jitter or packet loss in the actual operation of the communication link, thereby ensuring the continuity and stability of the haptic feedback during remote driving and avoiding discomfort or misleading to the driver due to sudden changes or distortions caused by information interruption.

[0078] Specifically, when the system detects increased latency in the communication link, abnormal data update intervals, or data packet loss, the local digital twin engine on the home simulation terminal automatically enters predictive compensation mode. In this mode, the engine no longer relies entirely on real-time transmitted data, but instead uses the most recently successfully received real vehicle status data as initial conditions for subsequent calculations. The status data includes at least key parameters such as the vehicle's current speed, three-axis acceleration, steering angle, and attitude information.

[0079] During the prediction process, the digital twin engine invokes a pre-built and calibrated vehicle dynamics model to recursively calculate the vehicle's motion trend over a short period. This dynamics model comprehensively considers factors such as vehicle mass distribution, suspension characteristics, tire lateral slip characteristics, and power output response. Using continuous-time or discrete-time integration methods, it predicts the vehicle's speed changes, attitude changes, and spatial trajectory during communication interruptions or delays. To improve prediction stability, it is preferable to employ model constraints with damping terms or introduce state estimation methods to prevent rapid error accumulation during the recursive process.

[0080] Based on the above prediction results, the system generates corresponding motion-sensing drive commands and inputs them into the control system of the high-dynamic motion-sensing simulation platform. After the drive commands are processed by smoothing filtering and acceleration limiting, the control system moves the motion-sensing platform continuously according to the predicted trajectory, so that the driver maintains consistent motion feedback in subjective perception, thereby avoiding sudden stops or abnormal jumps caused by data interruptions.

[0081] Once the communication link is restored, the system enters the state resynchronization phase. At this time, the digital twin engine compares the current attitude of the simulated cabin with the latest received attitude of the real vehicle and calculates the error between the two. To prevent dizziness or discomfort caused by sudden attitude changes, the system employs a progressive correction algorithm to eliminate the error in stages. For example, through interpolation transitions or exponential convergence, the simulated platform's state is gradually adjusted to match that of the real vehicle within a preset time window. This process also incorporates human factors engineering constraints, limiting the rate of change of angular velocity and acceleration to ensure a smooth and natural transition.

[0082] In addition, in some implementations, the system can also dynamically adjust the weights and compensation strategies of the prediction model according to the current network quality. For example, in a high-latency environment, the prediction-dominant weight can be appropriately increased, and in the case of slight jitter, a data and prediction fusion method can be adopted to further improve the robustness of the system.

[0083] Through the aforementioned network fluctuation compensation mechanism, this invention can maintain the consistency and continuity of haptic feedback and visual perception in complex communication environments, effectively reducing the impact of network instability on remote driving experience and safety.

[0084] Example 3 To implement the methods of the above embodiments, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the various steps of the methods described above.

[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0086] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0087] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A home-vehicle-interworking-based immersive remote driving simulation system, characterized by, include: The home simulation terminal includes a motion-sensing simulation platform for providing multi-degree-of-freedom motion, and a human-computer interaction cockpit set on the motion-sensing simulation platform; The vehicle actuator, which is installed on a remote real vehicle, includes a sensing unit for collecting information on the vehicle's own motion posture and the surrounding environment, and a drive-by-wire actuator for executing vehicle driving actions. A converged communication network aggregates multiple communication links to connect the home simulation terminal and the vehicle execution terminal; The vehicle actuator is used to send the collected vehicle motion posture data to the home simulation terminal in real time. The motion simulation platform reproduces the dynamic motion sensation consistent with the real vehicle in real time based on the received vehicle motion posture data.

2. The system of claim 1, wherein, The home simulation terminal also includes a local digital twin controller, which pre-stores a dynamic model corresponding to the real vehicle. This model is used to predict the driving commands of the motion simulation platform based on the last received vehicle state data when the communication link is delayed or interrupted, so as to achieve motion compensation.

3. The system of claim 1, wherein, The vehicle execution terminal also includes an edge computing gateway, which uses multi-link aggregation transmission technology to aggregate at least two heterogeneous communication links to transmit video streams, audio streams, and vehicle status data to the home simulation terminal in real time.

4. The system of claim 1, wherein, The home simulation terminal also includes a biometric monitoring unit, which includes one or more of an electrocardiogram monitoring electrode, a pressure distribution sensor, and a brain-computer interface electrode, for collecting the driver's physiological signals to identify driving intentions and fatigue status.

5. The system of claim 1, wherein, The home simulation terminal also includes a fuzzy shared controller, which is used to fuse driver control commands, biometric recognition results and vehicle active safety boundary information to generate optimized control commands and send them to the vehicle execution terminal.

6. The system of claim 1, wherein, The motion simulation platform is a six-degree-of-freedom electric servo motion platform with a dynamic response delay of no more than 20 ms and a maximum dynamic overload capacity of no less than 0.8 G.

7. The system of claim 1, wherein, The system supports the following two driving modes: Online mode: The home simulator and the vehicle actuator are connected in real time to achieve remote driving control; Training mode: Disconnect from the vehicle execution terminal, and the home simulation terminal runs independently in a virtual environment based on high-precision map data to conduct route rehearsals or driving training.

8. An immersive remote driving simulation method based on home-vehicle interconnection, applied to the system described in any one of claims 1 to 7, characterized in that, include: Downlink transmission steps: The remote real vehicle collects video streams, audio streams, and vehicle motion status data, and sends them to the home simulation terminal through the communication link; The home simulation terminal parses the data, outputs the video stream to the display system, and inputs the vehicle motion state data as driving commands into the motion simulation platform to control the human-computer interaction cockpit to reproduce the motion posture consistent with the real vehicle. Uplink transmission steps: The driver operates the control components in the human-machine interface cockpit to generate control commands, which are then sent to the remote real vehicle via the communication link, where the drive-by-wire execution unit performs the corresponding operations; Network fluctuation compensation steps: When a delay or packet loss occurs in the communication link, the vehicle motion trend is predicted using a dynamic model based on the last received vehicle status data, and a compensation command is generated to drive the motion simulation platform to achieve a smooth transition.

9. The method of claim 8, wherein, The uplink transmission step further includes: While collecting driver control commands, the biometric monitoring unit collects the driver's physiological signals, identifies driving intentions, filters out physiological vibrations in the control commands, and generates optimized control commands by combining the vehicle's active safety boundary information.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 8 or 9.

Images