Device simulation operating system and method based on high-precision ue modeling and motion capture

Abstract

This application relates to a device simulation operating system and method based on high-precision UE modeling and motion capture, belonging to the field of virtual reality technology. The system includes a simulation modeling unit, a motion capture data acquisition unit, a spatial fusion calibration unit, an interactive mapping unit, a collaborative perception unit, and a central processing unit. By combining UE simulation modeling, VR motion capture, virtual-real space dynamic mapping calibration and real-time interactive mapping, cross-vehicle collaborative perception, and synchronization of student status within the same vehicle, this system can achieve device simulation operation training with strong immersion, precise operation, high efficiency for multi-person collaboration, and good spatial positioning synchronization.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a device simulation operating system and method based on high-precision UE modeling and motion capture. Background Technology

[0002] In driver training, hands-on vehicle practice is a core component for improving trainees' driving skills. However, its efficient implementation is constrained by multiple factors, making it difficult to guarantee effective training outcomes. Currently, the main challenge in hands-on vehicle practice for trainees lies in the mismatch between the large number of trainees and the limited number of training vehicles. This results in each trainee not receiving sufficient practice time, significantly diminishing the effectiveness of skill improvement. Furthermore, hands-on vehicle practice places high demands on site specifications and equipment performance. Factors such as the rationality of site layout and the daily maintenance of equipment further limit the convenience and efficiency of training activities, ultimately leading to low training efficiency and failing to meet the needs of large-scale, routine driver education.

[0003] To address the numerous limitations of hands-on vehicle training, existing technologies commonly employ VR (Virtual Reality) platforms for virtual driving training. This virtual simulation replaces physical hands-on practice, alleviating resource pressure on practical vehicle training. However, this existing technology has significant drawbacks. It cannot meet the actual teaching needs of collaborative driving training for large engineering vehicles and other vehicles, failing to fundamentally resolve the current predicament. Specifically, existing VR platform technology cannot accurately define the real activity space of trainees wearing VR devices; it can only set general boundary walls in the virtual space, unable to achieve precise correspondence and synchronous matching between the virtual space and the trainees' real activity space. When multiple people conduct VR driving training collaboratively, the aforementioned technical deficiencies will cause a series of problems: due to the lack of precise spatial calibration, the virtual operation position and the real activity position of each trainee cannot be synchronized, which can easily lead to misalignment of multiple positions and asynchronous actions, and may even cause safety hazards such as collisions between trainees in the real space; at the same time, this deficiency will also seriously weaken the immersion and realism of virtual driving training, and cannot effectively simulate the collaborative cooperation scenarios in real vehicle operation, making it difficult to achieve the same training effect as physical operation, and failing to fundamentally solve the core dilemmas of resource shortage and low efficiency faced by vehicle operation training. Summary of the Invention

[0004] Therefore, it is necessary to provide a device simulation operating system and method based on high-precision UE modeling and motion capture, which has a strong sense of immersion, precise operation, high efficiency for multi-person collaboration, and good spatial positioning synchronization, in order to address the above-mentioned technical problems.

[0005] A device simulation operating system based on high-precision UE modeling and motion capture, the system comprising: The simulation modeling unit is used to construct a virtual operation training field and vehicle model using UE simulation. Each vehicle model is based on a physical bench and virtual buttons, and is adapted for two trainees to operate and train together. The whole includes three parts: the cockpit, the control cabin, and the external structure of the vehicle. The motion capture data acquisition unit is used to collect and preprocess the trainee's full-body posture data and hand movement data in real time using VR-based motion capture equipment. The spatial fusion calibration unit is used to set calibration stakes and use point cloud registration algorithms to fuse and calibrate the motion capture physical space and virtual operation training space, and to provide collision avoidance warnings for trainees in the same vehicle within the physical test bench based on the calibration results and full-body posture data. The interactive mapping unit is used to map the trainee's hand movement data to the virtual operation of the vehicle model, and introduces a Markov chain model to predict the fingertip trajectory. Based on the prediction results, the pre-response of the virtual button is realized, and the vehicle control command is generated through the real-time interactive touch between the fingertip and the virtual button to drive the vehicle model to move and operate in the virtual operation training field according to the real dynamic logic. The collaborative sensing unit is used to achieve accurate information delivery and cross-vehicle collaborative sensing between different trainees, and to achieve status synchronization among trainees in the same vehicle. The central processing unit serves as the computing center of the system, coordinating the collaborative operation of all units.

[0006] In one embodiment, the simulation modeling unit is configured to use UE simulation to construct a three-dimensional virtual operation training field and a vehicle model for two trainees to operate and train together. At the same time, it integrates a physics engine to simulate the mechanical feedback during vehicle driving and the interaction with the virtual operation training field, so that the vehicle model has realistic dynamic characteristics. In each vehicle model, one trainee is defined as the driver, who is located in the cockpit to perform virtual vehicle driving; the other is defined as the operator, who is located in the control room to perform virtual vehicle engineering operations.

[0007] In one embodiment, the VR-based motion capture device includes a wearable motion capture device, a high-speed infrared motion capture camera, and motion capture analysis software. The wearable motion capture device is equipped with optical marker balls and includes VR glasses, a motion capture suit, and finger motion capture gloves. Multiple high-speed infrared motion capture cameras are configured and deployed within a physical platform to collect coordinate data of all optical marker balls in real time. The motion capture analysis software is used to identify each optical marker ball in real time based on the coordinate data collected by the high-speed infrared motion capture cameras, form raw point cloud data, and solve it into full-body posture data and hand motion data containing the three-dimensional coordinates of joints and rotation angles through inverse kinematics algorithms.

[0008] In one embodiment, the spatial fusion calibration unit includes a coordinate system definition module, a calibration stake setting module, a spatial fusion module, and an anti-collision warning module; The coordinate system definition module is used to define the physical coordinate system based on the actual motion capture site where the motion capture equipment is located. The origin is set at the center of the actual motion capture site, and a virtual coordinate system is defined based on the UE. The origin is set as the center of the virtual operation training field; since the motion capture equipment will drift due to environmental interference, the physical coordinate system is a variable coordinate system, while the virtual coordinate system is locked and unchanged when it is constructed, and is a fixed coordinate system. The calibration stake setting module is used to fix four optical marker calibration stakes with unique coded IDs at the four corners of the actual motion capture site, and automatically reads the four optical marker calibration stakes when the system starts. Physical coordinates in and in Preset corresponding virtual coordinates ;in, ; The spatial fusion module is used to first use the coordinates of four optical markers to calibrate the points. The Kabsch-Umeyama point cloud registration algorithm is used to calculate the initial rotation matrix between the two coordinate systems. and the initial translation vector And establish an initial coordinate transformation model, represented as Then, at fixed time intervals, four optical markers are used to calibrate stakes in each time frame. Real-time physical coordinates With fixed virtual coordinates The rotation matrix R(t) and translation vector T(t) between the two coordinate systems are updated to construct a real-time coordinate transformation model, dynamically correct the mapping relationship between virtual and real spaces, and complete high-precision dynamic calibration between the motion capture physical space and the virtual training space to ensure... Deviation from the initial value Keep within the preset range; The collision avoidance warning module is used to calculate the distance vector of fellow students in the physical test bench in real time based on the real-time coordinate transformation model and the full-body posture data of fellow students. When the distance vector is less than the preset safety threshold, the other student is highlighted in the VR display screen of either student, and a collision avoidance warning is triggered by an auditory alarm or vibration.

[0009] In one embodiment, the interaction mapping unit includes a finger joint point cloud construction module, a mechanical feedback hot zone definition module, a fingertip trajectory prediction module, a pre-response module, and a touch detection and instruction generation module; The finger joint point cloud construction module is used to acquire the student's hand movement data and construct the finger joint point cloud in real time. The Mechanical Feedback Hotspot Definition module is used to define an invisible mechanical feedback hotspot for all virtual buttons on a vehicle model in the UE editor. Each mechanical feedback hotspot is an axis-aligned bounding box with attribute parameters including touch depth threshold and rebound force curve. The rebound force curve is a linear or exponential function used to describe the change in virtual resistance during the process of pressing the virtual button with a finger. The fingertip trajectory prediction module first acquires the fingertip position in the finger joint point cloud, and then divides the spatial region around the fingertip into multiple discrete states, represented as follows: , where a single state This represents a 3D grid cell; the 3D grid cell where the fingertip is currently located is the current state. , The number of 3D mesh units around the fingertip is determined; then, a transition probability matrix is ​​constructed using an online-learned or pre-built first-order Markov chain model. , elements in Indicates starting from the current state Transition to the next state The probability; then at each time frame, based on the current state of the fingertip. The most likely state of the fingertip in the next time frame is calculated using the transition probability matrix, and is represented as follows: It outputs the center coordinates of the 3D grid cell corresponding to the most likely state as the predicted fingertip position; The pre-response module is used to perform pre-touch detection by predicting the fingertip position and all mechanical feedback hot zones after completing the touch detection of the current time frame. If the predicted fingertip position falls within the range of a certain mechanical feedback hot zone, the corresponding virtual button is rendered as a pre-highlighted state in advance to prompt the student to make contact. At the same time, the system prepares the vibration feedback waveform data of the corresponding virtual button based on the rebound force curve in advance and caches it in the drive buffer of the finger motion capture glove in the motion capture device to ensure that the vibration is output without delay when the fingertip actually arrives. The touch detection and command generation module first calculates in real time whether the actual fingertip position in each time frame enters the range of any mechanical feedback heat zone. If it enters and the actual fingertip position coincides with the predicted fingertip position, the corresponding virtual button changes from a pre-highlighted state to a high-highlighted state; if it enters but the actual fingertip position does not coincide with the predicted fingertip position, the corresponding virtual button changes from a normally dark state to a high-highlighted state; if it does not enter, the corresponding virtual button remains in a normally dark state. Then, it records the touch depth and touch speed of the fingertip touching the mechanical feedback heat zone. When the touch depth is greater than or equal to the touch depth threshold of the corresponding mechanical feedback heat zone, and the touch speed exceeds the preset positive speed threshold, it is determined to be a valid press. The system immediately generates the corresponding vehicle control command and sends it to the simulation modeling unit to drive the UE to play the virtual button press animation. The preloaded vibration signal is sent to the fingertip through the drive buffer of the finger motion capture glove, and the vehicle model is driven to move and operate in the virtual operation training field according to the real dynamic logic. Otherwise, it is determined to be a false touch and no vehicle control command is generated.

[0010] In one embodiment, the collaborative perception unit includes a role-based data subscription module, an operation status encoding broadcast module, and a vehicle status synchronization module; The role-based data subscription module is used to obtain the student role-permission table maintained by the central processing unit. When a student logs into the system, a role tag is automatically assigned to them. Based on the role tag, each student is restricted to subscribing to and rendering data within their permissions. The operation status encoding and broadcasting module is used to encode the operation status of each vehicle model into a 2D texture card and broadcast it to the local area network. After each student's UE client receives the 2D texture card of other vehicle models from the local area network, it renders it as a flat floating UI in a corner of the VR display screen and allows the student to turn it on or off, thus realizing cross-vehicle collaborative perception. The vehicle status synchronization module is used to broadcast the core state machine data of the vehicle model to the students in the same vehicle at a fixed frequency, and the students' UE clients in the same vehicle will render it independently to ensure that the current state of the vehicle model in the VR display screen of the students in the same vehicle is consistent and synchronized.

[0011] In one embodiment, the central processing unit consists of a high-performance rendering server cluster, which integrates the software services required by the simulation modeling unit, motion capture data acquisition unit, spatial fusion calibration unit, interactive mapping unit, and collaborative perception unit. It is responsible for receiving upper-level scheduling and control instructions, coordinating the collaborative operation of all units, and ensuring that the entire system operates stably with low latency.

[0012] A device simulation operation method based on high-precision UE modeling and motion capture, the method being implemented based on the aforementioned device simulation operating system based on high-precision UE modeling and motion capture, includes the following steps: The simulation modeling unit uses UE simulation to construct a virtual operation training field and vehicle model; each vehicle model is based on a physical bench and virtual buttons, and is suitable for two trainees to operate and train together. The whole includes three parts: the cockpit, the control cabin and the external structure of the vehicle. The motion capture data acquisition unit uses VR-based motion capture equipment to collect and preprocess the trainee's full-body posture data and hand movement data in real time. The calibration stakes are set by the spatial fusion calibration unit and the point cloud registration algorithm is used to fuse and calibrate the motion capture physical space and the virtual operation training space. Based on the calibration results and whole-body posture data, collision avoidance warning is given to the students in the same vehicle in the physical test bench. The interactive mapping unit maps the trainee's hand movement data to the virtual operation of the vehicle model, and introduces a Markov chain model to predict the fingertip trajectory. Based on the prediction results, the virtual button pre-response is realized. Through the real-time interactive touch between the fingertip and the virtual button, vehicle control commands are generated to drive the vehicle model to move and operate in the virtual operation training field according to the real dynamic logic. The collaborative sensing unit enables precise information delivery and cross-vehicle collaborative sensing among different trainees, and achieves status synchronization among trainees in the same vehicle. The central processing unit acts as the computing center of the system, coordinating the collaborative operation of all units.

[0013] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: The simulation modeling unit uses UE simulation to construct a virtual operation training field and vehicle model; each vehicle model is based on a physical bench and virtual buttons, and is suitable for two trainees to operate and train together. The whole includes three parts: the cockpit, the control cabin and the external structure of the vehicle. The motion capture data acquisition unit uses VR-based motion capture equipment to collect and preprocess the trainee's full-body posture data and hand movement data in real time. The calibration stakes are set by the spatial fusion calibration unit and the point cloud registration algorithm is used to fuse and calibrate the motion capture physical space and the virtual operation training space. Based on the calibration results and whole-body posture data, collision avoidance warning is given to the students in the same vehicle in the physical test bench. The interactive mapping unit maps the trainee's hand movement data to the virtual operation of the vehicle model, and introduces a Markov chain model to predict the fingertip trajectory. Based on the prediction results, the virtual button pre-response is realized. Through the real-time interactive touch between the fingertip and the virtual button, vehicle control commands are generated to drive the vehicle model to move and operate in the virtual operation training field according to the real dynamic logic. The collaborative sensing unit enables precise information delivery and cross-vehicle collaborative sensing among different trainees, and achieves status synchronization among trainees in the same vehicle. The central processing unit acts as the computing center of the system, coordinating the collaborative operation of all units.

[0014] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: The simulation modeling unit uses UE simulation to construct a virtual operation training field and vehicle model; each vehicle model is based on a physical bench and virtual buttons, and is suitable for two trainees to operate and train together. The whole includes three parts: the cockpit, the control cabin and the external structure of the vehicle. The motion capture data acquisition unit uses VR-based motion capture equipment to collect and preprocess the trainee's full-body posture data and hand movement data in real time. The calibration stakes are set by the spatial fusion calibration unit and the point cloud registration algorithm is used to fuse and calibrate the motion capture physical space and the virtual operation training space. Based on the calibration results and whole-body posture data, collision avoidance warning is given to the students in the same vehicle in the physical test bench. The interactive mapping unit maps the trainee's hand movement data to the virtual operation of the vehicle model, and introduces a Markov chain model to predict the fingertip trajectory. Based on the prediction results, the virtual button pre-response is realized. Through the real-time interactive touch between the fingertip and the virtual button, vehicle control commands are generated to drive the vehicle model to move and operate in the virtual operation training field according to the real dynamic logic. The collaborative sensing unit enables precise information delivery and cross-vehicle collaborative sensing among different trainees, and achieves status synchronization among trainees in the same vehicle. The central processing unit acts as the computing center of the system, coordinating the collaborative operation of all units.

[0015] The aforementioned device simulation operating system and method based on high-precision UE modeling and motion capture has the following advantages compared to existing technologies: 1. By constructing a virtual operation training field and vehicle model through UE simulation, and combining physical test benches and VR motion capture equipment, the trainees' physical movements are mapped to the virtual scene, breaking down the barriers between the virtual and the real, providing a strong sense of immersion, and efficiently simulating the real vehicle driving and engineering operation environment.

[0016] 2. The motion capture data acquisition unit accurately collects and preprocesses the student's full-body posture and fine hand movements data. Combined with the spatial fusion calibration unit, it completes the virtual-real space fusion calibration. Combined with fingertip trajectory prediction technology, it can realize virtual button pre-response, making the mapping between hand movements and virtual operations smoother and more accurate, and significantly reducing operation delay and the probability of misoperation.

[0017] 3. The system is compatible with collaborative operation between two trainees in the same vehicle. Combined with the collaborative sensing unit, it can achieve accurate information delivery between different trainees, cross-vehicle collaborative sensing, and synchronization of trainee status in the same vehicle, ensuring information exchange and action coordination during multi-person operation. At the same time, the central processing unit acts as the computing center to coordinate the efficient operation of each unit, allowing seamless connection between simulation modeling, data acquisition, spatial calibration, interactive mapping, and other links, avoiding lag and asynchrony problems in collaborative training, making multi-person collaborative training more orderly and efficient. It can not only cultivate trainees' collaborative operation ability, but also improve the overall efficiency of training. It is suitable for large-scale and standardized equipment simulation operation training scenarios, and can fundamentally solve the problems of resource shortage and low efficiency faced by practical training of vehicles and other equipment. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structure of a device simulation operating system based on high-precision UE modeling and motion capture in one embodiment. Figure 2 This is a flowchart illustrating a device simulation operation method based on high-precision UE modeling and motion capture in one embodiment. Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] In one embodiment, such as Figure 1As shown, a device simulation operating system based on high-precision UE (virtual engine) modeling and motion capture is provided. It includes a simulation modeling unit for constructing a virtual operation training field and vehicle models using UE simulation. Each vehicle model is based on a physical platform and virtual buttons, suitable for collaborative operation training of two trainees, and comprises a cockpit, an operating cabin, and the vehicle's external structure. A motion capture data acquisition unit is used to collect and preprocess the trainees' full-body posture data and hand movement data in real time using VR-based motion capture equipment. A spatial fusion calibration unit is used to set calibration stakes and use a point cloud registration algorithm to fuse and calibrate the motion capture physical space and the virtual operation training space, based on... The calibration results and full-body posture data are used for collision avoidance warnings for fellow trainees within a physical test bench. The interactive mapping unit maps trainees' hand movement data to the virtual operation of the vehicle model, and introduces a Markov chain model to predict fingertip trajectories. Based on the prediction results, it implements pre-response for virtual buttons, and generates vehicle control commands through real-time interaction between fingertips and virtual buttons, driving the vehicle model to move and operate in the virtual operation training field according to real dynamic logic. The collaborative perception unit enables accurate information delivery and cross-vehicle collaborative perception between different trainees, and achieves state synchronization among trainees in the same vehicle. The central processing unit acts as the system's computing center, coordinating the collaborative operation of all units. The following provides a detailed introduction to each unit.

[0021] 1. Simulation Modeling Unit: Configured to use UE simulation to construct a 3D virtual operation training field and a vehicle model for collaborative operation training between two trainees. It also integrates a physics engine to simulate the mechanical feedback during vehicle operation (such as bumps and acceleration sensations) and interactions with the virtual operation training field (such as animated feedback from opening and closing doors and pressing virtual buttons). This gives the vehicle model realistic dynamic characteristics. Within each vehicle model, one trainee is defined as the driver, residing in the cockpit to perform virtual driving; the other is defined as the operator (including the instrument panel, center console, spectrum display, keyboard array, etc.), residing in the control panel to perform virtual engineering tasks.

[0022] In this embodiment, a 3D virtual operation training field and vehicle model can be constructed using Unreal Engine 5.1 or later. The model accuracy reaches L3 level (high-precision restoration level), the texture resolution reaches 4K, and real-time ray tracing is supported.

[0023] 2. Motion Capture Data Acquisition Unit: Responsible for acquiring trainees' full-body posture and fine hand movements using VR-based motion capture equipment. The VR-based motion capture equipment includes wearable motion capture devices, high-speed infrared motion capture cameras, and motion capture analysis software. The wearable motion capture device is equipped with optical marker balls, and includes VR glasses, a motion capture suit, and finger motion capture gloves. Multiple high-speed infrared motion capture cameras are configured and deployed within a physical platform to acquire the coordinate data of all optical marker balls in real time. The motion capture analysis software is used to identify each optical marker ball in real time based on the coordinate acquisition data from the high-speed infrared motion capture cameras, forming raw point cloud data, and then using inverse kinematics algorithms to calculate full-body posture data and hand movement data, including the three-dimensional coordinates of joints and rotation angles.

[0024] In this embodiment, the HTC VIVE Pro 2 VR glasses are selected. Six passive optical marker balls are evenly distributed on the glasses' frame. The motion capture suit has a total of 18 optical marker balls distributed at the main joint points (shoulder, elbow, wrist, knee, and ankle). The finger motion capture gloves also integrate 18 fiber optic bending sensors, with 3 optical marker balls distributed on the back of the hand. Eight to 12 high-speed infrared motion capture cameras are deployed, using the MC1300 series from Measurement Technology, with a sampling frequency of no less than 120Hz. The motion capture analysis software uses Motive 3.0, with output data in BVH or FBX format, sent to the central processing unit via UDP multicast.

[0025] 3. The spatial fusion calibration unit includes a coordinate system definition module, a calibration stake setting module, a spatial fusion module, and a collision avoidance warning module.

[0026] The coordinate system definition module is used to define the physical coordinate system based on the actual motion capture site where the motion capture equipment is located. The origin is set at the center of the actual motion capture site (in millimeters), and a virtual coordinate system is defined based on the UE. The origin is set as the center of the virtual operation training field (unit is Unreal Engine unit (1uu=1cm)); since the motion capture equipment will drift due to environmental interference (such as temperature, vibration or small displacement), the physical coordinate system is a variable coordinate system, while the virtual coordinate system is locked and unchanged when it is built, and is a fixed coordinate system.

[0027] The calibration stake setting module is used to fix four optical marker calibration stakes with unique coded IDs at the four corners of the actual motion capture site, and automatically reads the four optical marker calibration stakes when the system starts. Physical coordinates in and in Preset corresponding virtual coordinates ;in, .

[0028] The spatial fusion module is used to first use the coordinates of four optical markers to calibrate the points. The Kabsch-Umeyama point cloud registration algorithm is used to calculate the initial rotation matrix between the two coordinate systems. and the initial translation vector And establish an initial coordinate transformation model, represented as Then, according to a fixed time interval (the time interval is set in this embodiment). ), using 4 optical marker staking posts in each time frame Real-time physical coordinates With fixed virtual coordinates The rotation matrix R(t) and translation vector T(t) between the two coordinate systems are updated to construct a real-time coordinate transformation model, dynamically correct the mapping relationship between virtual and real spaces, and complete high-precision dynamic calibration between the motion capture physical space and the virtual training space to ensure... Deviation from the initial value Keep it within a preset range (e.g., 2 mm).

[0029] The collision avoidance warning module calculates the distance vector of fellow passengers within the physical test platform in real time based on a real-time coordinate transformation model and the full-body posture data of the passengers in the same vehicle. When the distance vector is less than a preset safety threshold, the system highlights the other passenger in the VR display of either passenger and triggers a collision avoidance warning via auditory alarm or vibration. For example, when the distance vector of fellow passengers within the physical test platform is less than the safety threshold of 500 mm, the system highlights the other passenger with a bright red outline in the VR display of either passenger, triggers an auditory alarm, and, if necessary, sends a brief vibration signal to the motion capture equipment via the SDK (Software Development Kit) in the central server.

[0030] 4. The interactive mapping unit includes a finger joint point cloud construction module, a mechanical feedback hot zone definition module, a fingertip trajectory prediction module, a pre-response module, and a touch detection and command generation module.

[0031] The finger joint point cloud construction module is used to acquire the trainee's hand movement data (including precise position and rotation information) and construct the finger joint point cloud in real time. Specifically, the system records the fingertip position in each frame at a frame rate of 120Hz.

[0032] The Mechanical Feedback Hotspot Definition module is used to define an invisible mechanical feedback hotspot for all virtual buttons on the vehicle model (including each key on the keyboard or each touch point on the spectrum display) in the UE editor. Each mechanical feedback hotspot is an axis-aligned bounding box with attribute parameters including a touch depth threshold (normally set to 3 mm to simulate the physical travel of the virtual button) and a rebound force curve. The rebound force curve is a linear or exponential function used to describe the change in virtual resistance during the process of pressing the virtual button with a finger.

[0033] The fingertip trajectory prediction module first acquires the fingertip position in the finger joint point cloud, and then divides the spatial region around the fingertip into multiple discrete states, represented as follows: , where a single state This represents a 3D grid unit, which can be set to 5mm×5mm×5mm. The 3D grid unit currently occupied by the fingertip is the current state. , The number of 3D mesh units around the fingertip is determined; then, a transition probability matrix is ​​constructed using an online-learned or pre-built first-order Markov chain model. , elements in Indicates starting from the current state Transition to the next state The probability matrix can be obtained through offline training with a large amount of historical operation data, or it can be updated in real time during system operation according to the user's personal operation habits (using maximum likelihood estimation); then at each time frame, based on the current state of the fingertip... The most likely state of the fingertip in the next time frame is calculated using the transition probability matrix, and is represented as follows: It outputs the center coordinates of the 3D grid cell corresponding to the most likely state as the predicted fingertip position.

[0034] The pre-response module is used to perform pre-touch detection by predicting the fingertip position and all mechanical feedback hot zones after completing the touch detection of the current time frame. If the predicted fingertip position falls within the range of a certain mechanical feedback hot zone, the corresponding virtual button is pre-highlighted (such as edge glow) to prompt the student that they are about to touch it. At the same time, the system prepares the vibration feedback waveform data of the corresponding virtual button based on the rebound force curve in advance and caches it in the drive buffer of the finger motion capture glove in the motion capture device to ensure that the vibration is output without delay when the fingertip actually arrives.

[0035] The touch detection and command generation module first calculates in real time whether the actual fingertip position in each time frame enters the range of any mechanical feedback heat zone. If it enters and the actual fingertip position coincides with the predicted fingertip position, the corresponding virtual button changes from a pre-highlighted state to a high-highlighted state, possibly accompanied by a color change. If it enters but the actual fingertip position does not coincide with the predicted fingertip position, the corresponding virtual button changes from a normally dark state to a high-highlighted state. If it does not enter, the corresponding virtual button remains in a normally dark state. Then, it records the touch depth and touch speed of the fingertip touching the mechanical feedback heat zone. When the touch depth is greater than or equal to the touch depth threshold of the corresponding mechanical feedback heat zone, and the touch speed exceeds the preset positive speed threshold, it is determined to be a valid press. The system immediately generates the corresponding vehicle control command and sends it to the simulation modeling unit to drive the UE to play the virtual button press animation. The preloaded vibration signal is sent to the fingertip through the drive buffer of the finger motion capture glove, and the vehicle model is driven to move and operate in the virtual operation training field according to the real dynamic logic. Otherwise, it is determined to be a false touch and no vehicle control command is generated.

[0036] 5. The collaborative perception unit includes a role-based data subscription module, an operation status coding broadcast module, and a vehicle status synchronization module.

[0037] The role-based data subscription module is used to obtain the student role-permission table maintained by the central processing unit. When a student logs into the system, a role tag is automatically assigned to them. Based on the role tag, each student is restricted to subscribing to and rendering data within their own permissions.

[0038] The operation status encoding and broadcasting module encodes the operation status of each vehicle model (such as parameter panel status and spectrum display waveform data) into 2D texture cards and broadcasts them to the local area network (LAN). Each student's UE client receives the 2D texture cards of other vehicle models from the LAN and renders them as a planar floating UI (user interface) in a corner of the VR display, which the student can then turn on or off, enabling cross-vehicle collaborative perception. For example, when vehicle A selects "View vehicle B status" via the controller menu, the system does not attempt to render the complete 3D model of vehicle B in vehicle A's scene. Instead, it extracts the current parameter panel screenshot texture (256×256 pixels) and the current spectrum display waveform data from vehicle B's status database. The waveform data is then encoded into a simplified 2D line graph texture. These two textures are combined into a 512x512 2D texture card, which is broadcast to the LAN at 10fps via UDP. After receiving the 2D texture card, the UE5 client of vehicle A renders it as a planar floating UI in the upper right corner of the student's field of view (without obstructing the main line of sight). Students can choose to keep this UI in their field of vision or turn it off.

[0039] The vehicle state synchronization module broadcasts core state machine data of the vehicle model, including its position, speed, engine speed, and instrument readings, to fellow learners at a fixed frequency (50Hz). Each learner's UE client then independently renders this data from their own perspective, ensuring that the current state of the vehicle model displayed in their VR view is consistent and synchronized. This approach avoids the complexity and network burden of directly synchronizing animations, ensuring consistency at both the logical and data levels.

[0040] 6. Central Processing Unit: Consists of a high-performance rendering server cluster, with hardware configuration including dual Intel Xeon Platinum 8380 processors, four NVIDIA RTX A6000 graphics cards, 128GB DDR4 memory, and a 10Gb fiber optic network card. Software-wise, it integrates the software services required by the simulation modeling unit, motion capture data acquisition unit, spatial fusion calibration unit, interactive mapping unit, and collaborative perception unit. It is responsible for receiving upper-level scheduling and control commands, coordinating the collaborative operation of all units, and ensuring stable operation of the entire system with low latency (end-to-end latency <50ms).

[0041] The aforementioned device simulation operating system, based on high-precision UE modeling and motion capture, constructs a virtual operation training field and vehicle model through UE simulation. Combined with a physical test bench and VR motion capture equipment, it maps the trainee's physical movements to the virtual scene, breaking down the barriers between virtual and reality, providing a strong sense of immersion, and efficiently simulating real vehicle driving and engineering operation environments. The motion capture data acquisition unit accurately collects and preprocesses the trainee's full-body posture and fine hand movements data, and the spatial fusion calibration unit completes virtual-real spatial fusion calibration. Combined with fingertip trajectory prediction technology, it enables virtual button pre-response, making the mapping between hand movements and virtual operations smoother and more accurate, significantly reducing operation latency and the probability of misoperation. The system is compatible with two trainees. Collaborative operation within the same vehicle, combined with a collaborative sensing unit, enables precise information delivery between different trainees, cross-vehicle collaborative sensing, and synchronization of trainee status within the same vehicle, ensuring information exchange and action coordination during multi-person operation. Simultaneously, a central processing unit acts as the computing center, coordinating the efficient operation of each unit and seamlessly connecting simulation modeling, data acquisition, spatial calibration, and interactive mapping, avoiding stuttering and asynchrony issues in collaborative training. This makes multi-person collaborative training more orderly and efficient, cultivating trainees' collaborative operation skills and improving overall training efficiency. It is suitable for large-scale, standardized equipment simulation operation training scenarios, fundamentally solving the resource shortage and low efficiency problems faced in practical training of vehicles and other equipment.

[0042] In one embodiment, such as Figure 2 As shown, a device simulation operation method based on high-precision UE modeling and motion capture is provided. This method is implemented based on the aforementioned device simulation operating system based on high-precision UE modeling and motion capture, and includes the following steps: The simulation modeling unit uses UE simulation to construct a virtual operation training field and vehicle model; each vehicle model is based on a physical bench and virtual buttons, and is suitable for two trainees to operate and train together. The whole includes three parts: the cockpit, the control cabin and the external structure of the vehicle. The motion capture data acquisition unit uses VR-based motion capture equipment to collect and preprocess the trainee's full-body posture data and hand movement data in real time. The calibration stakes are set by the spatial fusion calibration unit and the point cloud registration algorithm is used to fuse and calibrate the motion capture physical space and the virtual operation training space. Based on the calibration results and whole-body posture data, collision avoidance warning is given to the students in the same vehicle in the physical test bench. The interactive mapping unit maps the trainee's hand movement data to the virtual operation of the vehicle model, and introduces a Markov chain model to predict the fingertip trajectory. Based on the prediction results, the virtual button pre-response is realized. Through the real-time interactive touch between the fingertip and the virtual button, vehicle control commands are generated to drive the vehicle model to move and operate in the virtual operation training field according to the real dynamic logic. The collaborative sensing unit enables precise information delivery and cross-vehicle collaborative sensing among different trainees, and achieves status synchronization among trainees in the same vehicle. The central processing unit acts as the computing center of the system, coordinating the collaborative operation of all units.

[0043] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a device simulation operation method based on high-precision UE modeling and motion capture. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0044] Those skilled in the art will understand that Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0045] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps: The simulation modeling unit uses UE simulation to construct a virtual operation training field and vehicle model; each vehicle model is based on a physical bench and virtual buttons, and is suitable for two trainees to operate and train together. The whole includes three parts: the cockpit, the control cabin and the external structure of the vehicle. The motion capture data acquisition unit uses VR-based motion capture equipment to collect and preprocess the trainee's full-body posture data and hand movement data in real time. The calibration stakes are set by the spatial fusion calibration unit and the point cloud registration algorithm is used to fuse and calibrate the motion capture physical space and the virtual operation training space. Based on the calibration results and whole-body posture data, collision avoidance warning is given to the students in the same vehicle in the physical test bench. The interactive mapping unit maps the trainee's hand movement data to the virtual operation of the vehicle model, and introduces a Markov chain model to predict the fingertip trajectory. Based on the prediction results, the virtual button pre-response is realized. Through the real-time interactive touch between the fingertip and the virtual button, vehicle control commands are generated to drive the vehicle model to move and operate in the virtual operation training field according to the real dynamic logic. The collaborative sensing unit enables precise information delivery and cross-vehicle collaborative sensing among different trainees, and achieves status synchronization among trainees in the same vehicle. The central processing unit acts as the computing center of the system, coordinating the collaborative operation of all units.

[0046] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: The simulation modeling unit uses UE simulation to construct a virtual operation training field and vehicle model; each vehicle model is based on a physical bench and virtual buttons, and is suitable for two trainees to operate and train together. The whole includes three parts: the cockpit, the control cabin and the external structure of the vehicle. The motion capture data acquisition unit uses VR-based motion capture equipment to collect and preprocess the trainee's full-body posture data and hand movement data in real time. The calibration stakes are set by the spatial fusion calibration unit and the point cloud registration algorithm is used to fuse and calibrate the motion capture physical space and the virtual operation training space. Based on the calibration results and whole-body posture data, collision avoidance warning is given to the students in the same vehicle in the physical test bench. The interactive mapping unit maps the trainee's hand movement data to the virtual operation of the vehicle model, and introduces a Markov chain model to predict the fingertip trajectory. Based on the prediction results, the virtual button pre-response is realized. Through the real-time interactive touch between the fingertip and the virtual button, vehicle control commands are generated to drive the vehicle model to move and operate in the virtual operation training field according to the real dynamic logic. The collaborative sensing unit enables precise information delivery and cross-vehicle collaborative sensing among different trainees, and achieves status synchronization among trainees in the same vehicle. The central processing unit acts as the computing center of the system, coordinating the collaborative operation of all units.

[0047] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0048] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0049] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A device simulation operating system based on high-precision UE modeling and motion capture, characterized in that, The system includes: The simulation modeling unit is used to construct a virtual operation training field and vehicle model using UE simulation. Each vehicle model is based on a physical bench and virtual buttons, and is adapted for two trainees to operate and train together. The whole includes three parts: the cockpit, the control cabin, and the external structure of the vehicle. The motion capture data acquisition unit is used to collect and preprocess the trainee's full-body posture data and hand movement data in real time using VR-based motion capture equipment. The spatial fusion calibration unit is used to set calibration stakes and use point cloud registration algorithms to fuse and calibrate the motion capture physical space and virtual operation training space, and to provide collision avoidance warnings for trainees in the same vehicle within the physical test bench based on the calibration results and full-body posture data. The interactive mapping unit is used to map the trainee's hand movement data to the virtual operation of the vehicle model, and introduces a Markov chain model to predict the fingertip trajectory. Based on the prediction results, the pre-response of the virtual button is realized, and the vehicle control command is generated through the real-time interactive touch between the fingertip and the virtual button to drive the vehicle model to move and operate in the virtual operation training field according to the real dynamic logic. The collaborative sensing unit is used to achieve accurate information delivery and cross-vehicle collaborative sensing between different trainees, and to achieve status synchronization among trainees in the same vehicle. The central processing unit serves as the computing center of the system, coordinating the collaborative operation of all units.

2. The device simulation operating system based on high-precision UE modeling and motion capture as described in claim 1, characterized in that, The simulation modeling unit is configured to use UE simulation to construct a three-dimensional virtual operation training field and a vehicle model for two trainees to operate and train together. At the same time, it integrates a physics engine to simulate the mechanical feedback during vehicle driving and the interaction with the virtual operation training field, so that the vehicle model has realistic dynamic characteristics. In each vehicle model, one trainee is defined as the driver, who performs virtual driving in the cockpit; the other is defined as the operator, who performs virtual engineering operations in the control room.

3. The device simulation operating system based on high-precision UE modeling and motion capture as described in claim 1, characterized in that, The VR-based motion capture device includes a wearable motion capture device, a high-speed infrared motion capture camera, and motion capture analysis software. The wearable motion capture device is equipped with optical marker balls and includes VR glasses, a motion capture suit, and finger motion capture gloves. Multiple high-speed infrared motion capture cameras are configured and deployed within a physical platform to collect the coordinate data of all optical marker balls in real time. The motion capture analysis software is used to identify each optical marker ball in real time based on the coordinate data collected by the high-speed infrared motion capture cameras, form raw point cloud data, and use inverse kinematics algorithms to calculate full-body posture data and hand motion data containing the three-dimensional coordinates of joints and rotation angles.

4. The device simulation operating system based on high-precision UE modeling and motion capture as described in claim 1, characterized in that, The spatial fusion calibration unit includes a coordinate system definition module, a calibration stake setting module, a spatial fusion module, and an anti-collision warning module. The coordinate system definition module is used to define a physical coordinate system based on the actual motion capture site where the motion capture equipment is located. The origin is set at the center of the actual motion capture site, and a virtual coordinate system is defined based on the UE. The origin is set as the center of the virtual operation training field; since the motion capture equipment will drift due to environmental interference, the physical coordinate system is a variable coordinate system, while the virtual coordinate system is locked and unchanged when it is constructed, and is a fixed coordinate system. The calibration stake setting module is used to fix four optical marker calibration stakes with unique coded IDs at the four corners of the actual motion capture site, and automatically reads the four optical marker calibration stakes when the system starts. Physical coordinates in and in Preset corresponding virtual coordinates ;in, ; The spatial fusion module is used to first use the coordinates of four optical marker calibrators to... The Kabsch-Umeyama point cloud registration algorithm is used to calculate the initial rotation matrix between the two coordinate systems. and initial translation vector And establish an initial coordinate transformation model, represented as Then, at fixed time intervals, four optical markers are used to calibrate stakes in each time frame. Real-time physical coordinates With fixed virtual coordinates The rotation matrix R(t) and translation vector T(t) between the two coordinate systems are updated to construct a real-time coordinate transformation model, dynamically correct the mapping relationship between virtual and real spaces, and complete high-precision dynamic calibration between the motion capture physical space and the virtual training space to ensure... Deviation from the initial value Keep within the preset range; The collision avoidance warning module is used to calculate the distance vector of the students in the same vehicle within the physical platform in real time based on the real-time coordinate transformation model and the full-body posture data of the students in the same vehicle. When the distance vector is less than a preset safety threshold, the other student is highlighted in the VR display screen of either student, and a collision avoidance warning is triggered by an auditory alarm or vibration.

5. The device simulation operating system based on high-precision UE modeling and motion capture as described in claim 1, characterized in that, The interactive mapping unit includes a finger joint point cloud construction module, a mechanical feedback heat zone definition module, a fingertip trajectory prediction module, a pre-response module, and a touch detection and command generation module. The finger joint point cloud construction module is used to acquire the student's hand movement data and construct the finger joint point cloud in real time. The mechanical feedback hotspot definition module is used to define an invisible mechanical feedback hotspot for all virtual buttons of the vehicle model in the UE editor. Each mechanical feedback hotspot is an axis-aligned bounding box, and its attribute parameters include touch depth threshold and rebound force curve. The rebound force curve is a linear or exponential function used to describe the change in virtual resistance during the process of pressing the virtual button with a finger. The fingertip trajectory prediction module first acquires the fingertip position in the finger joint point cloud, and then divides the spatial region around the fingertip into multiple discrete states, represented as follows: , where a single state This represents a 3D grid cell; the 3D grid cell where the fingertip is currently located is the current state. , The number of 3D mesh units around the fingertip is used; then, a transition probability matrix is ​​constructed using an online-learned or pre-built first-order Markov chain model. , elements in Indicates starting from the current state Transition to the next state The probability; then at each time frame, based on the current state of the fingertip. The most likely state of the fingertip in the next time frame is calculated using the transition probability matrix, and is represented as follows: It outputs the center coordinates of the 3D grid cell corresponding to the most likely state as the predicted fingertip position; The pre-response module is used to perform pre-touch detection by predicting the fingertip position and all mechanical feedback hot zones after completing the touch detection of the current time frame. If the predicted fingertip position falls within the range of a certain mechanical feedback hot zone, the corresponding virtual button is rendered as a pre-highlighted state in advance to prompt the student to make contact. At the same time, the system prepares the vibration feedback waveform data of the corresponding virtual button based on the rebound force curve in advance and caches it in the drive buffer of the finger motion capture glove in the motion capture device to ensure that the vibration is output without delay when the fingertip actually arrives. The touch detection and command generation module first calculates in real time whether the actual fingertip position of each time frame enters the range of any mechanical feedback heat zone. If it enters and the actual fingertip position coincides with the predicted fingertip position, the corresponding virtual button changes from a pre-highlighted state to a high-highlighted state; if it enters but the actual fingertip position does not coincide with the predicted fingertip position, the corresponding virtual button changes from a normally dark state to a high-highlighted state; if it does not enter, the corresponding virtual button remains in a normally dark state. Then, it records the touch depth and touch speed of the fingertip touching the mechanical feedback heat zone. When the touch depth is greater than or equal to the touch depth threshold of the corresponding mechanical feedback heat zone and the touch speed exceeds the preset positive speed threshold, it is determined to be a valid press. The system immediately generates the corresponding vehicle control command and sends it to the simulation modeling unit to drive the UE to play the virtual button press animation. The preloaded vibration signal is sent to the fingertip through the drive buffer of the finger motion capture glove, and the vehicle model is driven to move and operate in the virtual operation training field according to the real dynamic logic. Otherwise, it will be judged as a mistake and no vehicle control command will be generated.

6. The device simulation operating system based on high-precision UE modeling and motion capture as described in claim 1, characterized in that, The collaborative perception unit includes a role-based data subscription module, an operation status encoding broadcast module, and a vehicle status synchronization module. The role-based data subscription module is used to obtain the student role-permission table maintained by the central processing unit. When a student logs into the system, a role tag is automatically assigned to them. Based on the role tag, each student is restricted to subscribing to and rendering data within their permissions. The operation status encoding and broadcasting module is used to encode the operation status of each vehicle model into a 2D texture card and broadcast it to the local area network. After each student's UE client receives the 2D texture card of other vehicle models from the local area network, it renders it as a flat floating UI in a corner of the VR display screen and allows the student to turn it on or off, thereby realizing cross-vehicle collaborative perception. The vehicle status synchronization module is used to broadcast the core state machine data of the vehicle model to the students in the same vehicle at a fixed frequency, and the students' UE clients in the same vehicle will render it independently to ensure that the current state of the vehicle model in the VR display screen of the students in the same vehicle is consistent and synchronized.

7. The device simulation operating system based on high-precision UE modeling and motion capture as described in claim 1, characterized in that, The central processing unit consists of a high-performance rendering server cluster, which integrates the software services required by the simulation modeling unit, motion capture data acquisition unit, spatial fusion calibration unit, interactive mapping unit, and collaborative perception unit. It is responsible for receiving upper-level scheduling and control instructions, coordinating the collaborative operation of all units, and ensuring the stable operation of the entire system with low latency.

8. A device simulation operation method based on high-precision UE modeling and motion capture, characterized in that, The method is implemented based on the device simulation operating system based on high-precision UE modeling and motion capture as described in any one of claims 1-7, and includes the following steps: The simulation modeling unit uses UE simulation to construct a virtual operation training field and vehicle model; each vehicle model is based on a physical bench and virtual buttons, and is suitable for two trainees to operate and train together. The whole includes three parts: the cockpit, the control cabin and the external structure of the vehicle. The motion capture data acquisition unit uses VR-based motion capture equipment to collect and preprocess the trainee's full-body posture data and hand movement data in real time. The calibration stakes are set by the spatial fusion calibration unit and the point cloud registration algorithm is used to fuse and calibrate the motion capture physical space and the virtual operation training space. Based on the calibration results and whole-body posture data, collision avoidance warning is given to the students in the same vehicle in the physical test bench. The interactive mapping unit maps the trainee's hand movement data to the virtual operation of the vehicle model, and introduces a Markov chain model to predict the fingertip trajectory. Based on the prediction results, the virtual button pre-response is realized. Through the real-time interactive touch between the fingertip and the virtual button, vehicle control commands are generated to drive the vehicle model to move and operate in the virtual operation training field according to the real dynamic logic. The collaborative sensing unit enables precise information delivery and cross-vehicle collaborative sensing among different trainees, and achieves status synchronization among trainees in the same vehicle. The central processing unit acts as the computing center of the system, coordinating the collaborative operation of all units.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 8.

Images