Immersive content production and simulation system for motion vehicle

Abstract

The invention discloses an immersive content production and simulation system for a motion vehicle, and aims to solve the problems that virtual vision and physical motion are difficult to match, the content production cost is high and the interactivity is insufficient. According to the system, a six-degree-of-freedom track of a target is planned through a content making simulation operation end, and a moving carrier is driven to execute; and meanwhile, real motion data of the carrier is collected, and comparison and iterative correction are carried out on the real motion data and the target trajectory until errors converge. The system also uses artificial intelligence to generate medium-long-shot and close-shot three-dimensional asset fusion. Through closed-loop correction driven by physical data, high-precision synchronization of motion and pictures is realized, the development period is remarkably shortened, the content cost is reduced, and the personalized interactive experience is improved.

Description

Technical Field

[0001] This invention relates to the fields of immersive flight theaters, cantilevered dome theaters, and multi-degree-of-freedom motion platforms. Specifically, it relates to an immersive content creation and simulation system and method that integrates a real-time interactive engine, AIGC-generated distant views and layered 3D reconstruction, physical vehicle motion data acquisition and iterative correction, and multi-sensory feedback and interactive playback. The system spatially merges near-view 3D interactive content generated by an interactive engine (such as UE / Unity) with distant views in a panoramic / infinite world generated by AIGC, and drives a virtual camera with real vehicle motion data to achieve highly immersive content development and simulation for devices such as flight theaters, robotic arm rides, cantilevered dome theaters, and six-degree-of-freedom motion platforms. It is applicable to scenarios such as theme parks, science education, and military training. Background Technology

[0002] With the increasing prevalence of immersive experiences in entertainment, education, and training, flying / dome theaters and multi-degree-of-freedom motion platforms rely on the precise matching of visual imagery and physical motion. However, existing technologies generally suffer from the following limitations:

[0003] 1. Inefficient vehicle motion and attitude design. Traditionally, camera paths (corresponding to vehicle motion) are designed manually, with many parameters that are coupled together, making it difficult to achieve complex trajectories while balancing excitement and comfort; individual hardware differences make it difficult for actual motion to perfectly match the design curve, easily inducing dizziness and discomfort during sharp turns, large rises and falls, and other actions.

[0004] 2. Time-consuming motion-image matching debugging. It is difficult to strictly synchronize the actual movement of the device with the virtual image, requiring a lot of trial and error and repeated debugging. The debugging cycle for complex trajectories can be several weeks or even longer, which significantly prolongs the development cycle and increases costs.

[0005] 3. The amount of scene modeling required is enormous. To achieve a fully immersive effect, a large-scale, high-precision 3D asset needs to be constructed. In particular, the demand for mid-to-far-view models in special scenes is huge and difficult to dynamically expand, which limits content updates and reuse.

[0006] 4. Insufficient interactivity. Existing vehicle interactions are mostly simple triggers, lacking personalized responses to real-time user data, thus limiting the user experience and immersion.

[0007] 5. High content update costs. Each update often requires rebuilding the model and re-tuning motion matching, and may even require hardware modifications, leading to a decline in appeal during the operational period. Summary of the Invention

[0008] I. Technical problems to be solved

[0009] This invention aims to provide an immersive content creation and simulation system and method for motion vehicles, in order to solve problems such as complex content development processes, difficulties in matching and debugging virtual vision and physical motion, high production costs of scene assets, and insufficient interactivity. It realizes immersive content production and simulation driven by physical motion data, with automatic iterative correction capabilities, and can be quickly reused and updated.

[0010] II. Technical Solution

[0011] To achieve the above objectives, the system provided by this invention includes: a motion vehicle, a content creation simulation computing terminal, a playback computing terminal, a display terminal, an interactive antenna, and a cloud system. The system and method include at least the following modules and processes:

[0012] (A) A motion vehicle (1), comprising:

[0013] A1. Motion Module (12): Receives the frame-by-frame spatial point position and motion control code of the viewing reference point output by the content creation simulation calculation terminal, and performs smooth motion.

[0014] A2. Motion Data Acquisition Module (11): Based on the servo motor encoder, the module acquires the angles and torques of each motion axis, and calculates the six-degree-of-freedom (6DOF) pose of the viewing reference point based on the forward kinematics. The module transmits timestamped 6DOF data streams at a high frequency during trajectory operation via a network connection (preferably TCP / UDP Socket) between the host computer and the vehicle controller. The preferred sampling period is 15ms (approximately 66.7Hz), with an optional range of 4ms to 20ms (250Hz to 50Hz).

[0015] A3. Signal processing and control module (13): After receiving the start trigger, it drives the motion module according to the input trajectory and outputs the synchronization signal and status word to the playback calculation terminal.

[0016] (B) Content creation simulation computing terminal (2), including:

[0017] B1. Motion trajectory planning simulation and output module (22):

[0018] B1.1 imports close-up or preliminary scenes into an interactive engine (such as UE / Unity), designs the optimal motion trajectory of the virtual camera based on stimulation, interest, comfort, and narrative rhythm, and previews and simulates the audience's perspective in real time. The trajectory satisfies the jerk-constrained S-curve constraint, and the maximum acceleration and angular acceleration settings are based on device capabilities and comfort standards.

[0019] B1.2 imports the target vehicle model and physical constraint parameters for each motion axis, completing the skeleton / joint binding. Based on the virtual camera trajectory, when linear or angular acceleration exceeds a threshold, key points of the vehicle are automatically generated on the timeline, and matching key points are generated based on amplitude, direction, and duration; manual fine-tuning based on rhythm and plot is supported; if hardware safety limits are exceeded, the system will prompt and prevent unsafe output at the key points.

[0020] B1.3 outputs frame-by-frame 6DOF motion data (position X / Y / Z and rotation Pitch / Yaw / Roll) at a set frame rate based on keypoint animation output, which is used as spline curve sampling points to drive physical devices to perform smooth motion. The preferred sampling frame rate is 60–240 fps.

[0021] B2. Physical motion comparison and correction module (21):

[0022] B2.1 Time axis alignment: The acquired real 6DOF data is aligned with the target data output from B1.3 using a unified time base. The preferred method is to use a timestamp-based linear / spline interpolation and sliding window cross correlation. Clock synchronization can use NTP / PTP. The end-to-end alignment error is preferably ≤10ms.

[0023] B2.2 Frame-by-frame spatial comparison: After alignment, the position and pose differences are calculated frame by frame. The unit of position difference is mm, and the unit of angle difference is degree or radian. The preferred thresholds are 5mm to 20mm for position and 0.05° to 0.3° for angle.

[0024] B2.3 Iterative Correction: For frames with differences exceeding the threshold, a reverse correction is applied to the corresponding coordinate / angle dimension; the correction value is the product of the difference value's scaling factor k, where k is preferably 0.5, and can be selected within the range of 0.2 to 0.8, to balance convergence speed and stability. After full-frame processing, each frame's 6DOF and corresponding cubic B-spline curve are reconstructed to generate a smooth correction trajectory, and the device is driven to re-sample; the "acquisition-comparison-correction" iteration is repeated until the convergence threshold is met or the maximum number of iterations Nmax (preferably 3 to 8 times) is reached.

[0025] B3. Hybrid Virtual Scene Generation and Rendering Module (23):

[0026] B3.1 Mid-to-Long-Range AIGC Environment Generation Submodule:

[0027] B3.1.1 Based on reference images and text descriptions, a diffusion model is used to generate 360° panoramic images / panoramic videos, and outputs equidistant cylindrical projections (2:1) for skyboxes; single image expansion is supported to generate horizontal 360° / vertical 180° panoramas.

[0028] B3.1.2 generates an infinite world model in real time based on AI, integrating panoramic generation and layered 3D reconstruction. It takes the description of the model required for the scene as input, integrates panoramic generation and semantic layered 3D reconstruction, and quickly generates a roamable 3D scene. The output is a Mesh that can be used for engine level building; it realizes semantic separation of foreground / background and ground / sky to facilitate combination and replacement.

[0029] B3.2 Close-up / Foreground Rendering Submodule: Creates core 3D assets and UI for close-up interaction in the interaction engine, supporting interactions such as attack, pick-up, and branch selection; and integrates lighting and color consistency with the mid-to-far view output from B3.1.

[0030] (C) Playback processing terminal (5), including:

[0031] C1. Playback Module (51): Simultaneously plays interactive content on the host computer and pushes it to the display device to achieve immersive screen output; supports subtitles / overlay UI.

[0032] C2. Interaction Module (52): Combines RFID sensing antenna and RFID wristband to sense the identity of the rider and request cloud player profile (level, nickname, faction, etc.); according to the set gameplay, overlays personalized UI on the screen and plays differentiated content or unlockable clips according to faction / level.

[0033] C3. Interactive Antenna (53): Reads the RFID wristband and outputs the RFID tag ID to the interactive module.

[0034] (D) Cloud system (4): Stores real-time user play data and historical archives, and supports personalized content distribution and statistical analysis.

[0035] (E) Display end (3): including projection-dome display module or LED display screen; optional multi-projection fusion, geometric correction and gamma correction to ensure consistency of dome / wide viewing angle display.

[0036] (F) User RFID wristband (6): A wristband containing an RFID tag for identifying the user ID.

[0037] III. Terminology Definitions

[0038] 1. Viewing reference point: refers to a calibration point that is approximately the same as the passenger's eye position, used to define the spatial correspondence between the virtual camera and the optimal viewing point of the physical vehicle.

[0039] 2.6DOF: refers to the six degrees of freedom of position X / Y / Z and attitude Pitch / Yaw / Roll in three-dimensional space.

[0040] 3. Iterative correction: refers to the closed-loop process in which the difference between the actual motion and the target motion is proportionally reversed and applied to the target trajectory under a unified time axis and coordinate system, and then the device is driven to run again after smooth reconstruction until convergence.

[0041] IV. Optional Implementation Points and Parameter Ranges

[0042] 1. Data Interface: Acquired data frames should be encapsulated in JSON or binary (e.g., Protobuf), with fields including at least: timestamp, posX / posY / posZ, pitch / yaw / roll, qualityFlag, and deviceState. A wired gigabit network connection is preferred to reduce jitter, and end-to-end latency should ideally be ≤25ms.

[0043] 2. Trajectory reconstruction: Cubic B-splines are used for smoothing, with the spline node spacing preferably between 2 and 6 frames, and natural boundary conditions are adopted for the boundaries.

[0044] 3. Synchronization strategy: The rendering frame rate and the motion sampling frame rate are matched using nearest neighbor / spline interpolation; the screen synchronization signal can be aligned using inter-frame timecode.

[0045] 4. Safety Limits: Maximum angular velocity, angular acceleration, and displacement acceleration are limited according to equipment capabilities. The system performs dual verification during both trajectory generation and output stages.

[0046] V. Beneficial Effects

[0047] The present invention has the following beneficial effects:

[0048] 1. Motion-Vision Frame-Level Synchronization: Based on hardware timestamps and frame-to-frame synchronization of image signals, it improves the consistency between motion data and image output, significantly reducing dizziness.

[0049] 2. Shortened debugging cycle: By comparing the differential limiting and smooth output of the calibration device, repeated joint debugging is reduced, shortening the cycle from creation to operation.

[0050] 3. High system integration: The integrated cabinet solution for sensing, control, display and interaction improves reliability and reusability.

[0051] 4. Controllable content update costs: AI generates mid-range and long-range interactive assets and reusable close-up interactive assets, combined with automatic iterative correction, which greatly reduces update costs.

[0052] 5. Personalized interactive experience: Combining RFID with player profiles enables personalized UI display and branching content playback, enhancing user engagement and gameplay diversity. Attached Figure Description

[0053] Figure 1This is a schematic diagram of the overall modules of an immersive content creation and simulation system according to an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of the overall structure of an immersive content creation and simulation system during operation, according to an embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram of the structure of a motion vehicle according to an embodiment of the present invention.

[0056] Figure 4 This is a schematic diagram of the module structure for immersive content creation and simulation according to an embodiment of the present invention.

[0057] Figure 5 This is a schematic diagram of an embodiment of the immersive content creation and simulation process of the present invention.

[0058] Figure 6 This is a schematic diagram of the module structure of an immersive content creation and simulation system during operation, according to an embodiment of the present invention.

[0059] Explanation of reference numerals in the attached figures

[0060] (1) Sports vehicle; (11) Motion data acquisition module; (12) Motion module; (13) Signal processing and control module; (2) Content production simulation calculation terminal; (21) Physical motion comparison and correction module; (22) Motion trajectory planning simulation and output module; (23) Hybrid virtual scene generation and rendering module; (3) Display terminal; (4) Cloud system; (5) Playback calculation terminal; (51) Playback module; (52) Interaction module; (53) Interaction antenna; (6) RFID wristband; A1 / A2 / A3 / A7 / A9 / A10 are connecting lines for signal connection. Detailed Implementation

[0061] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are for illustration only and not for limiting the scope of protection of the present invention.

[0062] Example 1: System Overall Structure

[0063] See Figures 1-6 The system provided by the present invention includes: a motion vehicle (1), which includes a motion data acquisition module (11), a motion module (12), and a signal processing and control module (13); a content creation simulation computing terminal (2), on which a physical motion comparison and correction module (21), a motion trajectory planning simulation and output module (22), and a hybrid virtual scene generation and rendering module (23) are run; a playback computing terminal (5), which includes an interactive antenna (53), an interactive module (52), and a playback module (51); a display terminal (3); a cloud system (4); and an RFID wristband (6) worn by the user.

[0064] During the content production simulation stage, a motion vehicle (1), a content production simulation computing terminal (2), and a display terminal (3) are required, as well as connecting lines A1, A2, and A9 between them; during the passenger operation stage, the motion vehicle (1), the playback computing terminal (5), the cloud (4), and the RFID wristband (6) are retained, and the connection between the signal processing and control module (13) and the playback module (51) (A7) and the connection between the playback module (51) and the display terminal (3) (A10) are retained.

[0065] Example 2: Motion Data Acquisition and Comparison Correction

[0066] The motion data acquisition module (11) is fixed on the motion axis of the motion vehicle. It acquires the angle and torque of each axis through the encoder controlled by the closed loop of the servo motor, and calculates the 6DOF pose of the passenger seat viewing reference point in real time through the kinematic forward solution to reduce the drift effect and high-speed interference of IMU-type sensors.

[0067] The physical motion comparison and correction module (21) of the content production simulation calculation terminal (2) performs frame-by-frame comparison between the actual viewing position and the target position output by the motion trajectory planning simulation and output module (22). A unified time axis and coordinate system are adopted, and alignment is achieved by using linear / spline interpolation and sliding window cross correlation. Then, the position and angle errors are detected by the threshold criterion, and the frames exceeding the threshold are reversed by the scaling factor k, and reconstructed into cubic B-splines to obtain a smooth correction trajectory. The correction trajectory is written back to the motion trajectory planning simulation and output module (22) to drive the vehicle to run again and collect data until the error is less than the set threshold or the number of iterations reaches Nmax.

[0068] Example 3: Generation of Hybrid Virtual Scenes

[0069] The hybrid virtual scene generation and rendering module (23) uses a diffusion model to generate seamless spherical / cube projections, combined with LoRA fine-tuning to output equidistant cylindrical projections (2:1) for the skybox, forming a mid-to-far-field environment; or further through layered 3D reconstruction / infinite world generation, text / image inputs are converted into roamable 3D scene meshes, achieving semantic separation between foreground / background and ground / sky, facilitating combination and replacement. Foreground / close-up assets and UI are created in the interaction engine and unified with the mid-to-far-field in terms of lighting and tone, outputting virtual camera footage for preview and export.

[0070] Example 4: Generation Process and Linkage

[0071] 1. Design virtual camera trajectory: Design and preview the trajectory with S-curve constraints in the interactive engine.

[0072] 2. Track to vehicle mapping: Import vehicle parameters and generate key points and frame-by-frame 6DOF.

[0073] 3. Safety verification: The maximum angular velocity / angular acceleration / linear acceleration are verified twice, and output is prevented if the limit is exceeded.

[0074] 4. Trial run and data collection: Drive the motion module (12) to run along the trajectory and collect real 6DOF data stream.

[0075] 5. Comparison and Iteration: Time axis alignment, error calculation, proportional reverse correction and spline reconstruction, iterating until convergence or Nmax is reached.

[0076] 6. Image generation: During the preview process, the image is synchronously output to the display terminal (3), and the source is exported after confirming visual-motion consistency.

[0077] Example 5: Playback and Interaction

[0078] During actual passenger operation, after receiving the start signal from the motion vehicle signal processing and control module (13), the playback module (51) of the playback computing terminal (5) reads the tag ID of the player's RFID wristband (6) through the interaction antenna (53) and requests the player profile from the cloud (4). Based on the player's level / faction, it overlays personalized UI on the screen or switches differentiated content segments, while maintaining synchronous playback with the motion signal.

[0079] Industrial applicability:

[0080] This invention is applicable to scenarios such as theme parks, science education, and military training. It can use real motion data in a closed loop for content production and simulation, shorten the development and debugging cycle, reduce the production cost of large-scale scene assets, and enhance personalized interactive experience. It has significant industrial application value and replicability.

Claims

1. A system for creating and simulating immersive content for sports vehicles, characterized in that, include: The sports vehicle includes: ■ The motion data acquisition module collects the angles and torques of each motion axis based on the servo motor encoder, calculates the six-degree-of-freedom pose of the viewing reference point through forward kinematics, and sends a time-stamped six-degree-of-freedom data stream to the host computer via the network according to the set sampling period; ■ Motion module, used to smoothly execute motion based on the input frame-by-frame motion control data; ■Signal processing and control module, used to drive the motion module after receiving the start trigger, and output synchronization signal and status word to the playback computing terminal; Content creation simulation computing terminal, including: ■ Motion trajectory planning simulation and output module, used to design virtual camera trajectory based on jerk-constrained S-curve constraints in the interactive engine, import vehicle model and physical constraint parameters, automatically generate vehicle key points and output target six-degree-of-freedom trajectory sampled at frame rate; ■ The physical motion comparison and correction module is used to align the real six-degree-of-freedom data output by the motion data acquisition module with the target six-degree-of-freedom trajectory in time axis and compare them frame by frame in space. It is used to reverse the over-threshold difference according to the proportional coefficient and reconstruct it smoothly by spline, and iterate until the convergence threshold is met or the maximum number of iterations is reached. ■ Hybrid virtual scene generation and rendering module, used to generate mid-to-far panoramic environment using diffusion model and / or layered 3D reconstruction, and overlay close-range 3D assets in the interactive engine to achieve composite rendering with consistent lighting and color tone. The playback processing unit includes: ■Playback module, used to synchronously play interactive content on the host computer and output it to the display terminal; ■ Interactive module, connected to interactive antenna, is used to read RFID tags and request player profiles from the cloud, overlay personalized interfaces or trigger differentiated content segments, and keep synchronized with the playback module and the signal processing and control module; The display end is either a projection dome display module or an LED display screen; A cloud system is used to store player profiles and gameplay data and provide personalized distribution to the interactive module; In the content creation simulation computing terminal, during the preview and simulation process, the virtual camera in the engine is driven by the real six degrees of freedom data output by the motion data acquisition module to render the image, and the image is frame-level aligned and compared with the image rendered according to the target six degrees of freedom trajectory.

2. The system according to claim 1, characterized in that, The sampling period of the motion data acquisition module is 4ms to 20ms, preferably about 15ms, and the data frames are sent via TCP or UDP Socket in JSON or binary encapsulation. The data fields include timestamp, posX, posY, posZ, pitch, yaw, roll, qualityFlag and deviceState. Wired gigabit is preferred to ensure that the end-to-end latency is ≤25ms.

3. The system according to claim 1, characterized in that, The threshold for frame-by-frame spatial comparison is 5mm to 20mm in position and 0.05° to 0.3° in angle; the scaling factor k for reverse correction is 0.2 to 0.8, preferably about 0.5; and the maximum number of iterations is 3 to 8.

4. The system according to claim 1, characterized in that, The spline smoothing reconstruction uses cubic B-splines with a node spacing of 2 to 6 frames and natural boundary conditions.

5. The system according to claim 1, characterized in that, When the motion trajectory planning simulation and output module detects that the linear or angular acceleration exceeds the set threshold, it automatically generates key points of the vehicle based on the amplitude, direction and duration, and performs dual safety checks on the maximum angular velocity, angular acceleration and linear acceleration to prevent the output of the out-of-limit trajectory.

6. The system according to claim 1, characterized in that, The motion trajectory planning simulation and output module outputs a frame-by-frame six-degree-of-freedom sampling frame rate of 30fps to 240fps.

7. The system according to claim 1, characterized in that, The hybrid virtual scene generation and rendering module generates 360° panoramic images or panoramic videos based on the diffusion model and LoRA fine-tuning, and outputs 2:1 equidistant cylindrical projections as skyboxes or panoramic images.

8. The system according to claim 1, characterized in that, The hybrid virtual scene generation and rendering module is based on AI to generate an infinite world model in real time, integrating panoramic generation and layered 3D reconstruction. It takes the description of the model required for the scene as input, integrates panoramic generation and semantic layered 3D reconstruction, and quickly generates a roamable 3D scene mesh, that is, a mid-to-far field scene mesh in an interactive scene.

9. The system according to claim 1, characterized in that, The playback processing terminal synchronizes the image through inter-frame timecode and uses nearest neighbor or spline interpolation to match the difference between the rendering frame rate and the motion sampling frame rate, thereby achieving motion-visual frame-level synchronization.

10. The system according to claim 1, characterized in that, The viewing reference point is established in the vehicle model through skeletal / joint binding to establish a spatial correspondence with the optical center of the virtual camera, which is used to map the real six degrees of freedom to the virtual camera pose one by one.

11. A method for creating and simulating immersive content for motion vehicles, characterized in that, Includes the following steps: S1: In the hybrid virtual scene generation and rendering module, a mid-to-far panoramic environment is generated using a diffusion model and / or layered 3D reconstruction, overlaid with near-field 3D assets, and the lighting and color tone are adjusted for consistency. Generate interactive, immersive scenes that can be loaded into the interaction engine; S2: In the interaction engine, import the constraint parameters of the immersive scene and the moving vehicle, generate the vehicle key points and output the target six-degree-of-freedom trajectory sampled at the frame rate; S3: Drive the motion vehicle to run along the target six-degree-of-freedom trajectory, collect the angle and torque of each motion axis based on the servo motor encoder, calculate the real six degrees of freedom of the viewing reference point through the forward kinematics solution, and send it as a data stream with timestamps; S4: Perform time axis alignment and frame-by-frame spatial comparison between the real six degrees of freedom and the target six degrees of freedom, reverse the over-threshold difference according to the proportional coefficient, and obtain the corrected trajectory through spline smoothing reconstruction; S5: Based on the corrected trajectory, repeat steps S2 to S4 until the error meets the convergence threshold or reaches the maximum number of iterations. S6: During the simulation process, the virtual camera is driven by the real six degrees of freedom to perform preview rendering, and the frame-level alignment comparison with the image rendered according to the target six degrees of freedom is performed to confirm visual-motion consistency. S7: Export the video source to the playback module; S8: During passenger operation, the playback computing terminal plays the video source after receiving the start synchronization signal. The interaction module reads the RFID tag and requests the player profile from the cloud, overlays a personalized interface or triggers differentiated content segments, and outputs them to the display terminal in sync with the motion signal.

12. The method according to claim 11, characterized in that, In step S4, the time axis alignment uses timestamp-based linear interpolation or spline interpolation combined with sliding window cross correlation, and clock synchronization uses NTP or PTP to ensure that the alignment error is ≤10ms; the position and angle error thresholds are 5mm~20mm and 0.05°~0.3°, respectively; the scaling factor k is 0.2~0.8, preferably about 0.5; and the maximum number of iterations is 3~8.

13. The method according to claim 12, characterized in that, The sampling period of step S2 is 4ms to 20ms, preferably about 15ms; the virtual camera rendering frame rate of steps S1 and / or S5 is 30fps to 240fps, and the frame rate is matched by nearest neighbor or spline interpolation.

14. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the method as described in any one of claims 11 to 13.

Images