Simulation control method and device, equipment and storage medium

Abstract

The application discloses a simulation control method and device, equipment and a storage medium, and relates to the technical field of head-mounted display devices. The simulation control method comprises the following steps: obtaining an original rendering frame corresponding to a simulator display picture, wherein the original rendering frame comprises a simulation device, a digital person wearing the simulation device, and a virtual environment in which the digital person is located; determining sensor data corresponding to the original rendering frame, determining real-time simulation parameters based on the sensor data, wherein the sensor data represents visual features detected by a sensor on the simulation device; determining a real-time display frame of the simulation device based on the real-time simulation parameters, and determining a simulation picture presented by the simulation device to the digital person based on the real-time display frame. The application can improve the simulation effect of the simulator.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of head-mounted display devices, and particularly relates to a simulation control method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of AR (Augmented Reality) technology, VR (Virtual Reality) technology and MR (Mixed Reality) technology, simulation technology has become the core engine driving the iterative upgrade of the design and development of software and hardware.

[0003] At present, the traditional virtual simulation technology usually generates a picture presented by a simulation device according to preset simulation parameters or a static model. When the virtual simulation picture changes in real time, the display frame of the simulation device cannot adapt to the changes in time, resulting in a deviation between the picture seen by a digital person and the real-time state of the virtual simulation picture, and further affecting the overall simulation effect.

[0004] Therefore, how to improve the simulation effect of a simulator is a problem to be solved at present. SUMMARY

[0005] The main purpose of the present application is to provide a simulation control method, device, equipment and storage medium, which aims to improve the simulation effect of a simulator.

[0006] To achieve the above purpose, the present application provides a simulation control method, which comprises the following steps:

[0007] obtaining an original rendering frame corresponding to a display picture of a simulator, wherein the original rendering frame comprises a simulation device, a digital person wearing the simulation device and a virtual environment in which the digital person is located;

[0008] determining sensor data corresponding to the original rendering frame, and determining real-time simulation parameters based on the sensor data, wherein the sensor data represents visual features detected by a sensor on the simulation device;

[0009] determining a real-time display frame of the simulation device based on the real-time simulation parameters, so as to determine a simulation picture presented by the simulation device to the digital person based on the real-time display frame.

[0010] In an embodiment, the step of determining real-time simulation parameters based on the sensor data comprises:

[0011] determining a current overall pose of the digital person based on the sensor data;

[0012] determining real-time simulation parameters based on the current overall pose.

[0013] In an embodiment, the step of determining the current overall pose of the digital human based on the sensor data comprises:

[0014] determining, based on fisheye camera data and inertial measurement data in the sensor data, a head-mounted pose of the simulation device and environment map data in the original rendering frame;

[0015] determining, based on color image information and depth camera data in the sensor data, video perspective data in the original rendering frame;

[0016] determining, based on fisheye camera data in the sensor data, a hand pose of the digital human in the original rendering frame;

[0017] determining, based on eye image data and infrared auxiliary imaging data in the sensor data, an eye pose of the digital human in the original rendering frame;

[0018] determining, based on fisheye camera data, handle ray data, and handle motion information in the sensor data, a handle pose of the simulation device in the original rendering frame;

[0019] determining, based on facial image data and infrared auxiliary imaging data in the sensor data, a facial pose of the digital human in the original rendering frame;

[0020] determining the current overall pose of the digital human based on the head-mounted pose, the environment map data, the video perspective data, the hand pose, the eye pose, the handle pose, and the facial pose.

[0021] In an embodiment, the real-time simulation parameters include screen distortion parameters, and the step of determining real-time simulation parameters based on the current overall pose comprises:

[0022] determining, based on the eye pose and the head-mounted pose in the current overall pose, a positional relationship between the eyes of the digital human, the lenses of the simulation device, and the display screen;

[0023] determining screen distortion parameters of the simulation device through the positional relationship.

[0024] In an embodiment, the real-time simulation parameters further include camera simulation parameters, and the step of determining real-time simulation parameters based on the current overall pose comprises:

[0025] determining a first pose of a camera on the simulation device based on the current overall pose, and determining a camera simulation screen based on the first pose and the virtual environment, wherein the camera is an imaging device for collecting two-dimensional visual information;

[0026] determine a second pose of a depth camera on the simulation device based on the current overall pose, and determine a real-time depth map based on the second pose and the virtual environment, wherein the depth camera is an imaging device for collecting two-dimensional visual information and depth information;

[0027] use the camera simulation picture and the real-time depth map as the camera simulation parameters.

[0028] In an embodiment, the real-time simulation parameters further include inertial measurement parameters, and the step of determining the real-time simulation parameters based on the current overall pose comprises:

[0029] determine a third pose of an inertial measurement unit on the simulation device based on the current overall pose;

[0030] determine a fourth pose of the inertial measurement unit based on a historical overall pose corresponding to a previous frame of the original rendering frame;

[0031] determine a fifth pose of the inertial measurement unit based on a future overall pose corresponding to a next frame of the original rendering frame;

[0032] determine motion information of the simulation device at the current time based on position information and rotation information of the inertial measurement unit at the third pose, the fourth pose and the fifth pose, wherein the motion information includes velocity, acceleration and angular velocity;

[0033] use the motion information as the inertial measurement parameters.

[0034] In an embodiment, the real-time simulation parameters further include light simulation parameters, and the step of determining the real-time simulation parameters based on the current overall pose comprises:

[0035] determine a sixth pose of a light source on a handle in the simulation device based on the current overall pose, and determine the light simulation parameters based on the sixth pose and a ray tracing technique.

[0036] In addition, to achieve the above-mentioned purposes, the application further provides a simulation control device, which comprises:

[0037] an acquisition module configured to acquire an original rendering frame corresponding to a simulator display picture, wherein the original rendering frame includes a simulation device, a digital human wearing the simulation device and a virtual environment in which the digital human is located;

[0038] a determination module configured to determine sensor data corresponding to the original rendering frame, and determine real-time simulation parameters based on the sensor data, wherein the sensor data represents visual features detected by a sensor on the simulation device;

[0039] A display module is configured to determine a real-time display frame of the simulation device based on the real-time simulation parameter, and determine a simulation picture presented by the simulation device to the digital human based on the real-time display frame.

[0040] In addition, to achieve the above object, the present application also provides a storage medium, which is a computer readable storage medium, and a program for implementing the simulation control method is stored on the computer readable storage medium, and the program is executed by a processor to implement the steps of the simulation control method.

[0041] In addition, to achieve the above object, the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the simulation control method.

[0042] The present application provides a simulation control method, and the present application first acquires an original rendering frame corresponding to a simulator display picture, wherein the original rendering frame is a rendering frame comprising a simulation device, a digital human wearing the simulation device, and a virtual environment in which the digital human is located; determines sensor data corresponding to the original rendering frame, wherein the sensor data represents visual features detected by sensors on the simulation device at a time corresponding to the original rendering frame; determines real-time simulation parameters based on the sensor data; determines a real-time display frame of the simulation device based on the real-time simulation parameters; and determines a simulation picture presented by the simulation device to the digital human based on the real-time display frame.

[0043] In summary, the present application determines visual features detected by sensors on the simulation device from a rendering frame comprising the simulation device, the digital human, and the virtual environment, determines real-time simulation parameters based on the visual features, and determines a display frame of the simulation device based on the real-time simulation parameters. In this way, compared with the conventional method of being unable to dynamically generate a display frame, the present application dynamically determines simulation parameters of the simulation device based on changes in the virtual simulation picture (for example, changes in the pose of the digital human, updates of the virtual environment, etc.), to generate a display frame matching the current rendering frame, thereby improving the overall simulation effect of the simulator. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced here. Obviously, for those skilled in the art, other drawings can also be obtained from these drawings without any creative labor.

[0046] Figure 1A flowchart of a first embodiment of the simulation control method of the present application;

[0047] Figure 2 A schematic diagram of a simulator architecture involved in an embodiment of the simulation control method of the present application;

[0048] Figure 3 A rendering frame schematic diagram involved in an embodiment of the simulation control method of the present application;

[0049] Figure 4 An optical simulation schematic diagram involved in an embodiment of the simulation control method of the present application;

[0050] Figure 5 A simulation control flow schematic diagram involved in an embodiment of the simulation control method of the present application;

[0051] Figure 6 A module structure schematic diagram of the simulation control device of the present application;

[0052] Figure 7 A device structure schematic diagram of the hardware running environment involved in the simulation control method in the embodiment of the present application.

[0053] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0055] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] The main solution of the present application is: obtaining an original rendering frame corresponding to a simulator display picture, wherein the original rendering frame includes a simulation device, a digital person wearing the simulation device and a virtual environment in which the digital person is located; determining sensor data corresponding to the original rendering frame, determining real-time simulation parameters based on the sensor data, wherein the sensor data represents visual features detected by a sensor on the simulation device; determining a real-time display frame of the simulation device based on the real-time simulation parameters, so as to determine a simulation picture presented by the simulation device to the digital person based on the real-time display frame.

[0057] At present, the traditional virtual simulation technology usually relies on preset simulation parameters or static models to generate the picture presented by the simulation device. When the virtual simulation picture changes in real time, the display frame of the simulation device cannot adapt to these changes in time, resulting in a deviation between the picture seen by the digital person and the real-time state of the virtual simulation picture, and further affecting the overall simulation effect.

[0058] Therefore, how to improve the simulation effect of the simulator is an urgent problem to be solved at present.

[0059] The simulation control method provided in the first embodiment of the present application comprises steps S10-S30.

[0060] It should be noted that the execution subject of the method in each embodiment of the simulation control method of the present application can be a simulation control system, a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, or a simulator capable of realizing the above functions, and the present embodiment does not make a specific limitation thereon. The following takes the simulation control system as the execution subject as an example to describe the present embodiment and each of the following embodiments.

[0061] Based on this, the simulation control method of the first embodiment of the present application is proposed, please refer to Figure 1 , the simulation control method comprises steps S10-S30:

[0062] Step S10, obtaining an original rendering frame corresponding to a simulator display picture, wherein the original rendering frame comprises a simulation device, a digital person wearing the simulation device and a virtual environment in which the digital person is located;

[0063] It should be noted that the rendering frame corresponding to the current display picture of the simulator is referred to as the original rendering frame for distinction. The original rendering frame comprises a simulation device, a digital person wearing the simulation device and a virtual environment in which the digital person is located. Among them, as Figure 2As shown in the schematic diagram of the simulator architecture, the simulator includes three elements of a VRMR (Virtual Reality / Mixed Reality) device (i.e., a simulation device), a Digital Human, and a virtual environment. The simulation device includes an HMD (Head-Mounted Display) and a controller. The HMD includes an optics system, an ET (Eye Tracking) system, a VST (Video SeeThrough) system, and a tracking system. The optics system includes a lens and a display. The ET system includes an LED and an ET camera. The VST system includes an RGB camera and a ToF camera (i.e., a depth camera). The tracking system includes a fisheye camera and an IMU. The controller includes an LED and an IMU. The Digital Human includes an eye, a face, a skeleton, and clothes. Specifically, the eye includes a sclera, a cornea, an iris, aqueous fluid, a crystalline lens, and a retina. The face includes a mouth, a nose, and a jaw. The skeleton includes a hand and a body. The clothes include tops, pants, and gloves. The virtual environment includes a calibration module, a lighting module, and user scene maps. Specifically, the calibration module is a factory calibration tool. The calibration module includes a chessboard, jigs, and a robotic arm.

[0064] As shown in the schematic diagram of the simulator architecture, the simulator includes three elements of a VRMR (Virtual Reality / Mixed Reality) device (i.e., a simulation device), a Digital Human, and a virtual environment. The simulation device includes an HMD (Head-Mounted Display) and a controller. The HMD includes an optics system, an ET (Eye Tracking) system, a VST (Video SeeThrough) system, and a tracking system. The optics system includes a lens and a display. The ET system includes an LED and an ET camera. The VST system includes an RGB camera and a ToF camera (i.e., a depth camera). The tracking system includes a fisheye camera and an IMU. The controller includes an LED and an IMU. The Digital Human includes an eye, a face, a skeleton, and clothes. Specifically, the eye includes a sclera, a cornea, an iris, aqueous fluid, a crystalline lens, and a retina. The face includes a mouth, a nose, and a jaw. The skeleton includes a hand and a body. The clothes include tops, pants, and gloves. The virtual environment includes a calibration module, a lighting module, and user scene maps. Specifically, the calibration module is a factory calibration tool. The calibration module includes a chessboard, jigs, and a robotic arm. Figure 3 As shown in the schematic diagram of the simulator architecture, the simulator includes three elements of a VRMR (Virtual Reality / Mixed Reality) device (i.e., a simulation device), a Digital Human, and a virtual environment. The simulation device includes an HMD (Head-Mounted Display) and a controller. The HMD includes an optics system, an ET (Eye Tracking) system, a VST (Video SeeThrough) system, and a tracking system. The optics system includes a lens and a display. The ET system includes an LED and an ET camera. The VST system includes an RGB camera and a ToF camera (i.e., a depth camera). The tracking system includes a fisheye camera and an IMU. The controller includes an LED and an IMU. The Digital Human includes an eye, a face, a skeleton, and clothes. Specifically, the eye includes a sclera, a cornea, an iris, aqueous fluid, a crystalline lens, and a retina. The face includes a mouth, a nose, and a jaw. The skeleton includes a hand and a body. The clothes include tops, pants, and gloves. The virtual environment includes a calibration module, a lighting module, and user scene maps. Specifically, the calibration module is a factory calibration tool. The calibration module includes a chessboard, jigs, and a robotic arm.

[0065] In step S20, sensor data corresponding to the original rendering frame is determined, and real-time simulation parameters are determined based on the sensor data, wherein the sensor data represents visual features detected by sensors on the simulation device in the scene displayed in the original rendering frame.

[0066] It should be noted that the sensor data corresponding to the original rendering frame refers to the visual features detected by the sensors on the simulation device in the scene displayed in the original rendering frame. The sensors on the simulation device are virtual sensors, including but not limited to four gray global exposure fisheye cameras, two high-resolution rolling shutter RGB (Red-Green-Blue) cameras, ToF (Time of Flight) depth cameras, infrared cameras, infrared light sensors, handle LED lights, handle IMU (Inertial Measurement Unit) and the IMU of the head-mounted reality device.

[0067] After determining the sensor data corresponding to the original rendering frame, real-time simulation parameters of the simulation device in the current frame are determined based on the sensor data (hereinafter referred to as real-time simulation parameters for distinction).

[0068] In step S30, real-time display frames of the simulation device are determined based on the real-time simulation parameters, so as to determine the simulation picture presented by the simulation device to the digital person based on the real-time display frames.

[0069] The real-time display frames of the simulation device at the current time are determined based on the real-time simulation parameters (hereinafter referred to as real-time display frames for distinction), the real-time display frames correspond to the original rendering frames, the simulation picture presented by the simulation device to the digital person is determined based on the real-time display frames, and in actual application, the simulation picture is an animation.

[0070] The embodiments of the present application determine the visual features detected by the sensors on the simulation device from the rendering frames containing the simulation device, the digital person and the virtual environment, determine the real-time simulation parameters based on the visual features, and determine the display frames of the simulation device based on the real-time simulation parameters. In this way, compared with the traditional way of not being able to dynamically generate display frames, the embodiments of the present application dynamically determine the simulation parameters of the simulation device based on the changes in the virtual simulation picture, to generate display frames matching the current rendering frames, thereby improving the overall simulation effect of the simulator.

[0071] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and will not be described hereinafter. On this basis, the step S20 can include:

[0072] Step S201, determining a current overall pose of the digital person based on the sensor data;

[0073] It should be noted that the sensor data includes data detected by the fisheye camera (hereinafter referred to as fisheye camera data for distinction), data detected by the inertial measurement unit (hereinafter referred to as inertial measurement data for distinction), data detected by the RGB camera (hereinafter referred to as color image information for distinction), data detected by the depth camera (hereinafter referred to as depth camera data for distinction), eye data detected by the infrared camera (hereinafter referred to as eye image data for distinction), data detected by the infrared lamp sensor (hereinafter referred to as infrared auxiliary imaging data for distinction), data detected by the handle LED lamp sensor (hereinafter referred to as handle light data for distinction), data detected by the handle IMU (hereinafter referred to as handle motion information for distinction), and face data detected by the infrared camera (hereinafter referred to as face image data for distinction).

[0074] In the embodiment, the step S201 can include:

[0075] Step A10, determining a head-mounted display pose of the simulation device and environment map data in the original rendering frame based on the fisheye camera data and the inertial measurement data in the sensor data.

[0076] Determine the pose (hereinafter referred to as head-mounted display pose for distinction) of the head-mounted display device of the simulation device and the environment map data in the original rendering frame based on the fisheye camera data and the inertial measurement data in the sensor data.

[0077] In an implementation, the head-mounted display pose and the environment map data are determined by a SLAM (Simultaneous Localization and Mapping) algorithm. Specifically, first, the fisheye camera data is subjected to distortion correction processing to obtain a corrected image, and the inertial measurement data is subjected to filtering and denoising processing to obtain preprocessed IMU data; feature points and descriptors are extracted from the corrected image, and the IMU data is integrated to obtain a pose change, and then the visual and inertial pose information is fused to preliminarily estimate the head-mounted display pose; then the current image feature points are matched with the existing environment map feature points, a new map point is created, and an optimization algorithm is used to optimize the head-mounted display pose and the environment map at the same time; finally, the head-mounted display pose of the simulation device and the environment map data in the original rendering frame are obtained.

[0078] Step A20, determining video perspective data in the original rendering frame based on the color image information and the depth camera data in the sensor data.

[0079] Based on the color image information and the depth camera data in the sensor data, video perspective data in the original rendering frame is determined. The video perspective data combines the real scene colors, textures and other contents presented by the color image, and the distance information of objects in the scene from the camera obtained by the depth camera. Through this fusion, the video perspective data can more comprehensively and truly reflect the spatial and visual features of the real scene, providing basic data for virtual reality, augmented reality and other applications, so that users can see real scene pictures with depth perception.

[0080] In an implementable embodiment, the video perspective data is determined by a VST algorithm. Specifically, the color image information is preprocessed, such as denoising, color correction, etc., and the depth camera data is filtered to remove noise interference; then the depth data is registered with the color image according to the calibration parameters, so that the depth information is accurately corresponding to the pixels of the color image; finally, based on the registered color image information and the depth camera data, the video perspective data containing real scene color information and corresponding depth information is generated by an image rendering algorithm.

[0081] Step A30, based on the fisheye camera data in the sensor data, determining the hand pose of the digital person in the original rendering frame;

[0082] Based on the fisheye camera data in the sensor data, the pose of the hand of the digital person in the original rendering frame is determined (hereinafter referred to as the hand pose for distinction).

[0083] In an implementable embodiment, the hand pose is determined by an HT (Hand Tracking) algorithm. Specifically, the collected fisheye image is subjected to distortion correction processing to eliminate the influence of lens distortion on the image and obtain a clear hand image; then an image processing algorithm is used, such as a target detection and key point extraction algorithm based on deep learning, to detect the hand region in the corrected image and extract the hand key points; then, according to the coordinates of the extracted hand key points and in combination with the camera calibration parameters, the pose information of the hand in the three-dimensional space is determined through coordinate transformation and other calculation methods; finally, the hand pose information is corresponded to the original rendering frame, realizing the determination of the hand pose of the digital person in the original rendering frame based on the fisheye camera data.

[0084] Step A40, based on the eye image data and the infrared auxiliary imaging data in the sensor data, determining the eye pose of the digital person in the original rendering frame;

[0085] Based on the eye image data and the infrared auxiliary imaging data in the sensor data, the pose of the eye of the digital person in the original rendering frame is determined (hereinafter referred to as the eye pose for distinction).

[0086] In an implementable embodiment, the eye pose is determined by an ET (Eye Tracking) algorithm. Specifically, image data containing the digital human eye is acquired by an eye image acquisition device in the sensor, and corresponding infrared auxiliary imaging data is acquired by an infrared auxiliary imaging device. The eye image data is preprocessed, such as denoising, contrast enhancement, etc., to improve the image quality; the infrared auxiliary imaging data is also processed, such as filtering, to reduce noise interference. Then, image feature extraction algorithms are used to extract key feature points of the eye from the eye image, such as the pupil center, the corner of the eye, etc.; at the same time, feature information related to the eye is extracted from the infrared auxiliary imaging data, such as infrared reflection features, etc. Then, the eye image features and the infrared auxiliary imaging features are fused and analyzed to more accurately locate the eye features by using the complementary information of the two. Finally, the pose of the digital human eye in the original rendering frame is determined by geometric transformation and coordinate calculation based on the extracted and fused feature information.

[0087] Step A50, determining the pose of the handle of the simulation device in the original rendering frame based on the fisheye camera data, handle ray data, and handle motion information in the sensor data.

[0088] Based on the fisheye camera data, handle ray data, and handle motion information in the sensor data, the pose of the handle of the simulation device in the original rendering frame is determined (hereinafter referred to as the handle pose for distinction).

[0089] In an implementable embodiment, the handle pose is determined by a CT (Controller tracking) algorithm. Specifically, the fisheye camera data is corrected for distortion to ensure the accuracy of the image. For the handle ray data, the propagation path and characteristics of the light rays are analyzed; for the handle motion information, filtering is performed to remove noise. Then, the position and feature points of the handle are detected in the corrected fisheye image. Then, the spatial information reflected by the handle ray data, the motion state embodied by the handle motion information, and the handle information detected in the fisheye image are combined, and these information are comprehensively processed and optimized by a multi-source data fusion algorithm, such as extended Kalman filtering. Finally, the accurate pose of the handle of the simulation device in the original rendering frame is determined.

[0090] Step A60, determining the face pose of the digital human in the original rendering frame based on the face image data and the infrared auxiliary imaging data in the sensor data.

[0091] Based on the face image data and the infrared auxiliary imaging data in the sensor data, the pose of the face of the digital human in the original rendering frame is determined (hereinafter referred to as the face pose for distinction).

[0092] In an implementable embodiment, the face pose is determined by an FT (Faceial tracking) algorithm. Specifically, the face image data is preprocessed, such as grayscale processing, noise reduction, and the like, to improve the image quality. The infrared auxiliary imaging data is also filtered and the like to reduce interference. Then, a face feature detection algorithm is used to extract key feature points from the face image, such as feature points of eyes, nose, mouth, and the like. Meanwhile, feature information related to the face is extracted from the infrared auxiliary imaging data, such as infrared thermal radiation features and the like. After that, the face image features and the infrared auxiliary imaging features are fused and analyzed to more accurately locate the face features by virtue of the complementary characteristics of the two. Finally, the pose of the digital person's face in the original rendering frame is determined by coordinate transformation and geometric calculation and the like based on the extracted and fused feature information.

[0093] Step A70, determining a current overall pose of the digital person based on the headset pose, the environment map data, the video perspective data, the hand pose, the eye pose, the handle pose, and the face pose.

[0094] In an implementable embodiment, the headset pose, the hand pose, the eye pose, the handle pose, and the face pose are first unified in data format and calibrated to ensure that the data are in the same coordinate system and are accuracy-matched. Then, the environment map data are taken as a spatial reference datum, and the headset pose is determined in the map to determine its relative spatial position. Then, the headset pose is further corrected according to the correspondence between the real scene and the virtual scene reflected by the video perspective data. Then, the hand pose, the eye pose, the handle pose, and the face pose are fused into the overall coordinate system centered on the headset by using coordinate transformation and spatial geometric relationship calculation, with the headset pose as a reference and in combination with the relative positional relationship between the hand, the eye, the handle, and the face and the headset. Finally, the fused positional data of the parts are optimized by a data fusion algorithm, such as weighted average or Kalman filtering, to determine the current overall pose of the digital person.

[0095] In this way, the headset pose and the environment map are determined by the fisheye camera and the inertial measurement data, which can provide a basic coordinate and a scene spatial reference for the positioning of the digital person. The video perspective data generated by the color image and the depth camera data can enhance the real scene perception to assist in positioning. The fisheye camera data are respectively used to determine the hand pose and the handle pose, and the eye pose and the face pose are determined in combination with the infrared data, which can accurately obtain the state of each part of the digital person. Finally, the current overall pose of the digital person is determined by comprehensively integrating all the positional data, the positioning and presentation of the digital person in the virtual scene are more accurate, comprehensive, and in line with the real state, and the realism and accuracy of the virtual interaction are improved.

[0096] Step S202, determining real-time simulation parameters based on the current overall pose.

[0097] determine real-time simulation parameters of the simulation device based on the current overall pose of the digital human.

[0098] In this embodiment, the real-time simulation parameters include picture distortion parameters, and the step S202 can include:

[0099] Step B10, determining a position relationship between the eyes of the digital human, the lens of the simulation device, and the display screen based on the eye pose and the head-mounted device pose in the current overall pose;

[0100] Step B20, determining picture distortion parameters of the simulation device through the position relationship.

[0101] It should be noted that the real-time simulation parameters include picture distortion parameters. Based on the eye pose and the head-mounted device pose in the current overall pose, the relative position relationship between the eyes of the digital human, the lens of the simulation device, and the display screen (i.e., the display) is determined, and based on the relative position relationship, a mapping relationship from an original picture to an imaging picture, i.e., picture distortion parameters, is determined.

[0102] In an implementable embodiment, as Figure 4 As shown in the optical simulation schematic diagram, after the relative position relationship between the eyes of the digital human, the lens, and the screen is determined, a light ray tracing technology is used to calculate discrete points of the imaging picture, to obtain a mapping relationship from the original picture Mesh0 to the imaging picture Mesh1 ; then a rendering picture is obtained from OpenXR, and a distortion mapping is added to the rendering picture through a GeometryShader to obtain the imaging picture of the current frame rate.

[0103] Further, in order to minimize the delay, the imaging picture of the current frame rate and the mapping function of the next frame can be calculated simultaneously in a GPU (Graphics Processing Unit, graphics processor).

[0104] In this embodiment, the step S202 can include:

[0105] Step C10, determining a first pose of a camera on the simulation device based on the current overall pose, and determining a camera simulation picture based on the first pose and the virtual environment, wherein the camera is an imaging device for collecting two-dimensional visual information;

[0106] Step C20, determining a second pose of a depth camera on the simulation device based on the current overall pose, and determining a real-time depth map based on the second pose and the virtual environment, wherein the depth camera is an imaging device for collecting two-dimensional visual information and depth information.

[0107] Step C30, the camera simulation picture and the real-time depth map are taken as the camera simulation parameters.

[0108] A camera and a depth camera are arranged on the simulation device, the camera is an imaging device for collecting two-dimensional visual information, and the depth camera is an imaging device for collecting two-dimensional visual information and depth information. A pose of the camera on the simulation device (hereinafter referred to as a first pose for distinction) is accurately extracted from a current overall pose, and the first pose is input into a rendering engine. The engine interacts with a virtual environment in which a digital person is located according to the pose, and performs real-time rendering to generate a camera simulation picture by using a real-time ray tracing technology under the premise of ensuring a high frame rate, and adds distortion effects to the picture through a preset shader. Then, a pose of the depth camera on the simulation device (hereinafter referred to as a second pose for distinction) is obtained from the current overall pose, and the second pose and a virtual environment model are input into the rendering engine. The engine calculates and generates a real-time depth map under the perspective of the depth camera according to the spatial relationship between the pose of the depth camera and the virtual environment model. Finally, the generated camera simulation picture and the real-time depth map are integrated as camera simulation parameters for subsequent processes.

[0109] In the embodiment, the real-time simulation parameters further include inertial measurement parameters, and the step S202 can include:

[0110] Step D10, determining a third pose of an inertial measurement unit on the simulation device based on the current overall pose.

[0111] A current pose (hereinafter referred to as a third pose for distinction) of the inertial measurement unit on the simulation device is determined based on the current overall pose.

[0112] Step D20, determining a fourth pose of the inertial measurement unit based on a historical overall pose corresponding to a previous frame of the original rendering frame.

[0113] An overall pose (hereinafter referred to as a historical overall pose for distinction) of the digital person corresponding to a previous frame of the original rendering frame is obtained, and a pose (hereinafter referred to as a fourth pose for distinction) of the inertial measurement unit at the time of the previous frame of the original rendering frame is determined based on the historical overall pose.

[0114] Step D30, determining a fifth pose of the inertial measurement unit based on a future overall pose corresponding to a next frame of the original rendering frame.

[0115] An overall pose (hereinafter referred to as a future overall pose for distinction) of the digital person corresponding to a next frame of the original rendering frame is determined, and a pose (hereinafter referred to as a fourth pose for distinction) of the inertial measurement unit at the time of the previous frame of the original rendering frame is determined based on the historical overall pose.

[0116] Step D40, determining motion information of the simulation device at the current time based on the position information and rotation information of the inertial measurement unit at the third pose, the fourth pose and the fifth pose, wherein the motion information comprises velocity, acceleration and angular velocity;

[0117] Determine motion information of the simulation device at the current time based on the position information and rotation information of the inertial measurement unit at the third pose, the fourth pose and the fifth pose, wherein the motion information comprises velocity, acceleration and angular velocity.

[0118] In an implementation, the inertial measurement unit is on the simulation device, the position information of the inertial measurement unit at the fourth pose is represented as , and the rotation information is represented as ; the position information of the inertial measurement unit at the third pose is represented as , and the rotation information is represented as ; and the position information of the inertial measurement unit at the fifth pose is represented as , and the rotation information is represented as . The step of determining velocity based on the above position information and rotation information comprises:

[0119]

[0120]

[0121]

[0122] wherein, is a time interval between two adjacent rendered frames, and is a step size. is a velocity of the interpolation between the original rendered frame and its previous rendered frame, is a velocity of the interpolation between the original rendered frame and its next rendered frame, is a velocity of the simulation device corresponding to the original rendered frame. The step of determining acceleration based on the above position information and rotation information comprises:

[0123]

[0124]

[0125] wherein, the rotation information is a rotation quaternion corresponding to the original rendered frame, including four components, x, y, z and w are real numbers. is a rotation matrix generated based on the quaternion. is an acceleration of the simulation device corresponding to the original rendered frame. The step of determining angular velocity based on the above position information and rotation information comprises:

[0126]

[0127]

[0128]

[0129]

[0130]

[0131] wherein the rotation information is a quaternion corresponding to the previous rendering frame, including four components, the rotation information is a quaternion corresponding to the next rendering frame, including four components. is an angular velocity of the interpolation between the original rendering frame and its previous frame rendering frame, is an angular velocity of the interpolation between the original rendering frame and its next frame rendering frame, is an angular velocity of the simulation device corresponding to the original rendering frame.

[0132] Step D50, taking the motion information as the inertial measurement parameter.

[0133] The speed, acceleration and angular velocity of the simulation device at the time corresponding to the original rendering frame are taken as the inertial measurement parameter.

[0134] In the embodiment, the real-time simulation parameter further includes a light simulation parameter, and the step S202 can include:

[0135] Step E10, determining a sixth pose of a light source on a handle in the simulation device based on the current overall pose, and determining the light simulation parameter based on the sixth pose and a ray tracing technique.

[0136] It should be noted that the real-time simulation parameter further includes a light simulation parameter. The handle in the simulation device includes an LED lamp (i.e., a light source).

[0137] The pose of the light source on the handle in the simulation device is determined based on the current overall pose (hereinafter referred to as the sixth pose for distinction), and the LED lamp is simulated based on the sixth pose and a ray tracing technique to obtain the light simulation parameter, which is used for the handle and the eye movement tracking simulation.

[0138] Exemplarily, as Figure 5The simulation control flowchart is shown. The simulator displays a picture including a simulation device (XR device), a digital person, and a virtual environment. The Unreal plugin obtains a rendering frame of the simulator display picture, determines sensor data in the rendering frame, and sends the sensor data to OpenXR. The sensor data is sent to the algorithm core through OpenXR, and the rendering pose of the digital person is calculated through SLAM algorithm, VST algorithm, HT algorithm, ET algorithm, CT algorithm, and FT algorithm. The real-time simulation parameters are determined based on the rendering pose of the digital person (i.e. the current overall pose), and the display frame is obtained. The real-time display frame is returned to the plugin, and the animation pose is generated through the plugin.

[0139] In this way, the embodiments of the present application determine the positional relationship among the eye, the lens, and the display screen according to the eye pose and the head-mounted device pose, and then determine the picture distortion parameters, so as to accurately correct the picture distortion, make the display picture more close to the real visual experience, and improve the authenticity of the visual experience. The determination of the camera simulation parameters obtains the poses of the camera and the depth camera based on the overall pose and generates the corresponding picture and depth map, which provides more rich data support for the construction and interaction of the virtual scene, and enhances the realism and interactivity of the virtual scene. The determination of the inertial measurement parameters determines the motion information by comprehensively considering the position and rotation information of the inertial measurement unit under the current, historical, and future overall poses, which can more accurately reflect the motion state of the simulation device and provide accurate data for motion control and interaction feedback. The determination of the light simulation parameters can make the lighting effect in the virtual scene closer to the real environment according to the handle light source pose and the ray tracing technology, and further improve the immersion and authenticity of the virtual scene. Overall, the embodiments of the present application significantly improve the simulation degree and interaction experience of the virtual scene by accurately determining the multi-dimensional simulation parameters. By applying the real-time ray tracing technology to the simulation of the VR optical imaging system and combining the photorealistic digital person driving technology, a real-time and photorealistic VR / MR system is realized. In addition, the simulation control method of the present application can also realize the verification from calibration to display effect.

[0140] The embodiments of the present application also provide a simulation control device, please refer to Figure 6 , the simulation control device comprises:

[0141] The acquisition module 10 is configured to acquire an original rendering frame corresponding to a simulator display picture, wherein the original rendering frame includes a simulation device, a digital person wearing the simulation device, and a virtual environment in which the digital person is located.

[0142] A determining module 20 is configured to determine sensor data corresponding to the original rendering frame, and determine real-time simulation parameters based on the sensor data, wherein the sensor data represents visual features detected by sensors on the simulation device;

[0143] A display module 30 is configured to determine a real-time display frame of the simulation device based on the real-time simulation parameters, and determine a simulation picture presented by the simulation device to the digital person based on the real-time display frame.

[0144] Optionally, the determining module 20 is further configured to:

[0145] determine a current overall pose of the digital person based on the sensor data;

[0146] determine real-time simulation parameters based on the current overall pose.

[0147] Optionally, the determining module 20 is further configured to:

[0148] determine a head-mounted display pose and environment map data of the simulation device in the original rendering frame based on fisheye camera data and inertial measurement data in the sensor data;

[0149] determine video perspective data in the original rendering frame based on color image information and depth camera data in the sensor data;

[0150] determine a hand pose of the digital person in the original rendering frame based on fisheye camera data in the sensor data;

[0151] determine an eye pose of the digital person in the original rendering frame based on eye image data and infrared auxiliary imaging data in the sensor data;

[0152] determine a handle pose of the simulation device in the original rendering frame based on fisheye camera data, handle light ray data and handle motion information in the sensor data;

[0153] determine a face pose of the digital person in the original rendering frame based on face image data and infrared auxiliary imaging data in the sensor data;

[0154] determine a current overall pose of the digital person based on the head-mounted display pose, the environment map data, the video perspective data, the hand pose, the eye pose, the handle pose and the face pose.

[0155] Optionally, the real-time simulation parameters include picture distortion parameters, and the determining module 20 is further configured to:

[0156] determine a position relationship between eyes of the digital person, a lens of the simulation device and a display screen based on the eye position and the head-mounted position in the current overall position;

[0157] determine a picture distortion parameter of the simulation device based on the position relationship.

[0158] Optionally, the real-time simulation parameter further comprises a camera simulation parameter, and the determination module 20 is further configured to:

[0159] determine a first position of a camera on the simulation device based on the current overall position, and determine a camera simulation picture based on the first position and the virtual environment, wherein the camera is an imaging device for collecting two-dimensional visual information;

[0160] determine a second position of a depth camera on the simulation device based on the current overall position, and determine a real-time depth map based on the second position and the virtual environment, wherein the depth camera is an imaging device for collecting two-dimensional visual information and depth information;

[0161] use the camera simulation picture and the real-time depth map as the camera simulation parameter.

[0162] Optionally, the real-time simulation parameter further comprises an inertial measurement parameter, and the determination module 20 is further configured to:

[0163] determine a third position of an inertial measurement unit on the simulation device based on the current overall position;

[0164] determine a fourth position of the inertial measurement unit based on a historical overall position corresponding to a previous frame of the original rendering frame;

[0165] determine a fifth position of the inertial measurement unit based on a future overall position corresponding to a next frame of the original rendering frame;

[0166] determine motion information of the simulation device at the current time based on position information and rotation information of the inertial measurement unit at the third position, the fourth position and the fifth position, wherein the motion information comprises velocity, acceleration and angular velocity;

[0167] use the motion information as the inertial measurement parameter.

[0168] Optionally, the real-time simulation parameter further comprises a light simulation parameter, and the determination module 20 is further configured to:

[0169] determine a sixth position of a light source on a handle in the simulation device based on the current overall position, and determine the light simulation parameter based on the sixth position and a ray tracing technology.

[0170] The simulation control device provided by the embodiments of the present application adopts the simulation control method in the above embodiments, and can solve the technical problem of how to improve the simulation effect of the simulator. Compared with the prior art, the simulation control device provided by the embodiments of the present application has the same beneficial effects as the simulation control method provided by the above embodiments, and other technical features in the simulation control device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0171] The present application provides a simulator, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the simulation control method in the above embodiment one.

[0172] Reference will now be made to the drawings, and Figure 7 which shows a structural schematic diagram of a simulator suitable for implementing the embodiments of the present application. Figure 7 The simulator shown is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0173] As Figure 7 shown, the simulator can include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the simulator are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the simulator to communicate with other devices wirelessly or by wire to exchange data. Although a simulator with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be alternatively implemented or provided.

[0174] In particular, according to the embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments of the present application are executed.

[0175] The simulator provided by the present application adopts the simulation control method in the above-mentioned embodiments, and can solve the technical problem of how to improve the simulation effect of the simulator. Compared with the prior art, the simulator provided by the present application has the same beneficial effects as the simulation control method provided by the above-mentioned embodiments, and other technical features in the simulator are the same as the features disclosed in the last embodiment method, which will not be repeated here.

[0176] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0177] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0178] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the simulation control method in the above-mentioned embodiments.

[0179] The computer readable storage medium provided in the application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electrical wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0180] The above computer readable storage medium can be included in the emulator, or can exist separately without being assembled into the emulator.

[0181] The above computer readable storage medium carries one or more programs, which, when executed by the emulator, cause the emulator to: obtain an original rendering frame corresponding to a display screen of the emulator, wherein the original rendering frame includes a simulation device, a digital person wearing the simulation device, and a virtual environment in which the digital person is located; determine sensor data corresponding to the original rendering frame, determine real-time simulation parameters based on the sensor data, wherein the sensor data represents visual features detected by a sensor on the simulation device; determine a real-time display frame of the simulation device based on the real-time simulation parameters, to determine a simulation screen presented by the simulation device to the digital person based on the real-time display frame.

[0182] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0183] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0184] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0185] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the simulation control method described above, and can solve the technical problem of how to improve the simulation effect of the simulator. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the simulation control method provided by the above embodiments, and will not be described here.

[0186] The embodiment of the present application provides a computer program product, comprising a computer program, which realizes the steps of the simulation control method when executed by a processor.

[0187] The computer program product provided by the embodiment of the present application can improve the simulation effect of the simulator. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as those of the simulation control method provided by the above-mentioned embodiment, and are not described herein.

[0188] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A simulation control method, characterized in that, The simulation control method is applied to a simulator of a head-mounted display device, and the simulation control method includes: Obtain the original rendering frame corresponding to the display screen of the simulator, wherein the original rendering frame includes the simulation device, the digital person wearing the simulation device and the virtual environment in which the digital person is located; Determine the sensor data corresponding to the original rendering frame, wherein the sensor data characterizes the visual features detected by the sensors on the simulation device; The current overall pose of the digital human is determined based on the sensor data, wherein the current overall pose is determined based on the head display pose, environmental map data, video perspective data, hand pose, eye pose, handle pose, and face pose. Real-time simulation parameters are determined based on the current overall pose, wherein the real-time simulation parameters include image distortion parameters, camera simulation parameters, inertial measurement parameters, and illumination simulation parameters; The real-time display frame of the simulation device is determined based on the real-time simulation parameters, and the simulation screen presented to the digital human is determined based on the real-time display frame.

2. The method as described in claim 1, characterized in that, The step of determining the current overall pose of the digital human based on the sensor data includes: Based on the fisheye camera data and inertial measurement data in the sensor data, the head-mounted display pose and environmental map data of the simulation device in the original rendering frame are determined; Based on the color image information and depth camera data in the sensor data, determine the video perspective data in the original rendered frame; Based on the fisheye camera data in the sensor data, determine the hand pose of the digital human in the original rendered frame; Based on the eye image data and infrared-assisted imaging data in the sensor data, the eye pose of the digital human in the original rendering frame is determined; Based on the fisheye camera data, handle light data, and handle motion information in the sensor data, the handle pose of the simulation device in the original rendering frame is determined. Based on the facial image data and infrared-assisted imaging data in the sensor data, the facial pose of the digital human in the original rendering frame is determined; Based on the head-mounted display pose, the environmental map data, the video perspective data, the hand pose, the eye pose, the handle pose, and the face pose, the current overall pose of the digital human is determined.

3. The method as described in claim 1, characterized in that, The real-time simulation parameters include image distortion parameters, and the step of determining the real-time simulation parameters based on the current overall pose includes: Based on the eye pose and head display pose in the current overall pose, determine the positional relationship between the eyes of the digital human, the lenses of the simulation device, and the display screen; The image distortion parameters of the simulation device are determined by the positional relationship.

4. The method as described in claim 1, characterized in that, The real-time simulation parameters also include camera simulation parameters. The step of determining the real-time simulation parameters based on the current overall pose includes: The first pose of the camera on the simulation device is determined based on the current overall pose, and the camera simulation image is determined based on the first pose and the virtual environment, wherein the camera is an imaging device for acquiring two-dimensional visual information. The second pose of the depth camera on the simulation device is determined based on the current overall pose, and a real-time depth map is determined based on the second pose and the virtual environment, wherein the depth camera is an imaging device used to acquire two-dimensional visual information and depth information; The camera simulation image and real-time depth map are used as the camera simulation parameters.

5. The method as described in claim 1, characterized in that, The real-time simulation parameters also include inertial measurement parameters, and the step of determining the real-time simulation parameters based on the current overall pose includes: The third pose of the inertial measurement unit on the simulation device is determined based on the current overall pose. Based on the historical overall pose corresponding to the previous rendering frame of the original rendering frame, the fourth pose of the inertial measurement unit is determined. The fifth pose of the inertial measurement unit is determined based on the future overall pose corresponding to the next rendering frame of the original rendering frame. Based on the position and rotation information of the inertial measurement unit in the third, fourth, and fifth poses, the motion information of the simulation device at the current moment is determined, wherein the motion information includes velocity, acceleration, and angular velocity; The motion information is used as the inertial measurement parameter.

6. The method as described in claim 1, characterized in that, The real-time simulation parameters also include lighting simulation parameters, and the step of determining the real-time simulation parameters based on the current overall pose includes: The sixth pose of the light source on the handle in the simulation device is determined based on the current overall pose, and the lighting simulation parameters are determined based on the sixth pose and ray tracing technology.

7. A simulation control device, characterized in that, The simulation control device is used in a simulator for a head-mounted display device, and the simulation control device includes: The acquisition module is used to acquire the original rendering frame corresponding to the display screen of the simulator, wherein the original rendering frame includes the simulation device, the digital person wearing the simulation device and the virtual environment in which the digital person is located; The determination module is used to determine the sensor data corresponding to the original rendering frame, wherein the sensor data characterizes the visual features detected by the sensors on the simulation device; determine the current overall pose of the digital human based on the sensor data, wherein the current overall pose is determined based on the head-mounted display pose, environmental map data, video perspective data, hand pose, eye pose, handle pose, and face pose; and determine real-time simulation parameters based on the current overall pose, wherein the real-time simulation parameters include image distortion parameters, camera simulation parameters, inertial measurement parameters, and lighting simulation parameters. The display module is used to determine the real-time display frame of the simulation device based on the real-time simulation parameters, so as to determine the simulation screen presented to the digital human by the simulation device based on the real-time display frame.

8. A simulator, characterized in that, The simulator includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the simulation control method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the simulation control method as described in any one of claims 1 to 6.

Images