A wind tunnel VR simulation method and device based on multi-modal collaborative control

Abstract

The application provides a wind tunnel VR simulation method and device for multi-modal collaborative control, relates to the technical field of virtual reality simulation, and comprises the following steps: acquiring multi-source sensor data of an experimenter in a virtual scene, and pre-processing the multi-source sensor data; performing fusion processing on the pre-processed multi-source sensor data, and calculating whole-body posture data of the experimenter; identifying the action intention of the experimenter based on the whole-body posture data, and evaluating the body state category of the experimenter; generating wind tunnel control instructions and VR scene control instructions according to the action intention and the body state category of the experimenter; updating the virtual scene according to the VR scene control instructions, and adjusting the wind speed of the wind tunnel according to the wind tunnel control instructions. The application is used for solving the problem that the existing simulation method lacks intelligent interaction, which seriously affects the immersion of the experimenter and may also cause the experimenter to feel dizzy and uncomfortable.

Description

Technical Field

[0001] This invention relates to the field of virtual reality simulation technology, and more specifically, to a wind tunnel VR simulation method and apparatus for multimodal collaborative control. Background Technology

[0002] Existing wind tunnel VR simulation methods typically consist of independent VR systems, wind tunnel simulation systems, and triggering systems. Due to the lack of real-time data exchange and collaborative control between these systems, it's difficult for the virtual environment, wind speed, and the user to interact effectively. Specifically, wind speed control in wind tunnel simulation systems is often manually preset and cannot adaptively adjust based on the user's body posture in real time; similarly, changes in the VR scene are often pre-programmed and unrelated to the user's actual movements and wind speed perception. This lack of intelligent interaction severely impacts the user's immersion and may even cause dizziness or other discomfort. Summary of the Invention

[0003] The purpose of this invention is to provide a wind tunnel VR simulation method and apparatus for multimodal collaborative control, so as to improve the above-mentioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0004] Firstly, this application provides a wind tunnel VR simulation method for multimodal cooperative control, including:

[0005] Acquire multi-source sensor data from the user in the virtual scene, and preprocess the multi-source sensor data;

[0006] The preprocessed multi-source sensor data is fused to calculate the full-body posture data of the user.

[0007] Based on the full-body posture data, the user's movement intentions are identified, and the user's physical state category is assessed.

[0008] Based on the user's intentions and physical condition, wind tunnel control commands and VR scene control commands are generated.

[0009] The virtual scene is updated according to the VR scene control instructions, and the wind tunnel wind speed is adjusted according to the wind tunnel control instructions.

[0010] Secondly, this application also provides a wind tunnel VR simulation device with multimodal cooperative control, comprising:

[0011] The preprocessing module is used to acquire multi-source sensor data from the user in the virtual scene and to preprocess the multi-source sensor data.

[0012] The fusion and calculation module is used to fuse the preprocessed multi-source sensor data to calculate the user's full-body posture data.

[0013] The identification and assessment module is used to identify the user's movement intentions based on the full-body posture data and to assess the user's physical state category.

[0014] The instruction generation module is used to generate wind tunnel control instructions and VR scene control instructions based on the user's action intentions and body state categories;

[0015] The execution module is used to update the virtual scene according to the VR scene control instructions and to adjust the wind tunnel wind speed according to the wind tunnel control instructions.

[0016] Thirdly, this application also provides a wind tunnel VR simulation device with multimodal collaborative control, comprising:

[0017] Memory, used to store computer programs;

[0018] A processor is used to implement the steps of the wind tunnel VR simulation method for multimodal cooperative control when executing the computer program.

[0019] Fourthly, this application also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the wind tunnel VR simulation method based on multimodal cooperative control described above.

[0020] The beneficial effects of this invention are as follows:

[0021] This invention, by recognizing the user's full-body posture data, can predict the user's movement intentions, enabling the user to control the wind tunnel speed and VR scene in real time with natural and continuous body movements. This ensures strict synchronization between the wind speed regulated by the wind tunnel and the virtual scene updated by VR rendering, achieving multimodal synchronization of vision, force, and proprioception. This fundamentally solves the problem of sensory fragmentation, significantly enhancing immersion and reducing dizziness.

[0022] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the wind tunnel VR simulation method for multimodal collaborative control described in this embodiment of the invention;

[0025] Figure 2 This is a schematic diagram of the structure of the wind tunnel VR simulation device with multimodal collaborative control as described in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of the wind tunnel VR simulation device with multimodal collaborative control as described in an embodiment of the present invention.

[0027] Marked in the image:

[0028] 800. Wind tunnel VR simulation equipment with multimodal collaborative control; 801. Processor; 802. Memory; 803. Multimedia component; 804. I / O interface; 805. Communication component. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0030] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] Example 1:

[0032] This embodiment provides a multimodal collaborative control method for wind tunnel VR simulation, mainly applied in fields such as flight training, parachute simulation, and special operations training. For example, in parachute simulation training, the participant wears equipment integrating multi-source sensors and enters a vertical wind tunnel. The system senses the participant's full-body movements and physiological state in real time, identifies intentions such as "forward acceleration" and "left turn," and simultaneously adjusts the wind tunnel wind force and VR scene, allowing the participant to feel the lift of real airflow, thereby obtaining a near-realistic immersive experience.

[0033] See Figure 1 The figure shows that this method includes:

[0034] S1. Acquire multi-source sensor data of the user in the virtual scene, and preprocess the multi-source sensor data. The multi-source sensor data is collected by a multimodal perception subsystem deployed on the user's body and in the wind tunnel environment. The multimodal perception subsystem specifically includes:

[0035] Inertial sensor unit: Composed of multiple nine-axis inertial sensors, each of which is worn on major body segments and joints of the user, such as: left and right upper arms, left and right forearms, left and right thighs, left and right calves, chest, back, hands, and feet. Each nine-axis inertial sensor is used to capture the local motion of its corresponding body part, generating angle and acceleration data for that part.

[0036] Optical sensor: Composed of multiple high-speed infrared optical cameras, which capture reflective markers fixed on the wearer's clothing at a high frame rate and calculate the three-dimensional spatial coordinates of each marker in the global coordinate system using the principle of triangulation.

[0037] Physiological sensors: worn on the user to monitor the user's heart rate, blood oxygen, respiratory rate, and respiratory depth.

[0038] Specifically, step S1 includes:

[0039] S11. Receive raw data streams from an optical sensor, a physiological sensor, and at least one inertial sensor, each raw data stream comprising several data points;

[0040] S12. Obtain a unified clock reference, synchronize the time of multiple raw data streams according to the unified clock reference, and attach a unified timestamp to each data point of each raw data stream. ;

[0041] In this embodiment, in order to achieve unified alignment of multi-source data on the time axis, multiple original data streams are synchronized in time according to a unified clock reference.

[0042] S13. Encapsulate time-synchronized data with the same or adjacent timestamps into data packets according to a predefined format;

[0043] Specifically, the timestamp should meet the requirements. The data points are used as a set of data within a time window, where... Represents a timestamp. Indicates a reference time point. This represents the time difference. Subsequently, multiple raw data points from the same time window—namely, angular velocity and acceleration data, spatial position data, and physiological data—are packaged into a standardized data packet.

[0044] Based on the above embodiments, this method further includes:

[0045] S2. The preprocessed multi-source sensor data is fused to calculate the full-body posture data of the user;

[0046] Specifically, step S2 includes:

[0047] S21. Extract the angular velocity and acceleration data of the inertial sensor and the spatial position data of the optical sensor from the preprocessed data;

[0048] Among them, for the first There are 1 inertial sensor, and their corresponding angular velocities and accelerations are as follows:

[0049] ;

[0050] In the formula, Represents the angular velocity vector. These represent the three axes respectively. Represents the acceleration vector. Indicates the first An inertial sensor, This indicates transpose.

[0051] For the The spatial locations of the marked points are:

[0052] ;

[0053] in, Represents spatial location coordinates, These represent the three axes respectively. Indicates the first One marker point, This indicates transpose.

[0054] S22. A fusion algorithm is used to fuse and correct the angular velocity, acceleration data and spatial position data. In this embodiment, data from different sensors with different characteristics and coordinate systems are combined to obtain a better state estimate.

[0055] Specifically, definition ;

[0056] in, Represents the state vector. A unit quaternion representing the orientation of body segments. Let be the zero bias vector of the gyroscope. Let the velocity vector be the velocity vector of the body segment. Let the position vector of the body segment be... The zero bias vector of the accelerometer. This indicates transposition. Here, the body segment refers to the rigid body corresponding to the skeletal segment, which is used for mechanical calculations; the thigh segment is an example.

[0057] State prediction is performed using the angular velocity and acceleration data:

[0058] ;

[0059] In the formula, Indicates the predicted posture, A unit quaternion representing the orientation of body segments. This represents the quaternion multiplication operator. Represents the angular velocity vector. This represents the zero bias vector of the gyroscope. Indicates the predicted speed. for The resulting rotation matrix It is the acceleration due to gravity. Indicates the predicted location. This is the velocity vector of the joint.

[0060] Constructing observation equations using spatial location data:

[0061] ;

[0062] In the formula, Represents the observation residual. Represents spatial location coordinates, This indicates the theoretical position of the marker point relative to the corresponding joint. Represents the observation matrix. Indicates the error status. To observe noise.

[0063] In this embodiment, the observation residual is utilized. For error state Perform corrections and output the optimal estimated state.

[0064] S23. Based on the preset human skeletal model, the corrected data is converted into a unified global coordinate system to calculate the full-body posture data of the experiencer.

[0065] Specifically, step S23 includes:

[0066] S231. Based on the joint connection relationships defined in the human skeletal model, establish a transformation relationship from the sensor local coordinate system to the global coordinate system;

[0067] Specifically, the transformation matrix is ​​obtained through calibration experiments, and the transformation matrix describes the rotation and translation relationship between the sensor coordinate system and the skeleton coordinate system.

[0068] S232. Based on the transformation relationship and spatial position data, calculate the spatial position of the human joint in the global coordinate system, and associate the angular velocity and acceleration data with the human joint that drives the body segment:

[0069] ;

[0070] In the formula, This indicates the coordinate position of the joint center point in the global coordinate system. Indicates the first One joint, These represent the position and orientation of the bone segment in the global coordinate system, respectively. Represents the rotation matrix. This indicates the geometric position offset.

[0071] The angular velocities and accelerations of each bone segment after fusion correction are assigned to the corresponding joints that drive the bone segment's motion. For example, the angular velocity of the thigh bone represents the component of the angular velocity of the hip and knee joints in the direction of the large bone.

[0072] S233. Based on the spatial position, angular velocity, and acceleration data of the joint, calculate the three-dimensional rotation angle and angular velocity of the joint;

[0073] Specifically, taking the elbow joint as an example, the elbow joint is defined by the upper arm bones and the forearm bones. First, the pose quaternions of the upper arm bones and the forearm bones in the global coordinate system are obtained. Then, the relative rotation quaternion from the forearm bone coordinate system to the upper arm bone coordinate system is calculated to obtain the three-dimensional rotation angle. The expression of the angular velocity of the forearm bones in the upper arm bone coordinate system is used as the angular velocity of the elbow joint.

[0074] S234. Obtain the mass distribution parameters of the human skeleton model, and calculate the position, velocity and acceleration of the experiencer's center of mass based on the spatial position of the joints and the mass distribution parameters of the human skeleton model;

[0075] Specifically, the mass and center of mass of each body segment are obtained from the mass distribution parameters of the human skeletal model, thereby calculating the position, velocity, and acceleration of the total center of mass of the whole body:

[0076] ;

[0077] In the formula, Indicates location, Indicates the quality of bodily components. Indicates the first Each body part, Indicates the location of the center of mass of a body segment. Indicates speed, To express differentiation, Indicates time, It represents acceleration.

[0078] Based on the above embodiments, this method further includes:

[0079] S3. Based on the full-body posture data, identify the user's movement intentions and assess the user's body state category;

[0080] Specifically, step S3 includes:

[0081] S31. Construct the whole-body posture data within a continuous time window into a temporal data sequence;

[0082] In this embodiment, the whole-body posture data is updated once in each processing cycle. The whole-body posture data includes the three-dimensional rotation angle and three-dimensional angular velocity of the main joints, as well as the position, velocity and acceleration of the whole body center of mass. The whole-body posture data of each processing cycle can be represented as a data vector with a D feature dimension.

[0083] When constructing a time series, set the time window length. Extracting data from the current moment, backtracking forward... The continuous data from each processing cycle is arranged in chronological order to form a two-dimensional time-series data matrix. The number of rows in the time-series data matrix is... The number of columns is D.

[0084] S32. Input the time-series data sequence into a pre-trained time-series deep learning model to generate the probability corresponding to each predefined action intention category;

[0085] In this embodiment, the temporal deep learning model is a lightweight recurrent neural network model containing gated recurrent unit (GRU) layers. The temporal deep learning model has been trained offline using a large amount of labeled training data. The training data was collected in a wind tunnel simulating various flight maneuvers, and each segment of temporal attitude data corresponds to a labeled high-level maneuver intent category label, such as forward acceleration, backward deceleration, left tilt in preparation for a left turn, limb tuck and roll, maintaining stability, etc.

[0086] The input layer of the temporal deep learning model receives a sequence of temporal data. After the GRU units learn the dynamic patterns and dependencies of pose changes over time, the output layer outputs a probability distribution vector. This probability distribution vector contains the probability corresponding to each action intent category.

[0087] S33. Based on the probability of each action intention, the action intention with the highest probability is taken as the current action intention of the experiencer;

[0088] For example, suppose there are 5 predefined action categories: [leaning forward to accelerate, leaning back to decelerate, turning left, turning right, stabilizing], and the output probability vector is [0.02, 0.05, 0.80, 0.10, 0.03]. Then, the user's current action intention should be "turn left".

[0089] Specifically, step S3 further includes:

[0090] S34. Based on the whole-body posture data, calculate at least one posture index, the posture index including at least one of body center of mass offset and joint angle change variance;

[0091] Specifically, the method for calculating the body's center of mass offset is as follows:

[0092] Obtain the projected coordinates of the experiencer's centroid on the horizontal plane in each frame of data within a time window, calculate the average position of the centroid's projected coordinates within the time window, and calculate the standard deviation of the centroid's projected point relative to the average position as the centroid offset.

[0093] Specifically, the method for calculating the variance of the joint angle change is as follows:

[0094] Multiple consecutive angle values ​​of a joint within a time window are obtained, and the variance of the joint angle change is calculated based on these multiple consecutive angle values. In this embodiment, a larger variance indicates more jittery and unstable movement.

[0095] S35. Obtain physiological data from the participant and extract at least one physiological state indicator from it, wherein the physiological state indicator includes at least one of heart rate variability and respiratory rate;

[0096] Specifically, peak detection is performed on the electrocardiogram signal to identify each heartbeat cycle, resulting in a continuous heartbeat interval sequence. By calculating the root mean square characteristics of the differences between adjacent heartbeat intervals in the heartbeat interval sequence, the heart rate variability index is obtained.

[0097] The heart rate variability is used to assess the participant's psychological stress, tension, or fatigue, while the respiratory rate is used to assess the participant's tension, excitement, or discomfort.

[0098] S36. Based on the at least one posture index and the at least one physiological state index, determine the physical state category of the experiencer according to a preset evaluation rule;

[0099] Specifically, the preset evaluation rules are shown in Table 1:

[0100] Table 1

[0101]

[0102] Based on the above rules, the current state of the user can be determined.

[0103] Based on the above embodiments, this method further includes:

[0104] S4. Generate wind tunnel control commands and VR scene control commands based on the user's action intentions and body state category;

[0105] Specifically, based on the user's action intention, the system queries the action intention-control parameter mapping database, which defines corresponding initial target values ​​for each preset action intention. For example, for the wind tunnel subsystem, the initial target values ​​include the basic target wind speed and the basic airflow direction angle. For the VR subsystem, the initial target values ​​include the basic motion acceleration and scene event commands.

[0106] Specifically, based on the user's physical state, a preset adjustment strategy is invoked to correct the initial target value in real time. For example, when the physical state is "tense" or "tending to be unbalanced," the sensitivity of changes in control commands is reduced, making changes in wind speed and virtual scene response smoother to help the user regain balance and reduce tension; when the state is "stable," a higher sensitivity is used to provide sensitive control feedback.

[0107] Based on the above embodiments, this method further includes:

[0108] S5. Update the virtual scene according to the VR scene control command, and adjust the wind tunnel wind speed according to the wind tunnel control command;

[0109] In this embodiment, after receiving the VR scene control command, the target wind speed setting value is obtained. Combined with real-time wind speed feedback, the fan speed of the real wind tunnel is adjusted by controlling the frequency converter, thereby changing the physical wind speed acting on the user.

[0110] Upon receiving VR scene control commands, the system calculates the spatial position and orientation of relevant objects in the scene in the next frame based on the aerodynamics, acceleration, and angular velocity parameters in the commands. Based on the calculation results, it renders the corresponding immersive 3D image at a rate of no less than 90 frames per second and presents it to the user through a VR headset. This ensures that the virtual scene movement seen by the user is consistent with the real wind speed changes felt by their body.

[0111] Based on the above embodiments, this method further includes:

[0112] S6. Based on the new body movements generated by the user in the updated virtual scene, repeatedly acquire the user's multi-source sensor data to form a closed-loop control, ensuring the immersion and safety of the experience.

[0113] Example 2:

[0114] like Figure 2 As shown, this embodiment provides a wind tunnel VR simulation device with multimodal collaborative control, the device comprising:

[0115] The preprocessing module is used to acquire multi-source sensor data from the user in the virtual scene and to preprocess the multi-source sensor data.

[0116] The fusion and calculation module is used to fuse the preprocessed multi-source sensor data to calculate the user's full-body posture data.

[0117] The identification and assessment module is used to identify the user's movement intentions based on the full-body posture data and to assess the user's physical state category.

[0118] The instruction generation module is used to generate wind tunnel control instructions and VR scene control instructions based on the user's action intentions and body state categories;

[0119] The execution module is used to update the virtual scene according to the VR scene control instructions and to adjust the wind tunnel wind speed according to the wind tunnel control instructions.

[0120] Based on the above embodiments, the preprocessing module includes:

[0121] A receiving unit is used to receive raw data streams from an optical sensor, a physiological sensor, and at least one inertial sensor;

[0122] The synchronization unit is used to synchronize multiple raw data streams in time and attach a unified timestamp to each data point;

[0123] The encapsulation unit is used to encapsulate time-synchronized data into data packets according to a predefined format.

[0124] Based on the above embodiments, the fusion calculation module includes:

[0125] The extraction unit is used to extract the angular velocity and acceleration data of the inertial sensor and the spatial position data of the optical sensor from the preprocessed data.

[0126] The correction unit is used to fuse and correct the angular velocity, acceleration data and spatial position data using a fusion algorithm;

[0127] The calculation unit is used to convert the corrected data into a unified global coordinate system based on a preset human skeletal model in order to calculate the full-body posture data of the user.

[0128] Based on the above embodiments, the calculation unit includes:

[0129] The transformation subunit is used to establish the transformation relationship from the sensor's local coordinate system to the global coordinate system based on the joint connection relationship defined in the human skeleton model;

[0130] The associated sub-unit is used to calculate the spatial position of human joints in the global coordinate system and associate angular velocity and acceleration data with human joints;

[0131] The joint parameter subunit is used to calculate the three-dimensional rotation angle and angular velocity of the joint based on the joint's spatial position, angular velocity, and acceleration data.

[0132] The center of mass state sub-unit is used to calculate the position, velocity, and acceleration of the experiencer's center of mass based on the spatial position of the joints and the mass distribution parameters of the human skeletal model.

[0133] Based on the above embodiments, the identification and evaluation module includes:

[0134] Sequence construction unit, used to construct a temporal data sequence from whole-body pose data within a continuous time window;

[0135] The model processing unit is used to input the time series data sequence into the pre-trained time series deep learning model to generate the probability corresponding to each predefined action intention category;

[0136] The intent determination unit is used to determine the action intent with the highest probability as the user's current action intent based on the probability of each action intent.

[0137] Based on the above embodiments, the identification and evaluation module further includes:

[0138] The posture index unit is used to calculate at least one posture index based on whole-body posture data, wherein the posture index includes at least one of body center of mass offset and joint angle variation variance.

[0139] A physiological indicator unit is used to extract at least one physiological state indicator from the physiological data of the participant, said physiological state indicator including at least one of heart rate variability and respiratory rate;

[0140] The state determination unit is used to determine the physical state category of the experiencer based on the at least one posture index and the at least one physiological state index, according to a preset evaluation rule.

[0141] It should be noted that the specific manner in which each module performs its operation in the apparatus described in the above embodiments has been described in detail in the embodiments of the method, and will not be elaborated here.

[0142] Example 3:

[0143] Corresponding to the above method embodiments, this embodiment also provides a wind tunnel VR simulation device with multimodal collaborative control. The wind tunnel VR simulation device with multimodal collaborative control described below and the wind tunnel VR simulation method with multimodal collaborative control described above can be referred to in correspondence.

[0144] Figure 3 This is a block diagram illustrating a multimodal collaborative control wind tunnel VR simulation device 800 according to an exemplary embodiment. Figure 3 As shown, the multimodal collaborative control wind tunnel VR simulation device 800 may include: a processor 801 and a memory 802. The multimodal collaborative control wind tunnel VR simulation device 800 may also include one or more of the following: a multimedia component 803, an I / O interface 804, and a communication component 805.

[0145] The processor 801 controls the overall operation of the multimodal collaborative control wind tunnel VR simulation device 800 to complete all or part of the steps in the aforementioned multimodal collaborative control wind tunnel VR simulation method. The memory 802 stores various types of data to support the operation of the multimodal collaborative control wind tunnel VR simulation device 800. This data may include, for example, instructions for any application or method operating on the multimodal collaborative control wind tunnel VR simulation device 800, as well as application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 802 can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 802 or transmitted via the communication component 805. The audio component also includes at least one speaker for outputting audio signals. I / O interface 804 provides an interface between processor 801 and other interface modules, such as keyboards, mice, and buttons. These buttons can be virtual or physical. Communication component 805 is used for wired or wireless communication between the multimodal collaborative control wind tunnel VR simulation device 800 and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof. Therefore, the corresponding communication component 805 may include a Wi-Fi module, a Bluetooth module, and an NFC module.

[0146] In an exemplary embodiment, the wind tunnel VR simulation device 800 with multimodal collaborative control can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the wind tunnel VR simulation method with multimodal collaborative control described above.

[0147] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the multimodal cooperative control wind tunnel VR simulation method described above. For example, the computer-readable storage medium may be the memory 802 including program instructions described above, which may be executed by the processor 801 of the multimodal cooperative control wind tunnel VR simulation device 800 to complete the multimodal cooperative control wind tunnel VR simulation method described above.

[0148] Example 4:

[0149] Corresponding to the above method embodiments, this embodiment also provides a readable storage medium. The readable storage medium described below can be referred to in relation to the wind tunnel VR simulation method of multimodal cooperative control described above.

[0150] A readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of the wind tunnel VR simulation method of multimodal cooperative control described in the above method embodiments are implemented.

[0151] The readable storage medium can specifically be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other readable storage medium capable of storing program code.

[0152] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0153] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A wind tunnel VR simulation method with multimodal cooperative control, characterized in that, include: Acquire multi-source sensor data from the user in the virtual scene, and preprocess the multi-source sensor data; The preprocessed multi-source sensor data is fused to calculate the full-body posture data of the user. Based on the full-body posture data, the user's movement intentions are identified, and the user's physical state category is assessed. Based on the user's intentions and physical condition, wind tunnel control commands and VR scene control commands are generated. The virtual scene is updated according to the VR scene control instructions, and the wind tunnel wind speed is adjusted according to the wind tunnel control instructions.

2. The wind tunnel VR simulation method with multimodal cooperative control according to claim 1, characterized in that, Acquire multi-source sensor data from the user in a virtual scene, and preprocess the multi-source sensor data, including: Receive raw data streams from optical sensors, physiological sensors, and at least one inertial sensor, each raw data stream comprising several data points; A unified clock reference is obtained, and multiple raw data streams are synchronized in time according to the unified clock reference. A unified timestamp is added to each data point of each raw data stream. Data that has been synchronized with time and has the same or adjacent timestamps is encapsulated into data packets according to a predefined format.

3. The wind tunnel VR simulation method for multimodal collaborative control according to claim 1 or 2, characterized in that, The preprocessed multi-source sensor data is fused to calculate the user's full-body posture data, including: The angular velocity and acceleration data of the inertial sensor and the spatial position data of the optical sensor are extracted from the preprocessed data. A fusion algorithm is used to fuse and correct the angular velocity, acceleration data, and spatial position data; Based on a preset human skeletal model, the corrected data is converted into a unified global coordinate system to calculate the full-body posture data of the user.

4. The wind tunnel VR simulation method for multimodal cooperative control according to claim 3, characterized in that, Based on a pre-defined human skeletal model, the corrected data is converted into a unified global coordinate system to calculate the subject's full-body posture data, including: Based on the joint connection relationships defined in the human skeletal model, a transformation relationship from the sensor's local coordinate system to the global coordinate system is established; Based on the transformation relationship and spatial position data, the spatial position of the human joint in the global coordinate system is calculated, and the angular velocity and acceleration data are associated with the human joint. Based on the spatial position, angular velocity, and acceleration data of the joint, the three-dimensional rotation angle and angular velocity of the joint are calculated. The mass distribution parameters of the human skeletal model are obtained, and the position, velocity, and acceleration of the experiencer's center of mass are calculated based on the spatial position of the joints and the mass distribution parameters of the human skeletal model.

5. The wind tunnel VR simulation method for multimodal cooperative control according to claim 1, characterized in that, Identifying the user's intention to move based on the full-body posture data includes: The full-body posture data within a continuous time window is constructed into a temporal data sequence; The time-series data sequence is input into a pre-trained time-series deep learning model to generate probabilities corresponding to each predefined action intention category; Based on the probability of each action intention, the action intention with the highest probability is taken as the current action intention of the experiencer.

6. A wind tunnel VR simulation device with multimodal collaborative control, characterized in that, include: The preprocessing module is used to acquire multi-source sensor data from the user in the virtual scene and to preprocess the multi-source sensor data. The fusion and calculation module is used to fuse the preprocessed multi-source sensor data to calculate the user's full-body posture data. The identification and assessment module is used to identify the user's movement intentions based on the full-body posture data and to assess the user's physical state category. The instruction generation module is used to generate wind tunnel control instructions and VR scene control instructions based on the user's action intentions and body state categories; The execution module is used to update the virtual scene according to the VR scene control instructions and to adjust the wind tunnel wind speed according to the wind tunnel control instructions.

7. The wind tunnel VR simulation device with multimodal collaborative control according to claim 6, characterized in that, The preprocessing module includes: A receiving unit is used to receive raw data streams from an optical sensor, a physiological sensor, and at least one inertial sensor; The synchronization unit is used to synchronize multiple raw data streams in time and attach a unified timestamp to each data point; The encapsulation unit is used to encapsulate time-synchronized data into data packets according to a predefined format.

8. The wind tunnel VR simulation device with multimodal collaborative control according to claim 6 or 7, characterized in that, The fusion solution module includes: The extraction unit is used to extract the angular velocity and acceleration data of the inertial sensor and the spatial position data of the optical sensor from the preprocessed data. The correction unit is used to fuse and correct the angular velocity, acceleration data and spatial position data using a fusion algorithm; The calculation unit is used to convert the corrected data into a unified global coordinate system based on a preset human skeletal model in order to calculate the full-body posture data of the user.

9. The wind tunnel VR simulation device with multimodal collaborative control according to claim 8, characterized in that, The solution unit includes: The transformation subunit is used to establish the transformation relationship from the sensor's local coordinate system to the global coordinate system based on the joint connection relationship defined in the human skeleton model; The associated sub-unit is used to calculate the spatial position of human joints in the global coordinate system and associate angular velocity and acceleration data with human joints; The joint parameter subunit is used to calculate the three-dimensional rotation angle and angular velocity of the joint based on the joint's spatial position, angular velocity, and acceleration data. The center of mass state sub-unit is used to calculate the position, velocity, and acceleration of the experiencer's center of mass based on the spatial position of the joints and the mass distribution parameters of the human skeletal model.

10. The wind tunnel VR simulation device with multimodal collaborative control according to claim 6, characterized in that, The identification and evaluation module includes: Sequence construction unit, used to construct a temporal data sequence from whole-body pose data within a continuous time window; The model processing unit is used to input the time series data sequence into the pre-trained time series deep learning model to generate the probability corresponding to each predefined action intention category; The intent determination unit is used to determine the action intent with the highest probability as the user's current action intent based on the probability of each action intent.

Images