Virtual reality experiment system, method and device for multi-modal physiological index collection

Abstract

The present disclosure relates to the technical field of virtual reality, including a virtual reality experiment system, method and device for multi-modal physiological index collection. A pre-created virtual reality experiment scene is displayed by a display device; multi-modal data generated by a test subject based on the virtual reality experiment scene is collected by a data collection device; the data collection device includes a positioner, and the multi-modal data includes positioning data collected by the positioner, the positioning data indicating a trunk pose and a head pose of the test subject; standardized data is obtained by processing the multi-modal data based on a pre-set standardization rule by a data processing device; the virtual reality experiment scene displayed by the display device is updated based on the standardized data, and an experiment response result of the test subject is determined; it can be ensured that the scene content displayed by the virtual reality scene is consistent with the turning action of the test subject, the generation of dizziness is reduced, and the experiment effect is improved. At the same time, the experiment response result can be determined based on the multi-modal data, which can improve the accuracy of the experiment response result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of virtual reality, and particularly relates to a virtual reality experiment system, method and device for multi-modal physiological index collection. BACKGROUND

[0002] At present, in order to test physiological reactions and behavior performances of the public in emergency, a virtual reality experiment system can be used to simulate a real emergency, collect actions taken by a subject based on the simulated emergency, and realize experiment on the subject.

[0003] A conventional virtual reality experiment system includes: collecting actions of a subject through a head-mounted reality device and / or a control device to control movement and turning of the subject in a virtual reality experiment scene.

[0004] However, the real object actions collected by the conventional virtual reality experiment system may not be consistent with the turning actions mapped to the virtual reality experiment scene, which may cause discomfort of the subject, such as physiological reactions of dizziness, nausea, etc., and result in poor experiment effect. SUMMARY

[0005] Therefore, the present disclosure provides a virtual reality experiment system, method and device for multi-modal physiological index collection, which can solve the problem that the scene content displayed in a virtual reality scene is inconsistent with the actions of a subject in a conventional virtual reality system, thereby causing dizziness and poor experiment effect.

[0006] According to an aspect of the present disclosure, a virtual reality experiment system for multi-modal physiological index collection is provided, which includes a display device, a data collection device, and a data processing device in communication with the display device and the data collection device, respectively.

[0007] The display device is configured to display a pre-created virtual reality experiment scene.

[0008] The data collection device is configured to collect multi-modal data generated by a subject based on the virtual reality experiment scene; the data collection device includes a position meter, and correspondingly, the multi-modal data includes positioning data collected by the position meter, the positioning data indicating a trunk pose and a head pose of the subject.

[0009] The data processing device is configured to process the multi-modal data based on a pre-set standardization rule to obtain standardized data, update the virtual reality experiment scene displayed by the display device based on the standardized data, and determine an experiment reaction result of the subject.

[0010] In a possible implementation, the updating of the virtual reality experiment scene displayed by the display device based on the standardized data comprises:

[0011] determining, based on the standardized trunk pose data, a scene position and a trunk orientation of the subject in the virtual reality experiment scene;

[0012] determining, based on the standardized head pose data, a head orientation of the subject;

[0013] controlling the display device to display scene content in the virtual reality experiment scene at a position indicated by the trunk orientation and the head orientation.

[0014] In a possible implementation, the positioner comprises a first sensing component arranged on the trunk of the subject, a second sensing component arranged on the head of the subject, and a signal emitting component arranged in a real space in which the subject is located; the positioner collects the positioning data, comprising:

[0015] sensing, by the first sensing component, a preset signal emitted by the signal emitting component; determining, based on the sensed preset signal, a first relative position of the signal emitting component relative to the first sensing component; determining, based on the first relative position and a setting position of the signal emitting component, a position and a direction of the trunk of the subject, to obtain trunk pose data;

[0016] sensing, by the second sensing component, a preset signal emitted by a different signal emitting component; determining, based on the sensed preset signal, a second relative position of the signal emitting component relative to the second sensing component; determining, based on the second relative position and a setting position of the signal emitting component, a position and a direction of the head of the subject, to obtain head pose data.

[0017] In a possible implementation, the data collection device further comprises an eye tracker, and correspondingly, the multi-modal data comprises two-dimensional eye movement data collected by the eye tracker; the data processing device is further configured to:

[0018] determining, based on the head pose data and the eye movement data, an eye position of the subject in the virtual reality experiment scene;

[0019] generating a ray along a line of sight of the subject based on the eye position and the virtual reality experiment scene;

[0020] determining a three-dimensional eye movement coordinate in the virtual reality environment based on a position of intersection of the ray and a scene object in the virtual reality experiment scene.

[0021] In a possible implementation, the data acquisition device further comprises at least one of the following:

[0022] An electromyography device configured to acquire a skin surface muscle electrical signal of the subject;

[0023] An electroencephalography device configured to acquire brain neuron electrical activity data of a scalp surface of the subject;

[0024] An action capture device configured to acquire whole-body spatial motion data of the subject;

[0025] A near-infrared device configured to acquire brain oxyhemoglobin and deoxyhemoglobin data of the subject.

[0026] In a possible implementation, the processing the multi-modal data based on the preset standardization rule to obtain standardized data comprises:

[0027] mapping data related to a scene position of the virtual reality experiment scene to a virtual scene coordinate system corresponding to the virtual reality experiment scene;

[0028] adjusting a collection frequency of the multi-modal data to a preset frequency;

[0029] converting the frequency-synchronized data into a preset format to obtain the standardized data.

[0030] In a possible implementation, the data processing device is further configured to:

[0031] store motion data indicated by the multi-modal data during interaction of the subject with the virtual reality experiment scene;

[0032] in a case where a process playback instruction is received, use the motion data to perform experiment process playback based on a playback technology.

[0033] According to another aspect of the present disclosure, a multi-modal physiological indicator acquisition virtual reality experiment method is provided, which is used for a data processing device in the virtual reality experiment system, and the method comprises:

[0034] controlling the display device to display a pre-created virtual reality experiment scene;

[0035] acquiring multi-modal data of a subject collected by the data acquisition device; the multi-modal data comprises positioning data collected by the positioner, and the positioning data indicates a trunk pose and a head pose of the subject;

[0036] processing the multi-modal data based on a preset standardization rule to obtain standardized data;

[0037] update the virtual reality experiment scene displayed by the display device based on the standardized data, and determine the experiment reaction result of the subject.

[0038] According to another aspect of the present disclosure, a virtual reality experiment device for multi-modal physiological index acquisition is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0039] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium having computer program instructions stored thereon is provided, wherein the computer program instructions are executed by a processor to implement the above method.

[0040] According to another aspect of the present disclosure, a computer program product is provided, comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above method.

[0041] A pre-created virtual reality experiment scene is displayed by a display device; multi-modal data generated by a subject based on the virtual reality experiment scene is collected by a data collection device; the data collection device comprises a position meter, and correspondingly, the multi-modal data comprises position data collected by the position meter, the position data indicating a trunk pose and a head pose of the subject; standardized data is obtained by processing the multi-modal data based on a pre-set standardization rule by a data processing device; the virtual reality experiment scene displayed by the display device is updated based on the standardized data, and an experiment reaction result of the subject is determined; the problem that the scene content of the virtual reality scene display is inconsistent with the action of the subject in the traditional virtual reality system, thereby generating a dizziness and a poor experiment effect can be solved; in this embodiment, the turning operation of the subject in the virtual reality scene can be determined by combining the trunk pose and the head pose, the scene content of the virtual reality scene display is ensured to be consistent with the action of the subject, the generation of dizziness is reduced, and the experiment effect is improved. At the same time, the accuracy of determining the experiment reaction result can be improved based on the multi-modal data.

[0042] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0044] Figure 1A structural schematic diagram of a virtual reality experiment system for multi-modal physiological index collection is shown according to an embodiment of the present disclosure.

[0045] Figure 2 A schematic diagram of an eye movement information heat map is shown according to an embodiment of the present disclosure.

[0046] Figure 3 A schematic diagram of a body part of a subject captured by a motion capture device is shown according to an embodiment of the present disclosure.

[0047] Figure 4 A flow chart of a virtual reality experiment method for multi-modal physiological index collection is shown according to an embodiment of the present disclosure.

[0048] Figure 5 A block diagram of a virtual reality experiment apparatus for multi-modal physiological index collection is shown according to an embodiment of the present disclosure.

[0049] Figure 6 A block diagram of a virtual reality experiment apparatus for multi-modal physiological index collection is shown according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] Various exemplary embodiments, features and aspects of the present disclosure will be explained in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0051] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0052] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated in the present disclosure can be omitted. It will be appreciated by those skilled in the art that the present disclosure can be practiced without these specific details. In some instances, well-known methods, structures, elements, and circuits have not been described in detail in order to avoid obscuring the purpose and features of the present disclosure.

[0053] Figure 1 A structural schematic diagram of a virtual reality experiment system for multi-modal physiological index collection is shown according to an embodiment of the present disclosure. As shown in the figure, the system includes a display apparatus 110, a data collection apparatus 120, and a data processing apparatus 130 connected in communication with the display apparatus 110 and the data collection apparatus 120, respectively. Figure 1

[0054] ​The display device 110 is configured to display a pre-created virtual reality experiment scene. In this embodiment, the subject can watch the virtual reality experiment scene displayed by the display device 110 in real time during the movement. Illustratively, the display device 110 is a head-mounted display, such as a virtual reality (VR) glasses.

[0055] The virtual reality experiment scene is configured to simulate a real experiment scene, so as to test the reaction of the subject to the virtual reality experiment scene and obtain an experiment reaction result. For example, the virtual reality experiment scene is configured to simulate a real disaster scene. The embodiment is not limited to the content of the virtual reality experiment scene.

[0056] Optionally, the virtual reality experiment scene is generated by the data processing device 130 and displayed by the display device 110, or the virtual reality experiment scene is generated by another device connected to the virtual reality experiment system in communication and sent to the virtual reality experiment system, and then displayed by the display device 110. The embodiment is not limited to the source of the virtual reality experiment scene.

[0057] Optionally, the display device 110 is installed with a three-dimensional engine, and the three-dimensional engine is called to render and display the virtual reality experiment scene, or the display device 110 communicates with the data processing device 130 to instruct the data processing device 130 to call the three-dimensional engine installed therein to render the virtual reality experiment scene, and the scene picture obtained by the rendering is fed back to the display device 110 for display. The embodiment is not limited to the way in which the display device 110 displays the virtual reality experiment scene.

[0058] The data acquisition device 120 is configured to acquire multi-modal data generated by the subject based on the virtual reality experiment scene. The multi-modal data includes data corresponding to different parts of the subject when the subject makes a certain reaction.

[0059] In this embodiment, the data acquisition device 120 includes a position detector, and correspondingly, the multi-modal data includes positioning data acquired by the position detector, which indicates the trunk pose and the head pose of the subject.

[0060] In a conventional virtual reality system, the head pose of the subject is generally acquired, such as installing a position detector in a head-mounted display to acquire the head pose. Then, the turning operation of the subject in the virtual reality scene is determined based on the head pose. However, the rotation of the head of the subject does not represent the real turning operation, and the turning of the subject in the virtual reality scene determined by the head pose will cause the scene content displayed by the virtual reality scene to be inconsistent with the action of the subject, thereby causing dizziness and poor experiment effect.

[0061] In the embodiment, the position detector can collect the trunk pose and the head pose of the subject, so that the turning operation of the subject in the virtual reality scene can be determined by combining the trunk pose and the head pose, the scene content displayed in the virtual reality scene is ensured to be consistent with the action of the subject, the generation of dizziness is reduced, and the experimental effect is improved.

[0062] In one example, the position detector includes a first sensing component arranged on the trunk of the subject, a second sensing component arranged on the head of the subject, and a signal emitting component arranged in the real space where the subject is located. Accordingly, the position detector collects positioning data, including:

[0063] The first sensing component senses the preset signal emitted by the signal emitting component; based on the sensed preset signal, the first relative position of the signal emitting component relative to the first sensing component is determined; based on the first relative position and the arrangement position of the signal emitting component, the position and direction of the trunk of the subject are determined, and the trunk pose data is obtained;

[0064] The second sensing component senses the preset signal emitted by different signal emitting components; based on the sensed preset signal, the second relative position of the signal emitting component relative to the second sensing component is determined; based on the second relative position and the arrangement position of the signal emitting component, the position and direction of the head of the subject are determined, and the head pose data is obtained.

[0065] The first sensing component and the second sensing component are used to sense the preset signal emitted by the signal emitting component. Illustratively, the signal emitting component is a laser emitter, and accordingly, the preset signal is a laser signal, and the first sensing component and the second sensing component are photosensitive sensors. In other embodiments, the signal emitting component can also emit other types of preset signals, and accordingly, the first sensing component and the second sensing component can also be adaptively arranged as other types of sensors, and the type of the first sensing component, the second sensing component and the signal emitting component is not limited in the embodiment.

[0066] Optionally, the first sensing component determines the first relative position of the signal emitting component relative to the first sensing component based on the signal intensity of the preset signal. At this time, the number of signal emitting components is at least three, and each signal emitting component is placed at a different arrangement position in the real space where the subject is located. At this time, the first sensing component can determine the distance between the first sensing component and each signal emitting component based on the signal intensity between the first sensing component and the at least three signal emitting components, and can determine the first relative position based on the distance between the first sensing component and each signal emitting component. In combination with the arrangement position of the signal emitting component, the position and direction of the first sensing component in the real space can be determined.

[0067] Alternatively, the first sensing component is capable of sensing the signal strength and signal direction of the preset signal, and determining the first relative position of the signal transmitting component relative to the first sensing component based on the signal strength and signal direction. At this time, the number of signal transmitting components is at least one. At this time, the first sensing component determines the first relative position based on the sensed signal strength and signal direction, and then, in combination with the setting position of the signal transmitting component, the position and direction of the first sensing component in the real space can be determined.

[0068] The first sensing component is arranged at a position on the test subject that can represent the trunk pose; for example, arranged in front of the waist of the test subject, or behind the waist, etc. At this time, the position and direction of the first sensing component in the real space can be regarded as the position and direction of the trunk in the real space, and the trunk pose data is obtained. The setting position of the first sensing component is not limited in the embodiment. Alternatively, the number of first sensing components is at least one. In the case that the number of first sensing components is at least two, the sensing data collected by the plurality of first sensing components is fused to obtain the trunk pose data.

[0069] The fusion manner includes but is not limited to: taking the first sensing component at the preset position on the trunk of the test subject as the reference sensing component, and taking the first sensing component at other positions as the auxiliary sensing component; determining the fusion weight based on the distance between the auxiliary sensing component and the reference sensing component, and the distance and the fusion weight are in a negative correlation relationship; and performing data fusion on the sensing data collected by each first sensing component based on the fusion weight to obtain the trunk pose data. The distance between the auxiliary sensing component and the reference sensing component and the fusion weight of the reference sensing component are preset values. The preset position can be in front of the waist of the test subject, or behind the waist, etc. The other positions can be the chest, back, arm or leg of the test subject, etc. The implementation manner of the preset position and the other positions is not limited in the embodiment. Alternatively, the first sensing components at different positions correspond to different preset weights. The sensing data collected by each first sensing component is fused according to the preset weight corresponding to the different positions to obtain the trunk pose data. The fusion manner is not limited in the embodiment.

[0070] Alternatively, the data fusion of the sensing data can be realized by the data processing device 130. At this time, the data processing device 130 obtains the sensing data collected by each first sensing component, and performs data fusion to obtain the trunk pose data. Alternatively, the data fusion of the sensing data can also be realized by one of the first sensing components. At this time, the first sensing component is in communication connection with other sensing components to obtain the sensing data collected by other sensing components for data fusion.

[0071] The manner of obtaining the head pose data is the same as that of obtaining the torso pose data, except that the second sensing component is arranged on the head of the subject, and the obtained data is the head pose data. Details are not repeated herein.

[0072] In other embodiments, the positioning instrument can also be implemented to include a sensing component arranged on the torso of the subject, and a signal transmitting component arranged in the real space where the subject is located; and further include a positioning sensor arranged on the head of the subject. At this time, the sensing component and the signal transmitting component cooperate to collect the torso pose data, and the positioning sensor works alone to collect the head pose data. That is, the positioning sensor collects the head pose data without relying on the preset signal transmitted by the signal transmitting component. Optionally, the positioning sensor includes but is not limited to an accelerometer, a gyroscope, and / or a magnetometer, etc., and the implementation manner of the positioning sensor is not limited herein.

[0073] In other embodiments, the manner of collecting the head pose data and the torso pose data can also be other manners, and the implementation manner of the head pose data and the torso pose data is not limited herein.

[0074] The data acquisition device 120 and the data processing device 130 are connected through wired communication or wireless communication. After the data acquisition device 120 collects the multi-modal data, the multi-modal data is sent to the data processing device 130.

[0075] Correspondingly, the data processing device 130 is configured to process the multi-modal data based on a preset standardization rule to obtain standardized data, update the virtual reality experiment scene displayed by the display device 110 based on the standardized data, and determine the experimental response result of the subject.

[0076] In one example, processing the multi-modal data based on the preset standardization rule to obtain the standardized data includes: mapping data related to a scene position of the virtual reality experiment scene to a virtual scene coordinate system corresponding to the virtual reality experiment scene; adjusting the collection frequency of the multi-modal data to a preset frequency; and converting the frequency-synchronized data into a preset format to obtain the standardized data.

[0077] The data related to the scene position includes positioning data. In the embodiment, the positioning data is mapped to a unified virtual scene coordinate system to ensure the accuracy of the interaction between the subject and the virtual reality experiment scene.

[0078] In addition, by adjusting the collection frequency to a unified preset frequency, the synchronization of the multi-modal data can be ensured.

[0079] In addition, by converting the frequency-synchronized data into a unified preset format, data compatibility between different types of data can be ensured, thereby ensuring normal operation of the virtual reality experiment.

[0080] Illustratively, the virtual reality experiment scene displayed by the display device 110 is updated based on the standardized data, including: determining, based on the standardized torso pose data, a scene position and a torso orientation of the subject object in the virtual reality experiment scene; determining, based on the standardized head pose data, a line-of-sight direction of the subject object; and controlling the display device 110 to display scene content in the virtual reality experiment scene at the scene position indicated by the torso orientation and the line-of-sight direction.

[0081] In this embodiment, the scene content displayed by the display device 110 is determined by the torso orientation and the line-of-sight direction, so that the scene content displayed by the display device 110 is more matched with the actions of the subject object, and the experimental effect of the virtual reality experiment is improved.

[0082] The data processing device 130 maps the standardized torso pose data to the virtual reality experiment scene based on a preset data mapping rule to obtain a scene position and a torso orientation corresponding to the subject object, and maps the standardized head pose data to the virtual reality experiment scene based on the data mapping rule to obtain a head orientation corresponding to the subject object.

[0083] The data processing device 130 controls the display device 110 to display scene content in the virtual reality experiment scene at the scene position indicated by the torso orientation and the head orientation, including: in the virtual reality experiment scene, determining scene content in a preset range in front of the scene position and the torso orientation as center, and taking the scene content as front view content corresponding to the scene position and the torso orientation; and in the virtual reality experiment scene, determining scene content obtained by rotating the head orientation by an angle as reference to obtain scene content corresponding to the head orientation. At this time, if the torso of the subject object does not rotate, only the head rotates, and when the head orientation is consistent with the torso orientation, the data processing device 130 controls the display device 110 to always display the front view content, and when the head orientation is inconsistent with the torso orientation, the data processing device 130 rotates the viewpoint of the virtual reality experiment scene with the front view content as reference, thereby ensuring that the scene content currently displayed by the display device 110 is consistent with the turning action of the subject object, reducing dizziness, and improving the virtual reality experiment effect.

[0084] Optionally, the data acquisition apparatus 120 further comprises an eye tracker, and correspondingly, the multi-modal data comprises eye movement data acquired by the eye tracker. The eye tracker is integrated in the display apparatus 110 (e.g., a VR headset) and faces the eyes of the subject. In one example, the eye tracker is configured to track the center of the pupil of the subject, and correspondingly, the eye movement data acquired by the eye tracker is two-dimensional eye movement data indicating the horizontal and vertical positions of the pupil in the screen or field of view, i.e., the eye movement data comprises horizontal and vertical coordinates in the screen coordinate system.

[0085] The data processing apparatus 130 is further configured to determine, based on the head pose data and the eye movement data, an eye position of the subject in the virtual reality experiment scene, generate a ray in the virtual reality experiment scene along a line of sight direction of the subject based on the eye position, and determine a three-dimensional eye movement coordinate in the virtual reality environment based on an intersection position of the ray and a scene object in the virtual reality experiment scene.

[0086] In this embodiment, by determining the three-dimensional eye movement coordinate of the subject in the virtual reality environment, the attention distribution of the subject during the virtual reality experiment can be determined, and based on the attention distribution, the experimental result of the subject can be determined.

[0087] Specifically, the data processing apparatus 130 determines, based on the head pose data and the eye movement data, an eye position of the subject in the virtual reality experiment scene, including: after processing the head pose data and the eye movement data according to a standardization rule, converting the standardized eye movement data from a screen coordinate system to a three-dimensional coordinate system corresponding to the head pose data to obtain a three-dimensional coordinate of the pupil; and mapping the three-dimensional coordinate of the pupil to the virtual reality experiment scene based on a preset data mapping rule to obtain the eye position. The conversion relationship between the screen coordinate system and the three-dimensional coordinate system is pre-stored in the data processing apparatus 130.

[0088] Optionally, the eye tracker is further configured to acquire a corneal reflection point of the subject, and the data processing apparatus 130 is configured to determine the line of sight direction based on the pupil-corneal reflection method using the eye movement data and the corneal reflection point; or the eye tracker is further configured to acquire a sclera edge and an iris edge, and the data processing apparatus 130 is configured to determine the line of sight direction based on the sclera-iris edge method using the positions of the sclera edge and the iris edge in the screen. The determination method of the line of sight direction is not limited in this embodiment.

[0089] After determining the eyeball position and the line-of-sight direction, the data processing apparatus 130 can emit a ray in the virtual reality experiment scene from the eyeball position to the line-of-sight direction, and the intersection position of the ray and a scene object in the virtual reality experiment scene is a three-dimensional eye movement coordinate in the virtual reality environment, which is used to indicate the scene object currently gazed at by the subject.

[0090] Optionally, after obtaining the three-dimensional eye movement coordinate, the data processing apparatus 130 determines the experimental reaction result of the subject, including: generating a heat map of the eye movement information of the subject according to the three-dimensional eye movement coordinate to obtain the attention distribution result of the subject. At this time, the experimental reaction result includes the attention distribution result of the subject. For reference Figure 2 The schematic diagram for determining the three-dimensional eye movement coordinate is shown in FIG. 6. According to the schematic diagram, the ray emitted by the pupil of the subject intersects with a virtual ground in the virtual reality experiment scene, and the three-dimensional coordinate of the intersection position in the virtual reality experiment scene is the three-dimensional eye movement coordinate, Figure 2 Figure 2 The heat map indicated by the rectangular frame region in the figure is the region observed by the subject.

[0091] Optionally, in order to further improve the accuracy of analyzing the behavior of the subject, the data acquisition apparatus 120 further includes at least one of the following:

[0092] 1. An electromyography device for acquiring a skin surface muscle electrical signal of the subject. The skin surface muscle electrical signal is used to indicate the muscle state of the subject, such as whether the subject is in a muscle tension state.

[0093] 2. An electroencephalography device for acquiring brain neuron electrical activity data of the scalp surface of the subject. The brain neuron electrical activity data is used to indicate the brain active state of the subject, such as whether the subject is in an active state.

[0094] 3. A motion capture device for acquiring whole-body spatial motion data of the subject. The whole-body spatial motion data is used to indicate the interactive action between the subject and the virtual reality experiment scene, such as moving in the virtual reality experiment scene, avoiding obstacles in the virtual reality experiment scene, etc.

[0095] ​Optionally, the motion capture device comprises an inertial motion capture device and / or an optical motion capture device, and the embodiment is not limited to the type of the motion capture device. The inertial motion capture device refers to a device for measuring the motion of different parts of the subject through a plurality of inertial measurement units, including but not limited to an accelerometer, a gyroscope, a magnetometer, etc. The optical motion capture device refers to a device for capturing the full range of images of the subject through a plurality of cameras.

[0096] The motion capture device captures the body part of the subject Figure 3 According to Figure 3 It can be seen that the motion capture device can capture the full-body motion data of the subject.

[0097] 4. Near-infrared device, for collecting the cerebral oxy-hemoglobin and deoxy-hemoglobin data of the subject. The cerebral oxy-hemoglobin and deoxy-hemoglobin data indicates the blood flow change of the cerebral cortex of the subject to determine the brain activity of the subject. The cerebral oxy-hemoglobin and deoxy-hemoglobin data includes but is not limited to the oxy-hemoglobin (Oxy-Hb) data, deoxy-hemoglobin (deoxy-hb) data, and / or total hemoglobin (total-hb) data of the subject, and the embodiment is not limited to the data content of the cerebral oxy-hemoglobin and deoxy-hemoglobin data.

[0098] The data processing apparatus 130 determines the experimental response result of the subject, including determining the cognitive state of the subject according to the standardized positioning data, the three-dimensional eye movement coordinates, the electromyography data, the electroencephalography data and / or the near-infrared data. The cognitive state includes but is not limited to the memory state, the action execution state, the attention concentration state, etc. The type of the cognitive state can be based on the experimental requirements, and the embodiment is not limited to the type of the cognitive state.

[0099] The data processing apparatus 130 pre-stores a pre-trained cognitive type recognition network, and inputs the standardized multi-modal data collected at the preset time into the cognitive type recognition network to obtain the cognitive state of the subject at the preset time.

[0100] The cognitive type recognition network is established based on a neural network model and is trained using sample multi-modal data and cognitive type labels corresponding to the sample multi-modal data.

[0101] In other embodiments, the data processing apparatus 130 can also use other mathematical models to calculate the cognitive state of the subject, and the embodiment is not limited to the determination method of the cognitive state of the subject.

[0102] In other embodiments, the data acquisition device 120 can also include more types of devices, and accordingly, the multi-modal data can also include other types of data, which are not listed one by one here.

[0103] Optionally, the data processing device 130 is also configured to: store the motion data indicated by the multi-modal data during the interaction of the subject with the virtual reality experiment scene; and in response to receiving an experiment process playback instruction, use the motion data to perform experiment process playback based on a playback technology.

[0104] The motion data indicated by the multi-modal data includes a motion trajectory and a motion direction of the subject.

[0105] In the VR field, the playback technology is used to reproduce the pre-stored motion data in the VR environment.

[0106] Optionally, the data processing device 130 can also store all the multi-modal data to analyze the cognitive state of the subject after the virtual reality experiment ends.

[0107] In summary, the virtual reality experiment system for multi-modal physiological indicator acquisition provided in this embodiment displays a pre-created virtual reality experiment scene through a display device; a data acquisition device acquires multi-modal data generated by a subject based on the virtual reality experiment scene; the data acquisition device includes a positioner, and accordingly, the multi-modal data includes positioning data acquired by the positioner, the positioning data indicating a trunk pose and a head pose of the subject; a data processing device processes the multi-modal data based on a pre-set standardization rule to obtain standardized data; the virtual reality experiment scene displayed by the display device is updated based on the standardized data, and an experiment response result of the subject is determined; the problem that the scene content displayed in the virtual reality scene in the traditional virtual reality system is inconsistent with the action of the subject, thereby causing dizziness and poor experiment results can be solved; in this embodiment, the trunk pose and the head pose are combined to determine the turning operation of the subject in the virtual reality scene, so as to ensure that the scene content displayed in the virtual reality scene is consistent with the action of the subject, reduce the generation of dizziness, and improve the experiment results.

[0108] In addition, by combining the positioning data with multi-modal data such as three-dimensional eye movement coordinates, skin surface muscle electrical signals, brain neuron electrical activity data, whole body spatial motion data, and / or brain oxygenated hemoglobin and deoxygenated hemoglobin data, the experiment response result of the subject can be determined, which can improve the accuracy of determining the experiment response result.

[0109] Figure 4 A flowchart of a multi-modal physiological indicator acquisition virtual reality experiment method according to an embodiment of the present disclosure is shown. This embodiment uses the method for Figure 1The data processing apparatus 130 shown is taken as an example for description, and in actual implementation, the method can also be applied to other electronic devices having processing capability and being connected in communication with the data processing apparatus 130. As shown in the figure, the method comprises the following steps. Figure 4

[0110] Step 401: controlling a display apparatus to display a pre-created virtual reality experiment scene.

[0111] Step 402: acquiring multi-modal data of a subject object collected by a data collection apparatus; the multi-modal data comprises positioning data collected by a positioner, the positioning data indicating a torso pose and a head pose of the subject object.

[0112] Illustratively, the multi-modal data further comprises at least one of the following: a skin surface muscle electrical signal of the subject object, a brain neuron electrical activity data of a brain scalp surface, whole-body spatial motion data, and brain oxygenated hemoglobin and deoxygenated hemoglobin data, and specific details are shown in the above embodiments, which are not repeated here.

[0113] Step 403: processing the multi-modal data based on a pre-set standardization rule to obtain standardized data.

[0114] Step 404: updating the virtual reality experiment scene displayed by the display apparatus based on the standardized data, and determining an experiment reaction result of the subject object.

[0115] The related description of the present embodiment is shown in the above system embodiments, which are not repeated here.

[0116] In summary, the virtual reality experiment method for multi-modal physiological index collection provided in the present embodiment updates the virtual reality experiment scene displayed by the display apparatus by collecting the torso pose and the head pose of the subject object; can solve the problem that the scene content of the virtual reality scene display is inconsistent with the action of the subject object in the traditional virtual reality system, thereby generating a dizziness, and the experiment effect is poor; and the present embodiment can determine the turning operation of the subject object in the virtual reality scene in combination with the torso pose and the head pose, ensure that the scene content of the virtual reality scene display is consistent with the action of the subject object, reduce the generation of dizziness, and improve the experiment effect.

[0117] Figure 5 A block diagram of a virtual reality experiment device for multi-modal physiological index collection according to an embodiment of the present disclosure is shown. The device at least comprises: a scene display module 510, a data collection module 520, a data standardization module 530, and a result determination module 540.

[0118] The scene display module 510 is configured to control a display apparatus to display a pre-created virtual reality experiment scene.

[0119] ​The data acquisition module 520 is configured to acquire multi-modal data of a subject collected by the data acquisition device. The multi-modal data includes positioning data collected by the positioner, and the positioning data indicates a trunk pose and a head pose of the subject.

[0120] The data standardization module 530 is configured to process the multi-modal data based on a preset standardization rule to obtain standardized data.

[0121] The result determination module 540 is configured to update a virtual reality experiment scene displayed by the display device based on the standardized data, and determine an experiment reaction result of the subject.

[0122] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be repeated here.

[0123] The embodiments of the present disclosure also propose a computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0124] The embodiments of the present disclosure also propose an electronic device, including a processor, a memory for storing processor-executable instructions, wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0125] The embodiments of the present disclosure also provide a computer program product, including computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of the electronic device, the processor in the electronic device executes the above method.

[0126] Figure 6 is a block diagram of a multi-modal physiological indicator acquisition virtual reality experiment device 1900 according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Referring to Figure 6 , the device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0127] The apparatus 1900 can also include a power supply component 1926 configured to perform power management for the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input output interface 1958 (I / O interface). The apparatus 1900 can operate in conjunction with a operating system, for example, Windows Server TM or the like, that supports running a three-dimensional engine.

[0128] In an example embodiment, there is also provided a non-transitory computer- readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the apparatus 1900 to perform the above method.

[0129] The above has described various embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical application, or technical improvement in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A virtual reality experimental system for acquiring multimodal physiological indicators, characterized in that, The system includes: a display device, a data acquisition device, and a data processing device that is communicatively connected to the display device and the data acquisition device respectively; The display device is used to display a pre-created virtual reality experimental scene; The data acquisition device is used to collect multimodal data generated by the test subject based on the virtual reality experimental scenario; the data acquisition device includes a positioning device, which includes a first sensing component disposed on the test subject's torso, a second sensing component disposed on the test subject's head, and a signal transmitting component disposed in the real space where the test subject is located; correspondingly, the multimodal data includes positioning data collected by the positioning device, which indicates the test subject's torso pose and head pose; The data processing device is used to map data related to the scene location of the virtual reality experimental scene to the virtual scene coordinate system corresponding to the virtual reality experimental scene; adjust the acquisition frequency of the multimodal data to a preset frequency; convert the frequency-synchronized data into a preset format to obtain standardized data; update the virtual reality experimental scene displayed on the display device based on the standardized data, and determine the experimental response results of the subject; store the motion data indicated by the multimodal data during the interaction between the subject and the virtual reality experimental scene; and, upon receiving a playback command for the experimental process, use the motion data to replay the experimental process based on playback technology. The process of updating the virtual reality experimental scene displayed on the display device based on the standardized data includes: Based on the standardized trunk pose data, the subject's scene position and trunk orientation in the virtual reality experimental scenario are determined. The head orientation of the subject was determined based on the standardized head position data. The display device is controlled to display the scene content in the virtual reality experimental scene, indicating the torso orientation and head orientation at the scene location; The positioning device collects the positioning data, including: The first sensing component senses a preset signal emitted by the signal transmitting component; based on the sensed preset signal, the first relative position of the signal transmitting component relative to the first sensing component is determined; based on the first relative position and the setting position of the signal transmitting component, the position and orientation of the subject's torso are determined, and torso pose data is obtained. The second sensing component senses preset signals emitted by different signal transmitting components; based on the sensed preset signals, the second relative position of the signal transmitting component relative to the second sensing component is determined; based on the second relative position and the setting position of the signal transmitting component, the position and orientation of the subject's head are determined, and head pose data is obtained. The data acquisition device further includes an eye tracker, and correspondingly, the multimodal data includes two-dimensional eye movement data acquired by the eye tracker; the data processing device is further used for: Based on the head pose data and the eye movement data, the eye positions of the subject in the virtual reality experimental scene are determined. Based on the eye position, a ray is generated along the subject's line of sight towards the virtual reality experimental scene; Based on the intersection position of the ray with the scene object in the virtual reality experimental scene, the three-dimensional eye-tracking coordinates in the virtual reality environment are determined.

2. The system according to claim 1, characterized in that, The data acquisition device further includes at least one of the following: An electromyography (EMG) device is used to collect electrical signals from the muscles on the skin surface of the subject. EEG device, used to collect data on the electrical activity of neurons in the brain on the surface of the scalp of the subject; Motion capture equipment is used to collect the whole-body spatial motion data of the subject; Near-infrared equipment was used to collect oxygenated and deoxygenated hemoglobin from the brain of the subject.

3. A virtual reality experimental method for acquiring multimodal physiological indicators, characterized in that, The method is used in the data processing device of the virtual reality experimental system according to claim 1 or 2, and the method includes: The display device is controlled to display a pre-created virtual reality experimental scene; The multimodal data of the subject collected by the data acquisition device is acquired; the multimodal data includes the positioning data collected by the positioning device, and the positioning data indicates the trunk pose and head pose of the subject. Data related to the scene location of the virtual reality experimental scene is mapped to the virtual scene coordinate system corresponding to the virtual reality experimental scene; the acquisition frequency of the multimodal data is adjusted to a preset frequency; the frequency-synchronized data is converted into a preset format to obtain standardized data; The virtual reality experimental scene displayed on the display device is updated based on the standardized data, and the experimental response results of the test subject are determined. Store the motion data indicated by the multimodal data during the interaction between the subject and the virtual reality experimental scene; Upon receiving a command to replay the experimental process, the motion data is used to replay the experimental process based on playback technology. The process of updating the virtual reality experimental scene displayed on the display device based on the standardized data includes: Based on standardized trunk pose data, the subject's scene position and trunk orientation in the virtual reality experimental scene are determined; based on standardized head pose data, the subject's head orientation is determined; the display device is controlled to display scene content in the virtual reality experimental scene, indicated by the trunk orientation and head orientation at the scene position; The positioning device includes a first sensing component disposed on the torso of the subject, a second sensing component disposed on the head of the subject, and a signal transmitting component disposed in the real space where the subject is located; the positioning device collects the positioning data, including: The first sensing component senses a preset signal emitted by the signal transmitting component; based on the sensed preset signal, the first relative position of the signal transmitting component relative to the first sensing component is determined; based on the first relative position and the setting position of the signal transmitting component, the position and orientation of the subject's torso are determined, and torso pose data is obtained. The second sensing component senses preset signals emitted by different signal transmitting components; based on the sensed preset signals, the second relative position of the signal transmitting component relative to the second sensing component is determined; based on the second relative position and the setting position of the signal transmitting component, the position and orientation of the subject's head are determined, and head pose data is obtained. The data acquisition device further includes an eye tracker, and correspondingly, the multimodal data includes two-dimensional eye movement data acquired by the eye tracker; the data processing device is further used for: Based on the head pose data and the eye movement data, the eye positions of the subject in the virtual reality experimental scene are determined. Based on the eye position, a ray is generated along the subject's line of sight towards the virtual reality experimental scene; Based on the intersection position of the ray with the scene object in the virtual reality experimental scene, the three-dimensional eye-tracking coordinates in the virtual reality environment are determined.

4. A virtual reality experimental device for acquiring multimodal physiological indicators, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of claim 3 when executing instructions stored in the memory.

5. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method of claim 3.

Images