A multimodal physiological data evaluation system based on virtual reality technology

Through the multimodal physiological data evaluation system of virtual reality technology, the problem that traditional paper evaluation and measurement tables cannot be feedback and adjusted in real time is solved, efficient and accurate physiological data collection and analysis are achieved, and real-time emotional and psychological evaluation is provided.

CN118486455BActive Publication Date: 2024-11-01HANGZHOU XUZHISHI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410947979.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-01
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

Traditional paper physiological evaluation and measurement forms cannot provide real-time feedback on user filling, it is difficult to make immediate error correction or adjustment, and it is impossible to provide real-time emotional or psychological state assessment.

Method used

Using a multimodal physiological data evaluation system based on virtual reality technology, including a virtual reality input module, a physiological data acquisition module, a physiological data analysis module and a physiological data interface interaction module, the environment is collected and analyzed by immersive physiological state evaluation. Data, real-time feedback on evaluation results and make adjustments.

Benefits of technology

It improves the efficiency and accuracy of data collection and processing, provides real-time emotional and psychological state assessment, reduces the workload of manual data input and processing, and enhances the reliability of user experience and evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118486455B_ABST
    Figure CN118486455B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimodal physiological data evaluation system based on virtual reality technology. The system includes: a virtual reality input module, a physiological data acquisition module, a physiological data analysis module and a physiological data interface interaction module connected in communication, wherein the virtual reality input module is used to create an immersive physiological state evaluation environment and capture the input information of the person to be tested; the physiological data acquisition module is used to collect the multimodal physiological data information of the person to be tested; the physiological data analysis module is used to select a preset data analysis model according to the multimodal physiological data information to perform data comprehensive analysis; the physiological data interface interaction module is used to provide real-time feedback on the evaluation results of the person to be tested according to a preset interaction method, and to make real-time feedback adjustments according to the evaluation results. It automatically records and analyzes the physiological data of the user during the evaluation process by integrating multimodal physiological data acquisition technology, thereby improving the efficiency and accuracy of data acquisition and processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of physiological evaluation, and in particular to a multimodal physiological data evaluation system based on virtual reality technology. Background Art

[0002] In the prior art, traditional paper-based physiological assessment scales are a widely used tool in psychological assessment and emotional research. Such scales are usually presented in printed form, and users answer questions by handwriting. Although paper-based scales have their own advantages, such as simple use, easy storage, and no device dependence. However, such scales are usually presented in printed form, and users answer questions by handwriting. There is no real-time feedback on the user's filling status, and it is difficult to make immediate error corrections or adjustments. It is also impossible to provide real-time emotional or psychological state assessment, which has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the present invention is to provide a multimodal physiological data evaluation system based on virtual reality technology to address the deficiencies in the prior art. It automatically records and analyzes the user's physiological data during the evaluation process by integrating multimodal physiological data acquisition technology, thereby reducing the workload of manual data input and processing and improving the efficiency and accuracy of data acquisition and processing.

[0004] An embodiment of the present application provides a multimodal physiological data evaluation system based on virtual reality technology, the system comprising:

[0005] A virtual reality input module, a physiological data acquisition module, a physiological data analysis module and a physiological data interface interaction module connected in communication, wherein:

[0006] The virtual reality input module is used to create an immersive physiological status assessment environment and capture input information of the person to be tested;

[0007] The physiological data acquisition module is used to collect multimodal physiological data information of the person to be tested;

[0008] The physiological data analysis module is used to select a preset data analysis model to perform comprehensive data analysis according to the multimodal physiological data information;

[0009] The physiological data interface interaction module is used to provide real-time feedback of the evaluation results of the person to be tested according to a preset interaction method, and to make real-time feedback adjustments according to the evaluation results.

[0010] Optionally, the virtual reality input module includes:

[0011] VR head display unit, input control unit and motion sensor unit, wherein:

[0012] The VR head display unit is used to provide an immersive physiological state assessment environment;

[0013] The input control unit is used to obtain the motion signal of the person to be tested;

[0014] The motion sensing unit is used to convert the motion signal of the person to be tested into an input electrical signal for physiological status assessment.

[0015] Optionally, the multimodal physiological data information includes:

[0016] The test person’s eye movement data, heart rate monitoring data, skin electrical response data, and brain wave data.

[0017] Optionally, the physiological data analysis module includes:

[0018] Multimodal data fusion unit and data analysis model unit, wherein:

[0019] The multimodal data fusion unit is used to perform feature fusion of multimodal physiological data information according to a multimodal data fusion algorithm; wherein the multimodal data fusion algorithm is expressed as: in, Represents the multimodal physiological data after feature fusion, represents a non-empty finite set of multimodal physiological data, represents a non-empty finite set of conditional attributes of multimodal physiological data, Represents modal data The weight set satisfies represents the feature fusion function;

[0020] The data analysis model unit is used to process the multimodal physiological data after feature fusion according to a preset data analysis model.

[0021] Optionally, the preset data analysis model includes:

[0022] Data preprocessing layer, feature extraction layer, and model evaluation and optimization layer, among which,

[0023] The data preprocessing layer is used to perform preprocessing of multimodal physiological data information; wherein the number of neurons in the data preprocessing layer is determined by the following formula: in, represents the number of neurons in the data preprocessing layer, Indicates the number of multimodal physiological data information, is a constant, and its value range is [1,10];

[0024] The feature extraction layer is used to perform feature extraction on the preprocessed multimodal physiological data information; wherein the output of the feature extraction layer is expressed as: in, Represents the multimodal physiological data information after feature extraction, represents the feature extraction layer activation function, is the weight, Input variables representing multimodal physiological data information, Indicates the threshold value;

[0025] The model evaluation and optimization layer is used to obtain the current evaluation results of the person to be tested according to the multimodal physiological data information, and to optimize the model based on the current evaluation results.

[0026] Optionally, the preset interaction mode includes:

[0027] At least one of a gesture recognition method of the person to be tested, a head movement interaction method of the person to be tested, and an external control device interaction method.

[0028] Optionally, the system further includes:

[0029] A physiological data storage and management module, wherein the physiological data storage and management module is used to use data encryption and anonymity processing technology to perform storage of multimodal physiological data information of the person to be tested.

[0030] Another embodiment of the present application provides a multimodal physiological data evaluation method based on virtual reality technology, the method comprising:

[0031] The multimodal physiological data evaluation system is used to collect the multimodal physiological data of the person to be tested in real time, and the evaluation results of the person to be tested are adjusted based on feedback to meet the physiological evaluation needs of the person to be tested.

[0032] Yet another embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, wherein the computer program is configured to implement the above-mentioned method when executed.

[0033] Another embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the above-mentioned method.

[0034] Compared with the prior art, the present invention proposes a multimodal physiological data evaluation system based on virtual reality technology, including a virtual reality input module, a physiological data acquisition module, a physiological data analysis module and a physiological data interface interaction module connected in communication, wherein the virtual reality input module is used to create an immersive physiological state evaluation environment and capture the input information of the person to be tested; the physiological data acquisition module is used to collect the multimodal physiological data information of the person to be tested; the physiological data analysis module is used to select a preset data analysis model according to the multimodal physiological data information to perform data comprehensive analysis; the physiological data interface interaction module is used to provide real-time feedback on the evaluation results of the person to be tested according to a preset interaction method, and to make real-time feedback adjustments according to the evaluation results. It automatically records and analyzes the physiological data of the user during the evaluation process by integrating multimodal physiological data acquisition technology, reduces the workload of manual data input and processing, and improves the efficiency and accuracy of data acquisition and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the structure of a multimodal physiological data evaluation system based on virtual reality technology provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the structure of another multimodal physiological data evaluation system based on virtual reality technology provided by an embodiment of the present invention;

[0037] Figure 3 A hardware structure block diagram of a computer terminal for a multimodal physiological data evaluation method based on virtual reality technology provided by an embodiment of the present invention;

[0038] Figure 4 A flowchart of a multimodal physiological data evaluation method based on virtual reality technology provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, but should not be construed as limiting the present invention.

[0040] See also Figure 1 , Figure 1 A schematic diagram of the structure of a multimodal physiological data evaluation system based on virtual reality technology provided by an embodiment of the present invention, Figure 1A multimodal physiological data evaluation system 200 based on virtual reality technology is shown, and the system includes: a virtual reality input module 201, a physiological data acquisition module 202, a physiological data analysis module 203 and a physiological data interface interaction module 204 that are communicatively connected, wherein the virtual reality input module 201 is used to create an immersive physiological state evaluation environment and capture input information of a person to be tested; the physiological data acquisition module 202 is used to collect multimodal physiological data information of the person to be tested; the physiological data analysis module 203 is used to select a preset data analysis model according to the multimodal physiological data information to perform comprehensive data analysis; the physiological data interface interaction module 204 is used to provide real-time feedback on the evaluation results of the person to be tested according to a preset interaction method, and to make real-time feedback adjustments based on the evaluation results.

[0041] The multimodal physiological data information may include: eye movement data, heart rate monitoring data, skin electrical response data and brain wave data of the person to be tested. The preset interaction mode may include: at least one of: gesture recognition mode of the person to be tested, head movement interaction mode of the person to be tested and interaction mode of an external control device.

[0042] Specifically, the virtual reality input module 201 is used to create an immersive physiological state assessment environment. It uses virtual reality technology to build a highly realistic virtual environment for users, simulates various physiological state assessment scenarios, and captures user behavior, movements, voice and other input information in the virtual environment through VR devices (such as VR helmets, handles, etc.).

[0043] It should be noted that the virtual reality input module 201 can support a variety of VR devices and platforms, allowing users to customize the virtual environment to meet different evaluation needs. Provide real-time feedback such as sound and touch to enhance user immersion. The physiological data acquisition module 202 can collect multimodal physiological data information of the person to be tested: including but not limited to electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), galvanic skin response (GSR), respiratory rate, heart rate, etc. The physiological data analysis module 203 selects a preset data analysis model according to the multimodal physiological data information to perform data comprehensive analysis, such as using machine learning, deep learning and other algorithms to process and analyze the collected physiological data to evaluate the user's physiological state. The physiological data interface interaction module 204 provides real-time feedback on the evaluation results of the person to be tested according to the preset interaction method, such as displaying the evaluation results to the user through graphics, text, sound, etc. Provide personalized suggestions, guidance or intervention measures for users based on the evaluation results.

[0044] In an optional embodiment, the virtual reality input module 201 may include: a VR head display unit, an input control unit and a motion sensing unit, wherein the VR head display unit is used to provide an immersive physiological state assessment environment; the input control unit is used to obtain the motion signal of the person to be tested; and the motion sensing unit is used to convert the motion signal of the person to be tested into an input electrical signal for physiological state assessment.

[0045] Specifically, the virtual reality input module 201 is intended to provide an immersive physiological state assessment environment for the person to be tested through virtual reality (VR) technology. Among them, the VR head display unit is the core part of the virtual reality system, which is responsible for providing an immersive visual experience for the wearer. Through the head-mounted display (HMD), the user can see a virtual world generated by a computer, as if they were there. The VR head display usually uses two high-resolution display screens to display images for the left and right eyes respectively to create stereoscopic vision. In addition, it may also include a position tracking system (such as a gyroscope, accelerometer, etc.) to detect the user's head movement and update the perspective in the virtual world in real time. The VR head display unit can also create a virtual environment similar to the real world, which is used to simulate various scenes (such as skydiving, deep-sea diving, etc.) to observe the physiological reactions of the person to be tested in various situations.

[0046] The input control unit is used to receive motion signals from the person under test, which can come from various input devices, such as handles, gloves, pedals, etc. The input control unit may include sensors, microcontrollers, and wireless communication modules. Sensors are used to detect the user's movements (such as gestures, footsteps, etc.), and the microcontroller is responsible for processing these signals and converting them into digital format, which are then sent to the computer through wireless communication modules (such as Bluetooth, Wi-Fi, etc.). The input control unit can capture the interactive behaviors of the person under test in the virtual environment, such as moving, operating virtual objects, etc. These behaviors can be used to evaluate their physiological states in terms of cognition, reaction speed, coordination, etc.

[0047] The motion sensing unit is responsible for converting the motion signals of the person under test into the input electrical signals required for physiological status assessment. The motion sensing unit usually contains sensors such as accelerometers, gyroscopes and magnetometers, which can detect information such as the user's motion acceleration, direction changes and geomagnetic field. After these data are converted into electrical signals, they can be further processed and analyzed to evaluate the user's physiological state. The motion sensing unit can monitor the body movements of the person under test in the virtual environment, such as step frequency, posture changes, etc. These data can be used to evaluate their physical strength, balance ability, movement coordination and other physiological states.

[0048] It should be noted that the multimodal physiological data information may include: eye movement data, heart rate monitoring data, skin electrical response data and brain wave data of the person to be tested.

[0049] In an optional implementation, the physiological data analysis module 203 may include: a multimodal data fusion unit and a data analysis model unit, wherein the multimodal data fusion unit is used to perform feature fusion of multimodal physiological data information according to a multimodal data fusion algorithm; wherein the multimodal data fusion algorithm is expressed as: in, Represents the multimodal physiological data after feature fusion, represents a non-empty finite set of multimodal physiological data, represents a non-empty finite set of conditional attributes of multimodal physiological data, Represents modal data The weight set satisfies represents the feature fusion function;

[0050] The data analysis model unit is used to process the multimodal physiological data after feature fusion according to a preset data analysis model.

[0051] The preset data analysis model includes: a data preprocessing layer, a feature extraction layer, and a model evaluation and optimization layer, wherein the data preprocessing layer is used to perform preprocessing of multimodal physiological data information; wherein the number of neurons in the data preprocessing layer is determined by the following formula: in, represents the number of neurons in the data preprocessing layer, Indicates the number of multimodal physiological data information, is a constant, and its value range is [1,10];

[0052] The feature extraction layer is used to perform feature extraction on the preprocessed multimodal physiological data information; wherein the output of the feature extraction layer is expressed as: in, Represents the multimodal physiological data information after feature extraction, represents the feature extraction layer activation function, is the weight, Input variables representing multimodal physiological data information, Indicates the threshold value;

[0053] The model evaluation and optimization layer is used to obtain the current evaluation results of the person to be tested according to the multimodal physiological data information, and to optimize the model based on the current evaluation results.

[0054] See also Figure 2 , Figure 2Another multimodal physiological data evaluation system 200 based on virtual reality technology is shown, and the system may also include: a physiological data storage and management module 205, wherein the physiological data storage and management module 205 is used to use data encryption and anonymity processing technology to perform storage of multimodal physiological data information of the person to be tested.

[0055] Specifically, the physiological data storage and management module 205 is responsible for receiving, storing, managing and protecting the multimodal physiological data information of the person to be tested. Such information may include electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), respiratory rate, galvanic skin response (GSR) and other physiological signal data.

[0056] To ensure data security, the module uses data encryption technology, which is a technology that protects data from unauthorized access or tampering during storage and transmission. Common encryption technologies include symmetric encryption (such as AES, DES) and asymmetric encryption (such as RSA). The module will select a suitable encryption algorithm to encrypt multimodal physiological data information to ensure that only authorized users can access and decrypt the data.

[0057] In addition to data encryption, this module can also use anonymization technology to protect the privacy of the person being tested. Anonymization is a technology that reduces or eliminates the association with personal identity-related information by removing or replacing identifiers. In this module, all information related to the identity of the person being tested (such as name, ID number, contact information, etc.) will be removed or replaced with anonymous identifiers. In this way, even if the data is leaked, the attacker cannot directly identify the specific individual identity.

[0058] The encrypted and anonymized physiological data will be stored in a secure and reliable database. The database should have the characteristics of high availability, scalability and fault tolerance to ensure the integrity and accessibility of the data. At the same time, the module will also provide data backup and recovery mechanisms to prevent data loss or damage. In terms of management, the module will provide data access control functions, allowing only authorized users to access and operate data after identity authentication and permission verification. In addition, the module will also record all data access and operation logs for auditing and tracking.

[0059] The physiological data storage and management module 205 is an important tool to ensure the security and privacy of multimodal physiological data information. By adopting data encryption and anonymization processing technology, the module can protect the privacy of the person being tested from being leaked and ensure the security of data during storage and transmission. At the same time, the module also provides reliable data storage and management functions, providing strong support for physiological status assessment.

[0060] Exemplarily, the basic process of the multimodal physiological data evaluation system based on virtual reality technology may include the following steps:

[0061] 1. System initialization: The user wears a VR headset and a handheld controller and enters the virtual reality environment through the virtual reality input module 201. The physiological data acquisition module 202 starts working, for example, the heart rate monitor, the galvanic skin response sensor and the brain wave sensor start to collect the user's physiological data.

[0062] 2. Evaluation in a virtual environment: In a virtual environment, users start filling out the psychological evaluation scale according to the system prompts. During the evaluation process, users can interact with the system through gesture recognition, handle or head movement to complete the questionnaire.

[0063] 3. Data collection and real-time analysis: The physiological data collection module 202 records the user's physiological data at each moment during the evaluation process in real time. The physiological data analysis module 203 can conduct a comprehensive analysis of the physiological data collected in real time and the user's self-reported data to evaluate the user's psychological response under different emotional states.

[0064] 4. Feedback and adjustment: The physiological data interface interaction module 204 provides real-time feedback on the user's filling status and preliminary evaluation results. The user can adjust his or her answer based on the feedback to ensure the accuracy and reliability of the evaluation results.

[0065] 5. Data storage and report generation: After the assessment is completed, all data are securely stored through the physiological data storage and management module 205. The system generates a comprehensive psychological assessment report, including the user's self-reported data and the analysis results of physiological data, to provide comprehensive support for psychological assessment.

[0066] 6. Privacy protection: During the entire working process, the system ensures the encryption and anonymization of user data to protect user privacy and security.

[0067] The user wears a virtual reality device to enter an immersive virtual environment, and the system captures the user's movements through a handheld controller and motion sensors. In the virtual environment, the user fills in the psychological assessment scale according to the system prompts, and the system provides a variety of interaction methods (such as gesture recognition, head movement interaction, and handle interaction) for the user to choose. During the assessment process, the physiological data acquisition module monitors the user's eye movements, heart rate, skin electrical response, brain waves and other physiological indicators in real time, and records the user's physiological reactions in different emotional states. The data processing and analysis module conducts a comprehensive analysis of the collected multimodal data, and combines the user's self-report data to provide comprehensive psychological assessment results. The system provides real-time feedback on the assessment results through the interactive interface module, and the user can adjust the answer based on the feedback to ensure the accuracy and reliability of the assessment.

[0068] It should be noted that the construction of the virtual reality environment uses virtual reality technology to create a highly immersive assessment scene to ensure that users are assessed in a private, undisturbed environment. Multimodal data collection integrates devices such as eye movement, heart rate monitors, galvanic skin response sensors, and brain wave sensors to achieve real-time collection of user physiological data. Advanced multimodal data fusion algorithms are used to comprehensively process the collected physiological data and self-reported data to generate a comprehensive psychological assessment report. By developing a variety of interactive methods, including gesture recognition, head movement interaction, and handle interaction, the user experience and participation are improved. Data encryption and anonymization processing technology are used to ensure the security and privacy protection of user data during transmission and storage.

[0069] For example, the real-time collection of eye movement data of the person to be tested can be achieved by wearing a virtual reality device, which may involve the use of eye tracking technology. Among them, the virtual reality device integrates an eye tracking function, such as a built-in eye tracking device, such as a built-in infrared camera and light source. The infrared light source is used to illuminate the eyeball, while the camera is responsible for capturing the movement of the eyeball, especially the position of the cornea and pupil of the person to be tested. Through the principle of infrared light reflection, the camera can detect the relative position changes of the cornea and pupil. The captured raw data includes video images of eye movement, which need to be further processed to determine the gaze point of the person to be tested. The raw data is processed by a series of algorithms, including image preprocessing, corneal reflection point and pupil center point detection, spatial calibration, gaze point calculation and other steps to accurately estimate the gaze point of the person to be tested and convert it into eye movement data.

[0070] It can be seen that this application has realized an efficient, comprehensive and interactive psychological assessment system through virtual reality technology and multimodal physiological data collection, significantly improved the efficiency of data collection and processing, provided real-time feedback and adjustment, enhanced emotion induction effects and privacy protection, and is highly adaptable, convenient and flexible, providing a more accurate and user-friendly solution for psychological assessment.

[0071] The embodiment of the present invention also provides a multimodal physiological data evaluation method based on virtual reality technology, which can be applied to electronic devices such as computer terminals, specifically ordinary computers, quantum computers, etc.

[0072] The following describes it in detail by taking running on a computer terminal as an example. Figure 3 The hardware structure block diagram of a computer terminal of a multimodal physiological data evaluation method based on virtual reality technology provided by an embodiment of the present invention. Figure 3 As shown, the computer terminal may include one or more ( Figure 3Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the computer terminal may also include a transmission device 106 for communication functions and an input and output device 108. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations are shown.

[0073] The memory 104 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to a multimodal physiological data evaluation method based on virtual reality technology in an embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned system. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to a computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0074] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0075] See also Figure 4 , Figure 4 A flowchart of a multimodal physiological data evaluation method based on virtual reality technology provided by an embodiment of the present invention, the method may include:

[0076] S401: Utilize the multimodal physiological data evaluation system to collect multimodal physiological data of the person to be tested in real time, and provide feedback to adjust the evaluation results of the person to be tested to meet the physiological evaluation needs of the person to be tested.

[0077] Specifically, the multimodal physiological data evaluation system is used to collect the multimodal physiological data of the person to be tested in real time, and the evaluation results are adjusted based on these data feedback to meet their physiological evaluation needs. This will help people better understand their own physiological state and make adjustments or improvements as needed to ensure the accuracy and reliability of the evaluation.

[0078] Compared with the prior art, the present invention proposes a multimodal physiological data evaluation system based on virtual reality technology, including a virtual reality input module, a physiological data acquisition module, a physiological data analysis module and a physiological data interface interaction module connected in communication, wherein the virtual reality input module is used to create an immersive physiological state evaluation environment and capture the input information of the person to be tested; the physiological data acquisition module is used to collect the multimodal physiological data information of the person to be tested; the physiological data analysis module is used to select a preset data analysis model according to the multimodal physiological data information to perform data comprehensive analysis; the physiological data interface interaction module is used to provide real-time feedback on the evaluation results of the person to be tested according to a preset interaction method, and to make real-time feedback adjustments according to the evaluation results. It automatically records and analyzes the physiological data of the user during the evaluation process by integrating multimodal physiological data acquisition technology, reduces the workload of manual data input and processing, and improves the efficiency and accuracy of data acquisition and processing.

[0079] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to implement the steps in the above method embodiment when running.

[0080] Specifically, in this embodiment, the above storage medium may be configured to store a computer program for performing the following steps:

[0081] S401: Utilize the multimodal physiological data evaluation system to collect multimodal physiological data of the person to be tested in real time, and provide feedback to adjust the evaluation results of the person to be tested to meet the physiological evaluation needs of the person to be tested.

[0082] Specifically, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0083] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in the above method embodiment.

[0084] Specifically, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0085] Specifically, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0086] S401: Utilize the multimodal physiological data evaluation system to collect multimodal physiological data of the person to be tested in real time, and provide feedback to adjust the evaluation results of the person to be tested to meet the physiological evaluation needs of the person to be tested.

[0087] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0088] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] In the several embodiments provided by the present invention, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0090] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0091] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0092] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0093] The embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multimodal physiological data evaluation system based on virtual reality technology, characterized in that: The system comprises: A virtual reality input module, a physiological data acquisition module, a physiological data analysis module and a physiological data interface interaction module connected in communication, wherein: The virtual reality input module is used to create an immersive physiological status assessment environment and capture input information of the person to be tested; The physiological data acquisition module is used to collect multimodal physiological data information of the person to be tested; The physiological data analysis module is used to select a preset data analysis model to perform comprehensive data analysis according to the multimodal physiological data information; The physiological data interface interaction module is used to provide real-time feedback of the evaluation results of the person to be tested according to a preset interaction method, and to make real-time feedback adjustments based on the evaluation results; The multimodal physiological data information includes: eye movement data, heart rate monitoring data, skin electrical response data and brain wave data of the person to be tested; The physiological data analysis module comprises: Multimodal data fusion unit and data analysis model unit, wherein: The multimodal data fusion unit is used to perform feature fusion of multimodal physiological data information according to a multimodal data fusion algorithm; wherein the multimodal data fusion algorithm is expressed as: in, Represents the multimodal physiological data after feature fusion, represents a non-empty finite set of multimodal physiological data, , represents an element in a non-empty finite set of multimodal physiological data, Indicates the element number, represents a non-empty finite set of conditional attributes of multimodal physiological data, represents the weight set of multimodal physiological data, represents the set of all weights of modal data, Represents modal data The weight set satisfies , Represents modal data The sequence number of the weight set, , A feature set representing multimodal physiological data, represents the feature fusion function; The data analysis model unit is used to process the multimodal physiological data after feature fusion according to a preset data analysis model.

2. The multimodal physiological data evaluation system according to claim 1, characterized in that: The virtual reality input module comprises: VR head display unit, input control unit and motion sensor unit, wherein: The VR head display unit is used to provide an immersive physiological state assessment environment; The input control unit is used to obtain the motion signal of the person to be tested; The motion sensing unit is used to convert the motion signal of the person to be tested into an input electrical signal for physiological status assessment.

3. The multimodal physiological data evaluation system according to claim 1, characterized in that: The preset data analysis model includes: Data preprocessing layer, feature extraction layer, and model evaluation and optimization layer, among which, The data preprocessing layer is used to perform preprocessing of multimodal physiological data information; wherein the number of neurons in the data preprocessing layer is determined by the following formula: in, represents the number of neurons in the data preprocessing layer, Indicates the number of multimodal physiological data information, is a constant, and its value range is [1,10]; The feature extraction layer is used to perform feature extraction on the preprocessed multimodal physiological data information; wherein the output of the feature extraction layer is expressed as: in, Represents the multimodal physiological data information after feature extraction, represents the feature extraction layer activation function, is the weight, Input variables representing multimodal physiological data information, Indicates the threshold value; The model evaluation and optimization layer is used to obtain the current evaluation results of the person to be tested according to the multimodal physiological data information, and to optimize the model based on the current evaluation results.

4. The multimodal physiological data evaluation system according to claim 3, characterized in that: The preset interaction mode includes: At least one of a gesture recognition method of the person to be tested, a head movement interaction method of the person to be tested, and an external control device interaction method.

5. The multimodal physiological data evaluation system according to any one of claims 1 to 4, characterized in that: The system further comprises: A physiological data storage and management module, wherein the physiological data storage and management module is used to use data encryption and anonymity processing technology to perform storage of multimodal physiological data information of the person to be tested.

6. A multimodal physiological data evaluation method based on virtual reality technology, characterized in that: The method comprises: The multimodal physiological data evaluation system as described in any one of claims 1 to 4 is used to collect multimodal physiological data of the person to be tested in real time, and the evaluation result of the person to be tested is fed back to adjust so as to meet the physiological evaluation needs of the person to be tested.

7. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to implement the method described in claim 6 when executed.

8. An electronic device, comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the computer program to implement the method described in claim 6.