Virtual reality cognitive training method and system based on multi-sensory information collaboration

Abstract

The invention discloses a virtual reality cognitive training method and system based on multi-sensory information collaboration. The virtual reality cognitive training method based on multi-sensory information collaboration comprises the steps of determining a virtual reality environment matched with a target cognitive training task based on the target cognitive training task; scene sensory stimulation information corresponding to the virtual reality environment is determined, and the scene sensory stimulation information comprises at least one of scene visual information and scene auditory information; auxiliary sensory stimulation information corresponding to the target cognitive training task is determined, and the auxiliary sensory stimulation information comprises at least one of target frequency light signal information and target mode sound signal information; and the scene sensory stimulation information and the auxiliary sensory stimulation information are coordinated and matched, and are output in combination with a virtual reality environment to obtain a virtual reality cognitive training mode, so that a trainer can perform cognitive training based on the virtual reality cognitive training mode.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a virtual reality cognitive training method and system based on multi-sensory information collaboration. Background Art

[0002] With the development of science and technology, virtual reality (VR) technology has gradually been applied to the field of cognitive function training. However, the current virtual reality training system mainly relies on a single stimulation method, which cannot fully stimulate the cognitive potential of the brain, and the stimulation effect is poor, resulting in poor cognitive training effect. Summary of the invention

[0003] The embodiments of the present application aim to solve at least one of the technical problems in the related art to a certain extent. To this end, the purpose of the embodiments of the present application is to propose a virtual reality cognitive training method, system, device, electronic device, storage medium and program product based on multi-sensory information collaboration.

[0004] The embodiment of the present application provides a virtual reality cognitive training method based on multi-sensory information collaboration, the method comprising: based on a target cognitive training task, determining a virtual reality environment matching the target cognitive training task; determining scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information comprises at least one of scene visual information and scene auditory information; determining auxiliary sensory stimulation information corresponding to the target cognitive training task, wherein the auxiliary sensory stimulation information comprises at least one of target frequency light signal information and target pattern sound signal information; coordinating the scene sensory stimulation information and the auxiliary sensory stimulation information, and outputting them in combination with the virtual reality environment to obtain a virtual reality cognitive training model, so that a trainee can perform cognitive training based on the virtual reality cognitive training model.

[0005] Exemplarily, the frequency of the target frequency optical signal information is within the range of 30 Hz-100 Hz, and the pattern of the target pattern sound signal information includes a binaural beat sound pattern.

[0006] Exemplarily, the scene sensory stimulation information and the auxiliary sensory stimulation information are coordinated and output in combination with the virtual reality environment, including: during the training process, the scene sensory stimulation information is output in combination with the virtual reality environment; before and / or during the training, the auxiliary sensory stimulation information is output in combination with the virtual reality environment.

[0007] Exemplarily, the auxiliary sensory stimulation information also includes force tactile feedback signal information.

[0008] Exemplarily, before and / or during training, the auxiliary sensory stimulation information is output in combination with the virtual reality environment, including: before and / or during training, at least one of the target frequency light signal information and the target pattern sound signal information is output in combination with the virtual reality environment, so as to induce stimulation of a specific brain area of ​​the trainee; during training, the force tactile feedback signal information is output in combination with the virtual reality environment, wherein the force tactile feedback signal information is used to induce stimulation of a specific brain area of ​​the trainee at a specific training opportunity and / or to simulate interactive information in the virtual reality environment.

[0009] Exemplarily, the method also includes: collecting physiological data and behavioral data of trainees during cognitive training based on the virtual reality cognitive training model; adjusting the auxiliary sensory stimulation information based on the physiological data and the behavioral data; coordinating the scene sensory stimulation information and the adjusted auxiliary sensory stimulation information, and outputting them in combination with the virtual reality environment to obtain an adjusted virtual reality cognitive training model, so that trainees can perform cognitive training based on the adjusted virtual reality cognitive training model.

[0010] Another embodiment of the present application provides a virtual reality cognitive training system based on multi-sensory information collaboration, the system comprising: a virtual reality display unit, used to provide a virtual reality environment, and output scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information; an auxiliary sensory stimulation unit, including at least one of a light stimulation unit and an audio stimulation unit, the light stimulation unit is used to output target frequency light signal information, and the audio stimulation unit is used to output target pattern sound signal information; a data processing unit, which performs data interaction with the virtual reality display unit and the auxiliary sensory unit, and the data processing unit is used to execute the above-mentioned virtual reality cognitive training method based on multi-sensory information collaboration.

[0011] Exemplarily, the auxiliary sensory stimulation unit also includes a tactile feedback unit for outputting force tactile feedback signal information.

[0012] Another embodiment of the present application provides a virtual reality cognitive training device based on multi-sensory information collaboration, the device comprising: a first determination module, used to determine a virtual reality environment matching a target cognitive training task based on the target cognitive training task; a second determination module, used to determine scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information; a third determination module, used to determine auxiliary sensory stimulation information corresponding to the target cognitive training task, wherein the auxiliary sensory stimulation information includes at least one of target frequency light signal information and target pattern sound signal information; an output module, used to coordinate the scene sensory stimulation information and the auxiliary sensory stimulation information, and output them in combination with the virtual reality environment to obtain a virtual reality cognitive training model, so that the training personnel can perform cognitive training based on the virtual reality cognitive training model.

[0013] Another embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the above embodiments are implemented.

[0014] Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0015] Another embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method described in any of the above embodiments.

[0016] In the above implementation, based on the target cognitive training task, a virtual reality environment matching the target cognitive training task is determined; scene sensory stimulation information corresponding to the virtual reality environment is determined, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information; auxiliary sensory stimulation information corresponding to the target cognitive training task is determined, wherein the auxiliary sensory stimulation information includes at least one of target frequency light signal information and target pattern sound signal information; the scene sensory stimulation information and the auxiliary sensory stimulation information are coordinated and output in combination with the virtual reality environment to obtain a virtual reality cognitive training model, so that the trainee can perform cognitive training based on the virtual reality cognitive training model. The virtual reality cognitive training method based on multi-sensory information collaboration of the present invention can perform collaborative training based on multi-sensory information in a virtual reality system to enhance the cognitive function and perception ability of the trainee and improve the effect of cognitive training. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a virtual reality cognitive training method based on multi-sensory information collaboration provided in an embodiment of the present application;

[0018] Figure 2 A flowchart of a virtual reality cognitive training method based on multi-sensory information collaboration provided by another embodiment of the present application;

[0019] Figure 3 A schematic diagram of a virtual reality cognitive training system based on multi-sensory information collaboration provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of a virtual reality cognitive training device based on multi-sensory information collaboration provided in an embodiment of the present application;

[0021] Figure 5 A block diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0022] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0023] With the development of science and technology, virtual reality (VR) technology has gradually been applied to the field of cognitive function training. However, the current virtual reality training system mainly relies on a single stimulation method, which cannot fully stimulate the cognitive potential of the brain, and the stimulation effect is poor, resulting in poor cognitive training effect. Therefore, it is necessary to develop a collaborative training system that can provide multi-sensory information stimulation to achieve a more comprehensive cognitive function improvement and improve the cognitive training effect.

[0024] In view of this, the embodiments of the present application provide an optimized virtual reality cognitive training method based on multi-sensory information collaboration, which can perform collaborative training based on multi-sensory information in a virtual reality system to enhance the cognitive function and perceptual ability of trainees and improve the effect of cognitive training.

[0025] Figure 1 A flowchart of a virtual reality cognitive training method based on multi-sensory information collaboration provided in an embodiment of the present application.

[0026] like Figure 1 As shown, the virtual reality cognitive training method 100 based on multi-sensory information collaboration provided in the embodiment of the present application includes steps S110-S140.

[0027] Step S110, based on the target cognitive training task, determining a virtual reality environment that matches the target cognitive training task.

[0028] Exemplarily, cognitive training tasks are varied, and trainees can improve cognitive abilities by completing cognitive training tasks. For example, cognitive training tasks can be tasks for memory abilities, and memory abilities can be improved by completing cognitive training tasks. For different trainees, target cognitive training tasks for trainees can be matched from multiple cognitive training tasks. Cognitive training tasks, for example, include travel training tasks, which require trainees to start from and arrive at the destination based on a specified travel route. Cognitive training tasks can also include home training tasks, which require trainees to complete specified daily actions (such as watching TV, picking up a cup, etc.) in a home environment to improve the cognitive abilities of trainees, which are particularly suitable for the field of cognitive dysfunction in the elderly in the medical field.

[0029] When conducting cognitive training, if the cognitive training task is based on actual scenarios, there are limitations of actual scenarios and high training costs, so training can be conducted in a virtual reality environment. For example, by providing a virtual reality environment, trainees complete the target cognitive training task in the virtual reality environment.

[0030] Step S120, determining scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information.

[0031] Specifically, a virtual reality environment that matches the target cognitive training task is determined, and then the scene sensory stimulation information corresponding to the virtual reality environment is determined. The scene sensory stimulation is, for example, information provided by the virtual reality environment. The scene sensory stimulation information includes, for example, at least one of scene visual information and scene auditory information. For example, when the target cognitive training task requires the trainee to travel from a specified start to a destination based on a specified travel route, the virtual reality environment provides a virtual road environment, and the trainee can travel from a specified start to a destination in the virtual road environment. At this time, the scene visual information is used to simulate obstacles and vehicles in the road environment, and the scene auditory information is used to simulate the vehicle horns and the noise of pedestrians in the road environment, so as to achieve immersive training.

[0032] Step S130, determining auxiliary sensory stimulation information corresponding to the target cognitive training task, wherein the auxiliary sensory stimulation information includes at least one of target frequency light signal information and target pattern sound signal information.

[0033] For example, if the target cognitive training task is used to improve the memory ability of the trainee, the trainee can be stimulated by auxiliary sensory stimulation information to induce the cognitive memory function of the trainee's brain and improve the memory training effect. The target frequency light signal information includes, for example, light flashing stimulation information outputting the target frequency, and the target pattern sound signal information includes sound stimulation information outputting the target pattern.

[0034] Step S140, the scene sensory stimulation information and the auxiliary sensory stimulation information are coordinated and output in combination with the virtual reality environment to obtain a virtual reality cognitive training model, so that the trainee can perform cognitive training based on the virtual reality cognitive training model.

[0035] Exemplarily, when training is conducted based on a virtual reality environment, scene sensory stimulation information and auxiliary sensory stimulation information are coordinated and combined with the virtual reality environment to obtain and output a virtual reality cognitive training model, which includes outputting scene sensory stimulation information and auxiliary sensory stimulation information in the virtual reality environment.

[0036] It can be understood that scene sensory stimulation information and auxiliary sensory stimulation information constitute multi-sensory information. The present application can enhance the cognitive function and perception ability of trainees and improve the effect of cognitive training by coordinating multi-sensory information for cognitive function training in a virtual reality environment.

[0037] In another example, the frequency of the target frequency optical signal information is within a target frequency range, and the frequencies within the target frequency range include 40 Hz, for example, the target frequency range may be 30 Hz-100 Hz. The pattern of the target pattern acoustic signal information includes a binaural beat pattern.

[0038] It can be understood that the virtual cognitive training method of the present application integrates acoustic, optical and electrical stimulation, and uses the sound and video of the virtual reality environment, auxiliary sound stimulation information and auxiliary light stimulation information as multi-sensory information for collaborative training. This method can provide a training environment with comprehensive sensory stimulation, especially by combining 40HZ optical and electrical stimulation information, binaural beat sound stimulation information and scene sensory stimulation information to conduct comprehensive cognitive training, thereby solving the limitations of the virtual reality environment and improving the cognitive training effect.

[0039] In another example, the scene sensory stimulation information and the auxiliary sensory stimulation information are coordinated and output in combination with the virtual reality environment, including: during the training process, the scene sensory stimulation information is output in combination with the virtual reality environment; before and / or during the training, the auxiliary sensory stimulation information is output in combination with the virtual reality environment.

[0040] For example, scene sensory stimulation information corresponding to a virtual reality environment can be output in combination with the virtual reality environment during training. Auxiliary sensory stimulation information can be output in combination with the virtual reality environment before training, or it can be output in combination with the virtual reality environment during training. Outputting auxiliary sensory stimulation information before training can induce and activate the cognitive functions of trainees in advance, which is convenient for improving the cognitive training performance of trainees in subsequent training processes and improving the training effect. Outputting auxiliary sensory stimulation information during training can match and comprehensively induce the cognitive functions of trainees in the current training process together with the scene sensory stimulation information, thereby improving the overall cognitive training effect.

[0041] In another example of the present application, the auxiliary sensory stimulation information may also include force tactile feedback signal information.

[0042] Among them, before and / or during training, auxiliary sensory stimulation information is output in combination with a virtual reality environment, including the following two methods.

[0043] The first method is to output at least one of the target frequency light signal information and the target pattern sound signal information in combination with the virtual reality environment before and / or during training, so as to induce stimulation to the specific brain area of ​​the trainee. Outputting at least one of the target frequency light signal information and the target pattern sound signal information before training can induce stimulation to the specific brain area of ​​the trainee in advance, so that the trainee can perform better in subsequent training. During the training process, outputting at least one of the target frequency light signal information and the target pattern sound signal information can induce stimulation to the specific brain area of ​​the trainee according to the real-time situation of the training during the training process, so as to improve the effect of the current training in real time.

[0044] The second method is to output force tactile feedback signal information in combination with a virtual reality environment during training, wherein the force tactile feedback signal information is used to induce stimulation of specific brain regions of trainees at specific training times and / or to simulate interactive information in a virtual reality environment. For example, trainees can wear force feedback gloves and vibration feedback vests, and output force tactile feedback signal information during training. On the one hand, the force tactile feedback signal information can simulate the tactile experience in reality, enhance the sense of immersion and improve the training effect. On the other hand, the force tactile feedback signal information can induce stimulation of specific brain regions of trainees according to the real-time training situation during training, and improve the effect of the current training in real time.

[0045] In the example of this application, 40HZ photoelectric stimulation information, binaural beat stimulation information, force tactile feedback signal information and virtual reality scene sensory stimulation information can be integrated and coordinated to conduct comprehensive cognitive training, solve the limitations in the virtual reality environment, and thus improve the cognitive training effect. It can be seen that by integrating 40HZ photoelectric stimulation information, binaural beat stimulation information, force tactile feedback signal information and virtual reality scene sensory stimulation information, a multi-sensory collaborative virtual reality training environment is created to enhance the individual's cognitive function and perception ability and improve the cognitive training effect.

[0046] Through the above-mentioned stimulation, specific brain area activities are induced for cognitive functions, and based on the principle of brain remodeling, the cognitive functions of trainees are improved or rehabilitated, which can be applied to the field of cognitive dysfunction training for the elderly in the medical field.

[0047] Figure 2 A flowchart of a virtual reality cognitive training method based on multi-sensory information collaboration provided in another embodiment of the present application.

[0048] like Figure 2 As shown, another virtual reality cognitive training method 200 based on multi-sensory information collaboration provided in an embodiment of the present application includes, for example, steps S210-S270.

[0049] Step S210, based on the target cognitive training task, determining a virtual reality environment that matches the target cognitive training task.

[0050] Step S220, determining scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information.

[0051] Step S230, determining auxiliary sensory stimulation information corresponding to the target cognitive training task, wherein the auxiliary sensory stimulation information includes at least one of target frequency light signal information and target pattern sound signal information.

[0052] Step S240, the scene sensory stimulation information and the auxiliary sensory stimulation information are coordinated and output in combination with the virtual reality environment to obtain a virtual reality cognitive training model, so that the trainee can perform cognitive training based on the virtual reality cognitive training model.

[0053] Among them, steps S210-S240 are the same as or similar to the above steps S110-S140 and are not repeated here.

[0054] Step S250, collecting physiological data and behavioral data of the trainee during cognitive training based on the virtual reality cognitive training model.

[0055] Exemplarily, physiological data include, for example, EEG data, ECG data, respiration data, blood oxygen data, human body nine-axis posture data, etc. Behavioral data include, for example, trainee motion data, facial expression data, data of interaction with the virtual reality environment, trainee operation data, etc.

[0056] Step S260, adjusting the auxiliary sensory stimulation information based on the physiological data and the behavioral data.

[0057] For example, the relevant parameters of the auxiliary sensory stimulation information are adjusted based on the physiological data and behavioral data. For example, if it is known based on the auxiliary sensory stimulation information that the training effect of the trainees is not good, the stimulation level of the auxiliary sensory stimulation information can be appropriately increased.

[0058] Step S270, the scene sensory stimulation information and the adjusted auxiliary sensory stimulation information are coordinated and output in combination with the virtual reality environment to obtain an adjusted virtual reality cognitive training model, so that the trainee can perform cognitive training based on the adjusted virtual reality cognitive training model.

[0059] It can be understood that the training effect can be monitored in real time through physiological data and behavioral data, and the virtual reality cognitive training mode can be dynamically adjusted according to the training effect to achieve personalized training for the trainees. By collecting the trainees' physiological data and behavioral performance data, the training effect can be evaluated, and the acoustic and optical stimulation parameters and tactile feedback mode can be adjusted according to the evaluation results to optimize subsequent training.

[0060] Figure 3 A schematic diagram of a virtual reality cognitive training system based on multi-sensory information collaboration provided in an embodiment of the present application.

[0061] like Figure 3 As shown, the virtual reality cognitive training device 300 based on multi-sensory information collaboration provided in an embodiment of the present application includes: a virtual reality display unit 310, an auxiliary sensory stimulation unit 320 and a data processing unit 330.

[0062] A virtual reality display unit 310, configured to provide a virtual reality environment and output scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information;

[0063] The auxiliary sensory stimulation unit 320 includes at least one of a light stimulation unit 321 and an audio stimulation unit 322, wherein the light stimulation unit 321 is used to output target frequency light signal information, and the audio stimulation unit 322 is used to output target pattern sound signal information;

[0064] The data processing unit 330 performs data interaction with the virtual reality display unit and the auxiliary sensory unit. The data processing unit is used to execute the above virtual reality cognitive training method based on multi-sensory information coordination.

[0065] For example, the data processing unit 330 is used to perform: based on the target cognitive training task, determine a virtual reality environment that matches the target cognitive training task; determine scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information; determine auxiliary sensory stimulation information corresponding to the target cognitive training task, wherein the auxiliary sensory stimulation information includes at least one of target frequency light signal information and target pattern sound signal information; coordinate the scene sensory stimulation information and the auxiliary sensory stimulation information, and output them in combination with the virtual reality environment to obtain a virtual reality cognitive training model, so that trainees can perform cognitive training based on the virtual reality cognitive training model.

[0066] The frequency of the target frequency optical signal information is within a target frequency range, the frequencies within the target frequency range include 40 Hz, and the mode of the target mode sound signal information includes a binaural beat sound mode.

[0067] The scene sensory stimulation information and the auxiliary sensory stimulation information are coordinated and output in combination with a virtual reality environment, including: during the training process, the scene sensory stimulation information is output in combination with the virtual reality environment; before and / or during the training process, the auxiliary sensory stimulation information is output in combination with the virtual reality environment.

[0068] Exemplarily, the auxiliary sensory stimulation unit 320 further includes a tactile feedback unit 323 for outputting force tactile feedback signal information. The tactile feedback unit 323 includes, for example, feedback gloves and a vibration feedback vest.

[0069] The data processing unit 330 is used to execute: before and / or during training, output at least one of target frequency light signal information and target pattern sound signal information in combination with the virtual reality environment, so as to induce stimulation of specific brain areas of the trainee; during training, output force tactile feedback signal information in combination with the virtual reality environment, wherein the force tactile feedback signal information is used to induce stimulation of specific brain areas of the trainee at specific training times and / or to simulate interactive information in a virtual reality environment.

[0070] The data processing unit 330 is used to execute the following: collecting physiological data and behavioral data of trainees during cognitive training based on a virtual reality cognitive training model; adjusting auxiliary sensory stimulation information based on the physiological data and behavioral data; coordinating scene sensory stimulation information with the adjusted auxiliary sensory stimulation information, and outputting them in combination with a virtual reality environment to obtain an adjusted virtual reality cognitive training model, so that trainees can perform cognitive training based on the adjusted virtual reality cognitive training model.

[0071] In another example, the present application also proposes a multi-sensory collaborative virtual reality cognitive function training system integrating acoustic and optical stimulation, including:

[0072] Virtual reality display unit: The virtual reality display unit includes a virtual display head display unit or a virtual display glasses unit, which is used to provide an immersive visual experience, and has a built-in gyroscope and accelerometer to achieve head movement tracking;

[0073] Audio device unit: Similar to the audio stimulation unit mentioned above, it is used to provide audio stimulation of a specific pattern (such as binaural beats), using sound signals of a specific frequency to enhance the cognitive function of the brain;

[0074] Photoelectric stimulation unit: generates light flashes of a specific frequency (such as 40HZ) through an LED light source, synchronized with the audio device unit, and provides visual stimulation;

[0075] Haptic feedback device: Similar to the haptic feedback unit mentioned above, it includes force feedback gloves and vibration feedback vests, which can simulate the tactile experience in reality and enhance the sense of immersion;

[0076] Control unit: integrated control of the operation of the above units or devices, synchronized acoustic and optical stimulation with tactile feedback, and adjusted stimulation parameters according to training results (such as physiological data and behavioral data);

[0077] Data analysis unit: collects physiological and behavioral data during training, evaluates training effects and optimizes training plans.

[0078] The control unit and the data analysis unit may constitute the above-mentioned data processing unit, for example.

[0079] In another example, the present application also proposes a multi-sensory collaborative virtual reality cognitive function training method integrating acoustic and optical stimulation, which is performed by a multi-sensory collaborative virtual reality cognitive function training system integrating acoustic and optical stimulation, and the method includes the following steps:

[0080] Start the virtual reality cognitive function training system, and train the personnel to wear the virtual reality display unit and tactile feedback device;

[0081] An audio signal of a specific pattern (such as binaural beats) is played through the audio device unit, and at the same time, the photoelectric stimulation unit generates synchronized light flashes of a specific frequency (such as 40 Hz);

[0082] The control unit synchronizes the acoustic and optical stimulation with the tactile feedback and outputs them according to a preset training program for training;

[0083] Trainees complete designated cognitive training tasks in a virtual reality environment while receiving multi-sensory stimulation;

[0084] The data analysis unit collects the user's physiological and behavioral data to evaluate the training effect;

[0085] According to the evaluation results, adjust the acoustic, optical and electrical stimulation parameters and tactile feedback mode to optimize subsequent training.

[0086] Figure 4 A schematic diagram of a virtual reality cognitive training device based on multi-sensory information collaboration provided in an embodiment of the present application.

[0087] like Figure 4 As shown, the virtual reality cognitive training device 400 based on multi-sensory information collaboration provided in the embodiment of the present application includes: a first determination module 410, a second determination module 420, a third determination module 430 and an output module 440.

[0088] A first determination module 410 is used to determine a virtual reality environment matching the target cognitive training task based on the target cognitive training task;

[0089] A second determination module 420 is used to determine scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information;

[0090] A third determination module 430 is used to determine auxiliary sensory stimulation information corresponding to the target cognitive training task, wherein the auxiliary sensory stimulation information includes at least one of target frequency light signal information and target pattern sound signal information;

[0091] The output module 440 is used to coordinate the scene sensory stimulation information and the auxiliary sensory stimulation information, and output them in combination with the virtual reality environment to obtain a virtual reality cognitive training model so that the trainees can perform cognitive training based on the virtual reality cognitive training model.

[0092] Exemplarily, the frequency of the target frequency optical signal information is within a target frequency range, the frequencies within the target frequency range include 40 Hz, and the pattern of the target pattern sound signal information includes a binaural beat sound pattern.

[0093] Exemplarily, the output module 440 is also used to: output scene sensory stimulation information in combination with the virtual reality environment during training; and output auxiliary sensory stimulation information in combination with the virtual reality environment before and / or during training.

[0094] Exemplarily, the auxiliary sensory stimulation information also includes force tactile feedback signal information.

[0095] Exemplarily, before and / or during training, auxiliary sensory stimulation information is output in combination with the virtual reality environment, including: before and / or during training, at least one of target frequency light signal information and target pattern sound signal information is output in combination with the virtual reality environment to induce stimulation of specific brain areas of the trainee; during training, force tactile feedback signal information is output in combination with the virtual reality environment, wherein the force tactile feedback signal information is used to induce stimulation of specific brain areas of the trainee at specific training times and / or to simulate interactive information in the virtual reality environment.

[0096] Exemplarily, the virtual reality cognitive training device 400 based on multi-sensory information collaboration also includes:

[0097] A collection module is used to collect physiological data and behavioral data of trainees during cognitive training based on the virtual reality cognitive training model;

[0098] An adjustment module, used for adjusting auxiliary sensory stimulation information based on physiological data and behavioral data;

[0099] The output adjustment module is used to coordinate the scene sensory stimulation information and the adjusted auxiliary sensory stimulation information, and output them in combination with the virtual reality environment to obtain the adjusted virtual reality cognitive training model, so that the trainees can perform cognitive training based on the adjusted virtual reality cognitive training model.

[0100] It can be understood that the specific implementation process of the virtual reality cognitive training device 400 based on multi-sensory information collaboration can refer to the implementation process of the virtual reality cognitive training method based on multi-sensory information collaboration above, and will not be repeated here.

[0101] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0102] One embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.

[0103] One embodiment of the present application provides a virtual reality cognitive training system based on multi-sensory information collaboration, including the following electronic device.

[0104] Figure 5 A block diagram of an electronic device provided for an embodiment of the present application.

[0105] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned virtual reality cognitive training method based on multi-sensory information collaboration is implemented.

[0106] like Figure 5 As shown, for ease of understanding, the embodiment of the present application shows a specific electronic device 500.

[0107] The electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0108] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0109] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods described above. For example, in some embodiments, any one or more of the various methods described above may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, any one or more of the various methods described above may be executed. One or more steps. Alternatively, in other embodiments, the computing unit 501 may be configured to perform any one or more of the various methods described above in any other appropriate manner (e.g., by means of firmware).

[0111] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this application, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0112] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0113] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0114] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0115] In addition, the terms "first", "second", etc. used in the embodiments of the present application are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present application can explicitly or implicitly indicate that at least one of the features is included in the embodiment. In the description of the present application, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0116] In this application, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "connected" and "fixed" etc. appearing in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific implementation situation.

[0117] In the present application, unless otherwise clearly specified and limited, a first feature being “above” or “below” a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being “above”, “above”, and “above” a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being “below”, “below”, and “below” a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0118] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A virtual reality cognitive training method based on multi-sensory information collaboration, characterized in that: The method comprises: Based on the target cognitive training task, determining a virtual reality environment matching the target cognitive training task; Determining scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information; Determining auxiliary sensory stimulation information corresponding to the target cognitive training task, wherein the auxiliary sensory stimulation information includes at least one of target frequency light signal information and target pattern sound signal information; The scene sensory stimulation information and the auxiliary sensory stimulation information are coordinated and output in combination with the virtual reality environment to obtain a virtual reality cognitive training model, so that trainees can perform cognitive training based on the virtual reality cognitive training model.

2. The method according to claim 1, characterized in that The frequency of the target frequency optical signal information is within the range of 30 Hz-100 Hz, and the mode of the target pattern sound signal information includes a binaural beat sound mode.

3. The method according to claim 1, characterized in that The step of coordinating the scene sensory stimulation information and the auxiliary sensory stimulation information and outputting them in combination with the virtual reality environment includes: During the training process, the scene sensory stimulation information is output in combination with the virtual reality environment; Before and / or during training, the auxiliary sensory stimulation information is output in combination with the virtual reality environment.

4. The method according to claim 3, characterized in that The auxiliary sensory stimulation information also includes force tactile feedback signal information.

5. The method according to claim 4, characterized in that The step of outputting the auxiliary sensory stimulation information in combination with the virtual reality environment before and / or during the training includes: Before and / or during training, outputting at least one of the target frequency light signal information and the target pattern sound signal information in combination with the virtual reality environment, so as to induce stimulation to a specific brain area of ​​the trainee; During the training process, the force tactile feedback signal information is output in combination with the virtual reality environment, wherein the force tactile feedback signal information is used to induce stimulation of specific brain areas of the trainee at specific training times and / or to simulate interactive information in the virtual reality environment.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Collecting physiological data and behavioral data of trainees during cognitive training based on the virtual reality cognitive training model; Adjusting the auxiliary sensory stimulation information based on the physiological data and the behavioral data; The scene sensory stimulation information and the adjusted auxiliary sensory stimulation information are coordinated and output in combination with the virtual reality environment to obtain an adjusted virtual reality cognitive training model, so that the training personnel can perform cognitive training based on the adjusted virtual reality cognitive training model.

7. A virtual reality cognitive training system based on multi-sensory information collaboration, characterized in that: The system comprises: A virtual reality display unit, configured to provide a virtual reality environment and output scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information; an auxiliary sensory stimulation unit, comprising at least one of a light stimulation unit and an audio stimulation unit, wherein the light stimulation unit is used to output target frequency light signal information, and the audio stimulation unit is used to output target pattern sound signal information; A data processing unit is used to perform data interaction with the virtual reality display unit and the auxiliary sensory unit, and the data processing unit is used to execute the method according to any one of claims 1 to 6.

8. The system according to claim 7, characterized in that The auxiliary sensory stimulation unit also includes a tactile feedback unit for outputting force tactile feedback signal information.

9. A virtual reality cognitive training device based on multi-sensory information collaboration, characterized in that: The device comprises: A first determination module is used to determine a virtual reality environment matching the target cognitive training task based on the target cognitive training task; A second determination module is used to determine scene sensory stimulation information corresponding to the virtual reality environment, wherein the scene sensory stimulation information includes at least one of scene visual information and scene auditory information; a third determination module, configured to determine auxiliary sensory stimulation information corresponding to the target cognitive training task, wherein the auxiliary sensory stimulation information includes at least one of target frequency light signal information and target pattern sound signal information; The output module is used to coordinate the scene sensory stimulation information and the auxiliary sensory stimulation information, and output them in combination with the virtual reality environment to obtain a virtual reality cognitive training model, so that the trainee can perform cognitive training based on the virtual reality cognitive training model.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images