A cultural and creative experience system and control method based on virtual reality technology

Abstract

The present application provides a cultural and creative experience system and control method based on virtual reality technology, by acquiring motor imagery brain waves; converting the motor imagery brain waves into motion timing EEG features and motion source domain EEG features respectively; determining the spatial dynamic entropy of the brain waves of the target user when performing motor imagery according to the motion source domain EEG features; acquiring the perceptual EEG of the target user when using the virtual reality cultural and creative experience equipment, and determining the neural response values ​​of various regions inside the brain according to the perceptual EEG and the spatial dynamic entropy; and providing motor imagery feedback to the target user's cultural and creative experience process according to the motion timing EEG features and all neural response values. The scheme of the present application can be used to extract the spatiotemporal features in the motor imagery EEG, and use them for feedback control of the cultural and creative experience system, reducing the action delay caused by the long response time of the input device, thereby enhancing the user's sense of immersion in the cultural and creative experience.

Description

Technical Field

[0001] The present application relates to the field of cultural and creative experience technology, and more specifically, to a cultural and creative experience system and control method based on virtual reality technology. Background Art

[0002] The existing cultural and creative experience system can use technologies such as virtual reality, augmented reality, and artificial intelligence to provide cultural and creative content. It aims to present cultural and creative content to users in a more vivid, interactive, and immersive way through technological means, thereby enhancing users' sense of participation and experience.

[0003] The cultural and creative experience system based on virtual reality technology refers to an immersive interactive experience platform that uses virtual reality technology to provide cultural and creative content. The system brings users into a virtual three-dimensional environment through virtual reality equipment and related software, allowing users to experience and interact with cultural and creative content in an immersive way. However, in the prior art, the cultural and creative experience system based on virtual reality technology usually uses input devices such as handles and hand tracking devices to capture user behavior, but these input devices have the problem of long response time, which will cause the user's actions to be out of sync with the feedback in the virtual world, reducing the user's immersion. Motor imagery EEG refers to the individual imagining a certain movement without performing the movement, thereby activating the brain area related to the movement, thereby generating specific EEG signals. These EEG signals can be collected by EEG equipment, and further analyzed and classified for various application scenarios. Therefore, how to reduce the action delay caused by the long response time of the input device, thereby enhancing the user's immersion in the cultural and creative experience has become a difficult problem faced by the industry. Summary of the invention

[0004] The present application provides a cultural and creative experience system and control method based on virtual reality technology to solve the technical problem of how to reduce the action delay caused by the long response time of the input device, thereby enhancing the user's immersion in the cultural and creative experience.

[0005] In a first aspect, the present application provides a cultural and creative experience control method based on virtual reality technology, comprising:

[0006] Collect the target user's motor imagination brain waves when using virtual reality cultural and creative experience equipment;

[0007] Performing multi-scale convolution on the motor imagery EEG to obtain the spatiotemporal convolution features of normal EEG signals of the target user when using the virtual reality device for cultural and creative experience, and then converting the motor imagery EEG into motion time sequence EEG features and motion source domain EEG features based on the spatiotemporal convolution features;

[0008] Determining the spatial dynamic entropy of the brain waves of the target user when performing motor imagery according to the motor source domain electroencephalogram characteristics;

[0009] Acquire a perceptual electroencephalogram of the target user when using the virtual reality cultural and creative experience device, determine the dynamic interaction degree of brain nerves in different regions of the target user's brain through the perceptual electroencephalogram, and then determine the neural response value of each region inside the brain of the target user when performing the cultural and creative experience according to all the dynamic interaction degrees and the spatial dynamic entropy;

[0010] A feedback adjustment sequence is determined when a target user uses a virtual reality device for a cultural and creative experience based on the motion timing EEG characteristics and all neural response values, and then feedback regulation is performed on the cultural and creative experience of the target user based on the feedback adjustment sequence.

[0011] In some embodiments, multi-scale convolution is performed on the motor imagery brain waves to obtain the spatiotemporal convolution features of normal brain wave signals of the target user when using the virtual reality device for cultural and creative experience, specifically including:

[0012] Determining a two-dimensional convolution domain when performing multi-scale convolution on the motor imagery brain waves;

[0013] Convolving the two-dimensional convolution domain through a convolutional neural network to obtain the spatial characteristics of the motor imagery brain wave;

[0014] Extracting the temporal features of the motor imagery brain waves from the two-dimensional convolutional domain based on a long short-term memory network;

[0015] The spatiotemporal convolution features of the normal brain wave signals of the target user when using the virtual reality device for cultural and creative experience are determined according to the spatial features and the temporal features.

[0016] In some embodiments, converting the motor imagery EEG waves into motion time sequence EEG features and motion source domain EEG features based on the spatiotemporal convolution features specifically includes:

[0017] Determining a motion time-series EEG feature according to the time-series feature in the spatiotemporal convolution feature;

[0018] The motion source domain EEG features are determined based on the spatial features in the spatiotemporal convolution features.

[0019] In some embodiments, determining the spatial dynamic entropy of the brain waves of the target user when performing motor imagery according to the motor source domain electroencephalogram features specifically includes:

[0020] Determine the spatial distribution domain of the brain wave signals at different electrode positions in the virtual reality device according to the brain wave characteristics of the motion source domain;

[0021] determining a spatial relative fluctuation between electrodes according to the spatial distribution domain;

[0022] The spatial dynamic entropy of the brain waves of the target user when performing motor imagery is determined based on all spatial relative fluctuations.

[0023] In some embodiments, determining the dynamic interaction degree of cranial nerves in different regions of the target user's brain through the perceived electroencephalogram specifically includes:

[0024] Determining the spectrum intensity of the perceptual brain wave signal in the perceptual electroencephalogram at each frequency component;

[0025] The dynamic interaction degree of the cranial nerves in different areas of the target user's brain is determined based on all spectrum intensities.

[0026] In some embodiments, determining the feedback adjustment sequence when the target user uses the virtual reality device for cultural and creative experience based on the motion timing EEG characteristics and all neural response values ​​specifically includes:

[0027] Performing motion synchronization detection on the motion time-series EEG characteristics to obtain long-range motion characteristics of the target user when using the virtual reality device for cultural and creative experience;

[0028] Determining an initial feedback sequence of each motion synchronization coefficient in the long-range motion feature according to all neural response values;

[0029] Based on all the initial feedback sequences, the feedback adjustment sequence when the target users use virtual reality devices for cultural and creative experience is determined.

[0030] In some embodiments, the motion synchronization detection is performed on the motion time sequence EEG characteristics to obtain the long-range motion characteristics of the target user when using the virtual reality device for cultural and creative experience, specifically including:

[0031] Extracting movement-related potentials from the movement time-series EEG features;

[0032] Analyzing the movement synchronization of the brain electrical activity at different time periods when the target user performs movement imagination according to the movement-related potential, and obtaining the movement synchronization coefficient of the brain electrical activity at different time periods;

[0033] Based on all motion synchronization coefficients, the long-range motion characteristics of the target users when using virtual reality devices for cultural and creative experiences are determined.

[0034] In the second aspect, the present application provides a cultural and creative experience system based on virtual reality technology, including:

[0035] The acquisition module is used to collect the target user's motor imagination brain waves when using the virtual reality cultural and creative experience equipment;

[0036] A processing module, configured to perform multi-scale convolution on the motor imagery EEG to obtain the spatiotemporal convolution features of normal EEG signals of the target user when the target user uses the virtual reality device for cultural and creative experience, and then convert the motor imagery EEG into motion time sequence EEG features and motion source domain EEG features based on the spatiotemporal convolution features;

[0037] The processing module is also used to determine the spatial dynamic entropy of the brain waves of the target user when performing motor imagination according to the brain wave characteristics of the motion source domain;

[0038] The processing module is also used to obtain a perceptual electroencephalogram of the target user when using the virtual reality cultural and creative experience device, determine the dynamic interaction degree of brain nerves in different regions of the target user's brain through the perceptual electroencephalogram, and then determine the neural response value of each region inside the brain of the target user when performing the cultural and creative experience according to all the dynamic interaction degrees and the spatial dynamic entropy;

[0039] An execution module is used to determine a feedback adjustment sequence when a target user uses a virtual reality device for cultural and creative experience based on the motion timing EEG characteristics and all neural response values, and then to perform feedback regulation on the cultural and creative experience of the target user based on the feedback adjustment sequence.

[0040] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned cultural and creative experience control method based on virtual reality technology.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned cultural and creative experience control method based on virtual reality technology is implemented.

[0042] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:

[0043] In the cultural and creative experience system and control method based on virtual reality technology provided by the present application, first, the motor imagery brain waves of the target user when using the virtual reality cultural and creative experience device are collected; the motor imagery brain waves are subjected to multi-scale convolution to obtain the spatiotemporal convolution features of the normal brain wave signals of the target user when using the virtual reality device for cultural and creative experience, and then the motor imagery brain waves are respectively converted into motion timing brain wave features and motion source domain brain wave features based on the spatiotemporal convolution features; the spatial dynamic entropy of the brain waves of the target user when performing motor imagery is determined according to the motion source domain brain wave features; the perceptual electroencephalogram of the target user when using the virtual reality cultural and creative experience device is obtained, the dynamic interaction degree of the cranial nerves in different regions of the target user's brain is determined through the perceptual electroencephalogram, and then the neural response values ​​of various regions inside the brain of the target user when performing cultural and creative experience are determined according to all the dynamic interaction degrees and the spatial dynamic entropy; the feedback adjustment sequence of the target user when using the virtual reality device for cultural and creative experience is determined according to the motion timing brain wave features and all the neural response values, and then the cultural and creative experience of the target user is feedback-regulated based on the feedback adjustment sequence.

[0044] It can be seen that the present application performs feedback regulation on the target user's cultural and creative experience based on the feedback regulation sequence; first, the target user's motor imagery brain waves are collected when using the virtual reality cultural and creative experience device; the spatiotemporal convolution characteristics of the normal brain wave signals of the target user when using the virtual reality device for cultural and creative experience are determined, and then the motor imagery brain waves are converted into motion timing EEG characteristics and motion source domain EEG characteristics based on the spatiotemporal convolution characteristics respectively; the spatial dynamic entropy of the brain waves of the target user when performing motor imagery is determined according to the motion source domain EEG characteristics; secondly, the perceptual EEG of the target user when using the virtual reality cultural and creative experience device is obtained, and the dynamic interaction degree of the cranial nerves in different areas of the target user's brain is determined through the perceptual EEG, and then the neural response values ​​of each area of ​​the target user's brain when performing the cultural and creative experience are determined according to all the dynamic interaction degrees and the spatial dynamic entropy. Determining the neural response values ​​of each area of ​​the target user's brain when performing the cultural and creative experience can obtain the intensity of the activity of each neuron in the brain, reflecting the activity level of each neuron within a specific time. The above steps The brain activity area of ​​the user during the cultural and creative experience can be accurately located, which is helpful to understand the brain area activated by the user in a specific motor task, so as to provide more targeted feedback; then, the feedback adjustment sequence when the target user uses the virtual reality device for the cultural and creative experience is determined according to the motor timing EEG characteristics and all neural response values. The feedback adjustment sequence when the target user uses the virtual reality device for the cultural and creative experience can be determined to obtain a series of feedback instructions generated after analyzing the EEG data generated when the user uses the virtual reality device for the cultural and creative experience, which are used to adjust and control the virtual reality environment in real time to enhance the user's immersive experience and avoid the problem of the user's action being out of sync with the feedback in the virtual world due to the long response time of the input device; finally, the cultural and creative experience of the target user is feedback-regulated based on the feedback adjustment sequence; in summary, based on the above scheme, the spatiotemporal characteristics in the motor imagery EEG can be extracted and used for feedback regulation of the cultural and creative experience system, reducing the action delay caused by the long response time of the input device, thereby enhancing the user's immersion in the cultural and creative experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is an exemplary flow chart of a cultural and creative experience control method based on virtual reality technology according to some embodiments of the present application;

[0046] Figure 2 is an exemplary flow chart of determining spatiotemporal convolution features according to some embodiments of the present application;

[0047] Figure 3 is an exemplary flow chart of determining spatial dynamic entropy according to some embodiments of the present application;

[0048] Figure 4is a schematic diagram of exemplary hardware and / or software of a cultural and creative experience system based on virtual reality technology according to some embodiments of the present application;

[0049] Figure 5 It is a structural schematic diagram of a computer device for implementing a cultural and creative experience control method based on virtual reality technology as shown in some embodiments of the present application. DETAILED DESCRIPTION

[0050] The embodiment of the present application provides a cultural and creative experience system and control method based on virtual reality technology, the core of which is to obtain motor imagery brain waves; convert the motor imagery brain waves into motion time series brain wave characteristics and motion source domain brain wave characteristics respectively; determine the spatial dynamic entropy of the brain waves of the target user when performing motor imagery according to the motion source domain brain wave characteristics; obtain the perceptual electroencephalogram of the target user when using the virtual reality cultural and creative experience equipment, and determine the neural response values ​​of various regions inside the brain according to the perceptual electroencephalogram and the spatial dynamic entropy; and provide motor imagery feedback to the cultural and creative experience process of the target user according to the motion time series brain wave characteristics and all neural response values. By adopting the scheme of the present application, the spatiotemporal characteristics in the motor imagery brain wave can be extracted and used for feedback control of the cultural and creative experience system, reducing the action delay caused by the long response time of the input device, thereby enhancing the user's immersion in the cultural and creative experience.

[0051] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of a cultural and creative experience control method based on virtual reality technology according to some embodiments of the present application. The cultural and creative experience control method 100 based on virtual reality technology mainly includes the following steps:

[0052] In step 101, the motor imagery brain waves of the target user when using the virtual reality cultural and creative experience device are collected.

[0053] It should be noted that, in the present application, the motor imagery brain wave is the electrophysiological activity signal of the target user when performing a cultural and creative experience; in specific implementation, the motor imagery record of the target user performing a preset number of cultural and creative experiences (the default is 200 times) can be obtained in the virtual reality device, and the collection of the brain wave signals of the target user performing motor imagery in the motor imagery record is used as the motor imagery brain wave, wherein the motor imagery record includes a preset motor imagery task, the brain wave signal of the target user performing motor imagery, and the electrode position of all electrodes on the virtual reality device, and the electrode position refers to the position coordinates of each electrode on the scalp of the virtual reality device when collecting the brain wave signal of the target user.

[0054] In step 102, multi-scale convolution is performed on the motor imagery EEG waves to obtain the spatiotemporal convolution features of normal EEG signals of the target user when using the virtual reality device for cultural and creative experience, and then based on the spatiotemporal convolution features, the motor imagery EEG waves are converted into motion timing EEG features and motion source domain EEG features respectively.

[0055] In some embodiments, reference Figure 2 , which is an exemplary flow chart of determining spatiotemporal convolution features according to some embodiments of the present application. In the present application, multi-scale convolution is performed on the motor imagery brain waves to obtain the spatiotemporal convolution features of normal brain wave signals of the target user when using a virtual reality device for cultural and creative experience. The following steps can be used to achieve this:

[0056] In step 1021, a two-dimensional convolution domain is determined when multi-scale convolution is performed on the motor imagery brain waves;

[0057] In step 1022, the two-dimensional convolution domain is convolved by a convolutional neural network to obtain the spatial features of the motor imagery brain waves;

[0058] In step 1023, the temporal features of the motor imagery brain waves are extracted from the two-dimensional convolutional domain based on the long short-term memory network;

[0059] In step 1024, the spatiotemporal convolution features of the normal brain wave signals of the target user when using the virtual reality device for cultural and creative experience are determined according to the spatial features and the temporal features.

[0060] It should be noted that, in the present application, the two-dimensional convolution domain is a two-dimensional matrix for performing a two-dimensional convolution operation on motor imagery brain waves; the spatial feature is a feature that describes the similarity between motor imagery brain waves at various spatial positions (such as electrode positions); the temporal feature is a feature that describes the changes of brain wave signals over time in motor imagery brain waves; and the spatiotemporal convolution feature is a feature that describes the changes of motor imagery brain waves in space and time.

[0061] In specific implementation, first, all noise-free signals in the motor imagery brain wave are converted into a time-frequency matrix through an existing time-frequency analysis method (such as short-time Fourier transform), and then all time-frequency matrices are stacked according to the electrode position of the noise-free signal corresponding to each time-frequency matrix as a two-dimensional convolution domain for multi-scale convolution of the motor imagery brain wave; secondly, an existing multi-layer convolutional neural network is loaded, and the two-dimensional convolution domain is used as the input of the multi-layer convolutional neural network. After executing the multi-layer convolutional neural network, the vector in the final output of the multi-layer convolutional neural network is used as the spatial feature of the motor imagery brain wave; then, an existing long short-term memory network is loaded, and the two-dimensional convolution domain is used as the input of the long short-term memory network. After executing the long short-term memory network, the hidden state of the last time step in the output result of the long short-term memory network is used as the temporal feature of the motor imagery brain wave; finally, the set of the spatial feature and the temporal feature is used as the spatiotemporal convolution feature of the normal brain wave signal when the target user uses the virtual reality device for cultural and creative experience.

[0062] In some embodiments, converting the motor imagery EEG into motion time sequence EEG features and motion source domain EEG features based on the spatiotemporal convolution features can be achieved by using the following steps:

[0063] Determining a motion time-series EEG feature according to the time-series feature in the spatiotemporal convolution feature;

[0064] The motion source domain EEG features are determined based on the spatial features in the spatiotemporal convolution features.

[0065] It should be noted that, in the present application, the motion source domain EEG features are EEG signals that reflect the electrical activity patterns of different brain regions of the target user during motor imagination; the motion timing EEG features are EEG signals that reflect the temporal changes in the target user's brain electrical activity during motor imagination.

[0066] In the specific implementation, first, an existing gated recurrent unit model is initialized, and the timing feature is used as the input of the gated recurrent unit model. After the gated recurrent unit model is executed, the time series data of the output in the fully connected layer of the gated recurrent unit model is used as the motion timing EEG feature; then, the electrode position of each electrode on the virtual reality device is obtained, and a three-dimensional space (the default size is 100 100 100) is initialized, the electrode position of each electrode is mapped to the three-dimensional space, and then all vector values ​​in the spatial feature are filled into the position of the corresponding electrode in the three-dimensional space to form a continuous spatial feature distribution, and the existing feature analysis algorithm (such as independent component analysis) is used to perform feature analysis on the spatial feature distribution, and the analysis result of the feature analysis algorithm can be used as the motion source domain EEG feature.

[0067] In step 103, the spatial dynamic entropy of the brain waves of the target user when performing motor imagery is determined according to the motion source domain brain wave characteristics.

[0068] In some embodiments, reference Figure 3 , which is an exemplary flow chart of determining spatial dynamic entropy according to some embodiments of the present application. In the present application, determining the spatial dynamic entropy of the brain waves of the target user when performing motion imagination according to the EEG characteristics of the motion source domain can be implemented by the following steps:

[0069] In step 1031, the spatial distribution domain of the EEG signals at different electrode positions in the virtual reality device is determined according to the EEG characteristics of the motion source domain;

[0070] In step 1032, the spatial relative fluctuation between the electrodes is determined according to the spatial distribution domain;

[0071] In step 1033 , the spatial dynamic entropy of the brain waves of the target user when performing motor imagery is determined based on all spatial relative fluctuations.

[0072] It should be noted that in this application, spatial dynamic entropy is an indicator that quantifies the volatility and complexity of EEG signals in the spatial dimension; spatial relative fluctuation refers to the changes between EEG signals at different electrode positions, reflecting the differences and interactions in the activities of different brain regions; the spatial distribution domain is the distribution pattern of EEG characteristics in the motion source domain at different electrode positions. The signal recorded by each electrode represents the electrical activity in a specific area of ​​the brain, and the spatial distribution domain reflects the spatial pattern of these electrical activities.

[0073] In the specific implementation, first, the electrode position of each electrode on the virtual reality device is obtained, and a three-dimensional space (the default size is 100 100 100) is initialized, the electrode position of each electrode is mapped to the three-dimensional space, and then each brain wave signal in the motion source domain EEG feature is filled into the position of the electrode corresponding to the brain wave signal in the three-dimensional space, and the filled three-dimensional space is used as the spatial distribution domain; then, for each pair of electrodes in the spatial distribution domain, the mean of the signal difference values ​​of the brain wave signals corresponding to the two electrodes at all time points is calculated, and the obtained mean is used as the spatial relative fluctuation between the two electrodes. The spatial relative fluctuation between each electrode can be obtained in the above manner; finally, the entropy value of each spatial relative fluctuation is calculated using the existing entropy calculation formula (such as Shannon entropy), and the set of all entropy values ​​is used as the spatial dynamic entropy of the brain wave when the target user performs motion imagination.

[0074] In step 104, the perceptual electroencephalogram of the target user when using the virtual reality cultural and creative experience device is obtained, and the dynamic interaction degree of the brain nerves in different areas of the target user's brain is determined through the perceptual electroencephalogram, and then the neural response value of each area inside the brain of the target user when performing the cultural and creative experience is determined according to all the dynamic interaction degrees and the spatial dynamic entropy.

[0075] It should be noted that in the present application, the perceptual EEG is an image that records the perceptual brain wave signals generated by the target user's perception of changes in the virtual reality environment during the cultural and creative experience. The perceptual brain wave signals may include visual perception brain waves, auditory perception brain waves, cognitive brain waves, and attention brain waves, etc.; in specific implementation, when the target user is experiencing the cultural and creative experience, the existing portable EEG headset can be used to collect the target user's brain wave signals until the end of the cultural and creative experience, and then all the collected brain wave signals are processed through existing EEG signal processing and visualization tools (such as EEGLAB) to obtain the corresponding perceptual EEG.

[0076] In some embodiments, determining the dynamic interaction degree of cranial nerves in different regions of the target user's brain through the perceived EEG can be achieved by the following steps:

[0077] Determining the spectrum intensity of the perceptual brain wave signal in the perceptual electroencephalogram at each frequency component;

[0078] The dynamic interaction degree of the cranial nerves in different areas of the target user's brain is determined based on all spectrum intensities.

[0079] It should be noted that in the present application, the dynamic interactivity is a parameter that reflects the degree of coordination of a neuron group during a cultural and creative experience; the spectral intensity is a parameter that describes the intensity characteristics of different frequency components in a frequency domain signal; in specific implementation, first, the brain wave signal in the perceptual electroencephalogram can be extracted, and the brain wave signal can be divided according to a time window (the default is 1 second), and the brain wave signal in each time window can be processed by an existing spectrum analysis method (such as short-time Fourier transform), and the output result is used as the spectrum intensity of each frequency component; then, the spectrum intensity of α waves (8-13 Hz), β waves (13-30 Hz), θ waves (4-7 Hz) and δ waves (less than 4 Hz) is extracted from all frequency components, and the entropy value of each spectrum intensity is calculated by an existing entropy calculation formula (such as Shannon entropy), and the entropy value obtained is used as the spectrum complexity of the frequency component corresponding to the spectrum intensity, and the spectrum complexity of each frequency component is used as the dynamic interactivity of the cranial nerves in the brain area where the corresponding frequency component can be detected in the brain, thereby obtaining the dynamic interactivity of the cranial nerves in different areas of the target user's brain.

[0080] In some embodiments, determining the neural response values ​​of various regions in the brain of the target user when the target user is experiencing the cultural and creative industry according to all dynamic interaction degrees and the spatial dynamic entropy can be achieved by using the following steps:

[0081] Determine the activity interaction map of the target user's brain area based on all dynamic interactions;

[0082] The neural response values ​​of various regions in the brain of the target user when the target user is experiencing cultural and creative activities are determined through the activity interaction graph and the spatial dynamic entropy.

[0083] It should be noted that, in the present application, the neural response value refers to the intensity of neuronal activity in the brain, and the neural response value reflects the activity level of neurons within a specific time; the activity interaction map is a map that records the dynamic interaction of brain nerves in different areas of the target user's brain; in specific implementation, first, all dynamic interactions can be mapped to the corresponding brain areas in the brain to obtain a two-dimensional image, and the two-dimensional image is used as an activity interaction map reflecting the activity complexity of different areas of the target user's brain; then, for each pair of electrodes in the perception electroencephalogram, the overlapping area of ​​the two electrodes' monitoring range corresponds to a brain area in the activity interaction map, and the entropy values ​​corresponding to the two electrodes in the spatial dynamic entropy are mapped to the corresponding brain area in the activity interaction map, and then the product of the spectrum complexity in the brain area and the entropy value is used as the neural response value of the brain area. The neural response value of the brain area corresponding to each pair of electrodes in the perception electroencephalogram can be obtained in the above manner, and then the neural response value of each area inside the brain of the target user when performing cultural and creative experience can be obtained.

[0084] In step 105, a feedback adjustment sequence for a target user to use a virtual reality device for a cultural and creative experience is determined based on the motion timing EEG characteristics and all neural response values, and then feedback regulation is performed on the target user's cultural and creative experience based on the feedback adjustment sequence.

[0085] In some embodiments, determining the feedback adjustment sequence when the target user uses the virtual reality device for cultural and creative experience based on the motion timing EEG characteristics and all neural response values ​​can be implemented by the following steps:

[0086] Performing motion synchronization detection on the motion time-series EEG characteristics to obtain long-range motion characteristics of the target user when using the virtual reality device for cultural and creative experience;

[0087] Determining an initial feedback sequence of each motion synchronization coefficient in the long-range motion feature according to all neural response values;

[0088] Based on all the initial feedback sequences, the feedback adjustment sequence when the target users use virtual reality devices for cultural and creative experience is determined.

[0089] It should be noted that, in the present application, the feedback adjustment sequence is a series of feedback instructions generated after analyzing the EEG data generated when the user uses the virtual reality device for the cultural and creative experience, which is used to adjust and control the virtual reality environment in real time to enhance the user's immersive experience; the initial feedback sequence is a feature that describes the activity feedback amount of the motor nerve source inside the brain under different motion synchronization coefficients; in specific implementation, first, the motion timing EEG characteristics are subjected to motion synchronization detection to obtain the long-range motion characteristics when the target user uses the virtual reality device for the cultural and creative experience, and then, the existing standardization technology (such as Z-Score standardization) can be used to standardize all the motion synchronization coefficients and all the neural response values ​​in the long-range motion characteristics, and A motion synchronization coefficient is selected from the long-range motion feature as the selected motion synchronization coefficient, and the average value between the selected motion synchronization coefficient and the neural response value of each area inside the brain is used as the activity feedback amount of the motor nerve source in the corresponding area, thereby obtaining the activity feedback amount of each motor nerve source, and the sequence composed of all activity feedback amounts is used as the initial feedback sequence of the selected motion synchronization coefficient, thereby obtaining the initial feedback sequence of each motion synchronization coefficient in the long-range motion feature; finally, the time point at which the motor imagination task corresponding to each motion synchronization coefficient ends and the initial feedback sequence are combined into a feedback vector, and then the collection of all feedback vectors is used as the feedback adjustment sequence when the target user uses the virtual reality device for cultural and creative experience.

[0090] In some embodiments, the following steps may be used to perform motion synchronization detection on the motion time sequence EEG characteristics to obtain the long-range motion characteristics of the target user when using the virtual reality device for cultural and creative experience:

[0091] Extracting movement-related potentials from the movement time-series EEG features;

[0092] Analyzing the movement synchronization of the brain electrical activity at different time periods when the target user performs movement imagination according to the movement-related potential, and obtaining the movement synchronization coefficient of the brain electrical activity at different time periods;

[0093] Based on all motion synchronization coefficients, the long-range motion characteristics of the target users when using virtual reality devices for cultural and creative experiences are determined.

[0094] It should be noted that in the present application, long-range motion characteristics refer to the electrical activity characteristics that reflect the target user's brain motor function and motor imagination process when the target user uses virtual reality equipment for cultural and creative experience, and can be used to analyze and provide feedback on the target user's neural activity during motor imagination; the motion synchronization coefficient is a parameter that represents the power changes in the target user's brain electrical activity at different times when the target user is performing motor imagination.

[0095] In the specific implementation, first, according to the start time and end time of each motor imagery task in a series of motor imagery tasks set in the above steps, the motor time series EEG characteristics are segmented to obtain multiple EEG segments, each EEG segment corresponds to a motor imagery task, and the EEG waves corresponding to the time period between every two motor imagery tasks are used as baseline segments, and the average power of all baseline segments is calculated using existing spectrum analysis methods (such as short-time Fourier transform), and the obtained average power is used as the baseline frequency band power, and then the start time of a motor imagery task (such as the first motor imagery task) is selected as the time zero point, each EEG segment is aligned, and the potential values ​​of all time points of the EEG segment are averaged. value, and obtain the motion-related potential of the EEG segment. The motion-related potential of all EEG segments can be obtained in the above manner. Then, the average power of each EEG segment is calculated using existing spectrum analysis methods (such as short-time Fourier transform), and the obtained power is used as the frequency band power of the EEG segment. The ratio of the difference between the frequency band power and the baseline frequency band power and the motion-related potential of the EEG segment can be used as the motion synchronization coefficient of the frequency band. The motion synchronization coefficient of the brain's EEG activity at different time periods can be obtained in the above manner. Finally, the sequence composed of the motion synchronization coefficients at each time period in chronological order is used as the long-range motion characteristics of the target user when using virtual reality equipment for cultural and creative experience.

[0096] Preferably, in some embodiments, feedback control of the target user's cultural and creative experience based on the feedback adjustment sequence can be achieved in the following manner, namely: when the target user performs a motor imagery task, first determine the corresponding initial feedback sequence in the feedback adjustment sequence according to the time point when the motor imagery task ends, and then control the virtual reality environment to provide feedback to the target user according to the activity feedback amount of each motor nerve source in the brain in the initial feedback sequence. For example, for an initial feedback sequence corresponding to a motor imagery task, if the activity feedback amount of the supplementary motor area in the brain is 0.8 and the activity feedback amount of the premotor area is 0.3, then adjust the movement speed of the virtual object corresponding to the target user in the virtual reality.

[0097] In addition, in another aspect of the present application, in some embodiments, the present application provides a cultural and creative experience system based on virtual reality technology, referring to Figure 4 , which is a schematic diagram of exemplary hardware and / or software of a cultural and creative experience system based on virtual reality technology according to some embodiments of the present application. The cultural and creative experience system 400 based on virtual reality technology includes: a collection module 401, a processing module 402 and an execution module 403, which are described as follows:

[0098] Collection module 401, in this application, collection module 401 is mainly used to collect the target user's motor imagination brain waves when using the virtual reality cultural and creative experience device;

[0099] Processing module 402, in the present application, processing module 402 is mainly used to perform multi-scale convolution on the motor imagery EEG to obtain the spatiotemporal convolution features of the normal EEG signals of the target user when using the virtual reality device for cultural and creative experience, and then convert the motor imagery EEG into motion time sequence EEG features and motion source domain EEG features based on the spatiotemporal convolution features;

[0100] It should be noted that the processing module 402 in the present application is also used to determine the spatial dynamic entropy of the brain waves of the target user when performing motor imagination according to the brain wave characteristics of the motion source domain;

[0101] In addition, it should be noted that the processing module 402 in the present application is also used to obtain the perceived electroencephalogram of the target user when using the virtual reality cultural and creative experience device, determine the dynamic interaction degree of the cranial nerves in different regions of the target user's brain through the perceived electroencephalogram, and then determine the neural response value of each region inside the brain of the target user when performing the cultural and creative experience according to all the dynamic interaction degrees and the spatial dynamic entropy;

[0102] Execution module 403. In the present application, execution module 403 is mainly used to determine the feedback adjustment sequence when the target user uses the virtual reality device for cultural and creative experience based on the motion timing EEG characteristics and all neural response values, and then to feedback and regulate the cultural and creative experience of the target user based on the feedback adjustment sequence.

[0103] In addition, the present application also provides a computer device, which includes a memory and a processor, the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned cultural and creative experience control method based on virtual reality technology.

[0104] In some embodiments, reference Figure 5 , which is a schematic diagram of the structure of a computer device for implementing a cultural and creative experience control method based on virtual reality technology according to some embodiments of the present application. The cultural and creative experience control method based on virtual reality technology in the above embodiment can be Figure 5 The computer device 500 shown in the figure is implemented, and the computer device 500 includes at least one processor 501, a communication bus 502, a memory 503 and at least one communication interface 504.

[0105] The processor 501 may be a general-purpose central processing unit (CPU) or an application specific integrated circuit (ASIC).

[0106] The communication bus 502 may be used to transmit information between the above-mentioned components.

[0107] The memory 503 may be a read only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read only memory (EEPROM), a compact disc read only memory (CD ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 503 may exist independently and be connected to the processor 501 via the communication bus 502. The memory 503 may also be integrated with the processor 501.

[0108] Among them, the memory 503 is used to store the program code for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the program code stored in the memory 503. The program code may include one or more software modules. The cultural and creative experience control method based on virtual reality technology in the above embodiment can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.

[0109] The communication interface 504 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0110] In a specific implementation, as an embodiment, a computer device may include multiple processors, each of which may be a single-core (single CPU) processor or a multi-core (multi CPU) processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0111] The above-mentioned computer device may be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device may be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer device.

[0112] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned cultural and creative experience control method based on virtual reality technology.

[0113] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0114] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A cultural and creative experience control method based on virtual reality technology, characterized in that: The steps include: Collect the target user's motor imagination brain waves when using virtual reality cultural and creative experience equipment; Performing multi-scale convolution on the motor imagery EEG to obtain the spatiotemporal convolution features of normal EEG signals of the target user when using the virtual reality device for cultural and creative experience, and then converting the motor imagery EEG into motion time sequence EEG features and motion source domain EEG features based on the spatiotemporal convolution features; Determining the spatial dynamic entropy of the brain waves of the target user when performing motor imagery according to the motor source domain electroencephalogram characteristics; Acquire a perceptual electroencephalogram of the target user when using the virtual reality cultural and creative experience device, determine the dynamic interaction degree of brain nerves in different regions of the target user's brain through the perceptual electroencephalogram, and then determine the neural response value of each region inside the brain of the target user when performing the cultural and creative experience according to all the dynamic interaction degrees and the spatial dynamic entropy; A feedback adjustment sequence is determined when a target user uses a virtual reality device for a cultural and creative experience based on the motion timing EEG characteristics and all neural response values, and then feedback regulation is performed on the cultural and creative experience of the target user based on the feedback adjustment sequence.

2. The method according to claim 1, characterized in that The motor imagery brainwaves are subjected to multi-scale convolution to obtain the spatiotemporal convolution features of normal brainwave signals of the target user when using the virtual reality device for cultural and creative experience, which specifically include: Determining a two-dimensional convolution domain when performing multi-scale convolution on the motor imagery brain waves; Convolving the two-dimensional convolution domain through a convolutional neural network to obtain the spatial characteristics of the motor imagery brain wave; Extracting the temporal features of the motor imagery brain waves from the two-dimensional convolutional domain based on a long short-term memory network; The spatiotemporal convolution features of the normal brain wave signals of the target user when using the virtual reality device for cultural and creative experience are determined according to the spatial features and the temporal features.

3. The method according to claim 1, characterized in that The steps of converting the motor imagery EEG waves into motion time sequence EEG features and motion source domain EEG features based on the spatiotemporal convolution features specifically include: Determining a motion time-series EEG feature according to the time-series feature in the spatiotemporal convolution feature; The motion source domain EEG features are determined based on the spatial features in the spatiotemporal convolution features.

4. The method according to claim 1, characterized in that Determining the spatial dynamic entropy of the brain waves of the target user when performing motor imagination according to the motor source domain electroencephalogram features specifically includes: Determine the spatial distribution domain of the brain wave signals at different electrode positions in the virtual reality device according to the brain wave characteristics of the motion source domain; determining a spatial relative fluctuation between electrodes according to the spatial distribution domain; The spatial dynamic entropy of the brain waves of the target user when performing motor imagery is determined based on all spatial relative fluctuations.

5. The method according to claim 1, characterized in that Determining the dynamic interaction degree of cranial nerves in different regions of the target user's brain through the perceived electroencephalogram specifically includes: Determining the spectrum intensity of the perceptual brain wave signal in the perceptual electroencephalogram at each frequency component; The dynamic interaction degree of the cranial nerves in different areas of the target user's brain is determined based on all spectrum intensities.

6. The method according to claim 1, characterized in that Determining the feedback adjustment sequence when the target user uses the virtual reality device for cultural and creative experience based on the motion timing EEG characteristics and all neural response values ​​specifically includes: Performing motion synchronization detection on the motion time-series EEG characteristics to obtain long-range motion characteristics of the target user when using the virtual reality device for cultural and creative experience; Determining an initial feedback sequence of each motion synchronization coefficient in the long-range motion feature according to all neural response values; Based on all the initial feedback sequences, the feedback adjustment sequence when the target users use virtual reality devices for cultural and creative experience is determined.

7. The method according to claim 6, characterized in that The motion synchronization detection is performed on the motion time sequence EEG characteristics to obtain the long-range motion characteristics of the target user when using the virtual reality device for cultural and creative experience, which specifically include: Extracting movement-related potentials from the movement time-series EEG features; Analyzing the movement synchronization of the brain electrical activity at different time periods when the target user performs movement imagination according to the movement-related potential, and obtaining the movement synchronization coefficient of the brain electrical activity at different time periods; Based on all motion synchronization coefficients, the long-range motion characteristics of the target users when using virtual reality devices for cultural and creative experiences are determined.

8. A cultural and creative experience system based on virtual reality technology, comprising: The acquisition module is used to collect the target user's motor imagination brain waves when using the virtual reality cultural and creative experience equipment; A processing module, configured to perform multi-scale convolution on the motor imagery EEG to obtain the spatiotemporal convolution features of normal EEG signals of the target user when the target user uses the virtual reality device for cultural and creative experience, and then convert the motor imagery EEG into motion time sequence EEG features and motion source domain EEG features based on the spatiotemporal convolution features; The processing module is also used to determine the spatial dynamic entropy of the brain waves of the target user when performing motor imagination according to the brain wave characteristics of the motion source domain; The processing module is also used to obtain a perceptual electroencephalogram of the target user when using the virtual reality cultural and creative experience device, determine the dynamic interaction degree of brain nerves in different regions of the target user's brain through the perceptual electroencephalogram, and then determine the neural response value of each region inside the brain of the target user when performing the cultural and creative experience according to all the dynamic interaction degrees and the spatial dynamic entropy; The execution module is used to determine the feedback adjustment sequence when the target user uses the virtual reality device for cultural and creative experience according to the motion timing EEG characteristics and all neural response values, and then perform feedback regulation on the cultural and creative experience of the target user based on the feedback adjustment sequence.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the cultural and creative experience control method based on virtual reality technology described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the steps of the cultural and creative experience control method based on virtual reality technology as described in any one of claims 1 to 7 are implemented.

Images