Virtual reality system, display mode control method, and electronic device

Abstract

The embodiment of the application provides a virtual reality system, a display mode control method and an electronic device, which comprise a master control module, an electroencephalogram signal acquisition module and a display module; the display module is used for displaying image data according to a current display mode; the electroencephalogram signal acquisition module is used for acquiring an electroencephalogram signal of a human brain in response to the image data displayed by the display module, and sending the electroencephalogram signal to the master control module; the master control module is used for determining a target display mode corresponding to the electroencephalogram signal, and sending the target display mode and to-be-displayed image data to the display module; and the display module is used for displaying the to-be-displayed image data according to the target display mode. After the master control module determines the target display mode corresponding to the electroencephalogram signal, the display module adjusts the to-be-displayed image data according to the target display mode, and displays the adjusted to-be-displayed image data, so that the display mode can be adaptively switched according to the actual VR experience demand of a user, and the VR visual experience of the user is improved.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to a virtual reality system, a display mode control method, and an electronic device. Background Technology

[0002] In virtual reality (VR) technology, 3D scene images are typically displayed using head-mounted VR displays to enhance the human-computer interaction experience. However, current VR devices have fixed display resolutions and refresh rates, making it impossible to adaptively switch display modes according to the user's actual VR experience needs, thus reducing the user's VR experience. Summary of the Invention

[0003] The purpose of this application is to provide a virtual reality system, a display mode control method, and an electronic device to improve the user's VR experience. The specific technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a virtual reality system, the system comprising:

[0005] Main control module, EEG signal acquisition module and display module;

[0006] The display module is used to display image data according to the current display mode;

[0007] The EEG signal acquisition module is used to acquire the EEG signals of the human brain in response to the image data displayed by the display module, and send the EEG signals to the main control module;

[0008] The main control module is used to determine the target display mode corresponding to the EEG signal and send the target display mode and the image data to be displayed to the display module;

[0009] The display module is further configured to display the image data to be displayed according to the target display mode.

[0010] In one possible implementation, the EEG signal acquisition module includes multiple acquisition electrodes, a signal amplifier, and an analog-to-digital converter;

[0011] The acquisition electrodes are used to acquire photoelectric signals from the human brain and send the photoelectric signals to the signal amplifier.

[0012] The signal amplifier is used to receive the photoelectric signal, filter and amplify the photoelectric signal to obtain a corrected photoelectric signal, and send the corrected photoelectric signal to the analog-to-digital converter.

[0013] The analog-to-digital converter is used to receive the amplified photoelectric signal, convert the amplified photoelectric signal into a digital form of EEG signal, and send the EEG signal to the main control module.

[0014] In one possible implementation, the EEG signal acquisition module includes 4-10 acquisition electrodes.

[0015] In one possible implementation, the main control module includes a central processing unit and a graphics processing unit; the display module includes a display control chip and a display screen.

[0016] The central processing unit is used to acquire the electroencephalogram (EEG) signal; extract features from the EEG signal to obtain EEG features; determine the target display mode corresponding to the EEG features; and configure the target display mode to the display control chip.

[0017] The graphics processor is used to preprocess image data to obtain image data to be displayed; and to send the image data to be displayed to the display control chip.

[0018] The display control chip is used to render the image data to be displayed according to the target display mode and display it on the display screen.

[0019] In one possible implementation, the EEG signal includes N channels of brain electronic signals, each corresponding to a acquisition electrode;

[0020] The central processing unit is specifically used to calculate the distance between data points with adjacent acquisition times in each EEG signal; calculate the distance between data points with the same acquisition time in each EEG signal; obtain a distance threshold; connect data points in the EEG signal whose distance is less than the distance threshold to obtain brain network data; and calculate the average number of network paths in the brain network data as EEG features.

[0021] In one possible implementation, the i-th brain electrical signal is represented as {x} i，1 x i，2 x i，3 , ..., x i，n}, where n is the number of data in the i-th brain electronic signal;

[0022] The central processing unit is specifically used to calculate the distance between adjacent data points in the i-th brain electronic signal using the following formula:

[0023]

[0024] in, Let x represent the Euclidean distance between the (j+1)th data point and the jth data point in the i-th brain electrical signal. i,j+1 Let x represent the (j+1)th data point in the i-th brain electrical signal. i,j This represents the j-th data point in the i-th brain electrical signal.

[0025] In one possible implementation, the central processing unit is specifically configured to acquire multiple data queues of a preset length from each of the brain signals using a sliding window of a preset length, wherein at least one data queue is acquired from each of the brain signals; calculate the energy value of each of the data queues respectively; and select the minimum energy value as a distance threshold.

[0026] In one possible implementation, the central processing unit is specifically configured to: calculate the energy value of each data queue using the following formula:

[0027]

[0028] Where E represents the energy value of the data queue, X i X represents the i-th data in the data queue. j This represents the j-th data item in the data queue.

[0029] In one possible implementation, the central processing unit is specifically configured to: calculate the average number of network paths as an EEG feature for the brain network data using the following formula:

[0030]

[0031] Where L represents the average number of paths in the network, M represents the number of data points in the brain network data, and D... ij This represents the number of edges on the shortest path between two data points.

[0032] Secondly, embodiments of this application provide a display mode control method, the method comprising:

[0033] Acquire the image data to be displayed and the brain signals of the human brain in response to the image data displayed by the display module;

[0034] Determine the target display mode corresponding to the electroencephalogram (EEG) signal;

[0035] The target display mode and the image data to be displayed are sent to the display module so that the display module displays the image data to be displayed according to the target display mode.

[0036] In one possible implementation, determining the target display pattern corresponding to the EEG signal includes:

[0037] Feature extraction is performed on the electroencephalogram (EEG) signals to obtain EEG features;

[0038] The target display mode corresponding to the EEG feature is determined according to the first correspondence between the pre-determined EEG feature and the display mode.

[0039] In one possible implementation, the EEG signal includes N channels of brain electronic signals, each corresponding to a acquisition electrode;

[0040] The step of extracting features from the electroencephalogram (EEG) signal to obtain EEG features includes:

[0041] For each brain electronic signal, calculate the distance between data points with adjacent acquisition times in that brain electronic signal channel;

[0042] Calculate the distance between data collected at the same time from each of the various brain electronic signals;

[0043] A distance threshold is obtained, and data in the EEG signals whose distance is less than the distance threshold are connected to obtain brain network data;

[0044] The average number of network paths in the brain network data is calculated as an EEG feature.

[0045] In one possible implementation, the i-th brain electrical signal is represented as {x} i，1 x i，2 x i，3 , ..., x i，n}, where n is the number of data in the i-th brain electronic signal;

[0046] For each brain signal, calculating the distance between data points with adjacent acquisition times in that brain signal channel includes:

[0047] For the i-th brain electronic signal, the distance between data points with adjacent acquisition times in the i-th brain electronic signal is calculated using the following formula:

[0048]

[0049] in, Let x represent the Euclidean distance between the (j+1)th data point and the jth data point in the i-th brain electrical signal. i,j+1 Let x represent the (j+1)th data point in the i-th brain electrical signal. i,j This represents the j-th data point in the i-th brain electrical signal.

[0050] In one possible implementation, obtaining the distance threshold includes:

[0051] Using a sliding window of a preset length, multiple data queues of a preset length are obtained from each of the brain electronic signals, wherein at least one data queue is obtained from each of the brain electronic signals;

[0052] Calculate the energy value of each of the data queues;

[0053] The minimum energy value is selected as the distance threshold.

[0054] In one possible implementation, calculating the energy value of each of the data queues includes:

[0055] For each data queue, the energy value of that data queue is calculated using the following formula:

[0056]

[0057] Where E represents the energy value of the data queue, X i X represents the i-th data in the data queue. j This represents the j-th data item in the data queue.

[0058] In one possible implementation, calculating the average number of network paths in the brain network data as an EEG feature includes:

[0059] For the aforementioned brain network data, the average number of network paths is calculated as an EEG feature using the following formula:

[0060]

[0061] Where L represents the average number of paths in the network, M represents the number of data points in the brain network data, and D... ij This represents the number of edges on the shortest path between two data points.

[0062] Thirdly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0063] Memory, used to store computer programs;

[0064] When a processor executes a program stored in memory, it implements any of the steps described in the second aspect above.

[0065] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the steps described in the second aspect above.

[0066] Beneficial effects of the embodiments in this application:

[0067] This application provides a virtual reality system, a display mode control method, and an electronic device. The virtual reality system includes a main control module, an electroencephalogram (EEG) signal acquisition module, and a display module. The display module is used to display image data according to the current display mode. The EEG signal acquisition module is used to acquire the EEG signals of the human brain in response to the image data displayed by the display module and send the EEG signals to the main control module. The main control module is used to determine the target display mode corresponding to the EEG signals and send the target display mode and the image data to be displayed to the display module. The display module is used to display the image data to be displayed according to the target display mode. After the main control module determines the corresponding target display mode based on the EEG signals, the display module adjusts the image data to be displayed according to the target display mode and displays the adjusted image data. Through the above-mentioned virtual reality (VR) system, the display mode can be adaptively switched according to the user's actual VR experience needs, improving the user's VR visual experience.

[0068] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0070] Figure 1 This is a schematic diagram of a first structure of a virtual reality system provided in an embodiment of this application;

[0071] Figure 2a This is a schematic diagram of a second structure of a virtual reality system provided in an embodiment of this application;

[0072] Figure 2b This diagram shows the names of the acquisition electrodes and their contact positions with the head.

[0073] Figure 3a This is a schematic diagram of a third structure of a virtual reality system provided in an embodiment of this application;

[0074] Figure 3b This is a schematic diagram of a fourth structure of a virtual reality system provided in an embodiment of this application;

[0075] Figure 4 This is a schematic diagram of a simple network containing 5 nodes and 5 edges / connections;

[0076] Figure 5 A flowchart illustrating a display mode control method provided in an embodiment of this application;

[0077] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0078] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0079] With the development of science and technology, virtual reality (VR) technology has matured and gained increasing popularity. Traditional VR technology typically uses head-mounted displays to show 3D scenes, enhancing the human-computer interaction experience. The fundamental principle is that the user's left and right eyes view two nearly identical images, differing only slightly in angle. These images are then processed by the user's brain to create a sense of depth, providing a virtual, three-dimensional environment. However, current VR devices have fixed display resolutions and refresh rates (fixed display module specifications). Therefore, in practical use, they cannot adaptively switch display modes according to the user's actual VR experience needs. For example, in interactive environments such as virtual games that require a realistic immersive experience, the system's resolution and refresh rate cannot be automatically adjusted based on the user's current VR experience. Furthermore, when users experience fatigue after prolonged viewing, the resolution cannot be adjusted to prevent the displayed image from becoming overly sharp.

[0080] To address the aforementioned issues, related technologies have provided a display mode switching method. However, this method only allows switching between the display modes of a first and second screen arranged side-by-side. Specifically, it involves obtaining a display mode switching command and switching the displayed image information of the first and second screens according to the display mode instruction. It cannot adaptively switch display modes based on the user's actual VR experience needs.

[0081] In order to adaptively switch the display mode according to the user's actual VR experience needs and improve the user's VR experience, this application provides a virtual reality system and a display mode control method.

[0082] First, a virtual reality system provided in the embodiments of this application will be described in detail, see [link to relevant documentation]. Figure 1 The virtual reality system includes:

[0083] Main control module 1, EEG signal acquisition module 2, and display module 3;

[0084] The display module 3 is used to display image data according to the current display mode;

[0085] The EEG signal acquisition module 2 is used to acquire the EEG signal of the human brain in response to the image data displayed by the display module, and send the EEG signal to the main control module 1;

[0086] The main control module 1 is used to determine the target display mode corresponding to the EEG signal and send the target display mode and the image data to be displayed to the display module 3.

[0087] The display module 3 is also used to display the image data to be displayed according to the target display mode.

[0088] During the operation of the virtual reality system, the display module displays image data according to the current display mode. This current display mode can be the preset display mode in the initial state of the display module, or it can be the display mode sent to the display module by the main control module; both are within the scope of protection of this application.

[0089] As a user views the images displayed on the screen, their brain responds by generating corresponding signals. These signals are acquired and preprocessed by an electroencephalogram (EEG) acquisition module to obtain the brain's electroencephalogram (EEG). EEG signals are obtained by amplifying and recording the spontaneous bioelectrical potentials of the cerebral cortex through devices applied to the scalp. They represent the spontaneous, rhythmic electrical activity of brain cell groups recorded by electrodes. EEG acquisition is a non-invasive method for recording cortical electrical signals and is a commonly used method for detecting brain activity.

[0090] The target display mode corresponding to EEG signals refers to the real-time acquisition of EEG signals generated by the user in the virtual reality (VR) environment, and the determination of the user's real-time VR experience needs based on the changing trends of the EEG signals within a certain time range. For example, when the user's EEG signals indicate that the currently viewed video is not clear enough, the target display mode should be high display resolution and high display refresh rate to meet the user's real-time need for a high-definition visual experience.

[0091] After receiving the target display mode and the image data to be displayed, the display module adjusts the image data according to the target display mode and then displays the adjusted image data. In one example, when the user's EEG signal indicates that the currently viewed video is not clear enough, the target display mode can be a high display resolution and a high refresh rate. The display module increases the display resolution and refresh rate of the image data to be displayed according to this target display mode to meet the user's real-time demand for a high-definition visual experience. In another example, when the user's EEG signal indicates that the currently viewed video is choppy but the clarity is just right, it indicates that the video decoding or rendering resources may be insufficient. In this case, the target display mode can be to maintain the clarity while reducing the video refresh rate. The display module maintains the resolution of the image data to be displayed unchanged while reducing the refresh rate to reduce video choppyness.

[0092] The image data to be displayed can be either compressed or the original image data, depending on the storage size of the image data. In one example, the image data to be displayed is an RGB (three primary colors) 24-bit image, with each pixel occupying three bytes and each byte being 8 bits (binary digit). The image data needs to be compressed before being sent to the display module.

[0093] In this embodiment, after the main control module determines the corresponding target display mode based on the EEG signal, the display module adjusts the image data to be displayed according to the target display mode and displays the adjusted image data. Therefore, the display mode can be adaptively switched according to the user's actual VR experience needs, which improves the user's VR visual experience.

[0094] In one possible implementation, see Figure 2a The EEG signal acquisition module 2 includes multiple acquisition electrodes 21, a signal amplifier 22, and an analog-to-digital converter 23;

[0095] The acquisition electrode 21 is used to acquire photoelectric signals from the human brain and send the photoelectric signals to the signal amplifier 22;

[0096] The signal amplifier 22 is used to receive the photoelectric signal, filter and amplify the photoelectric signal to obtain a corrected photoelectric signal, and send the corrected photoelectric signal to the analog-to-digital converter 23.

[0097] The analog-to-digital converter 23 is used to receive the amplified photoelectric signal, convert the amplified photoelectric signal into a digital form of EEG signal, and send the EEG signal to the main control module 1.

[0098] When users interact with VR scenes, event-related potentials (ERPs) are easily generated. ERPs are a special type of brain evoked potential, generated by intentionally assigning specific psychological meaning to stimuli, utilizing multiple or diverse stimuli to produce brain potentials. They reflect the neurophysiological changes in the brain during cognitive processes and are also known as cognitive potentials. Specifically, they refer to brain potentials recorded from the scalp surface when a user cognitively processes a VR scene. Acquisition electrodes obtain ERPs in real time by contacting the user's scalp. The naming of the acquisition electrodes and their corresponding positions on the head can be determined according to the "International Standard Lead" method, such as... Figure 2b As shown, the face is facing upwards. The location of the acquisition electrodes should be determined based on the measurement of skull landmarks, and should be as proportional as possible to the size and shape of the skull. The standard location of the acquisition electrodes should be appropriately distributed across all parts of the skull. The names of the acquisition electrode locations should be combined with brain regions (frontal region, temporal region, parietal region, occipital region). Electrodes are labeled with international Arabic numerals: odd numbers for the left hemisphere and even numbers for the right hemisphere. Zero represents the center of the skull, A1 and A2 represent the left and right earlobes, smaller numbers are used closer to the midline, and larger numbers are used further out. The naming and location correspondence of acquisition electrodes is shown in Table 1 below.

[0099] Table 1

[0100] forehead pre frontal lobe FP1, FP2 Lateral forehead Inferior frontal lobe F7, F8 Forehead area frontal lobe F3, F4, FZ central central lobe C3, C4, CZ Temporal region Tempal lobe T3, T4 Posttemporal posterior temperal lobe T5, T6 Top area parietal lobe P3, P4, PZ pillow area occipital lobe O1, O2 Ear auricular A1, A2

[0101] Before collecting EEG signals, the user needs to wear a virtual reality (VR) headset and fix the acquisition electrodes in the corresponding positions on the head, ensuring good contact and the ability to obtain photoelectric signals. After preparation, the user can start the VR system normally and monitor the electrode contact in the professional photoelectric signal acquisition application interface. Good electrode contact ensures that the EEG signal acquisition module can collect relatively clear EEG signals in real time.

[0102] In one possible implementation, the EEG signal acquisition module 2 includes 4-10 acquisition electrodes.

[0103] Insufficient electrode collection leads to insufficient EEG signal data, resulting in unconvincing assessments of the user's actual VR experience needs; conversely, excessive electrode collection leads to excessive EEG signal data, increasing computational load and overburdening the main control module, causing system latency. Therefore, according to... Figure 2b As shown, 4-10 acquisition electrodes are installed at the corresponding positions on the headband directly above the head-mounted device (used to collect photoelectric signals from the human brain). In one example, the number of acquisition electrodes can be 4. Figure 2aThe diagram illustrates four acquisition electrodes (first acquisition electrode 211, second acquisition electrode 212, third acquisition electrode 213, and fourth acquisition electrode 214). These electrodes are connected to a signal amplifier via adapter cables. The signal amplifier filters and amplifies the acquired photoelectric signals, then sends the corrected signals to an analog-to-digital converter (ADC). The ADC converts the corrected photoelectric signals into digital EEG signals and sends these signals to the main control module.

[0104] In this embodiment, digital EEG signals are acquired using acquisition electrodes, signal amplifiers, and analog-to-digital converters, and then sent to the main control module for processing.

[0105] In one possible implementation, see Figure 3a The main control module 1 includes a CPU (central processing unit) 11 and a GPU (graphics processing unit) 12; the display module 3 includes a display control chip 31 and a display screen 32.

[0106] The CPU 11 is used to acquire the EEG signal; extract features from the EEG signal to obtain EEG features; determine the target display mode corresponding to the EEG features; and configure the target display mode to the display control chip 31.

[0107] The GPU 12 is used to preprocess the image data to obtain the image data to be displayed; and to send the image data to be displayed to the display control chip 31.

[0108] The display control chip 31 is used to render the image data to be displayed according to the target display mode and display it on the display screen 32.

[0109] The main control module is used for the control and signal data calculation of the entire system, including the display module driver, the EEG signal acquisition module driver, EEG signal processing, and 3D scene rendering. The main control module includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and memory, with the memory used to store the image data. To ensure compatibility with a wider range of applications and to guarantee system computing speed and stability, the CPU in this application can use a high-performance Android processor. In one example, the Android processor can be a Qualcomm 865 processor or a Qualcomm XR2 processor, etc. This application does not make any specific limitations on this.

[0110] Features of EEG signals can be extracted by constructing brain network models. In one example, the average path length of the brain network at each stage can be used as the EEG feature of that stage. The path length is defined as the average distance between any two nodes in the brain network, which is used to characterize the global connectivity features. For example, the EEG feature of a certain stage can be 0.25-0.5.

[0111] In one example, when the EEG characteristic is between 0.25 and 0.5, the corresponding target display mode is 2160*1600p, 90Hz. That is, when the user's EEG characteristic is between 0.25 and 0.5, the target display mode should be a display resolution of 2160*1600p and a display refresh rate of 90Hz to meet the user's current visual experience needs.

[0112] In one possible implementation, the EEG features obtained by constructing a brain network model can be matched with the register values ​​corresponding to the display mode. That is, after acquiring and extracting EEG signals to obtain EEG features, the CPU identifies these features and writes the corresponding register values ​​into the registers of the display control chip according to the target display mode corresponding to those features. Figure 3b As shown in Table 2, the display control chip supports multiple display modes for various application scenarios. The chip pre-programs a list of display modes corresponding to different register values, setting different display resolutions and refresh rates based on these register values. By establishing a correspondence between EEG characteristics, display modes, and register values, the display mode can be adaptively adjusted according to the user's actual VR experience needs.

[0113] Table 2

[0114] 0～0.25 3840*2160p, 75Hz flag=0 0.25～0.5 2160*1600p, 90Hz flag=1 0.5～0.75 1920*1080p, 60Hz flag=2 0.75～1 1280*720p, 60Hz flag=3 …… …… ……

[0115] It should be noted that Table 2 only illustrates the correspondence between EEG characteristics, display patterns, and register values, and does not show all the correspondences.

[0116] In the virtual reality (VR) system of this application, frequent adaptive adjustments to the display mode would cause significant wear and tear on the main control module. Therefore, the display control chip can render the image data to be displayed. Rendering is the process of projecting a model in a 3D scene into a 2D digital image according to predefined environment, lighting, materials, and rendering parameters. When the register value corresponding to the target display mode is written into the register, the display control chip renders the image to be displayed according to the target display mode and displays it on the screen. The display control chip decompresses the received image data to be displayed and outputs the image to be displayed according to the display mode corresponding to the register value. The specific steps are as follows:

[0117] (1) The main control module can compress the image data to be displayed and then send it to the display control chip for decompression;

[0118] (2) After the display control chip decompresses the image data to be displayed using DSC (Display Stream Compression, a commonly used display compression technology), it performs information line processing, mainly reading the register values ​​written in the registers;

[0119] (3) The display control chip will have a list of display modes corresponding to different register values ​​pre-written in the chip. After reading different register values, it will set different display resolutions and display refresh rates.

[0120] (4) The display control chip adjusts the display resolution and refresh rate of the image data to be displayed according to the target display mode, and finally outputs the adjusted image data to be displayed on the display screen.

[0121] It is important to note that the registers corresponding to the display control chip have a memory function. If the register values ​​read from two consecutive reads are the same, it is assumed that the virtual reality (VR) system does not need to switch display modes, and can directly output the data after decompression, thus saving time.

[0122] See Figure 3b The EEG signal acquisition module can communicate with the CPU in the main control module via USB (Universal Serial Bus). In one example, the USB interface can be a Type-C interface, or other USB interfaces. The CPU in the main control module can communicate with the registers via the I2C communication interface (synchronous serial bus), and the GPU in the main control module can communicate with the display control chip via the MIPI PORT (Mobile Industry Processor Interface).

[0123] In this embodiment, after the CPU determines the corresponding target display mode based on the EEG characteristics, the display control chip adjusts the image data to be displayed according to the target display mode and displays the adjusted image data. Therefore, the display mode can be adaptively switched according to the user's actual VR experience needs, thereby improving the user's VR visual experience.

[0124] In one possible implementation, the EEG signal includes N channels of brain electronic signals, each corresponding to a acquisition electrode;

[0125] Specifically, the CPU is used to calculate the distance between data points with adjacent acquisition times in each EEG signal; calculate the distance between data points with the same acquisition time in each EEG signal; obtain a distance threshold; connect data points in the EEG signal whose distance is less than the distance threshold to obtain brain network data; and calculate the average number of network paths in the brain network data as an EEG feature.

[0126] To better monitor a user's electroencephalogram (EEG) signals over a period of time in a virtual reality (VR) environment, EEG signals can be collected at preset intervals. The duration of each collection can be set according to actual conditions and data volume requirements, as can the preset interval. For example, EEG signals can be collected every hour, with each collection lasting 90 seconds. This means that EEG signal monitoring is performed every hour the user uses the VR device, calculating EEG characteristics, and adaptively adjusting the display mode based on these characteristics. The EEG signal acquisition module can have a collection frequency of 1000Hz. In one example, if 90 seconds of EEG signals are collected each time, the length of a single EEG signal can be 90,000, allowing for the construction of a brain network model based on sufficient data.

[0127] The distance threshold can be determined based on the energy value in the brain network model. If the distance threshold is too large, it will lead to too many connections and network redundancy. If the distance threshold is too small, it will result in scattered points or subnetworks. The entire brain network model can be regarded as an elastic mechanical system. The minimum energy value (minimum average difference value) in the brain network model can be calculated as the distance threshold, thereby constructing a brain network model that better reflects the user's physiological characteristics.

[0128] Brain networks can be categorized into structural brain networks (composed of anatomical connections between neural units, reflecting the physiological structure of the brain), functional networks (describing the statistical connections between nodes, and are undirected networks), and causal networks (describing the mutual influence or information flow between nodes, and are directed networks). In the field of brain network research, the study of complex brain networks is the most extensive, also known as graph-based brain network research. In graph theory, a specific network can be abstracted as a graph composed of a set of vertices and a set of edges, where an edge represents a "relationship" between two connected nodes. If the edges are undirected, the network is called an undirected network; otherwise, it is called a directed network. If each edge has different weights, the network is called a weighted network; if the edges between nodes have both direction and weight, the network is called a directed weighted network. The degree of a node is defined as the number of edges connected to that node. In a directed network, the degree of a node is divided into out-degree (the number of edges from that node to other nodes) and in-degree (the number of edges from other nodes to that node). In a weighted network, the node strength corresponds to the degree and is defined as the sum of the weights of the edges connected to that node.

[0129] In brain network research, metrics such as characteristic path length, clustering coefficient, and betweenness are commonly used to measure the overall or local characteristics of a network. The characteristic path length (average network path length / average number of network paths) is defined as the average distance between any two nodes in the network, used to characterize global connectivity. When disconnected parts exist in the network (such as isolated points), the characteristic path length will be infinite. Therefore, efficiency (defined as the reciprocal of the harmonic mean of distances between all nodes in the network) is used to characterize network connectivity. A higher efficiency value indicates a lower cost for exchanging information or energy on the network. The clustering coefficient is used to characterize local connectivity. For a node of degree k, if the actual number of edges connecting it to k connected nodes is m, then the clustering coefficient is defined as the ratio of the actual number of edges connecting it to k connected nodes to the possible number of edges connecting k nodes (C = 2m / [k(k-1)]). The clustering coefficient of the entire network is defined as the mean of the clustering coefficients of each node, used to describe the tightness of connections between nodes in the network. The importance of each node in a network can be characterized by its betweenness, which is the number of shortest paths through a node (or edge). Nodes with high degree or betweenness are called hubs. Nodes (or edges) with high betweenness are usually important for maintaining the effectiveness of communication throughout the network. The importance of a particular node (or edge) to the network can be evaluated by calculating the efficiency of the "damaged" network after removing that node (or edge).

[0130] like Figure 4 The diagram shows a simple network containing 5 nodes and 5 edges / connections. "Circles" represent nodes in the network, and "connecting lines" between two nodes represent edges / connections.

[0131] Currently, research on complex brain networks is a hot topic in neuroscience and an important branch of complex network theory. Existing studies on complex brain networks based on brain imaging techniques such as electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), and diffusion tensor imaging (DTI) have demonstrated that complex network theory is a powerful tool for analyzing brain structure and function. However, due to advancements in medical imaging techniques, most current brain network studies require the assistance of molecular biology and neurophysiology for the diagnosis of brain function and diseases. These studies require specialized and expensive equipment, making them unsuitable for the virtual reality (VR) field. Therefore, this application's embodiment maps real-time acquired EEG signal time series into a complex network. It uses multiple (channel) EEG signals and establishes connections between EEG signals in the same channel and between EEG signals in different channels at the same sampling time based on distance thresholds, forming a complex brain network model.

[0132] In this embodiment, data in the EEG signal whose distance is less than the predetermined distance threshold are connected to obtain a brain network model; based on the brain network model, EEG features are obtained by calculating the average path length of the network.

[0133] In one possible implementation, the i-th brain electrical signal is represented as {x} i，1 x i，2 x i，3 , ..., x i，n}, where n is the number of data in the i-th brain electronic signal;

[0134] Specifically, the CPU is used to calculate the distance between adjacent data points in the i-th brain electronic signal using the following formula:

[0135]

[0136] in, Let x represent the Euclidean distance between the (j+1)th data point and the jth data point in the i-th brain electrical signal. i,j+1 Let x represent the (j+1)th data point in the i-th brain electrical signal. i,j This represents the j-th data point in the i-th brain electrical signal.

[0137] In one example, for the first brain signal, the value of j+1 can range from 20,000 to 90,000 (as shown in the above example, the length of a single brain signal can be 90,000). To represent the brain network model more clearly, chaotic system theory is introduced, meaning that the distance between data points acquired at adjacent times can be measured using Euclidean distance. Euclidean distance is a commonly used definition of distance, referring to the true distance between two points in m-dimensional space, or the natural length of a vector (i.e., the distance from that point to the origin). In two-dimensional and three-dimensional space, the Euclidean distance is the actual distance between two points.

[0138] In the embodiments of this application, for each brain electronic signal, the distance between data points with adjacent acquisition times in each brain electronic signal can be calculated using a formula.

[0139] In one possible implementation, the CPU is specifically configured to acquire multiple data queues of a preset length from each of the brain signals using a sliding window of a preset length, wherein at least one data queue is acquired from each of the brain signals; calculate the energy value of each of the data queues respectively; and select the minimum energy value as a distance threshold.

[0140] The preset length of the window can be set based on the length of each EEG signal and the precision of the distance threshold. A shorter preset length allows for a more suitable distance threshold selection, but increases the computational load; a longer preset length reduces the computational load, but may result in a less suitable distance threshold selection. In one example, the length of a single EEG signal can be 90,000, and the preset length of the sliding window can be set to 800, meaning that a sliding window of length 800 is used to randomly extract data from each EEG signal.

[0141] Select the minimum energy value as the distance threshold, when When the distance is less than or equal to the distance threshold, an edge is established between j and j+1; otherwise, no edge is established.

[0142] In this embodiment, a sliding window of preset length is used to obtain multiple data queues of preset length from various brain signals, and a distance threshold is obtained by calculating the energy value of each data queue.

[0143] In one possible implementation, the CPU is specifically configured to: calculate the energy value of each data queue using the following formula:

[0144]

[0145] Where E represents the energy value of the data queue, X i X represents the i-th data in the data queue. j This represents the j-th data item in the data queue.

[0146] The energy value of each data queue is calculated using the above formula, and the minimum energy value is used as the distance threshold. After obtaining the distance threshold, the distance between data acquired at the same time is calculated for each brain signal. If the distance is less than or equal to the distance threshold, an edge is established; otherwise, no edge is established. In one example, {x 11 ,x 21 ,x 31 ,x 41} represents the data corresponding to each of the four brain electronic signals at the first acquisition time. The distance between the above data is calculated using formula (1). If the distance is less than or equal to the distance threshold, an edge is established; otherwise, no edge is established. It should be noted that no edge needs to be established between brain electronic signals at different acquisition time points and between different brain electronic signals.

[0147] In this embodiment of the application, the energy value of each data queue can be calculated using a formula.

[0148] In one possible implementation, the CPU is specifically configured to: calculate the average number of network paths as an EEG feature for the brain network data using the following formula:

[0149]

[0150] Where L represents the average number of paths in the network, M represents the number of data points in the brain network data, and D... ij This represents the number of edges on the shortest path between two data points.

[0151] By connecting data in the EEG signals that are less than a distance threshold, brain network data is obtained. The average path length of the network is then calculated to reflect the EEG characteristics of the user at that stage, as shown in formula (3).

[0152] In this embodiment of the application, the user's EEG characteristics at this stage were obtained by calculating the average path length of the network.

[0153] Based on the same concept, this application also provides a display mode control method, see [link to relevant documentation]. Figure 5 This includes the following steps:

[0154] Step S501: Acquire the image data to be displayed and the brain signals of the human brain in response to the image data displayed by the display module.

[0155] Step S502: Determine the target display mode corresponding to the EEG signal.

[0156] Step S503: Send the target display mode and the image data to be displayed to the display module so that the display module displays the image data to be displayed according to the target display mode.

[0157] In this embodiment, after determining the corresponding target display mode based on the EEG signal, the display module adjusts the image data to be displayed according to the target display mode and displays the adjusted image data. Therefore, the display mode can be adaptively switched according to the user's actual VR experience needs, thereby improving the user's VR visual experience.

[0158] In one possible implementation, determining the target display pattern corresponding to the EEG signal includes:

[0159] Feature extraction is performed on the electroencephalogram (EEG) signals to obtain EEG features;

[0160] The target display mode corresponding to the EEG feature is determined according to the first correspondence between the pre-determined EEG feature and the display mode.

[0161] In this embodiment of the application, the target display mode corresponding to the EEG feature is determined by a first correspondence between the pre-determined EEG feature and the display mode.

[0162] In one possible implementation, the EEG signal includes N channels of brain electronic signals, each corresponding to a acquisition electrode;

[0163] The step of extracting features from the electroencephalogram (EEG) signal to obtain EEG features includes:

[0164] For each brain electronic signal, calculate the distance between data points with adjacent acquisition times in that brain electronic signal channel;

[0165] Calculate the distance between data collected at the same time from each of the various brain electronic signals;

[0166] A distance threshold is obtained, and data in the EEG signals whose distance is less than the distance threshold are connected to obtain brain network data;

[0167] The average number of network paths in the brain network data is calculated as an EEG feature.

[0168] In this embodiment, data in the EEG signal whose distance is less than the predetermined distance threshold are connected to obtain a brain network model; based on the brain network model, EEG features are obtained by calculating the average path length of the network.

[0169] In one possible implementation, the i-th brain electrical signal is represented as {x} i，1 x i，2 x i，3 , ..., x i，n}, where n is the number of data in the i-th brain electronic signal;

[0170] For each brain signal, calculating the distance between data points with adjacent acquisition times in that brain signal channel includes:

[0171] For the i-th brain electronic signal, the distance between data points with adjacent acquisition times in the i-th brain electronic signal is calculated using the following formula:

[0172]

[0173] in, Let x represent the Euclidean distance between the (j+1)th data point and the jth data point in the i-th brain electrical signal. i,j+1 Let x represent the (j+1)th data point in the i-th brain electrical signal. i,j This represents the j-th data point in the i-th brain electrical signal.

[0174] In the embodiments of this application, for each brain electronic signal, the distance between data points with adjacent acquisition times in each brain electronic signal can be calculated using a formula.

[0175] In one possible implementation, obtaining the distance threshold includes:

[0176] Using a sliding window of a preset length, multiple data queues of a preset length are obtained from each of the brain electronic signals, wherein at least one data queue is obtained from each of the brain electronic signals;

[0177] Calculate the energy value of each of the data queues;

[0178] The minimum energy value is selected as the distance threshold.

[0179] In this embodiment, a sliding window of preset length is used to obtain multiple data queues of preset length from various brain signals, and a distance threshold is obtained by calculating the energy value of each data queue.

[0180] In one possible implementation, calculating the energy value of each of the data queues includes:

[0181] For each data queue, the energy value of that data queue is calculated using the following formula:

[0182]

[0183] Where E represents the energy value of the data queue, X i X represents the i-th data in the data queue. j This represents the j-th data item in the data queue.

[0184] In this embodiment of the application, the energy value of each data queue can be calculated using a formula.

[0185] In one possible implementation, calculating the average number of network paths in the brain network data as an EEG feature includes:

[0186] For the aforementioned brain network data, the average number of network paths is calculated as an EEG feature using the following formula:

[0187]

[0188] Where L represents the average number of paths in the network, M represents the number of data points in the brain network data, and D... ij This represents the number of edges on the shortest path between two data points.

[0189] In this embodiment of the application, the user's EEG characteristics at this stage were obtained by calculating the average path length of the network.

[0190] This application also provides an electronic device, such as... Figure 6As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0191] Memory 603 is used to store computer programs;

[0192] When the processor 601 executes the program stored in the memory 603, it implements any of the display mode control methods in the above embodiments.

[0193] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0194] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0195] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0196] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0197] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described display mode control methods.

[0198] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the display mode control methods described above.

[0199] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0200] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0201] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of methods, electronic devices, storage media, and computer program products are basically similar to the system embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the system embodiments.

[0202] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A virtual reality system, characterized in that, The system includes: Main control module, EEG signal acquisition module and display module; The display module is used to display image data according to the current display mode; The EEG signal acquisition module is used to acquire the EEG signals of the human brain in response to the image data displayed by the display module, and send the EEG signals to the main control module; the EEG signals include N channels of brain electronic signals, and each channel of brain electronic signal corresponds to a collection electrode; The main control module includes a central processing unit (CPU) and a graphics processing unit (GPU); the display module includes a display control chip and a display screen; the CPU is used to acquire the EEG signals; for each EEG signal, calculate the distance between data points with adjacent acquisition times in that EEG signal; calculate the distance between data points with the same acquisition time in each EEG signal; acquire a distance threshold; connect the EEG signals whose distance is less than the distance threshold to obtain brain network data; calculate the average number of network paths of the brain network data as an EEG feature; determine the target display mode corresponding to the EEG feature, and configure the target display mode to the display control chip; the GPU is used to preprocess the image data to obtain image data to be displayed; and send the image data to be displayed to the display control chip; The display control chip is used to render the image data to be displayed according to the target display mode and display it on the display screen.

2. The system according to claim 1, characterized in that, The EEG signal acquisition module includes multiple acquisition electrodes, a signal amplifier, and an analog-to-digital converter; The acquisition electrodes are used to acquire photoelectric signals from the human brain and send the photoelectric signals to the signal amplifier. The signal amplifier is used to receive the photoelectric signal, filter and amplify the photoelectric signal to obtain a corrected photoelectric signal, and send the corrected photoelectric signal to the analog-to-digital converter. The analog-to-digital converter is used to receive the corrected photoelectric signal, convert the corrected photoelectric signal into a digital form of EEG signal, and send the EEG signal to the main control module.

3. The system according to claim 1, characterized in that, The i-th brain electrical signal is represented as {x} i，1 x i，2 x i，3 , ..., x i，n }, where n is the number of data in the i-th brain electronic signal; The central processing unit is specifically used to calculate the distance between adjacent data points in the i-th brain electronic signal using the following formula: ； in, Let x represent the Euclidean distance between the (j+1)th data point and the jth data point in the i-th brain electrical signal. i,j+1 Let x represent the (j+1)th data point in the i-th brain electrical signal. i,j This represents the j-th data point in the i-th brain electrical signal.

4. The system according to claim 1, characterized in that, The central processing unit is specifically used to acquire multiple data queues of a preset length from each of the brain signals using a sliding window of a preset length, wherein at least one data queue is acquired from each of the brain signals; calculate the energy value of each data queue; and select the minimum energy value as a distance threshold.

5. The system according to claim 4, characterized in that, The central processing unit is specifically used to: calculate the energy value of each data queue using the following formula: ； Where E represents the energy value of the data queue, X i X represents the i-th data in the data queue. j This represents the j-th data item in the data queue.

6. The system according to claim 1, characterized in that, The central processing unit is specifically used to: calculate the average number of network paths as an EEG feature for the brain network data using the following formula: ； Where L represents the average number of paths in the network, and M represents the number of data points in the brain network data. This represents the number of edges on the shortest path between two data points.

7. A display mode control method, characterized in that, The method includes: Acquire image data to be displayed and brain signals of the human brain in response to the image data displayed by the display module. The brain signals include N channels of brain electronic signals, and each channel of brain electronic signal corresponds to a collection electrode. For each EEG signal, the distance between data points with adjacent acquisition times is calculated; among the EEG signals, the distance between data points with the same acquisition time is calculated; a distance threshold is obtained, and data points with a distance less than the distance threshold in the EEG signals are connected to obtain brain network data; the average number of network paths in the brain network data is calculated as an EEG feature; according to a predetermined first correspondence between EEG features and display modes, the target display mode corresponding to the EEG feature is determined; The target display mode and the image data to be displayed are sent to the display module so that the display module displays the image data to be displayed according to the target display mode.

8. The method according to claim 7, characterized in that, The i-th brain electrical signal is represented as {x} i，1 x i，2 x i，3 , ..., x i，n }, where n is the number of data in the i-th brain electronic signal; For each brain signal, calculating the distance between data points with adjacent acquisition times in that brain signal channel includes: For the i-th brain electronic signal, the distance between data points with adjacent acquisition times in the i-th brain electronic signal is calculated using the following formula: ； in, Let x represent the Euclidean distance between the (j+1)th data point and the jth data point in the i-th brain electrical signal. i,j+1 Let x represent the (j+1)th data point in the i-th brain electrical signal. i,j This represents the j-th data point in the i-th brain electrical signal.

9. The method according to claim 7, characterized in that, The acquisition of the distance threshold includes: Using a sliding window of a preset length, multiple data queues of a preset length are obtained from each of the brain electronic signals, wherein at least one data queue is obtained from each of the brain electronic signals; Calculate the energy value of each of the data queues; Select the minimum energy value as the distance threshold; The calculation of the energy value of each of the data queues includes: For each data queue, the energy value of that data queue is calculated using the following formula: ； Where E represents the energy value of the data queue, X i X represents the i-th data in the data queue. j This represents the j-th data item in the data queue.

10. The method according to claim 7, characterized in that, The calculation of the average number of network paths in the brain network data as an EEG feature includes: For the aforementioned brain network data, the average number of network paths is calculated as an EEG feature using the following formula: ； Where L represents the average number of paths in the network, and M represents the number of data points in the brain network data. This represents the number of edges on the shortest path between two data points.

11. An electronic device, characterized in that, include: The system includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the method described in any one of claims 7-10.

Images