An adaptive black insertion display method based on human eye activity for virtual reality

Abstract

The present invention discloses an adaptive black insertion display method based on human eye activity for virtual reality, including: step 1, collecting user electrooculogram signals and device motion acceleration, and preprocessing the electrooculogram signals and device motion acceleration; step 2, combining the classification of user electrooculogram signals with the classification of device motion acceleration to obtain M user states; step 3, constructing a multi-period time domain convolution module; step 4, constructing a relationship network based on the multi-period time domain convolution module, and classifying the electrooculogram signal query set Q1 and the device motion acceleration query set Q2; step 5, obtaining the corresponding matrix WQ of each user state and the black insertion display duration of the general user; step 6, combining the corresponding matrix WQ of each user state and the black insertion display duration of the general user to obtain the black insertion display duration per second. The multi-period time domain convolution module proposed in the present application has fewer parameters, occupies less memory, and can retain timing information for a longer time.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality device display, and in particular to an adaptive black insertion display technology based on human eye activity to perform black insertion display on videos in virtual reality. Background Art

[0002] At present, the refresh rate of most virtual reality products is in the range of 90Hz to 180Hz. Low refresh rate will cause the problem of screen ghosting for users; at the same time, due to the low refresh rate, the picture received by the user's eyes will be inconsistent with the real information perceived by the vestibular system, and the user will feel dizzy. A high refresh rate will bring users smooth and coherent dynamic pictures, and at the same time can alleviate the user's dizziness. Therefore, high refresh rate has become a selling point for major virtual reality product manufacturers. In order to solve the above problems, the black insertion method is often used to solve it. The black insertion method is to insert a pure black frame between two video frames. Patent CN202111473884.9 proposes a method of using sensors such as gyroscopes to obtain acceleration and thus control the duration of black insertion display, which is used to solve the problem of reducing ghosting and flickering. However, there are two problems with this patent: first, the method does not explain in detail the mapping relationship between the acceleration of the virtual reality device and the duration of the black display; second, because the human eye receives different degrees of video at different levels of activity, when the human eye is tired, the video content received by the human eye is relatively less, which means that the duration of the black display can be slightly increased, so it is not scientific to control the duration of the black display solely from the acceleration of the virtual reality device. Nowadays, many people are anxious because of various pressures, and excessive anxiety can lead to insomnia. Therefore, it is important to propose a method to help people with insomnia fall asleep smoothly. Watching videos to help sleep is a common method, but existing sleep-aiding videos cannot ensure that they are effective for users.

[0003] If there is an adaptive black-insertion display technology based on human eye activity, users can switch between game mode and sleep mode according to their own requirements, and adaptively adjust the black-insertion display duration in different modes. First, in sleep mode, the black-insertion display duration is controlled according to human eye fatigue to achieve the effect of helping sleep; second, in game mode, the black-insertion display duration is controlled according to the device's acceleration data and human eye activity data to achieve a pleasant gaming experience. Then this virtual reality device can not only enrich the user's spare time life, but also improve the user's sleep quality, which can be said to be a must-have at home.

[0004] After consulting the existing data, it is found that the feature information within a certain period of time can be retained through the time domain convolution model. This patent proposes a multi-period time domain convolution model to solve the problem of long input time and many layers of traditional time domain convolution models when processing long time series information; further, the relational network has a better effect in small sample learning. Due to the differences in electrooculographic signals and device motion acceleration of different people, this patent proposes to use the cosine similarity function as the loss function of the relational network to improve the network accuracy. This patent can accurately identify the user's eye activity and device motion acceleration by constructing a multi-time domain convolution network, thereby obtaining a classification result. This patent formulates a mapping relationship library between the classification results and the black insertion display duration to obtain the black insertion display duration. Summary of the invention

[0005] Virtual reality products have problems such as screen smearing and eye fatigue caused by low refresh rates, and existing virtual reality products do not have a sleep mode for people with sleep disorders. This patent proposes an adaptive black insertion display technology based on human eye activity, and users can switch between game mode and sleep mode according to their own needs. In game mode, the black insertion display duration is obtained by analyzing the user's electrooculogram signal and acceleration; in sleep mode, the black insertion display duration is obtained by analyzing the user's electrooculogram signal. There are few training samples for electrooculogram signals, and traditional convolutional neural networks are prone to overfitting during training, and the network accuracy is not high; most of the computing resources of the computing module of virtual reality products are used for image rendering, and other auxiliary technologies cannot occupy too many computing resources. Occupying too many computing resources will cause the image to freeze, resulting in a very poor user experience. This patent proposes an application method of an adaptive black insertion display technology based on human eye activity in virtual reality. First, the electrooculogram signal features are processed using a multi-period time domain convolution module, which can not only retain the time series feature information for a long time, but also the time domain convolution module has fewer computing parameters than the GRU unit and LSTM module. Then, the relational network is used to replace the traditional recognition network, and the cosine similarity loss function is proposed. It can achieve excellent accuracy by learning a small number of samples. Finally, the judgment switch on the same network model is used to complete the user's mode switching instructions. There is no need to build different network models according to different modes. The amount of parameters that need to be calculated is reduced, and the response speed of the playback screen and mode adjustment is improved.

[0006] The present invention provides an adaptive black insertion display method for virtual reality based on human eye activity, comprising the following steps:

[0007] Step 1: Collect the user's electrooculogram signal and the device motion acceleration, and pre-process the electrooculogram signal and the device motion acceleration;

[0008] Step 2, classifying the user's electrooculogram signals and using them as the electrooculogram signal sample set S1;

[0009] Classify the device motion acceleration and use it as the device motion acceleration sample set S2;

[0010] The classification of the user's electrooculogram signal is combined with the classification of the device motion acceleration to obtain M types of user states.

[0011] Step 3: construct a multi-period time domain convolution module to extract the timing characteristics of electrooculogram signals and acceleration.

[0012] Step 4, constructing a first relational network based on a multi-time domain convolution module, combining the electrooculogram signal sample set S1, and classifying the electrooculogram signal query set Q1;

[0013] A second relational network based on a multi-time domain convolution module is constructed, and the device motion acceleration query set Q2 is classified in combination with the device motion acceleration sample set S2.

[0014] Step 5: Obtain the corresponding matrix WQ of M user states and the black insertion display duration of common users; WQ = [wq1, wq2, …wq λ …wq M ], where 1≤λ≤M, wq λ Indicates the black display duration of the general user corresponding to the λth user status;

[0015] Step 6, based on the classification of the electrooculogram signals obtained in step 4 and the classification of the device motion acceleration, the user status is combined, and the corresponding matrix WQ of each user status and the black insertion display duration of the general user in step 5 is combined to obtain the black insertion display duration per second.

[0016] Furthermore, in step 5, a corresponding matrix WQ of M user states and black insertion duration of common users is obtained; the specific process is as follows:

[0017] Obtain the corresponding matrix U of each user state and black display duration under the test of N volunteers;

[0018] U=(u1,u2,…u λ …u M )

[0019] u λ =[m λ,1 , m λ,2 ...m λ，i ...m λ,N ]

[0020] m λ,i represents the most comfortable black insertion display duration for the i-th volunteer in the λ-th user state, and N represents the number of volunteers.

[0021] Use the following formula to calculate uλ Calculate and obtain the most suitable black display duration wq per second for the general λth user state λ .

[0022]

[0023] Construct the corresponding matrix WQ of the black display duration of the general user in M ​​user states, WQ = [wq1, wq2, …wq λ …wq M ]. Further, in step 1, the user's electrooculogram signal and the device motion acceleration are collected, and the electrooculogram signal and the device motion acceleration are preprocessed. The specific steps are as follows:

[0024] Step 1.1, collect the forehead electrooculogram signal through n electrodes on the virtual reality device, obtain the original electrooculogram signal in the n-channel δ time period, and remove the baseline signal in the first τ time period of the original electrooculogram signal. Use the sampling frequency of VHz for downsampling, and then use the filter to filter the electrooculogram signal

[0025] Step 2.1: Use the gravity sensor to obtain the acceleration a of the device in three directions within a time period of δ x 、a y 、a z , and extract four other acceleration components, the calculation formula is as follows:

[0026]

[0027] The obtained 7 equipment motion acceleration data (a x ,a y ,a z ,f,h,α,β) for processing. First, the data in the first τ time is removed, and then down-sampled using a sampling frequency of VHz.

[0028] Furthermore, step 2 is specifically as follows: classifying the user's electrooculogram signals into three categories: awake, tired, and sleepy, and using the three categories as the electrooculogram signal sample set S1.

[0029] The device motion acceleration is classified into zero level, first level, second level, third level, and fourth level; as the device motion acceleration sample set S2

[0030] The three categories of user electrooculogram signals are combined with the five categories of device motion acceleration to obtain fifteen user states.

[0031] Furthermore, in step 3, a multi-period time domain convolution module is constructed to extract the timing characteristics of the electrooculogram signal and the device motion acceleration;

[0032] The multi-period time domain convolution module inputs the electrooculogram signal or device motion acceleration into the time domain convolution module in different time periods. First, it is divided into a segments according to the length of the time sequence. The input duration of each time period is θ.

[0033] The input of the time domain convolution module in the first period is E1 = [e0, e1…e t-1 ,e t ], where e t The size is D*n, where D represents each e t The number of electrooculogram signals in , n represents the number of channels.

[0034] The input of the time domain convolution module in the second period is E2 = [e t ,e t+1 …e 2t-1 ,e 2t ],

[0035] …………

[0036] The input of the time domain convolution module of segment a is E a =[e L-t ,e L-t+1 …e L-1 ,e L ];

[0037] The network output of the first segment is ω1, and so on, the network output of segment a is ω a , the first calculated ω a Store ω1, ω2…ω a Finally, the output of the multi-stage time domain convolution module is W = [ω1, ω2…ω a ].

[0038] Furthermore, causal convolution is used in the time domain convolution module, and the causal convolution calculation method is as follows:

[0039]

[0040] where k=(k0,k1,…,k c-1 ) is the convolution kernel, c represents the number of convolution kernels, E = (e0, e1, …, e t ) is the input segmented electrooculogram signal.

[0041] Furthermore, the calculation method of adding dilated convolution to causal convolution is as follows:

[0042]

[0043] Where d represents the expansion coefficient, d = {1, 2, ... 2 b}, b represents the number of network layers.

[0044] Furthermore, step 4 specifically includes the following contents:

[0045] The first relational network based on the multi-time domain convolution module includes a relational module and multiple groups of channel attention mechanisms and multi-period temporal convolution modules; the number of groups of channel attention mechanisms and multi-period temporal convolution modules in the first relational network is the same as the number of categories of the electrooculographic signal;

[0046] Different categories of user electrooculogram signal sample sets S1 are respectively input into different groups of channel attention mechanisms and multi-period temporal convolution modules to obtain S1 abstract features;

[0047] The user's electrooculogram signal query set Q1 is input into another set of channel attention mechanisms and multi-period temporal convolution modules to obtain Q1 abstract features, where the query set Q1 is the data to be classified.

[0048] Each type of feature in S1 abstract feature is concatenated with Q1 abstract feature and then input into the relationship module The distance between the sample set S1 and the query set Q1 is calculated to obtain the similarity score between the two, thereby classifying them.

[0049] The second relational network based on the multi-time domain convolution module includes a relational module As well as multiple groups of channel attention mechanisms and multi-period temporal convolution modules;

[0050] Different categories of user device motion acceleration sample sets S2 are respectively input into different groups of channel attention mechanisms and multi-period temporal convolution modules to obtain S2 abstract features; the number of groups of channel attention mechanisms and multi-period temporal convolution modules in the second relational network is the same as the number of categories of device motion acceleration;

[0051] The user device motion acceleration query set Q2 is input into another set of channel attention mechanisms and multi-period temporal convolution modules to obtain Q2 abstract features, where the query set Q2 is the data to be classified.

[0052] Each type of feature in S2 abstract feature is concatenated with Q2 abstract feature and then input into the relationship module The distance between sample set S2 and query set Q2 is calculated to obtain the similarity score between the two, thereby classifying them.

[0053] Furthermore, in the relational network, the loss function trains the network by calculating the similarity distance between the sample set S and the query set Q;

[0054] The calculation sample set S is the user electrooculogram signal sample set S1 or the device motion acceleration sample set S2, and the query set Q represents the user electrooculogram signal sample set Q1 or the device motion acceleration query set Q2.

[0055] The calculation formula is as follows:

[0056]

[0057] where r i,j is the probability score of the corresponding query set sample, and y represents the time series feature.

[0058] Furthermore, for the electrooculogram signal, the cosine similarity loss function is used to calculate the correlation of the cascade matrix. The calculation formula is as follows:

[0059]

[0060] Furthermore, the present invention provides an adaptive black insertion display method for virtual reality based on human eye activity, further comprising:

[0061] Step 7: Build a personalized user library, which is used to store the electrooculogram signal data and device motion acceleration data of multiple users, and their corresponding classification labels, specifically:

[0062] Step 7.1, the user logs in and uses the virtual reality device;

[0063] Step 7.2, user status identification;

[0064] Collect the user's electrooculogram signal data and device motion acceleration, and first compare them with the user data stored in the user database using the cosine similarity function;

[0065] When the similarity is lower than R1, the user database does not store this type of data for the user, and the process goes to step 7.3;

[0066] When the similarity is greater than or equal to R1, the classification in the user database is directly used to obtain the black display duration of the user according to the content in step 5.

[0067] Step 7.3, add user data to the user library

[0068] The user's electrooculogram signal data and device motion acceleration are scored respectively through the first relationship network and the second relationship network based on the multi-time domain convolution module, and the classification probability score is output. The electrooculogram signal data, device motion acceleration and their corresponding classifications with a classification probability score above R are stored in the user library.

[0069] Furthermore, the black insertion time for images with different refresh rates is different. This patent establishes a query library for the black insertion time of 90Hz, 110Hz, 130Hz, 150Hz, and 170Hz. For refresh rates of 120Hz, 140Hz and more, bilinear interpolation can be used to fill in the gaps.

[0070] Furthermore, the user switches between the sleep mode and the game mode by controlling the input amount. For the sleep mode, the device motion acceleration is assigned a zero-level category, and the user state is determined by the electrooculogram signal classification; for the game mode, the user state is determined by the user electrooculogram signal classification and the device motion acceleration classification.

[0071] This patent proposes an application method of adaptive black insertion display technology based on human eye activity in virtual reality. Users can adjust the virtual reality device to game mode and sleep mode according to their own needs. Then, this patent identifies the user's eye activity and the user's acceleration according to the user's set mode. This patent classifies the user's eye activity and acceleration, and the acceleration classification in sleep mode is defaulted to 0. Compared with the traditional network, the relational network based on the multi-period time domain convolution module proposed in this patent has fewer parameters, occupies less memory, and can retain time series information for a longer time; it proposes to use the cosine similarity function as the relational network loss function, so that the network can have excellent accuracy through the learning of a small number of samples. Secondly, this patent establishes a personalized user library to store the user's labeled time series information, which is convenient for direct use next time. Thirdly, this patent uses the judgment switch on the same network model to complete the user's mode switching instructions, without the need to build different network models according to different modes. Finally, this patent tests volunteers and calculates the mapping relationship between different classification results and black display duration. The black display duration can be obtained through query through the classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a multi-period time domain convolution module;

[0073] Figure 2 It is a relational network based on multi-period temporal convolutional modules;

[0074] Figure 3 It is the switch control structure for mode selection;

[0075] Figure 4 It is a user status classification table;

[0076] Figure 5 is the channel domain attention module DETAILED DESCRIPTION

[0077] The patent of this invention is based on the existing artificial intelligence technologies such as time domain convolution module, attention mechanism, relational network, etc., and is aimed at the problems of screen smearing and eye fatigue caused by low refresh rate of existing virtual reality products, and the fact that existing virtual reality products do not have a sleep mode for people with sleep disorders. This patent proposes an adaptive black insertion display technology based on human eye activity, and users can switch between game mode and sleep mode according to their own needs. In game mode, the black insertion display duration is obtained by analyzing the user's electrooculogram signal and acceleration; in sleep mode, the black insertion display duration is obtained by analyzing the user's electrooculogram signal. There are few training samples for electrooculogram signals, and traditional convolutional neural networks are prone to overfitting during training, and the network accuracy is not high; most of the computing resources of the computing module of virtual reality products are used for picture rendering, and other auxiliary technologies cannot occupy too many computing resources. Occupying too many computing resources will cause the picture to freeze, resulting in a very poor user experience. This patent proposes an application method of an adaptive black insertion display technology based on human eye activity in virtual reality. First, the multi-period time domain convolution module is used to process the characteristics of the electrooculogram signal, which can not only retain the time series feature information for a long time, but also has fewer calculation parameters compared with the GRU unit and LSTM module. Then, the relational network is used to replace the traditional recognition network, and the cosine similarity loss function is proposed. Through the learning of a small number of samples, it can have excellent accuracy; finally, the judgment switch on the same network model is used to complete the user's game mode and sleep mode switching instructions. There is no need to build different network models according to different modes, the number of parameters that need to be calculated is reduced, and the response speed of the playback screen and mode adjustment is improved.

[0078] An adaptive black insertion display method based on human eye activity for virtual reality comprises the following steps:

[0079] Step 1: Collect the user's electrooculogram signal and device motion acceleration, and pre-process the electrooculogram signal and device motion acceleration. The specific steps are as follows:

[0080] Step 1.1, collect the forehead electrooculogram signal through n electrodes on the virtual reality device, obtain the original electrooculogram signal in the δ time period of n channels, in this embodiment, n=4, remove the baseline signal in the first τ time period of the original electrooculogram signal. Then, since the initial frequency of the electrooculogram signal is IHz, in order to facilitate the subsequent data processing, this patent uses a sampling frequency of VHz for downsampling. Finally, this patent uses a 20-400Hz filter to filter the electrooculogram signal. In this embodiment, δ=60s, τ=3.195s, I=1000, V=200.

[0081] Step 2.1: Use a gravity sensor with a sampling frequency of 1 Hz to obtain the acceleration a of the device in three directions within a time period of δ. x 、ay 、a z , In order to enrich the data diversity, four acceleration components are extracted, and the calculation formula is as follows:

[0082]

[0083]

[0084] This patent obtains 7 data (a x ,a y ,a z ,f,h,α,β) for processing. First, the data in the first τ time is removed, and then down-sampled using a sampling frequency of VHz. In this embodiment, I=1000, δ=60s, τ=3.195s, and V=200.

[0085] Step 2: Since the production of the EO signal data set is relatively complicated, the EO signals of N people are collected and classified into three categories: awake, tired, and sleepy, as the EO signal sample set S1.

[0086] The device motion acceleration is classified into zero level, first level, second level, third level, and fourth level; as the device motion acceleration sample set S2

[0087] The three categories of user electrooculogram signals are combined with the five categories of device motion acceleration to obtain fifteen user states, such as Figure 4 shown.

[0088] like Figure 3 As shown, the user switches between the sleep mode and the game mode by controlling the input amount. For the sleep mode, the device motion acceleration is assigned a zero-level category, and the user state is determined by the electrooculogram signal classification; for the game mode, the user state is determined by the user electrooculogram signal classification and the device motion acceleration classification.

[0089] Step 3: construct a multi-period time domain convolution module to extract the timing characteristics of electrooculogram signals and acceleration.

[0090] The time domain convolution module can process time series information, and compared with LSTM and GRU algorithms, it has lower memory usage, more stable gradients, and parallelism. However, due to the parallelism of the time domain convolution module, the complete time series information needs to be input at one time during calculation, so it takes a lot of time to store when processing long time series information; when processing long time series information, the time domain convolution module needs to build more layers, and each layer needs to use a filter, which takes up a lot of memory and increases the amount of calculation.

[0091] Based on this, this patent proposes a multi-period time domain convolution module. This patent improves the structure of the original time domain convolution module, inputs the time series information in different time periods, processes the time series information in segments, and reduces the duration and memory. Figure 1 The input of the multi-period time domain convolution module is a vector of n*(γ*s), where n represents the number of channels, γ represents the duration, and s represents the number of electrooculogram signals per second. In this embodiment, n=4, γ=56.805s, and s=200.

[0092] The multi-period time domain convolution module inputs the electrooculogram signal into the time domain convolution module in different time periods to process the electrooculogram signal with a long time series; at the same time, it reduces the number of network layers and reduces the memory usage. The specific steps are as follows:

[0093] The size of the electrooculogram signal to be calculated is [(L+1)*D)]*n. First, it is divided into a segments according to the timing length of the electrooculogram signal. The input duration of each segment is θ. In the present invention, θ=1.995s and a=30 are used.

[0094] The input of the time domain convolution module in the first period is E1 = [e0, e1…e t-1 ,e t ], where e t The size is D*n, where D represents each e t The number of electrooculogram signals, n represents the number of channels. In this embodiment, D = 21, n = 4, t = 18, L = 540, and each e t 21 electrooculogram signals with a duration of 0.105 s are stored in it.

[0095] The input of the time domain convolution module in the second period is E2 = [e t ,e t+1 …e 2t-1 ,e 2t ],

[0096] …………

[0097] The input of the time domain convolution module of segment a is E a =[e L-t ,e L-t+1 …e L-1 ,e L ].

[0098] Each time period in the multi-time period time domain convolution module adopts the original time domain convolution module processing method.

[0099] The time domain convolution module uses causal convolution, which can well extract the timing features in the electrooculogram signal. The calculation method of causal convolution is as follows:

[0100]

[0101] where k=(k0,k1,…,k c-1 ) is the convolution kernel, c represents the number of convolution kernels, E = (e0, e1, …, e t ) is the input segmented electrooculogram signal.

[0102] In order to reduce the number of calculation layers of causal convolution, the time domain convolution module uses dilated convolution to reduce the amount of calculation and increase the receptive field. The calculation method of causal convolution with dilated convolution is as follows:

[0103]

[0104] Where d represents the expansion coefficient, generally d = {1,2,…2 b}, b represents the number of network layers.

[0105] The calculation method of the causal convolution with dilated convolution added in section a is as follows:

[0106]

[0107] In this embodiment, the first layer parameter setting of the time domain convolution module in the ath period is d=1, c=19. The second layer parameter setting of the time domain convolution module in the ath period is d=2, c=9. The third layer parameter setting of the time domain convolution module in the ath period is d=6, c=3.

[0108] In order to facilitate network training, the time domain convolution module adds a residual block structure, through the 1*1 convolution operation, so that e t It has the same number of channels as the output of the causal convolution with the dilated convolution added. In the multi-period temporal convolution module, the network output of the ath segment is ω a , because the calculation is performed in time periods, the ω calculated first a Store ω1, ω2…ω a Finally, the output of the multi-stage time domain convolution module is W = [ω1, ω2…ω a ].

[0109] The multi-period time domain convolution module inputs the device motion acceleration into the time domain convolution module in different time periods. The device motion acceleration input of the relational network of the multi-period time domain convolution module is y*(γ*), where y=7, representing a x ,a y ,a z ,f,h,α,β;γ represents the duration, s represents the number of electrooculogram signals per second, γ=56.805s, s=200. The input of the multi-time domain convolution module in each period is 7 acceleration data collected within 1.995s, and the data size is 7*399. There are 30 time domain convolution modules in total.

[0110] Step 4, construct a first relationship network based on a multi-time domain convolution module for identifying electrooculogram signals; construct a second relationship network based on a multi-time domain convolution module for identifying device motion acceleration.

[0111] Since there are too few samples in the data set, it is a small sample data set. Therefore, when using the traditional classification network for training, overfitting problems are prone to occur, resulting in low network accuracy. Based on this, this patent uses a relational network for subsequent classification.

[0112] Using the embedding module in the original relational network Extract the features of S and Q. Since this patent uses long time series information, Unable to extract the time series features in the time series information, this patent uses a multi-period time domain convolution module instead This patent adds an attention mechanism module to assign weights to the features of each channel. The attention mechanism module is not described here. After passing through the multi-period time domain convolution module, the abstract features of S and Q are obtained, and then each type of feature in S is spliced ​​with Q respectively. Finally, the relationship module is used The distance between S and Q is calculated to obtain the similarity score between the two, thereby achieving the classification effect.

[0113] like Figure 2 As shown, the first relational network based on the multi-time domain convolution module includes a relational module As well as four groups of channel attention mechanisms and multi-period temporal convolution modules;

[0114] The preprocessed three categories of user electrooculogram signal sample sets S1 are respectively input into three groups of channel attention mechanisms and multi-period temporal convolution modules to obtain S1 abstract features;

[0115] The preprocessed user electrooculogram signal query set Q1 is input into a set of channel attention mechanisms and multi-period temporal convolution modules to obtain Q1 abstract features, where the query set Q1 is the data to be classified.

[0116] Each type of feature in S1 abstract feature is concatenated with Q1 abstract feature and then input into the relationship module The distance between the sample set S1 and the query set Q1 is calculated to obtain the similarity score between the two, thereby achieving the classification effect.

[0117] The second relational network based on the multi-time domain convolution module includes a relational module As well as six groups of channel attention mechanisms and multi-period temporal convolution modules;

[0118] The preprocessed five categories of user device motion acceleration sample sets S2 are respectively input into five groups of channel attention mechanisms and multi-period temporal convolution modules to obtain S2 abstract features;

[0119] The preprocessed user device motion acceleration query set Q2 is input into a set of channel attention mechanisms and multi-period temporal convolution modules to obtain Q2 abstract features, where the query set Q2 is the data to be classified.

[0120] Each type of feature in S2 abstract feature is concatenated with Q2 abstract feature and then input into the relationship module The distance between the sample set S2 and the query set Q2 is calculated to obtain the similarity score between the two, thereby achieving the classification effect.

[0121] This patent directly uses the channel domain attention mechanism, such as Figure 5 As shown. Taking the electrooculogram signal from time 0 to t as an example, E1 is input into the channel domain attention mechanism module. First, after global average pooling, a real number sequence of size 1*1*4 can be obtained; then after 1*1 convolution, which is used to learn weight parameters, a 1*1*4 one-dimensional sequence can be obtained; finally, after the Sigmod function is used to increase the network nonlinearity, a 1*1*4 h1 can be obtained. All time periods are input in sequence to obtain h=[h1,h2…h 30 ] Then the abstract feature HW of S and Q can be expressed as Where hw1=h1×ω1.

[0122] In the relational network, the loss function trains the network by calculating the similarity distance between the sample set S and the query set Q. The calculated sample set S represents the user's electrooculogram signal sample set S1 or the device motion acceleration sample set S2, and the query set Q represents the user's electrooculogram signal sample set Q1 or the device motion acceleration query set Q2.

[0123] The calculation formula is as follows:

[0124]

[0125] where r i,j is the probability score of the corresponding query set sample, and y represents the time series feature.

[0126] However, there are differences in the electrooculographic signals of different people. Simply calculating the similarity distance of the features is too absolute. Although this can ensure that the classification results are correct at high scores, it cannot ensure that the classification results are wrong at low scores. Based on this, the present application proposes to use the cosine similarity loss function to calculate the correlation of the cascade matrix, which is applicable to both the first relationship network and the second relationship network based on the multi-time domain convolution module; the cascade matrix refers to the matrix obtained by splicing each type of feature in the S1 abstract feature with the Q1 abstract feature, or the matrix obtained by splicing each type of feature in the S2 abstract feature with the Q2 abstract feature.

[0127] The calculation formula is as follows:

[0128]

[0129] Step 5, obtaining the corresponding matrix u between the user status and the black insertion display duration;

[0130] The test was conducted on 12 volunteers, including 3 elderly people, 3 young people, 3 teenagers, and 3 myopic people. Among the 12 people, 6 were male and 6 were female. The volunteers were observed for Figure 4 It reflects the different black display durations in fifteen states.

[0131] After the volunteer scoring test, the corresponding matrix U is obtained, where U = (u1, u2, ... λ …u 15 ). λ =[m λ,1 ,m λ,2 …m λ,N ], m represents the most comfortable black insertion display duration of the volunteer in the λth user state, and N represents the number of volunteers. This patent uses formula (10) to calculate u λ Calculation is performed to obtain the optimal black insertion display duration corresponding to different user states applicable to general users, and a mapping matrix WQ between user states and black insertion display durations is added. In this embodiment, λ∈{1,2,…15}, N=12.

[0132] The most suitable black display duration per second for the general λth user state wq λ , the calculation method is as follows:

[0133]

[0134] The black insertion time for images with different refresh rates is different. This application has established a query library for the black insertion time of 90Hz, 110Hz, 130Hz, 150Hz, and 170Hz. For 120Hz, 140Hz and more refresh rates, bilinear interpolation can be used to fill in the gaps.

[0135] Step 6, according to the classification of the electrooculogram signal obtained in step 4 and the classification of the device motion acceleration, the user status is obtained by combining them, and the corresponding matrix U of each user status and the black insertion display duration is combined to obtain the black insertion display duration per second.

[0136] In addition, this application also includes building a personalized user database, which automatically stores the user's usage information after the user completes a virtual reality product use; specifically:

[0137] Step 7: Build a personalized user library, which is used to store the electrooculogram signal data and device motion acceleration data of multiple users, and their corresponding classification labels, specifically:

[0138] Step 7.1, the user logs in and uses the virtual reality device;

[0139] Step 7.2, user status identification;

[0140] Collect the user's electrooculogram signal data and device motion acceleration, and first compare them with the user data stored in the user database using the cosine similarity function;

[0141] When the similarity is lower than R1, the user database does not store this type of data for the user, and the process goes to step 7.3;

[0142] When the similarity is greater than or equal to R1, the classification in the user database is directly used, and the black insertion display duration of the user is obtained according to the content in step 5. In this embodiment, R1=95.

[0143] Step 7.3, add user data to the user library

[0144] The user's electrooculogram data and device motion acceleration are scored respectively by the first relational network and the second relational network based on the multi-time domain convolution module, and a probability score is output, and the electrooculogram data and device motion acceleration with a classification score above R are stored in the user database. In this embodiment, R=90.

[0145] This patent proposes an application method of adaptive black insertion display technology based on human eye activity in virtual reality. Users can adjust the virtual reality device to game mode and sleep mode according to their own needs. Then, this patent identifies the user's eye activity and the user's acceleration according to the user's setting mode. This application classifies the user's eye activity and the user's acceleration. The classification result of the user's acceleration in sleep mode is 0 by default. Compared with the traditional network, the relational network based on the multi-period time domain convolution module proposed in this patent has fewer parameters, occupies less memory, and can retain time series information for a longer time; it proposes to use the cosine similarity function as the relational network loss function, so that the network can have excellent accuracy through the learning of a small number of samples. Secondly, this patent establishes a personalized user library to store the user's labeled time series information, which is convenient for direct use next time. Thirdly, this patent uses the judgment switch on the same network model to complete the user's mode switching instructions, without the need to build different network models according to different modes. Finally, this patent tests volunteers and calculates the mapping relationship between different classification results and black display duration. The black display duration can be obtained through query through the classification results.

Claims

1. An adaptive black insertion display method based on human eye activity for virtual reality, characterized in that: The steps include: Step 1: Collect the user's electrooculogram signal and the device motion acceleration, and pre-process the electrooculogram signal and the device motion acceleration; Step 2, classifying the user's electrooculogram signals as the electrooculogram signal sample set S1; Classify the device motion acceleration as the device motion acceleration sample set S2; The classification of the user's electrooculogram signal is combined with the classification of the device's motion acceleration to obtain M types of user states; Step 3, construct a multi-period time domain convolution module to extract the timing characteristics of electrooculogram signals and acceleration; Step 4, constructing a first relational network based on a multi-time domain convolution module, combining the electrooculogram signal sample set S1, and classifying the electrooculogram signal query set Q1; Construct a second relational network based on a multi-time domain convolution module, and classify the device motion acceleration query set Q2 in combination with the device motion acceleration sample set S2; Step 5, obtain the corresponding matrix WQ of M user states and the black display duration of common users: WQ=[wq1,wq2,…wq λ …wq M ], where 1≤λ≤M, wq λ Indicates the black display duration of the general user corresponding to the λth user status; Step 6, based on the classification of the electrooculogram signals obtained in step 4 and the classification of the device motion acceleration, the user status is combined, and the corresponding matrix WQ of each user status and the black insertion display duration of the general user in step 5 is combined to obtain the black insertion display duration per second.

2. According to claim 1, an adaptive black insertion display method based on human eye activity for virtual reality, characterized in that: In step 1, the user's electrooculogram signal and the device motion acceleration are collected and preprocessed. The specific steps are as follows: Step 1.1, collect the forehead electrooculogram signal through n electrodes on the virtual reality device, obtain the original electrooculogram signal in the δ time period of n channels, remove the baseline signal in the first τ time period of the original electrooculogram signal; use the sampling frequency of VHz for downsampling, and then use the filter to filter the electrooculogram signal; Step 2.1: Use the gravity sensor to obtain the acceleration a of the device in three directions within a time period of δ x 、a y 、a z , and extract four other acceleration components, the calculation formula is as follows: The obtained 7 equipment motion acceleration data (a x ,a y ,a z ,f,h,α,β) for processing. First, the data in the first τ time is removed, and then down-sampled using a sampling frequency of VHz.

3. The method for adaptive black insertion display based on human eye activity for virtual reality according to claim 1, characterized in that: Step 2 specifically includes: classifying the user's electrooculogram signals into three categories: awake, tired, and sleepy, and using the three categories as the electrooculogram signal sample set S1; Classify the device motion acceleration into zero level, first level, second level, third level, and fourth level; and use them as the device motion acceleration sample set S2; The three categories of user electrooculogram signals are combined with the five categories of device motion acceleration to obtain fifteen user states.

4. The method for adaptive black insertion display based on human eye activity for virtual reality according to claim 1, characterized in that: In step 3, a multi-period time domain convolution module is constructed to extract the timing characteristics of the electrooculogram signal and the device motion acceleration; The multi-period time domain convolution module inputs the electrooculogram signal or the device motion acceleration into the time domain convolution module in different time periods. First, it is divided into a segments according to the length of the time sequence, and the input duration of each time period is θ; The network output of the first segment is ω1, and so on, the network output of segment a is ω a , ω1,ω2…ω a Finally, the output of the multi-stage time domain convolution module is W = [ω1, ω2…ω a ].

5. The method for adaptive black insertion display based on human eye activity for virtual reality according to claim 4, characterized in that: The time domain convolution module adopts causal convolution, and the causal convolution calculation method is as follows: where k=(k0,k1,…,k c-1 ) is the convolution kernel, c represents the number of convolution kernels, E = (e0, e1, …, e t ) is the input segmented electrooculogram signal.

6. The method for adaptive black insertion display based on human eye activity for virtual reality according to claim 5, characterized in that: The calculation method of adding dilated convolution to causal convolution is as follows: Where d represents the expansion coefficient, d = {1, 2, ... 2 b }, b represents the number of network layers.

7. The method for adaptive black insertion display based on human eye activity for virtual reality according to claim 5, characterized in that: Step 4 specifically includes the following: The first relational network based on the multi-time domain convolution module includes a relational module and multiple groups of channel attention mechanisms and multi-period temporal convolution modules; the number of groups of channel attention mechanisms and multi-period temporal convolution modules in the first relational network is the same as the number of categories of the electrooculographic signal; Different categories of user electrooculogram signal sample sets S1 are respectively input into different groups of channel attention mechanisms and multi-period temporal convolution modules to obtain S1 abstract features; The user's electrooculogram query set Q1 is input into another set of channel attention mechanisms and multi-period temporal convolution modules to obtain Q1 abstract features, where the query set Q1 is the data to be classified; Each type of feature in S1 abstract feature is concatenated with Q1 abstract feature and then input into the relationship module The distance between the sample set S1 and the query set Q1 is calculated to obtain the similarity score between the two, thereby classifying them; The second relational network based on the multi-time domain convolution module includes a relational module As well as multiple groups of channel attention mechanisms and multi-period temporal convolution modules; Different categories of user equipment motion acceleration sample sets S2 are respectively input into different groups of channel attention mechanisms and multi-period temporal convolution modules to obtain S2 abstract features; The number of groups of the channel attention mechanism and the multi-period temporal convolution module in the second relational network is the same as the number of categories of the device motion acceleration; The user device motion acceleration query set Q2 is input into another set of channel attention mechanisms and multi-period temporal convolution modules to obtain Q2 abstract features, where the query set Q2 is the data to be classified; Each type of feature in S2 abstract feature is concatenated with Q2 abstract feature and then input into the relationship module The distance between sample set S2 and query set Q2 is calculated to obtain the similarity score between the two, thereby classifying them.

8. The method for adaptive black insertion display based on human eye activity for virtual reality according to claim 1, characterized in that: In the relational network, the loss function trains the network by calculating the similarity distance between the sample set S and the query set Q; The calculation sample set S is the user electrooculogram signal sample set S1 or the device motion acceleration sample set S2, and the query set Q represents the user electrooculogram signal sample set Q1 or the device motion acceleration query set Q2; The calculation formula is as follows: where r i,j is the probability score of the corresponding query set sample, and y represents the time series feature.

9. The method for adaptive black insertion display based on human eye activity for virtual reality according to claim 1, characterized in that: For EOG signals, the cosine similarity loss function is used to calculate the correlation of the cascade matrix; The calculation formula is as follows:

10. The method for adaptive black insertion display based on human eye activity for virtual reality according to claim 1, characterized in that: Also includes: Step 7: Build a personalized user library, which is used to store the electrooculogram signal data and device motion acceleration data of multiple users, and their corresponding classification labels, specifically: Step 7.1, the user logs in and uses the virtual reality device; Step 7.2, user status identification; Collect the user's electrooculogram signal data and device motion acceleration, and first compare them with the user data stored in the user database using the cosine similarity function; When the similarity is lower than R1, the user data is not stored in the user database, and the process goes to step 7.3; When the similarity is greater than or equal to R1, the classification in the user database is directly adopted; Step 7.3, add user data to the user library; The user's electrooculogram data and device motion acceleration are scored respectively through the first relationship network and the second relationship network based on the multi-time domain convolution module, and the probability score is output. The electrooculogram data, device motion acceleration and their corresponding classifications with classification scores above R are stored in the user library.

Images