A method for evaluating visually-induced motion sickness based on an attention mechanism

Abstract

The application belongs to the technical field of digital image processing, and is especially a visual-induced motion sickness evaluation method based on an attention mechanism. First, a light flow video is extracted from a VR video, and then the VR video and the light flow video are preprocessed and input into an evaluation model. Then, the VR video and the light flow video are independently subjected to feature extraction and evaluation to evaluate the degree of visual-induced motion sickness. Finally, the results of the VR video and the light flow video are fused to obtain the evaluation result of the visual-induced motion sickness of the VR video. The application fuses a Transformer architecture, constructs a self-attention mechanism for time series and spatial series respectively, and captures the mutual dependence between sequence elements. Through a backend fusion technology, the integration of the outputs of the double-flow network is realized, which not only maintains the calculation efficiency, but also significantly improves the evaluation accuracy of the model.

Description

Technical Field

[0001] This invention relates to a visually induced motion sickness assessment method based on attention mechanisms, belonging to the field of digital image processing technology. Background Technology

[0002] Compared to two-dimensional video, virtual reality (VR) video contains depth information, providing viewers with a more realistic visual effect and immersive experience. In recent years, with the widespread application of virtual reality technology, such as 3D movies and virtual reality games, these technologies have profoundly impacted and transformed people's daily lives. As a rapidly developing technology, virtual reality faces many challenges. One of the most significant problems is that during a virtual reality experience, the user's visual system receives motion information while the body remains still. This mismatch between vision and the vestibular system can lead to symptoms such as headaches, dizziness, and even vomiting, a condition known as visual-induced motion sickness.

[0003] How to construct an assessment model for visually induced motion sickness (VIS) and effectively classify VR videos so that users can choose videos that better suit their individual needs is an increasingly important issue. The main factors that can be used to assess VIS in VR videos include: optical flow intensity, motion trajectory, texture information, area of ​​focus, and temporal information. By establishing a dual-flow assessment model that combines visual and motion flows with attention mechanisms, the severity of VIS can be effectively assessed. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] The purpose of this invention is to provide a method for assessing visually induced motion sickness based on an attention mechanism. When watching VR videos, the method assesses the degree of visually induced motion sickness based on the video content and extracted optical flow information. It also effectively classifies VR videos by combining the attention mechanism, allowing users to choose videos that better suit their individual experience needs.

[0006] (II) Technical Solution

[0007] The technical solution of this invention is implemented as follows: A method for assessing visually induced motion sickness based on attention mechanisms. First, optical flow video is extracted from VR video. Then, the VR video and optical flow video are preprocessed and input into the assessment model. Next, independent feature extraction and assessment are performed on the VR video and optical flow video to evaluate the degree of visually induced motion sickness. Finally, the VR video results and optical flow video results are fused to obtain the assessment result of visually induced motion sickness in VR video. The specific steps are as follows:

[0008] Step 1: Extract optical flow video from the VR video and perform preprocessing, including the following sub-steps:

[0009] Step 101: Extract optical flow maps from consecutive frames of the VR video, using the formula shown below:

[0010]

[0011] Where f(x,y,t) represents the optical flow vector at time t and position (x,y). x and v y These are the components of the optical flow vector in the x and y directions, representing the speed at which pixels in the image move over time in these two directions. and These represent the rate of change of position (x, y) over time in the image sequence.

[0012] Step 102: Divide the VR video and optical flow video into N patches of the same size, calculated as follows:

[0013] N = HW / P 2

[0014] Where the video input is X∈R H×W×F H and W represent the length and width of the input video frame, which are adjusted to 224×224 here. F represents the frame sampling size of the video, P represents the length and width of the split patch, and N is the number of split patches. After decomposing the input video, it can be represented as a vector. p=1,...,N; t=1,...,F.

[0015] Step 103: Group patches with the same p but different t into one category and name them Time patches; group patches with different p but the same t into another category and name them Time patches.

[0016] Step 2: Perform independent feature extraction and evaluation on VR video and optical flow video, including the following sub-steps:

[0017] Step 201: Input the time patch into the network's Embedding module to facilitate the subsequent calculation of the temporal correlation between VR video and optical flow video, and add location encoding information, as shown in the following formula:

[0018]

[0019] in The output result is named "token", where 0 represents the current network layer number, p represents the current patch number, and t represents the current frame number, p = 1, ..., N; t = 1, ..., F; Let P be a learnable matrix, representing the length and width of the split patch. For learnable spatial location encoding, D is the dimension of the patch.

[0020] Step 202: Input the Spac patch into the Embedding module of the network to facilitate the subsequent calculation of the spatial sequence correlation between VR video and optical flow video, and add positional encoding information, as shown in the following formula.

[0021]

[0022] in The output result is named "token", where 0 represents the current network layer number, p represents the current patch number, and t represents the current frame number, p = 1, ..., N; t = 1, ..., F; Let P be a learnable matrix, representing the length and width of the split patch. For learnable spatial location encoding, D is the dimension of the patch. The additional location code represents the overall classification result.

[0023] Step 203: After preprocessing the data through the Embedding module, the data is input into the core module of the network model: the Encoder module. The Encoder module consists of Attention layers and Multilayer Perceptron (MLP) layers. The network model contains L Encoder layers, and each Encoder module contains two Attention modules and one MLP module.

[0024] Step 204: The Attention layer in each Encoder layer is calculated as follows:

[0025]

[0026] Attention(Q,K,V)=AV

[0027] Where Q, K, and V are The set of , where A is the self-attention distribution calculated from Q and K. The calculation method is as follows:

[0028]

[0029]

[0030]

[0031] Where a = 1, ..., A represents the position in the attention heads, l = 1, ..., L represents the encoder layer number, p = 1, ..., N represents the patch position in the current video frame, t = 1, ..., F represents the current frame number, and D... h This represents the dimension of each head.

[0032] Step 205: After merging the attention points from all heads, perform a linear projection and feed it into a multilayer perceptron (MLP) to solve the non-linear problem that the attention module cannot handle. After passing through an L-layer encoder module, the position encoded as 0 is taken. The classification results for the current frame are fused together from the classification results of all frames to obtain the final classification result:

[0033]

[0034] in For the output results, l indicates that the current network layer is the final layer, p indicates that the current patch number is the initial number, t indicates the current frame number, p = 1, ..., N; t = 1, ..., F; LN represents a fully connected layer, MLP represents a multilayer perceptron, and MEAN represents averaging multiple results to obtain the final evaluation values ​​of VR video and optical flow video.

[0035] Step 3: Fuse the VR video results and the optical flow video results to obtain the final assessment results of visually induced motion sickness in the VR video.

[0036] Step 301: Using a back-end fusion method, the Softmax outputs of the appearance flow network and the motion flow network are weighted and averaged to smooth the results of each sub-network, as shown below:

[0037]

[0038] Where T is the number of subnetworks, x i For the classification results output by the corresponding sub-network, w i For the corresponding network weights, w i ≥0,

[0039] (III) Beneficial Effects

[0040] This application integrates the Transformer architecture and constructs self-attention mechanisms for both time series and spatial series to capture the interdependencies between sequence elements. Through backend fusion technology, it integrates the outputs of the two-stream network, which maintains computational efficiency and significantly improves the evaluation accuracy of the model. Attached Figure Description

[0041] Figure 1 A schematic diagram illustrating the data preprocessing for VR video extraction;

[0042] Figure 2 This is a flowchart for model evaluation. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Example

[0045] This embodiment applies a visually induced motion sickness assessment method based on attention mechanisms, selecting VR videos from the Padmanaban stereoscopic video database for evaluation. The specific steps are as follows:

[0046] Step 1: Extract optical flow video from the VR video and perform preprocessing, including the following sub-steps:

[0047] Step 101: Segment the VR video into video frames and place them in the 'jpg' folder. Create an index address for each video for easy retrieval later. Extract the optical flow map from consecutive frames of the VR video, using the formula shown below:

[0048]

[0049] Where f(x,y,t) represents the optical flow vector at time t and position (x,y). x and v y These are the components of the optical flow vector in the x and y directions, representing the speed at which pixels in the image move over time in these two directions. and These represent the rate of change of position (x, y) over time in the image sequence.

[0050] The extracted optical flow maps were converted to .jpg format, the same as the video frames, and placed in the 'flow' folder. The resolution of the optical flow maps was 704*448. An index address was created for the optical flow maps of each video for easy retrieval later.

[0051] Step 102: Divide the VR video and optical flow video into N patches of the same size, calculated as follows:

[0052] N = HW / P 2

[0053] Where the video input is X∈RH×W×F H and W represent the length and width of the input video frame. Due to the different resolutions of VR video and optical flow video, the model first reconstructs the input video, adjusting the input frame to 224×224. F represents the frame sampling size of the video, P represents the length and width of the split patch, and N is the number of split patches. After decomposing the input video, it can be represented as a vector. p=1,...,N; t=1,...,F.

[0054] Step 103: Group the patches with the same p but different t into one category and name it Time patch, which represents the time sequence information of the video; group the patches with the same t but different p into another category and name it Space, which represents the spatial sequence information of the video.

[0055] Step 2: Perform independent feature extraction and evaluation on VR video and optical flow video, including the following sub-steps:

[0056] Step 201: Input the time patch into the network's Embedding module to facilitate the subsequent calculation of the temporal correlation between VR video and optical flow video, and add location encoding information, as shown in the following formula:

[0057]

[0058] in The output result is named "token", where 0 represents the current network layer number, p represents the current patch number, and t represents the current frame number, p = 1, ..., N; t = 1, ..., F; Let P be a learnable matrix, representing the length and width of the split patch. This is a learnable spatial location encoding, where D is the dimension of the patch. The processed result is named Timetoken.

[0059] Step 202: Input the Spac patch into the Embedding module of the network to facilitate the subsequent calculation of the spatial sequence correlation between VR video and optical flow video, and add positional encoding information, as shown in the following formula.

[0060]

[0061] in The output result is named "token", where 0 represents the current network layer number, p represents the current patch number, and t represents the current frame number, p = 1, ..., N; t = 1, ..., F; Let P be a learnable matrix, representing the length and width of the split patch. For learnable spatial location encoding, D is the dimension of the patch. The additional location encoding symbolizes the degree of visually induced motion sickness in the video. This processed result is named the Space token.

[0062] Step 203: After preprocessing the data through the Embedding module, the data is input into the core module of the network model: the Encoder module. The Encoder module consists of an Attention layer and a Multilayer Perceptron (MLP) layer. The network model contains L Encoder layers, and each Encoder module contains two Attention modules and one MLP module. The network model is optimized using the stochastic gradient descent algorithm, with a weight decay of 0.0001 during backpropagation, a momentum factor of 0.9, a batch size of 8, and 120 training epochs. Before network training, data augmentation is required. Considering that conventional data augmentation methods such as image blurring, rotation, and flipping may cause visually induced motion sickness, this paper uses a sliding start frame method for data augmentation. When inputting the video, scale jittering is used to scale the shorter side of the input video to a random value in the range [256, 320], and then randomly offset and cropped into a 224×224 image region.

[0063] Step 204: Directly use the attention distribution obtained during the Transformer's attention mechanism calculation to represent the correlation between tokens. Since directly calculating the self-attention distribution of the video's tokens would lead to excessive computation and prolonged training time, this paper first calculates the self-attention distribution of the time series, then calculates the self-attention distribution of the spatial series, obtaining the correlation between all tokens in the video step by step. The processed Time token is input into the first Attention layer to calculate the temporal correlation between the video and the optical flow video. The processed Space token is input into the second Attention layer to calculate the spatial correlation between the video and the optical flow video. The calculation method of the Attention layer is as follows:

[0064]

[0065] Attention(Q,K,V)=AV

[0066] Where Q, K, and V are The set of , where A is the self-attention distribution calculated from Q and K. The calculation method is as follows:

[0067]

[0068]

[0069]

[0070] Where a = 1, ..., A represents the position in the attention heads, l = 1, ..., L represents the number of encoder layers, p = 1, ..., N represents the token position in the current video frame, t = 1, ..., F represents the current frame number, and D... h This represents the dimension of each head.

[0071] Step 205: After merging the attention points from all heads, perform a linear projection and feed it into a multilayer perceptron (MLP) to solve the non-linear problem that the attention module cannot handle. After passing through an L-layer encoder module, the position encoded as 0 is taken. The classification results for the current frame are fused together from the classification results of all frames to obtain the final classification result:

[0072]

[0073] in For the output results, l indicates that the current network layer is the final layer, p indicates that the current patch number is the initial number, t indicates the current frame number, p = 1, ..., N; t = 1, ..., F; LN represents a fully connected layer, MLP represents a multilayer perceptron, and MEAN represents averaging multiple results to obtain the final evaluation values ​​of VR video and optical flow video.

[0074] Step 3: Fuse the VR video results and the optical flow video results to obtain the final assessment results of visually induced motion sickness in the VR video.

[0075] Step 301: Using a back-end fusion method, the Softmax outputs of the appearance flow network and the motion flow network are weighted and averaged to smooth the results of each sub-network, as shown below:

[0076]

[0077] Where T is the number of subnetworks, x i For the classification results output by the corresponding sub-network, w i For the corresponding network weights, w i ≥0,

[0078] To better evaluate video content and provide it to users with varying levels of VIMS adaptation, this paper employs a four-category classification method to more precisely assess participants' experience. The four-category classification is as follows:

[0079]

[0080] VIMS indicates the degree of visually induced motion sickness. None means almost no discomfort, Low indicates mild discomfort, Medium reflects moderate discomfort, and High means extremely severe motion sickness, which is almost unbearable.

[0081] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for assessing visually induced motion sickness based on attention mechanisms, characterized in that: The VR videos selected from the Padmanaban stereoscopic video database were evaluated, and the specific steps are as follows; Step 1: Extract optical flow video from VR video and preprocess it, including splitting VR video and optical flow video into N patches of the same size; Step 2: Perform independent feature extraction and evaluation on VR video and optical flow video; Step 2 includes the following sub-steps: Step 201: Input the Time patch into the Embedding module of the network to facilitate the subsequent calculation of the temporal correlation between VR video and optical flow video, and add location encoding information, as shown in the following formula: ; in The output result is named "token," where 0 represents the current network layer number. This indicates the current patch number. Indicates the current frame number. =1, ..., N; =1, ..., F; Let P be a learnable matrix, representing the length and width of the split patch. For learnable spatial location encoding, D is the dimension of the patch. Represents a video vector; Step 202: Input the Spac patch into the Embedding module of the network to facilitate the subsequent calculation of the spatial sequence correlation between VR video and optical flow video, and add positional encoding information, as shown in the following formula. ; in The output result is named "token," where 0 represents the current network layer number. This indicates the current patch number. Indicates the current frame number. =1, ..., N; =1, ..., F; Let P be a learnable matrix, representing the length and width of the split patch. For learnable spatial location encoding, D is the dimension of the patch. The additional location code represents the overall classification result; Step 203: After preprocessing the data through the Embedding module, the data is input into the core module of the network model: the Encoder module; the Encoder module consists of Attention layers and Multilayer Perceptron (MLP) layers; the network model contains L Encoder layers, and each Encoder module contains 2 Attention modules and one MLP module; Step 204: The Attention layer in each Encoder layer is calculated as follows: ; ; in, , , for , , The set, for and The self-attention distribution obtained through calculation; where , , The calculation method is as follows: ; ; ; in =1, ..., A represents the position in the attention heads. =1, ..., L represents the number of encoder layers. =1, ..., N represent the patch positions in the current video frame. =1, ..., F represents the current frame number. This represents the dimension of each head; Step 205: After merging the attention points from all heads, perform a linear projection and feed it into a multilayer perceptron (MLP) to solve the non-linear problem that the attention module cannot handle; after passing through an L-layer encoder module, take the position encoded as 0. The classification results for the current frame are fused together from the classification results of all frames to obtain the final classification result: ; in For the output results, This indicates that the current network layer is the final layer. This indicates that the current patch count is the initial number. Indicates the current frame number. =1, ..., N; =1, ..., F; LN represents a fully connected layer, MLP represents a multilayer perceptron, and MEAN represents averaging multiple results to obtain the final evaluation values ​​of VR video and optical flow video; Step 3: Fuse the VR video results and the optical flow video results to obtain the final assessment results of visually induced motion sickness in the VR video.

2. The method for assessing visually induced motion sickness based on attention mechanisms according to claim 1, characterized in that: Step 1 includes the following sub-steps: Step 101: Extract optical flow maps from consecutive frames of the VR video, as shown in the following formula: ; in, Indicates time and location The optical flow vector at that location; and These are the components of the optical flow vector in the x and y directions, representing the speed at which pixels in the image move over time in these two directions; and Representing positions respectively Rate of change over time in the image sequence; Step 102: Divide the VR video and optical flow video into N patches of the same size, calculated as follows: ; Among them, the video input is H and W represent the length and width of the input video frame, which are adjusted to 224×224 here. F represents the frame sampling size of the video, P represents the length and width of the split patch, and N is the number of split patches. After the input video is decomposed, it can be represented as a vector. , =1, ..., N; =1, ...,F; Step 103: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] same Different patches are grouped together and named Time patch. different Patches with the same characteristics are grouped together and named Time patch.

3. The method for assessing visually induced motion sickness based on attention mechanisms according to claim 1, characterized in that: Step 3 includes the following sub-steps: Step 301: Using a back-end fusion method, the Softmax outputs of the appearance flow network and the motion flow network are weighted and averaged to smooth the results of each sub-network, as shown below: ; Where T is the number of subnetworks. The classification results are output by the corresponding sub-network. For the corresponding network weights, and , .

Images