Head-mounted device for motion sickness information detection and computer readable medium

Abstract

The embodiment of the invention discloses a head-mounted device for motion sickness information detection and a computer readable medium. The head-mounted device for motion sickness information detection comprises an image acquisition device configured to acquire a visual image data sequence; the processor is configured to execute the following processing: obtaining a preprocessed image data sequence; generating depth image data; generating optical flow image data; generating a horizontal feature map, a vertical feature map and a depth feature map; obtaining time sequence characteristic information; generating a motion sickness detection result; determining information to be displayed; a memory configured to store the motion sickness detection result; and the display is configured to display the to-be-displayed information. According to the embodiment, computing resources consumed during data processing can be reduced.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technologies, and more particularly, to a head-mounted device and a computer-readable medium for detecting motion sickness information. Background Art

[0002] Motion sickness is a common physiological reaction of users in a virtual reality environment. Detecting the degree of motion sickness symptoms of users in virtual reality and providing timely relief measures for users is an important technology to improve the user experience. Currently, when detecting motion sickness, the commonly used method is to capture the physiological reactions of users in a virtual reality environment by detecting the electroencephalogram, galvanic skin response, respiratory rate, etc. of users, so as to predict the motion sickness of users.

[0003] However, when detecting motion sickness in the above manner, the following technical problems often exist: The method of predicting motion sickness of users by detecting the electroencephalogram, galvanic skin response, respiratory rate, etc. of users requires users to wear special contact devices, which is likely to reduce the virtual reality experience of users. And when detecting, it is necessary to synchronously process various data such as the electroencephalogram, galvanic skin response, and respiratory rate of users, which is likely to cause a large consumption of computing resources during processing.

[0004] The above information disclosed in this background art section is only used to enhance the understanding of the background of the concept of the present disclosure, and thus, it may include information that does not form the prior art known to those of ordinary skill in the art in this country. Summary of the Invention

[0005] The content part of the present disclosure is used to introduce concepts in a brief form, and these concepts will be described in detail in the following detailed implementation part. The content part of the present disclosure is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

[0006] Some embodiments of the present disclosure propose a head-mounted device and a computer-readable medium for detecting motion sickness information to solve one or more of the technical problems mentioned in the above background art section.

[0007] In a first aspect, some embodiments of the present disclosure provide a head-mounted device for detecting motion sickness information. The head-mounted device for detecting motion sickness information includes: an image acquisition device configured to acquire a sequence of visual image data corresponding to a target user; a processor configured to perform the following processes: preprocess the sequence of visual image data to obtain a sequence of preprocessed image data; for each preprocessed image data in the sequence of preprocessed image data, generate depth image data based on the preprocessed image data; for every two frames of preprocessed image data in the sequence of preprocessed image data that meet a preset image condition, generate optical flow image data based on the two frames of preprocessed image data; generate a horizontal feature map, a vertical feature map, and a depth feature map based on the generated depth image data, the generated optical flow image data, and a feature extraction layer in a pre-trained motion sickness detection model, where the feature extraction layer includes a horizontal feature extraction network, a vertical feature extraction network, and a depth feature extraction network; input the horizontal feature map, the vertical feature map, and the depth feature map into a temporal feature extraction network in the motion sickness detection model to obtain temporal feature information; generate a motion sickness detection result based on the temporal feature information; determine information to be displayed in response to determining that the motion sickness detection result meets a preset detection result condition; a memory configured to store the motion sickness detection result; and a display configured to display the information to be displayed.

[0008] In a second aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, where the program, when executed by a processor, implements the method described in any implementation manner of the first aspect above.

[0009] The above-mentioned head-mounted device for motion sickness information detection according to the present disclosure has the following beneficial effects: By using the head-mounted device for motion sickness information detection according to the present disclosure, the consumption of computing resources during processing can be reduced. Specifically, the reason for the relatively large consumption of computing resources is that the method of predicting a user's motion sickness by detecting the user's electroencephalogram, galvanic skin response, respiratory rate, etc. requires the user to wear a dedicated contact device, which is likely to reduce the user's virtual reality experience. Moreover, when detecting, it is necessary to synchronously process various data such as the user's electroencephalogram, galvanic skin response, and respiratory rate, which is likely to result in a relatively large consumption of computing resources during processing. Based on this, the head-mounted device for motion sickness information detection according to the present disclosure, first, an image acquisition device, is configured to acquire a sequence of visual image data corresponding to a target user. Thus, the original data to be processed can be obtained. Secondly, a processor, is configured to perform the following processing: First, preprocess the above-mentioned sequence of visual image data to obtain a sequence of preprocessed image data. Thus, the original data can be preprocessed to reduce the interference of noise information in the data on subsequent processing. Secondly, for each preprocessed image data in the above-mentioned sequence of preprocessed image data, based on the above-mentioned preprocessed image data, generate depth image data. Thus, a depth map corresponding to the sequence of preprocessed image data can be obtained. Then, for every two frames of preprocessed image data in the above-mentioned sequence of preprocessed image data that meet the preset image conditions, based on the above-mentioned two frames of preprocessed image data, generate optical flow image data. Thus, an optical flow field map corresponding to the sequence of preprocessed image data can be obtained. Then, based on the generated respective depth image data, the generated respective optical flow image data, and the feature extraction layer in the pre-trained motion sickness detection model, generate a horizontal feature map, a vertical feature map, and a depth feature map, wherein the above-mentioned feature extraction layer includes a horizontal feature extraction network, a vertical feature extraction network, and a depth feature extraction network. Thus, respective feature maps corresponding to the sequence of preprocessed image data can be obtained. Then, input the above-mentioned horizontal feature map, the above-mentioned vertical feature map, and the above-mentioned depth feature map into the temporal feature extraction network in the above-mentioned motion sickness detection model to obtain temporal feature information. Thus, temporal features corresponding to the sequence of preprocessed image data can be obtained. Then, based on the above-mentioned temporal feature information, generate a motion sickness detection result. Thus, the motion sickness detection result of the user can be obtained through the temporal features. Then, in response to determining that the above-mentioned motion sickness detection result meets the preset detection result conditions, determine the information to be displayed. Then, a memory, is configured to store the above-mentioned motion sickness detection result. Thus, the motion sickness detection result can be stored. Finally, a display, is configured to display the above-mentioned information to be displayed. Thus, mitigation measures can be provided to users with relatively severe symptoms.Also, since it is possible to predict a user's motion sickness by obtaining a sequence of the user's visual image data and processing the sequence of visual image data, without the user having to additionally wear a dedicated contact device, the probability of a lower virtual experience of the user caused by wearing a dedicated contact device can be reduced. Also, since it is possible to directly predict a user's motion sickness through a pre-trained motion sickness detection model, without having to simultaneously process multiple types of data such as the user's electroencephalogram, galvanic skin response, and respiratory rate, the consumption of computing resources during processing can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and the elements and elements are not necessarily drawn to scale.

[0011] Figure 1 is a schematic structural diagram of some embodiments of a head-mounted device for detecting motion sickness information according to the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0012] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the scope of protection of the present disclosure.

[0013] In addition, it should be noted that only parts related to the relevant invention are shown in the drawings for the sake of convenience of description. Without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other.

[0014] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules, or units, and are not used to limit the order of the functions performed by these devices, modules, or units or their interdependent relationships.

[0015] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise clearly specified in the context, it should be understood as "one or more".

[0016] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0017] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0018] Figure 1 The structural schematic diagrams of some embodiments of a head-mounted device for detecting motion sickness information according to the present disclosure are shown. Figure 1 It includes an image acquisition device 1, a processor 2, a memory 3, and a display 4.

[0019] In some embodiments, the above-mentioned image acquisition device 1 may be configured to acquire a sequence of visual image data corresponding to a target user. Among them, the above-mentioned image acquisition device 1 may be a device for acquiring the above-mentioned sequence of visual image data. The above-mentioned image acquisition device 1 may include a camera and an attitude sensor. The above-mentioned attitude sensor may be a sensor capable of real-time monitoring of the head attitude of the target user in a virtual scene. For example, the above-mentioned attitude sensor may be an inertial measurement unit. Each visual image data in the above-mentioned sequence of visual image data may be an image corresponding to the virtual scene observed by the user when wearing the above-mentioned head-mounted device. The above-mentioned sequence of visual image data may be a continuous frame corresponding to the virtual scene. Each visual image data in the above-mentioned sequence of visual image data corresponds to an image width and an image height. The sizes of the respective visual image data in the above-mentioned sequence of visual image data are the same. The above-mentioned target user may be the user wearing the above-mentioned head-mounted device. The above-mentioned head-mounted device may be a device capable of constructing a virtual scene for the user and detecting the degree of dizziness of the user in the virtual scene. The above-mentioned head-mounted device may include the above-mentioned image acquisition device 1, processor 2, memory 3, and display 4. For example, the above-mentioned head-mounted device may be a VR glasses. In practice, the above-mentioned image acquisition device 1 may acquire a sequence of visual image data corresponding to the target user through the camera.

[0020] In some embodiments, the above-mentioned processor 2 may be configured to perform the following processing: First, preprocess the sequence of visual image data to obtain a sequence of preprocessed image data.

[0021] In some embodiments, the above-mentioned processor 2 may preprocess the above-mentioned visual image data sequence to obtain a preprocessed image data sequence. Among them, each preprocessed image data in the above-mentioned preprocessed image data sequence may be preprocessed visual image data. In practice, first, the above-mentioned processor 2 may perform normalization processing on the above-mentioned visual image data sequence through a normalization algorithm to obtain a normalized visual image data sequence as a normalized image data sequence. Then, the above-mentioned normalized image data sequence may be denoised through an image denoising algorithm to obtain a denoised normalized image data sequence as a preprocessed image data sequence. Among them, the above-mentioned normalization algorithm may be an algorithm capable of performing normalization processing on images. For example, the above-mentioned normalization algorithm may be Min-Max normalization (Min-Max Normalization). The above-mentioned image denoising algorithm may be an algorithm capable of denoising images. For example, the above-mentioned image denoising algorithm may be a Gaussian filtering method.

[0022] Second, for each preprocessed image data in the preprocessed image data sequence, based on the preprocessed image data, depth image data is generated.

[0023] In some embodiments, the above-mentioned processor 2 may, for each preprocessed image data in the above-mentioned preprocessed image data sequence, generate depth image data based on the above-mentioned preprocessed image data. Among them, the above-mentioned depth image data may be a depth map corresponding to the preprocessed image data.

[0024] In some optional implementation manners of some embodiments, the above-mentioned processor 2 may generate depth image data based on the above-mentioned preprocessed image data through the following steps: First step, perform feature extraction processing on the above-mentioned preprocessed image data to obtain a group of preprocessed feature maps. Among them, each preprocessed feature map in the above-mentioned group of preprocessed feature maps may be a feature map corresponding to the preprocessed image data. Each preprocessed feature map in the above-mentioned group of preprocessed feature maps may be a feature map of different dimensions corresponding to the above-mentioned preprocessed image data. In practice, the above-mentioned processor 2 may input the above-mentioned preprocessed image data into a feature extractor to obtain a group of preprocessed feature maps. Among them, the above-mentioned feature extractor may be a variational autoencoder.

[0025] Second step, for each preprocessed feature map in the above-mentioned group of preprocessed feature maps, perform the following steps: The first sub-step is to perform downsampling on a preset random noise map to obtain a downsampled noise map corresponding to the above-mentioned preprocessed feature map. Among them, the above-mentioned random noise map can be an image rendered by randomly generated pixel values. The above-mentioned downsampled noise map can be a randomly generated noise map after downsampling. The size of the above-mentioned downsampled noise map is the same as that corresponding to the above-mentioned preprocessed feature map. In practice, the above-mentioned processor 2 can reduce the above-mentioned random noise map to the size corresponding to the above-mentioned preprocessed feature map through image pooling technology to perform downsampling on the above-mentioned random noise map and obtain the downsampled random noise map as the downsampled noise map. Among them, the above-mentioned image pooling technology can be a technology capable of performing pooling processing on images. For example, the above-mentioned image pooling technology can be max pooling.

[0026] The second sub-step is to perform image splicing on the above-mentioned downsampled noise map and the above-mentioned preprocessed feature map to obtain a noise-spliced feature map. Among them, the above-mentioned noise-spliced feature map can be a preprocessed feature map with a downsampled noise map added. In practice, the above-mentioned processor 2 can input the above-mentioned downsampled noise map and the above-mentioned preprocessed feature map into a feature splicing function to obtain a noise-spliced feature map. Among them, the above-mentioned feature splicing function can be a function capable of splicing different feature maps. For example, the above-mentioned feature splicing function can be the concat function.

[0027] The third step is to determine each of the obtained noise-spliced feature maps as a group of noise-spliced feature maps.

[0028] Step 4: Generate depth image data based on the above noise-stitching feature map group. Among them, the above depth image data may be the depth map corresponding to the above preprocessed image data. In practice, first, the above processor 2 may perform sampling processing on each noise-stitching feature map in the above noise-stitching feature map group through an interpolation sampling technique to obtain each noise-stitching feature map after sampling processing as a sampled stitching feature map group. Among them, the sizes of the sampled stitching feature maps in the above sampled stitching feature map group are the same. The above interpolation sampling technique may be a technique capable of scaling the size of an image. For example, the above interpolation sampling technique may be bilinear interpolation upsampling. Then, the number of the above sampled stitching feature map group may be determined as the number of feature maps. Then, the reciprocal of the number of the above feature maps may be determined as the feature map ratio data. Then, for each sampled stitching feature map in the above sampled stitching feature map group, the product of the above sampled stitching feature map and the above feature map ratio data may be determined as a sampled ratio feature map through an image processing software. Then, the sum of the determined sampled ratio feature maps may be determined as the target feature map through the above image processing software. Among them, the above image processing software may be software capable of performing operations on images. For example, the above image processing software may be HALCON machine vision software. Finally, the above target feature map may be input into a convolutional layer to obtain depth image data.

[0029] Third, for every two frames of preprocessed image data in the preprocessed image data sequence that meet the preset image conditions, generate optical flow image data based on the two frames of preprocessed image data.

[0030] In some embodiments, the above processor 2 may, for every two frames of preprocessed image data in the preprocessed image data sequence that meet the preset image conditions, generate optical flow image data based on the two frames of preprocessed image data. Among them, the above optical flow image data may be an image used to represent the optical flow field corresponding to the two frames of preprocessed image data. The above preset image conditions may be that two frames of preprocessed image data are adjacent.

[0031] In some optional implementation manners of some embodiments, the above processor 2 may generate optical flow image data based on the two frames of preprocessed image data through the following steps: The first step is to perform image conversion processing on the above two frames of pre-processed image data to obtain two frames of visual grayscale image data. Among them, each frame of visual grayscale image data in the above two frames of visual grayscale image data can be a grayscale image of the corresponding pre-processed image data. In practice, the above processor 2 can convert the above two frames of pre-processed image data into a grayscale image through a grayscale image conversion function to obtain two frames of visual grayscale image data. Among them, the above grayscale image conversion function can be a function that can convert a color image into a grayscale image. For example, the above grayscale image conversion function can be the cvtColor function in OpenCV.

[0032] In the second step, the visual grayscale image data satisfying the preset image frame condition in the two frames of visual grayscale image data is determined as the first grayscale image data, wherein the image frame condition may be that the pre-processed image data corresponding to the visual grayscale image data ranks higher in the pre-processed image data sequence.

[0033] In the third step, the visual grayscale image data of the two frames of visual grayscale image data that does not meet the above image frame condition is determined as the second grayscale image data.

[0034] In the fourth step, for each pixel point included in the first grayscale image data, the following steps are performed: The first sub-step is to determine the above pixel point as the first pixel point.

[0035] The second sub-step is to determine the first pixel matrix corresponding to the first pixel point based on the first grayscale image data and the first pixel point. The first pixel matrix may be a third-order matrix composed of the pixel value corresponding to the first pixel point and the adjacent pixel values ​​corresponding to the first pixel point. Each of the adjacent pixel values ​​may be a pixel value corresponding to a pixel point adjacent to the first pixel point. For example, the adjacent pixel values ​​may be 8 pixel values ​​corresponding to 8 pixel points adjacent to the first pixel point with the first pixel point as the center. In practice, when there is a null value in each of the adjacent pixel values ​​corresponding to the first pixel point, the processor 2 may determine the preset fill value as the element corresponding to the null value. The preset fill value may be a pre-set value. For example, the preset fill value may be 0.

[0036] The third sub-step is to generate a first horizontal spatial gradient and a first vertical spatial gradient based on a preset horizontal operator matrix, a preset vertical operator matrix, and the above-mentioned first pixel matrix. Among them, the above-mentioned horizontal operator matrix can be a third-order matrix for detecting the horizontal edges of an image. For example, the above-mentioned horizontal operator matrix can be the convolution kernel of the Sobel operator in the horizontal direction. The above-mentioned vertical operator matrix can be a third-order matrix for detecting the vertical edges of an image. For example, the above-mentioned vertical operator matrix can be the convolution kernel of the Sobel operator in the vertical direction. The above-mentioned first horizontal spatial gradient can be the gradient amplitude in the horizontal direction corresponding to the above-mentioned first pixel point. The above-mentioned first vertical spatial gradient can be the gradient amplitude in the vertical direction corresponding to the above-mentioned first pixel point.

[0037] In practice, first, the above-mentioned processor 2 can determine the Hadamard product of the above-mentioned horizontal operator matrix and the above-mentioned first pixel matrix as the first horizontal matrix. Then, the sum of the elements of each matrix in the above-mentioned first horizontal matrix can be determined as the first horizontal spatial gradient.

[0038] In practice, first, the above-mentioned processor 2 can determine the Hadamard product of the above-mentioned vertical operator matrix and the above-mentioned first pixel matrix as the first vertical matrix. Then, the sum of the elements of each matrix in the above-mentioned first vertical matrix can be determined as the first vertical spatial gradient.

[0039] The fourth sub-step is to determine the pixel points corresponding to the above-mentioned first pixel point among the pixel points included in the above-mentioned second grayscale image data as the second pixel points. As an example, when the position of the above-mentioned first pixel point in the above-mentioned first grayscale image data is at an image height of 41 and an image width of 23, the second pixel point is the pixel point at an image height of 41 and an image width of 23 in the above-mentioned second grayscale image data.

[0040] The fifth sub-step is to determine the difference between the pixel value corresponding to the above-mentioned first pixel point and the pixel value corresponding to the above-mentioned second pixel point as the pixel difference data.

[0041] The sixth sub-step is to construct optical flow non-linear information based on a preset custom horizontal vector, a preset custom vertical vector, the above-mentioned first horizontal spatial gradient, the above-mentioned first vertical spatial gradient, and the above-mentioned pixel difference data. Among them, the above-mentioned custom horizontal vector can be an unknown representing the motion speed of the above-mentioned first pixel point in the horizontal direction. For example, the above-mentioned custom horizontal vector can be represented by u. The above-mentioned custom vertical vector can be an unknown representing the motion speed of the above-mentioned first pixel point in the vertical direction. For example, the above-mentioned custom vertical vector can be represented by v. The above-mentioned optical flow non-linear information can be an equation composed of the above-mentioned custom horizontal vector, the above-mentioned custom vertical vector, the above-mentioned first horizontal spatial gradient, the above-mentioned first vertical spatial gradient, and the above-mentioned pixel difference data.

[0042] In practice, first, the above-mentioned processor 2 can determine the product of the above-mentioned custom horizontal vector and the above-mentioned first horizontal spatial gradient as the horizontal term. Secondly, the product of the above-mentioned custom vertical vector and the above-mentioned first vertical spatial gradient can be determined as the vertical term. Then, the negative of the above-mentioned pixel difference data can be determined as the negative difference data. Finally, the sum of the above-mentioned horizontal term and the above-mentioned vertical term can be made equal to the above-mentioned negative difference data, thereby constructing an equation as the optical flow non-linear information.

[0043] The seventh sub-step is to determine each pixel point corresponding to the above-mentioned first pixel point among each pixel point included in the above-mentioned first grayscale image data as each target pixel point. Among them, each target pixel point among the above-mentioned target pixel points can be a pixel point adjacent to the above-mentioned first pixel point.

[0044] The eighth sub-step is to determine each target optical flow non-linear information corresponding to each of the above-mentioned target pixel points based on each of the above-mentioned target pixel points. Among them, each target optical flow non-linear information among the above-mentioned target optical flow non-linear information can be the optical flow non-linear information corresponding to the target pixel point. In practice, for each target pixel point among the above-mentioned target pixel points, the above-mentioned processor 2 can generate the optical flow non-linear information corresponding to the target pixel point as the target optical flow non-linear information. The method of generating the optical flow non-linear information corresponding to the target pixel point can refer to the specific implementation manner of generating the optical flow non-linear information corresponding to the above-mentioned first pixel point, which will not be elaborated here.

[0045] The ninth sub-step is to generate the optical flow horizontal data and the optical flow vertical data corresponding to the above-mentioned first pixel point based on the above-mentioned optical flow non-linear information and each of the above-mentioned target optical flow non-linear information. Among them, the above-mentioned optical flow horizontal data can be the solution of the above-mentioned custom horizontal vector. The above-mentioned optical flow vertical data can be the solution of the above-mentioned custom vertical vector. In practice, the above-mentioned processor 2 can solve the above-mentioned optical flow non-linear information and each of the above-mentioned target optical flow non-linear information by the polynomial fitting method to obtain the solution of the above-mentioned custom horizontal vector and the solution of the above-mentioned custom vertical vector. Among them, the above-mentioned polynomial fitting method can be the least squares method. Then, the solution of the above-mentioned custom horizontal vector can be determined as the optical flow horizontal data corresponding to the above-mentioned first pixel point. Then, the solution of the above-mentioned custom vertical vector can be determined as the optical flow vertical data corresponding to the above-mentioned first pixel point.

[0046] Step 5: Generate an optical flow level matrix based on the generated optical flow level data. Among them, the above optical flow level matrix can be a matrix composed of the above optical flow level data. In practice, the above processor 2 can determine the image matrix corresponding to the above first grayscale image data as the first image matrix. Then, according to the positions of the respective pixel points in the above first grayscale image data in the above first image matrix, the above optical flow level data can be combined into an optical flow level matrix. For example, when the position of the pixel point corresponding to the optical flow level data in the above first image matrix is the second column of the third row, the optical flow level data is the element corresponding to the position of the second column of the third row in the optical flow level matrix.

[0047] Step 6: Generate an optical flow vertical matrix based on the generated optical flow vertical data. Among them, the above optical flow vertical matrix can be a matrix composed of the above optical flow vertical data. In practice, the above processor 2 can combine the above optical flow vertical data into an optical flow vertical matrix according to the positions of the respective pixel points in the above first grayscale image data in the above first image matrix.

[0048] Step 7: Generate optical flow image data based on the above-mentioned optical flow horizontal matrix and the above-mentioned optical flow vertical matrix. In practice, for each element included in the above-mentioned optical flow horizontal matrix, first, the above-mentioned processor 2 can determine the above-mentioned element as the target horizontal element. Then, the element corresponding to the above-mentioned target horizontal element among the elements included in the above-mentioned optical flow vertical matrix can be determined as the element to be processed. Then, the square of the above-mentioned target horizontal element can be determined as the target square data. Then, the square of the above-mentioned element to be processed can be determined as the to-be-processed square data. Then, the sum of the above-mentioned target square data and the above-mentioned to-be-processed square data can be determined as the amplitude data corresponding to the above-mentioned target horizontal element. Then, the above-mentioned target horizontal element and the above-mentioned element to be processed are input into the two-parameter arctangent function, and the data output by the above-mentioned two-parameter arctangent function is obtained as the direction data corresponding to the above-mentioned target horizontal element. Finally, for the determined amplitude data, according to the positions of the above-mentioned amplitude data corresponding to the respective target horizontal elements in the above-mentioned optical flow horizontal matrix, the above-mentioned amplitude data are combined into a matrix as the optical flow amplitude matrix. For the obtained direction data, according to the positions of the target horizontal elements corresponding to the above-mentioned direction data in the above-mentioned optical flow horizontal matrix, the above-mentioned direction data are combined into a matrix as the optical flow direction matrix. Then, the above-mentioned optical flow amplitude matrix is input into the amplitude normalization function, so that the element values in the above-mentioned optical flow amplitude matrix are normalized to within [0, 255], and the normalized optical flow amplitude matrix is obtained as the normalized amplitude matrix. Among them, the above-mentioned amplitude normalization function can be a function that can normalize the element values in the matrix to within [0, 255]. For example, the above-mentioned amplitude normalization function can be the Cv2.Normalize function. Then, each element value in the above-mentioned optical flow direction matrix is normalized to within [0, 180] through the angle normalization function, and the normalized optical flow direction matrix is obtained as the normalized direction matrix. Among them, the above-mentioned angle normalization function can be a function that normalizes the elements in the matrix to within [0, 180]. For example, the above-mentioned angle normalization function can be the Cv2.cartToPolar function. Then, the above-mentioned normalized amplitude matrix, the above-mentioned normalized direction matrix, and the preset saturation matrix are input into the merging function to obtain the merged image. Among them, the above-mentioned merged image can be the HSV image corresponding to the above-mentioned normalized amplitude matrix and the above-mentioned normalized direction matrix. The above-mentioned merging function can be a function that can convert the above-mentioned normalized amplitude matrix and the above-mentioned normalized direction matrix into an HSV image. For example, the above-mentioned merging function can be the cv2.merge function. The above-mentioned saturation matrix can be a matrix with the same size as the above-mentioned optical flow horizontal matrix and all element values being the preset saturation value. The above-mentioned preset saturation value can be a numerical value between [0, 255].Finally, the merged image can be input into a color space conversion function to obtain a merged image converted into RGB format as optical flow image data. The color space conversion function can be a function that can convert an HSV image into an RGB image. For example, the color space conversion function can be a cv2.cvtColor function.

[0049] In the process of adopting technical solutions to solve the above technical problems, the following problems are often accompanied: Methods that predict motion sickness through the user's EEG and other data often require denoising, filtering, data fusion, and other processing of the detected EEG, galvanic skin response, respiratory rate and other data, which can easily lead to high computational complexity during data processing, and thus lead to large consumption of computing resources during data processing.

[0050] Faced with the above technical problems, we decided to adopt the following solutions: In some optional implementations of some embodiments, the processor 2 may generate optical flow image data based on the two frames of pre-processed image data by the following steps: In the first step, the pre-processed image data satisfying the preset image processing condition in the two frames of pre-processed image data is determined as the first pre-processed visual image data, wherein the image processing condition may be that the pre-processed image data ranks higher in the pre-processed image data sequence.

[0051] In the second step, the pre-processed image data of the two frames of pre-processed image data that does not meet the above image processing conditions is determined as the second pre-processed visual image data.

[0052] The third step is to perform grayscale conversion processing on the first preprocessed visual image data and the second preprocessed visual image data to obtain a first preprocessed visual grayscale image and a second preprocessed visual grayscale image. The first preprocessed visual grayscale image may be a grayscale image corresponding to the first preprocessed visual image data. The second preprocessed visual grayscale image may be a grayscale image corresponding to the second preprocessed visual image data. In practice, the processor 2 may input the first preprocessed visual image data and the second preprocessed visual image data into the grayscale image conversion function, respectively, to obtain a first preprocessed visual grayscale image and a second preprocessed visual grayscale image.

[0053] Step 4: Perform image transformation processing on the above first preprocessed visual grayscale image and the above second preprocessed visual grayscale image to obtain a first visual spectrum matrix and a second visual spectrum matrix. Among them, the above first visual spectrum matrix can be a complex matrix corresponding to the above first preprocessed visual grayscale image. The above second visual spectrum matrix can be a complex matrix corresponding to the above second preprocessed visual grayscale image. In practice, the above processor 2 can perform image transformation processing on the above first preprocessed visual grayscale image through an image transformation technique to obtain a first visual spectrum matrix. The above second preprocessed visual grayscale image can be subjected to image transformation processing through the above image transformation technique to obtain a second visual spectrum matrix. Among them, the above image transformation technique can be a fast Fourier transform technique.

[0054] Step 5: Generate a conjugate spectrum matrix based on the above second visual spectrum matrix. Among them, the above conjugate spectrum matrix can be a conjugate matrix corresponding to the above second visual spectrum matrix. In practice, the above processor 2 can convert the above second visual spectrum matrix into a conjugate spectrum matrix through a matrix conversion function. Among them, the above matrix conversion function can be a function capable of converting a complex matrix into a corresponding conjugate matrix. For example, the above matrix conversion function can be the conj function.

[0055] Step 6: Determine the product of the above first visual spectrum matrix and the above conjugate spectrum matrix as the conjugate visual spectrum matrix.

[0056] Step 7: Generate a first amplitude matrix based on the above first visual spectrum matrix. Among them, the above first amplitude matrix can be an amplitude spectrum corresponding to the above first visual spectrum matrix. In practice, the above processor 2 can input the above first visual spectrum matrix into an absolute value function to obtain a first amplitude matrix. Among them, the above absolute value function can be a function capable of generating an amplitude spectrum of a complex matrix. For example, the above absolute value function can be the abs function.

[0057] Step 8: Generate a second amplitude matrix based on the above second visual spectrum matrix. Among them, the above second amplitude matrix can be an amplitude spectrum corresponding to the above second visual spectrum matrix. In practice, the above processor 2 can input the above second visual spectrum matrix into the above absolute value function to obtain a second amplitude matrix.

[0058] Step 9: Determine the product of the above first amplitude matrix and the above second amplitude matrix as the visual amplitude matrix.

[0059] Step 10: Determine the inverse of the above visual amplitude matrix as the visual amplitude inverse matrix.

[0060] Step 11: Determine the product of the above conjugate visual spectrum matrix and the above visual amplitude inverse matrix as the visual amplitude spectrum matrix.

[0061] The twelfth step is to generate a visual phase matrix based on the above visual amplitude spectrum matrix. Among them, the above visual phase matrix can be the real value matrix corresponding to the above visual amplitude spectrum matrix. In practice, first, the above processor 2 can perform an inverse transformation process on the above visual amplitude spectrum matrix through an inverse transformation technology to obtain a transformed complex matrix as the inverse transformation matrix. Among them, the above inverse transformation technology can be an inverse fast Fourier transform technology. Then, the above inverse transformation matrix can be input into the above absolute value function to obtain the real value matrix corresponding to the above visual amplitude spectrum matrix as the visual phase matrix.

[0062] The thirteenth step is to determine the first peak data and the second peak data based on the above visual phase matrix. Among them, the above first peak data can be the row number corresponding to the element with the largest value in the above visual phase matrix. The above second peak data can be the column number corresponding to the element with the largest value in the above visual phase matrix. In practice, first, the above processor 2 can determine the element with the largest corresponding value in the above visual phase matrix as the target element, and then, the row number corresponding to the above target element can be determined as the first peak data. Then, the column number corresponding to the above target element can be determined as the second peak data.

[0063] The fourteenth step is to generate horizontal translation data and vertical translation data corresponding to the above first preprocessed visual image data based on the above first peak data and the above second peak data. Among them, the above horizontal translation data can represent the offset of the above first preprocessed visual image data relative to the above second preprocessed visual image data in the horizontal direction. The above vertical translation data can represent the offset of the above first preprocessed visual image data relative to the above second preprocessed visual image data in the vertical direction. In practice, first, the above processor 2 can determine the image height and image width corresponding to the above first preprocessed visual image data as the to-be-processed height data and the to-be-processed width data respectively. Secondly, the ratio of the above to-be-processed height data to a preset value can be determined as the central height data. The ratio of the above to-be-processed width data to the above preset value can be determined as the central width data. Then, the difference between the above first peak data and the above central height data can be determined as the vertical translation data. The difference between the above second peak data and the above central width data can be determined as the horizontal translation data.

[0064] Step 15: The above first pre-processed visual grayscale image is block-processed to obtain respective grayscale image blocks. Each of the grayscale image blocks among the above respective grayscale image blocks can be an image block corresponding to the first pre-processed visual grayscale image. In practice, the above processor 2 can input the above first pre-processed visual grayscale image into an image block function to obtain respective grayscale image blocks. Among them, the above image block function can be a function capable of block-processing an image. For example, the above image block function can be the view_as_blocks function.

[0065] Step 16: Based on the above horizontal translation data and the above vertical translation data, each of the above grayscale image blocks is initialized to obtain respective initialized image blocks. Each of the initialized image blocks among the above respective initialized image blocks can be a grayscale image block whose corresponding motion vector is confirmed as the above horizontal translation data and the above vertical translation data. In practice, for each of the grayscale image blocks among the above respective grayscale image blocks, the above processor 2 can determine the above horizontal translation data and the above vertical translation data as the motion vector corresponding to the above grayscale image block to initialize the above grayscale image block and obtain an initialized image block.

[0066] Step 17: For each of the above initialized image blocks among the above respective initialized image blocks, image block horizontal data and image block vertical data corresponding to the above initialized image block are generated. Among them, the above image block horizontal data can be the horizontal translation data corresponding to the above initialized image block. The above image block vertical data can be the vertical translation data corresponding to the above initialized image block. In practice, for each of the above initialized image blocks among the above respective initialized image blocks, the above processor 2 can take the generated horizontal translation data corresponding to the above initialized image block as the initialized horizontal data. The vertical translation data corresponding to the above initialized image block can be generated as the initialized vertical data. The manner of generating the horizontal translation data and the vertical translation data corresponding to the above initialized image block can refer to the specific implementation manner of generating the horizontal translation data and the vertical translation data corresponding to the above first pre-processed visual image data, which will not be elaborated here. Then, the above initialized horizontal data can be determined as the image block horizontal data. The above initialized vertical data can be determined as the image block vertical data.

[0067] Step 18: Based on the generated respective image block horizontal data and the generated respective image block vertical data, optical flow image data is generated.

[0068] In practice, first, for the horizontal data of each of the above image blocks, the processor 2 can combine the horizontal data of each of the above image blocks into a matrix as a local horizontal displacement matrix according to the corresponding positions of the above initial image blocks in the above first preprocessed visual grayscale image. Then, according to the corresponding positions of the above initial image blocks in the above first preprocessed visual grayscale image, the vertical data of each of the above image blocks can be combined into a matrix as a local vertical displacement matrix. Then, for each element among the elements included in the above local horizontal displacement matrix, first, the processor 2 can determine the product of a preset first ratio data and the above element as local horizontal element data. Then, the product of the above horizontal translation data and a preset second ratio data can be determined as horizontal element data. Then, the sum of the above local horizontal element data and the above horizontal element data can be determined as the horizontal data to be combined. Finally, according to the corresponding positions of the above elements in the above local horizontal displacement matrix, the determined horizontal data to be combined can be combined into a matrix as an optical flow horizontal component matrix. Then, an optical flow vertical component matrix corresponding to the above local vertical displacement matrix can be generated. Among them, the method for generating the optical flow vertical component matrix can refer to the specific implementation manner of generating the optical flow horizontal component matrix corresponding to the above local horizontal displacement matrix, which will not be elaborated here. Then, the above optical flow horizontal component matrix and the above optical flow vertical component matrix can be input into an image conversion function to obtain optical flow image data. Among them, the above image conversion function can be a function capable of converting an optical flow field into an image. For example, the above image conversion function can be the quiver function in matlab.

[0069] The above technical solution and its related content, as an inventive point of the embodiments of the present disclosure, solve the problem of "relatively large consumption of computing resources". The factors that lead to relatively large consumption of computing resources are often as follows: The method of predicting motion sickness through data such as the electroencephalogram of users often requires processing data such as detected electroencephalogram, galvanic skin response, and respiratory rate, such as denoising, filtering, and data fusion, which easily leads to a relatively high computational complexity during data processing, and thus leads to a relatively large consumption of computing resources during data processing. If the above factors are solved, the consumption of computing resources can be reduced. To achieve this effect, first, the present disclosure determines the preprocessed image data that meets the preset image processing conditions among the above two frames of preprocessed image data as the first preprocessed visual image data. Thus, the first frame of preprocessed image data can be determined. Secondly, the present disclosure determines the preprocessed image data that does not meet the above image processing conditions among the above two frames of preprocessed image data as the second preprocessed visual image data. Thus, the second frame of preprocessed image data can be determined. Then, the present disclosure performs grayscale conversion processing on the above first preprocessed visual image data and the above second preprocessed visual image data to obtain a first preprocessed visual grayscale image and a second preprocessed visual grayscale image. Thus, grayscale images of the two frames of preprocessed image data can be obtained. Then, the present disclosure performs image transformation processing on the above first preprocessed visual grayscale image and the above second preprocessed visual grayscale image to obtain a first visual frequency spectrum matrix and a second visual frequency spectrum matrix. Thus, frequency spectrum matrices corresponding to the two frames of preprocessed image data can be obtained. Then, based on the above second visual frequency spectrum matrix, a conjugate frequency spectrum matrix is generated. Then, the product of the above first visual frequency spectrum matrix and the above conjugate frequency spectrum matrix is determined as the conjugate visual frequency spectrum matrix. Then, based on the above first visual frequency spectrum matrix, a first amplitude matrix is generated. Then, based on the above second visual frequency spectrum matrix, a second amplitude matrix is generated. Then, the product of the above first amplitude matrix and the above second amplitude matrix is determined as the visual amplitude matrix. Then, the inverse of the above visual amplitude matrix is determined as the visual amplitude inverse matrix. Then, the product of the above conjugate visual frequency spectrum matrix and the above visual amplitude inverse matrix is determined as the visual amplitude frequency spectrum matrix. Thus, the cross-power spectrum between the two frames of preprocessed image data can be obtained. Then, based on the above visual amplitude frequency spectrum matrix, a visual phase matrix is generated. Then, based on the above visual phase matrix, first peak data and second peak data are determined. Then, based on the above first peak data and the above second peak data, horizontal translation data and vertical translation data corresponding to the above first preprocessed visual image data are generated. Thus, the displacement amounts of the first preprocessed visual grayscale image relative to the second preprocessed visual grayscale image in the horizontal and vertical directions can be obtained. Then, the above first preprocessed visual grayscale image is block-processed to obtain respective grayscale image blocks. Thus, the first preprocessed visual grayscale image can be block-processed. Then, based on the above horizontal translation data and the above vertical translation data, the above respective grayscale image blocks are initialized to obtain respective initialized image blocks.Thus, each initialized image block can be obtained. Then, for each of the above-mentioned initialized image blocks, image block horizontal data and image block vertical data corresponding to the above-mentioned initialized image block are generated. Finally, based on the generated image block horizontal data and the generated image block vertical data, optical flow image data is generated. Thus, an optical flow field corresponding to the visual image data can be obtained. Also, since the visual image data sequence corresponding to the user can be block-processed to generate each optical flow image data, and then the user's motion sickness can be predicted through each optical flow image data, without the need for data fusion processing such as electroencephalogram data, the computational complexity during data processing can be reduced, and the computational resources consumed during data processing can be reduced.

[0070] Fourth, based on the generated depth image data, the generated optical flow image data, and the feature extraction layer in the pre-trained motion sickness detection model, a horizontal feature map, a vertical feature map, and a depth feature map are generated.

[0071] In some embodiments, the above-mentioned processor 2 can generate a horizontal feature map, a vertical feature map, and a depth feature map based on the generated depth image data, the generated optical flow image data, and the feature extraction layer in the pre-trained motion sickness detection model.

[0072] Among them, the above-mentioned horizontal feature map can be a feature map corresponding to the motion vectors of the above-mentioned optical flow image data in the horizontal direction. The above-mentioned vertical feature map can be a feature map corresponding to the motion vectors of the above-mentioned optical flow image data in the vertical direction. The above-mentioned depth feature map can be a feature map corresponding to the above-mentioned depth image data.

[0073] The above-mentioned motion sickness detection model can be a neural network model that takes each depth image data and each optical flow image data as inputs and outputs a motion sickness data sequence. The above-mentioned motion sickness detection model can include a feature extraction layer, a temporal feature extraction network, and an output layer. The above-mentioned feature extraction layer can include a horizontal feature extraction network, a vertical feature extraction network, and a depth feature extraction network. Each motion sickness data in the above-mentioned motion sickness data sequence can be the probability distribution of the target user experiencing motion sickness in the virtual scene. The above-mentioned motion sickness data can include symptom severity and probability data. The above-mentioned symptom severity can represent the degree of the target user experiencing motion sickness. The symptom severity can include but is not limited to: asymptomatic, mild symptom, moderate symptom, severe symptom, extremely severe symptom. The above-mentioned probability data can represent the probability of the corresponding symptom severity occurring. For example, the above-mentioned motion sickness data sequence can be {symptom severity: asymptomatic, probability data: 10%; symptom severity: mild symptom, probability data: 10%; symptom severity: moderate symptom, probability data: 5%; symptom severity: severe symptom, probability data: 70%; symptom severity: extremely severe symptom, probability data: 5%}.

[0074] In some alternative implementations of some embodiments, the above-mentioned motion sickness detection model can be obtained by the above-mentioned processor 2 through the following steps: First step, obtain a sample set. Among them, the samples in the sample set include a sample depth image data set, a sample optical flow image data set, and a sample motion sickness data sequence. Each sample depth image data in the above-mentioned sample depth image data set can be depth image data for model training. Each sample optical flow image data in the above-mentioned sample optical flow image data set can be optical flow image data for model training. Each sample motion sickness data in the above-mentioned sample motion sickness data sequence can be the standard motion sickness data corresponding to the above-mentioned sample. Each sample motion sickness data in the above-mentioned sample motion sickness data sequence can include a sample symptom degree and sample probability data. The above-mentioned sample symptom degree can be the symptom degree corresponding to the sample motion sickness data. The above-mentioned sample probability data can represent the probability of the occurrence of the corresponding sample symptom degree.

[0075] Second step, based on the sample set, perform the following training steps: First sub-step, input the respective sample optical flow image data sets included in at least one sample in the sample set into the horizontal feature extraction network included in the feature extraction layer of the initial neural network, and obtain the horizontal feature map corresponding to each sample in the above-mentioned at least one sample. Among them, the above-mentioned horizontal feature extraction network can be a neural network that takes the respective sample optical flow image data sets as inputs and outputs the horizontal feature maps corresponding to the respective sample optical flow image data sets. For example, the above-mentioned horizontal feature extraction network can be a convolutional neural network.

[0076] Second sub-step, input the respective sample optical flow image data sets included in the above-mentioned at least one sample into the vertical feature extraction network included in the feature extraction layer of the initial neural network, and obtain the vertical feature map corresponding to each sample in the above-mentioned at least one sample. Among them, the above-mentioned vertical feature extraction network can be a neural network that takes the respective sample optical flow image data sets as inputs and outputs the vertical feature maps corresponding to the respective sample optical flow image data sets. For example, the above-mentioned vertical feature extraction network can be a convolutional neural network.

[0077] Third sub-step, input the respective sample depth image data sets included in the above-mentioned at least one sample into the depth feature extraction network included in the feature extraction layer of the initial neural network, and obtain the depth feature map corresponding to each sample in the above-mentioned at least one sample. Among them, the above-mentioned depth feature extraction network can be a neural network that takes the respective sample depth image data sets as inputs and outputs the depth feature maps corresponding to the respective sample depth image data sets. For example, the above-mentioned depth feature extraction network can be a convolutional neural network.

[0078] Fourth sub-step: Input the horizontal feature map, vertical feature map, and depth feature map corresponding to each of the above at least one sample into the temporal feature extraction network in the initial neural network to obtain the temporal feature information corresponding to each of the above at least one sample. Among them, the above temporal feature information may be the temporal features corresponding to the above horizontal feature map, the above vertical feature map, and the above depth feature map. The above temporal feature extraction network may be a neural network that takes the horizontal feature map, vertical feature map, and depth feature map as inputs and the temporal feature information as outputs. For example, the above temporal feature extraction network may be a long short-term memory network.

[0079] Fifth sub-step: Input the temporal feature information corresponding to each of the above at least one sample into the output layer in the initial neural network to obtain the motion sickness data sequence corresponding to each of the above at least one sample. Among them, the above output layer may be an activation function that takes the temporal feature information as input and the motion sickness data sequence as output. For example, the above activation function may be the Softmax function.

[0080] Sixth sub-step: Generate the distribution loss data corresponding to each of the above at least one sample based on the sample motion sickness data sequence and the motion sickness data sequence corresponding to each of the above at least one sample. Among them, the above distribution loss data may be the loss value between the sample motion sickness data sequence and the motion sickness data sequence.

[0081] Seventh sub-step: Generate the transformation loss data corresponding to each of the above at least one sample based on the preset weight data, the sample motion sickness data sequence and the motion sickness data sequence corresponding to each of the above at least one sample. Among them, the above weight data may be a preset value. Here, there is no limitation on the specific setting of the above weight data. The above transformation loss data may be the loss value between the change value of every two adjacent sample motion sickness data in the sample motion sickness data sequence and the change value of every two adjacent motion sickness data in the motion sickness data sequence.

[0082] Eighth sub-step: Generate the sample loss data based on the distribution loss data and the transformation loss data corresponding to each of the above at least one sample. Among them, the above sample loss data may be the loss value between each sample motion sickness data sequence corresponding to the above at least one sample and each motion sickness data sequence. In practice, for each of the above at least one sample, the processor 2 may determine the average of the distribution loss data and the transformation loss data corresponding to the sample as the average loss data. Then, the average of each average loss data corresponding to the above at least one sample may be determined as the sample loss data.

[0083] The ninth sub-step, in response to determining that the above sample loss data meets a preset loss condition, determines the initial neural network as a motion sickness detection model. Among them, the above loss condition may be that the above sample loss data is less than a preset loss value. The above loss value may be a preset numerical value. Here, there is no limitation on the specific setting of the above loss value.

[0084] The tenth sub-step, in response to determining that the above sample loss data does not meet the above loss condition, adjusts the network parameters of the initial neural network, and uses the unused samples to form a sample set, uses the adjusted initial neural network as the initial neural network, and executes the above training step again. In practice, the above processor 2 can adjust the network parameters of the above initial neural network through the BackPropagation Algorithm (BP algorithm) and the gradient descent method (such as the mini-batch gradient descent algorithm).

[0085] In some optional implementation manners of some embodiments, the above processor 2 can generate distribution loss data corresponding to each sample in the above at least one sample based on the sample motion sickness data sequence and the motion sickness data sequence corresponding to each sample in the above at least one sample through the following steps: For each sample in the above at least one sample, execute the following steps: The first step is to determine the sample motion sickness data sequence corresponding to the above sample as the target sample data sequence.

[0086] The second step is to execute the following steps for each motion sickness data in the motion sickness data sequence corresponding to the above sample: The first sub-step is to determine the logarithm of the probability data included in the above motion sickness data as the motion sickness logarithm data. Among them, the above logarithm may be a logarithm with base 2.

[0087] The second sub-step is to determine the target sample data corresponding to the above motion sickness data in the above target sample data sequence as the sample data to be processed.

[0088] The third sub-step is to determine the product of the above motion sickness logarithm data and the sample probability data included in the above sample data to be processed as the motion sickness loss data.

[0089] The third step is to determine the sum of the determined motion sickness loss data as the distribution loss data corresponding to the above sample.

[0090] In some optional implementation manners of some embodiments, the above processor 2 can generate transformation loss data corresponding to each sample in the above at least one sample based on preset weight data, the sample motion sickness data sequence and the motion sickness data sequence corresponding to each sample in the above at least one sample through the following steps: For each of the at least one sample above, perform the following steps: In the first step, based on the sample motion sickness data sequence corresponding to the above sample, for every two adjacent sample motion sickness data in the above sample motion sickness data sequence, determine the difference between the two sample probability data corresponding to the two sample motion sickness data as the sample motion sickness difference data.

[0091] In the second step, based on the determined sample motion sickness difference data, generate a sample motion sickness difference data sequence. In practice, the above processor 2 can arrange the above sample motion sickness difference data in the order corresponding to the above sample motion sickness data sequence as the sample motion sickness difference data sequence. As an example, when the above sample motion sickness data sequence is {sample motion sickness data 1, sample motion sickness data 2, sample motion sickness data 3}, and the sample motion sickness difference data corresponding to sample motion sickness data 1 and sample motion sickness data 2 is sample motion sickness difference data 1, and the sample motion sickness difference data corresponding to sample motion sickness data 2 and sample motion sickness data 3 is sample motion sickness difference data 2, the obtained sample motion sickness difference data sequence is {sample motion sickness difference data 1, sample motion sickness difference data 2}.

[0092] In the third step, based on the motion sickness data sequence corresponding to the above sample, for every two adjacent motion sickness data in the above motion sickness data sequence, determine the difference between the two probability data corresponding to the two motion sickness data as the motion sickness difference data.

[0093] In the fourth step, based on the determined motion sickness difference data, generate a motion sickness difference data sequence. In practice, the above processor 2 can arrange the above motion sickness difference data in the order corresponding to the above motion sickness data sequence as the motion sickness difference data sequence.

[0094] In the fifth step, for each sample motion sickness difference data in the above sample motion sickness difference data sequence, perform the following steps: In the first sub-step, determine the motion sickness difference data corresponding to the above sample motion sickness difference data in the above motion sickness difference data sequence as the target motion sickness difference data.

[0095] In the second sub-step, determine the difference between the above target motion sickness difference data and the above sample motion sickness difference data as the data to be processed difference data.

[0096] In the sixth step, determine the average of the determined data to be processed difference data as the data to be processed average data.

[0097] In the seventh step, determine the product of the above data to be processed average data and the above weight data as the transformation loss data corresponding to the above sample.

[0098] Fifth, input the horizontal feature map, vertical feature map, and depth feature map into the temporal feature extraction network in the motion sickness detection model to obtain temporal feature information.

[0099] In some embodiments, the above-mentioned processor 2 may input the above-mentioned horizontal feature map, the above-mentioned vertical feature map, and the above-mentioned depth feature map into the temporal feature extraction network in the above-mentioned motion sickness detection model to obtain temporal feature information.

[0100] Sixth, generate a motion sickness detection result based on the temporal feature information.

[0101] In some embodiments, the above-mentioned processor 2 may generate a motion sickness detection result based on the above-mentioned temporal feature information. Among them, the above-mentioned motion sickness detection result may be the symptom degree of motion sickness that occurs to the above-mentioned target user. In practice, first, the above-mentioned processor 2 may input the above-mentioned temporal feature information into the output layer in the above-mentioned motion sickness detection model to obtain a motion sickness data sequence corresponding to the above-mentioned temporal feature information. Then, the symptom degree with the largest corresponding probability data among the various symptom degrees included in the above-mentioned motion sickness data sequence may be determined as the motion sickness detection result.

[0102] Seventh, in response to determining that the motion sickness detection result meets a preset detection result condition, determine the information to be displayed.

[0103] In some embodiments, the above-mentioned processor 2 may, in response to determining that the above-mentioned motion sickness detection result meets a preset detection result condition, determine the information to be displayed. Among them, the above-mentioned detection result condition may be that the above-mentioned motion sickness detection result is not asymptomatic. The above-mentioned information to be displayed may be the information that needs to be displayed. In practice, the above-mentioned processor 2 may determine the preset motion sickness relief information as the information to be displayed. Among them, the above-mentioned motion sickness relief information may be information for providing measures to relieve dizziness to the user. For example, the above-mentioned motion sickness relief information may be "If you feel uncomfortable such as dizziness, you can reduce the rotation speed to relieve dizziness".

[0104] In some embodiments, the above-mentioned memory 3 may be configured to store the above-mentioned motion sickness detection result.

[0105] In some embodiments, the above-mentioned display 4 may be configured to display the above-mentioned information to be displayed.

[0106] The above-mentioned head-mounted device for motion sickness information detection of the present disclosure has the following beneficial effects: By using the head-mounted device for motion sickness information detection of the present disclosure, the consumption of computing resources during processing can be reduced. Specifically, the reason for the large consumption of computing resources is that the method of predicting the user's motion sickness by detecting the user's electroencephalogram, galvanic skin response, respiratory rate, etc. requires the user to wear a dedicated contact device, which is likely to reduce the user's virtual reality experience. Moreover, when detecting, it is necessary to synchronously process various data such as the user's electroencephalogram, galvanic skin response, and respiratory rate, which is likely to cause a large consumption of computing resources during processing. Based on this, the head-mounted device for motion sickness information detection of the present disclosure, firstly, an image acquisition device, is configured to acquire a sequence of visual image data corresponding to a target user. Thus, the raw data to be processed can be obtained. Secondly, a processor, is configured to perform the following processing: Firstly, preprocess the above-mentioned sequence of visual image data to obtain a sequence of preprocessed image data. Thus, the raw data can be preprocessed to reduce the interference of noise information in the data on subsequent processing. Secondly, for each preprocessed image data in the above-mentioned sequence of preprocessed image data, based on the above-mentioned preprocessed image data, generate depth image data. Thus, a depth map corresponding to the sequence of preprocessed image data can be obtained. Then, for every two frames of preprocessed image data in the above-mentioned sequence of preprocessed image data that meet the preset image conditions, based on the above-mentioned two frames of preprocessed image data, generate optical flow image data. Thus, an optical flow field map corresponding to the sequence of preprocessed image data can be obtained. Then, based on the generated respective depth image data, the generated respective optical flow image data, and the feature extraction layer in the pre-trained motion sickness detection model, generate a horizontal feature map, a vertical feature map, and a depth feature map, wherein the above-mentioned feature extraction layer includes a horizontal feature extraction network, a vertical feature extraction network, and a depth feature extraction network. Thus, respective feature maps corresponding to the sequence of preprocessed image data can be obtained. Then, input the above-mentioned horizontal feature map, the above-mentioned vertical feature map, and the above-mentioned depth feature map into the temporal feature extraction network in the above-mentioned motion sickness detection model to obtain temporal feature information. Thus, temporal features corresponding to the sequence of preprocessed image data can be obtained. Then, based on the above-mentioned temporal feature information, generate a motion sickness detection result. Thus, the motion sickness detection result of the user can be obtained through the temporal features. Then, in response to determining that the above-mentioned motion sickness detection result meets the preset detection result condition, determine the information to be displayed. Then, a memory, is configured to store the above-mentioned motion sickness detection result. Thus, the motion sickness detection result can be stored. Finally, a display, is configured to display the above-mentioned information to be displayed. Thus, relief measures can be provided to users with relatively severe symptoms.Also, since it is possible to predict a user's motion sickness by obtaining a sequence of the user's visual image data and processing the sequence of visual image data, without the user having to additionally wear a dedicated contact device, the probability of a lower virtual experience of the user caused by wearing a dedicated contact device can be reduced. Further, since it is possible to directly predict a user's motion sickness through a pre-trained motion sickness detection model, without having to simultaneously process multiple types of data such as the user's electroencephalogram, galvanic skin response, and respiratory rate, the consumption of computing resources during processing can be reduced.

[0107] It should be noted that the computer-readable medium described in some embodiments of the present disclosure may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0108] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0109] The above computer-readable medium may be included in the above head-mounted device for detecting motion sickness information; or it may exist separately and not be assembled into the above head-mounted device for detecting motion sickness information. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the above head-mounted device for detecting motion sickness information, the above head-mounted device for detecting motion sickness information is caused to: an image acquisition device configured to acquire a sequence of visual image data corresponding to a target user; a processor configured to perform the following processes: preprocess the above sequence of visual image data to obtain a sequence of preprocessed image data; for each preprocessed image data in the above sequence of preprocessed image data, generate depth image data based on the above preprocessed image data; for every two frames of preprocessed image data in the above sequence of preprocessed image data that meet a preset image condition, generate optical flow image data based on the above two frames of preprocessed image data; generate a horizontal feature map, a vertical feature map, and a depth feature map based on the generated respective depth image data, the generated respective optical flow image data, and a feature extraction layer in a pre-trained motion sickness detection model, wherein the above feature extraction layer includes a horizontal feature extraction network, a vertical feature extraction network, and a depth feature extraction network; input the above horizontal feature map, the above vertical feature map, and the above depth feature map into a temporal feature extraction network in the above motion sickness detection model to obtain temporal feature information; generate a motion sickness detection result based on the above temporal feature information; in response to determining that the above motion sickness detection result meets a preset detection result condition, determine information to be displayed; a memory configured to store the above motion sickness detection result; a display configured to display the above information to be displayed.

[0110] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet).

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0112] The functions described above in this document may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), and so on.

[0113] The above description is only some preferred embodiments of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) disclosed in the embodiments of the present disclosure that have similar functions.

Claims

1. A head-mounted device for detecting motion sickness information, comprising: An image acquisition device configured to acquire a sequence of visual image data corresponding to a target user; A processor configured to perform the following processing: Preprocess the sequence of visual image data to obtain a sequence of preprocessed image data; For each preprocessed image data in the sequence of preprocessed image data, generate depth image data based on the preprocessed image data; For every two frames of preprocessed image data in the sequence of preprocessed image data that meet a preset image condition, generate optical flow image data based on the two frames of preprocessed image data; Generate a horizontal feature map, a vertical feature map, and a depth feature map based on the generated depth image data, the generated optical flow image data, and a feature extraction layer in a pre-trained motion sickness detection model, wherein the feature extraction layer includes a horizontal feature extraction network, a vertical feature extraction network, and a depth feature extraction network; Input the horizontal feature map, the vertical feature map, and the depth feature map into a temporal feature extraction network in the motion sickness detection model to obtain temporal feature information; Generate a motion sickness detection result based on the temporal feature information; In response to determining that the motion sickness detection result meets a preset detection result condition, determine information to be displayed; A memory configured to store the motion sickness detection result; A display configured to display the information to be displayed.

2. The head-mounted device for detecting motion sickness information according to claim 1, wherein, The processor is further configured to generate depth image data based on the preprocessed image data through the following steps, including: Perform feature extraction processing on the preprocessed image data to obtain a group of preprocessed feature maps; For each preprocessed feature map in the group of preprocessed feature maps, perform the following steps: Perform downsampling processing on a preset random noise map to obtain a downsampled noise map corresponding to the preprocessed feature map; Perform image stitching processing on the downsampled noise map and the preprocessed feature map to obtain a noise-stitched feature map; Determine the obtained noise-stitched feature maps as a group of noise-stitched feature maps; Generate depth image data based on the group of noise-stitched feature maps.

3. The head-mounted device for detecting motion sickness information according to claim 1, wherein, The processor is further configured to generate optical flow image data based on the two frames of preprocessed image data through the following steps, including: Perform image conversion processing on the two frames of preprocessed image data to obtain two frames of visual grayscale image data; Determine the visual grayscale image data that meets a preset image frame condition in the two frames of visual grayscale image data as first grayscale image data; Determine the visual grayscale image data that does not meet the image frame condition in the two frames of visual grayscale image data as second grayscale image data; For each pixel point in each pixel point included in the first grayscale image data, perform the following steps: Determine the pixel point as a first pixel point; Based on the first grayscale image data and the first pixel point, determine a first pixel matrix corresponding to the first pixel point; Generate a first horizontal spatial gradient and a first vertical spatial gradient based on a preset horizontal operator matrix, a preset vertical operator matrix, and the first pixel matrix; Determine the pixel points corresponding to the first pixel point among the respective pixel points included in the second grayscale image data as second pixel points; Determine the difference between the pixel value corresponding to the first pixel point and the pixel value corresponding to the second pixel point as pixel difference data; Construct optical flow non-linear information based on a preset custom horizontal vector, a preset custom vertical vector, the first horizontal spatial gradient, the first vertical spatial gradient, and the pixel difference data; Determine the respective pixel points corresponding to the first pixel point among the respective pixel points included in the first grayscale image data as respective target pixel points; Determine respective target optical flow non-linear information corresponding to the respective target pixel points based on the respective target pixel points; Generate optical flow horizontal data and optical flow vertical data corresponding to the first pixel point based on the optical flow non-linear information and the respective target optical flow non-linear information; Generate an optical flow horizontal matrix based on the generated respective optical flow horizontal data; Generate an optical flow vertical matrix based on the generated respective optical flow vertical data; Generate optical flow image data based on the optical flow horizontal matrix and the optical flow vertical matrix; 4. The head-mounted device for detecting motion sickness information according to claim 1, wherein, The motion sickness detection model is obtained by training through the following steps: Obtain a sample set, wherein the samples in the sample set include a sample depth image data set, a sample optical flow image data set, and a sample motion sickness data sequence, and each sample motion sickness data in the sample motion sickness data sequence includes sample symptom degree and sample probability data; Based on the sample set, perform the following training steps: Input the respective sample optical flow image data sets included in at least one sample in the sample set into the horizontal feature extraction network included in the feature extraction layer of the initial neural network to obtain horizontal feature maps corresponding to each sample in the at least one sample; Input the respective sample optical flow image data sets included in the at least one sample into the vertical feature extraction network included in the feature extraction layer of the initial neural network to obtain vertical feature maps corresponding to each sample in the at least one sample; Input the respective sample depth image data sets included in the at least one sample into the depth feature extraction network included in the feature extraction layer of the initial neural network to obtain depth feature maps corresponding to each sample in the at least one sample; Input the horizontal feature maps, vertical feature maps, and depth feature maps corresponding to each sample in the at least one sample into the temporal feature extraction network in the initial neural network to obtain temporal feature information corresponding to each sample in the at least one sample; Input the temporal feature information corresponding to each sample in the at least one sample into the output layer in the initial neural network to obtain a motion sickness data sequence corresponding to each sample in the at least one sample, and each motion sickness data in the motion sickness data sequence includes symptom degree and probability data; Generate distribution loss data corresponding to each sample in the at least one sample based on the sample motion sickness data sequence and the motion sickness data sequence corresponding to each sample in the at least one sample; Generate the transformation loss data corresponding to each sample in the at least one sample based on the preset weight data, the sample motion sickness data sequence corresponding to each sample in the at least one sample, and the motion sickness data sequence; Generate sample loss data based on the distribution loss data and the transformation loss data corresponding to each sample in the at least one sample; In response to determining that the sample loss data meets the preset loss condition, determine the initial neural network as the motion sickness detection model; In response to determining that the sample loss data does not meet the loss condition, adjust the network parameters of the initial neural network, and use the unused samples to form a sample set. Use the adjusted initial neural network as the initial neural network and execute the training step again.

5. The head-mounted device for detecting motion sickness information according to claim 4, wherein, The processor is further configured to generate the distribution loss data corresponding to each sample in the at least one sample through the following steps based on the preset weight data, the sample motion sickness data sequence corresponding to each sample in the at least one sample, and the motion sickness data sequence, including: For each sample in the at least one sample, perform the following steps: Determine the sample motion sickness data sequence corresponding to the sample as the target sample data sequence; For each motion sickness data in the motion sickness data sequence corresponding to the sample, perform the following steps: Determine the logarithm of the probability data included in the motion sickness data as the motion sickness logarithm data; Determine the target sample data corresponding to the motion sickness data in the target sample data sequence as the sample data to be processed; Determine the product of the motion sickness logarithm data and the sample probability data included in the sample data to be processed as the motion sickness loss data; Determine the sum of the determined motion sickness loss data as the distribution loss data corresponding to the sample.

6. The head-mounted device for detecting motion sickness information according to claim 4, wherein, The processor is further configured to generate the transformation loss data corresponding to each sample in the at least one sample through the following steps based on the preset weight data, the sample motion sickness data sequence corresponding to each sample in the at least one sample, and the motion sickness data sequence, including: For each sample in the at least one sample, perform the following steps: Based on the sample motion sickness data sequence corresponding to the sample, for every two adjacent sample motion sickness data in the sample motion sickness data sequence, determine the difference between the two sample probability data corresponding to the two sample motion sickness data as the sample motion sickness difference data; Generate a sample motion sickness difference data sequence based on the determined sample motion sickness difference data; Based on the motion sickness data sequence corresponding to the sample, for every two adjacent motion sickness data in the motion sickness data sequence, determine the difference between the two probability data corresponding to the two motion sickness data as the motion sickness difference data; Generate a motion sickness difference data sequence based on the determined motion sickness difference data; For each sample motion sickness difference data in the sample motion sickness difference data sequence, perform the following steps: Determine the motion sickness difference data corresponding to the sample motion sickness difference data in the motion sickness difference data sequence as the target motion sickness difference data; Determine the difference between the target motion sickness difference data and the sample motion sickness difference data as the data to be processed for the difference; Determine the average of each of the determined difference data to be processed as the average data to be processed; Determine the product of the average data to be processed and the weight data as the transformation loss data corresponding to the sample.

7. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by a processor, it implements the method performed by the head-mounted device for detecting motion sickness information according to any one of claims 1-6.

Images