A virtual reality-based visual presentation method and related apparatus

Abstract

The application discloses a kind of visual presentation methods and related devices based on virtual reality, it is related to virtual reality technical field, the method includes: determining the user pupil position of each frame image based on image sequence data;Based on user pupil position, visual line focal point analysis is carried out to divide focal point area to virtual picture;Picture image data of focal point area is carried out sharpness rendering analysis;Based on motion sickness score and picture motion speed, picture image data of focal point area is carried out user motion sickness rendering analysis;Based on sharpness rendering data and user motion sickness rendering data, picture image data of focal point area is rendered processing;Based on the transmission of this picture image data to virtual reality head-mounted display in visual presentation by push stream engine combined with focal length adjustment.The application improves the fidelity of virtual picture, and provides more natural and comfortable visual experience for user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of virtual reality, in particular to a visual presentation method based on virtual reality and related device. BACKGROUND

[0002] Virtual reality technology is a technology that can create and experience a virtual world, which uses real life data, through computer technology to generate electronic signals, and combines with various output devices, that is, virtual and reality are combined with each other, so as to be converted into phenomena that can be felt by people. For virtual reality technology, the visual presentation of virtual pictures affects the immersion of users in the virtual environment, so how to better present virtual pictures is an extremely important consideration. At present, in the visual presentation of virtual pictures, there is a lack of analysis of the user's visual focus. Since the user's visual attention will move, gaze and scan on different objects in the real space, the user's visual attention focus will also have similar behaviors in the virtual reality space. However, the visual presentation of virtual pictures does not consider the visual focus characteristics of the user, resulting in insufficient comfort of the user's experience in the virtual environment. At the same time, due to the different sensitivities of the visual systems of different people to virtual pictures, they will more or less have motion sickness symptoms during the virtual environment experience. The current solution is basically that the user leaves the virtual environment for a period of time until the motion sickness symptoms disappear, but this solution has a great impact on the user's experience, and cannot bring the user a satisfactory virtual reality immersion experience. SUMMARY

[0003] The purpose of the present application is to overcome the shortcomings of the prior art, and the present application provides a visual presentation method based on virtual reality and related device, which improves the realism of virtual pictures and provides a more natural and comfortable visual experience for users.

[0004] In order to solve the above technical problems, the present application provides a visual presentation method based on virtual reality, applied to a virtual reality head-mounted display and a cloud server, the virtual reality head-mounted display is built-in with an eye movement tracking device; the method comprises:

[0005] The cloud server determines the user's pupil position of each frame of image based on the image sequence data collected by the eye movement tracking device;

[0006] Based on the user's pupil position, the visual focus is analyzed to obtain visual focus information, and the virtual picture corresponding to the virtual reality head-mounted display is divided into a focus area based on the visual focus information;

[0007] The picture image data of the focus area is analyzed for clarity rendering to obtain clarity rendering data;

[0008] Perform user motion sickness rendering analysis on the picture image data of the focus area based on the motion sickness score and the picture motion speed to obtain user motion sickness rendering data;

[0009] Perform rendering processing on the picture image data of the focus area based on the definition rendering data and the user motion sickness rendering data to obtain the picture image data after rendering processing;

[0010] Based on the push stream engine combined with the focal length adjustment, the picture image data after rendering processing is transmitted to the virtual reality head-mounted display for visual presentation.

[0011] Optionally, the user pupil position of each frame of image is determined based on the image sequence data collected by the eye movement tracking device, including:

[0012] Feature region extraction is performed on each frame of image of the image sequence data to obtain corresponding pupil region images, and binary processing is performed on each pupil region image to obtain corresponding binary pupil region images;

[0013] The pupil contour feature points of the binary pupil region image are extracted based on a clustering algorithm, and the first position data of the pupil is determined based on the pupil contour feature points;

[0014] The second position data of the pupil is determined based on the pupil point probability distribution map of each frame of image, and the user pupil position of each frame of image is determined based on the first position data and the second position data.

[0015] Optionally, the line of sight focal point analysis is performed based on the user pupil position to obtain line of sight focal point information, and the focus area of the virtual picture corresponding to the presentation of the virtual reality head-mounted display is divided based on the line of sight focal point information, including:

[0016] The eye movement trajectory and the eye stay duration are determined based on the user pupil position of each frame of image, and the line of sight focal point information is determined based on the eye movement trajectory and the eye stay duration;

[0017] The focus area of the virtual picture is divided based on the line of sight focal point information combined with the preset focus area range.

[0018] Optionally, the definition rendering analysis is performed on the picture image data of the focus area to obtain the definition rendering data, including:

[0019] The picture image data is grayed to obtain grayed picture image data, and the grayed picture image data is subjected to Gaussian filtering to obtain filtered picture image data;

[0020] Gradient operation is performed on the grayed picture image data and the filtered picture image data to obtain a first gradient image corresponding to the grayed picture image data and a second gradient image corresponding to the filtered picture image data;

[0021] calculate image sharpness data based on the first gradient image and the second gradient image, and analyze luminance data of the frame image data;

[0022] calculate a first difference value of the image sharpness data and a preset standard sharpness threshold value, calculate a second difference value of the luminance data and a preset standard luminance threshold value, and determine a tone rendering parameter and a sharpening effect rendering parameter based on the first difference value and the second difference value;

[0023] generate sharpness rendering data based on the tone rendering parameter and the sharpening effect rendering parameter.

[0024] Optionally, the user motion sickness rendering analysis of the frame image data of the focus point area based on the motion sickness score and the frame motion speed includes:

[0025] analyze the pixel displacement amount of adjacent frame image frames of the frame image data, and determine the frame motion speed based on the pixel displacement amount, and determine an adjustment parameter of the frame motion speed based on the motion sickness score when the user experiences the virtual frame;

[0026] perform attention heat analysis on the frame image data to obtain attention heat data, and determine frame motion sickness rendering parameters based on the attention heat data and the motion sickness score;

[0027] generate user motion sickness rendering data based on the adjustment parameter and the frame motion sickness rendering parameter.

[0028] Optionally, the rendering processing of the frame image data of the focus point area based on the sharpness rendering data and the user motion sickness rendering data includes:

[0029] determine the rendering resolution based on the field of view information and the anti-distortion information of the virtual reality head-mounted display;

[0030] perform rendering processing on the frame image data of the focus point area based on the rendering resolution, the sharpness rendering data and the user motion sickness rendering data to obtain the frame image data after rendering processing.

[0031] Optionally, the rendering processing of the frame image data of the focus point area based on the sharpness rendering data and the user motion sickness rendering data includes:

[0032] encode the frame image data after rendering processing based on the encoding engine to obtain encoded frame image data, and transmit the encoded frame image data to the virtual reality head-mounted display based on the streaming engine;

[0033] determine a pupil distance based on the user pupil position, and determine a focal point adjustment value of an inbuilt lens and an image center position adjustment value of an inbuilt image display source in a virtual reality head-mounted display based on the pupil distance;

[0034] The virtual reality head-mounted display visually presents the encoded picture image data based on the focal point adjustment value of the inbuilt lens and the image center position adjustment value of the inbuilt image display source.

[0035] In addition, the present application also provides a virtual reality-based visual presentation device applied to a virtual reality head-mounted display and a cloud server, wherein the virtual reality head-mounted display is internally provided with an eye movement tracking device; the device comprises:

[0036] a pupil position determination module configured to determine a user pupil position of each frame of image based on image sequence data collected by the eye movement tracking device;

[0037] a focal point area division module configured to perform a visual line focal point analysis based on the user pupil position, to obtain visual line focal point information, and to divide a focal point area of a virtual picture corresponding to the visual presentation of the virtual reality head-mounted display based on the visual line focal point information;

[0038] a definition analysis module configured to perform definition rendering analysis on picture image data of the focal point area, to obtain definition rendering data;

[0039] a motion sickness rendering analysis module configured to perform user motion sickness rendering analysis on the picture image data of the focal point area based on a motion sickness score and a picture motion speed, to obtain user motion sickness rendering data;

[0040] a rendering processing module configured to perform rendering processing on the picture image data of the focal point area based on the definition rendering data and the user motion sickness rendering data, to obtain picture image data after rendering processing;

[0041] a visual presentation module configured to transmit the picture image data after rendering processing to the virtual reality head-mounted display for visual presentation based on a streaming engine combined with focal length adjustment.

[0042] In addition, the present application also provides an electronic device comprising a processor and a memory, wherein the memory is configured to store instructions, and the processor is configured to call the instructions in the memory, so that the electronic device performs the virtual reality-based visual presentation method described above.

[0043] In addition, the present application also provides a computer readable storage medium storing computer instructions, when the computer instructions run on an electronic device, so that the electronic device performs the virtual reality-based visual presentation method described above.

[0044] In the embodiment of the present application, the line-of-sight focus analysis is performed based on the user pupil position of each frame image to obtain line-of-sight focus information, and the focus area of the virtual picture corresponding to the virtual reality head-mounted display is divided based on the line-of-sight focus information. The focus area of the virtual picture is divided considering the change of the user attention focus, which can be more close to the visual presentation of the user in the actual space. The definition rendering analysis is performed on the picture image data of the focus area, the user motion sickness rendering analysis is performed on the picture image data of the focus area based on the motion sickness score and the picture motion speed, the picture image data of the focus area is rendered based on the definition rendering data and the user motion sickness rendering data, which greatly reduces the user's dizziness and improves the user experience. At the same time, the picture quality is improved while avoiding affecting the immersion of the user in the virtual reality experience. The picture image data after the rendering processing is transmitted to the virtual reality head-mounted display for visual presentation based on the streaming engine combined with the focal length adjustment, which helps to reduce the user's eye fatigue and reduce the delay, and provides the user with a more natural and comfortable visual experience. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0046] Figure 1 is a flow diagram of the visual presentation method based on virtual reality in the embodiment of the present application;

[0047] Figure 2 is a flow diagram of the visual presentation method based on virtual reality in another embodiment of the present application;

[0048] Figure 3 is a structural composition diagram of the visual presentation device based on virtual reality in the embodiment of the present application;

[0049] Figure 4 is a structural composition diagram of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0051] Embodiment one

[0052] Referring to Figure 1 , Figure 1 is a flowchart of a virtual reality-based visual presentation method in an embodiment of the present application, the method being applied to a virtual reality head-mounted display and a cloud server, the virtual reality head-mounted display being built-in with an eye movement tracking device, and the method comprising:

[0053] S11: determining, by the cloud server, a user pupil position of each frame of image based on image sequence data collected by the eye movement tracking device;

[0054] In the specific implementation of the present application, the determination of the user pupil position of each frame of image based on the image sequence data collected by the eye movement tracking device comprises: feature region extraction on each frame of image of the image sequence data to obtain corresponding pupil region images, and binaryzation processing on each pupil region image to obtain corresponding binaryzation pupil region images; extraction of pupil contour feature points of the binaryzation pupil region images based on a clustering algorithm, and determination of first position data of the pupil based on the pupil contour feature points; determination of second position data of the pupil based on a pupil point probability distribution map of each frame of image, and determination of the user pupil position of each frame of image based on the first position data and the second position data.

[0055] Specifically, the cloud server is configured to receive data transmitted by the eye movement tracking device and the virtual reality head-mounted display for analysis, obtain rendering data of the image, and transmit the rendered image to the virtual reality head-mounted display. The cloud server is in communication connection with the virtual reality head-mounted display. The virtual reality head-mounted display is configured to display corresponding virtual images. The virtual reality head-mounted display is built-in with an eye movement tracking device, which is a visual acquisition device such as a camera or a video camera, and is configured to acquire eye images of a user. The virtual reality head-mounted display uses a high-resolution display screen to present more delicate and realistic images. The virtual reality head-mounted display uses lightweight devices to ensure that the user does not feel uncomfortable after long-time use.

[0056] The eye tracking device collects a plurality of frames of user eye images to form image sequence data, extracts feature regions from each frame of the image sequence data, extracts feature regions of pupils from each frame of the image sequence data through a pre-trained deep convolutional neural network model, and obtains corresponding pupil region images. The pupil region images are binarized, the binarization of the image can use the maximum inter-class variance method, each frame of image is divided into two classes according to the initial threshold, the inter-class variance between the two is calculated, the threshold is updated, the inter-class variance is recalculated, and when the inter-class variance is maximum, the threshold is the binarization threshold. The binarization threshold is used to binarize each frame of image to obtain corresponding binarized pupil region images. The pupil contour feature points of the binarized pupil region images are extracted based on a clustering algorithm, the pixel points belonging to the pupil region and the iris region are clustered, the clustering threshold can be determined according to the gray characteristics of the general pupil region image, the gray of the pixel points in the pupil region and the gray of the pixel points in the iris region are inconsistent, the pixel points belonging to the pupil region are determined through the clustering algorithm, the pixel points in the pupil region are searched for the pixel points in the iris region, the pixel points in the iris region are taken as the pupil contour feature points, and the first position data of the pupil is determined based on the pupil contour feature points. According to the pupil contour feature points, an ellipse is fitted, the pupil contour is fitted, the position of the pupil is calculated according to the pupil contour, and the first position data is obtained. The second position data of the pupil is determined based on the pupil point probability distribution map of each frame of image, the corresponding heat map is generated from each frame of image through the deep neural network, the brightness in the heat map is used to represent the probability value, the higher the brightness in the heat map, the higher the probability that the position is a pupil point, the probability value of the heat map is solved using the argmax function, the points with probability values reaching a preset probability threshold are extracted, the pupil point probability distribution map is generated according to the extracted points and the probability values, the second position data of the pupil is calculated according to the positions of the points in the pupil point probability distribution map, and the user pupil position of each frame of image is determined based on the first position data and the second position data. The user pupil position of each frame of image can be determined by weighted average of the first position data and the second position data.

[0057] S12: analyzing the line of sight focus based on the user pupil position, obtaining line of sight focus information, and dividing a focus region of a virtual picture presented by the virtual reality head-mounted display based on the line of sight focus information;

[0058] In the implementation of the present application, the gaze focus analysis based on the user pupil position, the gaze focus information obtained, and the division of the focus area of the virtual picture corresponding to the virtual reality head-mounted display based on the gaze focus information include: determining the eye movement trajectory and the eye stay duration based on the user pupil position of each frame of image, and determining the gaze focus information based on the eye movement trajectory and the eye stay duration; and dividing the focus area of the virtual picture based on the gaze focus information and the preset focus area range.

[0059] Specifically, the eye movement trajectory and the eye stay duration are determined based on the user pupil position of each frame of image, the eye movement trajectory is determined according to the change of the user pupil position of each frame of image, the eye stay duration is determined according to the duration of the unchanged user pupil position of each frame of image, and the gaze focus information is determined based on the eye movement trajectory and the eye stay duration. The focus area of the virtual picture is divided based on the gaze focus information and the preset focus area range. When the eye movement trajectory stops and the eye stay duration exceeds the preset duration, the position of the virtual picture corresponding to the user pupil position at that moment is taken as the center point, and the focus area of the virtual picture corresponding to the virtual reality head-mounted display is divided based on the preset focus area range.

[0060] S13: performing sharpness rendering analysis on the picture image data of the focus area to obtain sharpness rendering data;

[0061] In the implementation of the present application, the sharpness rendering analysis on the picture image data of the focus area to obtain the sharpness rendering data includes: performing gray-scale processing on the picture image data to obtain gray-scale picture image data, and performing Gaussian filtering on the gray-scale picture image data to obtain filtered picture image data; performing gradient operation on the gray-scale picture image data and the filtered picture image data to obtain a first gradient image corresponding to the gray-scale picture image data and a second gradient image corresponding to the filtered picture image data; calculating image sharpness data based on the first gradient image and the second gradient image, and analyzing the brightness data of the picture image data; calculating a first difference value between the image sharpness data and a preset standard sharpness threshold, calculating a second difference value between the brightness data and a preset standard brightness threshold, and determining the tone rendering parameter and the sharpening effect rendering parameter based on the first difference value and the second difference value; and generating the sharpness rendering data based on the tone rendering parameter and the sharpening effect rendering parameter.

[0062] Specifically, several frame picture images presented by the focus area are intercepted, such as a user focus area is a certain role scene in an animation scene, several action frame picture images of the role scene are intercepted, picture image data is formed, the picture image data is grayed, the three primary colors of the picture image data are weighted and averaged with different weights, the weight values are distributed according to the sensitivity of the human eye to the three primary colors, the gray scale of the image is processed according to the weighted average value, the gray scale picture image data is obtained, and the gray scale picture image data is Gaussian filtered, the gray scale picture image data is Gaussian filtered through a Gaussian filter, the image is Gaussian filtered to effectively suppress noise, and the filtered picture image data is obtained. Gradient operation is performed on the gray scale picture image data and the filtered picture image data, which can be performed by a Sobel operator. The Sobel operator is used to calculate the gradient of the image, which calculates the gradient of the image by convolution operation on the image, obtains the first gradient image corresponding to the gray scale picture image data and the second gradient image corresponding to the filtered picture image data. The image sharpness data is calculated based on the first gradient image and the second gradient image, the first gradient image and the second gradient image are processed by blocking, the first gradient image block corresponding to the first gradient image and the second gradient image block corresponding to the second gradient image are obtained, the gray scale picture image data and the filtered picture image data are processed by blocking, the first picture image block corresponding to the gray scale picture image data and the second picture image block corresponding to the filtered picture image data are obtained, the mean values of the first gradient image block, the second gradient image block, the first picture image block and the second picture image block are calculated, the mean values can be the mean values of the pixel values, the variances of the first gradient image block and the second gradient image block are calculated according to the mean values of the first gradient image block and the second gradient image block, and the covariance between the first gradient image block and the second gradient image block is calculated, the brightness comparison value between the first picture image block and the second picture image block is calculated according to the mean values of the first picture image block and the second picture image block combined with the brightness constant, the gradient contrast comparison value and the gradient structure information comparison value between the first gradient image block and the second gradient image block are calculated according to the variances of the first gradient image block and the second gradient image block and the covariance between the first gradient image block and the second gradient image block combined with the gradient contrast constant and the gradient structure information constant, the image sharpness is calculated according to the brightness comparison value, the gradient contrast comparison value and the gradient structure information comparison value, that is, the image sharpness data, and the brightness data of the picture image data is analyzed, the average brightness is calculated according to the brightness value corresponding to each pixel point of the picture image data, that is, the brightness data.The first difference value of the image definition data and the preset standard definition threshold is calculated, the second difference value of the brightness data and the preset standard brightness threshold is calculated, and the tone rendering parameter and the sharpening effect rendering parameter are determined based on the first difference value and the second difference value. The corresponding tone rendering parameter and sharpening effect rendering parameter are matched in the database through the first difference value and the second difference value. The tone rendering parameter can optimize the tone effect of the virtual picture image, and the sharpening effect rendering parameter can optimize the sharpening effect of the picture image. The definition rendering data is generated based on the tone rendering parameter and the sharpening effect rendering parameter, that is, the definition rendering data is composed of the tone rendering parameter and the sharpening effect rendering parameter.

[0063] S14: performing user motion sickness rendering analysis on the picture image data of the focus point area based on the motion sickness score and the picture motion speed to obtain user motion sickness rendering data;

[0064] In the specific implementation process of the present application, the user motion sickness rendering data is obtained by performing user motion sickness rendering analysis on the picture image data of the focus point area based on the motion sickness score and the picture motion speed, including: analyzing the pixel displacement amount of adjacent picture image frames of the picture image data, and determining the picture motion speed based on the pixel displacement amount, and determining the adjustment parameter of the picture motion speed based on the motion sickness score when the user experiences the virtual picture; attention heat data is obtained by performing attention heat analysis on the picture image data, and the picture motion sickness rendering parameter is determined based on the attention heat data and the motion sickness score; the user motion sickness rendering data is generated based on the adjustment parameter and the picture motion sickness rendering parameter.

[0065] Specifically, the pixel displacement amount of the adjacent picture image frames of the picture image data is analyzed, a plurality of pixel sampling regions are intercepted from the adjacent picture image frames, the proportion of the same pixels in the pixel sampling regions of the adjacent picture image frames is calculated, the proportion is taken as the similarity, the two groups of pixel sampling regions with the highest similarity are taken as the same regions between the adjacent picture image frames, the pixel displacement amount of each same pixel in the same regions in the adjacent picture image frames is calculated, the average value of the pixel displacement amounts of all the same pixels is calculated, the average value is taken as the pixel displacement amount, the picture motion speed is determined based on the pixel displacement amount, the picture motion speed is calculated according to the pixel displacement amount and the time of conversion between the adjacent picture image frames, the adjustment parameter of the picture motion speed is determined based on the motion sickness score of the user when experiencing the virtual picture, when the user uses the virtual reality head-mounted display, there is an experience process of the virtual picture, the virtual reality head-mounted display is provided with a user motion sickness feedback mechanism, which can quantify the motion sickness score of the user according to the experience process of the virtual picture, for example, the motion sickness score can be set as a 10-point system, 0 points represent no motion sickness symptoms, and 10 points represent obvious motion sickness symptoms, the corresponding motion sickness picture motion speed threshold is determined according to the motion sickness score, whether the picture motion speed is greater than the motion sickness picture motion speed threshold is judged, if the picture motion speed is greater than the motion sickness picture motion speed threshold, it indicates that the current picture change exceeds the bearing threshold of the user, which will cause the user to have motion sickness symptoms, the difference between the picture motion speed and the motion sickness picture motion speed threshold can be taken as the adjustment parameter, if the picture motion speed is less than or equal to the motion sickness picture motion speed threshold, the picture motion speed does not need to be adjusted.The attention heat analysis is performed on the picture image data, pixel data at the same position in adjacent picture image frames of the picture image data is analyzed, the pixel data is input into an attention heat analysis model for attention heat analysis of the user, the attention heat analysis model is a converged model obtained by inputting sample data into a deep neural network for training, attention heat data is obtained, the attention heat of the user is a length of time that the user stays in the focus area, and picture motion sickness rendering parameters are determined based on the attention heat data and the motion sickness score, according to the attention heat data, picture shaking speed and light and shade change degree of the picture image in the focus area under the length of time that the user stays in the focus area are determined, the motion sickness picture shaking speed threshold and the motion sickness light and shade change degree threshold are determined according to the motion sickness score, whether the picture shaking speed is greater than the motion sickness picture shaking speed threshold is judged, whether the light and shade change degree is greater than the motion sickness light and shade change degree threshold is judged, if the picture shaking speed is greater than the motion sickness picture shaking speed threshold and / or the light and shade change degree is greater than the motion sickness light and shade change degree threshold, then the difference between the picture shaking speed and the motion sickness picture shaking speed threshold and the difference between the light and shade change degree and the motion sickness light and shade change degree threshold are taken as the picture motion sickness rendering parameters, if the picture shaking speed is less than or equal to the motion sickness picture shaking speed threshold and the light and shade change degree is less than or equal to the motion sickness light and shade change degree threshold, then the picture shaking speed and the light and shade change degree do not need to be adjusted. User motion sickness rendering data is generated based on the adjustment parameters and the picture motion sickness rendering parameters, and the virtual picture can be effectively prevented from motion sickness by the user motion sickness rendering data.

[0066] S15: rendering processing is performed on the picture image data of the focus area based on the definition rendering data and the user motion sickness rendering data, and rendered picture image data is obtained.

[0067] In the specific implementation process of the present application, the rendering processing is performed on the picture image data of the focus area based on the definition rendering data and the user motion sickness rendering data, and the rendered picture image data is obtained, including: determining the rendering resolution based on the field of view angle information and the anti-distortion information of the virtual reality head-mounted display; performing rendering processing on the picture image data of the focus area based on the rendering resolution, the definition rendering data and the user motion sickness rendering data, and obtaining the rendered picture image data.

[0068] Specifically, the field of view angle information and the anti-distortion information of the virtual reality head-mounted display are used to determine the rendering resolution, the lens of the optical instrument is taken as the vertex, the angle formed by the two edges of the maximum range of the object image of the measured target through the lens is called the field of view angle, in the virtual reality device, the field of view angle information is the field of view angle of the lens of the virtual reality head-mounted display, the anti-distortion information is the corresponding relationship between the image coordinates of the original image and the screen coordinates of the screen of the virtual reality head-mounted display, the rendering resolution of the picture image in the focal point area is determined according to the field of view angle information and the anti-distortion information, the pixel coordinates of the corresponding field of view angle coordinates on the screen of the virtual reality head-mounted display are determined according to the field of view angle information and the anti-distortion information, and the rendering resolution of the picture image in the focal point area is calculated according to the pixel coordinates. The picture image data of the focal point area is rendered based on the rendering resolution, the definition rendering data and the user motion sickness rendering data, the picture image data of the focal point area is rendered in resolution through the rendering resolution, the picture image is rendered in tone and sharpening effect according to the definition rendering data, the picture image frame is rendered in picture motion speed, picture jitter speed and light and shade change degree according to the user motion sickness rendering data, and the picture image data after rendering is obtained.

[0069] S16: The picture image data after rendering is transmitted to the virtual reality head-mounted display for visual presentation based on the push stream engine combined with the focal length adjustment.

[0070] In the specific implementation process of the application, the picture image data after rendering is transmitted to the virtual reality head-mounted display for visual presentation based on the push stream engine combined with the focal length adjustment, including: encoding the picture image data after rendering based on the encoding engine to obtain encoded picture image data, and transmitting the encoded picture image data to the virtual reality head-mounted display based on the push stream engine; determining the pupil distance based on the user pupil position, and determining the focal point adjustment value of the built-in lens of the virtual reality head-mounted display and the image center position adjustment value of the built-in image display source based on the pupil distance; and the virtual reality head-mounted display visually presents the encoded picture image data based on the focal point adjustment value of the built-in lens and the image center position adjustment value of the built-in image display source.

[0071] Specifically, the picture image data after the rendering processing is encoded based on an encoding engine, the encoding engine calls an encode module to encode the picture image data after the rendering processing, the encode module is a module for video image encoding, the encode module is composed of a plurality of encoding channels, the encoding channels are used to complete protocol encoding, the encoding of the picture image data can be quickly completed through the encode module to obtain encoded picture image data, and the encoded picture image data is transmitted to the virtual reality head-mounted display based on a streaming engine, the streaming engine is used to push the video image to the virtual reality head-mounted display, the streaming engine can select a corresponding data transmission path according to a network state to reduce data transmission delay. The pupil distance is determined based on the user pupil position, that is, the distance between the two pupils is determined according to the user pupil position, and the focal point adjustment value of the built-in lens and the image center position adjustment value of the built-in image display source of the virtual reality head-mounted display are determined based on the pupil distance, that is, the deviation of the focal point of the lens and the pupil center point of the human eye and the deviation of the image display source and the pupil center point are analyzed according to the pupil distance and the user pupil position, the focal point adjustment value of the lens is determined according to the deviation of the focal point of the lens and the pupil center point of the human eye, and the image center position adjustment value of the image display source is determined according to the deviation of the image display source and the pupil center point. The virtual reality head-mounted display visually presents the encoded picture image data based on the focal point adjustment value of the built-in lens and the image center position adjustment value of the built-in image display source, adjusts the focal point of the lens and the image center position of the image display source of the virtual reality head-mounted display based on the focal point adjustment value of the built-in lens and the image center position adjustment value of the built-in image display source, better adjusts the focal length of the lens and the center point position of the image display source of the virtual reality head-mounted display, makes the pupil center point of the user and the focal point of the lens and the image center point of the image display source on the same straight line, brings better visual effect to the user, decodes the encoded picture image data, maps the decoded picture image data in the screen focal point area of the virtual reality head-mounted display for display, and realizes the visual presentation of the virtual picture with better effect.

[0072] In the embodiment of the present application, the line-of-sight focus analysis is performed based on the user pupil position of each frame image, the line-of-sight focus information is obtained, the focus area of the virtual picture corresponding to the virtual reality head-mounted display is divided based on the line-of-sight focus information, and the focus area of the virtual picture is divided considering the change of the user attention focus, which can be more close to the visual presentation of the user in the actual space. The definition rendering analysis is performed on the picture image data of the focus area, the user motion sickness rendering analysis is performed on the picture image data of the focus area based on the motion sickness score and the picture motion speed, the rendering processing is performed on the picture image data of the focus area based on the definition rendering data and the user motion sickness rendering data, which greatly reduces the user's dizziness and improves the user experience. At the same time of improving the picture quality, the immersion of the user to the virtual reality experience is avoided. The picture image data after the rendering processing is transmitted to the virtual reality head-mounted display for visual presentation based on the streaming engine combined with the focal length adjustment, which helps to reduce the user's eye fatigue and reduce the delay, and provides the user with a more natural and comfortable visual experience.

[0073] Embodiment two

[0074] Please refer to Figure 2 , Figure 2 is a flowchart of a virtual reality-based visual presentation method in another embodiment of the present application, the method is applied to a virtual reality head-mounted display and a cloud server, the virtual reality head-mounted display is built-in with an eye movement tracking device, and the method comprises the following steps:

[0075] S201: The cloud server determines the user pupil position of each frame image based on the image sequence data collected by the eye movement tracking device;

[0076] S202: The line-of-sight focus analysis is performed based on the user pupil position, the line-of-sight focus information is obtained, and the focus area of the virtual picture corresponding to the virtual reality head-mounted display is divided based on the line-of-sight focus information;

[0077] S203: The definition rendering analysis is performed on the picture image data of the focus area, and the definition rendering data is obtained;

[0078] S204: The adjacent picture image frames of the picture image data are analyzed for the pixel displacement amount, the picture motion speed is determined based on the pixel displacement amount, and the corresponding motion sickness picture motion speed threshold value is determined according to the motion sickness score;

[0079] S205: It is judged whether the picture motion speed is greater than the motion sickness picture motion speed threshold value;

[0080] S206: If the picture motion speed is greater than the motion sickness picture motion speed threshold value, the difference between the picture motion speed and the motion sickness picture motion speed threshold value is taken as the adjustment parameter;

[0081] S207: If the picture motion speed is less than or equal to the picture motion speed threshold, the picture motion speed does not need to be adjusted.

[0082] S208: Attention heat analysis is performed on the picture image data to obtain attention heat data, and picture motion rendering parameters are determined based on the attention heat data and the motion sickness score;

[0083] S209: User motion sickness rendering data is generated based on the adjustment parameters and the picture motion rendering parameters;

[0084] S210: The picture image data of the focus area is rendered based on the clarity rendering data and the user motion sickness rendering data, and the rendered picture image data is obtained.

[0085] S211: The rendered picture image data is transmitted to the virtual reality head-mounted display for visual presentation based on the streaming engine combined with the focal length adjustment.

[0086] In the embodiment of the application, the line of sight focus information is obtained based on the user pupil position of each frame of image, the focus area of the virtual picture corresponding to the virtual reality head-mounted display is divided based on the line of sight focus information, and the focus area of the virtual picture is divided considering the change of the user attention focus. The visual presentation of the user in the actual space can be more close. The picture image data of the focus area is analyzed for clarity rendering, the picture image data of the focus area is analyzed for user motion sickness rendering based on the motion sickness score and the picture motion speed, the picture image data of the focus area is rendered based on the clarity rendering data and the user motion sickness rendering data, which greatly reduces the user's dizziness and improves the user experience. At the same time, the picture quality is improved without affecting the user's immersive feeling of virtual reality experience. The rendered picture image data is transmitted to the virtual reality head-mounted display for visual presentation based on the streaming engine combined with the focal length adjustment, which helps to reduce the user's eye fatigue and reduce the delay, and provides the user with a more natural and comfortable visual experience.

[0087] Embodiment three

[0088] Please refer to Figure 3 , Figure 3 is a structural composition diagram of the visual presentation device based on virtual reality in the embodiment of the application. The device is applied to a virtual reality head-mounted display and a cloud server. The virtual reality head-mounted display is built-in with an eye tracking device. The device comprises:

[0089] The pupil position determination module 31 is used to determine the user pupil position of each frame of image based on the image sequence data collected by the eye tracking device.

[0090] The focus area division module 32 is configured to perform a line-of-sight focus analysis based on the user pupil position, obtain line-of-sight focus information, and divide a virtual picture corresponding to the presentation of the virtual reality head-mounted display into a focus area based on the line-of-sight focus information.

[0091] The definition analysis module 33 is configured to perform definition rendering analysis on the picture image data of the focus area, and obtain definition rendering data.

[0092] The motion sickness rendering analysis module 34 is configured to perform user motion sickness rendering analysis on the picture image data of the focus area based on the motion sickness score and the picture motion speed, and obtain user motion sickness rendering data.

[0093] The rendering processing module 35 is configured to perform rendering processing on the picture image data of the focus area based on the definition rendering data and the user motion sickness rendering data, and obtain picture image data after rendering processing.

[0094] The visual presentation module 36 is configured to transmit the picture image data after rendering processing to the virtual reality head-mounted display for visual presentation based on the streaming engine in combination with the focal length adjustment.

[0095] In the implementation process of the present application, the specific implementation of the device item can refer to the implementation of the above-mentioned method item, which will not be repeated here.

[0096] In the embodiment of the present application, the line-of-sight focus analysis is performed based on the user pupil position of each frame of image, the line-of-sight focus information is obtained, the virtual picture corresponding to the presentation of the virtual reality head-mounted display is divided into a focus area based on the line-of-sight focus information, and the focus area of the virtual picture is divided considering the change of the user attention focus, which can be more close to the visual presentation of the user in the actual space. The definition rendering analysis is performed on the picture image data of the focus area, the user motion sickness rendering analysis is performed on the picture image data of the focus area based on the motion sickness score and the picture motion speed, the rendering processing is performed on the picture image data of the focus area based on the definition rendering data and the user motion sickness rendering data, which greatly reduces the dizziness of the user and improves the user experience, improves the picture quality while avoiding affecting the immersion of the user in the virtual reality experience. The picture image data after rendering processing is transmitted to the virtual reality head-mounted display for visual presentation based on the streaming engine in combination with the focal length adjustment, which helps to reduce the eye fatigue of the user and reduce the delay, and provides a more natural and comfortable visual experience for the user.

[0097] The computer readable storage medium provided by the embodiment of the present application stores a computer program, and the program is executed by a processor to realize the virtual reality based visual presentation method of any one of the above embodiments. The computer readable storage medium includes but is not limited to any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the storage device includes any medium that stores or transmits information in a form capable of being read by a device (for example, a computer, a mobile phone), which can be a read-only memory, a magnetic disk or an optical disk, etc.

[0098] Embodiment Four

[0099] Please refer to Figure 4 , Figure 4 is a structural composition diagram of an electronic device in the embodiment of the present application.

[0100] The embodiment of the present application further provides an electronic device, as shown in Figure 4 , the electronic device includes a memory 41, a processor 43, and a computer program 42 stored in the memory 41 and executable on the processor 43. Those skilled in the art can understand that Figure 3The electronic device shown does not constitute a limitation on all devices, and can include more or fewer components than shown, or combine some components. The memory 41 can be used to store computer programs 42 and various functional modules, and the processor 43 runs the computer programs 42 stored in the memory 41 to perform various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include both the internal memory and the external memory. The internal memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. The external memory can include a hard disk, a floppy disk, a ZIP disk, a USB disk, a magnetic tape, etc. The processor 43 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip processor, or the processor 43 can also be any conventional processor, etc. The processor and the memory disclosed in the present application include but are not limited to these types of processors and memories. The processor and the memory disclosed in the present application are only examples and are not limited.

[0101] As an embodiment, the electronic device includes one or more processors 43, a memory 41, and one or more computer programs 42, wherein the one or more computer programs 42 are stored in the memory 41 and configured to be executed by the one or more processors 43, and the one or more computer programs 42 are configured to perform the virtual reality-based visual presentation method in any one of the above embodiments. For specific implementation process, please refer to the above embodiments, which will not be repeated here.

[0102] In the embodiment of the present application, the line-of-sight focus analysis is performed based on the user pupil position of each frame image to obtain line-of-sight focus information, the virtual picture corresponding to the virtual reality head-mounted display is divided into focus areas based on the line-of-sight focus information, and the focus areas of the virtual picture are divided considering the change of the user attention focus, which can be more close to the visual presentation of the user in the actual space. The definition rendering analysis is performed on the picture image data of the focus areas, the user motion sickness rendering analysis is performed on the picture image data of the focus areas based on the motion sickness score and the picture motion speed, the rendering processing is performed on the picture image data of the focus areas based on the definition rendering data and the user motion sickness rendering data, which greatly reduces the user's dizziness and improves the user experience. At the same time of improving the picture quality, the immersion of the user to the virtual reality experience is avoided. The picture image data after the rendering processing is transmitted to the virtual reality head-mounted display for visual presentation based on the streaming engine combined with the focal length adjustment, which helps to reduce the user's eye fatigue and reduce the delay, and provides the user with a more natural and comfortable visual experience.

[0103] In addition, the above describes in detail the method and related device for visual presentation based on virtual reality provided by the embodiment of the present application, and the principle and implementation mode of the present application are described by using specific examples in this paper. The above embodiment is only used to help understand the method and core idea of the present application; at the same time, for the general technical personnel in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A method of visual presentation based on virtual reality, characterized by, The application is applied to a virtual reality head-mounted display and a cloud server, and the virtual reality head-mounted display is internally provided with an eye movement tracking device; the method comprises the following steps: The cloud server determines the user pupil position of each frame image based on the image sequence data collected by the eye movement tracking device; Based on the user pupil position, the gaze focus is analyzed, the gaze focus information is obtained, and the virtual picture corresponding to the virtual reality head-mounted display is divided into a focus area based on the gaze focus information; The picture image data of the focus area is subjected to sharpness rendering analysis to obtain sharpness rendering data, including: the picture image data is subjected to grayscale processing to obtain grayscale picture image data, and the grayscale picture image data is subjected to Gaussian filtering to obtain filtered picture image data; the grayscale picture image data and the filtered picture image data are subjected to gradient operation to obtain a first gradient image corresponding to the grayscale picture image data and a second gradient image corresponding to the filtered picture image data; the image sharpness data is calculated based on the first gradient image and the second gradient image, and the brightness data of the picture image data is analyzed; the first difference value between the image sharpness data and the preset standard sharpness threshold value is calculated, the second difference value between the brightness data and the preset standard brightness threshold value is calculated, and the tone rendering parameter and the sharpening effect rendering parameter are determined based on the first difference value and the second difference value; the sharpness rendering data is generated based on the tone rendering parameter and the sharpening effect rendering parameter; The picture image data of the focus area is subjected to user motion sickness rendering analysis based on the motion sickness score and the picture motion speed to obtain user motion sickness rendering data, including: the pixel displacement amount of adjacent picture image frames of the picture image data is analyzed, and the picture motion speed is determined based on the pixel displacement amount; the adjustment parameter of the picture motion speed is determined based on the motion sickness score when the user experiences the virtual picture; the attention heat data is obtained by analyzing the attention heat of the picture image data, and the picture motion sickness rendering parameter is determined based on the attention heat data and the motion sickness score; the user motion sickness rendering data is generated based on the adjustment parameter and the picture motion sickness rendering parameter; The picture image data of the focus area is subjected to rendering processing based on the sharpness rendering data and the user motion sickness rendering data to obtain the picture image data after rendering processing; The picture image data after rendering processing is transmitted to the virtual reality head-mounted display for visual presentation based on the push stream engine combined with the focal length adjustment.

2. The virtual reality-based visual presentation method of claim 1, wherein, The user pupil position of each frame image is determined based on the image sequence data collected by the eye movement tracking device, including: The feature region of each frame image of the image sequence data is extracted to obtain the corresponding pupil region image, and the binary processing is performed on each pupil region image to obtain the corresponding binary pupil region image; The pupil contour feature points of the binary pupil region image are extracted based on the clustering algorithm, and the first position data of the pupil is determined based on the pupil contour feature points; The second position data of the pupil is determined based on the pupil point probability distribution map of each frame image, and the user pupil position of each frame image is determined based on the first position data and the second position data.

3. The virtual reality-based visual presentation method of claim 1, wherein, The gaze focus analysis based on the user pupil position is performed to obtain gaze focus information, and the virtual picture presented by the virtual reality head-mounted display is divided into a focus area based on the gaze focus information, including: The user pupil position of each frame of image is used to determine the eye movement trajectory and the eye stay time, and the gaze focus information is determined based on the eye movement trajectory and the eye stay time; The focus area of the virtual picture is divided based on the gaze focus information and the preset focus area range.

4. The virtual reality-based visual presentation method of claim 1, wherein, The picture image data of the focus area is rendered based on the clarity rendering data and the user motion sickness rendering data to obtain the rendered picture image data, including: The rendering resolution is determined based on the field of view angle information and the anti-distortion information of the virtual reality head-mounted display; The picture image data of the focus area is rendered based on the rendering resolution, the clarity rendering data and the user motion sickness rendering data to obtain the rendered picture image data.

5. The virtual reality-based visual presentation method of claim 1, wherein, The rendered picture image data is transmitted to the virtual reality head-mounted display for visual presentation based on the push stream engine and the focal length adjustment, including: The rendered picture image data is encoded based on the encoding engine to obtain encoded picture image data, and the encoded picture image data is transmitted to the virtual reality head-mounted display based on the push stream engine; The pupil distance is determined based on the user pupil position, and the focal point adjustment value of the built-in lens and the image center position adjustment value of the built-in image display source of the virtual reality head-mounted display are determined based on the pupil distance; The virtual reality head-mounted display visually presents the encoded picture image data based on the focal point adjustment value of the built-in lens and the image center position adjustment value of the built-in image display source.

6. A virtual reality-based visual presentation device, characterized by The application is applied to a virtual reality head-mounted display and a cloud server, and the virtual reality head-mounted display is built-in with an eye movement tracking device; the device includes: A pupil position determination module is used to determine the user pupil position of each frame of image based on the image sequence data collected by the eye movement tracking device; A focus area division module is used to perform gaze focus analysis based on the user pupil position to obtain gaze focus information, and divide the virtual picture presented by the virtual reality head-mounted display into a focus area based on the gaze focus information. The sharpness analysis module is configured to perform sharpness rendering analysis on the picture image data of the focus area to obtain sharpness rendering data, including: performing grayscale on the picture image data to obtain grayscale picture image data, and performing Gaussian filtering on the grayscale picture image data to obtain filtered picture image data; performing gradient operation on the grayscale picture image data and the filtered picture image data to obtain a first gradient image corresponding to the grayscale picture image data and a second gradient image corresponding to the filtered picture image data; calculating image sharpness data based on the first gradient image and the second gradient image, and analyzing brightness data of the picture image data; calculating a first difference value between the image sharpness data and a preset standard sharpness threshold, calculating a second difference value between the brightness data and a preset standard brightness threshold, and determining a tone rendering parameter and a sharpening effect rendering parameter based on the first difference value and the second difference value; and generating the sharpness rendering data based on the tone rendering parameter and the sharpening effect rendering parameter; The motion sickness rendering analysis module is configured to perform user motion sickness rendering analysis on the picture image data of the focus area based on a motion sickness score and picture motion speed to obtain user motion sickness rendering data, including: analyzing pixel displacement of adjacent picture image frames of the picture image data, and determining the picture motion speed based on the pixel displacement, determining an adjustment parameter of the picture motion speed based on the motion sickness score when a user experiences a virtual picture; performing attention heat analysis on the picture image data to obtain attention heat data, and determining picture motion sickness rendering parameters based on the attention heat data and the motion sickness score; and generating the user motion sickness rendering data based on the adjustment parameter and the picture motion sickness rendering parameters; The rendering processing module is configured to perform rendering processing on the picture image data of the focus area based on the sharpness rendering data and the user motion sickness rendering data to obtain rendered picture image data. The visual presentation module is configured to transmit the rendered picture image data to a virtual reality head-mounted display for visual presentation based on a streaming engine combined with focus adjustment. 7.An electronic device comprising a processor and a memory, wherein The memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory, so that the electronic device performs the virtual reality-based visual presentation method in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, which, when executed on an electronic device, cause the electronic device to perform the virtual reality-based visual presentation method in any one of claims 1 to 5.

Images