An image rendering method, device, virtual reality device and storage medium

Abstract

The application discloses an image rendering method and device, a virtual reality device and a storage medium, and relates to the technical field of virtual reality devices. The method comprises the following steps: when it is detected that the gaze point data of a user is lost, obtaining historical gaze point data sets and current scene information; determining first gaze point data according to the historical gaze point data sets and the current scene information, and determining second gaze point data according to the current scene information; determining target gaze point data according to the first gaze point data and the second gaze point data; and performing image rendering according to the target gaze point data. The method solves the problem that image rendering cannot be performed after the gaze point data is lost, and, after the gaze point data of a user is detected to be lost, the gaze point data is predicted according to historical gaze point data sets and current scene information, so that the problem that image rendering cannot be performed after the gaze point data is lost is avoided. Image rendering is performed according to the target gaze point data obtained through prediction, the fluency of a picture is ensured, the picture does not appear to be stuck, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image rendering method, apparatus, virtual reality device, and storage medium. Background Technology

[0002] Current virtual reality (VR) foveated rendering methods are based on obtaining eye-related data and then rendering high-resolution, sub-high-resolution, and low-resolution areas by estimating the foveated point. This requires communication between the foveated point and the computer's graphics card system interface. The graphics card obtains the foveated coordinates and performs the rendering work to differentiate between different resolutions. Therefore, foveated direction data must be input before image rendering can begin.

[0003] Focal rendering requires gaze coordinates or direction, but occasionally, gaze data may be lost due to interference with the eye information acquisition sensor or other reasons. Without gaze data, the focal rendering workflow cannot proceed normally, resulting in stuttering and causing dizziness, thus affecting the user experience. Therefore, how to render images when focal data is missing is a problem that needs to be solved. Summary of the Invention

[0004] This invention provides an image rendering method, apparatus, virtual reality device, and storage medium to solve the problem of image rendering being impossible after gaze point data is lost.

[0005] According to one aspect of the present invention, an image rendering method is provided, the method comprising:

[0006] Once the user's gaze data is detected to be lost, the historical gaze data set and current scene information are retrieved.

[0007] The first gaze point data is determined based on the historical gaze point dataset and the current scene information, and the second gaze point data is determined based on the current scene information;

[0008] Determine the target gaze point data based on the first gaze point data and the second gaze point data;

[0009] Image rendering is performed based on the target gaze point data.

[0010] According to another aspect of the present invention, an image rendering apparatus is provided, the apparatus comprising:

[0011] The information acquisition module is used to acquire historical gaze point datasets and current scene information when the user's gaze point data is detected to be lost.

[0012] The gaze point data acquisition module is used to determine first gaze point data based on the historical gaze point dataset and current scene information, and to determine second gaze point data based on the current scene information;

[0013] The target data determination module is used to determine target gaze point data based on the first gaze point data and the second gaze point data.

[0014] The image rendering module is used to render images based on the target gaze point data.

[0015] According to another aspect of the present invention, a virtual reality device is provided, the virtual reality device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image rendering method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the image rendering method according to any embodiment of the present invention.

[0020] The technical solution of this invention, upon detecting the loss of user gaze data, acquires a historical gaze data set and current scene information; determines first gaze data based on the historical gaze data set and current scene information; determines second gaze data based on the current scene information; determines target gaze data based on the first and second gaze data; and performs image rendering based on the target gaze data. This solves the problem of being unable to render images after gaze data loss. After detecting the loss of user gaze data, gaze data is predicted based on the historical gaze data set and current scene information; first gaze data is determined based on the historical gaze data set and current scene information; second gaze data is determined based on the current scene information; and then target gaze data is determined based on the first and second gaze data. This achieves accurate prediction of gaze data, avoids the problem of being unable to render images after gaze data loss, and ensures smooth image rendering based on the predicted target gaze data, preventing image stuttering and improving user experience.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an image rendering method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of an image rendering method provided according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of an image rendering apparatus according to Embodiment 3 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of a virtual reality device that implements the image rendering method of this invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," "target," "historical," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1 The flowchart of an image rendering method provided in Embodiment 1 of the present invention is applicable to the situation of image rendering when gaze point data is lost. The method can be executed by an image rendering device, which can be implemented in hardware and / or software and can be configured in electronic devices such as VR devices and AR devices.

[0031] It should be noted that the application environment for the image rendering method in this embodiment can be described as follows: images are displayed through VR devices or ordinary smart devices. Taking the images displayed during a game as an example, the user's gaze position changes as the scene changes during gameplay; that is, the position of the gaze point may also be different depending on the scene. The displayed image is rendered by collecting the user's gaze point information. In existing implementations, since image rendering requires gaze point data, rendering cannot be performed if the gaze point data is lost, resulting in screen stuttering and affecting the user experience.

[0032] The image rendering method provided in this embodiment can predict gaze point data after gaze point data is lost, and perform image rendering based on the predicted gaze point data, thereby overcoming the problems existing in the prior art.

[0033] like Figure 1 As shown, the method includes:

[0034] S101. When the user's gaze data is detected to be lost, obtain the historical gaze data set and the current scene information.

[0035] In this embodiment, gaze point data can be specifically understood as information about a specific point on an object that the user's gaze is focused on. For example, gaze point data could be the coordinates of the gaze point, the user's gaze direction, the time of the gaze, etc. The user refers to a user using (or wearing) a VR device, eye-tracking device, or other smart device with gaze point data collection capabilities. The historical gaze point dataset can be specifically understood as a collection of data storing historical gaze point data. Historical gaze point data represents the gaze point data of a user when looking at an object before the current time. The historical gaze point data in the historical gaze point dataset can be from one user or multiple users. Current scene information can be specifically understood as the scene currently displayed in the image. For example, when a user is playing a game while wearing a VR device, the current scene information could be the enemy's position, the user's aiming point, and the positions of objects or characters within the scene when the user is aiming at the enemy before shooting.

[0036] This application takes an eye-control device that collects gaze point data as an example. The eye-control device collects the user's gaze point data in real time. When sensor malfunction or other reasons prevent the collection of gaze point data, or when the gaze point data is abnormal, it is determined that the gaze point data is lost. Simultaneously, while the eye-control device collects the user's gaze point data in real time, the gaze point data is stored. The storage method can be local storage, server storage, etc. To facilitate data retrieval, the gaze point data is preferably stored in chronological order to form a historical gaze point dataset. Since different users have corresponding historical data, the historical gaze point dataset can be associated with the user when forming the historical gaze point dataset. When the loss of a user's gaze point data is detected, the historical gaze point dataset associated with the user is retrieved from the corresponding storage space. Alternatively, when not distinguishing between different users' historical gaze point data, historical gaze point data can be stored in a single historical gaze point dataset, and the historical gaze point dataset can be directly retrieved. Current scene information can be directly obtained from the electronic device or server displaying the image. For example, when displaying images through a VR device, the VR device pre-stores all scene information during the game, and the current scene information corresponding to the current moment can be directly retrieved.

[0037] In this application embodiment, the historical gaze point dataset is preferably a dataset associated with the user.

[0038] S102. Determine the first gaze point data based on the historical gaze point dataset and the current scene information, and determine the second gaze point data based on the current scene information.

[0039] In this embodiment, the first gaze point data can be understood as gaze point data predicted based on both historical gaze point datasets and current scene information. The second gaze point data can be understood as gaze point data predicted based on only the current scene information. Both the first and second gaze point data include at least the location information of the gaze point, such as gaze point coordinates.

[0040] Specifically, the historical gaze point dataset includes the user's historical gaze point data, reflecting all points the user has gazed at before the current time. Based on the current scene information, this historical gaze point data is filtered to obtain the first gaze point data matching the current scene information. During image display, the scene information is fixed. However, the scene may contain special points, which can be fixed coordinates used to guide the user or coordinates that the user might be interested in. For example, when an enemy appears, coordinates can be generated within the enemy's body area as special points to guide the user. Before displaying games or other images, the images are pre-determined. Therefore, special points in each scene can be pre-determined based on the scene. After determining the current scene information, the corresponding special point data is determined based on the current scene information, and this data is used as the second gaze point data.

[0041] S103. Determine the target gaze point data based on the first gaze point data and the second gaze point data.

[0042] In this embodiment, the target gaze point data can be specifically understood as the final predicted gaze point data. By comparing the difference between the first gaze point data and the second gaze point data, it is determined whether the first gaze point data and the second gaze point data match. If they match, the predicted first gaze point data and the second gaze point data are determined to be accurate. Either data is selected from the first gaze point data and the second gaze point data as the target gaze point data, or the target gaze point data is obtained by performing calculations on the first gaze point data and the second gaze point data, such as averaging or weighting. When the first gaze point data and the second gaze point data do not match, there is inaccurate data in the first gaze point data and the second gaze point data. The target gaze point data is determined based on the reliability of the first gaze point data and the second gaze point data.

[0043] S104. Render the image based on the target gaze point data.

[0044] After determining the target gaze point data, the target gaze point data is used as the user's gaze point to render the image. For example, based on the target gaze point data, the image is divided into high-resolution, second-high-resolution, and low-resolution regions, and then each region is rendered according to its corresponding resolution.

[0045] This application provides an image rendering method that, upon detecting the loss of user gaze point data, acquires a historical gaze point dataset and current scene information; determines first gaze point data based on the historical gaze point dataset and current scene information; determines second gaze point data based on the current scene information; determines target gaze point data based on the first and second gaze point data; and performs image rendering based on the target gaze point data. This method solves the problem of being unable to render images after the loss of gaze point data. After detecting the loss of user gaze point data, the method predicts gaze point data based on the historical gaze point dataset and current scene information, determines the first gaze point data based on the historical gaze point dataset and current scene information, determines the second gaze point data based on the current scene information, and then determines the target gaze point data based on the first and second gaze point data. This achieves accurate prediction of gaze point data, avoids the problem of being unable to render images after the loss of gaze point data, and ensures smooth image rendering based on the predicted target gaze point data, preventing image stuttering and improving user experience.

[0046] Example 2

[0047] Figure 2 This is a flowchart of an image rendering method provided in Embodiment 2 of the present invention. This embodiment is a refinement based on the above embodiments. Figure 2 As shown, the method includes:

[0048] S201. When the user's gaze point data is detected to be lost, obtain the historical gaze point dataset and the current scene information.

[0049] The system collects and analyzes user gaze data in real time. When it detects that user gaze data is lost, it retrieves historical gaze datasets and current scene information.

[0050] S202. Filter the historical gaze point datasets according to time to determine the candidate historical datasets.

[0051] In this embodiment, the candidate historical dataset can be specifically understood as a dataset storing candidate historical gaze point data, which consists of gaze point data from different scenarios. Since the data in the historical gaze point dataset can be gaze point data collected from repeatedly viewing the same image, each historical gaze point data is labeled with a time. Taking a shooting game as an example, each time a user plays the game, historical gaze point data is recorded, and all historical gaze point data from a single game can constitute a subset of the dataset. When a user repeatedly plays the same game, multiple gaze point information will appear in the same scenario; that is, the gaze point may be different or the same each time. The historical gaze point dataset is filtered according to time to obtain the latest historical gaze point subset for each scenario, forming the candidate historical dataset. For the same game session, if the scenario remains unchanged but the user's gaze point changes, since the latest historical gaze point data is selected, the resulting historical gaze point subset is also the latest gaze point data for each scenario.

[0052] S203. Based on the current scenario information, filter the candidate historical datasets to determine the target historical data corresponding to the current scenario information.

[0053] In this embodiment, the target historical data can be specifically understood as historical gaze point data that matches scene information. The candidate historical dataset stores historical gaze point data under different scenes. After determining the current scene information, the candidate historical dataset is filtered according to the identifier of the current scene information to obtain the target historical data corresponding to the current scene information. For example, in a game, each scene corresponds to one frame of data. The scene information in the candidate historical dataset that matches the current scene information is determined according to the identifier of the data frame, and the historical gaze point data corresponding to this scene information is used as the target historical data.

[0054] S204. Determine the target's historical data as the first fixation point data.

[0055] It's important to know that S202-S204 determines the first fixation point data, and S205-S209 determines the second fixation point data. There's no strict order in their execution; S202-S204 can be executed first, followed by S205-S209; or S205-S209 can be executed first, followed by S202-S204; or they can be executed in parallel. The specific order can be set according to the requirements in practical applications. Figure 2 Take parallel execution as an example.

[0056] S205. Determine whether the current scene information has a corresponding stimulus point. If yes, execute S206; otherwise, execute S208.

[0057] In this embodiment, a stimulus point can be specifically understood as the coordinate point of an object that guides the user's gaze and stimulates the user's vision. Each scene can be pre-set with its associated stimulus points. For example, in a shooting game, the bullseye in the scene is the stimulus point; or, for example, if an object suddenly appears or moves in the game, that object can be understood as a stimulus point. If there are no coordinate points in the scene that the user needs to pay special attention to, no stimulus point can be set. Based on the association between scene information and stimulus points, it is determined whether the current scene information has a corresponding stimulus point. Usually, a maximum of one stimulus point is set in a scene. If multiple stimulus points are set in the same scene, their priorities can be set. For example, the priority of the stimulus point in the middle can be set to high priority, and the order of stimulus points can be determined according to the order of their appearance.

[0058] S206. Determine the stimulation point data based on the current scene information.

[0059] In this embodiment, stimulus point data can be specifically understood as the corresponding data of the stimulus point, such as the coordinates and size of the stimulus point. The stimulus point data is set simultaneously with the pre-set stimulus points to accurately describe them. The stimulus points are determined based on the correlation between the current scene information and the stimulus points; correspondingly, the stimulus point data can also be determined from the pre-set data.

[0060] S207. Use the stimulus point data as the second fixation point data.

[0061] The stimulus point data is directly used as the second fixation point data to predict the fixation point, and S210 is executed.

[0062] S208. Determine the user's gaze point of interest based on the current scene information.

[0063] In this embodiment, the gaze point of interest can be specifically understood as the coordinate point that the user is interested in. The current scene information may contain points of interest to the user; for example, in a shooting game, the position the gun is pointing at is the point of interest; in a shooting game, the aiming point is the point of interest. The user's gaze point of interest is determined by analyzing the current scene information. This analysis can be done through prediction using a neural network model. The neural network model is pre-trained based on the image corresponding to the scene and the labeled gaze points of interest to obtain a neural network model capable of predicting gaze points of interest.

[0064] S209. Use the fixation data of the point of interest as the second fixation data.

[0065] S210. Determine whether the first fixation point data and the second fixation point data match. If yes, execute S211; otherwise, execute S212.

[0066] The method to determine whether the first fixation point data and the second fixation point data match can be to calculate the distance between the two points. If the distance is within a certain range, the first fixation point data and the second fixation point data are determined to match, and S211 is executed; if the distance is not within a certain range, the first fixation point data and the second fixation point data are determined to not match, and S212 is executed.

[0067] As an optional embodiment of this example, this optional embodiment further optimizes the determination of whether the first gaze point data and the second gaze point data match to A1-A3:

[0068] A1. Calculate the coordinate difference between the first fixation point data and the second fixation point data.

[0069] In this embodiment, the coordinate difference can be the difference between the horizontal coordinates, the difference between the vertical coordinates, or a combination of the horizontal and vertical coordinates. The coordinate difference between the first gaze point data and the second gaze point data can be calculated by calculating the difference between the horizontal coordinates, calculating the difference between the vertical coordinates, or calculating a combination of the horizontal and vertical coordinates. In this application, the combination of the horizontal and vertical coordinates can be the distance between the two points, that is, the distance between the first gaze point and the second gaze point.

[0070] A2. When the coordinate difference is within the preset difference range, determine that the first gaze point data and the second gaze point data are matched.

[0071] A pre-defined range of difference values ​​is used. When calculating multiple types of coordinate differences, different types of coordinate differences can correspond to different ranges. The coordinate differences are compared to their corresponding ranges. If only one type of coordinate difference exists, and the difference falls within the range, the first and second gaze point data are considered to match. When multiple types of coordinate differences exist, the condition for matching the first and second gaze point data can be that more than a preset number of coordinate differences fall within the corresponding range. For example, when there are three types of coordinate differences, the first and second gaze point data are considered to match if at least two types of coordinate differences fall within the corresponding range.

[0072] A3. When the coordinate difference is not within the preset difference range, it is determined that the first gaze point data and the second gaze point data do not match.

[0073] If there is only one type of coordinate difference, and the coordinate difference is outside the range of the difference value, it is determined that the first gaze point data and the second gaze point data do not match. When there are multiple types of coordinate differences, the condition for the first gaze point data and the second gaze point data to not match can be: the number of coordinate differences within the corresponding difference value range does not exceed a preset number.

[0074] S211. Use the first gaze point data or the second gaze point data as the target gaze point data.

[0075] If the first gaze point data and the second gaze point data match, and there is little difference between the first gaze point data and the second gaze point data, then any one of the gaze point data can be selected as the target gaze point data.

[0076] S212. Determine the target gaze point data based on the priority of the first gaze point data and the second gaze point data.

[0077] The priorities of the first and second fixation point data are preset, and the target fixation point data is determined by comparing the priorities.

[0078] As an optional embodiment of this example, this optional embodiment further optimizes the target fixation point data to B1-B2 based on the priority of the first fixation point data and the second fixation point data:

[0079] B1. Determine the priority of the first fixation point data and the second fixation point data.

[0080] The priorities of the first and second fixation point data are predetermined, and the priorities of the first and second fixation point data can be directly obtained when determining the target fixation point data.

[0081] B2. Prioritize high-priority fixation data as target fixation data.

[0082] The priorities of the first and second fixation point data are compared, and the fixation point data with higher priority is determined and used as the target fixation point data. This application uses priority settings to reflect the reliability of the data; data with higher priority is more reliable.

[0083] S213. Render the image based on the target gaze point data.

[0084] This application provides an image rendering method. This embodiment refines the process of determining first gaze point data, second gaze point data, and target gaze point data. It filters historical gaze point datasets using time and current scene information to determine the first gaze point data. Then, it determines stimulus point data for stimuli or gaze points of interest using current scene information, thereby determining the second gaze point data. This solves the problem of image rendering failure due to lost gaze point data. Upon detecting lost user gaze point data, it predicts gaze point data based on historical gaze point datasets and current scene information, achieving accurate prediction and avoiding the inability to render images after gaze point data loss. By matching the first and second gaze point data, the target gaze point data is determined, ensuring its accuracy. Image rendering is performed based on the predicted target gaze point data, ensuring smooth image rendering without stuttering and improving user experience.

[0085] Example 3

[0086] Figure 3 This is a schematic diagram of the structure of an image rendering device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an information acquisition module 31, a gaze point data acquisition module 32, a target data determination module 33, and an image rendering module 34;

[0087] Among them, the information acquisition module 31 is used to acquire historical gaze point dataset and current scene information when the user's gaze point data is detected to be lost;

[0088] The fixation point data acquisition module 32 is used to determine first fixation point data based on the historical fixation point dataset and current scene information, and to determine second fixation point data based on the current scene information;

[0089] Target data determination module 33 is used to determine target gaze point data based on the first gaze point data and the second gaze point data;

[0090] The image rendering module 34 is used to render the image based on the target gaze point data.

[0091] This application provides an image rendering apparatus that solves the problem of image rendering failure after gaze point data loss. Upon detecting the loss of user gaze point data, the apparatus predicts gaze point data based on historical gaze point datasets and current scene information. It then determines first gaze point data based on the historical gaze point dataset and current scene information, determines second gaze point data based on the current scene information, and finally determines target gaze point data based on the first and second gaze point data. This achieves accurate prediction of gaze point data, avoiding the problem of image rendering failure after gaze point data loss. Image rendering is performed based on the predicted target gaze point data, ensuring smooth image playback and preventing stuttering, thus improving the user experience.

[0092] Optionally, the gaze point data acquisition module 32 includes:

[0093] The alternative data filtering unit is used to filter the historical gaze point dataset according to time to determine alternative historical datasets;

[0094] The target data filtering unit is used to filter the candidate historical dataset based on the current scene information to determine the target historical data corresponding to the current scene information;

[0095] The first data determination unit is used to determine the target historical data as the first gaze point data.

[0096] Optionally, the gaze point data acquisition module 32 includes:

[0097] A stimulus data determination unit is used to determine the stimulus point data of the stimulus point based on the current scene information if the current scene information has a corresponding stimulus point.

[0098] The second data determination unit is used to use the stimulation point data of the stimulation point as the second fixation point data.

[0099] Optionally, the gaze point data acquisition module 32 includes:

[0100] The point of interest determination unit is used to determine the user's gaze point of interest based on the current scene information if the current scene information does not have a corresponding stimulus point;

[0101] The third data determination unit is used to use the gaze point data of the gaze point of interest as the second gaze point data.

[0102] Optionally, the target data determination module 33 is specifically used to: determine whether the first gaze point data and the second gaze point data match; if so, use the first gaze point data or the second gaze point data as the target gaze point data; otherwise, determine the target gaze point data according to the priority of the first gaze point data and the second gaze point data.

[0103] Optionally, the method for determining whether the first gaze point data and the second gaze point data match may be: calculating the coordinate difference between the first gaze point data and the second gaze point data; when the coordinate difference is within a preset range, determining that the first gaze point data and the second gaze point data match; when the coordinate difference is not within the preset range, determining that the first gaze point data and the second gaze point data do not match.

[0104] Optionally, the method for determining the target gaze point data based on the priority of the first gaze point data and the second gaze point data may be: determining the priority of the first gaze point data and the second gaze point data; and using the gaze point data with higher priority as the target gaze point data.

[0105] The image rendering apparatus provided in the embodiments of the present invention can execute the image rendering method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0106] Example 4

[0107] Figure 4 A schematic diagram of the structure of a virtual reality device 40 that can be used to implement embodiments of the present invention is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0108] like Figure 4 As shown, the virtual reality device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 and a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer programs stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0109] Multiple components in the virtual reality device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, optical disk, etc.; and a communication unit 49, such as a network card, modem, wireless transceiver, etc. The communication unit 49 allows the virtual reality device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0110] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as image rendering methods.

[0111] In some embodiments, the image rendering method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the virtual reality device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the image rendering method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the image rendering method by any other suitable means (e.g., by means of firmware).

[0112] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0115] To provide interaction with the user, the systems and techniques described herein can be implemented on a virtual reality device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the virtual reality device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).

[0116] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0117] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0118] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image rendering method, characterized in that, include: Once the user's gaze data is detected to be lost, the historical gaze data set and current scene information are retrieved. The first gaze point data is determined based on the historical gaze point dataset and the current scene information, and the second gaze point data is determined based on the current scene information; Determine the target gaze point data based on the first gaze point data and the second gaze point data; Image rendering is performed based on the target gaze point data; The step of determining the first gaze point data based on the historical gaze point dataset and the current scene information includes: The historical gaze point dataset is filtered based on time to determine candidate historical datasets; Based on the current scene information, the candidate historical dataset is filtered to determine the target historical data corresponding to the current scene information; The target historical data is determined as the first gaze point data; The step of determining the second gaze point data based on the current scene information includes: If the current scene information has a corresponding stimulus point, determine the stimulus point data of the stimulus point based on the current scene information; The stimulation point data of the stimulation point is used as the second fixation point data; If the current scene information does not have a corresponding stimulus point, determine the user's gaze point of interest based on the current scene information; The gaze point data of the gaze point of interest is used as the second gaze point data.

2. The method according to claim 1, characterized in that, The step of determining the target gaze point data based on the first gaze point data and the second gaze point data includes: Determine whether the first gaze point data and the second gaze point data match. If they do, use either the first gaze point data or the second gaze point data as the target gaze point data. Otherwise, the target gaze point data is determined based on the priority of the first gaze point data and the second gaze point data.

3. The method according to claim 2, characterized in that, The step of determining whether the first gaze point data and the second gaze point data match includes: Calculate the coordinate difference between the first fixation point data and the second fixation point data; When the coordinate difference is within a preset difference range, it is determined that the first gaze point data and the second gaze point data match. When the coordinate difference is not within the preset difference range, it is determined that the first gaze point data and the second gaze point data do not match.

4. The method according to claim 2, characterized in that, The step of determining the target gaze point data based on the priority of the first gaze point data and the second gaze point data includes: Determine the priority of the first gaze point data and the second gaze point data; Use high-priority fixation data as target fixation data.

5. An image rendering apparatus, characterized in that, include: The information acquisition module is used to acquire historical gaze point datasets and current scene information when the user's gaze point data is detected to be lost. The gaze point data acquisition module is used to determine first gaze point data based on the historical gaze point dataset and current scene information, and to determine second gaze point data based on the current scene information; The target data determination module is used to determine target gaze point data based on the first gaze point data and the second gaze point data. The image rendering module is used to render images based on the target gaze point data; The gaze point data acquisition module includes: The alternative data filtering unit is used to filter the historical gaze point dataset according to time to determine alternative historical datasets; The target data filtering unit is used to filter the candidate historical dataset based on the current scene information to determine the target historical data corresponding to the current scene information; The first data determining unit is used to determine the target historical data as the first gaze point data; The gaze point data acquisition module further includes: A stimulus data determination unit is used to determine the stimulus point data of the stimulus point based on the current scene information if the current scene information has a corresponding stimulus point. The second data determination unit is used to use the stimulation point data of the stimulation point as the second fixation point data; The gaze point data acquisition module further includes: The point of interest determination unit is used to determine the user's gaze point of interest based on the current scene information if the current scene information does not have a corresponding stimulus point; The third data determination unit is used to use the gaze point data of the gaze point of interest as the second gaze point data.

6. A virtual reality device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image rendering method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the image rendering method according to any one of claims 1-4.

Images