A method for rendering video images in a VR scene

Abstract

This application discloses a method and system for rendering video images in a VR scene. Based on the user's viewpoint trajectory information, this application dynamically renders the video images at each time point within the current time period. The resulting VR scene, which is then presented to the user, takes into account the user's line of sight and improves the rendering quality of the video images.

Description

Technical Field

[0001] This application relates to virtual reality (VR) technology, and in particular to a method and system for rendering video images in a VR scene. Background Technology

[0002] With the development of VR technology, VR-enabled devices such as VR glasses or VR headsets are gradually becoming commercially available. VR devices acquire video images, process them for VR presentation, and then present these VR video images to the user. This VR-applicable processing involves rendering the video image. However, due to the limited computing resources of VR devices, the rendering rate is often slower than the rendering rate, resulting in latency issues when presenting VR video images.

[0003] When VR devices render video images, the quality of the rendering directly affects the quality of the subsequent VR video images. Therefore, under the premise of saving the computing resources of VR devices, it has become an urgent technical problem to solve which rendering method with high rendering quality is adopted to render video images so that the rendered VR video images are presented without delay, do not cause dizziness to users when using VR devices, and improve the user experience. Summary of the Invention

[0004] This application provides a method for rendering video images in a VR scene. This method can take into account the user's viewpoint trajectory information when rendering video images in a VR scene, thereby improving the rendering quality of the video images.

[0005] This application also provides a system for rendering video images in a VR scene. This system can take into account the user's viewpoint trajectory information when rendering video images in a VR scene, thereby improving the rendering quality of the video images.

[0006] In one embodiment of this application, a method for rendering video images in a VR scene is provided, the method comprising:

[0007] A video image at the current time point is provided. The video image at the current time point is divided into multiple sub-regions. The image feature information of the sub-regions and the obtained user viewpoint feature information are input into the trained attention model for processing to obtain the attention coefficient of the sub-region. The attention coefficient of the sub-region reflects the probability value that the user viewpoint at the next time point may fall into the sub-region. The sub-region is rendered based on the attention coefficient of the sub-region to obtain the rendered video image at the current time point.

[0008] The attention coefficient and image feature information of the sub-region are input into the trained user gaze trajectory prediction model for processing to obtain the user gaze trajectory information for the current time period.

[0009] For video images at subsequent time points within the current time period, the video images at subsequent time points are divided into multiple sub-regions. Based on the user's gaze trajectory information within the current time period, the attention coefficient of each sub-region in the video image at each subsequent time point within the current time period is calculated. The corresponding sub-region is rendered based on the attention coefficient of the sub-region to obtain the rendered video image at each subsequent time point.

[0010] In the above embodiments, the method further includes:

[0011] Release the rendered video images at each time point in the time period in chronological order;

[0012] The system collects user viewpoint information at corresponding time points. When the user viewpoint information falls into a sub-region of the rendered video image, the corresponding rendered sub-region is used to form a VR scene for presentation.

[0013] In the above embodiments, the acquired user viewpoint feature information includes visual behavior factor information and contextual factor information, wherein,

[0014] The visual behavior factor information includes: the texture information (Textures) of the sub-region, the mesh information (Meshes) of the sub-region, and the position information of the sub-region;

[0015] The contextual factors include: user intent expression data, text data, voice dialogue data, system guidance data, and task orientation data.

[0016] In the above embodiments, the method used to divide the video image at the current time point into multiple sub-regions and the method used to divide the video image at subsequent time points into multiple sub-regions are the same, including:

[0017] The video image is mapped to a two-dimensional video image, which is a VR scene within the user's field of vision range defined by the sum of the user's field of view (FOV) and a set angle α.

[0018] The image feature information of the two-dimensional video image is input into the trained segmentation model to obtain multiple sub-regions and corresponding user viewpoint feature information.

[0019] In the above embodiments, before obtaining the attention coefficient of the sub-region, the method further includes:

[0020] After processing the sub-region based on the set concave principle, the attention coefficient of the sub-region is obtained.

[0021] In the above embodiments, the training process of the user gaze trajectory prediction model includes:

[0022] The attention coefficient and image feature information of the sub-region in the video image at the current time point are input into the user trajectory prediction model established based on user visual habit information for training, and the probability value of the user's gaze trajectory of the sub-region is output.

[0023] The true bounding box (GT) of the user gaze trajectory prediction model is determined by the user's gaze trajectory from the sub-region to the adjacent sub-region.

[0024] During training, the user trajectory prediction model is adjusted based on the recorded user visual habit information until training is complete.

[0025] In the above embodiments, obtaining the user's gaze trajectory information for the current time period further includes:

[0026] Based on the user's actual gaze trajectory information for the current time period, it is determined whether the user gaze trajectory information obtained directly through the user gaze trajectory prediction model for the current time period is accurate.

[0027] If accurate, the user gaze trajectory prediction model will be used to process the data, and the resulting user gaze trajectory information for the current time period will be used as the obtained user gaze trajectory information for the current time period.

[0028] If inaccurate, the actual user gaze trajectory information for the current time period is used as the obtained user gaze trajectory information for the current time period, and the user gaze trajectory prediction model is optimized and trained based on the actual user gaze trajectory information for the current time period.

[0029] In the above embodiments, the step of calculating the attention coefficient of a sub-region in the video image at each subsequent time point within the current time period based on the user's gaze trajectory information during the current time period includes:

[0030] For each sub-region in the video image at each subsequent time point within the current time period, based on the user's gaze trajectory information for the current time period, it is determined whether the user's gaze falls into the sub-region. If so, the attention coefficient of the sub-region is enhanced by a set amount based on the attention coefficient of the corresponding previous time point; if not, the attention coefficient of the sub-region is weakened by a set amount based on the attention coefficient of the corresponding previous time point.

[0031] In the above embodiments, rendering the sub-region based on the attention coefficient of the sub-region includes:

[0032] Set an attention coefficient threshold and determine whether the attention coefficient of the sub-region exceeds the set attention coefficient threshold. If it does, render the sub-region using the set high-level rendering method; otherwise, render it using the set low-level rendering method.

[0033] In another embodiment of this application, a system for rendering video images of a VR scene is provided. The system includes: a video image processing unit for the current time point, a user gaze trajectory information prediction unit, and a video image processing unit for the current time period, wherein...

[0034] The video image processing unit at the current time point is used to provide a video image at the current time point, divide the video image at the current time point into multiple sub-regions, input the image feature information of the sub-regions and the acquired user viewpoint feature information into the trained attention model for processing, obtain the attention coefficient of the sub-region, the attention coefficient of the sub-region reflects the probability value of the user viewpoint falling into the sub-region at the next time point, and render the sub-region based on the attention coefficient of the sub-region to obtain the rendered video image at the current time point;

[0035] The user gaze trajectory information prediction unit is used to input the attention coefficient of the sub-region and the image feature information of the sub-region into the trained user gaze trajectory prediction model for processing, so as to obtain the user gaze trajectory information for the current time period.

[0036] The video image processing unit for the current time period is configured to divide the video images of subsequent time points within the current time period into multiple sub-regions, calculate the attention coefficient of each sub-region in the video image of each subsequent time point within the current time period based on the user's gaze trajectory information of the current time period, and render the corresponding sub-region based on the attention coefficient of the sub-region to obtain the rendered video image of each subsequent time point.

[0037] In another embodiment of this application, an electronic device is provided, comprising:

[0038] processor;

[0039] A memory storing a program configured to implement, when executed by the processor, a method for rendering video images in a VR scene as described above.

[0040] As seen above, the embodiments of this application dynamically render video images at each time point within the current time period based on user viewpoint trajectory information. The resulting VR scene presented to the user takes into account the user's line of sight, thus improving the rendering quality of the video images. Attached Figure Description

[0041] Figure 1 A flowchart illustrating a method for rendering video images in a VR scene according to an embodiment of this application;

[0042] Figure 2 A schematic diagram illustrating the relationship between factor information affecting user viewpoint feature information of a sub-region and the attention coefficient of the sub-region, as provided in an embodiment of this application.

[0043] Figure 3 This is a schematic diagram illustrating the mapping of a video image to a two-dimensional video image, provided as an embodiment of this application.

[0044] Figure 4 A schematic diagram illustrating the implementation of the partitioning model provided in the embodiments of this application;

[0045] Figure 5 A flowchart illustrating the process of calculating the attention coefficient of the sub-region at the current time point using an attention model, provided in an embodiment of this application;

[0046] Figure 6 A schematic diagram of the attention coefficients of a sub-region at the current time point provided in this application embodiment;

[0047] Figure 7 A schematic diagram illustrating the relationship between the user's gaze trajectory and the attention coefficient of a sub-region of a video image, provided in an embodiment of this application;

[0048] Figure 8 A schematic diagram illustrating the process of training a user gaze trajectory prediction model provided in an embodiment of this application;

[0049] Figure 9 A flowchart illustrating a specific example of a method for rendering video images provided in this application embodiment;

[0050] Figure 10a A flowchart illustrating the target prediction process in a specific example provided in this application embodiment;

[0051] Figure 10b A schematic diagram illustrating the interaction between the user terminal and the system backend of the VR device in a specific example provided in this application embodiment;

[0052] Figure 11 A flowchart illustrating the scheme for predicting user gaze trajectory information in a specific example two provided in this application embodiment;

[0053] Figure 12aThe third specific example provided in this application embodiment combines user visual habit information;

[0054] Figure 12b A schematic diagram illustrating the process of rendering different sub-regions in a video image based on user gaze trajectory information, as shown in Specific Example 3 of this application;

[0055] Figure 13 The implementation process diagram of Specific Example 4 provided in the embodiments of this application;

[0056] Figure 14 This is a schematic diagram of the system structure for rendering video images of a VR scene according to an embodiment of this application;

[0057] Figure 15 This is a schematic diagram of an electronic device provided for another embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0060] The technical solution of this application will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0061] Currently, to overcome the latency issue in rendering VR video images on VR devices, a buffer can be set up in the VR device to cache the rendered VR video images, which are then retrieved from the buffer during rendering. While this method can improve the rendering effect of VR video images, it doesn't fundamentally solve the problem of excessive computing resources consumed by VR devices during video rendering. Excessive computing resources used for rendering video images still lead to a decrease in rendering quality, affecting the overall presentation quality of the rendered VR video images.

[0062] In this context, to overcome the problem of excessive computing resources consumed by VR devices in rendering video images, the rendering method can be optimized to save computational load on VR devices. For example, one approach is to use a region-based rendering method: rendering the central concave region and the peripheral region of the video image separately, giving the central concave region a higher resolution than the peripheral region. Another approach is to predict the motion region in the video image and then render that region: breaking down the motion of the video image into camera translation, object motion, and camera rotation; using the parallax mapping principle, processing the camera translation motion to obtain the first extrapolated image; using a camera translation correction algorithm and a fixed-point iteration method, using the first extrapolated image to extrapolate the object motion image; and finally, after calculation using a smoothing algorithm, obtaining the rendered VR video image.

[0063] The methods described above all reduce the computational load on VR devices by lowering the rendering quality when rendering video images. While the second method considers the dynamic impact of camera movement on object motion during rendering, thus improving video image rendering quality compared to the static rendering of the first method, both methods only offer limited improvements in video image rendering quality. The degree of improvement is limited by the accuracy of the method used and cannot substantially enhance the overall rendering quality of the video image.

[0064] It can be seen that, given limited computing resources, VR devices improve the rendering of VR video images to ensure presentation quality and avoid issues such as rendering latency. However, the degree of improvement in image rendering quality is limited by the accuracy of the improved rendering method and cannot substantially enhance the overall rendering quality. Furthermore, user experience issues such as dizziness when viewing VR video images presented in VR scenes are not only due to rendering latency but also to the low rendering quality of the improved rendering method. Whether a user experiences dizziness from a presented VR video image largely depends on their gaze. The aforementioned rendering improvements do not base their rendering on user gaze interaction information, resulting in unnatural interactions such as selecting content while looking at the VR video image, leading to dizziness and a poor user experience.

[0065] Therefore, in order to overcome the above problems, this application embodiment dynamically renders the video images at each time point within the current time period based on the user's viewpoint trajectory information. The resulting VR scene presented to the user takes into account the user's line of sight, thus improving the rendering quality of the video images.

[0066] Figure 1 The flowchart of the method for rendering video images in a VR scene provided in this application embodiment includes the following specific steps:

[0067] Step 101: Provide a video image at the current time point, divide the video image at the current time point into multiple sub-regions, input the image feature information of the sub-regions and the obtained user viewpoint feature information into the trained attention model for processing, and obtain the attention coefficient of the sub-region. The attention coefficient of the sub-region reflects the probability value that the user viewpoint at the next time point may fall into the sub-region. Render the sub-region based on the attention coefficient of the sub-region to obtain the rendered video image at the current time point.

[0068] Step 102: Input the attention coefficient and image feature information of the sub-region into the trained user gaze trajectory prediction model for processing to obtain the user gaze trajectory information for the current time period;

[0069] Step 103: For video images at subsequent time points within the current time period, divide the video images at subsequent time points into multiple sub-regions. Based on the user gaze trajectory information of the current time period, calculate the attention coefficient of each sub-region in the video image at each subsequent time point within the current time period. Render the corresponding sub-region based on the attention coefficient of the sub-region to obtain the rendered video image at each subsequent time point.

[0070] Following the method, the following is also included:

[0071] Release the rendered video images at each time point in the time period in chronological order;

[0072] The system collects user viewpoint information at corresponding time points. When the user viewpoint information falls into a sub-region of the rendered video image, the corresponding rendered sub-region is used to form a VR scene for presentation.

[0073] In this method, rendering and presentation are both achieved by VR devices such as VR glasses or VR headsets.

[0074] In this embodiment of the application, the attention coefficient of the sub-region reflects the probability value that the user's viewpoint will fall into the sub-region at the next time point. When the user's viewpoint still falls into the sub-region at the next time point, the attention coefficient of the sub-region is 1.

[0075] In this embodiment of the application, the video image at the current time point within the current time period, as well as the video images at subsequent time points, present a VR scene based on the user's gaze. For each video image at the current time point and the video images at subsequent time points, a preset segmentation model is used to divide it into multiple sub-regions, and the multiple sub-regions are pre-rendered based on the calculated attention coefficients.

[0076] As can be seen, the embodiments of this application divide each video image within the current time period into multiple sub-regions, and combine the image feature information of each sub-region with the user's gaze trajectory information within the current time period to determine the attention coefficient of each sub-region and pre-render each video image accordingly. This can improve the quality of VR device rendering video images, thereby improving the presentation quality of subsequent VR video images. This prevents users from experiencing dizziness when watching VR video images, thus improving the user experience.

[0077] In this embodiment of the application, based on the analysis of the user's intent to select a sub-region in the video image, it is known that the factors influencing the acquisition of user viewpoint feature information include: visual behavior factors and contextual factors. For example... Figure 2 As shown, Figure 2 This diagram illustrates the relationship between factor information influencing user viewpoint feature information of a sub-region and the attention coefficient of that sub-region, as provided in an embodiment of this application. Figure 2 In this context, visual behavioral information includes textures, meshes, and positional information. Textures specifically refer to the hue, saturation, and brightness of the sub-region; meshes specifically refer to the size and shape of the sub-region; and positional information includes the distance and changes in the sub-region. When processing this information using an attention model, both similarity-based and reflexive attention mechanisms are employed. With the similarity-based attention mechanism, the similarity between the visual information of the sub-region and the user's current viewpoint information is compared: if the visual information of the sub-region matches the user's current viewpoint information, the attention coefficient of that sub-region is enhanced; otherwise, the attention coefficient remains unchanged. With the reflexive attention mechanism, the attention coefficient of the sub-region is enhanced when it is close to the user's viewpoint information or / and when the subject in the sub-region is moving. Contextual information includes user intent data, text data, voice dialogue data, system guidance data, and task-oriented data. From the contextual information, user intent expression data, text data, and voice dialogue data are used to extract descriptions related to the subject in the current VR scene. When the extracted descriptions are similar to the image features of the sub-region, the attention coefficient of that sub-region is enhanced; when they are dissimilar, the attention coefficient is weakened. From the system guidance data and Task directionality data in the aforementioned information, descriptions of the subject emphasized by the system in the current VR scene are extracted. When these descriptions are similar to the image features of the sub-region, the attention of that sub-region is enhanced.

[0078] In this embodiment of the application, the method used to divide the video image at the current time point into multiple sub-regions and the method used to divide the video image at subsequent time points into multiple sub-regions are the same, specifically:

[0079] The video image is mapped to a two-dimensional video image, which is a VR scene within the user's field of vision range defined by the sum of the user's field of view (FOV) and a set angle (α);

[0080] The image feature information of the two-dimensional video image is input into the trained segmentation model to obtain multiple sub-regions and their corresponding user viewpoint feature information. Here, the corresponding user viewpoint feature information can be represented using mask features.

[0081] In this embodiment of the application, a schematic diagram of mapping a video image to a two-dimensional video image is shown below. Figure 3 As shown. The sum of the user's field of view (FOV) and α defines the VR scene within the user's field of view, forming a two-dimensional video image, which has undergone... Figure 4 The segmentation model shown divides the data into sub-regions, resulting in multiple sub-regions and their masks. The value of α is set such that it is slightly larger than the user's field of view (FOV) but smaller than the VR scene range that the VR device can display, for example, less than 3°. The segmentation model can employ image segmentation neural networks, such as segmentation models.

[0082] In the embodiments of this application, Figure 1 The attention model in [the context] can be implemented using a convolutional neural network (CNN), such as... Figure 5 As shown, Figure 5 This application provides a flowchart illustrating the process of calculating the attention coefficients of sub-regions at the current time point using an attention model. The process involves extracting image feature information (represented as RGB in the diagram) and extracted user viewpoint feature information (represented as a mask in the diagram) from the sub-region, encoding them, and inputting them into the attention model for attention coefficient calculation. After concatenation through a concatenation layer in the attention model, the model performs CNNblock processing to obtain the attention coefficients of each sub-region in the video image at the current time point, which are then output through the output layer of the attention model. The attention coefficients of each sub-region in the video image reflect the confidence level of the user's gaze at the next time point, thus allowing prediction of which sub-region the user's gaze trajectory might move to at the next time point. The output of the attention coefficients of each sub-region in the video image at the current time point is presented as an attention map, where the attention map ∈ R... n*n , where R represents each subregion, and each subregion consists of n*n elements, where n can take the value 3.

[0083] In this embodiment, before applying the attention coefficients for each sub-region, the sub-regions are further processed based on the concave principle. For example, the attention coefficients from the center sub-region to the edge sub-region of the video image are attenuated by a set amplitude based on the concave principle.

[0084] Figure 6This is a schematic diagram of the attention coefficients of a sub-region at the current time point provided in an embodiment of this application. As shown in the figure, the attention coefficients of each sub-region of the video image are calculated by considering not only the user viewpoint feature information extracted from the sub-region, but also the fovea principle, thereby determining the subsequent user gaze trajectory and providing a basis for calculating the attention coefficients of the sub-regions at subsequent time points.

[0085] In specific implementation, such as Figure 7 As shown, Figure 7 This diagram illustrates the relationship between the user's gaze trajectory and the attention coefficients of sub-regions in a video image, as provided in this embodiment. The video image, mapped to a two-dimensional image, is divided into multiple sub-regions. Each sub-region contains 3*3 elements, and each sub-region represents a patch. Each element contains multiple pixels, and each element is represented as a cell. The attention coefficient of each cell is obtained by a weighted average of the attention coefficients of its constituent pixels. The user's viewpoint motion trajectory is recorded using a VR device. The user's gaze trajectory from the current sub-region to adjacent sub-regions is statistically analyzed and used as the ground truth (GT) of the user gaze trajectory prediction model for training.

[0086] In this embodiment of the application, the training process of the user gaze trajectory prediction model is as follows:

[0087] The attention coefficient and image feature information of the sub-region in the video image at the current time point are input into the user trajectory prediction model established based on user visual habit information for training, and the probability value of the user's gaze trajectory of the sub-region is output.

[0088] The ground truth (GT) of the user gaze trajectory prediction model is determined by the user's gaze trajectory from the current sub-region to the adjacent sub-region.

[0089] During training, user visual habit information is recorded, and the user trajectory prediction model is adjusted until training is complete.

[0090] The above process is as follows Figure 8 As shown, Figure 8 This is a schematic diagram illustrating the process of training a user gaze trajectory prediction model according to an embodiment of this application. The user gaze trajectory prediction model can employ a CNN model. The image feature information of the sub-region is represented as RGB of View Patch, and the attention coefficient of the sub-region is represented as Attention Patch. The ground truth (GT) of the user trajectory prediction model is determined based on user visual habit information (Record the real trajectory of the next cell).

[0091] In this embodiment of the application, obtaining the user's gaze trajectory information for the current time period further includes:

[0092] Based on the user's actual gaze trajectory information for the current time period, it is determined whether the user gaze trajectory information obtained directly through the user gaze trajectory prediction model for the current time period is accurate.

[0093] If accurate, the user gaze trajectory prediction model will be used to process the data, and the resulting user gaze trajectory information for the current time period will be used as the obtained user gaze trajectory information for the current time period.

[0094] If inaccurate, the actual user gaze trajectory information for the current time period is used as the obtained user gaze trajectory information for the current time period, and the user gaze trajectory prediction model is optimized and trained based on the actual user gaze trajectory information for the current time period.

[0095] In other words, when the user gaze trajectory prediction model's prediction of the current time period is inaccurate, to improve user experience, the model directly uses the user's actual gaze trajectory information for the current time period for subsequent execution, instead of using the predicted gaze trajectory information. This also indicates that the user gaze trajectory prediction model's accuracy is insufficient and needs to be optimized and trained based on the user's actual gaze trajectory information for the current time period to improve its prediction accuracy in subsequent executions.

[0096] Of course, when the user gaze trajectory information for the current time period directly predicted by the user gaze trajectory prediction model is inaccurate, the user gaze trajectory model can be optimized and trained based on the actual user gaze trajectory information for the current time period. Then, the optimized and trained user gaze trajectory model can be used to make predictions again, and the direct prediction result obtained can be used as the user gaze trajectory information for the current time period.

[0097] In this embodiment of the application, the step of calculating the attention coefficient of a sub-region in the video image at each subsequent time point within the current time period based on the user's gaze trajectory information during the current time period includes:

[0098] For each sub-region in the video image at each subsequent time point within the current time period, based on the user's gaze trajectory information for the current time period, it is determined whether the user's gaze falls into the sub-region. If so, the attention coefficient of the sub-region is increased by a set amount based on the attention coefficient of the corresponding previous time point; otherwise, the attention coefficient of the sub-region is decreased by a set amount based on the attention coefficient of the corresponding previous time point. Here, the set amount can be set according to requirements and is not limited.

[0099] In this embodiment of the application, rendering the corresponding sub-region based on the attention coefficient of the sub-region includes:

[0100] An attention coefficient threshold is set, and it is determined whether the attention coefficient of the sub-region exceeds the set threshold. If it does, the sub-region is rendered using the set high-level rendering method; otherwise, it is rendered using the set low-level rendering method. Here, the attention coefficient threshold can be set to 80%, the high-level rendering method is rendering for a pre-set time period, and / or rendering using the set high-level resolution; the low-level rendering method is normal rendering, or rendering using the set low-level resolution.

[0101] As can be seen, in order to implement the embodiments of this application, three neural network models were trained: a segmentation model, an attention model, and a user gaze trajectory prediction model. Through the organic cooperation of the three neural network models, the rendering of video images in the VR scene in the current time period was realized.

[0102] Figure 9 A flowchart illustrating a specific example of a method for rendering video images provided in this application embodiment, the specific steps of which include:

[0103] Step 801: Obtain the VR scene within the current time period;

[0104] In this step, the VR scene is composed of video images at the current time and video images at subsequent times, and these video images are acquired one by one;

[0105] Step 802: Extract user viewpoint feature information from the VR scene;

[0106] This step includes information such as textures, meshes, and location information;

[0107] Step 803: Divide the video image at the current time point into multiple sub-regions;

[0108] Step 804: For each sub-region, the image feature information and user viewpoint feature information of the sub-region are input into the attention model for processing to obtain the attention coefficient of the sub-region, which forms the attention coefficient map of each sub-region of the video image at the current time point.

[0109] Step 805: At the current time point, capture the user's eye movement and viewpoint data;

[0110] Steps 806-807: Divide the VR scene into sub-regions and determine the user viewpoint feature information;

[0111] Step 808: Obtain user visual habit information and dynamically adjust the user trajectory prediction model;

[0112] Here, the user trajectory prediction model can be set and dynamically adjusted based on user visual habit information. Specifically, the parameters in the user trajectory prediction model can be adjusted.

[0113] Step 809: Determine the user viewpoint feature information by taking the attention coefficient map and molecular region of each sub-region of the video image at the current time point, input it into the user gaze trajectory prediction model for processing, and obtain the user gaze trajectory information for the current time period.

[0114] Step 810: For each time point in the VR scene, based on the user's gaze trajectory information in the current time period, predict the user's viewpoint information, and then determine whether the predicted user's viewpoint information is correct. If it is correct, proceed to step 811; if it is incorrect, proceed to the next step based on the user's actual viewpoint information in the current time period, save the user's actual gaze trajectory information, return to step 808, and continue to optimize and train the user's gaze trajectory prediction model based on the user's actual gaze trajectory information.

[0115] In this step, when the predicted user viewpoint information is incorrect, the user's actual viewpoint information for the current time period can be directly obtained for subsequent execution. That is, for each sub-region in the video image at subsequent time points of the current time period, the user's actual gaze trajectory information for the current time period is rendered. After that, the user's actual gaze trajectory information is recorded again for subsequent optimization training of the user gaze trajectory prediction model to enhance the accuracy of the user trajectory prediction model.

[0116] In this step, if the predicted user viewpoint information is incorrect, the user gaze trajectory prediction model can be directly optimized and trained based on the user's actual gaze trajectory information in the current time period, and then the optimized and trained user gaze trajectory prediction model can be used to make predictions again.

[0117] Step 811: Execute the prediction result based on the user's gaze trajectory information in the current time period;

[0118] In this step, executing the prediction results involves rendering each sub-region of the video image at subsequent time points within the current time period, based on the user's gaze trajectory information within the current time period.

[0119] The following are some specific examples to illustrate the embodiments of this application in detail.

[0120] In a specific example, this embodiment uses an attention model to determine the user's intent regarding the subject they are currently looking at and selects whether to render it. Figure 10a The flowchart of the target prediction process in a specific example provided in this application includes:

[0121] Step 1: Only the sub-region of the user's current visual focal point is pre-rendered;

[0122] Step 2: Obtain user viewpoint feature information;

[0123] The diagram illustrates the data acquisition process.

[0124] Step 3: Perform data analysis using an attention model;

[0125] The diagram shows that during data analysis, visual behavioral factors and contextual factors were analyzed.

[0126] Step 4: Obtain the attention coefficient for each sub-region based on the classification results;

[0127] Figure 10b The interaction diagram between the user terminal and the system backend of the VR device in Specific Example 1 provided in this application embodiment includes:

[0128] Step 5: Based on the attention coefficient of each sub-region, pre-render each sub-region and cache the rendering results;

[0129] Step 6: Present the rendering results to the user and perform data analysis accordingly;

[0130] In this step, based on the user's actual behavior, when the user's visual focus is predicted to fall on a certain sub-region of the target, the rendered sub-region can be presented to the user.

[0131] The second specific example is a solution for predicting user gaze trajectory information. Figure 11 The flowchart of the scheme for predicting user gaze trajectory information in Specific Example 2 provided in the embodiments of this application includes:

[0132] Step 1: Render the sub-region of the user's current visual focus;

[0133] Step 2: Process the user gaze trajectory prediction model to obtain the analysis results;

[0134] In this step, the model analysis results include: based on the data in the scene, determining that the car flying in the sky in the scene is the final point of view of the user in the current time period;

[0135] Step 3-1: Evaluate the attention coefficient of each sub-region based on the analysis results;

[0136] In this step, the attention coefficient evaluation results shown in the right figure are provided based on the analysis results;

[0137] Step 3-2: Simultaneously with step 3-1, train the user gaze trajectory prediction model;

[0138] During training, user visual habit information is introduced, so that the user gaze trajectory prediction model obtains a different user gaze trajectory than the one obtained in the initial state during training.

[0139] The third specific example is a method for predicting user eye trajectory information by combining user visual habit information. Figure 12a A schematic diagram illustrating the process of predicting user gaze trajectory information by combining user visual habit information in a specific example three provided in this application embodiment includes:

[0140] Step 1: Render the sub-region of the user's current visual focus;

[0141] Step 2: Train the user's gaze trajectory prediction model;

[0142] During the training process, information on different user visual habits is incorporated;

[0143] Step 3: Evaluate the eye trajectory of different users obtained by the model in order to adjust the user eye trajectory model in the future.

[0144] Figure 12b A schematic diagram illustrating the process of rendering different sub-regions in a video image based on user gaze trajectory information, as shown in Specific Example 3 of this application, includes:

[0145] Step 4: Based on the user gaze trajectories of different users obtained from the analysis, determine the attention coefficient of each sub-region in the video image, and pre-render the sub-regions with attention coefficients exceeding 80% of the attention coefficient threshold to generate multi-angle videos and store them.

[0146] Step 5: Based on the user's actual behavior, when the user's gaze falls on the corresponding sub-area, release the pre-rendered result of that sub-area stored in order to complete the presentation in the VR scene.

[0147] In a specific example, the scheme for predicting the user's line of sight in this application can also be applied to autonomous vehicle driving. By predicting the user's line of sight in advance, potential safety hazards that may be overlooked can be discovered and warnings can be issued. Figure 13 The implementation process diagram of Specific Example Four provided in the embodiments of this application includes:

[0148] Step 1: Equip the vehicle with an infrared camera that can detect the driver's eye movement data;

[0149] Step 2: During driving, dynamically analyze the driver's current gaze point, and use an attention model and a user gaze trajectory prediction model to predict the user's gaze trajectory information.

[0150] Step 3: Compare the user's line-of-sight trajectory information with the driving path information to extract potential safety hazards that may exist outside the range of the user's line-of-sight trajectory information.

[0151] In this step, attention coefficients of each sub-region in the video images at subsequent time points can be obtained based on the user's eye trajectory information. Based on this, the video images of the subsequent driving path can be compared to determine the safe driving hazards that the user has not noticed.

[0152] Step 4: Use voice or other means to remind the user of potential safety hazards during subsequent driving.

[0153] like Figure 14 As shown, Figure 14 This is a schematic diagram of a system structure for rendering video images of a VR scene according to an embodiment of this application. The system includes: a video image processing unit for the current time point, a user gaze trajectory information prediction unit, and a video image processing unit for the current time period, wherein...

[0154] The video image processing unit at the current time point is used to provide a video image at the current time point, divide the video image at the current time point into multiple sub-regions, input the image feature information of the sub-regions and the acquired user viewpoint feature information into the trained attention model for processing, obtain the attention coefficient of the sub-region, the attention coefficient of the sub-region reflects the probability value of the user viewpoint falling into the sub-region at the next time point, and render the sub-region based on the attention coefficient of the sub-region to obtain the rendered video image at the current time point;

[0155] The user gaze trajectory information prediction unit is used to input the attention coefficient of the sub-region and the image feature information of the sub-region into the trained user gaze trajectory prediction model for processing, so as to obtain the user gaze trajectory information for the current time period.

[0156] The video image processing unit for the current time period is configured to divide the video images of subsequent time points within the current time period into multiple sub-regions, calculate the attention coefficient of each sub-region in the video image of each subsequent time point within the current time period based on the user's gaze trajectory information of the current time period, and render the corresponding sub-region based on the attention coefficient of the sub-region to obtain the rendered video image of each subsequent time point.

[0157] The system also includes a presentation unit, which releases rendered video images at various time points in the time period in chronological order; collects user viewpoint information at corresponding time points; and when the user viewpoint information falls into a sub-region of the rendered video image, forms the corresponding rendered sub-region into a VR scene for presentation.

[0158] As can be seen from the embodiments of this application, by predicting the user's gaze trajectory information, the content the user wants to view can be rendered in advance. In certain specific application scenarios, such as VR device display, this provides preparation time for hardware implementation and also ensures the user experience. By predicting the user's gaze trajectory information, natural interaction in certain visual scenarios can be achieved, making the human-computer interaction method more acceptable to users.

[0159] In another embodiment of this application, a non-transitory computer-readable storage medium is provided, which stores instructions that, when executed by a processor, cause the processor to execute a method for rendering video images in a VR scene as described in the foregoing embodiments.

[0160] Figure 15 This is a schematic diagram of an electronic device provided for another embodiment of this application. (See diagram below.) Figure 15 As shown, another embodiment of this application also provides an electronic device, which may include a processor 1501, wherein the processor 1501 is used to perform the steps of the above-described method for rendering video images in a VR scene. From Figure 15 It can also be seen that the electronic device provided in the above embodiments further includes a non-transitory computer-readable storage medium 1502, on which a computer program is stored, and the computer program is executed by the processor 1501 to perform the steps of the above-described method for rendering video images in a VR scene.

[0161] Specifically, the non-transitory computer-readable storage medium 1502 can be a general-purpose storage medium, such as a mobile disk, hard disk, FLASH, read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or portable compact disk read-only memory (CD-ROM), etc. When the computer program on the non-transitory computer-readable storage medium 1502 is run by the processor 1501, it can cause the processor 1501 to execute the various steps of the above-described method for rendering video images in a VR scene.

[0162] In practical applications, the non-transitory computer-readable storage medium 1502 may be included in the device / apparatus / system described in the above embodiments, or it may exist independently without being assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, can perform the various steps of the method for rendering video images in a VR scene.

[0163] Another embodiment of this application provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method for rendering video images in a VR scene.

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments disclosed in this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings. For example, two blocks shown connectedly may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0165] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, and all such combinations and / or combinations fall within the scope of this application.

[0166] This document uses specific embodiments to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application, and are not intended to limit this application. For those skilled in the art, changes can be made to the specific implementation methods and application scope based on the ideas, spirit and principles of this application. Any modifications, equivalent substitutions, improvements, etc., made should be included within the scope of protection of this application.

Claims

1. A method for rendering video images in a VR scene, characterized in that, The method includes: A video image at the current time point is provided. The video image at the current time point is divided into multiple sub-regions. The image feature information of the sub-regions and the obtained user viewpoint feature information are input into the trained attention model for processing to obtain the attention coefficient of the sub-region. The attention coefficient of the sub-region reflects the probability value that the user viewpoint at the next time point may fall into the sub-region. The sub-region is rendered based on the attention coefficient of the sub-region to obtain the rendered video image at the current time point. The attention coefficient and image feature information of the sub-region are input into the trained user gaze trajectory prediction model for processing to obtain the user gaze trajectory information for the current time period. For video images at subsequent time points within the current time period, the video images at subsequent time points are divided into multiple sub-regions. Based on the user's gaze trajectory information within the current time period, the attention coefficient of each sub-region in the video image at each subsequent time point within the current time period is calculated. The corresponding sub-region is rendered based on the attention coefficient of the sub-region to obtain the rendered video image at each subsequent time point.

2. The method as described in claim 1, characterized in that, The method further includes: Release the rendered video images at each time point in the time period in chronological order; The system collects user viewpoint information at corresponding time points. When the user viewpoint information falls into a sub-region of the rendered video image, the corresponding rendered sub-region is used to form a VR scene for presentation.

3. The method as described in claim 1, characterized in that, The acquired user viewpoint feature information includes visual behavior factor information and contextual factor information, wherein, The visual behavior factor information includes: the texture information (Textures) of the sub-region, the mesh information (Meshes) of the sub-region, and the position information of the sub-region; The contextual factors include: user intent expression data, text data, voice dialogue data, system guidance data, and task orientation data.

4. The method as described in claim 1, characterized in that, The method used to divide the video image at the current time point into multiple sub-regions and the method used to divide the video image at subsequent time points into multiple sub-regions are the same, including: The video image is mapped to a two-dimensional video image, which is a VR scene within the user's field of vision range defined by the sum of the user's field of view (FOV) and a set angle α. The image feature information of the two-dimensional video image is input into the trained segmentation model to obtain multiple sub-regions and corresponding user viewpoint feature information.

5. The method as described in claim 1, characterized in that, Before obtaining the attention coefficients of the sub-region, the process also includes: After processing the sub-region based on the set concave principle, the attention coefficient of the sub-region is obtained.

6. The method as described in claim 1, characterized in that, The training process of the user gaze trajectory prediction model includes: The attention coefficient and image feature information of the sub-region in the video image at the current time point are input into the user trajectory prediction model established based on user visual habit information for training, and the probability value of the user's gaze trajectory of the sub-region is output. The true bounding box (GT) of the user gaze trajectory prediction model is determined by the user's gaze trajectory from the sub-region to the adjacent sub-region. During training, the user trajectory prediction model is adjusted based on the recorded user visual habit information until training is complete.

7. The method as described in claim 1, characterized in that, The method of obtaining the user's gaze trajectory information for the current time period also includes: Based on the user's actual gaze trajectory information for the current time period, it is determined whether the user gaze trajectory information obtained directly through the user gaze trajectory prediction model for the current time period is accurate. If accurate, the user gaze trajectory prediction model will be used to process the data, and the resulting user gaze trajectory information for the current time period will be used as the obtained user gaze trajectory information for the current time period. If inaccurate, the actual user gaze trajectory information for the current time period is used as the obtained user gaze trajectory information for the current time period, and the user gaze trajectory prediction model is optimized and trained based on the actual user gaze trajectory information for the current time period.

8. The method as described in claim 1, characterized in that, The calculation of attention coefficients for sub-regions in the video image at each subsequent time point within the current time period, based on the user gaze trajectory information of the current time period, includes: For each sub-region in the video image at each subsequent time point within the current time period, based on the user's gaze trajectory information for the current time period, it is determined whether the user's gaze falls into the sub-region. If so, the attention coefficient of the sub-region is enhanced by a set amount based on the attention coefficient of the corresponding previous time point; if not, the attention coefficient of the sub-region is weakened by a set amount based on the attention coefficient of the corresponding previous time point.

9. The method as described in claim 1, characterized in that, Rendering the sub-region based on the attention coefficient of the sub-region includes: Set an attention coefficient threshold and determine whether the attention coefficient of the sub-region exceeds the set attention coefficient threshold. If it does, render the sub-region using the set high-level rendering method; otherwise, render it using the set low-level rendering method.

10. A system for rendering video images of a VR scene, characterized in that, The system includes: a video image processing unit for the current time point, a user gaze trajectory information prediction unit, and a video image processing unit for the current time period, wherein... The video image processing unit at the current time point is used to provide a video image at the current time point, divide the video image at the current time point into multiple sub-regions, input the image feature information of the sub-regions and the acquired user viewpoint feature information into the trained attention model for processing, obtain the attention coefficient of the sub-region, the attention coefficient of the sub-region reflects the probability value of the user viewpoint falling into the sub-region at the next time point, and render the sub-region based on the attention coefficient of the sub-region to obtain the rendered video image at the current time point; The user gaze trajectory information prediction unit is used to input the attention coefficient of the sub-region and the image feature information of the sub-region into the trained user gaze trajectory prediction model for processing, so as to obtain the user gaze trajectory information for the current time period. The video image processing unit for the current time period is configured to divide the video images of subsequent time points within the current time period into multiple sub-regions, calculate the attention coefficient of each sub-region in the video image of each subsequent time point within the current time period based on the user's gaze trajectory information of the current time period, and render the corresponding sub-region based on the attention coefficient of the sub-region to obtain the rendered video image of each subsequent time point.

11. An electronic device, characterized in that, include: processor; A memory storing a program configured to, when executed by the processor, implement a method for rendering video images in a VR scene as described in any one of claims 1 to 9.

Images