Rendering method, device and equipment of static scene under dynamic visual angle and medium

Abstract

The invention relates to the technical field of computer graphics, and provides a rendering method and device for a static scene under a dynamic view angle, equipment and a medium, and the method comprises the steps: obtaining a future view angle probability cloud picture of a preset frame number according to the historical behavior data of a user and a hybrid neural network view angle prediction model; dividing a pre-rendering static scene of at least one target pre-rendering view angle based on a preset dynamic semantic weight matrix to obtain layered baking data; obtaining target optical flow data of a target adjacent view angle adjacent to the current view angle from a preset optical flow mapping table; during view angle switching, dynamically mixing a static scene rendered in real time at the current view angle and a static scene pre-rendered according to a pixel displacement relation based on target optical flow data to obtain scene mixed data; and rendering based on the scene mixed data to obtain a target static scene. The problem of real-time high-quality rendering of the static scene under the dynamic view angle can be solved.

Description

Technical Field

[0001] The present application relates to the technical field of computer graphics, and in particular to a method for rendering a static scene under a dynamic perspective, a device for rendering a static scene under a dynamic perspective, a corresponding electronic device, and a corresponding computer-readable storage medium. Background Art

[0002] Among pre-rendering technologies, light baking only supports static lighting, meaning it only supports pre-rendering from a fixed perspective. It cannot adapt to dynamic perspective changes. Because dynamic perspectives require recalculating indirect lighting, data must be reloaded when switching perspectives, causing lag. Consequently, lighting and shadow inconsistencies between static pre-rendering and dynamic real-time rendering elements can easily create visual disjunction, leading to an unnatural fusion of static and dynamic elements and making it difficult to render static scenes from a dynamic perspective. Summary of the Invention

[0003] The present application provides a method, apparatus, device and medium for rendering static scenes under a dynamic perspective, which can solve the problem of real-time high-quality rendering of static scenes under a dynamic perspective, and balance storage overhead, loading speed and visual quality.

[0004] In one aspect, the present application provides a method for rendering a static scene under a dynamic perspective, the method comprising:

[0005] Obtain user historical behavior data and pre-built hybrid neural network perspective prediction model;

[0006] Obtaining a view probability cloud map for a preset number of frames in the future based on the user historical behavior data and the hybrid neural network view prediction model; the view probability cloud map includes a hotspot area, and the hotspot area is used to indicate at least one target pre-rendered view whose allocation probability exceeds a preset probability threshold;

[0007] Dividing the pre-rendered static scene of the at least one target pre-rendered perspective based on a preset dynamic semantic weight matrix to obtain layered baking data; using the layered baking data to generate optical flow data between the respective target pre-rendered perspectives, and storing the optical flow data in a preset optical flow mapping table;

[0008] Obtaining target optical flow data of a target adjacent viewpoint adjacent to the current viewpoint from the preset optical flow mapping table; the target optical flow data is used to indicate a pixel displacement relationship between the current viewpoint and the target adjacent viewpoint;

[0009] When switching the perspective, dynamically mixing the static scene rendered in real time at the current perspective and the static scene pre-rendered according to the pixel displacement relationship based on the target optical flow data to obtain scene mixing data;

[0010] A target static scene is obtained by rendering based on the scene mixing data.

[0011] In another aspect, the present application provides a device for rendering a static scene under a dynamic perspective, the device comprising:

[0012] A viewpoint prediction model acquisition module, which is used for user historical behavior data and a pre-built hybrid neural network viewpoint prediction model;

[0013] A view probability cloud map generation module is configured to generate a view probability cloud map for a preset number of future frames based on the user's historical behavior data and the hybrid neural network view prediction model; the view probability cloud map includes a hotspot area, which is used to indicate at least one target pre-rendered view whose allocation probability exceeds a preset probability threshold;

[0014] a static scene segmentation module, configured to segment the pre-rendered static scene of the at least one target pre-rendered perspective based on a preset dynamic semantic weight matrix to obtain layered baked data; the layered baked data is used to generate optical flow data between the respective target pre-rendered perspectives, and the optical flow data is stored in a preset optical flow mapping table;

[0015] an optical flow data acquisition module, configured to acquire target optical flow data of a target adjacent viewpoint adjacent to the current viewpoint from the preset optical flow mapping table; the target optical flow data is used to indicate a pixel displacement relationship between the current viewpoint and the target adjacent viewpoint;

[0016] A scene mixing module is used to dynamically mix the static scene rendered in real time at the current perspective and the static scene pre-rendered according to the pixel displacement relationship based on the target optical flow data when the perspective is switched, to obtain scene mixing data;

[0017] The target static scene rendering module is used to render the target static scene based on the scene mixed data.

[0018] On the other hand, the present application also provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, any one of the methods for rendering a static scene under a dynamic perspective is implemented.

[0019] On the other hand, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements any of the methods for rendering a static scene under a dynamic perspective.

[0020] On the other hand, the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method for rendering a static scene under a dynamic perspective as described in the above aspects.

[0021] The present application provides a method, apparatus, device, and medium for rendering static scenes under dynamic perspectives. By acquiring a constructed hybrid neural network perspective prediction model, a perspective probability cloud map for a preset number of future frames is obtained based on user historical behavior data and the hybrid neural network perspective prediction model. The obtained perspective probability cloud map includes hotspot areas, which can be used to primarily indicate at least one target pre-rendered perspective whose allocation probability exceeds a preset probability threshold. The pre-rendered static scene for the at least one target pre-rendered perspective can then be divided based on a preset dynamic semantic weight matrix to obtain layered baking data. The layered baking data can be used to generate optical flow data between each target pre-rendered perspective. The generated optical flow data can be stored in a preset optical flow mapping table. Target optical flow data for a target adjacent perspective adjacent to the current perspective can then be obtained from the preset optical flow mapping table. The target optical flow data indicates the pixel displacement relationship between the current perspective and the target adjacent perspective. When switching between the current perspective and the target adjacent perspective, the static scene rendered in real time from the current perspective and the static scene pre-rendered according to the pixel displacement relationship can be dynamically blended based on the target optical flow data to eliminate visual jumps, thereby obtaining scene blending data. The target static scene can then be rendered based on the scene blending data, thereby achieving rendering of static scenes under dynamic perspectives. Through the collaborative solution of perspective prediction, layered pre-rendering and optical flow compensation, perspective prediction is combined with layered pre-rendering to solve the problem of dynamic perspective adaptation, and visual jumps are eliminated based on optical flow compensation, realizing the fusion of dynamic and static rendering, solving the problem of real-time high-quality rendering of static scenes under dynamic perspective, and balancing storage overhead, loading speed and visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flowchart of a method for rendering a static scene under a dynamic perspective provided by an embodiment of the present application;

[0023] Figure 2 Schematic diagram of the training process of the hybrid neural network perspective prediction model provided in an embodiment of the present application;

[0024] Figure 3 This is a schematic diagram of the generation process of layered baking data provided by an embodiment of the present application;

[0025] Figure 4 This is a structural block diagram of a device for rendering a static scene under a dynamic perspective provided by an embodiment of the present application;

[0026] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present application;

[0027] Figure 6 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0029] Pre-rendering technologies typically utilize Unity's lighting baking system. However, this technology only supports static lighting, meaning it supports pre-rendering for fixed viewpoints and cannot adapt to dynamic viewpoint changes. Since dynamic viewpoints require recalculating indirect lighting, data must be reloaded when switching viewpoints, causing lag. Consequently, lighting and shadow inconsistencies between static pre-rendered and dynamic real-time rendered elements can create visual disjunction, leading to an unnatural blend of static and dynamic elements and making it difficult to render static scenes under dynamic viewpoints. Furthermore, Epic Games' Nanite technology, which aims to revolutionize the way geometric details are handled in real-time rendering, relies on GPU (Graphics Processing Unit) hardware acceleration and cannot run smoothly on low-end devices. Indiscriminate pre-rendering results in redundant data in invisible areas, which consumes storage and bandwidth, leading to wasted resources. Furthermore, high-precision pre-rendering of the entire scene requires terabytes of storage, while lower-precision solutions result in reduced visual quality, creating a storage-performance trade-off.

[0030] The embodiment of the present application combines perspective prediction with layered pre-rendering through a hybrid neural network perspective prediction model based on user historical behavior data and a spatiotemporal consistency transition mechanism of optical flow compensation, based on a collaborative scheme of prediction, blocking, and compensation, to solve the problem of dynamic perspective adaptation. Based on the proposed optical flow compensation mechanism, it breaks through the technical bottleneck of the fusion of dynamic and static rendering, and further solves the problem of real-time high-quality rendering of static scenes under dynamic perspective, balancing storage overhead, loading speed and visual quality.

[0031] Reference Figure 1 , shows a flowchart of a method for rendering a static scene under a dynamic perspective provided by an embodiment of the present application, which may specifically include the following steps:

[0032] Step S101: Obtain user historical behavior data and a pre-built hybrid neural network perspective prediction model.

[0033] In some embodiments of the present application, perspective prediction is manifested as prediction of dynamic perspectives, which can be specifically implemented based on user historical behavior data and a pre-built hybrid neural network perspective prediction model.

[0034] Among them, the neural network structure of the hybrid neural network perspective prediction model can be a two-branch hybrid structure, and the hybrid neural network perspective prediction model can be obtained by training the first branch structure and the second branch structure in the two-branch hybrid structure based on the user's historical behavior data. Among them, the first branch structure can be LSTM (Long Short-Term Memory, long short-term memory network), and the second branch structure can be TCN (Temporal Convolutional Network, temporal convolutional network), that is, the hybrid neural network perspective prediction model has a hybrid network of LSTM layer and TCN layer. The specific training process of the hybrid neural network perspective prediction model is not limited in the embodiment of the present application.

[0035] User historical behavior data may refer to relevant data such as the user's historical trajectory, gaze point, and operating habits. Optionally, the user's historical behavior data may be in the form of a time series data sequence, which includes multiple time frames. For each frame of data, it may include three-dimensional position coordinates, viewing direction quaternion, and operation intensity. Among them, the three-dimensional position coordinates can be used to indicate the user's historical trajectory, and its data input source can be, for example, a VR (Virtual Reality) helmet posture sensor; the viewing direction quaternion can be used to indicate the user's gaze point, and its data input source can be an eye tracking gaze point; the operation intensity can be used to indicate the user's operating habits, such as zoom / pan speed, etc., and its data input source can be a mouse trajectory heat map.

[0036] Step S102: obtaining a view probability cloud map of a preset number of frames in the future based on the user's historical behavior data and the hybrid neural network view prediction model.

[0037] The hybrid neural network perspective prediction model can predict the perspective probability distribution of a preset number of frames in the future based on user historical behavior data.

[0038] Specifically, the view probability distribution for a preset number of future frames can be represented based on a view probability cloud map. Optionally, the user's historical behavior data can be input into a pre-built hybrid neural network view prediction model to output a view probability cloud map for a preset number of future frames. The preset number of future frames can be a preset number of future frames based on historical time frames in the time series of the user's historical behavior data, for example, the next 30 frames.

[0039] Optionally, the three-dimensional position coordinates, viewing direction quaternion and operation intensity in each frame data can be normalized to obtain normalized data, and then the normalized data can be processed based on the first branch structure in the dual-branch hybrid structure to obtain a first output, and the normalized data can be processed based on the second branch structure in the dual-branch hybrid structure to obtain a second output; at this time, the first output and the second output can be spliced ​​in the feature dimension to obtain a feature fusion result, and the feature fusion result can be mapped to an output of a preset dimension through the fully connected layer of the dual-branch hybrid structure, and the output of the preset dimension is processed using a preset activation function to obtain a viewing probability cloud map of a preset number of frames in the future, and then the viewing probability distribution of a preset number of frames in the future can be predicted based on the viewing probability cloud map.

[0040] Step S103 : dividing the pre-rendered static scene of at least one target pre-rendered perspective based on a preset dynamic semantic weight matrix to obtain layered baking data.

[0041] The view probability cloud map may include a hotspot area, which generally indicates a high-probability spatial location of an event, and may specifically be used to indicate at least one target pre-rendering view whose allocation probability exceeds a preset probability threshold.

[0042] In an embodiment of the present application, perspective prediction can be combined with layered pre-rendering to solve the problem of dynamic perspective adaptation.

[0043] In some embodiments of the present application, a pre-rendered static scene of at least one target pre-rendering perspective can be divided based on a preset dynamic semantic weight matrix to obtain layered baking data. Specifically, the division of the pre-rendered static scene of each target pre-rendering perspective is performed by dividing each pre-rendered scene into units of different levels, where the units of different levels correspond to different baking accuracies, thereby generating layered baking data, which is then used as the basis for layered pre-rendering.

[0044] Optionally, first, a scene analysis can be performed on the pre-rendered static scenes of each target pre-rendering perspective to obtain a scene analysis result. Then, based on the scene analysis result, a first target pre-rendered static scene can be obtained in which the scene volume of the pre-rendered static scenes of each target pre-rendering perspective exceeds a preset first threshold and the proportion of dynamic objects is less than or equal to a preset second threshold. Then, the first target pre-rendered static scene can be divided based on a preset dynamic semantic weight matrix to obtain layered baking data.

[0045] The use of the preset dynamic semantic weight matrix in the division of pre-rendered static scenes can be manifested as obtaining the regional staticity score, occlusion probability and historical attention of at least one first-target pre-rendered static scene, and then calculating the regional staticity score, occlusion probability and historical attention of each first-target pre-rendered static scene based on the preset dynamic semantic weight matrix to obtain the weight value of each first-target pre-rendered static scene, and then dividing each first-target pre-rendered static scene into units of different levels based on each weight value, and using a technology stack that matches the baking accuracy to bake the corresponding first-target pre-rendered static scene to generate pre-rendering metadata.

[0046] For the second target pre-rendered static scene in the pre-rendered static scene of each target pre-rendered perspective, the pre-rendered scene division operation is not performed; the second target pre-rendered static scene may refer to a pre-rendered static scene whose scene volume is greater than a preset first threshold, or whose dynamic object ratio is greater than a preset second threshold. The preset rendering data is baked based on the second target pre-rendered static scene. In this case, layered baking data can be obtained based on the pre-rendering metadata and the preset rendering data.

[0047] In a preferred embodiment of the present application, the layered baking data can be used to generate optical flow data between each target pre-rendering perspective, and the optical flow data can be stored in a preset optical flow mapping table.

[0048] Specifically, the baking data of each target pre-rendering perspective can be obtained from the layered baking data. Then, based on the baking data of any target pre-rendering perspective, the brightness of the preset image position of any target pre-rendering perspective at the preset first moment and the brightness of the preset image position at the preset second moment can be obtained. By adopting the preset dense optical flow algorithm, the preset image position and the brightness of the preset image position of any target pre-rendering perspective at the preset first moment and the brightness of the preset image position at the preset second moment are calculated to obtain the optical flow data of adjacent perspectives adjacent to the corresponding target pre-rendering perspective.

[0049] Furthermore, the optical flow data of any target pre-rendered perspective and adjacent perspectives adjacent to the corresponding target pre-rendered perspective can be stored in a preset optical flow mapping table. The preset optical flow mapping table is a structured data storage table for recording optical flow information at different perspectives, which can be easily retrieved and called. For example, the calculated optical flow information can be stored in FP16 compression format, which is not limited in this embodiment of the present application.

[0050] The optical flow data represents the motion vector of the pixel points in the image between adjacent frames, specifically the optical flow displacement vector, that is, the optical flow components in the x and y directions, which reflects the dynamic changes of the target object or scene.

[0051] Step S104: Obtain target optical flow data of target adjacent viewing angles adjacent to the current viewing angle from a preset optical flow mapping table.

[0052] The optical flow data stored in the preset optical flow map is a hybrid rendering technology that combines pre-calculated lighting information with real-time perspective interpolation, which can be used to optimize lighting consistency under dynamic perspective.

[0053] The preset second moment and the preset first moment may be adjacent time intervals. In practical applications, optical flow data between the current view and a target view adjacent to the current view is calculated and stored in a preset optical flow mapping table. That is, the preset optical flow mapping table stores target optical flow data for target view adjacent to the current view.

[0054] In some embodiments of the present application, target optical flow data of a target adjacent perspective adjacent to the current perspective may be obtained from a preset optical flow mapping table.

[0055] Step S105 , when switching the perspective, dynamically mix the static scene rendered in real time at the current perspective and the static scene pre-rendered according to the pixel displacement relationship based on the target optical flow data to obtain scene mixed data.

[0056] When switching the current perspective to the target adjacent perspective, the visual jump can be eliminated based on optical flow compensation to achieve the fusion of dynamic and static rendering.

[0057] Optionally, the target optical flow data can be used to indicate the pixel displacement relationship between the current view and the target adjacent view. In this case, a static scene rendered in real time from the current view and a pre-rendered static scene based on the pixel displacement relationship can be dynamically blended based on the target optical flow data to generate scene blending data, which can then be used to render the target static scene.

[0058] In some embodiments of the present application, the target optical flow data may include a target optical flow displacement vector, specifically an optical flow displacement vector, that is, an optical flow component in the x and y directions, which can reflect the dynamic changes of the target object or scene.

[0059] In practical applications, the perspective switching speed can be obtained, and the perspective switching speed and the target optical flow displacement vector can be used to dynamically adjust the mixing weight. Then, the mixing weight is used to mix the current real-time rendered static scene and the static scene pre-rendered according to the pixel displacement relationship to obtain scene mixing data. The scene mixing data is used as the basis of the target static scene under the dynamic perspective, thereby achieving the purpose of eliminating visual jumps.

[0060] In a preferred embodiment of the present application, the scene mixing data involves the processing of continuous frames. When processing continuous frames, such as when the perspective moves or the scene changes dynamically, there is usually a large amount of redundant information between adjacent frames. The aforementioned time redundancy can be utilized through motion compensation compression technology to improve the compression efficiency. Specifically, adjacent frame difference data of adjacent frame difference areas in the scene mixing data can be obtained. The adjacent frame difference data can refer to the redundant data between the current perspective and the target adjacent perspective adjacent to the current perspective. At this time, the adjacent frame difference data can be compressed using motion compensation to obtain compressed data; and a decoding table can be pre-generated for the compressed data. The generated decoding table can be stored in the constant memory of the graphics processor GPU for rendering the target static scene, and the generated decoding table can be parallel to the decoding block data.

[0061] Step S106: Rendering to obtain a target static scene based on the scene mixing data.

[0062] The generated decoding table can be parallel to the decoding block data, and can be decoded in parallel by the graphics processor GPU to obtain scene mixing data to render the target static scene, thereby realizing the rendering of static scenes under dynamic perspective.

[0063] In some embodiments of the present application, in order to facilitate those skilled in the art to further understand the rendering implementation of static scenes under dynamic perspectives provided in the embodiments of the present application, the following description is given in conjunction with the specific implementation process:

[0064] Reference Figure 2 , shows a schematic diagram of the training process of the hybrid neural network perspective prediction model provided by the embodiment of the present application. Figure 2 As shown, the hybrid neural network perspective prediction model may include a module for preparing input data, which may be used to input corresponding user historical behavior data based on the selection of a data input source. The data input source may include a VR helmet posture sensor, an eye tracking gaze point, and a mouse trajectory heat map. For example, the VR helmet posture sensor may be used to input position / posture data, such as three-dimensional position coordinates, which may be used to indicate the user's historical trajectory; the eye tracking gaze point may be used to input the perspective direction quaternion, which may be used to indicate the user's gaze point, i.e., the eye tracking focus; the mouse trajectory heat map / mouse movement heat map may be used to input the operation intensity, which may be used to indicate the user's operation habits, such as zoom / pan speed, etc.

[0065] The neural network structure of the hybrid neural network perspective prediction model can be a two-branch hybrid structure, where the first branch structure can be an LSTM and the second branch structure can be a TCN. That is, the hybrid neural network perspective prediction model comprises a hybrid network of LSTM and TCN layers. In this case, the hybrid network of LSTM and TCN layers can be trained based on the user's historical behavior data input by the preparation input data module, so that the trained hybrid neural network perspective prediction model can predict the perspective probability distribution of a preset number of future frames.

[0066] Optionally, the user historical behavior data can be input in the form of a time series data sequence, which includes n time frames. For example, the time series data sequence can be A={a1, a2, ..., a n}, where a1, a2, ..., a n These are the frame data for each time frame. Each frame data may include three-dimensional position coordinates (x, y, z), viewing direction quaternion (qw, qx, qy, qz), and operation intensity such as zoom / pan speed.

[0067] The training process of the hybrid neural network perspective prediction model can be expressed as normalizing the three-dimensional position coordinates (x, y, z), the perspective direction quaternion (qw, qx, qy, qz) and the operation intensity such as the zoom / translation speed in each frame data. For example, after the aforementioned frame data is normalized, it is normalized to the range of [-1, 1] to obtain normalized data; then, the LSTM branch and the TCN branch in the dual-branch hybrid structure can be trained based on the normalized data, and the first output of the first branch structure and the second output of the second branch structure are spliced ​​on the feature dimension to obtain a feature fusion result, and then the feature fusion result is mapped to an output of a preset dimension through the fully connected layer of the dual-branch hybrid structure, and the output of the preset dimension is processed using a preset activation function to obtain a perspective probability cloud map, so as to train the hybrid neural network perspective prediction model.

[0068] For example, the pseudo code implementation of the view prediction model training can be as follows:

[0069]

[0070] Among them, the LSTM branch can be used to capture long-term time dependencies. Its configuration can use a single-layer LSTM with 128 hidden units and return_sequences=True to retain the output of all time steps, which can be expressed as: lstm_layer=LSTM(units=128,return_sequences=True); the TCN branch can use dilated convolution to expand the receptive field and efficiently capture multi-scale temporal patterns. Assuming that it has a configuration with 64 filters, a convolution kernel size of 3, and an expansion factor of [1,2,4], it forms three layers of dilated convolution, covering dependencies of 1, 2, and 4 steps respectively, which can be expressed as: tcn_layer =TCN(n_filters=64, kernel_size=3,dilations=[1,2,4]); The fusion layer can achieve full connection after splicing the outputs of the two branches. Specifically, the outputs of the LSTM and TCN branches can be spliced ​​in the feature dimension for feature fusion. Then, it can be mapped to a 30-dimensional output through a fully connected layer (Dense) and use the Softmax activation function to finally output the view probability distribution of the next 30 frames.

[0071] The hybrid neural network perspective prediction model can predict the perspective probability distribution of a preset number of future frames using historical user behavior data. That is, after training the hybrid neural network perspective prediction model, the user's historical behavior data can be input into the hybrid neural network perspective prediction model, and the output is a perspective probability cloud map for a preset number of future frames.

[0072] The view probability cloud map can include hotspots, which indicate at least one target pre-rendered view whose probability of being assigned exceeds a preset probability threshold. The pre-rendered static scene for the at least one target pre-rendered view can then be partitioned based on a preset dynamic semantic weight matrix to generate layered baked data. This layered baked data forms the basis for layered pre-rendering, enabling the integration of view prediction and layered pre-rendering to address dynamic view adaptation.

[0073] Specifically, refer to Figure 3 , which shows a schematic diagram of the layered baking data generation process provided by an embodiment of the present application. The pre-rendered static scenes for each target pre-rendering perspective are divided into units of different levels. Units of different levels correspond to different baking accuracies, thereby generating layered baking data, which is then used as the basis for layered pre-rendering.

[0074] In some embodiments of the present application, first, a scene analysis can be performed on the pre-rendered static scenes of each target pre-rendering perspective. For the first target pre-rendered static scene whose scene volume exceeds a preset first threshold and whose proportion of dynamic objects is less than or equal to a preset second threshold, the first target pre-rendered static scene can be divided based on a preset dynamic semantic weight matrix; for the second target pre-rendered static scene whose scene volume is greater than the preset first threshold or whose proportion of dynamic objects is greater than the preset second threshold, the pre-rendering scene division operation is not performed.

[0075] Optionally, the division of the pre-rendered static scene can be achieved by performing octree segmentation. For example, assuming that the first threshold is preset to be 100m 3 , the second threshold is preset to 70%, such as Figure 3 As shown, for scene volumes ≥ 100m 3 For pre-rendered static scenes with dynamic objects accounting for ≤70%, the dynamic semantic weight matrix can be calculated by performing octree segmentation to achieve spatial octree partitioning; for scene volumes <100m 3 Alternatively, for pre-rendered static scenes with dynamic objects accounting for more than 70%, skip segmentation.

[0076] For example, for the calculation of the dynamic semantic weight matrix, the pseudo code of its algorithm logic can be as follows:

[0077]

[0078] The specific formula of the preset dynamic semantic weight matrix can be shown as follows:

[0079]

[0080] Among them, x, y, z are the three-dimensional space point coordinates of a certain first target pre-rendered static scene, W (x, y, z) is the weight value of a certain first target pre-rendered static scene, Score the staticness of a region for a first target pre-rendered static scene, Occlusion probability of pre-rendered static scene for a first target, The historical attention of pre-rendering a static scene for a first target, α, β, and γ are the weight proportions of each semantic respectively.

[0081] For example, the regional staticness score can be calculated based on the statistical movement variance of scene elements in historical frames. The specific calculation formula can be as follows:

[0082]

[0083] Among them, S static(i) is the normalized static score of scene element i in a first target pre-rendered static scene, with a value range of [0,1]; Var(x i ,y i ,z i ) represents the position variance of scene element i in a first target pre-rendered static scene in the historical frames; The maximum position variance value in a pre-rendered static scene for a first target.

[0084] The occlusion probability can be obtained by using the ray casting method to detect the visibility of the area. The pseudo code of its algorithm logic can be shown as follows:

[0085]

[0086] For example, rays can be emitted from multiple perspectives to a region of a first target pre-rendered static scene, and the ratio of the number of times the rays hit the region (i.e., the number of times the rays are not blocked) to the total number of rays is calculated. This ratio can be the visibility. In the weight calculation, the region with a high probability of occlusion is usually given a low weight, and the region with a high probability of occlusion S is given a low weight. occlusion It can be: occlusion probability = (1-visibility).

[0087] The historical attention can be obtained based on the Gaussian fuzzy of the user's historical operation heat map. The specific calculation formula can be as follows:

[0088]

[0089] Among them, x, y, z are the three-dimensional space coordinates of a first target pre-rendered static scene; S attention (x, y, z) is the historical attention of a first target pre-rendered static scene 3D space point x, y, z; (x t ,y t ) is the user operation position, that is, the gaze point; t is the time sequence identifier of the historical operation record; T is the total number of historical operation points considered; σ is the spatial influence radius; w(t) is the time attenuation weight, which can be based on the attenuation coefficient of temporal proximity to ensure that the weight of recent operation points is higher.

[0090] After calculating the weight values ​​of each first-target pre-rendered static scene, a weight value judgment can be performed. For example, each pre-rendered scene can be divided into four levels, L0-L3, based on each weight value. For example, if W>0.7, the corresponding first-target pre-rendered static scene can be divided into the L0 level; if 0.4≤W≤0.7, the corresponding first-target pre-rendered static scene can be divided into the L1 level; if W<0.4, the corresponding first-target pre-rendered static scene can be divided into the L3 level.

[0091] Different levels of units correspond to different baking precisions, as shown in Table 1 below:

[0092] Table 1 Baking accuracy of different levels of units

[0093]

[0094] Optionally, a technology stack that matches the baking accuracy can be used to bake the corresponding first target pre-rendered static scene. For example, for L0 level baking, there is a baking accuracy requirement for all 4K HDR elements, and ray tracing global illumination can be used for baking; for L1 level baking, there is a baking accuracy requirement for 2K PBR materials and reflections, and spherical harmonic spectra and emission probes can be used for baking; for L3 level baking, there is a baking accuracy requirement for 512px depth profiles, and depth buffer and template cache can be used for baking. At this point, data compression and storage operations can be performed on the data obtained from the baking process to generate pre-rendering metadata.

[0095] For scene volumes less than 100m 3 For pre-rendered static scenes with dynamic objects accounting for more than 70%, skip the octree segmentation and use the dynamic optimization mode for baking to obtain the preset rendering data.

[0096] The resulting layered baking data may include pre-rendering metadata generated by baking the first target pre-rendered static scene, and preset rendering data obtained by baking the second target pre-rendered static scene. It should be noted that the layered baking data has a hierarchical structure, enabling hierarchical storage of scene lighting information in the spatial or frequency domain, which is not limited in this embodiment of the present application.

[0097] In some embodiments of the present application, the technical bottleneck of dynamic and static rendering fusion can be broken through based on the proposed optical flow compensation mechanism.

[0098] The optical flow compensation mechanism is manifested as optical flow calculation and interpolation hybrid calculation, which enables the subsequent rendering of static scenes to achieve the effect of temporal and spatial consistency compensation. The pseudo code of its algorithm logic can be shown as follows:

[0099]

[0100] Optionally, the layered baked data can be used to generate optical flow data between each target pre-rendered view, and the optical flow data can be stored in a preset optical flow mapping table. The preset optical flow mapping table can store target optical flow data for the current view and target adjacent views adjacent to the current view, and the target optical flow data can be used to indicate the pixel displacement relationship between the current view and the target adjacent view.

[0101] Optical flow data represents the motion vector of pixels in an image between adjacent frames, specifically the optical flow displacement vector, that is, the optical flow components in the x and y directions, reflecting the dynamic changes of the target object or scene.

[0102] For example, the Farneback dense optical flow algorithm can be used to calculate the optical flow displacement vector of adjacent perspectives. The specific calculation formula of the optical flow calculation can be as follows:

[0103]

[0104] Among them, I(x,y,t) represents the brightness of the image position (x,y) at time t; I(x+u,y+v,t+1) represents the brightness of the image position (x+u,y+v) at time t+1; x refers to the spatial gradient of the image in the horizontal direction, y refers to the spatial gradient of the image in the vertical direction; t refers to the temporal gradient of the image in the time dimension; u and v are the optical flow components in the horizontal and vertical directions, respectively, that is, the displacement to be solved, then, the optical flow displacement vector of the adjacent perspective can be obtained based on the aforementioned displacement, which can be specifically represented by (u,v).

[0105] When switching from the current perspective to the target adjacent perspective, optical flow compensation can be used to eliminate visual jumps and achieve the fusion of dynamic and static rendering. Optionally, a blending weight can be used to blend the current real-time rendered static scene with a pre-rendered static scene based on pixel displacement relationships to obtain scene blending data for rendering the target static scene. This dynamic blending can be achieved by using a transition interpolation algorithm to calculate interpolation weights.

[0106] For example, the specific calculation formula for interpolation weight calculation can be as follows:

[0107]

[0108] in, is the mixing weight; λ is the attenuation coefficient, usually 0.1; (u, v) is the optical flow displacement vector; Δt is the view switching time interval, which can be used to define the view switching speed. That is, the mixing weight can be dynamically adjusted according to the view switching speed and the optical flow displacement vector (u, v). .

[0109] Optionally, the scene mixing data involves processing of consecutive frames, in which case block encoding and GPU decoding optimization can be performed.

[0110] Block coding can be specifically performed by dividing the texture in the scene mixed data into 8x8 blocks, and then performing DCT transform → quantization → Zigzag scan → Huffman encoding on each block. It should be noted that texture blocking is divided according to the principles of hardware architecture. Compared with the SIMD (Single Instruction, Multiple Data Register) register capacity evaluation of the CPU (Central Processing Unit) and GPU, 8x8 blocks are the optimal example, and the embodiments of this application are not limited to this.

[0111] GPU decoding optimization can be specifically manifested in using motion compensation to compress the difference data of adjacent frames to obtain compressed data, and pre-generating a decoding table for the compressed data. The generated decoding table can be stored in the constant memory of the graphics processing unit GPU for rendering the target static scene, and the generated decoding table is parallel to the decoding block data.

[0112] For example, the pseudo code of its algorithm logic can be shown as follows:

[0113]

[0114] It should be noted that CPU decoding will become a performance bottleneck in high-resolution texture scenarios (4K texture decoding requires 10-20ms). The parallel decoding of the GPU in the embodiment of the present application can reduce the time consumption to within 1ms, which can solve the bottleneck of real-time decoding of texture compression data on the GPU side.

[0115] Alternatively, the rendering solution provided in the embodiments of the present application can be applied to any application scenario where static scene rendering is required under dynamic viewing angles. Taking the smart city digital twin platform as an example, assuming its hardware environment is NVIDIA RTX 4090, 64GB RAM.

[0116] For the smart city digital twin platform, its input data can, for example, be expressed as input of urban BIM (Building Information Modeling Model, such as buildings, roads, vegetation, etc.), dynamic traffic flow data, etc. Before inputting the aforementioned data, the aforementioned data can also be pre-processed, for example, using Simplygon to simplify the model to reduce its polygon count by 80%, etc. The embodiments of the present application are not limited to this.

[0117] When applying the hybrid neural network perspective prediction model to the smart city digital twin platform, historical user behavior data can be represented by the collection of 100,000 user operation records. These collected user operation records can be used to train the hybrid neural network with a prediction accuracy of 92%. Based on the hybrid neural network perspective prediction model, a perspective probability cloud map for the next one second can be output in real time. This perspective probability cloud map contains hot spots, also known as a perspective heat map. For the core area in the perspective heat map, usually the red area of ​​the heat map, L0 level full-precision baking can be triggered to generate ray tracing light maps. For the edge area in the perspective heat map, usually the blue area of ​​the heat map, L3 level depth profile maps can be generated. The layered baking data generated above can then be used for real-time fusion rendering.

[0118] Real-time fusion rendering can be expressed as dynamically loading pre-baked blocks into the video memory. In the smart city digital twin scenario, real-time shadow mapping can be implemented for dynamic objects such as vehicles and pedestrians, and multi-channel synthesis can be performed. Then, when switching perspectives, the static scene rendered in real time from the current perspective and the pre-rendered static scene can be dynamically mixed, and the target static scene is rendered based on the scene mixing data, thereby realizing the rendering of static scenes under dynamic perspectives in the smart city digital twin platform.

[0119] It should be noted that for the specific rendering process, you can refer to the content of the method embodiment, and the embodiments of this application will not be described in detail here; and the rendering of static scenes under dynamic perspectives in the smart city digital twin platform has better frame time, video memory occupancy and transition error than mainstream solutions such as Unity baking system and Nanite technology, which can solve the problem of real-time high-quality rendering of static scenes under dynamic perspectives, and balance storage overhead, loading speed and visual quality.

[0120] In an embodiment of the present application, a constructed hybrid neural network view prediction model is obtained, and a view probability cloud map for a preset number of future frames is obtained based on user historical behavior data and the hybrid neural network view prediction model. The obtained view probability cloud map includes a hotspot area, which can be mainly used to indicate at least one target pre-rendered view whose allocation probability exceeds a preset probability threshold. Then, based on a preset dynamic semantic weight matrix, the pre-rendered static scene of the at least one target pre-rendered view can be divided to obtain layered baking data. The layered baking data can be used to generate optical flow data between each target pre-rendered view. The generated optical flow data can be stored in a preset optical flow mapping table. Target optical flow data of a target adjacent view adjacent to the current view is obtained from the preset optical flow mapping table. The target optical flow data indicates the pixel displacement relationship between the current view and the target adjacent view. When switching between the current view and the target adjacent view, the static scene rendered in real time from the current view and the static scene pre-rendered according to the pixel displacement relationship can be dynamically mixed based on the target optical flow data to eliminate visual jumps, thereby obtaining scene mixing data. Then, the target static scene is rendered based on the scene mixing data, thereby achieving rendering of static scenes under dynamic view. Through the collaborative solution of perspective prediction, layered pre-rendering and optical flow compensation, perspective prediction is combined with layered pre-rendering to solve the problem of dynamic perspective adaptation, and visual jumps are eliminated based on optical flow compensation, realizing the fusion of dynamic and static rendering, solving the problem of real-time high-quality rendering of static scenes under dynamic perspective, and balancing storage overhead, loading speed and visual quality.

[0121] It should be noted that for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0122] Reference Figure 4 , shows a structural block diagram of a rendering device for a static scene under a dynamic perspective provided by an embodiment of the present application, which may specifically include the following modules:

[0123] A viewpoint prediction model acquisition module 401 is used to acquire user historical behavior data and a pre-built hybrid neural network viewpoint prediction model;

[0124] A view probability cloud map generation module 402 is configured to generate a view probability cloud map for a preset number of future frames based on historical user behavior data and a hybrid neural network view prediction model; the view probability cloud map includes a hotspot region, which indicates at least one target pre-rendered view whose allocation probability exceeds a preset probability threshold;

[0125] A static scene segmentation module 403 is configured to segment the pre-rendered static scene of at least one target pre-rendered view based on a preset dynamic semantic weight matrix to obtain layered baked data; the layered baked data is used to generate optical flow data between the target pre-rendered views, and the optical flow data is stored in a preset optical flow mapping table;

[0126] The optical flow data acquisition module 404 is used to obtain target optical flow data of a target adjacent viewpoint adjacent to the current viewpoint from a preset optical flow mapping table; the target optical flow data is used to indicate the pixel displacement relationship between the current viewpoint and the target adjacent viewpoint;

[0127] The scene mixing module 405 is used to dynamically mix the static scene rendered in real time at the current perspective and the static scene pre-rendered according to the pixel displacement relationship based on the target optical flow data when the perspective is switched to obtain scene mixing data;

[0128] The target static scene rendering module 406 is configured to render the target static scene based on the scene mixing data.

[0129] In some embodiments of the present application, the neural network structure of the hybrid neural network view prediction model is a dual-branch hybrid structure; the user historical behavior data includes a time series data sequence, which includes multiple time frames, wherein each frame data includes three-dimensional position coordinates, a view direction quaternion, and an operation intensity; the view probability cloud map generation module 402 may include the following submodules:

[0130] The view probability cloud map generation submodule is used to normalize the three-dimensional position coordinates, view direction quaternion and operation intensity in each frame data to obtain normalized data; based on the first branch structure in the dual-branch hybrid structure, the normalized data is processed to obtain a first output; and, based on the second branch structure in the dual-branch hybrid structure, the normalized data is processed to obtain a second output; the first output and the second output are spliced ​​in the feature dimension to obtain a feature fusion result; the feature fusion result is mapped to an output of a preset dimension through the fully connected layer of the dual-branch hybrid structure, and the output of the preset dimension is processed using a preset activation function to obtain a view probability cloud map of a preset number of frames in the future.

[0131] In some embodiments of the present application, the static scene division module 403 may include the following submodules:

[0132] The layered baking data generation submodule is used to perform scene analysis on the pre-rendered static scenes of each target pre-rendering perspective to obtain scene analysis results; based on the scene analysis results, obtain the first target pre-rendered static scene in the pre-rendered static scenes of each target pre-rendering perspective, in which the scene volume exceeds a preset first threshold and the proportion of dynamic objects is less than or equal to a preset second threshold; divide the first target pre-rendered static scene based on a preset dynamic semantic weight matrix to obtain layered baking data.

[0133] In some embodiments of the present application, the layered baking data generation submodule may include the following units:

[0134] A layered baking data generation unit is used to obtain the regional staticity score, occlusion probability and historical attention of at least one first-target pre-rendered static scene; based on a preset dynamic semantic weight matrix, the regional staticity score, occlusion probability and historical attention of each first-target pre-rendered static scene are calculated respectively to obtain the weight value of each first-target pre-rendered static scene; based on each weight value, each first-target pre-rendered static scene is divided into units of different levels; units of different levels correspond to different baking precisions; a technology stack matching the baking precision is used to bake the corresponding first-target pre-rendered static scene to generate pre-rendering metadata; layered baking data is obtained based on the pre-rendering metadata and preset rendering data; wherein the preset rendering data is obtained by baking based on the second-target pre-rendered static scene, and the second-target pre-rendered static scene is a pre-rendered static scene with a scene volume greater than a preset first threshold, or a dynamic object ratio greater than a preset second threshold.

[0135] In some embodiments of the present application, after obtaining the layered baking data, the apparatus provided by the embodiments of the present application may further include the following modules:

[0136] An optical flow data storage module is used to obtain baking data of each target pre-rendering perspective from the layered baking data; based on the baking data of any target pre-rendering perspective, obtain the brightness of the preset image position of any target pre-rendering perspective at a preset first moment and the brightness of the preset image position at a preset second moment; the preset second moment and the preset first moment are adjacent time intervals; using a preset dense optical flow algorithm, the preset image position, the brightness of the preset image position of any target pre-rendering perspective at the preset first moment and the brightness of the preset image position at the preset second moment are calculated to obtain optical flow data of adjacent perspectives adjacent to the corresponding target pre-rendering perspective; and the optical flow data of any target pre-rendering perspective and the adjacent perspectives adjacent to the corresponding target pre-rendering perspective are stored in a preset optical flow mapping table.

[0137] In some embodiments of the present application, the target optical flow data includes a target optical flow displacement vector; the scene blending module 405 may include the following submodules:

[0138] The scene mixing submodule is used to obtain the perspective switching speed and dynamically adjust the mixing weight using the perspective switching speed and the target optical flow displacement vector; the mixing weight is used to mix the current real-time rendered static scene and the static scene pre-rendered according to the pixel displacement relationship to obtain scene mixing data.

[0139] In some embodiments of the present application, after obtaining the scene mixing data, the apparatus provided by the embodiment of the present application may further include the following modules:

[0140] The decoding optimization module is used to obtain adjacent frame difference data of adjacent frame difference areas in the scene mixed data; compress the adjacent frame difference data using motion compensation to obtain compressed data; pre-generate a decoding table for the compressed data; and store the decoding table in the constant memory of the graphics processor for rendering the target static scene.

[0141] In an embodiment of the present application, a constructed hybrid neural network view prediction model is obtained, and a view probability cloud map for a preset number of future frames is obtained based on user historical behavior data and the hybrid neural network view prediction model. The obtained view probability cloud map includes a hotspot area, which can be mainly used to indicate at least one target pre-rendered view whose allocation probability exceeds a preset probability threshold. Then, based on a preset dynamic semantic weight matrix, the pre-rendered static scene of the at least one target pre-rendered view can be divided to obtain layered baking data. The layered baking data can be used to generate optical flow data between each target pre-rendered view. The generated optical flow data can be stored in a preset optical flow mapping table. At this time, target optical flow data of a target adjacent view adjacent to the current view can be obtained from the preset optical flow mapping table. The target optical flow data indicates the pixel displacement relationship between the current view and the target adjacent view. When switching between the current view and the target adjacent view, the static scene rendered in real time from the current view and the static scene pre-rendered according to the pixel displacement relationship can be dynamically mixed based on the target optical flow data to eliminate visual jumps, thereby obtaining scene mixing data. Then, the target static scene is rendered based on the scene mixing data, thereby achieving rendering of static scenes under dynamic perspectives. Through the collaborative solution of perspective prediction, layered pre-rendering and optical flow compensation, perspective prediction is combined with layered pre-rendering to solve the problem of dynamic perspective adaptation, and visual jumps are eliminated based on optical flow compensation, realizing the fusion of dynamic and static rendering, solving the problem of real-time high-quality rendering of static scenes under dynamic perspective, and balancing storage overhead, loading speed and visual quality.

[0142] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0143] The present application also provides an electronic device, Figure 5The provided electronic device 500 includes a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and capable of running on the processor 520. When the computer program 511 is executed by the processor, the various processes of the embodiment of the rendering method of the static scene under the above-mentioned dynamic perspective are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0144] The present application also provides a computer-readable storage medium. Figure 6 The computer-readable storage medium 600 provided stores a computer program 511. When the computer program 511 is executed by the processor, the various processes of the embodiment of the rendering method of the static scene under the above-mentioned dynamic perspective are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0145] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0146] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than that shown or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules that appears in the embodiments of the present application is only a logical division. In actual applications, there may be other division methods. For example, multiple modules can be combined into or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in the embodiments of the present application. Moreover, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed into multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0147] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0148] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0149] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0150] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0151] In addition, the functional modules in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into a module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0152] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0153] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that can be stored on a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, hard disk, or magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0154] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0156] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0157] Finally, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0158] The above is a detailed introduction to the technical solutions provided in the embodiments of the present application. Specific examples are used in the embodiments of the present application to illustrate the principles and implementation methods of the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

Claims

1. A method for rendering a static scene under a dynamic perspective, characterized in that: The method comprises: Obtain user historical behavior data and pre-built hybrid neural network perspective prediction model; Obtaining a view probability cloud map for a preset number of frames in the future based on the user historical behavior data and the hybrid neural network view prediction model; the view probability cloud map includes a hotspot area, and the hotspot area is used to indicate at least one target pre-rendered view whose allocation probability exceeds a preset probability threshold; Dividing the pre-rendered static scene of the at least one target pre-rendered perspective based on a preset dynamic semantic weight matrix to obtain layered baking data; using the layered baking data to generate optical flow data between the respective target pre-rendered perspectives, and storing the optical flow data in a preset optical flow mapping table; Obtaining target optical flow data of a target adjacent viewpoint adjacent to the current viewpoint from the preset optical flow mapping table; the target optical flow data is used to indicate a pixel displacement relationship between the current viewpoint and the target adjacent viewpoint; When switching the perspective, dynamically mixing the static scene rendered in real time at the current perspective and the static scene pre-rendered according to the pixel displacement relationship based on the target optical flow data to obtain scene mixing data; A target static scene is obtained by rendering based on the scene mixing data.

2. The method according to claim 1, characterized in that The neural network structure of the hybrid neural network perspective prediction model is a dual-branch hybrid structure; the user historical behavior data includes a time series data sequence, and the time series data sequence includes multiple time frames, wherein each frame data includes three-dimensional position coordinates, perspective direction quaternion and operation intensity; The method of obtaining a view probability cloud map of a preset number of frames in the future based on the user historical behavior data and the hybrid neural network view prediction model includes: Normalizing the three-dimensional position coordinates, the viewing direction quaternion, and the operation intensity in each frame of data to obtain normalized data; Performing data processing on the normalized data based on the first branch structure in the dual-branch hybrid structure to obtain a first output; and performing data processing on the normalized data based on the second branch structure in the dual-branch hybrid structure to obtain a second output; Concatenating the first output and the second output in the feature dimension to obtain a feature fusion result; The feature fusion result is mapped to an output of a preset dimension through the fully connected layer of the dual-branch hybrid structure, and the output of the preset dimension is processed using a preset activation function to obtain a perspective probability cloud map of a future preset number of frames.

3. The method according to claim 1, characterized in that The step of dividing the pre-rendered static scene of the at least one target pre-rendered perspective based on a preset dynamic semantic weight matrix to obtain layered baking data includes: Performing scene analysis on the pre-rendered static scene of each target pre-rendered perspective to obtain scene analysis results; Based on the scene analysis result, obtaining, among the pre-rendered static scenes of each target pre-rendered perspective, a first target pre-rendered static scene whose scene volume exceeds a preset first threshold and whose dynamic object ratio is less than or equal to a preset second threshold; The first target pre-rendered static scene is divided based on a preset dynamic semantic weight matrix to obtain layered baking data.

4. The method according to claim 3, characterized in that The dividing the first target pre-rendered static scene based on a preset dynamic semantic weight matrix to obtain layered baking data includes: Obtaining a regional staticity score, an occlusion probability, and a historical attention of at least one of the first target pre-rendered static scenes; Based on a preset dynamic semantic weight matrix, the regional static score, occlusion probability, and historical attention of each first target pre-rendered static scene are calculated to obtain a weight value of each first target pre-rendered static scene; Divide each first target pre-rendered static scene into units of different levels based on each weight value; units of different levels correspond to different baking accuracies; Bake the corresponding first target pre-rendered static scene using a technology stack that matches the baking accuracy to generate pre-rendering metadata; Based on the pre-rendering metadata and the preset rendering data, layered baking data is obtained; wherein the preset rendering data is obtained by baking based on a second target pre-rendered static scene, and the second target pre-rendered static scene is a pre-rendered static scene with a scene volume greater than the preset first threshold, or a dynamic object ratio greater than the preset second threshold.

5. The method according to claim 1, wherein After obtaining the layered baking data, the method further includes: Obtaining baking data of each target pre-rendering perspective from the layered baking data; Based on the baked data of any target pre-rendering view, obtaining the brightness of a preset image position at a preset first moment and the brightness of the preset image position at a preset second moment for any target pre-rendering view, wherein the preset second moment and the preset first moment are adjacent time intervals; Using a preset dense optical flow algorithm, the preset image position and the brightness of the preset image position of any target pre-rendered view at a preset first moment and the brightness of the preset image position at a preset second moment are calculated to obtain optical flow data of adjacent views adjacent to the corresponding target pre-rendered view; The optical flow data of any target pre-rendering perspective and adjacent perspectives adjacent to the corresponding target pre-rendering perspective are stored in the preset optical flow mapping table.

6. The method according to claim 1 or 5, characterized in that The target optical flow data includes a target optical flow displacement vector; the scene mixing data is obtained by dynamically mixing a static scene rendered in real time from a current perspective and a static scene pre-rendered according to the pixel displacement relationship based on the target optical flow data, including: Obtaining a viewing angle switching speed, and dynamically adjusting a blending weight using the viewing angle switching speed and the target optical flow displacement vector; The mixing weight is used to mix the static scene currently rendered in real time and the static scene pre-rendered according to the pixel displacement relationship to obtain scene mixing data.

7. The method according to claim 1, characterized in that After obtaining the scene mixing data, the method further includes: Acquire adjacent frame difference data of adjacent frame difference areas in the scene mixture data; compressing the adjacent frame difference data using motion compensation to obtain compressed data; A decoding table is pre-generated for the compressed data; the decoding table is stored in a constant memory of a graphics processor for rendering the target static scene.

8. A device for rendering a static scene under a dynamic perspective, characterized in that: The device comprises: A viewpoint prediction model acquisition module is used to obtain user historical behavior data and a pre-built hybrid neural network viewpoint prediction model; A view probability cloud map generation module is configured to generate a view probability cloud map for a preset number of future frames based on the user's historical behavior data and the hybrid neural network view prediction model; the view probability cloud map includes a hotspot area, and the hotspot area is configured to indicate at least one target pre-rendered view angle whose allocation probability exceeds a preset probability threshold; a static scene segmentation module, configured to segment the pre-rendered static scene of the at least one target pre-rendered perspective based on a preset dynamic semantic weight matrix to obtain layered baked data; the layered baked data is used to generate optical flow data between the respective target pre-rendered perspectives, and the optical flow data is stored in a preset optical flow mapping table; an optical flow data acquisition module, configured to acquire target optical flow data of a target adjacent viewpoint adjacent to the current viewpoint from the preset optical flow mapping table; the target optical flow data is used to indicate a pixel displacement relationship between the current viewpoint and the target adjacent viewpoint; A scene mixing module is used to dynamically mix the static scene rendered in real time at the current perspective and the static scene pre-rendered according to the pixel displacement relationship based on the target optical flow data when the perspective is switched, to obtain scene mixing data; The target static scene rendering module is used to render the target static scene based on the scene mixed data.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for rendering a static scene under a dynamic perspective as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for rendering a static scene under a dynamic perspective as claimed in any one of claims 1 to 7 is implemented.

Images