A method, device, equipment and medium for rendering a static scene under a dynamic view angle

Abstract

The application relates to the technical field of computer graphics, and provides a rendering method and device for a static scene under a dynamic view angle, equipment and a medium. The method comprises the following steps: obtaining a view angle probability cloud diagram of a future preset frame number according to user historical behavior data and a hybrid neural network view angle prediction model; dividing a pre-rendered static scene of at least one target pre-rendered view angle based on a preset dynamic semantic weight matrix to obtain layered baking data; obtaining target optical flow data of a target adjacent view angle adjacent to a current view angle from a preset optical flow mapping table; when the view angle is switched, dynamically mixing a static scene rendered in real time according to the current view angle and a static scene pre-rendered according to a pixel displacement relationship based on the target optical flow data to obtain scene mixing data; and rendering the target static scene based on the scene mixing data. The application can solve the problem of real-time high-quality rendering of a static scene under a dynamic view angle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer graphics, in particular to a rendering method of a static scene under a dynamic view angle, a rendering device of a static scene under a dynamic view angle, a corresponding electronic device and a corresponding computer readable storage medium. BACKGROUND

[0002] In the related art of pre-rendering, the light baking technology only supports static light, that is, only supports pre-rendering of a fixed view angle, and cannot adapt to dynamic view angle changes. Since dynamic view angles need to recalculate indirect light, data needs to be reloaded when the view angle is switched, causing lag. Based on this, there is an inconsistency in light and shadow between static pre-rendering and dynamic real-time rendering elements, which easily causes visual fragmentation, leading to unnatural blending of dynamic and static scenes, and making it difficult to achieve rendering of a static scene under a dynamic view angle. SUMMARY

[0003] The present application provides a rendering method, device, equipment and medium of a static scene under a dynamic view angle, which can solve the problem of real-time high-quality rendering of a static scene under a dynamic view angle, and balance storage overhead, loading speed and visual quality.

[0004] In one aspect, the present application provides a rendering method of a static scene under a dynamic view angle, the method comprising:

[0005] obtaining user historical behavior data and a pre-constructed hybrid neural network view angle prediction model;

[0006] obtaining a view angle probability cloud map of a future preset frame number according to the user historical behavior data and the hybrid neural network view angle prediction model; the view angle probability cloud map contains a hot spot area, and the hot spot area is used to indicate at least one target pre-rendering view angle with an allocation probability exceeding a preset probability threshold;

[0007] dividing a pre-rendered static scene of the at least one target pre-rendering view angle based on a preset dynamic semantic weight matrix to obtain layered baking data; the layered baking data is used to generate optical flow data between each target pre-rendering view angle, and the optical flow data is stored in a preset optical flow mapping table;

[0008] obtaining target optical flow data of a target adjacent view angle adjacent to a current view angle from the preset optical flow mapping table; the target optical flow data is used to indicate a pixel displacement relationship between the current view angle and the target adjacent view angle;

[0009] when the view angle is switched, dynamically mixing a static scene rendered in real time based on the current view angle and a static scene pre-rendered according to the pixel displacement relationship based on the target optical flow data to obtain scene mixing data;

[0010] render a target static scene based on the scene mixing data.

[0011] In another aspect, the present application provides a device for rendering a static scene under a dynamic view angle, the device comprising:

[0012] a view angle prediction model acquisition module for user historical behavior data and a pre-constructed hybrid neural network view angle prediction model;

[0013] a view angle probability cloud map generation module for obtaining a view angle probability cloud map of a future preset frame number based on the user historical behavior data and the hybrid neural network view angle prediction model; the view angle probability cloud map comprises a hotspot area, and the hotspot area is used to indicate at least one target pre-rendering view angle with an allocation probability exceeding a preset probability threshold;

[0014] a static scene division module for dividing a pre-rendering static scene of the at least one target pre-rendering view angle based on a preset dynamic semantic weight matrix to obtain layered baking data; the layered baking data is used to generate optical flow data between each target pre-rendering view angle, and the optical flow data is stored in a preset optical flow mapping table;

[0015] an optical flow data acquisition module for obtaining target optical flow data of a target adjacent view angle adjacent to a current view angle from the preset optical flow mapping table; the target optical flow data is used to indicate a pixel displacement relationship between the current view angle and the target adjacent view angle;

[0016] a scene mixing module for dynamically mixing a static scene rendered in real time in the current view angle and a pre-rendered static scene according to the pixel displacement relationship based on the target optical flow data when a view angle is switched to obtain scene mixing data;

[0017] a target static scene rendering module for rendering a target static scene based on the scene mixing data.

[0018] In yet another aspect, the present application further provides an electronic device comprising a processor, a memory, and a computer program stored on the memory and capable of running on the processor, wherein the computer program is executed by the processor to implement any of the rendering methods of a static scene under a dynamic view angle.

[0019] In yet another aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement any of the rendering methods of a static scene under a dynamic view angle.

[0020] In yet another aspect, the present application also provides a computer program product comprising instructions which, when executed on a computer, cause the computer to perform the rendering method of a static scene under a dynamic view angle as described in the above aspects.

[0021] The rendering method of a static scene under a dynamic view angle provided by the present application, by obtaining a constructed hybrid neural network view angle prediction model, obtaining a view angle probability cloud map of a future preset frame number according to user historical behavior data and the hybrid neural network view angle prediction model, wherein the obtained view angle probability cloud map contains a hot spot area, the hot spot area can be mainly used to indicate at least one target pre-rendering view angle with an allocation probability exceeding a preset probability threshold, then the pre-rendering static scene of the at least one target pre-rendering view angle can be divided based on a preset dynamic semantic weight matrix to obtain layered baking data, the layered baking data can be used to generate light flow data between each target pre-rendering view angle, and the generated light flow data can be stored in a preset light flow mapping table; at this time, target light flow data of a target adjacent view angle adjacent to a current view angle can be obtained from the preset light flow mapping table, the target light flow data indicates a pixel displacement relationship between the current view angle and the target adjacent view angle, when view angle switching between the current view angle and the target adjacent view angle is performed, the static scene rendered in real time based on the current view angle and the static scene pre-rendered according to the pixel displacement relationship can be dynamically mixed based on the target light flow data to eliminate visual jump, to obtain scene mixing data, and then the target static scene is rendered based on the scene mixing data, to realize the rendering of a static scene under a dynamic view angle. Through the collaborative scheme of view angle prediction, layered pre-rendering and light flow compensation, the view angle prediction is combined with the layered pre-rendering, the dynamic view angle adaptation problem is solved, the visual jump is eliminated based on the light flow compensation, the fusion of dynamic rendering and static rendering is realized, the real-time high-quality rendering problem of a static scene under a dynamic view angle is solved, and the storage overhead, loading speed and visual quality are balanced. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a step flowchart of a rendering method of a static scene under a dynamic view angle provided by an embodiment of the present application;

[0023] Figure 2 is a training process schematic diagram of a hybrid neural network view angle prediction model provided by an embodiment of the present application;

[0024] Figure 3 is a generation process schematic diagram of layered baking data provided by an embodiment of the present application;

[0025] Figure 4 is a structural block diagram of a rendering device of a static scene under a dynamic view angle provided by an embodiment of the present application;

[0026] Figure 5 is a structural block diagram of an electronic device provided by an embodiment of the present application;

[0027] Figure 6 FIG. 1 is a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work fall within the scope of protection of the present application.

[0029] In the related art of pre-rendering, Unity light baking system is usually used for rendering. However, the foregoing light baking technology only supports static light, that is, only supports pre-rendering of fixed view angle, and cannot adapt to dynamic view angle change. Since dynamic view angle needs to recalculate indirect light, data needs to be reloaded when the view angle is switched, causing lag. Based on this, there is a problem of inconsistency of light and shadow between static pre-rendering and dynamic real-time rendering elements, which is easy to cause visual fragmentation, leading to unnatural fusion of dynamic and static, and it is difficult to realize rendering of static scene under dynamic view angle. In addition, the Nanite technology of Epic Games (aiming to completely change the processing method of geometric details in real-time rendering) relies on GPU (Graphics Processing Unit, graphics processor) hardware acceleration and cannot run smoothly on low-end devices. Non-discriminatory pre-rendering leads to a large amount of redundant data in invisible areas, which will occupy storage and bandwidth, causing resource waste. In addition, full-scene high-precision pre-rendering requires TB-level storage, while a low-precision solution will lead to a decrease in visual quality, causing a problem of contradiction between storage and performance.

[0030] The embodiment of the present application solves the problem of dynamic view angle adaptation by combining view angle prediction and layered pre-rendering based on the collaborative scheme of prediction, blocking and compensation based on the mixed neural network view angle prediction model based on user historical behavior data and the spatio-temporal consistency transition mechanism of optical flow compensation. Based on the proposed optical flow compensation mechanism, the technical bottleneck of dynamic and static rendering fusion is broken through, and the real-time high-quality rendering problem of static scene under dynamic view angle is solved, balancing storage overhead, loading speed and visual quality.

[0031] Reference Figure 1 FIG. 1 is a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application.

[0032] Step S101, obtaining user historical behavior data and a pre-constructed mixed neural network view angle prediction model.

[0033] In some embodiments of the present application, the view angle prediction is a prediction of a dynamic view angle, which can be implemented based on user historical behavior data and a pre-constructed hybrid neural network view angle prediction model.

[0034] The neural network structure of the hybrid neural network view angle prediction model can be a double-branch hybrid structure, and the hybrid neural network view angle prediction model can be trained based on user historical behavior data to obtain the first branch structure and the second branch structure in the double-branch hybrid structure. The first branch structure can be an LSTM (Long Short-Term Memory) network, and the second branch structure can be a TCN (Temporal Convolutional Network) network, i.e., the hybrid neural network view angle prediction model has an LSTM layer and a TCN layer hybrid network. The specific training process of the hybrid neural network view angle prediction model is not limited in the embodiments of the present application.

[0035] The user historical behavior data can refer to historical trajectory, gaze point, and operation habit of the user, and other related data. Optionally, the user historical behavior data can be in the form of a time series data sequence, which contains multiple time frames. For each frame of data, it can contain three-dimensional position coordinates, view angle direction quaternions, and operation intensity, etc. The three-dimensional position coordinates can be used to indicate the historical trajectory of the user, and the data input source can be, for example, a VR (Virtual Reality) helmet pose sensor, etc. The view angle direction quaternions can be used to indicate the gaze point of the user, and the data input source can be an eye tracking gaze point. The operation intensity can be used to indicate the operation habit of the user, such as zoom / pan speed, etc., and the data input source can be a mouse trajectory heat map.

[0036] In step S102, a view angle probability cloud map of a future preset number of frames is obtained based on the user historical behavior data and the hybrid neural network view angle prediction model.

[0037] The hybrid neural network view angle prediction model can predict the view angle probability distribution of the future preset number of frames based on the user historical behavior data.

[0038] Specifically, the view angle probability distribution of the future preset number of frames can be embodied based on the view angle probability cloud map. Optionally, the user historical behavior data can be input into the pre-constructed hybrid neural network view angle prediction model, and the view angle probability cloud map of the future preset number of frames is output, wherein the future preset number of frames can be a future preset number of frames based on the historical time frames in the time series of the user historical behavior data, such as 30 future frames.

[0039] Optionally, the three-dimensional position coordinates, the view direction quaternion and the operation intensity in each frame of data can be normalized to obtain normalized data, and then the normalized data can be processed based on the first branch structure in the double-branch hybrid structure to obtain a first output, and the normalized data can be processed based on the second branch structure in the double-branch hybrid structure to obtain a second output; at this time, the first output and the second output can be spliced in the feature dimension to obtain a feature fusion result, and the feature fusion result can be mapped to an output of a preset dimension through a fully connected layer of the double-branch hybrid structure, the output of the preset dimension can be processed using a preset activation function to obtain a view probability cloud diagram of a future preset number of frames, and then the prediction of the view probability distribution of the future preset number of frames can be realized based on the view probability cloud diagram.

[0040] In step S103, the pre-rendered static scene of the at least one target pre-rendered view is divided based on the preset dynamic semantic weight matrix to obtain layered baking data.

[0041] The view probability cloud diagram can include a hot spot area, which generally represents a high-probability spatial position of an event, and can be used to indicate at least one target pre-rendered view with an allocation probability exceeding a preset probability threshold.

[0042] In the embodiments of the present application, the view prediction can be combined with layered pre-rendering to solve the dynamic view adaptation problem.

[0043] In some embodiments of the present application, the pre-rendered static scene of the at least one target pre-rendered view can be divided based on the preset dynamic semantic weight matrix to obtain layered baking data. Specifically, the division of the pre-rendered static scene of each target pre-rendered view is to divide each pre-rendered scene into units of different levels, and the units of different levels correspond to different baking accuracies, thereby generating layered baking data, and then using the layered baking data as the basis for layered pre-rendering.

[0044] Optionally, first, the pre-rendered static scene of each target pre-rendered view can be analyzed to obtain a scene analysis result, then based on the scene analysis result, a first target pre-rendered static scene with a scene volume exceeding a preset first threshold and a dynamic object proportion less than or equal to a preset second threshold can be obtained from the pre-rendered static scene of each target pre-rendered view, and then the first target pre-rendered static scene can be divided based on the preset dynamic semantic weight matrix to obtain layered baking data.

[0045] For the use of the preset dynamic semantic weight matrix in the division of the pre-rendering static scene, the following operations can be performed: obtaining the region staticity score, the occlusion probability and the historical attention degree of at least one first target pre-rendering static scene, and then calculating the region staticity score, the occlusion probability and the historical attention degree of each first target pre-rendering static scene based on the preset dynamic semantic weight matrix to obtain the weight value of each first target pre-rendering static scene, and then dividing each first target pre-rendering static scene into units of different levels based on the weight value, and performing baking processing on the corresponding first target pre-rendering static scene by using a technology stack matched with the baking precision to generate pre-rendering metadata.

[0046] In the preset rendering data, the second target pre-rendering static scene is baked, and the layered baking data can be obtained based on the pre-rendering metadata and the preset rendering data.

[0047] In a preferred embodiment of the present application, the layered baking data can be used to generate optical flow data between the target pre-rendering perspectives, and the optical flow data can be stored in a preset optical flow mapping table.

[0048] Specifically, the baking data of each target pre-rendering perspective can be obtained from the layered baking data, and then the brightness of any target pre-rendering perspective at a preset image position at a preset first time and the brightness of the target pre-rendering perspective at the preset image position at a preset second time can be obtained based on the baking data of any target pre-rendering perspective. The preset image position, the brightness of any target pre-rendering perspective at the preset image position at a preset first time and the brightness of the target pre-rendering perspective at the preset image position at a preset second time are calculated by using a preset dense optical flow algorithm to obtain the optical flow data of a neighboring perspective adjacent to the corresponding target pre-rendering perspective.

[0049] In addition, the optical flow data of any target pre-rendering perspective and the neighboring perspective adjacent to the corresponding target pre-rendering perspective can be stored in a preset optical flow mapping table. The preset optical flow mapping table is a kind of structured data storage table used for recording optical flow information under different perspectives, which can facilitate quick retrieval and calling. For example, the calculated optical flow information can be stored in FP16 compression format, which is not limited in the embodiments of the present application.

[0050] The optical flow data represents the motion vector of the pixel points in the image between adjacent frames, specifically the optical flow displacement vector, i.e. the optical flow components in the x and y directions, reflecting the dynamic changes of the target object or scene.

[0051] Step S104, obtaining target optical flow data of the target adjacent view angle adjacent to the current view angle from the preset optical flow mapping table.

[0052] The optical flow data stored in the preset optical flow mapping table belongs to a hybrid rendering technology combining pre-computed lighting information and real-time view interpolation, and can be used for lighting consistency optimization under dynamic view.

[0053] The preset second moment and the preset first moment can be adjacent time intervals, and in actual application, it is manifested as that the optical flow data between the current view angle and the target adjacent view angle adjacent to the current view angle is calculated and stored in the preset optical flow mapping table. That is, the target optical flow data of the target adjacent view angle adjacent to the current view angle is stored in the preset optical flow mapping table.

[0054] In some embodiments of the present application, the target optical flow data of the target adjacent view angle adjacent to the current view angle can be obtained from the preset optical flow mapping table.

[0055] Step S105, when the view angle is switched, the static scene rendered in real time based on the current view angle and the static scene pre-rendered according to the pixel displacement relationship are dynamically mixed based on the target optical flow data to obtain scene mixing data.

[0056] When the current view angle is switched to the target adjacent view angle, the visual jump can be eliminated based on optical flow compensation, and the fusion of dynamic and static rendering is realized.

[0057] Optionally, the target optical flow data can be used to indicate the pixel displacement relationship between the current view angle and the target adjacent view angle. At this time, the static scene rendered in real time based on the current view angle and the static scene pre-rendered according to the pixel displacement relationship are dynamically mixed based on the target optical flow data to obtain scene mixing data, and then the target static scene is rendered based on the scene mixing data.

[0058] In some embodiments of the present application, the target optical flow data can include a target optical flow displacement vector, specifically an optical flow displacement vector, i.e. the optical flow components in x and y directions, which can reflect the dynamic change of the target object or scene.

[0059] In actual application, the view switching speed can be obtained, the mixing weight is dynamically adjusted using the view switching speed and the target optical flow displacement vector, and then the current real-time rendered static scene and the static scene pre-rendered according to the pixel displacement relationship are mixed using the mixing weight to obtain scene mixing data, so as to eliminate the visual jump by taking the scene mixing data as the basis of the target static scene under dynamic view.

[0060] In a preferred embodiment of the present application, the scene mixing data involves processing of consecutive frames. When processing consecutive frames, such as view angle movement or scene dynamic change, there is usually a large amount of redundant information between adjacent frames. The aforementioned temporal redundancy can be utilized by motion compensation compression technology to improve compression efficiency. Specifically, adjacent frame difference data of the difference region of adjacent frames in the scene mixing data can be obtained. The adjacent frame difference data can refer to redundant data between the current view angle and the target adjacent view angle adjacent to the current view angle. At this time, motion compensation can be used to compress the adjacent frame difference data to obtain compressed data. In addition, a decoding table can be pre-generated for the compressed data. The generated decoding table can be stored in the constant memory of the graphics processing unit (GPU) for rendering the target static scene. Moreover, the generated decoding table can be parallel to the decoding block data.

[0061] In step S106, the target static scene is rendered based on the scene mixing data.

[0062] The generated decoding table can be parallel to the decoding block data, and can be decoded in parallel by the GPU to obtain the scene mixing data for rendering the target static scene, thereby realizing rendering of the static scene under the dynamic view angle.

[0063] In some embodiments of the present application, in order to further understand the rendering of the static scene under the dynamic view angle provided by the embodiments of the present application by those skilled in the art, the following description is made in combination with a specific implementation process:

[0064] Referring to Figure 2 , a training process diagram of the hybrid neural network view angle prediction model provided by the embodiments of the present application is shown. As Figure 2 indicated, the hybrid neural network view angle prediction model can include a preparation input data module. The preparation input data module can be used to input the corresponding user historical behavior data based on the selection of the data input source. The data input source can include a VR helmet pose sensor, an eye tracking gaze point, a mouse trajectory heat map, etc. For example, the position / posture data, such as three-dimensional position coordinates, can be input by the VR helmet pose sensor, and the three-dimensional position coordinates can be used to indicate the historical trajectory of the user. The view angle direction quaternion can be input by the eye tracking gaze point, and the view angle direction quaternion can be used to indicate the gaze point of the user, i.e., the eye tracking focus point. The operation intensity can be input by the mouse trajectory heat map / mouse movement heat map, and the operation intensity can be used to indicate the operation habit of the user, such as zooming / panning speed, etc.

[0065] The hybrid neural network perspective prediction model can have a dual-branch hybrid structure, with the first branch being an LSTM and the second branch being a TCN. In other words, the hybrid neural network perspective prediction model has a hybrid network of LSTM and TCN layers. In this case, the hybrid network of LSTM and TCN layers can be trained based on the user's historical behavior data input from the input data module, enabling the trained hybrid neural network perspective prediction model to predict the perspective probability distribution for a preset number of future frames.

[0066] Optionally, user historical behavior data can be input in the form of a time-series data sequence, which contains n time frames. For example, the time-series data sequence can be A={a1,a2,...,a...} n}, where a1, a2, ..., a n These are the frame data for each time frame. Each frame data may include three-dimensional position coordinates (x, y, z), view direction quaternions (qw, qx, qy, qz), and operation intensity such as zoom / translation speed.

[0067] The training process of the hybrid neural network view prediction model can be represented by normalizing the three-dimensional position coordinates (x, y, z), view direction quaternions (qw, qx, qy, qz), and operation intensity such as scaling / translation speed in each frame of data. For example, after standardizing the aforementioned frame data, it is normalized to the range of [-1, 1] to obtain normalized data. Then, the LSTM branch and TCN branch in the dual-branch hybrid structure can be trained based on the normalized data. The first output of the first branch structure and the second output of the second branch structure are concatenated in the feature dimension to obtain the feature fusion result. Then, the feature fusion result is mapped to the output of the preset dimension through the fully connected layer of the dual-branch hybrid structure, and the output of the preset dimension is processed using a preset activation function to obtain the view probability cloud map, so as to train the hybrid neural network view prediction model.

[0068] For example, the pseudocode implementation of training a view prediction model can be shown below:

[0069]

[0070] The LSTM branch can be used to capture long-term time dependencies, and its configuration can use a single-layer LSTM with 128 hidden units and return_sequences=True to retain the output of all time steps, which can be represented as: lstm_layer = LSTM(units = 128, return_sequences = True); the TCN branch can use dilated convolution (DilatedConvolution) to expand the receptive field and efficiently capture multi-scale temporal patterns, assuming that it has a configuration of 64 filter numbers, a convolution kernel size of 3, and dilations of [1, 2, 4], i.e., forming three layers of dilated convolution, covering dependencies of 1, 2, and 4 steps, respectively, which can be represented as: tcn_layer = TCN(n_filters = 64, kernel_size = 3, dilations = [1, 2, 4]); for the fusion layer, the outputs of the LSTM and TCN branches can be concatenated in the feature dimension for feature fusion, and then mapped to a 30-dimensional output through a dense layer (Dense) and activated by a softmax function, and finally output the perspective probability distribution of the next 30 frames.

[0071] The mixed neural network perspective prediction model can predict the perspective probability distribution of a future preset number of frames based on user historical behavior data. That is, after training the mixed neural network perspective prediction model, the user historical behavior data can be input into the mixed neural network perspective prediction model to output the perspective probability cloud map of the future preset number of frames.

[0072] The perspective probability cloud map can include a hot spot area, which is used to indicate at least one target pre-rendering perspective with an allocation probability exceeding a preset probability threshold. At this time, the pre-rendering static scene of the at least one target pre-rendering perspective can be divided based on a preset dynamic semantic weight matrix to obtain layered baking data, which is the basis for layered pre-rendering and can combine perspective prediction with layered pre-rendering to solve the dynamic perspective adaptation problem.

[0073] Specifically, referring to Figure 3 , a generation process diagram of layered baking data provided by an embodiment of the present application is shown. The division of the pre-rendering static scene of each target pre-rendering perspective is manifested as dividing each pre-rendering scene into different levels of units, different levels of units corresponding to different baking accuracy, thereby generating layered baking data, and further taking layered baking data as the basis for layered pre-rendering.

[0074] In some embodiments of this application, firstly, scene analysis can be performed on the pre-rendered static scenes of each target pre-rendered viewpoint. For the first target pre-rendered static scene whose scene volume exceeds a preset first threshold and whose dynamic object ratio is less than or equal to a preset second threshold, the first target pre-rendered static scene can be divided based on a preset dynamic semantic weight matrix. For the second target pre-rendered static scene whose scene volume is greater than the preset first threshold or whose dynamic object ratio is greater than the preset second threshold, no pre-rendered scene division operation is performed.

[0075] Optionally, the partitioning of the pre-rendered static scene can be achieved by performing octree partitioning. For example, assume a preset first threshold of 100m. 3 The second threshold is preset to 70%, such as Figure 3 As shown, for scene volumes ≥ 100m 3 For pre-rendered static scenes where dynamic objects account for ≤70%, spatial octree partitioning can be achieved by calculating the dynamic semantic weight matrix through octree segmentation; for scenes with a volume <100m 3 Alternatively, for pre-rendered static scenes where dynamic objects account for more than 70%, skip the segmentation.

[0076] For example, the pseudocode for the algorithm logic of calculating the dynamic semantic weight matrix can be shown below:

[0077]

[0078] The specific formula for the preset dynamic semantic weight matrix can be shown below:

[0079]

[0080] Where x, y, z are the 3D spatial coordinates of a point in a pre-rendered static scene for a first target, and W(x, y, z) is the weight value of a pre-rendered static scene for a first target. Score the staticity of a region in a pre-rendered static scene for a given primary objective. Pre-render the occlusion probability of a static scene for a given primary target. The historical attention of a static scene is pre-rendered for a certain primary objective, with α, β, and γ representing the weight percentages of each semantic element.

[0081] For example, the regional staticity score can be calculated based on the movement variance of statistical scene elements in historical frames, and the specific calculation formula is as follows:

[0082]

[0083] Among them, S static(i) is a normalized static score of scene element i in a certain first target pre-rendered static scene, with a value range of [0, 1]; Var(x i ,y i ,z i ) represents the position variance of scene element i in a certain first target pre-rendered static scene in the historical frame; is the maximum position variance value in a certain first target pre-rendered static scene.

[0084] The occlusion probability can be obtained by detecting the region visibility using the ray casting method, and the pseudo code of the algorithm logic can be as follows:

[0085]

[0086] Exemplarily, a ray can be emitted from multiple perspectives to the region of a certain first target pre-rendered static scene, and the proportion of the number of times the ray hits the region (i.e., the number of times not occluded) to the total number of rays can be the visibility; in the weight calculation, generally, the weight of the region with high occlusion probability is low, and the occlusion probability S occlusion may be: occlusion probability = (1-visibility).

[0087] The historical attention can be obtained based on the user historical operation heat map Gaussian blur, and the specific calculation formula can be as follows:

[0088]

[0089] Wherein, x, y, z are the three-dimensional space point coordinates of a certain first target pre-rendered static scene; S attention (x, y, z) is the historical attention of the three-dimensional space point x, y, z of a certain first target pre-rendered static scene; (x t ,y t ) is the user operation position, that is, the gaze point; t is the time sequence identifier of the historical operation record; T is the total number of historical operation points considered; σ is the spatial influence radius; w(t) is the time decay weight, which can be based on the decay coefficient of time near and far, to ensure that the weight of recent operation points is higher.

[0090] After calculating the weight values of each first target pre-rendered static scene, weight value judgment can be performed, exemplarily, each pre-rendered scene can be divided into L0-L3 four-level units based on each weight value. For example, if W>0.7, the corresponding first target pre-rendered static scene can be divided into L0 level; if 0.4≤W≤0.7, the corresponding first target pre-rendered static scene can be divided into L1 level; if W<0.4, the corresponding first target pre-rendered static scene can be divided into L3 level.

[0091] Different levels of units correspond to different baking precision, which can be as shown in the following table 1:

[0092] Table 1 Baking precision of different levels of units

[0093]

[0094] Optionally, a technology stack matching the baking precision can be used to perform baking processing on the corresponding first target pre-rendering static scene. For example, for L0 level baking, there is a 4K HDR full element baking precision requirement, and ray tracing global illumination can be used for baking processing; for L1 level baking, there is a 2K PBR material and reflection baking precision requirement, and spherical harmonic spectrum and emission probe can be used for baking processing; for L3 level baking, there is a 512px depth contour baking precision requirement, and depth buffer and stencil buffer can be used for baking processing. At this time, data compression storage operation can be performed on the data obtained by baking processing to generate pre-rendering metadata.

[0095] For a pre-rendering static scene with a scene volume <100m 3 , octree partitioning thereof is skipped, and a full scene L0 baking precision matching technology stack can be used for baking processing; for a pre-rendering static scene with a dynamic object ratio >70%, octree partitioning thereof is skipped, and a dynamic optimization mode can be used for baking processing to obtain preset rendering data.

[0096] The finally output layered baking data can include pre-rendering metadata generated after baking processing based on the first target pre-rendering static scene, and preset rendering data obtained after baking processing based on the second target pre-rendering static scene. It should be noted that the layered baking data has a layered structure, which can realize spatial or frequency domain layered storage of scene lighting information, and the embodiments of the present application do not limit this.

[0097] In some embodiments of the present application, the optical flow compensation mechanism can break through the technical bottleneck of dynamic and static rendering fusion.

[0098] The optical flow compensation mechanism is manifested as optical flow calculation and interpolation mixing calculation, so that the subsequent rendering of the static scene can achieve the effect of spatiotemporal consistency compensation. The pseudo code of the algorithm logic can be as follows:

[0099]

[0100] Optionally, the layered baking data can be used to generate optical flow data between the respective target pre-rendered view, and the optical flow data can be stored in a preset optical flow mapping table. Wherein, the preset optical flow mapping table can store the target optical flow data of the current view and the target adjacent view adjacent to the current view, and the target optical flow data can be used to indicate the pixel displacement relationship between the current view and the target adjacent view.

[0101] The optical flow data represents the motion vector of the pixel points in the image between adjacent frames, specifically the optical flow displacement vector, i.e. the optical flow components in x and y directions, reflecting the dynamic changes of the target object or scene.

[0102] For example, the Farneback dense optical flow algorithm can be used to calculate the optical flow displacement vector of the adjacent view, and the specific calculation formula of the optical flow calculation can be as follows:

[0103]

[0104] Wherein, I(x,y,t) represents the brightness of the image position (x,y) at time t; I(x+u,y+v,t+1) represents the brightness of the image position (x+u,y+v) at time t+1; x represents the spatial gradient of the image in the horizontal axis direction, y represents the spatial gradient of the image in the vertical axis direction; t represents the time gradient of the image in the time dimension; u and v are the optical flow components in the horizontal and vertical directions, i.e. the displacement to be solved, and then the optical flow displacement vector of the adjacent view can be obtained based on the foregoing displacement, which can be specifically represented by (u,v).

[0105] When the current view is switched to the target adjacent view, the visual jump can be eliminated based on the optical flow compensation to realize the fusion of dynamic and static rendering. Optionally, the static scene of the current real-time rendering and the static scene pre-rendered according to the pixel displacement relationship can be mixed by using a mixing weight to obtain scene mixing data for rendering the target static scene, and the dynamic mixing can be realized by using a transition interpolation algorithm to calculate the interpolation weight.

[0106] For example, the specific calculation formula of the interpolation weight calculation can be as follows:

[0107]

[0108] Wherein, is the mixing weight; λ is the attenuation coefficient, usually 0.1; (u,v) is the optical flow displacement vector; Δt is the view switching time interval, which can be used to define the view switching speed. That is, the mixing weight can be dynamically adjusted according to the view switching speed and the optical flow displacement vector (u,v) .

[0109] Optionally, the scene mixing data involves processing of consecutive frames, at which point block encoding and GPU decoding optimization can be performed.

[0110] Block encoding, which can specifically be manifested as dividing the texture in the scene mixing data into 8x8 blocks, and then performing DCT transformation, quantization, Zigzag scanning and Huffman coding on each block. It should be noted that texture blocking is divided according to the principle of hardware architecture, and compared with the evaluation of the SIMD (Single Instruction, Multiple Data Register) register capacity of the CPU (Central Processing Unit) and GPU, 8x8 blocks are the optimal example, and the embodiments of the present application do not limit this.

[0111] GPU decoding optimization, which can specifically be manifested as using motion compensation to compress the adjacent frame difference data to obtain compressed data, and a decoding table can be pre-generated for the compressed data, the generated decoding table can be stored in the constant memory of the graphics processor GPU for rendering the target static scene, and the generated decoding table is parallel with the decoding block data.

[0112] Exemplarily, the pseudo code of the algorithm logic can be as follows:

[0113]

[0114] It should be noted that CPU decoding will become a performance bottleneck (4K texture decoding requires 10-20 ms) in high-resolution texture scenes, and the parallel decoding of the GPU in the embodiments of the present application can reduce the time consumption to within 1 ms, which can solve the real-time decoding bottleneck of texture compression data on the GPU end.

[0115] Optionally, the rendering scheme provided in the embodiments of the present application can be applied to any application scenario of static scene rendering under dynamic viewing angle. Taking the smart city digital twin platform as an example, assuming that the hardware environment is NVIDIA RTX 4090, 64GB RAM.

[0116] For the smart city digital twin platform, the input data can exemplarily be manifested as input of city BIM (Building Information Modeling Model, building information model, such as containing buildings, roads, vegetation, etc.), dynamic traffic flow data, etc. Before inputting the foregoing data, the foregoing data can also be preprocessed, such as using a Simplygon simplified model to reduce the number of polygons by 80%, etc., and the embodiments of the present application do not limit this.

[0117] In the scenario of applying the hybrid neural network perspective prediction model to the smart city digital twin platform, the user historical behavior data can be represented as the collection of 100,000 user operation records. The collected user operation records can be used to train the hybrid neural network, and the prediction accuracy reaches 92%. Then, the hybrid neural network perspective prediction model can be used to output the perspective probability cloud map in the next 1 second in real time. The perspective probability cloud map includes hot spot areas, which can also be referred to as a perspective heat map. For the core area in the perspective heat map, which is usually the red area of the heat map, an L0 level full-precision baking can be triggered to generate a ray tracing light map. For the edge area in the perspective heat map, which is usually the blue area of the heat map, an L3 level depth contour map can be generated. Subsequently, the aforementioned generated layered baking data can be used to realize real-time fusion rendering.

[0118] Real-time fusion rendering can be represented as dynamically loading pre-baked blocks to the video memory. In the scenario of the smart city digital twin, real-time shadow mapping can be performed on dynamic objects such as vehicles and pedestrians, and multi-channel synthesis can be performed. When the perspective is switched, the static scene rendered in the current perspective in real time and the pre-rendered static scene can be dynamically mixed. The target static scene is rendered based on the scene mixing data, and the rendering of the static scene under the dynamic perspective in the smart city digital twin platform is realized.

[0119] It should be noted that for the specific rendering process, the content of the method embodiment can be referred to. The present embodiment does not add to the rendering process. The rendering of the static scene under the dynamic perspective in the smart city digital twin platform has a frame time, a video memory occupation, and a transition error that is superior to mainstream solutions such as the Unity baking system and the Nanite technology. The real-time high-quality rendering problem of the static scene under the dynamic perspective can be solved, and the storage overhead, the loading speed, and the visual quality are balanced.

[0120] In the embodiment of the present application, by obtaining the constructed mixed neural network view prediction model, the view probability cloud map for a future preset frame number is obtained according to the user historical behavior data and the mixed neural network view prediction model, wherein the obtained view probability cloud map contains a hot area, and the hot area can be used to indicate at least one target pre-rendering view whose allocation probability exceeds a preset probability threshold. Then, the pre-rendering static scene of the at least one target pre-rendering view can be divided based on a preset dynamic semantic weight matrix to obtain layered baking data, and the layered baking data can be used to generate light flow data between the target pre-rendering views, and the generated light flow data can be stored in a preset light flow mapping table. The target light flow data of a target adjacent view adjacent to the current view is obtained from the preset light flow mapping table, and the target light flow data indicates the pixel displacement relationship between the current view and the target adjacent view. When the view switching between the current view and the target adjacent view is performed, the static scene rendered in real time based on the target light flow data and the static scene pre-rendered according to the pixel displacement relationship can be dynamically mixed to eliminate visual jump, so as to obtain scene mixing data, and then the target static scene is rendered based on the scene mixing data, thereby realizing the rendering of the static scene under the dynamic view. Through the collaborative scheme of view prediction, layered pre-rendering and light flow compensation, the view prediction is combined with the layered pre-rendering to solve the dynamic view adaptation problem, and the light flow compensation is used to eliminate the visual jump, thereby realizing the fusion of dynamic and static rendering, solving the real-time high-quality rendering problem of the static scene under the dynamic view, and balancing the storage overhead, loading speed and visual quality.

[0121] It should be noted that, for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the action order described, because according to the embodiments of the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the present application.

[0122] Referring to Figure 4 , a structural block diagram of a rendering device for a static scene under a dynamic view is shown, which can specifically include the following modules:

[0123] The view prediction model acquisition module 401 is configured to acquire user historical behavior data and a pre-constructed mixed neural network view prediction model.

[0124] The view probability cloud map generation module 402 is configured to obtain a view probability cloud map for a future preset frame number according to the user historical behavior data and the mixed neural network view prediction model. The view probability cloud map contains a hot area, and the hot area is used to indicate at least one target pre-rendering view whose allocation probability exceeds a preset probability threshold.

[0125] The static scene division module 403 is configured to divide the pre-rendered static scene of at least one target pre-rendered view based on a preset dynamic semantic weight matrix, to obtain layered baking data; the layered baking data is used to generate optical flow data between each target pre-rendered view, and the optical flow data is stored in a preset optical flow mapping table.

[0126] The optical flow data acquisition module 404 is configured to acquire target optical flow data of a target adjacent view adjacent to the current view from the preset optical flow mapping table; the target optical flow data is used to indicate a pixel displacement relationship between the current view and the target adjacent view.

[0127] The scene mixing module 405 is configured to dynamically mix the static scene rendered in real time based on the current view and the static scene pre-rendered according to the pixel displacement relationship based on the target optical flow data when the view is switched, to obtain scene mixing data.

[0128] The target static scene rendering module 406 is configured to render the target static scene based on the scene mixing data.

[0129] In some embodiments of the present application, the neural network structure of the mixed neural network view prediction model is a double-branch mixed structure; the user historical behavior data includes a time series data sequence, and the time series data sequence contains multiple time frames, wherein each frame of data contains a three-dimensional position coordinate, a view direction quaternion and an operation intensity; the view probability cloud map generation module 402 can include the following sub-modules:

[0130] The view probability cloud map generation sub-module is configured to normalize the three-dimensional position coordinate, the view direction quaternion and the operation intensity in each frame of data to obtain normalized data; perform data processing on the normalized data based on a first branch structure in the double-branch mixed structure to obtain a first output; and perform data processing on the normalized data based on a second branch structure in the double-branch mixed structure to obtain a second output; splice the first output and the second output in the feature dimension to obtain a feature fusion result; map the feature fusion result to an output of a preset dimension through a fully connected layer of the double-branch mixed structure, and process the output of the preset dimension using a preset activation function to obtain a view probability cloud map of a future preset number of frames.

[0131] In some embodiments of the present application, the static scene division module 403 can include the following sub-modules:

[0132] The hierarchical baking data generation submodule is configured to perform scene analysis on the pre-rendered static scene of each target pre-rendering view angle to obtain a scene analysis result; based on the scene analysis result, a first target pre-rendered static scene in which the scene volume exceeds a preset first threshold and the dynamic object proportion is less than or equal to a preset second threshold is obtained from the pre-rendered static scene of each target pre-rendering view angle; and the first target pre-rendered static scene is divided based on a preset dynamic semantic weight matrix to obtain hierarchical baking data.

[0133] In some embodiments of the present application, the hierarchical baking data generation submodule can include the following units:

[0134] The hierarchical baking data generation unit is configured to obtain the region staticity score, the occlusion probability and the historical attention degree of at least one first target pre-rendered static scene; based on the preset dynamic semantic weight matrix, the region staticity score, the occlusion probability and the historical attention degree of each first target pre-rendered static scene are calculated respectively to obtain a weight value of each first target pre-rendered static scene; each first target pre-rendered static scene is divided into units of different levels based on the weight value; the units of different levels correspond to different baking accuracies; a technical stack matching the baking accuracy is used to perform baking processing on the corresponding first target pre-rendered static scene to generate pre-rendering metadata; and the hierarchical baking data is obtained based on the pre-rendering metadata and preset rendering data; wherein the preset rendering data is obtained by baking processing based on a second target pre-rendered static scene, and the second target pre-rendered static scene is a pre-rendered static scene in which the scene volume is greater than the preset first threshold or the dynamic object proportion is greater than the preset second threshold.

[0135] In some embodiments of the present application, after obtaining the hierarchical baking data, the device provided by the embodiments of the present application can further include the following modules:

[0136] The optical flow data storage module is configured to obtain the baking data of each target pre-rendering view angle from the hierarchical baking data; based on the baking data of any target pre-rendering view angle, the brightness of the preset image position of any target pre-rendering view angle at a preset first time and the brightness of the preset image position at a preset second time are obtained; the preset second time and the preset first time are adjacent time intervals; a preset dense optical flow algorithm is used to calculate the brightness of the preset image position, the brightness of the preset image position of any target pre-rendering view angle at the preset first time and the brightness of the preset image position at the preset second time to obtain the optical flow data of the adjacent view angle adjacent to the corresponding target pre-rendering view angle; and the optical flow data of any target pre-rendering view angle and the adjacent view angle adjacent to the corresponding target pre-rendering view angle are stored in a preset optical flow mapping table.

[0137] In some embodiments of the present application, the target optical flow data includes a target optical flow displacement vector; and the scene mixing module 405 can include the following submodules:

[0138] The scene mixing sub-module is configured to obtain a view switching speed, dynamically adjust a mixing weight according to the view switching speed and a target optical flow displacement vector, and mix a current real-time rendered static scene and a static scene pre-rendered according to a pixel displacement relationship according to the mixing weight to obtain scene mixing data.

[0139] In some embodiments of the present application, after obtaining the scene mixing data, the apparatus provided by the embodiments of the present application can further include the following modules:

[0140] The decoding optimization module is configured to obtain adjacent frame difference data of an adjacent frame difference region in the scene mixing data, compress the adjacent frame difference data by using motion compensation to obtain compressed data, and pre-generate a decoding table for the compressed data, and store the decoding table in a constant memory of a graphics processing unit for rendering a target static scene.

[0141] In the embodiments of the present application, by obtaining the constructed mixed neural network view prediction model, the view probability cloud map of a future preset frame number is obtained according to the user historical behavior data and the mixed neural network view prediction model, wherein the obtained view probability cloud map contains a hot spot region, and the hot spot region can be mainly used to indicate at least one target pre-rendering view angle whose allocation probability exceeds a preset probability threshold. Then, the pre-rendering static scene of the at least one target pre-rendering view angle can be divided based on a preset dynamic semantic weight matrix to obtain layered baking data, and the layered baking data can be used to generate optical flow data between the target pre-rendering view angles, and the generated optical flow data can be stored in a preset optical flow mapping table. At this time, the target optical flow data of a target adjacent view angle adjacent to the current view angle can be obtained from the preset optical flow mapping table, and the target optical flow data indicates the pixel displacement relationship between the current view angle and the target adjacent view angle. When the view angle switching between the current view angle and the target adjacent view angle is performed, the static scene rendered in real time according to the current view angle and the static scene pre-rendered according to the pixel displacement relationship can be dynamically mixed based on the target optical flow data to eliminate visual jump, and scene mixing data is obtained, and then the target static scene is rendered based on the scene mixing data, so as to realize the rendering of the static scene under the dynamic view angle. Through the collaborative scheme of view prediction, layered pre-rendering and optical flow compensation, the view prediction is combined with the layered pre-rendering to solve the dynamic view angle adaptation problem, and the optical flow compensation is used to eliminate the visual jump, so as to realize the fusion of dynamic and static rendering, solve the real-time high-quality rendering problem of the static scene under the dynamic view angle, and balance the storage overhead, loading speed and visual quality.

[0142] For the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the related parts are described in the part of the method embodiments.

[0143] The embodiments of the present application also provide an electronic device, which refers to Figure 5The electronic device 500 provided in the embodiment of the present application includes a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and capable of running on the processor 520. The computer program 511, when executed by the processor, implements the processes of the embodiment of the method for rendering a static scene under a dynamic view angle described above and achieves the same technical effects. To avoid repetition, details are not described herein.

[0144] The embodiment of the present application also provides a computer readable storage medium. Referring to Figure 6 The computer readable storage medium 600 provided in the embodiment of the present application stores the computer program 511. The computer program 511, when executed by the processor, implements the processes of the embodiment of the method for rendering a static scene under a dynamic view angle described above and achieves the same technical effects. To avoid repetition, details are not described herein.

[0145] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0146] It should be noted that the terms "first", "second", and the like in the specification and claims of the embodiments of the present application are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or modules does not necessarily have to be limited to those steps or modules clearly listed, but can include other steps or modules not clearly listed or inherent to these processes, methods, products or devices. The division of modules in the embodiments of the present application is only a logical division, and in actual application, another division mode can be used, for example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, and the indirect coupling or communication connection between the modules can be electrical or other similar forms, which are not limited in the embodiments of the present application. In addition, the modules or sub-modules described as separate components can be or can not be physically separated, and can be or can not be physical modules, or can be distributed to a plurality of circuit modules, and some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0147] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and module described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0149] In several embodiments provided in the embodiments of the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are merely schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0150] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, that is, can be located in one place, or can be distributed to a plurality of network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0151] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can be stored in a computer readable storage medium.

[0152] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of a computer program product in whole or in part.

[0153] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0154] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes; these computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process.Figure 1 one or more processes and / or functions specified in one or more blocks Figure 1 one or more blocks or any combination thereof.

[0156] Although the preferred embodiments of the present application have been described, those skilled in the art who understand the basic inventive concept can make additional changes and modifications to the embodiments. Therefore, the appended claims are intended to include the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0157] Finally, it should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant national and regional laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0158] The above describes the technical solutions provided by the embodiments of the present application in detail. The principles and implementation manners of the embodiments of the present application are described by applying specific examples. The above description of the embodiments is only for helping to understand the method and its core idea of the embodiments of the present application; at the same time, for those skilled in the art, according to the idea of the embodiments of the present application, the specific implementation manner and application range will be changed; in summary, the content of the specification should not be understood as a limitation of the embodiments of the present application.

Claims

1. A method for rendering a static scene from a dynamic perspective, characterized in that, The method includes: Acquire user historical behavior data and a pre-built hybrid neural network perspective prediction model; Based on the user's historical behavior data and the hybrid neural network perspective prediction model, a perspective probability cloud map is obtained for a future preset number of frames; the perspective probability cloud map includes hotspot areas, which are used to indicate at least one target pre-rendered perspective with an allocation probability exceeding a preset probability threshold. The pre-rendered static scene of the at least one target pre-rendered view is divided based on a preset dynamic semantic weight matrix to obtain layered baking data; the layered baking data is used to generate optical flow data between each target pre-rendered view, and the optical flow data is stored in a preset optical flow mapping table. Target optical flow data of the target's adjacent viewpoints adjacent to the current viewpoint are obtained from the preset optical flow map table; the target optical flow data is used to indicate the pixel displacement relationship between the current viewpoint and the target's adjacent viewpoints; When switching viewpoints, the static scene rendered in real-time from the current viewpoint and the static scene pre-rendered according to the pixel displacement relationship are dynamically mixed based on the target optical flow data to obtain scene mixing data; wherein, the target optical flow data includes a target optical flow displacement vector; the specific steps include: obtaining the viewpoint switching speed, dynamically adjusting the mixing weight using the viewpoint switching speed and the target optical flow displacement vector; and mixing the static scene rendered in real-time and the static scene pre-rendered according to the pixel displacement relationship using the mixing weight to obtain scene mixing data; The target static scene is obtained by rendering the mixed data of the scene.

2. The method according to claim 1, characterized in that, The neural network structure of the hybrid neural network perspective prediction model is a two-branch hybrid structure; the user historical behavior data includes a time-series data sequence, which contains multiple time frames, wherein each frame data includes three-dimensional position coordinates, perspective direction quaternions, and operation intensity. The step of obtaining a view probability cloud map for a preset number of future frames based on the user's historical behavior data and the hybrid neural network view prediction model includes: The three-dimensional position coordinates, the viewpoint direction quaternion, and the operation intensity in each frame of data are normalized to obtain normalized data; The normalized data is processed based on the first branch structure in the dual-branch hybrid structure to obtain a first output; and the normalized data is processed based on the second branch structure in the dual-branch hybrid structure to obtain a second output. The first output and the second output are concatenated along the feature dimension to obtain the feature fusion result; The feature fusion result is mapped to a preset dimension output through the fully connected layer of the dual-branch hybrid structure, and the preset dimension output is processed using a preset activation function to obtain a view probability cloud map for a preset number of future frames.

3. The method according to claim 1, characterized in that, The process of dividing the pre-rendered static scene of the at least one target pre-rendered viewpoint based on a preset dynamic semantic weight matrix to obtain layered baking data includes: Scene analysis is performed on the pre-rendered static scenes from various target pre-rendered perspectives to obtain scene analysis results; Based on the scene analysis results, the first target pre-rendered static scene is obtained in each target pre-rendered viewpoint where the scene volume exceeds a preset first threshold and the proportion of dynamic objects is less than or equal to a preset second threshold. The first target pre-rendered static scene is divided based on a preset dynamic semantic weight matrix to obtain layered baking data.

4. The method according to claim 3, characterized in that, The step of dividing the first target pre-rendered static scene based on a preset dynamic semantic weight matrix to obtain layered baking data includes: Obtain at least one region staticity score, occlusion probability, and historical attention of the first target pre-rendered static scene; Based on the preset dynamic semantic weight matrix, the regional staticity score, occlusion probability and historical attention of each first target pre-rendered static scene are calculated to obtain the weight value of each first target pre-rendered static scene. Based on various weight values, each first target pre-rendered static scene is divided into units of different levels; different levels of units correspond to different baking precisions. A technology stack that matches the baking precision is used to bake the corresponding first target pre-rendered static scene to generate pre-rendered metadata. Based on the pre-rendered metadata and preset rendering data, layered baking data is obtained; wherein, the preset rendering data is obtained by baking a second target pre-rendered static scene, and the second target pre-rendered static scene is a pre-rendered static scene whose scene volume is greater than the preset first threshold, or whose dynamic object ratio is greater than the preset second threshold.

5. The method according to claim 1, characterized in that, After obtaining the layered baking data, the method further includes: From the layered baking data, obtain the baking data for each target pre-rendered viewpoint; Based on the baking data of any target pre-rendered viewpoint, obtain the brightness of the target pre-rendered viewpoint at a preset image position at a preset first time and the brightness of the preset image position at a preset second time; the preset second time and the preset first time are adjacent time intervals; A preset dense optical flow algorithm is used to calculate the brightness of the preset image position, the brightness of any target pre-rendered viewpoint at the preset image position at a preset first time, and the brightness of the preset image position at a preset second time, to obtain the optical flow data of the adjacent viewpoints adjacent to the corresponding target pre-rendered viewpoints. The optical flow data of any target pre-rendered view and the adjacent view adjacent to the corresponding target pre-rendered view are stored in the preset optical flow mapping table.

6. The method according to claim 1, characterized in that, After obtaining the scene mixing data, the method further includes: Obtain adjacent frame difference data in the adjacent frame difference region of the scene mixed data; The difference data between adjacent frames is compressed using motion compensation to obtain compressed data; A decoding table is pre-generated for the compressed data; the decoding table is stored in the constant memory of the graphics processor for rendering the target static scene.

7. A rendering device for a static scene from a dynamic perspective, characterized in that, The device includes: The perspective prediction model acquisition module is used to acquire user historical behavior data and a pre-built hybrid neural network perspective prediction model. The view probability cloud map generation module is used to obtain a view probability cloud map for a future preset number of frames based on the user's historical behavior data and the hybrid neural network view prediction model; the view probability cloud map includes hotspot areas, which are used to indicate at least one target pre-rendered view with an allocation probability exceeding a preset probability threshold. The static scene segmentation module is used to segment the pre-rendered static scene of the at least one target pre-rendered viewpoint based on a preset dynamic semantic weight matrix to obtain layered baking data; the layered baking data is used to generate optical flow data between each target pre-rendered viewpoint, and the optical flow data is stored in a preset optical flow mapping table; An optical flow data acquisition module is used to acquire target optical flow data of a target adjacent viewpoint adjacent to the current viewpoint from the preset optical flow mapping table; the target optical flow data is used to indicate the pixel displacement relationship between the current viewpoint and the target adjacent viewpoint. A scene blending module is used to dynamically blend a static scene rendered in real-time from the current viewpoint and a static scene pre-rendered according to the pixel displacement relationship based on the target optical flow data when switching viewpoints, to obtain scene blending data; wherein, the target optical flow data includes a target optical flow displacement vector; the specific steps include obtaining the viewpoint switching speed, dynamically adjusting the blending weights using the viewpoint switching speed and the target optical flow displacement vectors; and blending the static scene rendered in real-time and the static scene pre-rendered according to the pixel displacement relationship using the blending weights to obtain scene blending data; The target static scene rendering module is used to render the target static scene based on the scene mixed data.

8. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the rendering method for a static scene under a dynamic perspective as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the rendering method for a static scene under a dynamic perspective as described in any one of claims 1 to 6.

Images