Special effect real-time rendering method and system based on deep learning

Abstract

This invention provides a deep learning-based real-time special effects rendering method and system, relating to the field of computer science. The method includes analyzing the temporal motion characteristics and shading complexity of the special effects source data, generating a comprehensive load map and spatially partitioning the scene into core and edge regions. The core region is rendered with full precision, while the edge regions are rendered with reduced precision and compensated for prediction bias. Finally, the final rendering result is synthesized. This method effectively reduces the computational load while maintaining rendering quality, meeting the requirements of high frame rate real-time rendering.

Description

Technical Field

[0001] This invention relates to computer technology, and more particularly to a method and system for real-time rendering of special effects based on deep learning. Background Technology

[0002] In the field of computer graphics, real-time special effects rendering, especially effects combining complex physical simulations and lighting effects, places extremely high demands on computing resources. In existing technologies, to achieve real-time rendering with limited hardware resources, the conventional approach is to use a unified rendering pipeline to process the entire special effects scene. Specifically, the system acquires the source data of the special effects and performs complete shading and lighting calculations for each frame of the entire scene according to a preset, fixed rendering precision level. To handle dynamic scenes, some solutions introduce motion estimation based on historical frames to assist rendering, but such estimations are often limited to simple optical flow or block matching techniques, aiming to reuse some computational results to reduce overhead. Another common optimization strategy is to use spatial adaptive sampling, such as allocating different computing resources to different regions based on the geometric or textural complexity of the scene. However, the adaptive decision-making logic of these methods is usually based on static, pre-analyzed scene attributes, such as triangle density or texture resolution, and maintains a relatively fixed partition during the rendering process.

[0003] These conventional approaches have significant drawbacks. A major one is that their adaptive strategies lack precise perception and response to the dynamic characteristics of special effects. Special effects, especially simulated effects such as particles and fluids, exhibit drastic changes in motion patterns and visual importance across the spatiotemporal dimensions. Existing load partitioning methods based on static attributes or simple motion estimation struggle to capture the core complexity of these dynamic changes accurately and in real time. This leads to inappropriate allocation of computational resources, either wasting valuable resources in visually insensitive or slow-moving areas, or forcing quality reduction in visually critical areas with intense motion due to resource shortages, ultimately affecting the stability and visual fidelity of the overall rendering effect. Another drawback is the rigidity of the strategy for balancing rendering quality and computational overhead. When implementing reduced-precision rendering, the compensation or post-processing mechanisms of existing methods are often disconnected from the front-end load partitioning decisions, failing to form a closed-loop optimization guided by the final visual output quality. This easily introduces visible rendering flaws in dynamic scenes, such as ghosting, loss of detail, or discontinuous boundaries, especially when motion prediction is inaccurate or the scene changes abruptly. Summary of the Invention

[0004] This invention provides a method and system for real-time rendering of special effects based on deep learning, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a real-time special effects rendering method based on deep learning, comprising:

[0006] Obtain the special effects source data and frame rate constraint parameters of the scene to be rendered;

[0007] Time series analysis is performed on the special effects source data to extract motion feature vectors, a predicted position sequence is generated based on the motion feature vectors, and the spatial gradient divergence of the motion feature vectors is calculated to generate a motion complexity distribution map.

[0008] The coloring complexity of the special effects source data is estimated to generate an initial load map. The motion complexity distribution map is coupled with the initial load map to generate a comprehensive load map.

[0009] The load threshold is determined based on the comprehensive load graph and frame rate constraint parameters, and the comprehensive load is then applied accordingly. Figure 2 Value-based generation of spatial partitioning mask, and separation of special effects source data into core region data and edge region data based on spatial partitioning mask;

[0010] Perform full-precision rendering on the core area data to generate core rendering results, and perform reduced-precision rendering on the edge area data to generate edge rendering results;

[0011] The predicted position sequence is mapped to the core rendering result to extract the rendering value. The difference between the rendering value and the predicted value of the predicted position sequence is calculated to generate a prediction bias field. The edge rendering result is compensated based on the prediction bias field to generate a compensated edge rendering result.

[0012] Combine the core rendering results with the compensated edge rendering results to output the final rendering result.

[0013] The process involves time-series analysis of the special effects source data to extract motion feature vectors, generating predicted position sequences based on these vectors, and calculating the spatial gradient divergence of the motion feature vectors to generate a motion complexity distribution map.

[0014] A time-series sliding window is constructed to extract motion feature sequences from the special effects source data. The time difference of the motion feature sequences is calculated to generate motion change rate features. The motion change rate features are used as weights to weight the motion feature sequences to generate motion feature vectors.

[0015] The motion feature vector is input into the recursive prediction network. In the recursive prediction network, the motion feature vector is fused with the historical prediction state to generate an updated prediction state. Based on the updated prediction state, the spatial displacement vector is decoded and accumulated with the previous prediction position to generate the current prediction position. The current prediction positions of multiple time steps are arranged in time order to generate a prediction position sequence.

[0016] The motion feature vector is spatially expanded on the pixel grid of the special effects source data to obtain the motion feature value corresponding to each pixel position. A neighborhood difference window is constructed for the motion feature value of each pixel position, and the difference between the motion feature value of the center pixel position and the neighboring pixel positions is calculated to generate the directional gradient vector.

[0017] The spatial gradient divergence value of each pixel position is generated by summing the difference calculations of the directional gradient vector in the horizontal and vertical directions respectively. The spatial gradient divergence value is then filled into the corresponding pixel position to generate a motion complexity distribution map.

[0018] The process involves estimating the colorization complexity of the special effects source data to generate an initial load map, and then coupling the motion complexity distribution map with the initial load map to generate a comprehensive load map, including:

[0019] Pixel-level shading analysis is performed on the special effects source data. The number of texture samplings and shading fusions at each pixel position are counted and summed to generate the shading computation amount. The shading computation amount at each pixel position is then filled into the corresponding pixel position to generate the initial load map.

[0020] The coloring computation of each pixel position in the initial load map is extracted and a spatial neighborhood window is constructed. The ratio of the standard deviation to the mean of the coloring computation within the spatial neighborhood window is calculated to generate the load dispersion. Based on the load dispersion, the coloring computation of each pixel position in the initial load map is nonlinearly amplified to generate an adjusted initial load map.

[0021] Extract the motion complexity value of each pixel position in the motion complexity distribution map and the coloring computation of each pixel position in the adjusted initial load map, and calculate the mutual information of the motion complexity value and the coloring computation in the spatial distribution to generate coupling strength weights.

[0022] Based on the coupling strength weight, an asymmetric fusion ratio for motion complexity value and shading computation is constructed. The motion complexity value and shading computation are fused according to the asymmetric fusion ratio to generate a comprehensive load value. The comprehensive load value of each pixel position is filled into the corresponding pixel position to generate a comprehensive load map.

[0023] The load threshold is determined based on the comprehensive load graph and frame rate constraint parameters, and the comprehensive load is then applied accordingly. Figure 2 Value-based generation of spatial partitioning masks, and separation of special effects source data into core region data and edge region data based on the spatial partitioning masks, including:

[0024] Extract the overall load value of each pixel position in the overall load map, sort the overall load values ​​by numerical value and count the number of pixels corresponding to each overall load value, accumulate the number of pixels and calculate the proportion of the accumulated number of pixels to the total number of pixels to generate a load accumulation curve, extract the target accumulation ratio corresponding to the frame rate constraint parameter, and find the overall load value corresponding to the target accumulation ratio on the load accumulation curve to determine the load threshold.

[0025] The comprehensive load values ​​of each pixel position in the comprehensive load map are compared based on the load threshold. Pixel positions with comprehensive load values ​​greater than the load threshold are assigned as core identifiers, and pixel positions with comprehensive load values ​​less than or equal to the load threshold are assigned as edge identifiers. The core identifiers and edge identifiers are filled into the corresponding pixel positions to generate a spatial partitioning mask.

[0026] Extract the core and edge identifiers of each pixel position in the spatial partitioning mask. Extract and aggregate the pixel data corresponding to the core identifiers in the special effects source data to generate core region data, and extract and aggregate the pixel data corresponding to the edge identifiers in the special effects source data to generate edge region data.

[0027] Performing full-precision rendering on core region data to generate core rendering results, and performing reduced-precision rendering on edge region data to generate edge rendering results includes:

[0028] Extract the core pixel data of each pixel position in the core region data, perform texture sampling on the core pixel data according to multiple texture levels, and extract the corresponding texture information from each texture level to generate multi-layer texture features;

[0029] Multi-channel shading calculations are performed on the texture information of each texture level in the multi-layer texture features, and the transparency channel is retained to generate complete rendering pixel values. The complete rendering pixel values ​​of each pixel position are aggregated to generate the core rendering result.

[0030] Extract edge pixel data at each pixel position in the edge region data, perform texture sampling on the edge pixel data only at the basic texture level, and extract texture information from the basic texture level to generate single-layer texture features;

[0031] The texture information of the basic texture level in the single-layer texture feature is subjected to single-channel shading calculation and the transparency channel is removed to generate simplified rendering pixel values. The simplified rendering pixel values ​​of each pixel position are aggregated to generate the edge rendering result.

[0032] The predicted position sequence is mapped to the core rendering result to extract the rendered value. The difference between the rendered value and the predicted value of the predicted position sequence is calculated to generate a prediction bias field. Based on the prediction bias field, the edge rendering result is compensated to generate a compensated edge rendering result, including:

[0033] Extract the spatial coordinates and predicted values ​​of each predicted position in the predicted position sequence, map the spatial coordinates to the core rendering result, and extract the complete rendered pixel value of the corresponding pixel position as the rendered value.

[0034] The difference between the rendered value and the predicted value is calculated to generate the positional deviation, and the positional deviation is associated with the spatial coordinates to construct the prediction deviation field;

[0035] Extract the spatial coordinates and positional deviations of each prediction location in the prediction deviation field, calculate the spatial distance between prediction locations to generate distance weights, and perform spatial interpolation on the positional deviations according to the distance weights to generate a global deviation distribution.

[0036] Extract the spatial coordinates and simplified rendering pixel values ​​of each pixel position in the edge rendering result, and query the deviation value corresponding to the spatial coordinates of each pixel position from the global deviation distribution;

[0037] Calculate the nearest distance from the spatial coordinates of each pixel location to the spatial coordinates of the predicted location in the prediction deviation field to generate an attenuation coefficient, and scale the deviation value according to the attenuation coefficient to generate a deviation correction amount;

[0038] The simplified rendering pixel values ​​are superimposed with the deviation correction amount to generate the compensated rendering pixel values, and the compensated rendering pixel values ​​are aggregated to generate the compensated edge rendering result.

[0039] The composite core rendering result and the compensated edge rendering result are combined to output the final rendering result, which includes:

[0040] The spatial adjacency boundary between the core rendering result and the compensated edge rendering result is detected. The complete rendering pixel value sequence and the compensated rendering pixel value sequence are extracted on both sides of the spatial adjacency boundary respectively. The distribution difference between the complete rendering pixel value sequence and the compensated rendering pixel value sequence in the color space and the brightness space is calculated to generate the dual-domain boundary gradient.

[0041] The color transition band width and brightness transition band width are calculated based on the color component and brightness component of the dual-domain boundary gradient, respectively. The maximum value of the color transition band width and brightness transition band width is taken to determine the adaptive transition band width. Within the adaptive transition band width range, the normal distance from each pixel position to the spatial adjacent boundary is extracted. Based on the normal distance, a nonlinear decay function is constructed to calculate the bidirectional fusion weight. The complete rendered pixel value and the compensated rendered pixel value are asymmetrically weighted and fused according to the bidirectional fusion weight to generate the transition pixel value.

[0042] The final rendering result is output by splicing together the complete rendered pixel values ​​from the core rendering result, the compensated rendered pixel values ​​from the compensated edge rendering result, and the transition pixel values ​​according to spatial coordinates.

[0043] A second aspect of this invention provides a real-time special effects rendering system based on deep learning, comprising:

[0044] The data acquisition unit is used to acquire the special effects source data and frame rate constraint parameters of the scene to be rendered;

[0045] The feature analysis unit is used to perform time-series analysis on the special effects source data to extract motion feature vectors, generate predicted position sequences based on the motion feature vectors, and calculate the spatial gradient divergence of the motion feature vectors to generate a motion complexity distribution map.

[0046] The load calculation unit is used to estimate the coloring complexity of the special effects source data to generate an initial load map, and to couple the motion complexity distribution map with the initial load map to generate a comprehensive load map.

[0047] The region segmentation unit is used to determine the load threshold based on the comprehensive load map and frame rate constraint parameters, and to apply the load threshold to the comprehensive load. Figure 2 Value-based generation of spatial partitioning mask, and separation of special effects source data into core region data and edge region data based on spatial partitioning mask;

[0048] The hierarchical rendering unit is used to perform full-precision rendering on the core area data to generate core rendering results, and to perform reduced-precision rendering on the edge area data to generate edge rendering results.

[0049] The deviation compensation unit is used to map the predicted position sequence to the core rendering result to extract the rendering value, calculate the difference between the rendering value and the predicted value of the predicted position sequence to generate a prediction deviation field, and compensate the edge rendering result based on the prediction deviation field to generate a compensated edge rendering result.

[0050] The result synthesis unit is used to synthesize the core rendering result and the compensated edge rendering result, and output the final rendering result.

[0051] A third aspect of the present invention provides an electronic device, comprising:

[0052] processor;

[0053] Memory used to store processor-executable instructions;

[0054] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0055] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0056] In this embodiment, by performing refined modeling and partitioned scheduling of the rendering process, the overall rendering performance and visual quality can be significantly improved while ensuring real-time performance. On one hand, by jointly modeling motion features and shading complexity, accurate assessment of scene computational load is achieved, allowing rendering resources to be prioritized for highly dynamic and complex areas, thereby reducing unnecessary computational overhead and improving rendering efficiency per unit time. Through the coordinated processing of high-precision rendering of core areas and low-precision rendering of edge areas, the detail representation of key visual areas can be maintained without significantly increasing the computational burden, achieving a dynamic balance between rendering quality and performance. At the same time, a deviation compensation mechanism based on predicted position is introduced, allowing low-precision areas to use high-precision results for error correction, effectively reducing detail loss and temporal discontinuity caused by simplified rendering, and improving image stability and continuity. Furthermore, boundary transition blending processing weakens visual abrupt changes between areas of different precision, ensuring that the final image maintains good coherence and immersion in dynamic scenes. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the real-time special effects rendering method based on deep learning, as described in an embodiment of the present invention.

[0058] Figure 2 This is a flowchart illustrating the smooth transition and fusion process of core and edge rendering in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0061] Figure 1 This is a flowchart illustrating the real-time special effects rendering method based on deep learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0062] Obtain the special effects source data and frame rate constraint parameters of the scene to be rendered;

[0063] Time series analysis is performed on the special effects source data to extract motion feature vectors, a predicted position sequence is generated based on the motion feature vectors, and the spatial gradient divergence of the motion feature vectors is calculated to generate a motion complexity distribution map.

[0064] The coloring complexity of the special effects source data is estimated to generate an initial load map. The motion complexity distribution map is coupled with the initial load map to generate a comprehensive load map.

[0065] The load threshold is determined based on the comprehensive load graph and frame rate constraint parameters, and the comprehensive load is then applied accordingly. Figure 2 Value-based generation of spatial partitioning mask, and separation of special effects source data into core region data and edge region data based on spatial partitioning mask;

[0066] Perform full-precision rendering on the core area data to generate core rendering results, and perform reduced-precision rendering on the edge area data to generate edge rendering results;

[0067] The predicted position sequence is mapped to the core rendering result to extract the rendering value. The difference between the rendering value and the predicted value of the predicted position sequence is calculated to generate a prediction bias field. The edge rendering result is compensated based on the prediction bias field to generate a compensated edge rendering result.

[0068] Combine the core rendering results with the compensated edge rendering results to output the final rendering result.

[0069] The process involves time-series analysis of the special effects source data to extract motion feature vectors, generating predicted position sequences based on these vectors, and calculating the spatial gradient divergence of the motion feature vectors to generate a motion complexity distribution map.

[0070] A time-series sliding window is constructed to extract motion feature sequences from the special effects source data. The time difference of the motion feature sequences is calculated to generate motion change rate features. The motion change rate features are used as weights to weight the motion feature sequences to generate motion feature vectors.

[0071] The motion feature vector is input into the recursive prediction network. In the recursive prediction network, the motion feature vector is fused with the historical prediction state to generate an updated prediction state. Based on the updated prediction state, the spatial displacement vector is decoded and accumulated with the previous prediction position to generate the current prediction position. The current prediction positions of multiple time steps are arranged in time order to generate a prediction position sequence.

[0072] The motion feature vector is spatially expanded on the pixel grid of the special effects source data to obtain the motion feature value corresponding to each pixel position. A neighborhood difference window is constructed for the motion feature value of each pixel position, and the difference between the motion feature value of the center pixel position and the neighboring pixel positions is calculated to generate the directional gradient vector.

[0073] The spatial gradient divergence value of each pixel position is generated by summing the difference calculations of the directional gradient vector in the horizontal and vertical directions respectively. The spatial gradient divergence value is then filled into the corresponding pixel position to generate a motion complexity distribution map.

[0074] Special effects source data typically includes particle position coordinates, velocity vectors, color attributes, and transparency information, organized in a time-series format. The temporal sliding window is constructed using a fixed-length window, set to 8 to 16 frames, to extract particle motion information from consecutive frames of the special effects source data in chronological order, forming a motion feature sequence. This sequence includes the particle position coordinates x in each frame. t y t and velocity component v xt v yt Temporal difference calculation employs feature value subtraction between adjacent frames, specifically the difference between the corresponding features of frame t and frame t-1, yielding the changes in position and velocity. These two values ​​are combined to form the motion rate of change feature. The motion rate of change feature is normalized to the interval 0 to 1 and used as a weighting coefficient. This coefficient is then multiplied element-wise with the feature values ​​at corresponding moments in the motion feature sequence, and finally, a weighted average is calculated over the time dimension to generate a motion feature vector with dimensions ranging from 64 to 128.

[0075] The recursive prediction network employs a gated recurrent unit structure, including a reset gate and an update gate. The motion feature vector serves as the input for the current time step, while the historical prediction state stores the hidden state vector from the previous time step, with the same dimension as the motion feature vector. The reset gate is obtained by calculating a linear combination of the motion feature vector and the historical prediction state, followed by sigmoid activation. The update gate follows a similar computational process. During fusion, the historical prediction state is first modulated by the reset gate and then concatenated with the motion feature vector. A hyperbolic tangent activation is then used to generate candidate states. The update gate controls the mixing ratio of the historical prediction state and the candidate states to obtain the updated prediction state. The decoder consists of two fully connected layers, taking the updated prediction state as input and outputting a two-dimensional spatial displacement vector (Δx, Δy). The initial value of the preceding prediction position is set to the current position of the particle in the special effects source data. The current prediction position is obtained by accumulating the spatial displacement vector. The network recursively executes 5 to 10 time steps, storing the prediction positions from each step in chronological order to form a prediction position sequence.

[0076] The spatial unfolding operation maps the components of the motion feature vector to the pixel grid of the effect source data. The mapping rule determines the pixel affiliation based on the particle position coordinates. When multiple particles are mapped to the same pixel, the maximum value of the motion feature vector is taken as the motion feature value of that pixel. The neighborhood difference window uses a 3×3 size, with the center pixel position denoted as (i, j), and the neighboring pixel positions covering 8 positions from (i-1, j-1) to (i+1, j+1). The difference between the motion feature value of the center pixel and the motion feature values ​​of each neighboring pixel is calculated, and 8 difference components are generated according to the direction of the neighboring pixels relative to the center, which are combined into a directional gradient vector.

[0077] The horizontal differential calculation subtracts the difference components of the left and right neighbors in the directional gradient vector, while the vertical differential calculation subtracts the difference components of the upper and lower neighbors. The sum of these two subtractions yields the spatial gradient divergence value. This value reflects the drastic change in the motion characteristics of a pixel position; a larger value indicates higher motion complexity. After calculating the spatial gradient divergence value for all pixels in the source data, the spatial gradient divergence value is filled into the corresponding pixel coordinates, forming a motion complexity distribution map consistent with the spatial resolution of the source data. The pixel values ​​are normalized to a grayscale range of 0 to 255.

[0078] The process involves estimating the colorization complexity of the special effects source data to generate an initial load map, and then coupling the motion complexity distribution map with the initial load map to generate a comprehensive load map, including:

[0079] Pixel-level shading analysis is performed on the special effects source data. The number of texture samplings and shading fusions at each pixel position are counted and summed to generate the shading computation amount. The shading computation amount at each pixel position is then filled into the corresponding pixel position to generate the initial load map.

[0080] The coloring computation of each pixel position in the initial load map is extracted and a spatial neighborhood window is constructed. The ratio of the standard deviation to the mean of the coloring computation within the spatial neighborhood window is calculated to generate the load dispersion. Based on the load dispersion, the coloring computation of each pixel position in the initial load map is nonlinearly amplified to generate an adjusted initial load map.

[0081] Extract the motion complexity value of each pixel position in the motion complexity distribution map and the coloring computation of each pixel position in the adjusted initial load map, and calculate the mutual information of the motion complexity value and the coloring computation in the spatial distribution to generate coupling strength weights.

[0082] Based on the coupling strength weight, an asymmetric fusion ratio for motion complexity value and shading computation is constructed. The motion complexity value and shading computation are fused according to the asymmetric fusion ratio to generate a comprehensive load value. The comprehensive load value of each pixel position is filled into the corresponding pixel position to generate a comprehensive load map.

[0083] When performing pixel-level shading analysis on the special effects source data, the shader call records for each pixel position in the current frame are traversed. For each pixel position (x, y), the number of texture sampling operations T(x, y) triggered during the rendering process is counted, which is obtained from the number of texture read instructions in the shader execution log. Simultaneously, the number of shading blending operations F(x, y) participated in by this pixel is also counted, including the cumulative number of alpha blending, additive blending, and other operations. The two are then added together to obtain the shading computational load L. i (x, y) = T(x, y) + F(x, y). The calculated shading computation is filled into a two-dimensional matrix with the same spatial resolution as the source data of the special effects. Each element of the matrix corresponds to the shading computation at a pixel position. This matrix is ​​the initial load map.

[0084] When performing load dispersion analysis on the initial load map, a 7×7 spatial neighborhood window is constructed centered on each pixel location (x, y). Forty-nine shading computation samples are extracted from the window, and the standard deviation σ and mean μ of the sample set are calculated. The load dispersion is calculated as D(x, y) = σ / (μ + ε), where ε = 0.01 to avoid division by zero errors. The shading computation is nonlinearly amplified based on the load dispersion, using the amplification formula L. a (x, y) = L i (x, y)·(1+0.3·tanh(2·D(x, y))), this amplification operation gives higher weight values ​​to areas with uneven load distribution, so that they are preferentially allocated to high-precision rendering areas in subsequent partitioning.

[0085] When calculating the spatial distribution mutual information between motion complexity values ​​and coloring computation, the motion complexity distribution map and the adjusted initial load map are discretized into 16 levels. A two-dimensional joint histogram matrix H is constructed, and the joint occurrence frequency of each pixel position in the two feature dimensions is statistically analyzed. The edge probability distribution P is calculated. m and P s and the joint probability distribution P j The formula for calculating mutual information I is as follows: Where i represents the value level index of the discretized motion complexity distribution map, ranging from 1 to 16, j represents the value level index of the discretized adjusted initial load map, also ranging from 1 to 16, and P j (i, j) represents the joint probability of the i-th level motion complexity and the j-th level coloring computation occurring simultaneously, P m (i) represents the marginal probability of the i-th level of motion complexity, P s (j) represents the marginal probability of the j-th level coloring computation. The mutual information is normalized to the [0, 1] interval and used as the coupling strength weight W. e .

[0086] When constructing the asymmetric fusion ratio, the fusion weight for the motion complexity value is set to α = 0.4 + 0.3·W. e The fusion weights for shading computation are set to β = 1 - α to ensure the sum of the weights is 1. For each pixel location, the motion complexity value M(x, y) and the adjusted shading computation value L are extracted. a (x, y) are weighted and fused according to the asymmetric fusion ratio to calculate the comprehensive load value L. c (x, y) = α·M(x, y) + β·L a (x, y). The combined load values ​​of all pixel locations are filled into a matrix with the same resolution as the source data of the effects, forming a combined load map. This combined load map reflects both the motion complexity of the scene and the computational burden of shading, providing an accurate load basis for subsequent spatial partitioning.

[0087] The load threshold is determined based on the comprehensive load graph and frame rate constraint parameters, and the comprehensive load is then applied accordingly. Figure 2 Value-based generation of spatial partitioning masks, and separation of special effects source data into core region data and edge region data based on the spatial partitioning masks, including:

[0088] Extract the overall load value of each pixel position in the overall load map, sort the overall load values ​​by numerical value and count the number of pixels corresponding to each overall load value, accumulate the number of pixels and calculate the proportion of the accumulated number of pixels to the total number of pixels to generate a load accumulation curve, extract the target accumulation ratio corresponding to the frame rate constraint parameter, and find the overall load value corresponding to the target accumulation ratio on the load accumulation curve to determine the load threshold.

[0089] The comprehensive load values ​​of each pixel position in the comprehensive load map are compared based on the load threshold. Pixel positions with comprehensive load values ​​greater than the load threshold are assigned as core identifiers, and pixel positions with comprehensive load values ​​less than or equal to the load threshold are assigned as edge identifiers. The core identifiers and edge identifiers are filled into the corresponding pixel positions to generate a spatial partitioning mask.

[0090] Extract the core and edge identifiers of each pixel position in the spatial partitioning mask. Extract and aggregate the pixel data corresponding to the core identifiers in the special effects source data to generate core region data, and extract and aggregate the pixel data corresponding to the edge identifiers in the special effects source data to generate edge region data.

[0091] During spatial partitioning, all pixel positions in the comprehensive load map are traversed first, and the comprehensive load value at each pixel position is read. The comprehensive load values ​​corresponding to all pixel positions are stored in a one-dimensional array, and a quicksort algorithm is applied to this array to sort them in ascending order of value. After sorting, the sorted array is traversed, and the number of pixels corresponding to each different comprehensive load value is counted. Specifically, a hash table is created, using the comprehensive load value as the key and the number of times that value appears as the value; the counter in the hash table is incremented while traversing and sorting the array.

[0092] After completing the statistics, the number of pixels is accumulated starting from the minimum overall load value. The initial value of the accumulation variable is set to zero, and the number of pixels corresponding to each overall load value is added to the accumulation variable sequentially. After each accumulation, the ratio of the current accumulated pixel count to the total pixel count is calculated; this ratio is the cumulative percentage. Each overall load value and its corresponding cumulative percentage are stored sequentially, forming discrete data points for the load accumulation curve. This curve reflects the relationship between the overall load value and spatial distribution, with the overall load value on the horizontal axis and the cumulative percentage on the vertical axis.

[0093] The target cumulative percentage is determined based on the frame rate constraint parameters. These parameters reflect the allocation of system computing resources. If a frame rate constraint parameter of 30 frames per second corresponds to a certain baseline value, the target cumulative percentage can be set to 0.65; if a frame rate constraint parameter of 60 frames per second corresponds to another baseline value, the target cumulative percentage can be adjusted to 0.50. Among the discrete data points of the load accumulation curve, the data point where the cumulative percentage first exceeds or equals the target cumulative percentage is found. The comprehensive load value corresponding to this data point is extracted and determined as the load threshold. If no precisely matching cumulative percentage exists among the discrete data points, a linear interpolation method is used to calculate the load threshold, i.e., interpolation is performed between data points on both sides of the target cumulative percentage.

[0094] After determining the load threshold, all pixel positions in the composite load map are traversed again. For each pixel position, its composite load value is read and compared with the load threshold. If the composite load value > the load threshold, the pixel position is assigned a core identifier in a mask matrix of the same size, which can be represented by the integer 1; if the composite load value ≤ the load threshold, the pixel position is assigned an edge identifier in the mask matrix, which can be represented by the integer 0. After the traversal is complete, all pixel positions in the mask matrix are filled with 0 or 1, and this mask matrix is ​​the spatial partitioning mask. The spatial partitioning mask has the same spatial resolution and pixel position correspondence as the composite load map.

[0095] After generating the spatial partitioning mask, the core and edge markers are extracted, and a separation operation is performed on the effect source data. All pixel positions in the spatial partitioning mask are traversed, and positions with a marker value of 1 are identified. The corresponding pixel data from the effect source data is then read. Pixel data includes complete attributes required for rendering, such as color channel values, transparency values, and depth values. The pixel data corresponding to all core marker positions are aggregated according to their original spatial coordinates to generate a core region data structure. This data structure retains the spatial index information of the pixels, facilitating accurate positioning during subsequent full-precision rendering.

[0096] The edge marker locations are handled in the same way. The spatial partitioning mask is traversed to identify pixel locations with a marker value of 0, and the corresponding pixel data is extracted from the effect source data. All pixel data corresponding to edge marker locations are aggregated to generate an edge region data structure. The edge region data also retains its spatial index to ensure accurate backfilling to the original position after reduced-precision rendering. Through the above separation operations, the effect source data is completely divided into two parts: core region data and edge region data. These two parts are spatially complementary and non-overlapping, providing a data foundation for subsequent differentiated rendering strategies.

[0097] Performing full-precision rendering on core region data to generate core rendering results, and performing reduced-precision rendering on edge region data to generate edge rendering results includes:

[0098] Extract the core pixel data of each pixel position in the core region data, perform texture sampling on the core pixel data according to multiple texture levels, and extract the corresponding texture information from each texture level to generate multi-layer texture features;

[0099] Multi-channel shading calculations are performed on the texture information of each texture level in the multi-layer texture features, and the transparency channel is retained to generate complete rendering pixel values. The complete rendering pixel values ​​of each pixel position are aggregated to generate the core rendering result.

[0100] Extract edge pixel data at each pixel position in the edge region data, perform texture sampling on the edge pixel data only at the basic texture level, and extract texture information from the basic texture level to generate single-layer texture features;

[0101] The texture information of the basic texture level in the single-layer texture feature is subjected to single-channel shading calculation and the transparency channel is removed to generate simplified rendering pixel values. The simplified rendering pixel values ​​of each pixel position are aggregated to generate the edge rendering result.

[0102] When performing differentiated rendering on core region data and edge region data, a full-precision rendering strategy is adopted for the core region data. Core pixel data is extracted pixel-by-pixel from the core region data. This core pixel data contains complete geometric information such as the pixel's 3D coordinates, normal vector, UV coordinates, and material index. For each core pixel data, texture sampling is performed according to the Mipmap hierarchy, typically including three texture levels: base level, medium detail level, and high detail level. Base texture information with a resolution of 512×512 is extracted at the base texture level, medium detail texture information with a resolution of 1024×1024 is extracted at the medium detail level, and high detail texture information with a resolution of 2048×2048 is extracted at the high detail level. The texture information from these three levels together constitutes a multi-layered texture feature.

[0103] When performing multi-channel shading calculations on multi-layered texture features, the shading contribution values ​​of the diffuse, specular, ambient occlusion, and alpha channels are calculated separately. The diffuse channel calculates diffuse intensity based on the angle between the light source direction and the pixel normal vector; the specular channel calculates specular intensity based on the angle between the viewing direction and the reflection direction; and the ambient occlusion channel calculates the occlusion coefficient based on the surrounding geometric occlusion relationships. The alpha channel extracts the transparency value from the texture's alpha component and retains this channel information. The calculation results of the four channels are then fused according to weights to form a complete rendered pixel value, which includes RGB color information and alpha transparency information. The complete rendered pixel values ​​corresponding to all pixel positions within the core region are arranged and aggregated according to their spatial positions to form the image matrix of the core rendering result.

[0104] A reduced-precision rendering strategy is employed for edge region data. Edge pixel data is extracted from the edge region data. This edge pixel data still contains the pixel's geometric and material information, but texture sampling is only performed at the base texture level. Texture information is extracted from the base texture level with a resolution of 512×512 to generate a single-layer texture feature. This single-layer texture feature only contains the base level's color and normal information and does not include the fine texture variations at high-detail levels.

[0105] When performing simplified shading calculations on single-layer texture features, only the shading contribution value of the diffuse channel is calculated, ignoring the calculations of the specular and ambient occlusion channels. Diffuse intensity is directly obtained by the dot product of the light source intensity and the pixel normal vector, without considering complex illumination transport models. Simultaneously, the calculation and storage of the alpha channel are actively eliminated, and the alpha value of edge pixels is uniformly set to 1.0, representing a completely opaque state. This simplification reduces the shading computation to one-quarter of that in the core region. The simplified rendered pixel values ​​of each pixel location within the edge region are aggregated to form an image matrix of the edge rendering result. Through this differentiated rendering strategy, the core region maintains visual quality, while the edge region significantly reduces the rendering load through precision reduction, providing foundational data for subsequent prediction bias field compensation.

[0106] The predicted position sequence is mapped to the core rendering result to extract the rendered value. The difference between the rendered value and the predicted value of the predicted position sequence is calculated to generate a prediction bias field. Based on the prediction bias field, the edge rendering result is compensated to generate a compensated edge rendering result, including:

[0107] Extract the spatial coordinates and predicted values ​​of each predicted position in the predicted position sequence, map the spatial coordinates to the core rendering result, and extract the complete rendered pixel value of the corresponding pixel position as the rendered value.

[0108] The difference between the rendered value and the predicted value is calculated to generate the positional deviation, and the positional deviation is associated with the spatial coordinates to construct the prediction deviation field;

[0109] Extract the spatial coordinates and positional deviations of each prediction location in the prediction deviation field, calculate the spatial distance between prediction locations to generate distance weights, and perform spatial interpolation on the positional deviations according to the distance weights to generate a global deviation distribution.

[0110] Extract the spatial coordinates and simplified rendering pixel values ​​of each pixel position in the edge rendering result, and query the deviation value corresponding to the spatial coordinates of each pixel position from the global deviation distribution;

[0111] Calculate the nearest distance from the spatial coordinates of each pixel location to the spatial coordinates of the predicted location in the prediction deviation field to generate an attenuation coefficient, and scale the deviation value according to the attenuation coefficient to generate a deviation correction amount;

[0112] The simplified rendering pixel values ​​are superimposed with the deviation correction amount to generate the compensated rendering pixel values, and the compensated rendering pixel values ​​are aggregated to generate the compensated edge rendering result.

[0113] During real-time special effects rendering, the predicted position sequence is generated by the motion simulation module, containing the spatial coordinates and predicted rendering values ​​of multiple key prediction points. For each predicted position, the corresponding actual rendering value needs to be extracted from the core rendering result. The mapping process first determines the spatial coordinates of the predicted position. For example, the spatial coordinates of the predicted position P are (127.5, 85.2), which are projected onto the image space of the core rendering result through homogeneous coordinate transformation. Since the coordinates may fall at pixel intervals, bilinear interpolation is used to extract accurate rendering values ​​from the surrounding four pixels. If the RGB rendering values ​​of the four pixels surrounding point P are (0.92, 0.65, 0.43), (0.90, 0.63, 0.41), (0.93, 0.67, 0.45), and (0.91, 0.64, 0.42), respectively, the actual rendering value of point P is calculated using bilinear interpolation as (0.914, 0.647, 0.427).

[0114] After obtaining the actual rendered value, the difference between it and the predicted rendered value is calculated to generate the positional deviation. If the predicted rendered value of point P is (0.88, 0.61, 0.39), then the positional deviation is (0.034, 0.037, 0.037). This difference reflects the degree of deviation between the prediction model and the actual rendered result. The same operation is performed for each predicted position in the special effects rendering, obtaining a series of positional deviation values, and associating these deviation values ​​with the corresponding spatial coordinates to form a prediction deviation field. The prediction deviation field is represented in data structure as a set containing spatial positions and deviation values, with each element containing coordinates and the corresponding RGB channel deviation.

[0115] To generate a continuous global deviation distribution from a discrete prediction deviation field, spatial interpolation is required. First, the spatial distance between prediction locations is calculated, generating distance-based weighting coefficients. The distance is calculated using the Euclidean distance formula; if the spatial coordinates of two prediction locations are (x1y1) and (x2, y2), the straight-line distance between them is calculated. The distance weights are calculated using an inverse proportional function, with closer locations receiving larger weights. In actual interpolation calculations, only the 5-8 closest prediction locations are considered to reduce computational load while maintaining interpolation accuracy. For each interpolation point, the deviation values ​​of surrounding prediction locations are weighted and averaged according to the distance weights to obtain the estimated deviation value for that point. In this way, deviation estimates can be generated for any location in the effects rendering space, forming a continuous global deviation distribution.

[0116] When compensating for edge rendering results, the spatial coordinates and simplified rendering pixel values ​​of each pixel in the edge rendering result are first extracted. Edge rendering results are typically obtained using low-precision rendering methods, such as reducing the number of lighting calculations, lowering the sampling rate, or simplifying special effects models. For pixel Q in the edge rendering result, its spatial coordinates are (175.0, 132.0). The corresponding deviation value for this coordinate is retrieved from the global deviation distribution. If this coordinate is not a point in the predicted position sequence, the same spatial interpolation method is used to calculate the deviation value. Assume the spatial coordinates of the three nearest predicted locations P1, P2, and P3 around point Q are (171.2, 130.5), (177.8, 129.3), and (174.1, 135.6), respectively, with corresponding deviation values ​​of (0.034, 0.037, 0.037), (0.028, 0.032, 0.030), and (0.031, 0.035, 0.033). The calculated distances are 4.5, 3.8, and 3.7, respectively, with corresponding distance weights of 0.049, 0.068, and 0.071. The interpolation deviation value for point Q is calculated using a weighted average: (0.031, 0.035, 0.033).

[0117] To ensure that the compensation effect naturally diminishes with increasing distance from the predicted location, a distance-based attenuation mechanism is introduced. The distance from the pixel location to the nearest predicted location is calculated, and an exponential attenuation function is used to generate the attenuation coefficient. This function's characteristic is that the attenuation becomes more significant with increasing distance, ensuring that areas far from the predicted location are not overcompensated. If the distance from point Q to the nearest predicted location is 3.7, and the average spacing between predicted locations is 10, then the calculated attenuation coefficient is 0.985. Multiplying the interpolation deviation value by the attenuation coefficient yields the actual deviation correction amount used, which for point Q is (0.030, 0.034, 0.032).

[0118] The simplified rendered pixel values ​​in the edge rendering result are superimposed with the deviation correction amount to generate compensated rendered pixel values. If the simplified rendered pixel value of point Q is (0.85, 0.58, 0.36) and the deviation correction amount is (0.030, 0.034, 0.032), then the compensated rendered pixel value is (0.880, 0.614, 0.392). To prevent the compensation result from exceeding the valid range, the result is cropped to ensure that all channel values ​​are within the valid range. The above compensation operation is performed on all pixels in the edge rendering result, and the compensated pixel values ​​are organized according to the original arrangement to generate the compensated edge rendering result.

[0119] To optimize computational efficiency, a pre-computation strategy can be adopted, discretizing the global deviation distribution into a lookup table in the form of a regular grid. Compensation calculations for pixels in edge regions can directly retrieve the deviation value from the table, reducing the overhead of real-time interpolation calculations. The resolution of the lookup table can be adjusted according to actual needs, generally set to 1 / 4 to 1 / 2 of the original rendering resolution, reducing storage and query burdens while ensuring compensation quality.

[0120] In practical applications, the predicted position sequence can be obtained by selecting key points of the special effects particle system, typically with a denser distribution in areas of high particle density or drastic motion changes. The core rendering result is generated using the complete special effects rendering pipeline, including all lighting, shadow, and material details, while the edge rendering result uses a simplified shading model, omitting complex lighting calculations and high-precision texture sampling. The computational cost of the compensation process is far less than that of the full rendering, thus significantly improving overall rendering performance.

[0121] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the smooth transition and fusion process of core and edge rendering in an embodiment of the present invention.

[0122] The composite core rendering result and the compensated edge rendering result are combined to output the final rendering result, which includes:

[0123] The spatial adjacency boundary between the core rendering result and the compensated edge rendering result is detected. The complete rendering pixel value sequence and the compensated rendering pixel value sequence are extracted on both sides of the spatial adjacency boundary respectively. The distribution difference between the complete rendering pixel value sequence and the compensated rendering pixel value sequence in the color space and the brightness space is calculated to generate the dual-domain boundary gradient.

[0124] The color transition band width and brightness transition band width are calculated based on the color component and brightness component of the dual-domain boundary gradient, respectively. The maximum value of the color transition band width and brightness transition band width is taken to determine the adaptive transition band width. Within the adaptive transition band width range, the normal distance from each pixel position to the spatial adjacent boundary is extracted. Based on the normal distance, a nonlinear decay function is constructed to calculate the bidirectional fusion weight. The complete rendered pixel value and the compensated rendered pixel value are asymmetrically weighted and fused according to the bidirectional fusion weight to generate the transition pixel value.

[0125] The final rendering result is output by splicing together the complete rendered pixel values ​​from the core rendering result, the compensated rendered pixel values ​​from the compensated edge rendering result, and the transition pixel values ​​according to spatial coordinates.

[0126] After rendering the core and edge regions independently, the two rendering results need to be spatially composited. Since the core region uses full-precision rendering while the edge region uses reduced-precision rendering, visual discontinuities may exist at their boundary. The spatial adjacency boundary between the core rendering result and the compensated edge rendering result is detected; this boundary corresponds to the position in the aforementioned spatial partitioning mask where the value jumps from 1 to 0. A complete rendered pixel value sequence with a width of 3 to 5 pixels is extracted along this boundary towards the inner part of the core region, and a compensated rendered pixel value sequence of the same width is extracted towards the outer part of the edge region. The mean and standard deviation of each channel of these two pixel value sequences are calculated in the RGB color space, and the difference in color distribution is measured using the root mean square distance. In the luminance space, after converting RGB to grayscale values, the ratio of the mean difference to the standard deviation of luminance is calculated; these two are combined to form a dual-domain boundary gradient.

[0127] The color component of the dual-domain boundary gradient reflects the degree of abrupt changes in hue and saturation, while the luminance component reflects the intensity of contrast. A threshold of 0.15 is set for the color component; when the component value exceeds this threshold, the transition band width is expanded proportionally. The color transition band width is calculated by multiplying the base width of 3 pixels by the ratio of the color component to the threshold, with a maximum of 8 pixels. A threshold of 0.2 is set for the luminance component, and the luminance transition band width is also expanded proportionally from a base of 3 pixels to a maximum of 10 pixels. The color transition band width and the luminance transition band width are compared, and the larger value is taken as the adaptive transition band width to ensure coverage of the more visually sensitive gradient dimensions.

[0128] Within a defined adaptive transition band width, the Euclidean distance from the spatial coordinates of each pixel to the nearest point on the adjacent boundary is calculated as the normal distance d. A nonlinear decay function w(d) = 0.5 + 0.5 × cos(π × d / W) is constructed, where W is the adaptive transition band width. This function takes a value of 0.5 at the boundary to achieve symmetrical blending, decaying to 0 or 1 towards both sides. The blending weight on the core region side is w(d), and on the edge region side it is 1 - w(d), forming a bidirectional blending weight. For pixels within the transition band, the corresponding weight value is obtained according to their normal distance, and the weighted sum of the full-rendered pixel value and the compensated-rendered pixel value is obtained to obtain the transition pixel value. When a pixel is located on the core region side of the transition band, the contribution weight of the full-rendered pixel decreases, and the contribution weight of the compensated-rendered pixel increases; the opposite is true on the edge region side, achieving a smooth asymmetric transition.

[0129] After calculating the transition zone pixels, the final rendering result is assembled according to spatial coordinate indices. The original complete rendering pixel values ​​are retained within the core rendering region, the compensated rendering pixel values ​​are retained in the compensated edge rendering region, and the calculated transition pixel values ​​are filled into the transition zone region. These three parts of data are written to the output texture through frame buffer coordinate mapping, generating a visually continuous and computationally efficient final rendering frame. This compositing scheme avoids excessive blurring or visible seams caused by fixed-width transition zones, and adaptively adjusts the transition strategy based on actual rendering differences, ensuring consistent rendering quality across different scenes.

[0130] A second aspect of this invention provides a real-time special effects rendering system based on deep learning, the system comprising:

[0131] The data acquisition unit is used to acquire the special effects source data and frame rate constraint parameters of the scene to be rendered;

[0132] The feature analysis unit is used to perform time-series analysis on the special effects source data to extract motion feature vectors, generate predicted position sequences based on the motion feature vectors, and calculate the spatial gradient divergence of the motion feature vectors to generate a motion complexity distribution map.

[0133] The load calculation unit is used to estimate the coloring complexity of the special effects source data to generate an initial load map, and to couple the motion complexity distribution map with the initial load map to generate a comprehensive load map.

[0134] The region segmentation unit is used to determine the load threshold based on the comprehensive load map and frame rate constraint parameters, and to apply the load threshold to the comprehensive load. Figure 2 Value-based generation of spatial partitioning mask, and separation of special effects source data into core region data and edge region data based on spatial partitioning mask;

[0135] The hierarchical rendering unit is used to perform full-precision rendering on the core area data to generate core rendering results, and to perform reduced-precision rendering on the edge area data to generate edge rendering results.

[0136] The deviation compensation unit is used to map the predicted position sequence to the core rendering result to extract the rendering value, calculate the difference between the rendering value and the predicted value of the predicted position sequence to generate a prediction deviation field, and compensate the edge rendering result based on the prediction deviation field to generate a compensated edge rendering result.

[0137] The result synthesis unit is used to synthesize the core rendering result and the compensated edge rendering result, and output the final rendering result.

[0138] A third aspect of the present invention provides an electronic device, comprising:

[0139] processor;

[0140] Memory used to store processor-executable instructions;

[0141] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0142] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0143] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time special effects rendering method based on deep learning, characterized in that, include: Obtain the special effects source data and frame rate constraint parameters of the scene to be rendered; Time series analysis is performed on the special effects source data to extract motion feature vectors, a predicted position sequence is generated based on the motion feature vectors, and the spatial gradient divergence of the motion feature vectors is calculated to generate a motion complexity distribution map. The coloring complexity of the special effects source data is estimated to generate an initial load map. The motion complexity distribution map is coupled with the initial load map to generate a comprehensive load map. The load threshold is determined based on the comprehensive load map and frame rate constraint parameters. The comprehensive load map is binarized according to the load threshold to generate a spatial partitioning mask. Based on the spatial partitioning mask, the special effects source data is separated into core region data and edge region data. Perform full-precision rendering on the core area data to generate core rendering results, and perform reduced-precision rendering on the edge area data to generate edge rendering results; The predicted position sequence is mapped to the core rendering result to extract the rendering value. The difference between the rendering value and the predicted value of the predicted position sequence is calculated to generate a prediction bias field. The edge rendering result is compensated based on the prediction bias field to generate a compensated edge rendering result. Combine the core rendering results with the compensated edge rendering results to output the final rendering result.

2. The method according to claim 1, characterized in that, The process involves time-series analysis of the special effects source data to extract motion feature vectors, generating predicted position sequences based on these vectors, and calculating the spatial gradient divergence of the motion feature vectors to generate a motion complexity distribution map. A time-series sliding window is constructed to extract motion feature sequences from the special effects source data. The time difference of the motion feature sequences is calculated to generate motion change rate features. The motion change rate features are used as weights to weight the motion feature sequences to generate motion feature vectors. The motion feature vector is input into the recursive prediction network. In the recursive prediction network, the motion feature vector is fused with the historical prediction state to generate an updated prediction state. Based on the updated prediction state, the spatial displacement vector is decoded and accumulated with the previous prediction position to generate the current prediction position. The current prediction positions of multiple time steps are arranged in time order to generate a prediction position sequence. The motion feature vector is spatially expanded on the pixel grid of the special effects source data to obtain the motion feature value corresponding to each pixel position. A neighborhood difference window is constructed for the motion feature value of each pixel position, and the difference between the motion feature value of the center pixel position and the neighboring pixel positions is calculated to generate the directional gradient vector. The spatial gradient divergence value of each pixel position is generated by summing the difference calculations of the directional gradient vector in the horizontal and vertical directions respectively. The spatial gradient divergence value is then filled into the corresponding pixel position to generate a motion complexity distribution map.

3. The method according to claim 1, characterized in that, The process involves estimating the colorization complexity of the special effects source data to generate an initial load map, and then coupling the motion complexity distribution map with the initial load map to generate a comprehensive load map, including: Pixel-level shading analysis is performed on the special effects source data. The number of texture samplings and shading fusions at each pixel position are counted and summed to generate the shading computation amount. The shading computation amount at each pixel position is then filled into the corresponding pixel position to generate the initial load map. The coloring computation of each pixel position in the initial load map is extracted and a spatial neighborhood window is constructed. The ratio of the standard deviation to the mean of the coloring computation within the spatial neighborhood window is calculated to generate the load dispersion. Based on the load dispersion, the coloring computation of each pixel position in the initial load map is nonlinearly amplified to generate an adjusted initial load map. Extract the motion complexity value of each pixel position in the motion complexity distribution map and the coloring computation of each pixel position in the adjusted initial load map, and calculate the mutual information of the motion complexity value and the coloring computation in the spatial distribution to generate coupling strength weights. Based on the coupling strength weight, an asymmetric fusion ratio for motion complexity value and shading computation is constructed. The motion complexity value and shading computation are fused according to the asymmetric fusion ratio to generate a comprehensive load value. The comprehensive load value of each pixel position is filled into the corresponding pixel position to generate a comprehensive load map.

4. The method according to claim 1, characterized in that, The load threshold is determined based on the comprehensive load map and frame rate constraint parameters. A spatial partitioning mask is generated by binarizing the comprehensive load map according to the load threshold. Based on the spatial partitioning mask, the special effects source data is separated into core region data and edge region data, including: Extract the overall load value of each pixel position in the overall load map, sort the overall load values ​​by numerical value and count the number of pixels corresponding to each overall load value, accumulate the number of pixels and calculate the proportion of the accumulated number of pixels to the total number of pixels to generate a load accumulation curve, extract the target accumulation ratio corresponding to the frame rate constraint parameter, and find the overall load value corresponding to the target accumulation ratio on the load accumulation curve to determine the load threshold. The comprehensive load values ​​of each pixel position in the comprehensive load map are compared based on the load threshold. Pixel positions with comprehensive load values ​​greater than the load threshold are assigned as core identifiers, and pixel positions with comprehensive load values ​​less than or equal to the load threshold are assigned as edge identifiers. The core identifiers and edge identifiers are filled into the corresponding pixel positions to generate a spatial partitioning mask. Extract the core and edge identifiers of each pixel position in the spatial partitioning mask. Extract and aggregate the pixel data corresponding to the core identifiers in the special effects source data to generate core region data, and extract and aggregate the pixel data corresponding to the edge identifiers in the special effects source data to generate edge region data.

5. The method according to claim 1, characterized in that, Performing full-precision rendering on core region data to generate core rendering results, and performing reduced-precision rendering on edge region data to generate edge rendering results includes: Extract the core pixel data of each pixel position in the core region data, perform texture sampling on the core pixel data according to multiple texture levels, and extract the corresponding texture information from each texture level to generate multi-layer texture features; Multi-channel shading calculations are performed on the texture information of each texture level in the multi-layer texture features, and the transparency channel is retained to generate complete rendering pixel values. The complete rendering pixel values ​​of each pixel position are aggregated to generate the core rendering result. Extract edge pixel data at each pixel position in the edge region data, perform texture sampling on the edge pixel data only at the basic texture level, and extract texture information from the basic texture level to generate single-layer texture features; The texture information of the basic texture level in the single-layer texture feature is subjected to single-channel shading calculation and the transparency channel is removed to generate simplified rendering pixel values. The simplified rendering pixel values ​​of each pixel position are aggregated to generate the edge rendering result.

6. The method according to claim 1, characterized in that, The predicted position sequence is mapped to the core rendering result to extract the rendered value. The difference between the rendered value and the predicted value of the predicted position sequence is calculated to generate a prediction bias field. Based on the prediction bias field, the edge rendering result is compensated to generate a compensated edge rendering result, including: Extract the spatial coordinates and predicted values ​​of each predicted position in the predicted position sequence, map the spatial coordinates to the core rendering result, and extract the complete rendered pixel value of the corresponding pixel position as the rendered value. The difference between the rendered value and the predicted value is calculated to generate the positional deviation, and the positional deviation is associated with the spatial coordinates to construct the prediction deviation field; Extract the spatial coordinates and positional deviations of each prediction location in the prediction deviation field, calculate the spatial distance between prediction locations to generate distance weights, and perform spatial interpolation on the positional deviations according to the distance weights to generate a global deviation distribution. Extract the spatial coordinates and simplified rendering pixel values ​​of each pixel position in the edge rendering result, and query the deviation value corresponding to the spatial coordinates of each pixel position from the global deviation distribution; Calculate the nearest distance from the spatial coordinates of each pixel location to the spatial coordinates of the predicted location in the prediction deviation field to generate an attenuation coefficient, and scale the deviation value according to the attenuation coefficient to generate a deviation correction amount; The simplified rendering pixel values ​​are superimposed with the deviation correction amount to generate the compensated rendering pixel values, and the compensated rendering pixel values ​​are aggregated to generate the compensated edge rendering result.

7. The method according to claim 1, characterized in that, The composite core rendering result and the compensated edge rendering result are combined to output the final rendering result, which includes: The spatial adjacency boundary between the core rendering result and the compensated edge rendering result is detected. The complete rendering pixel value sequence and the compensated rendering pixel value sequence are extracted on both sides of the spatial adjacency boundary respectively. The distribution difference between the complete rendering pixel value sequence and the compensated rendering pixel value sequence in the color space and the brightness space is calculated to generate the dual-domain boundary gradient. The color transition band width and brightness transition band width are calculated based on the color component and brightness component of the dual-domain boundary gradient, respectively. The maximum value of the color transition band width and brightness transition band width is taken to determine the adaptive transition band width. Within the adaptive transition band width range, the normal distance from each pixel position to the spatial adjacent boundary is extracted. Based on the normal distance, a nonlinear decay function is constructed to calculate the bidirectional fusion weight. The complete rendered pixel value and the compensated rendered pixel value are asymmetrically weighted and fused according to the bidirectional fusion weight to generate the transition pixel value. The final rendering result is output by splicing together the complete rendered pixel values ​​from the core rendering result, the compensated rendered pixel values ​​from the compensated edge rendering result, and the transition pixel values ​​according to spatial coordinates.

8. A real-time special effects rendering system based on deep learning, used to implement the method of any one of claims 1-7, characterized in that, include: The data acquisition unit is used to acquire the special effects source data and frame rate constraint parameters of the scene to be rendered; The feature analysis unit is used to perform time-series analysis on the special effects source data to extract motion feature vectors, generate predicted position sequences based on the motion feature vectors, and calculate the spatial gradient divergence of the motion feature vectors to generate a motion complexity distribution map. The load calculation unit is used to estimate the coloring complexity of the special effects source data to generate an initial load map, and to couple the motion complexity distribution map with the initial load map to generate a comprehensive load map. The region segmentation unit is used to determine the load threshold based on the comprehensive load map and frame rate constraint parameters, and to generate a spatial segmentation mask by binarizing the comprehensive load map according to the load threshold. Based on the spatial segmentation mask, the special effects source data is separated into core region data and edge region data. The hierarchical rendering unit is used to perform full-precision rendering on the core area data to generate core rendering results, and to perform reduced-precision rendering on the edge area data to generate edge rendering results. The deviation compensation unit is used to map the predicted position sequence to the core rendering result to extract the rendering value, calculate the difference between the rendering value and the predicted value of the predicted position sequence to generate a prediction deviation field, and compensate the edge rendering result based on the prediction deviation field to generate a compensated edge rendering result. The result synthesis unit is used to synthesize the core rendering result and the compensated edge rendering result, and output the final rendering result.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images