Game picture dynamic interactive layout method and system

Abstract

The present application relates to the field of computer graphics and human-computer interaction technology, especially to a game picture dynamic interaction layout method and system, the method comprising: obtaining depth, normal, illumination, color and user event data to generate a field of attention; extracting an interest region according to the field of attention, obtaining two displacement fields by using neural optimal transport and quasi-conformal mapping respectively, and fusing the two displacement fields into a unified displacement field according to curvature weight; after deforming pixel coordinates by using the unified displacement field, inputting a voxel set of the interest region into a neural radiance field network to generate a compensation color, and combining the compensation color with a deformed color map to generate a fusion picture; inputting a user event through two-stage inverse mapping to hit an entity, and when the system energy is low, frame skipping is used for multiplexing, and when the system energy is high, threshold, conformal coefficient and damping are adjusted through time sequence difference reinforcement learning to realize cross-frame self-adaptation. The present application takes into account local magnification clarity and global continuity, reduces dizziness and saves computing power.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer graphics and human-computer interaction technology, and particularly relates to a game picture dynamic interactive layout method and system. BACKGROUND

[0002] In real-time interactive entertainment, the game picture not only bears information presentation, but also directly determines the operation accuracy and immersion of the player. The traditional "fixed interface + lens zoom" or "single viewport clipping" scheme can only magnify the whole or switch scenes, and it is difficult to balance the highlighting of local details and the coherence of the whole, resulting in experience defects such as too small target, interactive misplacement or dizziness in high-speed shooting, multi-player competition and mobile touch games. The industry generally tries to use multi-layer UI superposition, off-screen rendering or multi-resolution buffer to solve the problem, but these means either increase bandwidth and delay or destroy the consistency of the rendering pipeline.

[0003] The existing method usually magnifies the interesting area based on single-scale texture resampling or by reconstructing the viewport after clipping: a fixed scaling matrix is used, ignoring the screen geometric continuity, which is easy to produce "tearing" at the boundary. Some schemes use a mask map to magnify the interesting area, and then use bilinear interpolation to fill the holes. Due to the lack of global quality conservation, it leads to background holes or ghosting. There are also schemes that use convolution super-resolution to improve local clarity, but the network inference overhead is large and cannot maintain the correspondence of real pixels, and the hit detection is still inaccurate. The root cause is that they do not consider the three constraints of "global quality conservation, local angle preservation and real-time interactive coordinate consistency" at the same time, so distortion, delay or misjudgment inevitably occurs in high-speed zoom or multi-player scenes. SUMMARY

[0004] In view of the many problems existing in the prior art, the present application provides a game picture dynamic interactive layout method and system. The present application drives the attention field, and successively constructs a global quality conservation displacement field and a local angle-preserving displacement field, fuses them into a unified displacement field according to the curvature weight, deforms the screen pixels, and then uses a neural radiation field to real-time re-render the color of the interesting area, and adjusts the parameters online through a system energy-reinforcement learning closed loop.

[0005] A game picture dynamic interactive layout method, comprising the following steps:

[0006] Obtain a depth map, a normal map, an illumination map, a color map and user input event data every frame, input the obtained data into a neural network and generate an attention field combined with symplectic integration;

[0007] Determine an interesting area based on the attention field, construct a source probability density and a target probability density, respectively obtain a first displacement field and a second displacement field, and fuse the first displacement field and the second displacement field according to a preset weight to generate a unified displacement field;

[0008] The uniform displacement field is used to deform pixel coordinates, and a voxel set corresponding to the region of interest is input into a neural radiance field network to obtain a compensated color map, which is combined with the deformed color map to generate a fusion picture.

[0009] For a user input event on the fusion picture, original screen coordinates are determined through inverse operation of the uniform displacement field and the second displacement field, an entity identifier is obtained by querying the deformed entity identifier map, and the attention field threshold, the second displacement field parameter and the damping parameter are adjusted when the system energy exceeds a preset interval, so as to realize cross-frame adaptive feedback.

[0010] Preferably, when the attention field is generated, the neural network is sequentially composed of a multi-head self-attention layer and a feedforward layer, and the feature vectors of the depth map, the normal map, the illumination map and the color map are included in the same attention window in the multi-head self-attention layer.

[0011] Preferably, the first displacement field is obtained by performing Sinkhorn iteration on a discrete grid to solve the optimal transport mapping between the source probability density and the target probability density, and the mapping matrix is normalized by rows and columns after each iteration.

[0012] Preferably, the second displacement field is obtained by calculating the Beltrami coefficient using the grayscale value of the attention field and maintaining the screen boundary identity mapping in the Koebe iteration process.

[0013] Preferably, the preset weight is allocated according to the Laplace curvature of the first displacement field and the second displacement field on the pixel grid, and the first displacement field and the second displacement field are linearly superimposed to generate the uniform displacement field during fusion.

[0014] Preferably, the voxel set corresponding to the region of interest is obtained by writing the pixel coordinates into a hash grid after back-projection to three-dimensional coordinates, and the three-dimensional coordinates and the light direction are respectively Fourier-encoded as inputs of the neural radiance field network.

[0015] Preferably, when the original screen coordinates are determined, the screen space inverse displacement is first performed using the uniform displacement field, and then the quasi-isogonal inverse mapping is performed using the second displacement field, wherein the inverse displacement is calculated by bilinear interpolation.

[0016] Preferably, the system energy is obtained by weighting the sum of the attention field variation and the uniform displacement field curvature on the same pixel grid by a fixed coefficient; when the attention field threshold, the second displacement field parameter and the damping parameter are adjusted, a time-difference reinforcement learning algorithm is used to reduce the difference between the system energies of adjacent two frames as the target.

[0017] Preferably, when the system energy is lower than the preset lower limit and the attention field change amount is lower than the preset threshold, the last frame uniform displacement field and the last frame second displacement field are reused, and the steps of determining the interest region based on the attention field and deforming the pixel coordinates based on the uniform displacement field are skipped.

[0018] A game picture dynamic interactive layout system for implementing the game picture dynamic interactive layout method, the system comprises:

[0019] An attention field generation module is configured to acquire a depth map, a normal map, an illumination map, a color map and user input event data in each frame, input the data into a neural network and combine the Simpson integral to generate an attention field consistent with the screen resolution;

[0020] A displacement fusion module is configured to determine an interest region according to the attention field, construct a source probability density and a target probability density, obtain a first displacement field and a second displacement field respectively, and fuse the first displacement field and the second displacement field according to a preset weight to generate a uniform displacement field;

[0021] A compensation rendering module is configured to deform pixel coordinates based on the uniform displacement field, input a voxel set corresponding to the interest region into a neural radiance field network to obtain a compensation color map, and synthesize and fuse the compensation color map and the deformed color map into a fusion picture;

[0022] An interactive feedback module is configured to determine original screen coordinates by sequentially performing inverse operations on the uniform displacement field and the second displacement field on a user input event on the fusion picture, query an entity identification map to obtain an entity identification, and adjust the attention field threshold, the second displacement field parameter and the damping parameter when the system energy exceeds a preset interval to realize cross-frame adaptive feedback.

[0023] Compared with the prior art, the application has the following advantages and beneficial effects:

[0024] Through neural optimal transmission mapping + quasi-conformal mapping pixel-level fusion, a uniform displacement field that magnifies the interest region while maintaining global quality conservation and local angle continuity is realized; through voxel hashing + neural radiance field compensation rendering, real-time reconstruction and seamless fusion of texture details in the magnified region are realized; through two-level inverse mapping and system energy adaptive parameter adjustment, closed-loop control of user input hit accuracy and cross-frame visual smoothness is realized; through the frame skipping reuse strategy, power saving and frame rate stability in a static scene are realized. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The flowchart of the application is shown in the figure;

[0026] Figure 2 The displacement field fusion schematic diagram in the application is shown in the figure;

[0027] Figure 3 This is a schematic diagram of the two-level inverse mapping in this invention;

[0028] Figure 4 This is a structural block diagram of the system of the present invention. Detailed Implementation

[0029] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation.

[0030] like Figure 1 As shown, a method for dynamic interactive layout of game screens includes the following steps:

[0031] For each frame, a depth map, normal map, illumination map, color map, and user input event data are acquired. The acquired data is then input into a neural network and combined with symplectic integrals to generate an attention field.

[0032] After each frame is rendered, this invention simultaneously copies the depth map, surface normal map, lighting map, and color map, and reads the user input event data at the current moment. The four types of images are located in the same view frustum, with corresponding pixels matched one-to-one. They can be directly concatenated into a multi-channel tensor in video memory and fed into the attention field generation network. This network adopts a multi-head self-attention structure: for each pixel location, the depth value, normal components, lighting intensity, and color components are merged to obtain an 8-dimensional feature vector, which is then linearly transformed and mapped to a unified hidden dimension before entering the self-attention operator. The self-attention operator calculates feature correlations in the spatial dimension, ensuring that the impact of lighting changes on depth occlusion is synchronously transmitted with user input hotspots, forming a fused attention weight that includes geometric, material, and interaction information.

[0033] To avoid the lag caused by relying solely on the static estimation of the previous frame, this invention introduces symplectic integral prediction. Let the difference between the attention fields of two frames be... ,Will Treating it as generalized velocity, the current field of attention As a generalized coordinate system, a diagonal mass matrix is ​​introduced. With potential energy term Using the Lewis-Frog format, for In time step Internal execution:

[0034]

[0035]

[0036]

[0037] in is the attention field, and is its conjugate momentum. The symplectic scheme guarantees the conservation of Hamiltonian in the discrete time step, which suppresses the drift of long-time integration and provides smooth prediction when the user quickly slides or the camera dramatically transforms. The predicted attention field is linearly fused with the network output attention field by 7 to 3, which preserves the real-time feature and continuity.

[0038] The attention field plays a dual role in the subsequent steps of the present application: on the one hand, it determines the region of interest by threshold segmentation, providing input for the probability density mapping and displacement field calculation; on the other hand, it measures the frame difference participates in the system energy evaluation, driving the adaptive adjustment logic. Since the resolution of the attention field is consistent with the screen, the downstream displacement field solving does not need to be rescaled, reducing the interpolation error.

[0039] In the adaptation experiment of mobile devices, when the rendering resolution is reduced to 1280x720 and the hidden dimension is reduced to 64, the total calculation time delay of attention field generation and prediction is maintained within 2 milliseconds; although the model size is halved, the symplectic integral prediction can still maintain smooth tracking, reducing the aiming point jitter by about 33% in touch shooting. The results show that the present application has good scalability for hardware resources.

[0040] Preferably, when generating the attention field, the neural network is sequentially composed of a multi-head self-attention layer and a feedforward layer, and the feature vectors of the depth map, the normal map, the illumination map and the color map are included in the same attention window in the multi-head self-attention layer.

[0041] In the game screen dynamic interactive layout method of the present application, the attention field is responsible for depicting the attention intensity of each pixel in the screen in the current frame, and is the only weight source for subsequent interest region extraction, displacement field solving and interactive feedback. In order to consider the geometric depth, surface material, lighting conditions and player input intention, the present application uses a neural network sequentially stacked by a multi-head self-attention layer and a feedforward layer to generate the attention field, and puts the feature vectors of the depth map, the normal map, the illumination map and the color map into the same attention window in the multi-head self-attention layer, realizing cross-modal fusion.

[0042] The network input is first spliced by pixels in the video memory: the depth map provides 1-dimensional distance values; the normal map provides 3-dimensional direction cosines; the illumination map gives 1-dimensional illuminance per pixel; the color map provides 3-dimensional color components in the sRGB space. The total of 8-dimensional features are projected to a unified hidden dimension by a linear layer, and then enter the multi-head self-attention layer. Each attention head independently calculates the query vector, the key vector and the value vector. Let the hidden dimension be , the query matrix, the key matrix and the value matrix are denoted as , , , which means that the pixel embedding is the result of three linear transformations under the current head. The self-attention weight is obtained by the following formula:

[0043]

[0044] in This is a normalization factor used to prevent gradient vanishing as the dimension increases. In the formula... This represents the similarity matrix between the query and the key; the softmax function ensures that the sum of the pixel weights after row normalization is always equal to 1. Multiple attention heads are computed in parallel and then concatenated by channel, and then linearly mapped back to the hidden dimension to form fused features.

[0045] In multimodal input scenarios, depth and normal information describe scene geometry, lighting describes local brightness differences, and color maps record texture details. Self-attention weights explicitly model the cross-pixel correlations of these features, avoiding the limitation of traditional convolutions that can only capture local neighborhoods. For example, when a player uses a flashlight to illuminate an enemy in a dim environment, bright spots in the lighting map and foreground pixels highlighted in the depth map will receive higher weights through the self-attention mechanism, thus correctly identifying the interaction focus.

[0046] A multi-head self-attention layer is followed by two feedforward layers. The feedforward layers first use an activation function to enhance the nonlinear expression, and then use residual connections and layer normalization to maintain gradient stability. The network outputs a single-channel grayscale image, which is then mapped to 0 to 1 after sigmoid activation, resulting in an attention field consistent with the screen resolution. The attention field is not only used for thresholding but also participates in system energy measurement, therefore it must have inter-frame continuity. To this end, this invention introduces symplectic integral prediction on the network output: the difference between the attention fields of the previous two frames is regarded as generalized velocity, and the attention field of the current frame is regarded as coordinates, constructing a harmonic potential energy:

[0047]

[0048] in This indicates the attention field grayscale tensor. Its spatial gradient. Using the Lewis-Frog form... Performing half-step momentum update, full-step coordinate update, and then half-step momentum recovery ensures the conservation of discrete Hamiltonian. The prediction results are fused with the network output at a 7:3 ratio, providing one to two frames of look-ahead smoothing when the user is quickly dragging the camera or using multi-finger zooming, while avoiding excessive lag in static scenes.

[0049] In the motion sequence of the player cutting the gun and dodging sideways at the same time, the attention field generated by the application can accurately locate the position of the gun holder and the gun sight in the first frame; if the sinc integral is disabled, only the convolution attention network is relied on, and the weight can only be stabilized after two frames, resulting in a lag in the extraction of the interesting region.

[0050] In the motion sequence of the player cutting the gun and dodging sideways at the same time, the attention field generated by the application can accurately locate the position of the gun holder and the gun sight in the first frame; if the sinc integral is disabled, only the convolution attention network is relied on, and the weight can only be stabilized after two frames, resulting in a lag in the extraction of the interesting region.

[0051] As shown in Figure 2 , the source probability density and the target probability density are constructed based on the determination of the interesting region based on the attention field, the first displacement field and the second displacement field are obtained, and the first displacement field and the second displacement field are fused according to a preset weight to generate a unified displacement field;

[0052] Before generating the unified displacement field, the application first determines the interesting region according to the attention field. The attention field is a gray-scale image consistent with the screen resolution, and the greater the pixel value, the more worthy of highlighting the current frame. The attention field is binarized using a fixed threshold, and the connected pixel blocks constitute the interesting region. The interesting region is preserved in the pixel coordinate system to avoid precision loss in subsequent coordinate transformation.

[0053] In order to ensure that the interesting region is visually continuous with the background region after magnification, and at the same time avoid cutting off the player's global spatial sense, the application regards the interesting region and the background region as quality distributions respectively: the interesting region counts the number of pixels and the average weight of pixels, and the background region is obtained by normalizing the remaining pixel weights. Divide the entire attention field by the total weight to obtain the source probability density; after superimposing all the interesting regions according to the weighted Gaussian kernel, divide by the total weight to obtain the target probability density. The source probability density and the target probability density are defined on the same pixel grid, and the sum of the probability values at each pixel is 1.

[0054] In order to accurately map from source probability density to target probability density, the present application adopts neural optimal transport method. Optimal transport is a theory of finding the mapping between probability distributions under a given cost function; the present application takes pixel Euclidean distance square as the cost function. Double random matrix is initialized in memory, then Sinkhorn iteration is performed on discrete grid, i.e. normalizing matrix by row and by column alternately, accompanied by coefficient decay, to realize approximate entropy regularized optimal transport. Neural optimal transport network expands Sinkhorn iteration into trainable layers, uses synthesized region of interest for self-supervision in training phase, and only needs a small number of iterations to converge in inference phase. After convergence, a displacement vector map corresponding to pixels is obtained, called first displacement field, which guarantees mass conservation in global range, i.e. no holes or excessive overlap after pixel block enlargement.

[0055] However, the first displacement field only focuses on probability conservation and does not guarantee local angle relationship. To avoid texture distortion within the region of interest, the present application introduces a second displacement field. The second displacement field is obtained through quasi-conformal mapping: first, calculate the Beltrami coefficient according to the grayscale value of the attention field. The Beltrami coefficient is used to measure the size of angle distortion of the mapping at each pixel, and the closer the value is to 0, the closer it is to conformal. Take the Beltrami coefficient as input and use Koebe iteration to solve quasi-conformal mapping. Koebe iteration fixes the boundaries around the screen, so the background can still maintain complete envelope after mapping. After completing the iteration, the second displacement field is obtained, which approximately maintains the angle within the region of interest and reduces the problem of bending of vertical lines of building or stretching of character face after enlargement.

[0056] The first displacement field focuses on global mass conservation, and the second displacement field focuses on local angle preservation, with different goals. The present application dynamically allocates fusion weight according to the Laplace curvature of the two displacement fields on the pixel grid. The Laplace curvature can approximately represent the deformation intensity of the displacement field in the local area, and the greater the curvature, the more dramatic the displacement field changes. Let the curvature of the first displacement field be , and the curvature of the second displacement field be , then the fusion weight is defined as: , and the fusion formula is: , wherein is the unified displacement field, is the first displacement field, is the second displacement field. Here all refer to pixel displacement vector map, is a pixel-level scalar weight. Using pixel-level weight instead of a single global coefficient can make the weight transition smoothly at the boundary and avoid displacement field creases at the edge of the region of interest.

[0057] The uniform displacement field is stored in a texture in the video memory and is directly sampled in the subsequent morphing rendering stage. In the rendering pipeline, the uniform displacement field is used to perform screen space displacement for each vertex, so that the region of interest is enlarged on the screen. Since the uniform displacement field has been smoothed at the pixel level, the background and the region of interest are continuous after morphing, and the texture distortion in the region of interest is controlled.

[0058] Preferably, the first displacement field is obtained by performing Sinkhorn iteration on a discrete grid to solve the optimal transport mapping between the source probability density and the target probability density, and the mapping matrix is row-normalized and column-normalized after each iteration.

[0059] After the attention field is generated and the region of interest is obtained, the present application needs to controllably enlarge the region of interest in the screen space without destroying the connectivity of the entire frame. To this end, the present application first regards the entire attention field as a quality distribution, divides the pixel gray scale by the total gray scale to obtain the source probability density, and then constructs the target probability density using the statistical information of the region of interest. The construction process of the target probability density is as follows: the pixel centroid and the covariance of each region of interest are extracted, a Gaussian kernel is written into the pixel grid, and then the pixel grid is weighted according to the average gray scale of the region of interest, and finally normalized. In this way, the source probability density and the target probability density are defined on the same grid, and the integrals of the two are both 1.

[0060] To accurately map the source probability density to the target probability density at the pixel level, the present application uses the entropy regularized optimal transport theory. The core of optimal transport is to determine a mapping matrix, the rows and columns of which correspond to the pixel positions on the source grid and the target grid respectively, and the elements of which represent the quality flow of a single pixel pair. The present application selects the square of the pixel Euclidean distance as the unit flow cost, constructs a cost matrix and initializes a doubly random matrix with non-negative elements in the video memory of the graphics processing unit; then the optimal solution is approximated by Sinkhorn iteration. In each iteration, the matrix is first row-normalized, then column-normalized, and the two steps are alternated to gradually approximate the optimal transport mapping while keeping the row and column marginals unchanged.

[0061] To facilitate end-to-end fine-tuning in the training stage, the present application expands the Sinkhorn iteration into a differentiable network layer. Let the mapping matrix be , the row normalization vector be , and the column normalization vector be , then one round of iteration can be abstracted as , where represents the cost matrix, is the regularization coefficient, represents writing a vector into a diagonal matrix. The row vector and the column vector are obtained by left or right multiplying the mapping matrix by the target marginal and then taking the inverse. Afterwards, the displacement vector of each pixel can be extracted from the mapping matrix, forming the first displacement field.

[0062] Since all operations are performed on the pixel grid without vertex or fragment supersampling, the first displacement field is naturally aligned with the original frame resolution and satisfies global quality conservation: after zooming in the region of interest, its pixels come from the shrinking background region instead of being interpolated out of thin air, avoiding the generation of fuzzy rings or jagged gaps.

[0063] The purpose of unfolding Sinkhorn iteration into a trainable layer is to reduce inference time. By performing self-supervised training on synthetic random regions of interest, the network can approach 90% row-by-column balanced accuracy after 3 iterations, while the traditional 20-step iteration consumes more GPU bandwidth. In the inference stage, the source and target marginal vectors are stored in the GPU memory at one time, and the linear bandwidth is proportional to the resolution, which can meet the real-time demand of 60 frames at 4K resolution.

[0064] Example 3: In a first-person shooter game with a resolution of 1920x1080, the first displacement field is generated by selecting 0.01. During offline training, the network is iterated for 8 rounds using 50,000 synthetic heat maps; during online inference, 3 row and column normalization operations are performed, consuming 0.56 milliseconds. The generated first displacement field produces a screen space displacement of about 50 pixels at the center of the region of interest, which can enlarge the enemy's face from 24 pixels to about 40 pixels; at the same time, the background pixels automatically shrink according to the mapping matrix row and column normalization, and there are no gaps between pixels. Compared with the scheme using only nearest neighbor interpolation to enlarge, the former improves the peak signal-to-noise ratio by 2.4 decibels and reduces the edge misregistration rate by 38%.

[0065] In the mobile terminal experiment, the resolution is reduced to 1280x720, and remains unchanged. The inference time is reduced to 0.28 milliseconds, and the mapping matrix occupies less than 12 megabytes of GPU memory, realizing the cooperation of region of interest enlargement and background shrinkage. The results show that it is feasible to use Sinkhorn iteration to solve the optimal transport mapping on a graphics processing unit, and the proposed method has higher pixel consistency and real-time performance compared with the traditional convolutional enlargement or viewport cropping scheme.

[0066] Preferably, the second displacement field is obtained by calculating the Beltrami coefficient using the grayscale value of the attention field and maintaining the screen boundary identity mapping during the Koebe iteration process.

[0067] After the first displacement field completes global mass conservation, the present application introduces a second displacement field to reduce the angular distortion inside the region of interest. The core idea is to use quasi-conformal mapping (also known as quasi-conformal mapping) to locally approximate the shape, thereby avoiding stretching of text, character faces or building columns. The implementation process is divided into three steps: pixel-level Beltrami coefficient calculation, Koebe iteration to solve the mapping, and boundary identity constraint to maintain panoramic connectivity.

[0068] The first step is to regard the attention field gray scale image as a scalar function, and the higher the gray scale, the more the pixel needs to be enlarged. Map the pixel coordinates to the complex plane, and denote each pixel point as a complex variable . Set the control quantity of the mapping function as the Beltrami coefficient . The Beltrami coefficient essentially measures the degree of angular distortion of the mapping at that point, and its definition is: , where and represent the complex partial derivatives of and its conjugate , respectively. If is 0, the mapping is completely conformal; when is close to 1, the mapping distortion is maximum. To ensure that the region of interest is flexibly enlarged and the background changes smoothly, the present application normalizes the attention field gray scale to the interval through linear mapping, and then multiplies it by a radius weight that increases in the region of interest and decreases at the edge, to obtain the pixel-level .

[0069] The second step is to use Koebe iteration to solve the quasi-conformal mapping. Koebe iteration updates the mapping function on a discrete grid in the complex plane, so that the new mapping converges to the Beltrami equation . The pseudo code of the iteration can be expressed as: initialize ; for each pixel, perform a linear transformation: , where is the step factor, is the conjugate of the current gradient. After updating, do Laplacian smoothing on to suppress high-frequency noise. Iterating 3 to 5 times can make the mapping gradient change less than 0.5 pixels. Since Koebe iteration only depends on local difference operators, it can be completed in parallel by a computing shader on a graphics processor, and the time complexity is linearly related to the number of pixels.

[0070] The third step is the boundary identity constraint: to prevent the out-of-screen frame from tearing due to internal deformation, the present application fixes the mapping of the outermost 1 pixel as identity before each Koebe iteration, i.e. Simultaneously, only pixels within the boundary are updated during iteration. This ensures that the central region of interest can deform freely, while the surrounding background remains in situ and continuous. After iterative convergence, from... Get the pixel displacement vector: Thus, the second displacement field is obtained. Here and These represent the real and imaginary parts of a complex number, respectively.

[0071] In a text-based reading game, an annotation was designated as a region of interest. If only the first displacement field was used for magnification, the text strokes appeared bent. Adding a second displacement field, which uses conformal components primarily within the text, maintained the original stroke thickness, improving the reading comfort score from 3.2 to 4.5. The hardware environment was a desktop graphics processor with a resolution of 2560×1440. Four Koebe iterations took 0.62 milliseconds, representing 2.5% of the total rendering budget.

[0072] Another example is in racing games, where the region of interest (ROI) is positioned at the entrance / exit of the curve when players are driving at high speeds. Conformal components keep the road surface texture flat, reducing dizziness caused by the sense of speed. Compared to a solution without conformal components, players reported a 28% reduction in dizziness scores in continuous curve scenarios. Mobile experiments at a resolution of 1600×900 reduced the number of iterations to 3 and the execution time to 0.35 milliseconds, while still achieving similar distortion suppression effects.

[0073] By adjusting the Koebe iteration using pixel-level Beltrami coefficients, this invention provides local angle preservation in the region of interest (ROI) while maintaining overall screen coherence through boundary identity, providing controllable conformal components for the unified displacement field. After fusion with the global mass-conserving displacement field, it ensures that the ROI is sufficiently magnified while suppressing detail distortion, significantly improving the dynamic layout experience.

[0074] Preferably, the preset weights are allocated according to the Laplacian curvature of the first displacement field and the second displacement field on the pixel grid, and the first displacement field and the second displacement field are linearly superimposed during fusion to generate a unified displacement field.

[0075] The unified displacement field must inherit the global mass conservation of the first displacement field while preserving the local angular continuity of the second displacement field. This invention employs pixel-level weighted linear superposition to achieve synergy between the two, where the weights are adaptively allocated based on the magnitude of the Laplacian curvature of the two displacement fields on the same pixel grid. The Laplacian curvature measures the deformation intensity in the displacement vector field; essentially, it is the second-order Laplacian curvature that measures the deformation intensity in the displacement vector field, which is essentially the divergence of the second-order difference with respect to the displacement gradient. Let the first displacement field be in pixel... The horizontal and vertical components are denoted as follows: and The corresponding component of the second displacement field is denoted as With The approximation of the discrete Laplacian to the horizontal component can be written as:

[0076]

[0077] The same operation is done for the vertical component, resulting in The curvature at a pixel is defined as:

[0078]

[0079] Curvature of the second displacement field The same formula is used. To ensure that the contribution of the two displacement fields to the unified displacement field varies smoothly with the local deformation strength, the present invention performs one spatial smoothing on the curvature map using bilateral filtering in the graphics processor, with edge preservation to avoid sharp jumps at the boundaries of the region of interest. The pixel-level weight is given in inverse proportion to the curvature:

[0080]

[0081] where is a small constant to prevent the denominator from being zero. The closer to 1, the greater the contribution of the second displacement field at that pixel; the closer to 0, the first displacement field dominates. Substitute into the linear superposition:

[0082]

[0083]

[0084] The horizontal and vertical components of the unified displacement field are obtained as , This weighting method usually presents a distribution of large curvature of the first displacement field and small curvature of the second displacement field inside the region of interest, so is biased to 0.6 to 0.8, making the angle-preserving component dominant; at the background with fine texture, the curvatures of the two fields are close to about 0.5, maintaining the balance between quality conservation and angle preservation.

[0085] After the generation of the unified displacement field, pixel-level noise is easy to appear. The present invention adds one cubic spline filtering in the graphics processor: to , The three-point moving average is performed along the row and column directions respectively, and then a bidirectional interpolation is performed again; this operation does not destroy the mass conservation or the conformal property, but can significantly reduce the jagged edge. The filtered displacement field is written back to the video memory in the form of a double-channel 16-bit floating-point texture format for subsequent mesh shader sampling. Since all calculations are local convolution and linear combination, the time complexity increases linearly with the number of pixels, and the desktop resolution of 1920x1080 takes about 0.21 milliseconds, and the mobile resolution of 1280x720 takes about 0.11 milliseconds.

[0086] In the text adventure game, when the player clicks the screen, the attention field generates a weight peak in the dialog box area. The first-order curvature analysis shows that the average curvature of the first displacement field at the edge of the dialog box is 3.4 pixels, and the average curvature of the second displacement field is 1.1 pixels, and the weight distribution generated is between 0.24 and 0.38. The uniform displacement field makes the dialog box magnify by about 1.4 times, and the font does not appear to be tilted. Compared with the no-curvature weight scheme, the edge pen thickness increases the error from 21% to 7%.

[0087] In the high-speed turning scene of the racing game, the area of interest falls on the front inner bend. The curvature peak of the first displacement field reaches 7.8 pixels, and the peak of the second displacement field is 2.0 pixels, and the curvature weight reaches 0.78 at the center pixel and decreases to 0.46 at the edge pixel. In the rendering result, the shoulder line remains straight and the texture does not appear distorted; when using only the first displacement field scheme, the shoulder appears obvious wave shape. The player's dizziness score in the test of 10 consecutive bends is reduced by 25%.

[0088] The deformed pixel coordinates are obtained by using the uniform displacement field, and a voxel set corresponding to the area of interest is input into a neural radiance field network to obtain a compensated color graph, and the deformed color graph is synthesized into a fusion picture.

[0089] After the uniform displacement field is generated, the present application directly samples the displacement texture in the mesh shading stage, performs screen space deformation for each vertex, and the deformed pixel coordinates participate in subsequent color synthesis. In order to avoid the appearance of jagged edges, blurring or loss of details after the area of interest is magnified, the present application does not simply use the original color graph, but introduces a neural radiance field network to re-render the voxel set corresponding to the area of interest, obtains a compensated color graph, and generates a fusion picture by pixel coverage with the deformed color graph.

[0090] First, the principle of voxel set construction is introduced. The area of interest before deformation is a set of pixel indexes in screen space; by using the depth map, these pixels are back-projected to the camera coordinate system to obtain a three-dimensional point cloud. In order to efficiently organize in the video memory, the present application uses a hash grid to map the three-dimensional coordinates to hash buckets, and stores the density and color parameter index of the fixed-size voxel block in each bucket. This approach can be written in parallel on the graphics processor side at one time, avoiding thread divergence caused by pointer jumping.

[0091] The voxel set is then sent into the neural radiance field network. Neural radiance field is a kind of implicit field that predicts spatial density and direction-dependent radiance simultaneously through a multi-layer fully connected network. The present application selects an 8-layer fully connected structure with 128 hidden units per layer. In order to improve the convergence speed, the three-dimensional position coordinates and the line-of-sight direction are respectively Fourier encoded, so that the high-frequency geometric details are more easily fitted by the network. The input rays are uniformly sampled at 48 depth points within the voxel block, and the voxel density of each sampling point output by the network is denoted as , and the color vector corresponding to the line-of-sight direction is denoted as . According to the volume rendering formula:

[0092]

[0093]

[0094] The pixel-level compensation color can be accumulated, wherein is the distance between adjacent samples. represents the spatial density of the sampling point, represents the radiance color of the sampling point, represents the contribution weight of the sample to the line-of-sight transmittance. The whole ray summation process is implemented using parallel reduction in the fragment shader, and the calculation delay is linearly related to the sampling points, with an average time consumption of 0.74 milliseconds on the desktop.

[0095] When the compensation color map and the deformed color map are synthesized, the principle of "interest region priority" is followed: if the pixel falls within the screen range of the deformed interest region, the compensation color is used; otherwise, the deformed color is retained. In order to ensure smooth transition, the present application inserts a 4-pixel-wide blending band at the edge of the interest region, and blends the compensation color and the original color according to the bilinear weight, avoiding the hard edge interface.

[0096] In the dialogue close-up scene of the role-playing game, the face of the character is set as the interest region. The uniform displacement field is enlarged by about 1.8 times. After compensation by the neural radiance field, the facial skin texture retains high-frequency details, and the hair edge is free of jaggies; compared with the bilinear enlargement using only the original color map, the peak signal-to-noise ratio is improved by 2.7 decibels, and the structural similarity index is improved by 0.06. When the mobile terminal resolution is 1280x720, the voxel block edge length is set to 32, the number of sampling points is reduced to 32, the compensation delay is controlled to be 0.38 milliseconds, and the total rendering frame rate is stable at 60 frames. In the perception test, the average score of 20 players in the clarity of the fast zoom-in scene is improved by 27%.

[0097] Preferably, the voxel set corresponding to the interest region is obtained by writing the pixel coordinates into a hash grid after back-projection into three-dimensional coordinates, and the three-dimensional coordinates and the ray direction are respectively Fourier encoded and used as the input of the neural radiance field network.

[0098] After constructing the unified displacement field, this invention employs a neural radiation field network to resynthesize the colors of the regions of interest (ROIs) to avoid texture blurring or jagged edges caused by magnification of the ROIs. For the neural radiation field to function in real-time, the primary task is to accurately map screen-space pixels to three-dimensional space, push them into video memory in sparse voxel form, and use an efficient encoding method as network input.

[0099] First, let's explain the principle of back projection. During the rasterization stage of the rendering pipeline, the system outputs a depth value for each fragment. This depth value, combined with known camera intrinsic parameters, allows the reconstruction of the fragment's spatial position in the camera coordinate system. Let the screen resolution be... pixel coordinates are The back projection formula is:

[0100]

[0101] in It is a three-dimensional coordinate vector. For depth map values, for Intrinsic parameter matrix, This is the inverse matrix of the intrinsic parameters. Parallel back projection can instantly transform pixels in the region of interest into a 3D point cloud. Next, we introduce the hash grid organization. The number of pixels in the region of interest can reach hundreds of thousands; directly storing voxels as arrays would lead to memory fragmentation and random access latency. This invention uses fixed voxel side lengths. The three-dimensional space is divided into cubic grids, and the grids are indexed. Using a multiplicative mixed hash function: ,in For bitwise XOR, To shift to the left, The number of buckets is specified. Hash collisions are resolved using open addressing, with collision chains not exceeding 3 in length. Each bucket stores the density and color index of a voxel block. When the region of interest changes, only the corresponding point cloud in the hash bucket needs to be updated, avoiding global reconstruction. Experiments show that at a resolution of 1920×1080, using 32×32×32 voxel blocks in the hash grid can keep the voxel storage size within 24 megabytes.

[0102] The third step is Fourier encoding. Neural radiation field networks use a multi-layer fully connected structure, and it has been verified that low-frequency inputs often lead to difficulties in learning high-frequency details. Therefore, this invention addresses this by using three-dimensional position vectors... Perform multi-frequency Fourier mapping: Simultaneously, the unit vector of the line of sight. Do the same encoding, where This refers to the number of frequency bands. This invention sets... The input dimension is increased from 3 to 36, but the GPU memory consumption is still controllable due to the use of half-precision floating-point storage. The fitting ability of the Fourier encoding lifting network to high-frequency geometric details is particularly important when magnifying texture edge lines.

[0103] The fourth step describes the real-time rendering effect. After being organized by the hash grid, the voxel set of the interest area is sent to the neural radiance field network together with the encoding vector for inference. The network structure is 8 layers of full connection, with 128 hidden units per layer, and residual connections are used to alleviate gradient disappearance. The ray stepping uses uniform sampling, with a total of 48 depth points. The density and color are accumulated to obtain the pixel compensation color by the body rendering formula, and are overlaid on the color map after deformation.

[0104] Example 7: In a role-playing game, the player clicks to magnify the head of the character. The number of voxels in the interest area is about The hash grid allocation takes 0.12 milliseconds, the encoding and network inference take 0.62 milliseconds, and the overall delay is less than 1 millisecond. In the fusion picture, the hair strands and facial fine lines remain sharp, and the peak signal-to-noise ratio is improved by 2.8 decibels compared to the bilinear magnification scheme.

[0105] Example 8: In a cartoon rendering shooting game, the player quickly slides the touch screen to magnify the target. The mobile resolution is 1280x720, the voxel block length is 16, the number of ray sampling points is reduced to 32, and the total inference delay is 0.35 milliseconds. Due to the control of the hash bucket collision chain to 2, the random access memory hit rate remains 93%. The player's subjective score shows that there is no obvious jagged edge when the target is magnified, and the shooting accuracy is improved by 17%.

[0106] As shown in Figure 3 , for the user input event on the fusion picture, the original screen coordinates are determined by the inverse operation of the uniform displacement field and the second displacement field, the entity identifier is obtained by querying the deformed entity identifier map, and the attention field threshold, the second displacement field parameter and the damping parameter are adjusted when the system energy exceeds the preset interval, to realize cross-frame adaptive feedback.

[0107] After the fusion picture is generated, all input events of the player (including mouse clicks, touch point presses or handle cursor stays) occur in the deformed two-dimensional coordinate system. If these coordinates are directly sent to the traditional hit detection, they will be out of sync with the logical world: the interest area has been magnified by the uniform displacement field, the background has been compressed, and the real hit box of the entity has been dislocated from the screen pixels. The present application restores the original screen coordinates of the events through two-level inverse mapping, and then queries the entities, to ensure the interaction accuracy, and at the same time, the layout parameters of the next frame are adaptively adjusted with the help of the system energy closed loop, to ensure the visual comfort and stable computing power.

[0108] The first inverse mapping uses a uniform displacement field. The invention writes the uniform displacement field into a dual-channel 16-bit floating-point texture in the mesh shading stage, recording the horizontal and vertical displacement vectors of each pixel. When an event occurs, the displacement is obtained using the bilinear interpolation of the graphics processor with the event screen coordinates as the sampling center, and then the first step inverse displacement is obtained by coordinate subtraction. Because the uniform displacement field has been smoothed by cubic spline when generated, the sampling noise is constrained within 0.3 pixels; the bilinear interpolation is hardware accelerated on the native hardware unit of the graphics processor and can be completed within 0.01 milliseconds.

[0109] The second inverse mapping uses a second displacement field. The second displacement field is generated by a quasi-conformal mapping, which maintains local angle continuity. After the first-order inverse displacement, the inverse function of the quasi-conformal mapping is called again on the pixel coordinates. The quasi-conformal inverse mapping is essentially solving a composite function When the Koebe iteration is used to obtain the forward mapping, the invention simultaneously generates a sparse lookup table: the screen is divided into 64x64 grids, and the initial value of the reverse mapping is stored at the center of each grid; after the event coordinates fall into a certain grid, the initial value is obtained and iterated for 2 steps of Newton correction to converge, with an average time consumption of 0.05 milliseconds. The combined error of the two-level inverse transformation is measured to be no more than 0.5 pixels.

[0110] After obtaining the original screen coordinates, the invention queries the entity in the deformed entity identification map. The entity identification map is deformed synchronously with the color map, and the pixel value stores the entity number, which can cover 256 entity categories with a single byte. With the help of the event coordinate index map, the number can be returned in constant time, and then the event is packaged as <timestamp, entity number, event type, auxiliary parameter> and pushed to the logical thread. After the logical thread is processed, it will affect the scene state, thereby affecting the generation of the attention field and the displacement field in the next frame.

[0111] In order to prevent visual fatigue caused by excessive deformation, the invention calculates the system energy at the end of the logical thread. The system energy is composed of two parts: the attention field change and the curvature of the uniform displacement field. Let the current frame attention field be , the last frame attention field be , and the curvature of the uniform displacement field be , then:

[0112]

[0113] where , are constant coefficients. is the system energy scalar, is the total absolute difference of pixels, is the total absolute value of curvature, and both are unified in units of pixels. The energy measures the degree of visual deformation and the jitter of attention distribution, and the 1-norm of the two can be quickly and parallelly summed.

[0114] The invention sets an upper limit on the energy lower limit When above the upper limit, it indicates that the deformation amplitude is too large or the attention field jump is too severe, which may cause dizziness; when below the lower limit, it indicates that the picture is not enough to highlight the interactive focus, and the interest area effect is attenuated. In order to maintain a dynamic balance between the two thresholds, the application uses a time difference reinforcement learning algorithm to adjust the attention field threshold, the second displacement field Beltrami coefficient ratio and the damping parameter. The state vector of the reinforcement learning is , and the action vector is the parameter increment step. The objective function is: , where is the average of the upper limit and the lower limit. The network uses a two-layer perceptron to realize the policy approximation, and the number of parameters is 512, which does not increase the obvious memory pressure. The learning rate is updated once every frame online, and the learning rate is exponentially decayed after 500 frames. The actual measurement of the desktop update time is 0.03 milliseconds.

[0115] Preferably, when determining the original screen coordinates, first perform screen space inverse displacement using the uniform displacement field, and then perform quasi-conformal inverse mapping using the second displacement field, wherein the inverse displacement is calculated by bilinear interpolation.

[0116] After the fusion picture is generated, all user input events, whether mouse clicks, touch point presses or handle cursor drop points, are located in the screen coordinate system that has been deformed. If these coordinates are directly used for hit detection, the logic layer will misjudge the input position because the interest area has been enlarged and the background has been compressed. To ensure interaction accuracy, the application designs a "two-level inverse mapping" process: first, perform screen space inverse displacement using the uniform displacement field, and then perform inverse mapping using the second displacement field (quasi-conformal mapping). After two-level inverse operation, the original screen coordinates corresponding to the event can be obtained, and then the unique entity number is determined in combination with the deformed entity identification map, and finally a stable and accurate interaction loop is formed.

[0117] The first inverse mapping is based on the uniform displacement field. The uniform displacement field is stored in the video memory in the form of a double-channel 16-bit floating-point texture, and each pixel saves the horizontal displacement and the vertical displacement. When the event arrives, the event screen coordinates are used as the sampling center, the displacement vector is obtained by calling the graphics processor hardware bilinear interpolation function, and then the vector is directly subtracted from the event coordinates to obtain the first step inverse displacement. Since the uniform displacement field is subjected to three times of spline filtering in the generation stage, its gradient is smooth, and the bilinear interpolation error is less than 0.3 pixels; the hardware performs one interpolation only needs 4 times of texture reading, and the time consumption is about 0.01 milliseconds, which meets the requirements of high frame rate games.

[0118] The second inverse mapping depends on the second displacement field. The second displacement field is generated by quasi-conformal mapping, which maintains local angle continuity, but its forward function It is not necessarily easy to find the explicit inverse function. To ensure real-time performance, the present invention generates a sparse reverse lookup table simultaneously when Koebe iteration is used to find the forward mapping: divide the screen into 64x64 grid, and store the screen coordinates after forward mapping for each grid center pixel. After the event coordinates fall into a certain grid, select the nearest table entry as the inverse initial value, and use Newton correction for two iterations to converge. The Newton correction step only uses The numerical gradient in the neighborhood can be obtained by difference approximation, which does not increase the complexity of analytical derivation. The average time consumption of this inverse solution stage is 0.05 milliseconds, and the error is not more than 0.2 pixels.

[0119] Original screen coordinates After determination, the system queries the pixel value in the deformed entity identification map to obtain the entity number. The entity identification map and the color map are deformed by the same vertex shader to ensure consistency of the index; the pixel stores an 8-bit integer number, which can cover 256 entity categories. The lookup table operation is completed in the texture unit of the graphics processor, and the single access delay is less than 100 nanoseconds. At this point, the event is packaged as <timestamp, entity number, event type, and accompanying parameters>, and pushed to the logical thread.

[0120] In order to prevent visual fatigue caused by excessive deformation, the present invention evaluates the system energy at the end of the logical thread. The system energy is composed of two parts weighted by the frame difference of the attention field and the curvature of the uniform displacement field. Let the current frame of attention field be , the last frame of attention field be , and the frame difference be defined as . The absolute value of the uniform displacement field curvature is calculated by the discrete Laplace of the four-neighborhood and summed up. The energy calculation formula is: , wherein and are fixed coefficients. is the system energy scalar, and are both calculated in the pixel grid norm. The energy measures the intensity of visual deformation and the jitter of attention distribution, and the larger the value is, the more likely it is to trigger player discomfort.

[0121] The present invention sets the upper limit and the lower limit . When exceeds the upper limit, the system considers that the visual load is too high; when is lower than the lower limit, the system considers that the picture lacks a prominent focus. A time difference reinforcement learning algorithm is used to adjust the three key parameters: attention field threshold (affects the size of the area of interest), the proportion of the second displacement field Beltrami coefficient (affects the angle preservation intensity), and the damping coefficient (affects the interpolation smoothness of the displacement field). The state vector is taken as , and the action vector is the step size of the three parameters. The reward is set as: , wherein The two-layer perception machine strategy network outputs an action, and a logical thread is used to The learning rate is updated online; the network only contains 512 weights, which does not significantly increase the CPU / GPU burden. In practice, it converges within 300 frames online.

[0122] Example 9: Desktop first-person shooting, resolution 1920x1080, refresh rate 144 frames. When the gun is quickly swung, the system energy rises to 1.5 times the upper limit. Under reinforcement learning, the Beltrami coefficient is reduced by 15%, and the damping is increased by 10%, and the energy of the next frame is reduced to the safe area. Compared with the fixed parameter scheme, the dizziness score is reduced by 0.8 points, and the hit rate is increased by 5%.

[0123] Example 10: Mobile card game long time static. The system energy is continuously lower than the lower limit for 50 frames. The strategy network increases the interest area threshold by 8% and reduces the damping by 12%, and the card details are enlarged, and the reading comfort score is improved by 23%. All parameter tuning operations take 0.03 milliseconds, and do not affect 60 frame rendering.

[0124] Preferably, the system energy is obtained by weighting the sum of the attention field change amount and the curvature of the uniform displacement field on the same pixel grid by a fixed coefficient.

[0125] After the uniform displacement field and the attention field complete the layout, the present application quantifies the current frame visual load by the system energy, thereby driving the adaptive parameter tuning of the next frame. The system energy measures the picture "dynamic complexity" from two dimensions: one is the attention field change amount, which reflects the jumping amplitude of the picture focus in space; the other is the total curvature of the uniform displacement field, which reflects the strength of the screen geometric deformation. After the two are summed on the pixel grid and weighted by a fixed coefficient, a frame-level scalar is obtained. This scalar can not only simply evaluate the risk of dizziness, but also facilitate the reinforcement learning algorithm to call within a millisecond time window.

[0126] The attention field change amount directly corresponds to the displacement speed of the "player's visual attention area". If the interest is concentrated in the center of the screen in the last frame, and suddenly shifts to the edge of the screen in the next frame, the absolute value of the attention field gray difference is large, indicating that the visual attention center jumps sharply; a large number of experiments show that such jumps are more likely to trigger motion sickness if accompanied by large geometric deformation. The curvature of the uniform displacement field measures the second derivative of the displacement vector field by a discrete Laplace operator: high curvature means that the deformation gradient difference between pixels is large, and local stretching or contraction is obvious. If the total curvature sum rises within a frame, it means that the local bending or texture compression of the screen is intensified, which also increases the visual load.

[0127] To avoid the imbalance of absolute values with resolution changes, the present application uniformly calculates the two quantities at the final rendering resolution: the attention field change amount takes the pixel absolute difference and calculates the 1-norm; the total curvature calculates the discrete Laplace for the horizontal and vertical components of the displacement, respectively, takes the absolute value, and then calculates the 1-norm.

[0128]

[0129]

[0130] On the graphics processor, both can be completed by parallel reduction, without multiple memory round trips. and The system energy is obtained by combination: , the coefficient , The balance is obtained by offline calibration: if the game type emphasizes rapid shooting, the attention field weight is increased; if the scene roaming is emphasized, the curvature weight is increased, so that the values of the two are of the same order of magnitude. The attention field gray scale is , The displacement component Laplacian is and are two energy terms, respectively.

[0131] In terms of implementation, the application first reads two textures in the calculation shader of the graphics processor: one is the current frame attention field, and the other is the last frame attention field. After difference, the absolute value instruction is used for parallel summation; then the uniform displacement field texture is read, and the five-point template is applied to calculate the discrete Laplacian in the horizontal direction and the vertical direction, and the absolute value is taken and accumulated. Shared memory is used for block summation in the two reduction operations, and the block size is usually set to 16x16 to balance the delay and occupation. Under the resolution of 1920x1080, the two-step accumulation takes about 0.08 milliseconds.

[0132] After obtaining the system energy, it is compared with the fixed upper and lower limits. If the energy falls within the interval, the existing parameters are kept unchanged; if it exceeds the upper limit or is lower than the lower limit, the reinforcement learning strategy network is triggered to output the parameter adjustment action. The action space designed by the application includes: reducing or increasing the attention field threshold, scaling the Beltrami coefficient ratio, and adjusting the damping coefficient. The amplitude of each action is ±5% of the current value, ensuring that the picture is continuous frame by frame. The strategy network learns online, and the reward is set to the negative value of the center of the energy distance interval, encouraging the energy to quickly return to the interval.

[0133] In the high-speed shooting game, the player continuously sweeps the screen, causing the view angle to rotate at a speed of 720 degrees per second. This scene causes the attention field peak to jump from the left screen to the right screen in 5 frames, rising to 8.3x10x4; at the same time, the displacement field curvature also increases to 3.9x10x4 due to the continuous enlargement of the interest area. The two terms are weighted The system energy exceeds the upper limit by 1.2 times. The reinforcement learning instantly outputs the action of "reducing the Beltrami coefficient by 10% and increasing the damping by 5%"; the energy of the next frame decreases by 23%, and the energy of the third frame returns to the center of the interval. Compared with the version without feedback, the player's dizziness score is reduced by 0.7 points, and the shooting accuracy is increased by 4.5%.

[0134] In the card game static card drawing interface, the player's fingertips only swipe on the card group. It is noted that the amount of field change is stably , the curvature sum , and the energy is continuously lower than the lower limit for 40 frames. After the network outputs the action of "increasing the interest region threshold by 12% and reducing the damping by 8%", the card scaling amplitude increases, the energy returns to the center of the interval, and the player's score for the readability of the card font is increased by 22%.

[0135] Through the present application, the system energy covers both local deformation and global focus jump, and potential visual fatigue can be measured within one frame. Pure pixel specification operation is required, without global sorting or complex optimization algorithm, which is friendly to the bandwidth of the graphics processor. With reinforcement learning, the energy out of bounds is adjusted and converged, which can maintain comfort whether in intense battles or in static scenes. The calculation time is measured to be 0.24 milliseconds under 4K resolution; the time is 0.05 milliseconds under 1280x720 resolution on a mobile terminal, which is easy to deploy in a 60 frame and 144 frame rendering budget.

[0136] Preferably, when adjusting the attention field threshold, the second displacement field parameter and the damping parameter, a time difference reinforcement learning algorithm is used to reduce the difference between the system energies of adjacent two frames as the target.

[0137] In a continuous rendering scene, if the uniform displacement field or attention field parameter changes suddenly, the player is likely to feel dizzy or have the illusion of "frame skipping". The present application abstracts the picture state into "system energy" and uses a time difference reinforcement learning algorithm to adjust the parameters online within milliseconds, so that the difference between the system energies of adjacent two frames converges to zero as much as possible, thereby obtaining a stable, comfortable and highlighted dynamic layout.

[0138] State design and energy difference calculation, system energy has been defined in the foregoing steps as the result of linear weighting of the attention field frame difference term and the uniform displacement field curvature term by a fixed coefficient, and the higher the value, the greater the visual load of the frame. In order to monitor the energy continuity, the present application further defines the energy difference: , which corresponds to the load increase or decrease respectively. The environment state vector of the time difference learning is: , wherein represents the absolute value of the uniform displacement field curvature and is used to reflect the total amount of geometric deformation. The three components are all normalized to on the pixel grid, ensuring that the values are stable under different resolutions.

[0139] Action space and parameter adjustable range, action vector Three adjustable parameters: 1, attention field threshold Determine the size of the interest area; 2, the second displacement field Beltrami coefficient scaling factor Determine the local isometric intensity; 3, damping coefficient Determine the displacement field cubic spline filtering amplitude. Each action can take Three values, Fixed at 5% of the current parameters. So discrete action space scale , both easy to quickly traverse and retain enough adjustment granularity.

[0140] Reward function and target, in order to let the energy difference converge, the invention sets the reward as the negative absolute value of the energy difference:

[0141]

[0142] If the energy difference of adjacent frames approaches zero, the reward is greater; if the energy jumps, the reward is smaller, and reinforcement learning will actively seek strategies to reduce the jump.

[0143] Learning rules and core formula, using time difference SARSA Update the value function. Learning rate Take 0.05, discount factor Take 0.95, trace decay coefficient Take 0.6. The core update formula is:

[0144]

[0145] Where Indicates the expected cumulative reward of executing action In state . The current state is , the current action is , the immediate reward is , and the value function is . The value function is approximated by two layers of perception machines, with a hidden width of 64 and a total of 512 parameters. It is stored in the constant cache of the graphics processor, and the inference delay is less than 0.02 milliseconds.

[0146] Online inference and execution process, after each frame, the process is as follows: 1, calculate the new frame system energy And the energy difference . 2, construct state , call the policy network to output action according to Start 0.2, decay to 0.05 after 500 frames. 3, adjust and immediately applied to the next frame layout pipeline.4, record instant rewards , update the value function weight using the formula.

[0147] Example 13: Desktop first-person shooter, resolution 1920x1080, refresh rate 144fps. Fast gun swinging causes initial energy difference peak up to 1.5, upper limit set to 1. After 300 frames online, energy difference absolute value slides down to around 0.2 and stabilizes, player subjective dizziness score drops from 3.8 to 3.1.

[0148] Example 14: Mobile role-playing game, resolution 1280x720, 60fps. Long static scenes make energy difference long-term below lower limit 0.2. Learning algorithm automatically increases interest region threshold and reduces damping, energy difference rises to 0.3, card text clarity score improves by 25%. There is no significant increase in overall power consumption, and the inference stage power consumption rises by less than 0.5 watts.

[0149] Preferably, when the system energy is below the preset lower limit and the attention field change amount is below the preset threshold, the unified displacement field of the previous frame and the second displacement field of the previous frame are reused, and the steps of determining the interest region based on the attention field and deforming the pixel coordinates using the unified displacement field are skipped.

[0150] The system energy and the attention field change amount are at a low level at the same time, indicating that there is no significant visual deformation between consecutive frames, and there is no new interactive focus to highlight. The present application triggers the "frame skipping reuse" mechanism in this case: directly use the unified displacement field and the second displacement field generated in the previous frame, and omit the computationally intensive steps of interest region extraction, probability mapping and pixel deformation, thereby reducing the load on the graphics processor and stabilizing the output frame rate.

[0151] The triggering conditions include two threshold judgments. The first is the system energy threshold . The system energy has been weighted by the attention field frame difference term and the displacement field curvature term, and when its value is less than for two consecutive frames, it can be inferred that the current picture is basically static. The second is the attention field change threshold , quantified by the absolute difference , if the difference is less than , it means that the focus center of gravity has not moved. When both conditions are met, the frame skipping flag is set to true, and the layout pipeline enters the simplified mode.

[0152] In the simplified mode, the rendering main thread no longer calls the subprogram for determining the interest region based on the attention field, and no longer performs the neural optimal transport mapping, the quasi-conformal mapping and the post-cubic spline filtering; instead, the displacement texture already written into the video memory in the last frame is directly bound to the mesh shader, so that the screen morphing logic remains unchanged. In this way, unnecessary floating-point convolution and iterative operations are avoided, and cache invalidation caused by frequent texture rewriting is also avoided. According to statistics, under the same resolution condition, the omission of the above steps can reduce the per-frame calculation delay by about 1.4 milliseconds.

[0153] In order to prevent the frame skipping multiplexing from causing "aging" errors in a long-time static scene, the application still maintains a soft timeout after entering the simplified mode: if the number of continuous multiplexed frames exceeds 60, a complete layout step is forced to be performed to refresh the displacement field and ensure realignment with possible minimal scene drift. In addition, as long as the system energy or the attention field changes exceed the respective thresholds, the frame skipping flag is immediately cleared, and the next frame resumes full calculation, ensuring that the dynamic scene responds without delay.

[0154] Example 15: In a static room scene of a puzzle game, the player stops to observe the wall prompts. Since the character does not move, the camera does not shake, and the system energy remains at 0.12, which is lower than , the attention field frame difference remains at 0.017, which is lower than . The system enters the frame skipping mode from the 5th frame, and the average frame delay decreases from 6.4 milliseconds to 4.9 milliseconds, with a 9% reduction in energy consumption, a decrease in fan speed, and a reduction in noise.

[0155] Example 16: In a waiting hall of a multiplayer shooting game, the character is in a static standing state. After 48 frames of frame skipping mode, a teammate suddenly jumps into the center of the lens, causing the attention field peak to move and the frame difference to rise to 0.18, which is higher than the threshold. The layout pipeline automatically restores full calculation in the next frame, regenerates a new displacement field, and ensures that the teammate model is enlarged without misalignment. It is observed that the additional delay is only 1 frame, and the player does not perceive any lag.

[0156] As shown in Figure 4 , a game picture dynamic interactive layout system for implementing the game picture dynamic interactive layout method, the system comprises:

[0157] The attention field generation module is configured to obtain a depth map, a normal map, an illumination map, a color map and user input event data for each frame, input the data into a neural network and generate an attention field consistent with a screen resolution in combination with a symplectic integral. The module mainly runs on a graphics processor deep learning core and a parallel computing unit. A rendering pipeline stores a depth, a normal, an illumination and a color buffer in a frame buffer in a pixel shading stage and directly accesses a video memory through a PCIe BAR0. A multi-head self-attention network performs inference on a tensor core and uses a high-bandwidth memory cross strip to ensure aligned reading of an 8-dimensional input. A symplectic integral prediction is completed by a parallel computing shader. A shared memory is used to store gradient and momentum vectors for a 32*32 thread block. A single integral delay is less than 0.3 milliseconds.

[0158] The displacement fusion module is configured to determine an interest region according to the attention field, construct a source probability density and a target probability density, obtain a first displacement field and a second displacement field respectively, and fuse the first displacement field and the second displacement field according to a preset weight to generate a unified displacement field. The module relies on a graphics processor computing shader to expand Sinkhorn and Koebe iteration. Source / target probability density is stored in a half-precision texture. Row and column normalization is completed by a warp-matrix-multiply instruction of a TensorCore. Beltrami coefficients and Laplace curvature are quickly derived by a texture differential instruction of a graphics processor. Curvature weight calculation is performed in parallel reduction of a shared memory. The fusion result is written back as a two-channel FP16 displacement texture, which can be directly sampled by a subsequent rendering stage.

[0159] The compensation rendering module is configured to deform pixel coordinates using the unified displacement field, input a voxel set corresponding to the interest region into a neural radiance field network to obtain a compensation color map, and synthesize and fuse the compensation color map and the deformed color map into a final picture. Interest voxels are directly written into a hash grid buffer by a raster unit through depth back projection. A hash table is resident in a video memory and uses atomic locking to index a bucket. The neural radiance field network performs batch inference on a tensor core. Three-dimensional coordinates and a line-of-sight vector are first encoded in a Fourier code in a constant buffer and then sent to a multiply-accumulate pipeline. Ray integration uses a high-concurrency sampling of a ray tracing hardware unit. Compensation color and deformed color are quickly interpolated by a blending unit in a fragment stage according to a mask. The integrated bandwidth consumption is lower than that of a traditional off-screen synthesis.

[0160] The interactive feedback module is configured to determine original screen coordinates by sequentially performing inverse operations on the user input event on the fusion picture through the uniform displacement field and the second displacement field, query an entity identifier from the deformed entity identifier map, and adjust the attention field threshold, the second displacement field parameter and the damping parameter when the system energy exceeds a preset interval, so as to realize cross-frame adaptive feedback. The module is deployed in a central processing unit and a graphics processing unit in cooperation: the input event is written into a ring buffer by an interrupt controller, and a uniform displacement and a conformal lookup table are read by a graphics processing unit texture unit to realize two-stage inverse mapping; the entity identifier is immediately sent back to a logical core through NVLink zero-copy after sampling. The system energy and the curvature are obtained once in a calculation shader, a reinforcement learning strategy network is resident in a display memory in a CUDA graph mode, only 512 weights are completed once in a tensor core, and if frame skipping reuse is triggered, the last frame texture is directly reused through a graphics processing unit event fence, data moving and iterative calculation are saved, and it is ensured that the hardware pipeline is continuous and the delay is controllable.

[0161] The above merely provides an embodiment of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A game screen dynamic interactive layout method, characterized in that, The method comprises the following steps: Each frame obtains a depth map, a normal map, an illumination map, a color map, and user input event data, inputs the obtained data into a neural network, and generates an attention field in combination with symplectic integration; Based on the attention field, a region of interest is determined, a source probability density and a target probability density are constructed, a first displacement field and a second displacement field are obtained respectively, and a unified displacement field is generated by fusing the first displacement field and the second displacement field according to a preset weight; The pixel coordinates are deformed using the unified displacement field, a voxel set corresponding to the region of interest is input into a neural radiance field network to obtain a compensated color map, and the compensated color map is combined with the deformed color map to form a fusion picture; For the user input event on the fusion picture, the original screen coordinates are determined through inverse operation of the unified displacement field and the second displacement field, the entity identifier is obtained by querying the deformed entity identifier map, and when the system energy exceeds a preset interval, the attention field threshold, the second displacement field parameter and the damping parameter are adjusted to realize cross-frame adaptive feedback.

2. The method of claim 1, wherein, When generating the attention field, the neural network is sequentially composed of a multi-head self-attention layer and a feedforward layer, and the feature vectors of the depth map, the normal map, the illumination map and the color map are included in the same attention window in the multi-head self-attention layer.

3. The method of claim 1, wherein, The first displacement field is obtained by performing Sinkhorn iteration on a discrete grid to solve the optimal transport mapping between the source probability density and the target probability density, and the mapping matrix is normalized by rows and columns after each iteration.

4. The method of claim 1, wherein, The second displacement field is obtained by calculating the Beltrami coefficient using the grayscale value of the attention field and maintaining the screen boundary identity mapping in the Koebe iteration process.

5. The method of claim 1, wherein, The preset weight is distributed according to the Laplace curvature of the first displacement field and the second displacement field on the pixel grid, and the first displacement field and the second displacement field are linearly superimposed to generate the unified displacement field during fusion.

6. The method of claim 1, wherein, The voxel set corresponding to the region of interest is obtained by back-projecting the pixel coordinates into three-dimensional coordinates and writing them into a hash grid, and the three-dimensional coordinates and the ray direction are respectively Fourier-encoded as inputs of the neural radiance field network.

7. The method of claim 1, wherein, When determining the original screen coordinates, first, screen space inverse displacement is performed using the unified displacement field, and then quasi-conformal inverse mapping is performed using the second displacement field, wherein the inverse displacement is calculated by bilinear interpolation.

8. The method of claim 1, wherein, The system energy is obtained by summing the attention field variation and the unified displacement field curvature on the same pixel grid and weighting them by a fixed coefficient; when adjusting the attention field threshold, the second displacement field parameter and the damping parameter, a time difference reinforcement learning algorithm is used to reduce the difference between the system energies of adjacent two frames as the target.

9. The method of claim 1, wherein, When the system energy is lower than the preset lower limit and the attention field variation is lower than the preset threshold, the unified displacement field of the previous frame and the second displacement field of the previous frame are reused, and the steps of determining the region of interest based on the attention field and deforming the pixel coordinates using the unified displacement field are skipped.

10. A game screen dynamic interactive layout system for implementing the game screen dynamic interactive layout method of any one of claims 1-9, characterized in that, The system comprises: An attention field generation module is configured to obtain a depth map, a normal map, an illumination map, a color map, and user input event data at each frame, and input the data into a neural network and generate an attention field consistent with the screen resolution in combination with symplectic integration; The displacement fusion module is configured to determine an interest region according to the attention field, construct a source probability density and a target probability density, obtain a first displacement field and a second displacement field respectively, and fuse the first displacement field and the second displacement field according to a preset weight to generate a unified displacement field; The compensation rendering module is configured to use the unified displacement field to deform pixel coordinates, input a voxel set corresponding to the interest region into a neural radiance field network to obtain a compensation color map, and synthesize the compensation color map and a deformed color map to obtain a fusion picture; The interactive feedback module is configured to, for a user input event on the fusion picture, sequentially perform inverse operations on the unified displacement field and the second displacement field to determine original screen coordinates, query a deformed entity identification map to obtain an entity identification, and adjust the attention field threshold, the second displacement field parameter and the damping parameter when system energy exceeds a preset interval, so as to realize cross-frame adaptive feedback.

Images