A game interaction scene material image enhancement processing method based on virtual reality

Abstract

The application discloses a virtual reality-based game interaction scene material image enhancement processing method, mainly relates to the technical field of image processing, and aims to solve the problems that the existing scheme cannot distinguish the user interface from the three-dimensional scene and cannot perceive the structural differences of different semantic areas. The method comprises the following steps: obtaining modulation feature maps of various scales by using an encoding feature map and a user interface boundary distance field; obtaining a full-resolution decoding feature map by using the modulation feature maps; predicting a dynamic filtering weight and a dynamic bias term of each pixel position by using the full-resolution decoding feature map; obtaining an enhanced image based on a basic enhancement image, the dynamic filtering weight and the dynamic bias term; calculating a composite loss function according to the enhanced image and a corresponding processed high-quality image; and optimizing trainable parameters in a dynamic pixel-by-pixel filtering enhancement network by using the composite loss function to obtain a trained dynamic pixel-by-pixel filtering enhancement network.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for enhancing the image processing of game interactive scene materials based on virtual reality. Background Technology

[0002] Current technologies for processing interactive scene images in virtual reality and cloud gaming generally employ a globally uniform sharpening and contrast enhancement strategy, treating the entire frame as a homogeneous region for uniform processing. This method ignores the fundamental differences in visual characteristics between user interface (UI) elements and 3D rendered scenes: UI areas are characterized by high contrast, low texture, and sharp boundaries, while 3D scenes contain complex multi-directional textures, continuous lighting, and non-uniform materials. This "one-size-fits-all" enhancement approach results in over-sharpened jagged edges in the UI, while the fine textures in the 3D scene become blurred and distorted due to insufficient enhancement, failing to meet the demands of a high-quality immersive experience.

[0003] Existing methods rely on static convolutional kernels to perform fixed filtering operations on the entire image, failing to dynamically adjust the receptive field and weight distribution based on local structure. In directional material regions (such as leaf veins in vegetation and brick seams in buildings), static kernels easily distort texture continuity; in flat regions (such as UI buttons and sky curtains), excessive smoothing introduces pseudo-textures and noise. Due to the lack of explicit modeling of structural priors such as the distance field of user interface boundaries and texture orientation, the network cannot distinguish semantic regions in the feature space, resulting in a lack of spatial awareness in the enhancement process and difficulty in achieving pixel-level adaptive optimization. Summary of the Invention

[0004] This application provides a method for enhancing the image of game interactive scene materials based on virtual reality, in order to solve the problems of existing solutions.

[0005] This application provides a method for enhancing and processing images of game interactive scene materials based on virtual reality, the method including:

[0006] Collect original game interaction scene material images and construct corresponding preset high-definition reference images; perform preprocessing to obtain the processed original image and the processed high-definition image;

[0007] The processed original image is converted to a preset color space to obtain the main color tone matrix of the user interface; based on the processed original image and the main color tone matrix of the user interface, the color components of the user interface, scene texture variables and residual lighting variables are decomposed to obtain the color components of the user interface, scene texture variables and residual lighting variables.

[0008] Using original game interaction scene material images, the user interface soft mask variables, scene texture variables, and residual lighting variables are initialized; the user interface soft mask variables, scene texture variables, and residual lighting variables are updated using an alternating optimization method until the preset stop update condition is met, thus obtaining the user interface mask, scene texture layer, and residual lighting layer.

[0009] The multimodal input tensor is obtained based on the user interface mask, scene texture layer, and residual lighting layer;

[0010] A multimodal input tensor is input into a dynamic pixel-wise filtering enhancement network. Using the multimodal input tensor, encoded feature maps at several scales are obtained. Using the user interface boundary distance field, user interface boundary distance fields at several scales are obtained. Using the encoded feature maps and user interface boundary distance fields, modulation feature maps at each scale are obtained.

[0011] Using the modulation feature map, a full-resolution decoded feature map is obtained; using the full-resolution decoded feature map, the dynamic filtering weights and dynamic bias terms at each pixel location are predicted; based on the base enhanced image, dynamic filtering weights, and dynamic bias terms, the enhanced image is obtained.

[0012] Based on the enhanced image and the corresponding processed high-quality image, a composite loss function is calculated; the trainable parameters in the dynamic pixel-by-pixel filtering enhancement network are optimized using the composite loss function to obtain the trained dynamic pixel-by-pixel filtering enhancement network.

[0013] In one implementation of this application, original game interaction scene material images are acquired, and corresponding preset high-quality reference images are constructed; preprocessing is performed to obtain the processed original image and the processed high-quality image, specifically including:

[0014] Collect original game interaction scene material images, and then... Each original game interaction scene material image is defined as , The size is The red, green, and blue three-channel image, in which, Indicates the sample index; Indicates the image height; Indicates the image width;

[0015] For each original game interaction scene material image, a corresponding preset high-quality reference image is constructed;

[0016] The original game interactive scene material images and the preset high-quality reference images are used to form training sample pairs;

[0017] Perform spatial alignment between the original game interactive scene material images and the preset high-definition reference images;

[0018] The pixel values ​​of the original game interactive scene material images and the preset high-quality reference images, after spatial dimension alignment, are normalized to... The range is used to obtain the original image after processing and the high-quality image after processing.

[0019] In one implementation of this application, the processed original image is converted to a preset color space to obtain the main color tone matrix of the user interface, specifically including:

[0020] The processed original image is converted to a preset color space to obtain a perceptually uniform color map.

[0021] For each pixel position of the perceived uniform color map Calculate the local color variance and local gradient magnitude;

[0022] Pixels with local color variance less than a preset color variance threshold and local gradient magnitude less than a preset magnitude threshold are selected as flat candidate pixels.

[0023] Perform color clustering on the perceptually uniform color vectors corresponding to flat candidate pixels to obtain K color clusters;

[0024] According to preset filtering rules, candidate primary color clusters for the user interface are determined from K color clusters;

[0025] Calculate the user interface primary color vector based on the candidate primary color clusters of the user interface, and copy the user interface primary color vector to the entire image space to obtain the user interface primary color matrix.

[0026] In one implementation of this application, based on the processed original image and the user interface main color matrix, the user interface color components, scene texture variables, and residual lighting variables are decomposed to obtain the following:

[0027] Based on the processed original image, define user interface soft mask variables, scene texture variables, and residual lighting variables;

[0028] Copy the user interface soft mask variable along the red, green and blue channel directions to obtain a three-channel user interface soft mask variable;

[0029] Perform pixel-by-pixel, channel-by-channel multiplication between the three-channel user interface soft mask variable and the user interface primary color matrix to obtain the user interface color components.

[0030] Construct a joint separation target that combines the user interface color components, scene texture variables, and residual lighting variables with the original game interaction scene material image.

[0031] In one implementation of this application, the user interface soft mask variables, scene texture variables, and residual lighting variables are initialized using original game interaction scene material images; the user interface soft mask variables, scene texture variables, and residual lighting variables are updated using an alternating optimization method until a preset stop update condition is met, thereby obtaining the user interface mask, scene texture layer, and residual lighting layer, specifically including:

[0032] Through the formula:

[0033] Calculate the initial user interface soft mask value;

[0034] in, Indicates position The initial soft mask value for the user interface at that location; Indicates position The color distance between the original game interactive scene material image and the main color matrix of the user interface in a perceived uniform color space; This represents the color distance attenuation parameter; Indicates position The local color variance after normalization; Represents a numerical clipping function;

[0035] Initialize the scene texture variable to the high-frequency residual of the original game interactive scene material image minus the initial user interface color components;

[0036] The residual lighting variables are initialized to the low-frequency components of the original game interactive scene material images obtained by large-scale guided filtering or Gaussian filtering.

[0037] An alternating optimization approach is adopted, in which two variables are fixed and one variable is optimized each time among the three variables: user interface soft mask variable, scene texture variable, and residual lighting variable. This process is repeated to update the user interface soft mask variable, scene texture variable, and residual lighting variable. The update of the user interface soft mask variable, scene texture variable, and residual lighting variable is repeated until a preset number of iterations is reached or the change in reconstruction error is less than a preset threshold, thus obtaining the user interface mask, scene texture layer, and residual lighting layer.

[0038] In one implementation of this application, a multimodal input tensor is obtained based on a user interface mask, a scene texture layer, and a residual lighting layer, specifically including:

[0039] The user interface mask is binarized to obtain a binary user interface mask.

[0040] Extract the user interface boundary map from the user interface binary mask;

[0041] Perform a distance transformation on the user interface boundary map, calculate the pixel distance from each pixel to the nearest user interface boundary, and truncate and normalize the distances to obtain the user interface boundary distance field;

[0042] Extract the scene texture layer luminance channel from the scene texture layer;

[0043] Construct a multi-directional Gabor filter bank and calculate the local texture response of the scene texture layer's brightness channel in different directions;

[0044] Compare the response intensity of the texture response map in each direction at each pixel location, and select the direction with the largest response intensity as the main direction of the local texture;

[0045] Calculate the texture confidence map based on the proportion of the main directional response in all directional responses;

[0046] Generate a base enhanced image based on the scene texture layer and residual lighting layer;

[0047] Continuous directional encoding is performed on the main direction field of scene texture, and confidence-weighted texture direction cosine diagram and confidence-weighted texture direction sine diagram are calculated;

[0048] The base augmented image, user interface boundary distance field, confidence-weighted texture direction cosine map, confidence-weighted texture direction sine map, and texture confidence map are concatenated along the channel dimension to obtain the multimodal input tensor.

[0049] In one implementation of this application, the user interface boundary distance field is downsampled to several scales to obtain user interface boundary distance fields at several scales; modulation feature maps at each scale are obtained using the encoded feature map and the user interface boundary distance field, specifically including:

[0050] The multimodal input tensor is input into the lightweight encoder-decoder network to obtain encoded feature maps at several scales; the user interface boundary distance field is downsampled to several scales to obtain user interface boundary distance fields at several scales.

[0051] Downsample the confidence-weighted texture orientation cosine map, confidence-weighted texture orientation sine map, and texture confidence map to the [number]th [level]. The scale is then spliced ​​along the channel dimension to obtain the first scale. Texture structure priors at various scales;

[0052] The first The user interface boundary distance field at each scale and the first The texture structure prior map at the nth scale is input to the nth scale. The prior modulator subnetwork of the i-th scale generates the i-th scale. Spatial variable gain diagram at each scale and the Spatial variable offset map at various scales;

[0053] According to the Spatial variable gain plot at each scale and the first The spatially variable bias map at each scale is for the first... Modulate the encoded feature map at the i-th scale to obtain the i-th scale. Modulation feature map at each scale.

[0054] In one implementation of this application, a full-resolution decoded feature map is obtained using the modulation feature map; the dynamic filtering weights and dynamic bias terms at each pixel location are predicted using the full-resolution decoded feature map; and an enhanced image is obtained based on the base enhanced image, the dynamic filtering weights, and the dynamic bias terms, specifically including:

[0055] The modulation feature maps at each scale are input into the decoder for step-by-step upsampling and feature fusion to obtain the full-resolution decoded feature map.

[0056] The full-resolution decoded feature map is input into the dynamic filtering prediction head to predict the dynamic filtering weights and dynamic bias terms at each pixel location.

[0057] Perform boundary padding on the base augmented image to obtain the padded base augmented image;

[0058] The dynamic filtering weights corresponding to each pixel position are applied to the local neighborhood of the base enhanced image, and a dynamic bias term is added to obtain the enhanced image.

[0059] In one implementation of this application, a composite loss function is calculated based on the enhanced image and the corresponding processed high-quality image; the trainable parameters in the dynamic pixel-by-pixel filtering enhancement network are optimized using the composite loss function to obtain a trained dynamic pixel-by-pixel filtering enhancement network, specifically including:

[0060] Based on the enhanced image and the corresponding processed high-quality image, calculate the pixel fidelity loss, user interface edge consistency loss, scene texture high-frequency consistency loss, and global color stability loss.

[0061] The composite loss function is obtained by weighted summing of pixel fidelity loss, user interface edge consistency loss, scene texture high frequency consistency loss and global color stability loss.

[0062] The goal is to optimize the set of trainable parameters of the network by minimizing the average composite loss function over the training sample set. The parameter optimization objective is expressed as ;

[0063] in, Represents the optimal set of network parameters. This represents the set of trainable parameters of the network. This represents the total number of training samples. Indicates the first The composite loss function value corresponding to each training sample;

[0064] When the preset stopping conditions are met, a well-trained dynamic pixel-by-pixel filtering enhancement network is obtained.

[0065] As can be seen from the above technical solutions, this application has the following advantages:

[0066] By decomposing the original image into user interface color components, scene texture variables, and residual lighting variables, and combining this with alternating optimization of user interface soft mask variables, the system explicitly separates the UI region from the 3D scene region in the feature space, thus providing a semantically aware structural prior for the subsequent enhancement process. Based on this, a dynamic pixel-by-pixel filtering enhancement network jointly generates a modulation feature map based on the user interface boundary distance field and multimodal encoded feature map, enabling the filtering weights and biases of each pixel to be dynamically adjusted according to the semantic attributes of its region (such as UI edges, texture direction, and lighting gradient). This mechanism completely abandons the traditional globally unified filtering strategy, achieving precise sharpening of the UI region and faithful enhancement of the 3D scene texture, fundamentally avoiding the jagged and blurry distortion problems caused by region confusion.

[0067] This scheme introduces a distance field at the user interface boundary and fuses it with a multi-scale encoded feature map to generate a modulation feature map sensitive to spatial structure. This allows the shape and weight distribution of the filter kernel to be determined in real-time by the local geometry and semantic structure of the image, rather than relying on a pre-defined static convolution kernel. In UI boundary regions, the distance field guides the filter to focus on edge preservation; in directional material regions (such as leaf veins or brick seams), texture variables and the modulation feature map work together to suppress pseudo-textures and orientation distortions; in flat regions (such as a sky curtain), residual illumination variables suppress noise introduced by over-smoothing. This mechanism achieves pixel-level adaptation of the receptive field and filter intensity, significantly improving the accuracy and consistency of the enhancement process in response to complex visual structures.

[0068] This method constructs a composite loss function that includes both the enhanced image and the high-quality reference image. This directly drives the joint optimization of trainable parameters in the multimodal input tensor, creating a closed loop between the decomposition process of the user interface mask, texture layer, and lighting layer, and the parameter learning of the dynamic filtering network. This framework eliminates the need for manual design of enhancement rules or staged parameter tuning, ensuring that the enhancement strategy and the target image quality standard automatically align during training. The system can automatically adapt to the optimal enhancement parameters when facing different game scenarios, UI layouts, and lighting conditions, significantly improving the robustness and engineering practicality of the algorithm in diverse virtual reality interactive scenarios. Attached Figure Description

[0069] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a flowchart of a method for enhancing images of game interactive scene materials based on virtual reality, provided in an embodiment of this application.

[0071] Figure 2 This is a user interface mask heatmap provided in an embodiment of this application.

[0072] Figure 3 This is a scene texture layer map provided in the embodiments of this application.

[0073] Figure 4 This is a residual illumination layer map provided in an embodiment of this application.

[0074] Figure 5 This is a modulation feature map provided in an embodiment of this application.

[0075] Figure 6 This is an enhanced image provided in an embodiment of this application. Detailed Implementation

[0076] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.

[0078] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0079] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0080] The embodiment provides a method for enhancing the image processing of game interactive scene materials based on virtual reality, such as... Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps:

[0081] Step 110: Collect original game interaction scene material images and construct corresponding preset high-quality reference images; perform preprocessing to obtain the processed original image and the processed high-quality image.

[0082] Collect original game interaction scene material images, and then... Each original game interactive scene material image is defined as , The size is The red, green, and blue three-channel image, in which, This represents the sample index, with a value ranging from 1 to... ; This represents the total number of training samples, with an example value of 50000. This represents the image height and is used to limit the number of pixels vertically; an example value is 1080. This represents the image width and is used to limit the number of pixels horizontally. An example value is 1920.

[0083] In practical implementation, original game interaction scene images can be collected from real-time rendered screens of the game engine, game screenshots, images exported from the asset editor, or historical project asset libraries. To improve the generalization ability of the augmented model, the training samples should cover different game types, different user interface layouts, different lighting conditions, different material styles, and different compression qualities. For example, scenes from role-playing games, shooting games, strategy games, racing games, and open-world games can all be used as sample sources.

[0084] For each original game interaction scene material image, a corresponding high-quality reference image is constructed, and the first... A high-quality reference image is defined as , The size is The red, green, and blue three-channel images are used to provide supervised targets for enhancing network training.

[0085] In one implementation, a high-definition reference image This can be achieved by re-exporting the same game scene at high rendering quality settings, such as enabling higher texture quality, higher anti-aliasing levels, higher shadow quality, and higher post-processing quality. If re-rendering is not possible, it can also be obtained from manually refined images, downsampled images of high-resolution footage, multi-frame high-quality fused images, or images processed by professional image enhancement tools. High-quality reference images. Images should be consistent with the original game's interactive scene assets. Maintain consistency in user interface layout, scene content, and viewpoint to prevent the network from mislearning content differences as differences in image quality enhancement.

[0086] Original game interactive scene material images and high-definition reference images Form training sample pairs and create a training sample set. Training sample set It refers to a data set consisting of multiple original game interactive scene material images and their corresponding high-definition reference images, i.e. .in, Represents the training sample set, Indicates the first Original game interactive scene material images, Indicates the first A high-resolution reference image.

[0087] It should be noted that the training sample set The original game interaction scene images should ideally include semi-transparent user interfaces, small text, complex icons, brightly lit scenes, dark scenes, directional material textures, and low-texture smooth areas. By covering these scenes, the subsequent reinforcement network can learn the differentiated processing rules between user interface protection, material texture enhancement, and low-texture area artifact suppression.

[0088] Original game interactive scene material images and high-definition reference images Perform spatial alignment. Specifically, if the two images have different spatial dimensions, a high-quality reference image will be used. Resampled to original game interaction scene material images The spatial dimensions can be adjusted, or both can be cropped to the same training size. During the training phase, the size can be randomly cropped to [specific dimensions]. or The image blocks are retained at their original resolution during the deployment phase.

[0089] In practical implementation, resampling can employ bicubic interpolation or Lanczos interpolation. For user interface areas containing small text, Lanczos interpolation is preferred to reduce text edge blurring; for training cropping, the original game interaction scene images... and high-definition reference images The same clipping coordinates must be used to ensure that the supervised target corresponds one-to-one with the input content.

[0090] Original game interactive scene material images and high-definition reference images Pixel values ​​normalized to Range. Specifically, if the input pixel value is an 8-bit integer, the pixel value of each channel is divided by 255; if the input pixel value is a 10-bit integer, the pixel value of each channel is divided by 1023. The normalized pixel values ​​are used for subsequent color distance calculations, layer separation optimization, and enhanced network training.

[0091] During the training phase, images of the original game interaction scenes were used. Perform data perturbations to improve the network's adaptability to different source materials. Specifically, data perturbations can include random horizontal flipping, random cropping, random brightness perturbation, random contrast perturbation, random saturation perturbation, and random compression degradation. Random compression degradation can use a JPEG compression quality factor of 40 to 95 to simulate compression impairments during game asset export, platform upload, and network transmission.

[0092] It should be noted that text and icons in user interface elements are typically sensitive to geometric deformations; therefore, large-angle rotations, perspective distortions, or non-uniform scaling should not be used as regular data perturbations. By limiting the types of data perturbations, we can avoid artificially disrupting the user interface structure during training, thereby improving the stability of the user interface enhancement results.

[0093] Step 120: Convert the processed original image to a preset color space to obtain the main color tone matrix of the user interface; based on the processed original image and the main color tone matrix of the user interface, decompose to obtain the color components of the user interface, scene texture variables and residual lighting variables.

[0094] It should be noted that the original game interactive scene assets include both user interface elements and 3D rendered scene content, and the two differ significantly in color distribution, boundary morphology, texture direction, and lighting variations. Conventional image enhancement methods apply uniform sharpening, uniform contrast enhancement, or uniform noise reduction strategies to the entire image, making it difficult to simultaneously ensure the clarity of text icons, the stability of user interface colors, the continuity of scene material textures, and the naturalness of low-texture areas.

[0095] This application first estimates the dominant color matrix of the user interface from the original game interaction scene material images, and then jointly separates the user interface mask, scene texture layer, and residual lighting layer; then it generates the user interface boundary distance field, scene texture principal direction field, and texture confidence map; finally, it constructs a multimodal input tensor to provide color, boundary, direction, and texture reliability priors for the structure prior-guided dynamic pixel-by-pixel filtering enhancement network. The specific steps are as follows:

[0096] User interface elements typically consist of button background colors, status bar colors, text fill colors, icon background colors, and menu background colors. The color range is relatively concentrated, and color variations within areas are minimal. Directly calculating the dominant hue in the red-green-blue color space is easily affected by variations in brightness, channel non-uniformity, and large areas of background color in 3D scenes.

[0097] This application converts the original game interaction scene material images to a perceptually uniform color space and clusters the flat candidate pixels to obtain the main color tone matrix of the user interface. The specific steps are as follows:

[0098] 1> Define the original game interactive scene material images as , The size is The red, green, and blue three-channel image represents the game interaction scene material image to be processed. First, Pixel values ​​normalized to The range, and then the normalized range. Convert to CIELAB or OKLab color space to obtain a perceived uniform color map. Perceive uniform color map The size is The three-channel color map is used to make colors that are perceived as close by the human eye more closely in numerical distance, thereby improving the stability of the main color clustering of the user interface.

[0099] Furthermore, for each pixel position Calculate the local color variance and local gradient magnitude, where, Indicates spatial location in the image. Indicates the vertical pixel index. This represents the horizontal pixel index; the local neighborhood size can be taken as... or The local color variance is used to determine whether the current region is close to a solid color fill, and the local gradient magnitude is used to determine whether the current region is near a regular boundary. Pixels with a local color variance less than a preset color variance threshold and a local gradient magnitude that is not too high are selected as flat candidate pixels. The preset color variance threshold is denoted as […]. , Used to control the stringency of screening flat candidate pixels, with values ​​ranging from 0.005 to 0.02.

[0100] 2> Perform color clustering on the perceptually uniform color vectors corresponding to the flat candidate pixels to obtain multiple color clusters, where the total number of color clusters is defined as... , This indicates the number of candidate categories for the primary color tone; an example value is 12. The set of pixels in a color cluster is defined as ,in, This represents the color cluster index, with values ​​ranging from 1 to... ; Used to store the first A set of pixel positions for each color cluster.

[0101] In one implementation, mean-shift clustering can be used to cluster flat candidate pixels. The bandwidth of mean-shift clustering is defined as... , This controls the intensity of color cluster merging, with values ​​ranging from 6 to 12 perceptual color distance units. Alternatively, K-means clustering can be used, with the number of clusters directly set to... After clustering, the number of pixels, average color, intra-cluster color variance, number of connected regions, and median area of ​​connected regions are calculated for each color cluster. The number of pixels is used to measure the frequency of color occurrence, the intra-cluster color variance is used to determine whether the color is close to a solid color fill, and the number of connected regions and the median area of ​​connected regions are used to distinguish repetitive elements in the user interface from large areas of background in the 3D scene.

[0102] 3> Select candidate primary color clusters for the user interface based on the color cluster statistics. The index of the candidate primary color clusters for the user interface is defined as follows: That is, the color cluster index that is ultimately used to calculate the primary color vector of the user interface.

[0103] In practical implementation, color clusters with a pixel proportion greater than the background proportion threshold and a median connected region area greater than 5% of the total image area are first eliminated. The background proportion threshold can be set between 0.25 and 0.40. This process reduces the risk of large-area 3D scene colors such as sky, fog, ground, or walls being mistakenly selected as the main color of the user interface. Then, from the remaining color clusters, color clusters that meet the criteria of small intra-cluster color variance, small median connected region area, and a high number of pixels are selected as candidate color clusters for the user interface. From these candidate color clusters, the color cluster with the largest number of pixels is selected as the candidate main color cluster for the user interface. If no candidate color cluster meets the criteria, the color cluster with the largest number of pixels among the flat candidate pixels is directly selected as the candidate main color cluster for the user interface to ensure that the process can be implemented.

[0104] 4> Calculate the dominant color vector of the user interface based on the candidate dominant color clusters, and copy the dominant color vector of the user interface to the entire image space to obtain the dominant color matrix of the user interface. User interface main color matrix The size is A three-channel matrix is ​​used to provide color priors for the user interface. Each spatial location has the same primary hue color value, which can characterize the color features of solid or near-solid color areas of the user interface, such as buttons, status bars, icon backgrounds, text fills, and menu backgrounds.

[0105] In one implementation, the user interface primary color vector and the user interface primary color matrix are represented as follows:

[0106] ;

[0107] ;

[0108] in, This represents the primary color vector of the user interface, which is the average value of the red, green, and blue color channels. This indicates the number of pixels in the candidate primary color cluster for the user interface; This represents the pixel position index, used to traverse pixels in the candidate primary color cluster of the user interface; Represents original game interactive scene material images At pixel position The red, green, and blue color vectors at that location; This represents a spatial broadcast operator used to copy the user interface primary color vector to a height of [missing information]. And the width is The entire image space.

[0109] It should be noted that the spatial broadcast operator creates a space with a size of The matrix, and the input three-channel color vector Copy to every spatial location of the matrix. Above, in other words, for all and All have .

[0110] In one embodiment, as an example, the original game interactive scene material images It includes a red health bar, blue skill buttons, white text, and a gray menu background. After conversion to the CIELAB color space, mean-shift clustering is performed on the flat candidate pixels to obtain 12 color clusters. If the red blood bar color cluster appears repeatedly in multiple user interface areas, and the intra-cluster color variance of this color cluster is small and the median area of ​​the connected region is less than 5% of the total image area, then this red blood bar color cluster is determined as a candidate dominant color cluster for the user interface. The red, green, and blue means are calculated and broadcast to the entire image space to obtain the dominant color matrix of the user interface. User interface main color matrix It can provide stable color priors for the joint separation of semantic layers in subsequent user interface scenarios.

[0111] User interface elements in game interactive scene material images may be overlaid on the 3D rendered scene in the form of solid color, semi-transparent, anti-aliased edges or shadow outlines. If only color threshold is relied upon, the highlighted scene area will be mistakenly identified as the user interface area; if only edge detection is relied upon, the text edge and the scene material edge will be confused.

[0112] This application constructs a joint separation objective function that includes a color reconstruction term, a mask sparse edge regularization term, a texture total variation regularization term, and a lighting smoothing regularization term. It then uses an alternating optimization approach to decompose the original game interactive scene material image into user interface color components, scene texture variables, and residual lighting variables, thereby achieving semantic-level separation of heterogeneous layers. The specific steps are as follows:

[0113] 1> Define user interface soft mask variables Scene texture variables and residual illumination variables Among them, the user interface soft mask variable The size is Single-channel variables are used to estimate the confidence level that each pixel belongs to the user interface layer; scene texture variables. The size is The three-channel variables are used to estimate material textures, geometric edges, and local structure in a 3D rendered scene; residual lighting variables... The size is The three-channel variables are used to estimate the smooth changes caused by low-frequency lighting, color gradation, shadow transitions, and translucent overlays.

[0114] Furthermore, the user interface soft mask variable Copy along the red, green, and blue channels to obtain a three-channel user interface soft mask variable. Three-channel user interface soft mask variable The size is The three-channel variables are used in conjunction with the user interface main color matrix. Perform pixel-by-pixel, channel-by-channel multiplication. Then, through... The calculation operation yields the color components of the user interface, where, This represents the pixel-by-pixel, channel-by-channel multiplication operator.

[0115] 2> Construct a joint separation target that allows the user interface color components, scene texture variables, and residual lighting variables to jointly reconstruct the original game interactive scene material image. At the same time, constrain the user interface soft mask variables. It features smooth interior regions, sparse boundaries, and regular contours, constraining scene texture variables. Preserve edges and suppress random noise, constrain residual illumination variables The main focus is on low-frequency continuous changes. The joint separation objective is represented as:

[0116] ;

[0117] in, This represents the Frobenius norm, used to measure the overall energy of image reconstruction errors; This represents the first-order spatial gradient operator, used to calculate the transverse and longitudinal gradients. Represents the soft mask variable of the user interface. The spatial gradient mixing norm is used to encourage user interface masks to present structures with fewer boundaries and continuous contours. Represents scene texture variables The first-order gradient norm is used to suppress random noise and preserve scene texture boundaries; This represents the second-order spatial gradient operator, used to calculate residual illumination variables. The curvature change; This represents a residual illumination smoothing constraint, used to limit residual illumination variables. Absorbs high-frequency texture details; This represents the sparse edge regularization coefficient of the user interface mask, used to control the user interface soft mask variable. The degree of boundary smoothness, with values ​​ranging from 0.08 to 0.15; These represent the total variational regularization coefficients of the scene texture, used to control scene texture variables. The noise suppression strength, with values ​​ranging from 0.02 to 0.06; This represents the residual illumination smoothing regularization coefficient, used to control residual illumination variables. The low-frequency smoothness is represented by values ​​ranging from 0.01 to 0.04.

[0118] Step 130: Initialize the user interface soft mask variable, scene texture variable, and residual lighting variable using the original game interaction scene material image; update the user interface soft mask variable, scene texture variable, and residual lighting variable using an alternating optimization method until the preset stop update condition is met, and obtain the user interface mask, scene texture layer, and residual lighting layer; obtain the multimodal input tensor based on the user interface mask, scene texture layer, and residual lighting layer.

[0119] It should be noted that the soft mask variable for the user interface Scene texture variables and residual illumination variables Initialization is performed. Specifically, the original game interaction scene asset images are first calculated. With the main color matrix of the user interface The smaller the color distance and the flatter the local area, the higher the confidence level of the initial user interface; the larger the color distance or the stronger the local texture change, the lower the confidence level of the initial user interface.

[0120] In practical implementation, location The initial user interface soft mask value at the location can be calculated using the following formula:

[0121] ;

[0122] in, Indicates position The initial soft mask value for the user interface at that location; Indicates position Original game interactive scene material images With the main color matrix of the user interface Color distance in a perceived uniform color space; It is the Euclidean distance calculated in a perceptually uniform color space (such as CIELAB or OKLab), used to measure the difference between the pixel colors of the original image and the estimated dominant color of the user interface. In specific implementation, firstly, the dominant color vector of the user interface in the red-green-blue color space is... Convert to The same perceptually uniform color space yields a perceptually uniform dominant hue vector. Then, regarding the position Its color distance The calculation method is as follows ;in, Represents a perceived uniform color map In position The three-dimensional color vector at that location, This represents the L2 norm (i.e., Euclidean distance) of a vector. This represents the color distance attenuation parameter, which controls the degree of influence of color similarity on the initial user interface soft mask value. The example value is 8 to 15 perceptual color distance units. Indicates position The normalized local color variance, with a value ranging from 0 to 1, is used to reduce the initial user interface confidence level in complex scene texture areas; This represents a numerical clipping function used to restrict the result to a specific range. Within the range.

[0123] Furthermore, the scene texture variable Initialize to original game interactive scene material images The high-frequency residuals after subtracting the initial user interface color components can be obtained through guided filtering residuals, local mean residuals, or high-pass filtering. Furthermore, the residual illumination variables... Initialize to original game interactive scene material images The low-frequency components obtained through large-scale guided filtering or Gaussian filtering can have a filtering radius of 15 to 31 pixels. This initialization method allows for a reasonable allocation of the user interface, textures, and lighting at the beginning of the iteration, reducing the risk of the joint optimization getting stuck in erroneous separation results.

[0124] In one implementation, the scene texture variable Initialize to original game interactive scene material images The guided filter residual after subtracting the initial user interface color components; specifically, first calculate... Then, guided filtering is performed on the residual image, and finally, the high-frequency residual is obtained by subtracting the guided filtering result from the residual image. The initial value. The guide graph for guided filtering can be used. The brightness channel, the regularization coefficient is taken as The filter radius is set to 4 pixels.

[0125] In one implementation, the residual illumination variable Initialize to original game interactive scene material images The result obtained after large-scale guided filtering, the guided graph of the guided filter is used The filter radius is set to 15 to 31 pixels, and the regularization coefficient is set to... .

[0126] Update the user interface soft mask variables using an alternating optimization approach. Scene texture variables and residual illumination variables Specifically, first fix the scene texture variables. and residual illumination variables Update the user interface soft mask variable ; In updating the user interface soft mask variable At that time, based on the current reconstruction residual, determine whether each pixel is more suitable for the main color matrix of the user interface. The explanation then goes on to explain how to suppress isolated noise and jagged edges by maintaining smoothness at the edges.

[0127] In a convenient implementation method, the first The update method for the user interface soft mask data items in the next iteration is represented as follows:

[0128] ;in, Indicates the first Position in the next iteration The soft mask value for the user interface is obtained based on the data item; An iterative index indicating a union separation; This represents the color channel index, with a value ranging from 1 to 3; Represents original game interactive scene material images In position and color channels Pixel value at; Represents the main color matrix of the user interface In position and color channels The color value at that location; Indicates the first Scene texture variables in the next iteration In position and color channels The value at; Indicates the first Residual illumination variables in the next iteration In position and color channels The value at; This represents the stability constant for updating the soft mask in the user interface, used to prevent the denominator from being zero. An example value is shown below. .

[0129] Furthermore, regarding Perform guided filtering to obtain the updated user interface soft mask variables. The guiding graph for guided filtering can use original game interactive scene material images. For the luminance channel, the filter radius can be taken as 5 to 9 pixels, and the regularization coefficient can be taken as... Guided filtering can smooth mask values ​​within the user interface area while minimizing excessive diffusion of text, icons, and button boundaries.

[0130] It should be noted that guided filtering is a commonly used edge-preserving smoothing filter. Its core idea is to assume that the output image is a local linear transformation of the guide image. The linear coefficients are solved by minimizing the difference between the input image and the output image within a local window, thereby smoothing the image while preserving the edge structure of the guide image.

[0131] Fixed user interface soft mask variable and residual illumination variables Update scene texture variables Updating scene texture variables First, the original game interactive scene material images... The color components and residual lighting variables in the user interface cannot be determined. The explained portion is used as a texture candidate residual, i.e. Then, total variation denoising preserves structural edges and suppresses compression noise and color burrs.

[0132] in, Indicates the first The texture candidate residual in the next iteration is of size . The three-channel image. Then... Performing total variation denoising yields... .

[0133] In practical implementation, total variation denoising can be achieved using the Chambolle projection algorithm, alternating direction multiplier method, or other commonly used gradient soft thresholding shrinkage algorithms, with the number of iterations ranging from 10 to 30. This processing can reduce scene texture variables. The material structure of walls, floors, clothing, weapons, and vegetation is preserved, while the compression block effect and random noise are reduced in the subsequent texture orientation estimation process.

[0134] Fixed user interface soft mask variable and scene texture variables Update residual lighting variables In updating residual lighting variables First, calculate the candidate residuals of illumination.

[0135] Right now Then, large-scale smoothing filtering is used to preserve low-frequency brightness variations, shadow transitions, and color gradations, while suppressing high-frequency textures from entering residual lighting variables. .

[0136] in, Indicates the first The candidate illumination residuals in the next iteration are of size . Three-channel image.

[0137] In specific implementation, Perform large-scale guided filtering or Gaussian filtering to obtain The filtering radius can be taken from 21 to 41 pixels, and the regularization coefficient can be taken as... This process enables residual illumination variables to be adjusted. The main purpose is to express the smooth changes caused by scene lighting and semi-transparent overlays, avoiding the incorrect allocation of material particles, text edges, and icon outlines to residual lighting variables. middle.

[0138] Repeatedly update the user interface soft mask variables, scene texture variables, and residual lighting variables until a preset number of iterations is reached or the reconstruction error change is less than a preset threshold. The preset number of iterations is defined as follows: , This represents the maximum number of iterations for layer unification and separation, with values ​​ranging from 20 to 40. The reconstruction error variation threshold is defined as follows: , Used to determine whether the joint separation has converged; example values ​​are shown below. .

[0139] After the iteration is complete, the final user interface soft mask variable is defined as the user interface mask. The final scene texture variables are defined as the scene texture layer. The final residual illumination variable is defined as the residual illumination layer. Among them, the user interface mask The size is A single-channel image is used to characterize the confidence level of each pixel belonging to the user interface layer; scene texture layer. The size is The three-channel image is used to preserve material textures and local structures in the 3D rendered scene; residual lighting layer The size is The three-channel plot is used to characterize low-frequency illumination, color gradation, and shadow transition.

[0140] In one embodiment, as an example, an original game interactive scene material image A red health bar appears against a dark rock background. If segmented only by the red threshold, the warm reflections in the dark rocks might be mistaken for the health bar; if segmented only by edges, rock cracks might be mistaken for user interface boundaries. This application uses a user interface primary color matrix... Provide color priors via soft mask variables in the user interface. The boundary sparse constraints maintain the continuity of the health bar outline, using scene texture variables. Preserving rock cracks, through residual light variables It absorbs changes in shadow lighting, thereby assigning the health bar area, rock texture, and shadow lighting to different layers.

[0141] It should be noted that the joint separation of semantic layers in the user interface scene does not simply rely on color similarity, but simultaneously utilizes color reconstruction, boundary sparsity, texture fidelity, and lighting smoothing constraints. This design can reduce the probability of text, icons, buttons, and status bars being mistakenly enhanced as scene textures, and also reduce the probability of wall textures, ground textures, and character material edges being mistakenly included in the user interface area, thus providing a reliable structural prior for subsequent differential enhancement.

[0142] In one embodiment, such as Figure 2 As shown, the user interface mask is displayed using a heatmap. The specific pixel values ​​of the heatmap represent the confidence level of each pixel belonging to the user interface layer. The higher the brightness, the more prominent the user interface areas such as text, icons, buttons, status bars, and menu panels are.

[0143] Key areas for user interface enhancement are typically concentrated around text edges, icon outlines, button borders, status bar edges, and menu panel boundaries, while the interior areas of the user interface away from the boundaries should generally maintain smooth colors and avoid over-sharpening.

[0144] The user interface mask is converted into a boundary distance representation, enabling the augmentation network to explicitly perceive the location and influence range of the user interface boundary. The specific steps are as follows:

[0145] 1> User Interface Mask Perform binarization to obtain a binary mask for the user interface. User interface binary mask The size is A single-channel binary image is used to mark user interface areas and non-user interface areas.

[0146] In the specific implementation, the user interface mask binarization threshold is defined as follows: , Used to determine whether a pixel belongs to the user interface area; an example value is 0.5. When At that time, Set to 1; when At that time, Set to 0. For semi-transparent user interface areas, you can first apply a user interface mask. implement Gaussian smoothing is applied, with a Gaussian standard deviation of 0.8, followed by binarization to reduce breakage at the anti-aliasing edges.

[0147] 2> Based on the user interface binary mask Extracting the user interface boundary map User interface boundary diagram The size is A single-channel binary image is used to mark the boundary between the user interface area and the non-user interface area.

[0148] In one implementation, using Structural elements as binary masks for user interfaces Perform morphological dilation to obtain a dilated mask; use the same structuring element to create a binary mask for the user interface. Perform morphological erosion to obtain an erosion mask; then subtract the erosion mask from the dilation mask to obtain the user interface boundary map. This allows it to cover the edges of text strokes, icon outlines, button frames, and status bar boundaries.

[0149] 3> User Interface Boundary Map Perform a distance transformation, calculate the pixel distance from each pixel to the nearest user interface boundary, and truncate and normalize the distances to obtain the user interface boundary distance field. User interface boundary distance field The size is The single-channel image is used to represent the normalized distance of each pixel from the nearest user interface boundary. The closer the value is to 0, the closer it is to the user interface boundary, and the closer the value is to 1, the farther it is from the user interface boundary.

[0150] In one implementation, the definition is... Represents the distance field from the user interface boundary. The element in the i-th row and j-th column is calculated as follows: .in, Indicates position The pixel distance to the nearest user interface boundary; This represents the distance truncation threshold, used to limit the influence range of the user interface boundary. An example value is 16 pixels. Through the above processing, pixels located at the user interface boundary have a distance of 0, and positions with a distance greater than or equal to 16 pixels have a distance of 1.

[0151] In one embodiment, as an example, a user interface mask for the health bar area. When the value is greater than 0.5, the region is identified as the user interface region. After morphological boundary extraction of the blood bar region, distance transformation is used to calculate the distances from pixels inside the blood bar and neighboring pixels outside the blood bar to the blood bar outline. The user interface boundary distance field of pixels near the blood bar boundary is then calculated. Values ​​close to 0 are used, while values ​​close to 1 are used in the areas inside the blood bar far from the boundary. Subsequent enhancement networks can obtain clear boundary cues near the blood bar boundary, thereby improving the clarity of the outline; in the area inside the blood bar, the tendency for over-sharpening enhancement can be reduced, thereby maintaining smooth color filling.

[0152] In 3D rendered scenes, stone walls, wood grain, metal scratches, fabric wrinkles, tile edges, and grass textures are typically directional, while the sky, fog, low-frequency reflections from water, and solid-color backgrounds are less directional. If the enhancement network does not distinguish the reliability of texture direction, it is easy to introduce pseudo-textures in smooth areas and to disrupt texture continuity in directional material areas.

[0153] The scene texture principal orientation field and texture confidence map are estimated using a multi-directional Gabor filter bank. The specific steps are as follows:

[0154] 1> From scene texture layer Extract the brightness channel of the scene texture layer Scene texture layer brightness channel The size is A single-channel image is used to characterize the scene texture layer. The brightness structure is obtained by reducing the interference of color changes on texture direction estimation, and the acquisition method is represented as follows: .in, Represents scene texture layer The red channel Represents scene texture layer Green channel Represents scene texture layer The blue channel.

[0155] 2> Construct a multi-directional Gabor filter bank and calculate the brightness channels of the scene texture layer separately. Local texture response in different directions. The total number of directions is defined as follows: , Used to control the accuracy of direction analysis, with an example value of 8; The angle of each filtering direction is defined as , Used to determine the first The direction of a Gabor filter is denoted as... ,in, Indicates the filter direction index, with a value range of 1 to... The spatial scale parameter of the Gabor filter is defined as follows: , Used to control the local sensing range of the filter, with an example value of 3.0; the Gabor filter frequency parameters are defined as follows: , Used to control the position of the filter's response to the texture frequency; an example value is 0.25.

[0156] In the specific implementation, the scene texture layer brightness channel respectively with A Gabor filter is convolved, and the absolute value of the convolution result is taken as the texture response map in different directions. Among them, the th Gabor filter... The texture response map for each filtering direction is defined as follows: , The size is A single-channel image is used to characterize the brightness channel of the scene texture layer. In the Local frequency response in each direction. To reduce the impact of noise, one can... implement Mean filtering or Gaussian smoothing.

[0157] 3> Compare the response intensity of the texture response maps in each direction at each pixel location, and select the direction with the highest response intensity as the local texture master direction. The master direction index is defined as follows: , Used to indicate location The index of the filtering direction with the strongest response, i.e. The main direction field of scene texture is defined as follows: , The size is A single-channel map is used to characterize the orientation of the most salient texture structure in the neighborhood of each pixel, defining... ,in, Indicates the first Texture response map of each filtering direction In position The response value at the location; This indicates the filtering direction angle corresponding to the direction of the strongest response.

[0158] 4> Calculate the texture confidence map based on the proportion of the principal direction response among all direction responses. Texture confidence map The size is The single-channel image is used to characterize the reliability of local texture direction. The larger the value, the more obvious the directional texture; the smaller the value, the weaker the directionality or the closer the responses in each direction. The calculation method is expressed as follows:

[0159] ;

[0160] in, Indicates position The response value corresponding to the main direction; This represents the stability constant used in texture confidence calculations to prevent the denominator from being zero. An example value is shown below. .

[0161] Furthermore, if the location The sum of responses in all directions is less than a preset texture energy threshold. Then Setting it to 0 avoids unreliable directional priors from low-texture areas such as sky, fog, and solid-color backgrounds. The preset texture energy threshold is used for this purpose. Used to determine whether the current position has sufficient texture energy, with values ​​ranging from 0.01 to 0.03.

[0162] In one embodiment, for example, for areas of stone walls, wood grain, grass, and metal scratches, the multi-directional Gabor filter bank produces a strong response in one direction or a few adjacent directions, in which case the texture confidence map... Higher values ​​allow subsequent enhancement networks to recover continuous material details along the main texture direction. For areas like sky, fog, and solid-color menu backgrounds, responses in all directions are similar or the overall response is weak; in these cases, the texture confidence map... The value is set relatively low, and the subsequent enhancement network reduces the directional sharpening intensity to reduce pseudo-textures and noise amplification.

[0163] In one embodiment, such as Figure 3 As shown, the scene texture layer represents the material texture, geometric edges, and local structure in the 3D rendered scene. The stronger the response, the more obvious the high-frequency textures in areas such as stone walls, ground, vegetation, weapons, and character clothing are.

[0164] To enable the augmented network to simultaneously perceive basic color content, user interface boundary location, scene texture orientation, and texture orientation reliability, this application concatenates the basic augmented image, user interface boundary distance field, and texture orientation encoding map along the channel dimension to form a multimodal input tensor. The specific steps are as follows:

[0165] Based on scene texture layer and residual light layer Generate basic enhanced image Basic Enhanced Image The size is The three-channel image is used to provide initial brightness correction and scene texture enhancement results.

[0166] In the specific implementation, the scene texture layer is first processed. The luminance channel performs limited contrast adaptive histogram equalization, and the grid size can be set to... The contrast clipping threshold can be set from 2.0 to 4.0 to improve the visibility of material textures and local edges; then, the residual lighting layer... Perform gamma correction, the gamma coefficient is defined as follows: , This is used to adjust the low-frequency brightness level, with values ​​ranging from 0.8 to 1.2. Then, the enhanced scene texture result is fused with the gamma-corrected residual lighting layer (specifically, fusion is performed element-wise) to obtain the basic scene enhancement result.

[0167] It should be noted that contrast-limited adaptive histogram equalization is a commonly used image enhancement technique. It improves the local contrast of an image by performing histogram equalization in local grid regions of the image and cropping the histogram to limit noise amplification.

[0168] It should also be noted that gamma correction is a commonly used non-linear image processing operation that adjusts the brightness of an image by applying a power-law transformation to pixel values.

[0169] Furthermore, the user interface mask Copy along the color channel direction to create a three-channel user interface mask and according to Original game interactive scene material images The image is fused with the scene augmentation results to obtain the base augmentation image. This allows for the retention of more original game interaction scene images in the user interface area. In non-user interface areas, more scene-based enhancement results are used to avoid color overflow or edge jaggedness in text, icons, and status bars during the basic enhancement stage, as represented by... ,in, Enhance the results based on the scene. The scalar 1 in the value is broadcast to the same size as the three-channel user interface soft mask, and element-wise subtraction is performed.

[0170] For the main direction field of scene texture Continuous direction encoding is performed. Since texture direction is undirected, 0 degrees and 180 degrees represent the same texture direction. Directly inputting angle values ​​can easily cause numerical jumps. This application uses bi-angle cosine encoding and bi-angle sine encoding to express texture direction, so as to ensure that texture directions close to 0 degrees and close to 180 degrees are continuous in numerical expression.

[0171] In practical implementation, through Calculate the confidence-weighted texture orientation cosine map using the following method and through Calculate the confidence-weighted texture orientation sine map using the following method ,in, The size is A single-channel image is used to represent the biangular cosine components of the main texture direction; The size is A single-channel image is used to represent the bi-angle sine component of the main texture direction; Represents the cosine function; Represents a sine function. (Based on texture confidence maps) Weighting the direction encoding can strengthen the direction prior in reliable direction regions and reduce the strength of the direction prior in unreliable direction regions.

[0172] Enhance the base image User interface boundary distance field Confidence-weighted texture orientation cosine map Confidence-weighted texture orientation sine map and texture confidence map By concatenating along the channel dimension, a multimodal input tensor is obtained. Multimodal input tensor The size is The tensor is used as input to a structure-prior-guided dynamic pixel-by-pixel filtering enhancement network, enabling the enhancement network to simultaneously perceive image color content, user interface boundary locations, scene texture orientation, and texture reliability. .in, This represents the channel stitching operator, used to combine multiple feature maps of the same spatial size along the channel dimension; base enhancement image. Provides 3 channels, user interface boundary distance field Provides one channel, confidence-weighted texture orientation cosine map Provides one channel, confidence-weighted texture orientation sine curve Provides one channel, texture confidence map It provides 1 channel, which can be spliced ​​together to form 7 channels.

[0173] It should be noted that multimodal input tensors Instead of simply adding auxiliary channels, the algorithm provides the user interface boundary distance and scene texture direction as structural priors to the augmentation network. This enables the augmentation network to obtain clear boundary cues near the boundaries of text, icons, buttons, and status bars, directional cues in areas with wood grain, stone walls, metal scratches, and fabric wrinkles, and to reduce the intensity of directional augmentation in areas with sky, fog, and solid colors, thereby improving the regional adaptation capability of the augmentation results.

[0174] In one embodiment, such as Figure 4 As shown, the residual lighting layer represents low-frequency lighting, color gradation, and shadow transitions.

[0175] Step 140: Input the multimodal input tensor into the dynamic pixel-wise filtering enhancement network, and use the multimodal input tensor to obtain coded feature maps at several scales; use the user interface boundary distance field to obtain user interface boundary distance fields at several scales; use the coded feature maps and user interface boundary distance fields to obtain modulation feature maps at each scale.

[0176] Multimodal input tensor Including basic color content, user interface boundary distance prior, scene texture direction prior, and texture confidence prior, conventional encoder-decoder enhancement networks usually only enhance based on pixel color and convolution features, making it difficult to distinguish user interface edges, text outlines, directional material textures, and low-frequency lighting areas. This can easily lead to overly sharp user interface areas, discontinuous texture directions, and overall color shifts.

[0177] This application constructs a structure-prior-guided dynamic pixel-by-pixel filtering enhancement network. By constructing a multi-scale feature extraction path modulated by the structure prior and predicting the output path of the dynamic filtering kernel, the enhancement network can adopt different enhancement methods for different semantic structures.

[0178] To continuously utilize user interface boundary distance priors and scene texture orientation priors within the network, this application sets up structure prior modulation subnetworks at multiple scales in the encoder-decoder network. These subnetworks generate spatially variable gain maps and spatially variable bias maps based on the user interface boundary distance field and texture structure prior map at the current scale, and adaptively recalibrate the encoded feature maps. The specific steps are as follows:

[0179] 1> Input multimodal tensors Inputting the lightweight encoder-decoder network yields encoded feature maps at multiple scales. Among them, the... The encoded feature map at each scale is defined as , The size is The feature map is used to represent color, edge, texture and semantic information at the current scale; This represents the scale index, with values ​​ranging from 1 to... ; This represents the total number of scales, with an example value of 4. Indicates the first Feature map height at each scale; Indicates the first Feature map width at each scale; Indicates the first The number of feature channels at each scale has the following values: 32, 64, 128, and 256.

[0180] In one implementation, the encoder includes two encoders per scale. Convolutional layers, non-linear activation functions, and downsampling layers; downsampling can be achieved using convolutions with a stride of 2 or... Max pooling. Each scale of the decoder includes an upsampling layer, a skip connection fusion layer, and... Convolutional layers. This network structure can preserve user interface edges and local textures in shallow layers, and obtain a wider range of lighting and semantic context in deeper layers. In this implementation, the specific structure of the lightweight encoder-decoder network is as follows:

[0181] a> Encoder: contains One scale, The first metric is to... After two 3×3 convolutional layers (stride 1, padding 1), followed by a ReLU activation function, downsampling is performed using a 2×2 max pooling layer with a stride of 2 to obtain the feature map. Its number of channels The space dimensions are And so on, the first Each scale will After passing through convolution and pooling layers of the same structure, we obtain Its number of channels The numbers are 64, 128, and 256 respectively, with spatial dimensions of... ;

[0182] b>Decoder: Contains One scale; the first scale pair Bilinear interpolation upsampling is performed to double the spatial size, followed by two 3×3 convolutional layers, and then compared with the modulated feature map of the corresponding scale. The data is stitched and blended along the channel dimension; this process is repeated four times, and after four upsampling and feature fusion operations, the final output spatial size is restored to its original value. The number of channels is Full-resolution decoding feature map , .

[0183] 2> Distance field of user interface boundary Downsampling to the The metric is obtained to get the ... User interface boundary distance field at each scale , No. User interface boundary distance field at each scale The size is A single-channel image is used to locate the user interface boundary region at the current scale. In one implementation, downsampling can be performed using bilinear interpolation, while maintaining the numerical range. .

[0184] Furthermore, the confidence-weighted texture orientation cosine map is then used. Confidence-weighted texture orientation sine map and texture confidence map Downsampling to the The scale is then spliced ​​along the channel dimension to obtain the first scale. Texture structure priors at various scales . No. Texture structure priors at various scales The size is The feature map is used to express the texture orientation, texture orientation continuity, and orientation reliability at the current scale.

[0185] 3> The first User interface boundary distance field at each scale and the Texture structure priors at various scales Enter the first A priori modulation subnetwork of a single scale , generate the first Spatial variable gain diagram at each scale and the Spatial variable bias map at various scales Among them, the first A priori modulation subnetwork of a single scale This is used to map structural priors to feature modulation parameters, with convolutional kernel weights being trainable parameters; the... Spatial variable gain diagram at each scale The size is The feature map is used to control the enhancement magnitude of the current-scale encoded feature map at different spatial locations and different channels; the first Spatial variable bias map at various scales The size is The feature map is used to compensate for differences in brightness, texture, and edge response in different regions.

[0186] In one implementation, a priori modulated subnetwork is constructed. It consists of one 1×1 convolution, one 3×3 depthwise separable convolution, and two parallel 1×1 convolution branches with gain / bias. Specifically, the... A priori modulation subnetwork of a single scale First and By stitching along the channel dimensions, a 4-channel structural prior image is obtained; then... Convolution maps a 4-channel prior map to the number of intermediate channels, where the number of intermediate channels can be chosen from... Or the larger of 16; then use Depthwise separable convolution extracts the local relationship between the user interface boundary and texture orientation; finally, it is divided into gain branch and bias branch: the gain branch outputs the first... Spatial variable gain diagram at each scale And use the hyperbolic tangent function to Limited to Range; bias branch output number Spatial variable bias map at various scales It employs a linear mapping or a hyperbolic tangent function with a scaling factor to limit the bias amplitude. This implementation has a small number of parameters, making it suitable for batch enhancement and real-time enhancement tasks of game assets.

[0187] 4> According to the first Spatial variable gain diagram at each scale and the Spatial variable bias map at various scales For the first Encoded feature maps at various scales Modulation is performed to obtain the first Modulation feature map at each scale , No. Modulation feature map at each scale The size is The feature map is used to pass enhanced features with user interface boundary awareness and texture orientation awareness to the decoder.

[0188] In one implementation, the first Modulation feature map at each scale It is obtained by performing spatial adaptive gain modulation and bias compensation on the encoded feature map, and is expressed as:

[0189] ;

[0190] in, This represents the modulation intensity coefficient, used to control the magnitude of the influence of structural priors on the encoded feature map; an example value is 0.5. Because... Restricted to Range, when At that time, the gain range is This approach avoids excessive perturbation of the encoded feature map by structural priors. Feature channels near the user interface boundary can obtain higher edge responses, directionally relevant channels in reliable texture orientation regions can obtain stronger activations, and the enhancement tendency in unreliable or smooth regions is appropriately suppressed.

[0191] It should be noted that, as Figure 5 As shown, the structure prior-driven multi-scale feature modulation injects the user interface boundary distance prior and scene texture direction prior into the network's intermediate features, instead of just splicing auxiliary channels at the input end. This enables shallow features to process text edges and icon outlines more accurately, and allows deep features to combine a wide range of context to adjust material textures and low-frequency lighting, thereby improving the augmented network's ability to adapt to heterogeneous regions.

[0192] Step 150: Using the modulation feature map, obtain the full-resolution decoded feature map; using the full-resolution decoded feature map, predict the dynamic filtering weights and dynamic bias terms at each pixel location; based on the base enhanced image, dynamic filtering weights, and dynamic bias terms, obtain the enhanced image.

[0193] It should be noted that different game interaction scene areas require different filtering methods. The edges of text and the outlines of icons need to be kept sharp and burrs should be avoided. The directional material textures need to restore continuous details along the main direction. The sky, fog and solid color areas need to suppress noise.

[0194] This application predicts the dynamic filtering weights and dynamic bias terms for each pixel location based on the decoder output features, and applies the dynamic filtering weights to the base enhancement image. The enhanced image is obtained by analyzing the local neighborhood of the image. The specific steps are as follows:

[0195] 1> Input the modulation feature maps at each scale into the decoder (i.e., the decoder part in the lightweight encoder-decoder network) for step-by-step upsampling and feature fusion to obtain the full-resolution decoded feature map. Full-resolution decoding feature map The size is The feature map is used to summarize user interface boundaries, scene texture orientation, basic colors, and multi-scale contextual information. This indicates the number of feature channels for full-resolution decoding, with values ​​of 32 or 64.

[0196] In practical implementation, each stage of upsampling in the decoder can be achieved using bilinear interpolation. Convolution, or transposed convolution, can be used. To avoid checkerboard artifacts, bilinear interpolation combined with convolution is preferred. After each level of upsampling, the corresponding scale of the modulated feature map is... By using a skip connection to the input decoder, user interface boundary information and texture direction information can participate in dynamic filtering weight prediction at different resolutions.

[0197] 2> Decode the feature map at full resolution Input the dynamic filter prediction head to predict the dynamic filter weights at each pixel location. and dynamic bias terms Among them, dynamic filter weights The dynamic bias term is used to control the adaptive fusion intensity of the current pixel with its neighboring pixels. Used to compensate for local brightness and color differences. The dynamic filter radius is defined as follows: , Used to determine the pixel-by-pixel filtering neighborhood range, with an example value of 1, corresponding to... Local filter kernel.

[0198] In its implementation, the dynamic filtering prediction head consists of two 3×3 convolutional ReLU layers and one 1×1 convolutional output layer. The 1×1 convolutional output layer maps the feature channels to... Each of the three color channels corresponds to the position of each pixel in the three color channels. One dynamic filter weight and one dynamic bias term. When At this time, each color channel needs to predict 9 dynamic filter weights, for a total of 27 dynamic filter weights across the red, green, and blue color channels, plus an additional 3 dynamic bias terms. To avoid brightness divergence caused by dynamic filtering, Softmax normalization can be performed on the 9 dynamic filter weights at the same pixel location and within the same color channel, ensuring the sum of the neighborhood weights is 1; the dynamic bias terms can be constrained using the hyperbolic tangent function. Scope, of which, This indicates the upper limit of the dynamic bias amplitude, with an example value of 0.05.

[0199] 3> Enhance the base image Boundary padding is performed to ensure that edge pixels also have their complete neighborhood. Boundary padding can be achieved using mirror padding or copy padding, and the padding width is equal to the dynamic filtering radius. Mirror fill can reduce abrupt artifacts at image edges, making it suitable for enhancing edge areas of game assets.

[0200] 4> At each pixel location, assign the corresponding dynamic filtering weight. Acting on the basic enhanced image The local neighborhood, plus a dynamic bias term. To obtain an enhanced image Enhance the image The size is The three-channel image refers to a high-quality game interactive scene material image after being dynamically filtered pixel by pixel under the guidance of structural prior.

[0201] In one implementation, the definition is... Indicates enhanced image In position and color channels The pixel value at that location is calculated as follows:

[0202] ;

[0203] in, This represents the vertical neighborhood offset index, used to traverse the neighborhood positions in the vertical direction of the current pixel, with a value range of [value missing]. to ; This represents the horizontal neighborhood offset index, used to traverse the neighboring positions in the left and right directions of the current pixel. Its value range is... to ; Indicates position Color channels Neighborhood offset The corresponding dynamic filter weights; Represents the basic enhanced image In the neighborhood and color channels Pixel value at; Indicates position and color channels The corresponding dynamic bias term.

[0204] Furthermore, numerical cropping is performed on the dynamic filtering output to enhance the image. Limited to The range can be determined, or the valid pixel value can be obtained through the output Sigmoid function, thereby avoiding pixel values ​​going out of bounds due to dynamic filtering weights and dynamic bias terms.

[0205] In one embodiment, for example, when the current position is near the edge of text or the outline of an icon, the user interface boundary distance field... The location is near the user interface boundary, and the dynamic filtering prediction head emphasizes the dynamic filtering weights of high-contrast pixels on both sides of the edge, thereby improving the clarity of text and icon outlines. When the current location is in a directional texture area such as wood grain or metal scratches, the texture structure prior map provides the main direction information, and the dynamic filtering weights are distributed along the main texture direction, thus restoring continuous material details; when the current location is in a sky, fog, or solid color background area, the texture confidence map... When the value is low, the dynamic filtering weights tend to be smooth and uniform, thereby suppressing noise and artifacts.

[0206] In one embodiment, such as Figure 6 As shown, the response strength of the dynamic filtering weights represents the adaptive fusion strength of the dynamic pixel-by-pixel filtering enhancement network at different pixel positions. A higher response indicates that the region needs more structure-aware enhancement for user interface boundaries or scene textures.

[0207] Step 160: Calculate the composite loss function based on the enhanced image and the corresponding processed high-quality image; optimize the trainable parameters in the dynamic pixel-by-pixel filtering enhancement network using the composite loss function to obtain the trained dynamic pixel-by-pixel filtering enhancement network.

[0208] To achieve a balance between overall image quality, user interface edges, high-frequency details of scene textures, and global color stability in the augmentation network, a high-quality reference image is used as the supervision target during training, and a composite loss function is constructed. The specific steps are as follows:

[0209] 1> Original game interaction scene material images in each training sample Perform step S2 to obtain the multimodal input tensor. User interface mask User interface boundary distance field and texture confidence map Then, the multimodal input tensor Input the image into the dynamic pixel-by-pixel filtering enhancement network to obtain the enhanced image. At the same time, high-quality reference images will be used. Used to provide monitoring objectives.

[0210] 2> Calculate pixel fidelity loss Pixel fidelity loss Used for constrained image enhancement With high-definition reference images Overall consistency in pixel values.

[0211] In the specific implementation, the enhanced image and high-definition reference images The absolute error is calculated pixel-by-pixel and channel-by-channel and then averaged. .in, Indicates pixel fidelity loss; This represents the L1 norm, used to calculate the absolute error. Using the L1 norm can reduce the excessive influence of a small number of local outliers on training, allowing the network to learn overall image quality improvements more stably.

[0212] 3> Calculate the user interface edge consistency loss User interface edge consistency loss The edge strength of user interface areas such as text, icons, buttons, and status bars is used to ensure consistency with the high-quality reference image.

[0213] In the specific implementation, the enhanced images are first processed separately. and high-definition reference images The gradient map of the enhanced image is obtained by calculating the gradient map using the Sobel gradient operator. Gradient map of high-resolution reference image Then, use a user interface mask. and user interface boundary distance field Constructing the User Interface Edge Weight Graph ,Right now ,in, This represents the edge weight map of the user interface, which is of size [size missing]. A single-channel image is used to highlight areas near the boundaries in the user interface area; then, Copy along the color channel direction to obtain a three-channel user interface edge weight map. Then the user interface edge consistency loss. The gradient consistency between the enhanced image and the high-quality reference image in the edge region of the user interface is calculated by constraining the gradient consistency. This is represented as... By calculating the gradient error using this weight map, the network can focus on learning text strokes, icon outlines, button borders, and status bar boundaries, without over-sharpening the solid color filled areas inside the user interface.

[0214] 4> Calculate the high-frequency consistency loss of scene textures High-frequency consistency loss of scene texture Used to constrain material particles, texture grooves, fine edges, and local abrupt changes.

[0215] In the specific implementation, the enhanced image is first... Convert to enhance image luminance channel (Obtained through standard color space conversion), high-quality reference image Convert to high-quality reference image luminance channel Then, the high-frequency response is extracted using the Laplacian high-frequency operator to obtain the high-frequency response of the enhanced image and the high-quality reference image; finally, the texture confidence map is used. Weighting of high-frequency errors, i.e. .in, This indicates an enhanced image luminance channel, which is a channel with a size of Single-channel images are used to characterize enhanced images. The brightness structure; This represents the luminance channel of a high-quality reference image, and its size is [size missing]. A single-channel image is used to provide a reference texture structure; This represents the Laplacian high-frequency operator (i.e., the standard Laplacian operator), used to extract information about material particles, texture grooves, fine edges, and local abrupt changes in an image.

[0216] It should be noted that, through texture confidence maps After weighting, the network prioritizes high-frequency structure recovery in reliable directional or high-frequency texture areas such as stone walls, ground, character clothing, weapons, and vegetation, while reducing the intensity of high-frequency constraints in sky, fog, and solid color areas to prevent the network from generating pseudo textures to match high-frequency losses.

[0217] 5> Calculate the global color stability loss Global color stability loss Used for constrained image enhancement With high-definition reference images The global mean difference across the red, green, and blue channels is calculated as follows: .in, This represents the channel mean operator, used to calculate the global mean of an image across the red, green, and blue channels (achieved by calculating the arithmetic mean of all pixel values ​​in the red, green, and blue channels of the input image, respectively). By constraining the differences in channel means, it can reduce overall color cast without suppressing local detail enhancement like strong per-pixel color constraints.

[0218] 6> Reduce pixel fidelity loss User interface edge consistency loss High-frequency consistency loss of scene texture and global color stability loss By performing a weighted summation, we obtain the composite loss function. ,Right now Composite loss function Used for joint measurement of enhanced images With high-definition reference images The differences between pixels, edges, textures, and color statistics.

[0219] in, This represents the user interface edge consistency weight, used to control the user interface edge constraints in the composite loss function. The contribution in the calculation is 0.8, for example. This represents the high-frequency consistency weight of the texture, used to control the high-frequency structure constraints of the texture confidence region in the composite loss function. The contribution in the calculation is 0.6, for example. The color stability weights are used to control the global color statistical constraints in the composite loss function. The contribution is shown in the example, with a value of 0.1.

[0220] In one embodiment, as an example, the user interface edge consistency weight When the value is 0.8, the network places greater emphasis on edge consistency in user interface areas such as text, buttons, health bars, and icons; texture high-frequency consistency weight. When the weight is 0.6, the network prioritizes the high-frequency structure restoration of texture areas such as walls, floors, weapons, character clothing, and vegetation; color stability weight... When the value is 0.1, the network can reduce the overall color bias while retaining the local enhancement degree of freedom.

[0221] Parameter optimization can:

[0222] To ensure that the structure-prior-guided dynamic pixel-wise filtering enhancement network can operate stably under different game screens, user interface layouts, and material scenes, this application performs end-to-end parameter optimization based on the training sample set. The specific steps are as follows:

[0223] 1> Define all trainable parameters of the structure-prior-guided dynamic pixel-wise filtering enhancement network as After training, the optimal set of network parameters is obtained. , Used to generate enhanced images based on new original game interaction scene material images during the deployment phase.

[0224] 2> For the training sample set For each training sample in the dataset, a forward propagation is performed, where the first... The composite loss function value corresponding to each training sample is defined as follows: , Used to measure the The augmentation error of each training sample.

[0225] 3> Using the training sample set The objective is to minimize the average composite loss function and optimize the set of trainable parameters of the network. The parameter optimization objective is expressed as .in, Represents the optimal set of network parameters. This represents the set of trainable parameters of the network. This represents the total number of training samples. Indicates the first The composite loss function value corresponding to each training sample.

[0226] During training, an adaptive moment estimation optimizer is used to update the set of trainable parameters of the network. The initial learning rate can be set to... The batch size can be set to 8, the number of training rounds can be set to 200, and the input image cropping size can be set to... During training, a learning rate decay strategy can be adopted based on the decrease in validation set loss. For example, if the validation set loss does not decrease for 5 consecutive rounds, the learning rate can be multiplied by 0.5.

[0227] Optionally, the modulation intensity coefficient can be reduced in the early stages of training. Alternatively, partial structural prior modulation sub-network parameters can be frozen, allowing the network to first learn basic color and texture recovery capabilities. Full structural prior modulation can then be enabled later in the training process, enabling the network to gradually learn the impact of user interface boundaries and scene texture orientation on dynamic filtering weights. This training strategy can reduce the risk of color flickering, edge ringing, and local noise amplification caused by instability in dynamic filtering during the early stages of training.

[0228] 4. After each training epoch, evaluate the augmented network using validation samples. Validation metrics can include peak signal-to-noise ratio (PSNR), structural similarity, perceptual similarity, and user interface edge consistency metrics. User interface edge consistency metrics can be found in the user interface edge weight graph. Gradient error is calculated within the weighted region to specifically evaluate the quality of boundary recovery for text, icons, buttons, and status bars. The optimal set of network parameters corresponding to the lowest composite loss function or the best overall performance index on the validation set is saved. .

[0229] It should be noted that the composite loss function The user interface edge consistency loss and scene texture high-frequency consistency loss are not uniform enhancements of the entire graph, but are instead derived from the user interface edge weight graph. and texture confidence map Constraints are imposed, which enable the network to enhance contours in user interface boundary areas, recover high-frequency structures in reliable texture areas, and suppress artifacts in low-texture areas, thereby improving the visual naturalness and structural accuracy of the enhancement results.

[0230] It should also be noted that the dynamic pixel-wise filtering enhancement network consists of a lightweight encoder-decoder network, a multi-scale structural prior modulation subnetwork, and a dynamic filtering prediction head. Specifically, the multimodal input tensor... First, the code passes through a 4-layer encoder, with each layer containing two 3×3 convolutions, ReLU activation, and a 2×2 max pooling with a stride of 2, resulting in encoded feature maps at four scales. At each scale between the encoder and decoder, the downsampled structural prior map ( and Input structure prior modulator network Generate a spatially variable gain map and bias map ,right Modulation is performed to obtain Then, a 4-layer decoder progressively upsamples the deepest feature maps, improving resolution through bilinear interpolation and 3×3 convolutions, and fuses the corresponding scales through skip connections. The final output is a full-resolution decoded feature map. . Then input the dynamic filtering prediction head to predict the dynamic filtering kernel for each pixel. and bias and apply it to the base enhancement image. The final enhanced image is obtained from the above. .

[0231] Game interactive scene asset image deployment enhancement process:

[0232] After training is complete, it will have the optimal set of network parameters. The structure-prior-guided dynamic pixel-by-pixel filtering enhancement network is deployed to game asset processing systems, game engine post-processing modules, asset batch enhancement tools, or cloud image processing services. During the deployment phase, the new original game interaction scene asset images undergo the same preprocessing and enhancement process as during the training phase. The specific steps are as follows:

[0233] 1> Obtain the original game interaction scene material images to be enhanced. Original game interaction scene assets images that need enhancement The size is The red, green, and blue three-channel image, in which, This indicates the height of the input image during the deployment phase. This indicates the width of the input image during the deployment phase. Pixel values ​​normalized to Range, while maintaining the original spatial resolution.

[0234] 2> Enhanced original game interaction scene assets images The execution steps S2, including user interface dominant color matrix estimation, joint separation of user interface scene semantic layers, generation of user interface boundary distance field, estimation of scene texture principal direction field and texture confidence map, and construction of multimodal input tensors, yield the multimodal input tensor for the deployment phase. Multimodal input tensors during deployment phase The size is The tensor is used as input to the trained, structure-prior-guided, dynamic pixel-by-pixel filtering enhancement network.

[0235] It should be noted that the color clustering parameters, user interface mask binarization threshold, distance truncation threshold, number of Gabor filter directions, and texture confidence calculation method during the deployment phase should be consistent with those during the training phase to ensure that the numerical distribution of the structural prior matches that of the training phase.

[0236] 3> Deployment phase multimodal input tensor Input has the optimal set of network parameters The structure-prior-guided dynamic pixel-by-pixel filtering enhancement network sequentially performs multi-scale feature modulation, decoding feature fusion, dynamic filter weight prediction, and dynamic pixel-by-pixel filtering to obtain the enhanced image at the deployment stage. Enhanced images during deployment phase The size is The three-channel image refers to the original game interactive scene material image to be enhanced. The result of the enhanced structure awareness.

[0237] 4> Enhanced images during the deployment phase Perform post-processing of the output. First, ... Cut to The range is then converted to an 8-bit integer image, a 10-bit integer image, or a floating-point texture image based on the target output format; if the deployment scenario requires the alpha channel to be preserved, the original alpha channel is then combined with the enhanced image from the deployment stage. Reassemble to obtain an enhanced material with an alpha channel.

[0238] In one embodiment, for example, for a resolution of The game screenshots are generated during the deployment phase. First, the dominant color matrix of the user interface is estimated, and the user interface mask, scene texture layer, and residual lighting layer are separated. Then, the user interface boundary distance field, texture orientation encoding map, and texture confidence map are generated and stitched with the base augmented image. The deployment phase uses a multimodal input tensor; finally, it inputs a trained, structure-prior-guided, dynamic pixel-by-pixel filtering enhancement network, outputting an enhanced image for the deployment phase. The enhancement results can improve the texture clarity of 3D rendered scene areas such as rocks, floor tiles, weapons, character clothing, and vegetation while maintaining the stability of health bars, skill icons, and text colors, and reduce pseudo-textures in the sky, fog, and solid color panels.

[0239] Based on the above description, this application proposes a layer semantic separation preprocessing method, which decomposes the image into a user interface mask, a scene texture layer and a residual lighting layer through joint optimization, providing a structural basis for the differentiated enhancement of heterogeneous regions.

[0240] A multimodal structural prior is constructed, and the boundary influence range is explicitly located using the user interface boundary distance field. At the same time, the scene material orientation and reliability are described by a confidence-weighted texture direction encoding field.

[0241] A multi-scale feature transformation network with structural prior modulation is adopted to generate spatial adaptive gain and bias in each coding layer based on boundary and orientation priors, thereby realizing region-aware recalibration at the feature level.

[0242] The output is enhanced by dynamic pixel-by-pixel filtering. The network predicts the adaptive filter kernel and bias term for each position, enabling text boundary sharpening, directional texture continuous restoration, and flat area noise suppression to be completed collaboratively within the same framework.

[0243] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for enhancing the image processing of game interactive scene materials based on virtual reality, characterized in that, The method includes: Collect original game interaction scene material images and construct corresponding preset high-definition reference images; perform preprocessing to obtain the processed original image and the processed high-definition image; The processed original image is converted to a preset color space to obtain the main color tone matrix of the user interface; based on the processed original image and the main color tone matrix of the user interface, the color components of the user interface, scene texture variables and residual lighting variables are decomposed to obtain the color components of the user interface, scene texture variables and residual lighting variables. Using original game interactive scene material images, the user interface soft mask variables, scene texture variables, and residual lighting variables are initialized; the user interface soft mask variables, scene texture variables, and residual lighting variables are updated using an alternating optimization method until a preset stopping update condition is met, thereby obtaining the user interface mask, scene texture layer, and residual lighting layer; based on the user interface mask, scene texture layer, and residual lighting layer, the multimodal input tensor is obtained; A multimodal input tensor is input into a dynamic pixel-wise filtering enhancement network. Using the multimodal input tensor, encoded feature maps at several scales are obtained. Using the user interface boundary distance field, user interface boundary distance fields at several scales are obtained. Using the encoded feature maps and user interface boundary distance fields, modulation feature maps at each scale are obtained. Using the modulation feature map, a full-resolution decoded feature map is obtained; using the full-resolution decoded feature map, the dynamic filtering weights and dynamic bias terms at each pixel location are predicted; based on the base enhanced image, dynamic filtering weights, and dynamic bias terms, the enhanced image is obtained. Based on the enhanced image and the corresponding processed high-quality image, a composite loss function is calculated; the trainable parameters in the dynamic pixel-by-pixel filtering enhancement network are optimized using the composite loss function to obtain the trained dynamic pixel-by-pixel filtering enhancement network.

2. The method for enhancing game interactive scene material images based on virtual reality according to claim 1, characterized in that, Collect original game interaction scene material images and construct corresponding preset high-definition reference images; Perform preprocessing to obtain the original image and the high-quality image after processing, specifically including: Collect original game interaction scene material images, and then... Each original game interactive scene material image is defined as , The size is The red, green, and blue three-channel image, in which, Indicates the sample index; Indicates the image height; Indicates the image width; For each original game interaction scene material image, a corresponding preset high-quality reference image is constructed; The original game interactive scene material images and the preset high-quality reference images are used to form training sample pairs; Perform spatial alignment between the original game interactive scene material images and the preset high-definition reference images; The pixel values ​​of the original game interactive scene material images and the preset high-quality reference images, after spatial dimension alignment, are normalized to... The range is used to obtain the original image after processing and the high-quality image after processing.

3. The method for enhancing game interactive scene material images based on virtual reality according to claim 1, characterized in that, The processed original image is converted to a preset color space to obtain the main color matrix of the user interface, which specifically includes: The processed original image is converted to a preset color space to obtain a perceptually uniform color map. For each pixel position of the perceived uniform color map Calculate the local color variance and local gradient magnitude; Pixels with local color variance less than a preset color variance threshold and local gradient magnitude less than a preset magnitude threshold are selected as flat candidate pixels. Perform color clustering on the perceptually uniform color vectors corresponding to flat candidate pixels to obtain K color clusters; According to preset filtering rules, candidate primary color clusters for the user interface are determined from K color clusters; Calculate the user interface primary color vector based on the candidate primary color clusters of the user interface, and copy the user interface primary color vector to the entire image space to obtain the user interface primary color matrix.

4. The method for enhancing game interactive scene material images based on virtual reality according to claim 1, characterized in that, Based on the processed original image and the user interface dominant color matrix, the color components of the user interface, scene texture variables, and residual lighting variables are decomposed and obtained, specifically including: Based on the processed original image, define user interface soft mask variables, scene texture variables, and residual lighting variables; Copy the user interface soft mask variable along the red, green and blue channel directions to obtain a three-channel user interface soft mask variable; The user interface color components are obtained by performing pixel-by-pixel and channel-by-channel multiplication on the three-channel user interface soft mask variable and the user interface main color matrix.

5. The method for enhancing game interactive scene material images based on virtual reality according to claim 1, characterized in that, Using original game interaction scene material images, the user interface soft mask variables, scene texture variables, and residual lighting variables are initialized. An alternating optimization method is then used to update the user interface soft mask variables, scene texture variables, and residual lighting variables until a preset stopping condition is met, resulting in the user interface mask, scene texture layer, and residual lighting layer. Specifically, this includes: Through the formula: Calculate the initial user interface soft mask value; in, Indicates position The initial soft mask value for the user interface at that location; Indicates position The color distance between the original game interactive scene material image and the main color matrix of the user interface in a perceived uniform color space; This represents the color distance attenuation parameter; Indicates position The local color variance after normalization; Represents a numerical clipping function; Initialize the scene texture variable to the high-frequency residual of the original game interactive scene material image minus the initial user interface color components; The residual lighting variables are initialized to the low-frequency components of the original game interactive scene material images obtained by large-scale guided filtering or Gaussian filtering. An alternating optimization approach is adopted, in which two variables are fixed and one variable is optimized each time among the three variables: user interface soft mask variable, scene texture variable, and residual lighting variable. This process is repeated to update the user interface soft mask variable, scene texture variable, and residual lighting variable. The update of the user interface soft mask variable, scene texture variable, and residual lighting variable is repeated until a preset number of iterations is reached or the change in reconstruction error is less than a preset threshold, thus obtaining the user interface mask, scene texture layer, and residual lighting layer.

6. The method for enhancing game interactive scene material images based on virtual reality according to claim 1, characterized in that, The multimodal input tensor is obtained based on the user interface mask, scene texture layer, and residual lighting layer, specifically including: The user interface mask is binarized to obtain a binary user interface mask. Extract the user interface boundary map from the user interface binary mask; Perform a distance transformation on the user interface boundary map, calculate the pixel distance from each pixel to the nearest user interface boundary, and truncate and normalize the distances to obtain the user interface boundary distance field; Extract the scene texture layer luminance channel from the scene texture layer; Construct a multi-directional Gabor filter bank and calculate the local texture response of the scene texture layer's brightness channel in different directions; Compare the response intensity of the texture response map in each direction at each pixel location, and select the direction with the largest response intensity as the main direction of the local texture; Calculate the texture confidence map based on the proportion of the main directional response in all directional responses; Generate a base enhanced image based on the scene texture layer and residual lighting layer; Continuous directional encoding is performed on the main direction field of scene texture, and confidence-weighted texture direction cosine diagram and confidence-weighted texture direction sine diagram are calculated; The base augmented image, user interface boundary distance field, confidence-weighted texture direction cosine map, confidence-weighted texture direction sine map, and texture confidence map are concatenated along the channel dimension to obtain the multimodal input tensor.

7. The method for enhancing game interactive scene material images based on virtual reality according to claim 1, characterized in that, The user interface boundary distance field is downsampled to several scales to obtain user interface boundary distance fields at several scales; using the encoded feature map and the user interface boundary distance field, modulation feature maps at each scale are obtained, specifically including: The multimodal input tensor is input into the lightweight encoder-decoder network to obtain encoded feature maps at several scales; the user interface boundary distance field is downsampled to several scales to obtain user interface boundary distance fields at several scales. Downsample the confidence-weighted texture orientation cosine map, confidence-weighted texture orientation sine map, and texture confidence map to the [number]th [level]. The scale is then spliced ​​along the channel dimension to obtain the first scale. Texture structure priors at various scales; The first The user interface boundary distance field at each scale and the first The texture structure prior map at the nth scale is input to the nth scale. The prior modulator subnetwork of the i-th scale generates the i-th scale. Spatial variable gain diagram at each scale and the Spatial variable offset map at various scales; According to the Spatial variable gain plot at each scale and the first The spatially variable bias map at each scale is for the first... Modulate the encoded feature map at the i-th scale to obtain the i-th scale. Modulation feature map at each scale.

8. The method for enhancing game interactive scene material images based on virtual reality according to claim 1, characterized in that, Using the modulation feature map, a full-resolution decoded feature map is obtained; using the full-resolution decoded feature map, the dynamic filtering weights and dynamic bias terms at each pixel location are predicted. The enhanced image is obtained based on the base enhanced image, dynamic filter weights, and dynamic bias terms, specifically including: The modulation feature maps at each scale are input into the decoder for step-by-step upsampling and feature fusion to obtain a full-resolution decoded feature map. The full-resolution decoded feature map is input into the dynamic filtering prediction head to predict the dynamic filtering weights and dynamic bias terms at each pixel location. Perform boundary padding on the base augmented image to obtain the padded base augmented image; The dynamic filtering weights corresponding to each pixel position are applied to the local neighborhood of the base enhanced image, and a dynamic bias term is added to obtain the enhanced image.

9. The method for enhancing game interactive scene material images based on virtual reality according to claim 1, characterized in that, Based on the enhanced image and the corresponding processed high-quality image, a composite loss function is calculated. The trainable parameters in the dynamic pixel-by-pixel filtering enhancement network are then optimized using the composite loss function to obtain a trained dynamic pixel-by-pixel filtering enhancement network. This process specifically includes: Based on the enhanced image and the corresponding processed high-quality image, calculate the pixel fidelity loss, user interface edge consistency loss, scene texture high-frequency consistency loss, and global color stability loss. The composite loss function is obtained by weighted summing of pixel fidelity loss, user interface edge consistency loss, scene texture high frequency consistency loss, and global color stability loss. The goal is to optimize the set of trainable parameters of the network by minimizing the average composite loss function over the training sample set. The parameter optimization objective is expressed as ; in, Represents the optimal set of network parameters. This represents the set of trainable parameters of the network. This represents the total number of training samples. Indicates the first The composite loss function value corresponding to each training sample; When the preset stopping conditions are met, a well-trained dynamic pixel-by-pixel filtering enhancement network is obtained.

Images