An anti-aliasing method and system based on structural similarity guidance

Abstract

This invention discloses an anti-aliasing method and system based on structural similarity guidance, belonging to the field of computer graphics technology. In the input phase, information from the current frame and context information from the previous frame are read. The current frame information includes an RGB image, a depth map, and a motion vector image. The context information from the previous frame includes a 3-channel RGB image and a 5-channel feature map. A neural network model is embedded in a feedback loop. In the output phase, this neural network model generates a 5-channel feature map and a 3-channel anti-aliasing result image. These two results are concatenated into an 8-channel image, which is then passed as context information for the next frame for processing. This feedback loop is implemented through a recurrent neural network. The method of this invention employs an encoder-decoder architecture, achieving efficient anti-aliasing through dynamic kernel prediction and multi-scale feature fusion, and constructing feature reweighting fusion coefficients to solve ghosting and flickering issues in the TAA algorithm.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and in particular to an anti-aliasing method and system based on structural similarity guidance. Background Technology

[0002] Aliasing is a significant problem in computer graphics. During rendering, due to the limitations of low sampling rates, pixels are typically sampled one at a time, leading to undersampling. Undersampling causes image distortion, such as jagged edges, spatial noise, and flickering.

[0003] Multisampling anti-aliasing (MSAA) achieves smooth transitions at geometric edges by performing visibility tests on sub-sampled points of polygon boundaries. However, it lacks handling for aliasing generated during the shading stage and is difficult to apply in deferred rendering pipelines, thus becoming impractical. Single-frame algorithms (AXAA, FXAA) that rely on morphological filters and edge detection detect jagged features in images and perform different filtering operations for different features. While they offer excellent real-time performance, they generally suffer from excessive blurring and loss of detail. Temporal anti-aliasing (TAA) eliminates aliasing artifacts in high-frequency signals by applying subpixel jitter to camera projection and accumulating it in the temporal domain. However, it often causes temporal flicker due to false rejection of samples or motion blur due to conservative blending strategies.

[0004] In recent years, deep learning-based algorithms have become popular for handling aliasing problems. However, the technical challenge of combining deep learning with anti-aliasing algorithms remains to be solved. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the present invention aims to provide an anti-aliasing method and system based on structural similarity guidance. It adopts an encoder-decoder architecture, achieves efficient anti-aliasing through dynamic kernel prediction and multi-scale feature fusion, and constructs feature reweighting fusion coefficients to solve ghosting and flickering in the TAA algorithm.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] An anti-aliasing method based on structural similarity guidance is provided, including the following steps:

[0008] S1: Obtain the image data of the current frame and the context information of the previous frame. The image data of the current frame includes an RGB image, a depth map, and a motion vector map. The context information of the previous frame includes an RGB image and a feature map. Perform reprojection processing on the context information of the previous frame to obtain the reprojection result.

[0009] S2: Fuse the reprojection result with the RGB image of the current frame to generate the input image;

[0010] S3: Input the input image into the neural network model, which adopts an encoder-decoder architecture, including a decoupler, an SSIM predictor, a kernel predictor, and a time accumulator;

[0011] S4: The decoupler performs multi-scale feature extraction on the input image and outputs a first feature image and a second feature image. The first feature image is used for the SSIM predictor, and the second feature image is used for the kernel predictor.

[0012] S5: The SSIM predictor performs structural similarity index prediction on the first feature image and generates a single-channel SSIM value image;

[0013] S6: The kernel predictor performs convolution kernel prediction on the SSIM value image and the second feature image, generating dynamic convolution kernels for the current frame and the previous frame respectively;

[0014] S7: The time accumulator, based on the depth map and motion vector map, and combined with the two dynamic convolution kernels, performs convolution operations and fusion on the RGB image of the current frame and the reprojected RGB image to obtain the anti-aliasing result and feature map of the current frame. The feature map serves as the context information for the next frame, realizing a feedback loop.

[0015] Specifically, the decoupler achieves multi-scale feature extraction through a three-scale downsampling network. Each scale network uses three layers of residual blocks connected together, and max pooling is used to downsample the image. From the first scale to the third scale, the output channels of each layer are set to 16, 32, and 64, respectively.

[0016] Specifically, the SSIM predictor is a three-layer network. The input is a 32-channel image feature that has been downsampled by 2×2, and the output is a single-channel SSIM value image. Each pixel in this image corresponds to the SSIM value of a 2×2 pixel region in the input image.

[0017] Specifically, the kernel predictor obtains the features of the SSIM predicted image through a three-layer network, and extracts the image features from the decoupler. The two features are fused and a 25-channel convolutional kernel matrix is ​​generated through the three-layer network, where the 25 channels correspond to the convolutional kernels of the 5×5 window of the pixel.

[0018] Furthermore, the time accumulator projects the context information of the previous frame onto the coordinate space of the current frame using motion vectors, employs depth information as a constraint, and selects the point closest to the camera in a 3×3 neighborhood window as the sampling coordinates to achieve image alignment. Its reprojection function is:

[0019]

[0020] in:

[0021] (x,y) are the pixel coordinates of the current frame;

[0022] and These are the components of the motion vector in the x and y directions;

[0023] Represents a 3×3 neighborhood window centered on the coordinates pointed to by the motion vector;

[0024] The location of the pixel with the smallest depth value in the neighborhood is calculated, and the motion vector of the pixel with the smallest depth value is selected. Furthermore, a loss function for training the neural network model is also included, wherein the loss function is:

[0025] L total =αL ssim +βL rgb +γL t ;

[0026] in:

[0027] α=0.04, β=0.16, γ=0.8;

[0028] L ssim Predict L1 loss for SSIM;

[0029] L rgb To reconstruct the L1 loss;

[0030] L rgb This results in a loss of time.

[0031] Specifically, the expression for the SSIM prediction of L1 loss is as follows:

[0032]

[0033] in:

[0034] and To calculate the SSIM values ​​for the predicted image and the ground truth image;

[0035] Let i be the predicted image of the i-th frame;

[0036] The image is the actual image of the i-th frame;

[0037] N is the total number of pixels in the image;

[0038] i is the pixel index;

[0039]

[0040] M d =|Δdepth|<0.01;

[0041] M p =max(|ΔRGB|)<f(B,D);

[0042] F t , For frame t and frame t-1 of the reprojection.

[0043] The recurrent neural network of the time accumulator implements a feedback loop, which concatenates the output anti-aliasing result image and the feature map to obtain a channel image, which is then used as the context information for the next frame.

[0044] This invention also discloses an anti-aliasing system based on structural similarity guidance, used in the method, comprising:

[0045] Input processing module: used to acquire the image data of the current frame and the context information of the previous frame. The image data of the current frame includes an RGB image, a depth map and a motion vector map. The context information of the previous frame includes an RGB image and a feature map. The module performs reprojection processing on the context information of the previous frame to obtain a reprojection result. The module then fuses the reprojection result with the RGB image of the current frame to generate an input image.

[0046] Decoupler module: used to input the input image into the neural network model, which adopts an encoder-decoder architecture and includes a decoupler, an SSIM predictor, a kernel predictor, and a time accumulator; the decoupler performs multi-scale feature extraction on the input image and outputs a first feature image and a second feature image, the first feature image being used by the SSIM predictor and the second feature image being used by the kernel predictor;

[0047] SSIM prediction module: used to predict the structural similarity index of the first feature image and generate a single-channel SSIM value image;

[0048] Kernel predictor module: used to combine SSIM prediction results and separated features to generate a 5×5 convolution kernel matrix;

[0049] The temporal accumulator module is used to perform convolution operations and fuse the RGB image of the current frame and the reprojected RGB image based on the depth map and motion vector map, combined with the two dynamic convolution kernels, to obtain the anti-aliasing result and feature map of the current frame. The feature map serves as the context information for the next frame, realizing a feedback loop.

[0050] The feedback loop implemented by the recurrent neural network of the time accumulator module in the above system is to stitch the output anti-aliasing result image and the feature map to obtain the channel image, which is then used as the context information for the next frame.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0052] 1. This invention uses SSIM-guided dynamic kernel prediction to accurately perceive the severity of aliasing in each pixel region and generate adaptive convolution kernels, effectively handling aliasing problems in geometric edges and the shading stage.

[0053] 2. This invention significantly reduces flickering and ghosting phenomena by redesigning the mixed weight calculation method for sample accumulation and combining it with the IF-SSIM time loss function. Attached Figure Description

[0054] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0055] Figure 1 This is a network framework diagram of the present invention;

[0056] Figure 2 This is the training dataset for the present invention;

[0057] Figure 3 The graph shows the PSNR variation over time for each algorithm in scenario 1 of this invention.

[0058] Figure 4 The visual comparison of the present invention on five test datasets using TAA, TSR, MCNN, and SGNN;

[0059] Figure 5 This is a visual comparison of Network0 (the original neural network model) and Network1 (the neural network model with the SSIM prediction module removed) in the Scene 1 dataset. Detailed Implementation

[0060] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0061] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0062] This embodiment provides an anti-aliasing method based on structural similarity guidance, such as... Figure 1 As shown, in the input stage, the 3-channel result from the previous frame and the 5-channel feature image are reprojected and combined with the color image of the current frame to form an 11-channel input. This input is then processed by the network for SSIM prediction and kernel prediction, generating a 50-channel convolutional kernel. This kernel is used as the convolutional kernel for the 5x5 window of the current frame and the previous frame, respectively, for convolution operations. The final output is a 3-channel anti-aliasing result and a 5-channel feature map. This invention uses a temporal anti-aliasing algorithm (SGNN) guided by structural similarity index (SSIM), taking the color buffer from the rendering engine and several related gbuffers as input for each frame. Specifically, the system input includes an RGB color frame with subpixel dithering, the corresponding motion vector frame, and a depth buffer. The system output is the anti-aliased RGB image of the current frame. For each frame, the neural network model reprojects the outgoing context information from the previous frame using motion vectors, and then fuses the reprojected result with the color buffer of the current frame. Next, the neighborhood anti-aliasing module in the neural network model evaluates the neighborhood information of each pixel in the current frame to determine the SSIM value of each pixel relative to the target pixel, and incorporates it into the reconstruction network using attention blocks. The network output is two convolutional kernel matrices, which produce the final image by applying predicted convolutional kernels to the RGB images of the previous and current frames. The anti-aliasing result of the i-th frame of the TAA algorithm is shown below. The calculation is as follows:

[0063]

[0064] in:

[0065] This represents the anti-aliasing result for the i-th frame;

[0066] The original, un-de-aliased image of the i-th frame;

[0067] The motion vector from frame (i-1) to frame (i).

[0068] Warp(·) is used to calculate based on the motion vector. Project onto the current frame coordinate space;

[0069] W i W i-1 Mixed weights;

[0070] * indicates the weights are convolved with the image.

[0071] The specific steps of this method are as follows:

[0072] S1: Obtain the image data of the current frame and the context information of the previous frame. The image data of the current frame includes a 3-channel RGB image, a 1-channel depth map, and a 2-channel motion vector image. The context information of the previous frame includes a 3-channel RGB image and a 5-channel feature map. Perform reprojection processing on the context information of the previous frame to obtain the reprojection result. In the initial stage, since there is no previous frame to provide a reference, set the RGB image of the previous frame to be the same as that of the current frame, and initialize the feature map to 0.

[0073] S2: Fuse the reprojection result with the RGB image of the current frame to generate an 11-channel input image;

[0074] S3: Input the 11-channel input image into the neural network model. The neural network model adopts an encoder-decoder architecture, including a decoupler, an SSIM predictor, a kernel predictor, and a time accumulator.

[0075] S4: The decoupler performs multi-scale feature extraction on the 11-channel input image and outputs a first feature image and a second feature image. The first feature image is used for the SSIM predictor, and the second feature image is used for the kernel predictor.

[0076] S5: The SSIM predictor performs structural similarity index prediction on the first feature image and generates a single-channel SSIM value image;

[0077] S6: The kernel predictor performs convolution kernel prediction on the SSIM value image and the second feature image, generating dynamic convolution kernels with 25 channels for the current frame and the previous frame respectively. Each channel of the dynamic convolution kernel matrix corresponds to a 5×5 window convolution kernel.

[0078] S7: The time accumulator uses the depth map and motion vector map, combined with two dynamic convolution kernels, to perform convolution operations and fusion on the RGB image of the current frame and the reprojected RGB image to obtain the anti-aliasing result of the current frame and a 5-channel feature map. The 5-channel feature map serves as the context information for the next frame.

[0079] Specifically:

[0080] Decoupler: The decoupler aims to extract critical features from an image and separate potential SSIM information from the input image. It consists of a series of convolutional layers and activation functions used for feature extraction and separation. The decoupler achieves multi-scale feature extraction through a three-scale downsampling network. Each scale's network uses three connected Bottleneck blocks, and Maxpooling is used to downsample the image. From the first scale to the third scale, the output channels of each layer are set to 4, 8, and 16, respectively. The decoupler is shared by the image reconstruction and SSIM prediction branches. The image reconstruction branch uses 8-channel, downsampled image features, while the SSIM prediction branch uses 16-channel image features.

[0081] SSIM Predictor: The SSIM predictor is a three-layer network used to predict the Structural Similarity Index (SSIM) of an image. It uses 8-channel image features downsampled from the original image (2x2 size) as input and outputs a single-channel SSIM value image. The purpose of the SSIM predictor is to accurately predict the SSIM value of each part of the image, thereby guiding the reconstruction network to better predict kernels. A 2x2 downsampled SSIM image means that the value of each pixel corresponds to the SSIM value of 2x2 images. Using too large a feature image can lead to excessive penalties, while using too small an image can result in blurred SSIM features. Experiments have shown that a 2x2 downsampled SSIM image size is a good choice, effectively reducing computational complexity while ensuring the accuracy of the prediction results.

[0082] Kernel Predictor: The kernel predictor is used to predict the convolutional kernels for image anti-aliasing. The kernel predictor obtains the features of the SSIM-predicted image through a three-layer network, and at the same time extracts the image features from the decoupler. The two features are fused and then processed by the three-layer network to generate a 25-channel convolutional kernel matrix, where the 25 channels correspond to the convolutional kernels of a 5×5 window of pixels.

[0083] Temporal accumulator: The temporal accumulator is implemented using motion vectors and a depth buffer, which can be directly obtained from the rendering engine. For each frame F... i-1 By applying motion vector M i-1 By combining the context information of the previous frame and the current frame, the distortion function W aligns the RGB image of the current frame with the RGB image of the previous frame, creating a new 11-channel frame F. w This alignment ensures that the neural network model can utilize information from past frames to enhance the temporal coherence of the current frame. Directly using motion vectors to sample the previous frame may result in information distortion due to occlusion; therefore, depth information is used as a constraint, selecting the point closest to the camera within a 3x3 neighborhood window of the sampling point as the sampling coordinates. The formula for the reprojection function warp is:

[0084]

[0085] in:

[0086] (x,y) are the pixel coordinates of the current frame;

[0087] and These are the components of the motion vector in the x and y directions;

[0088] Represents a 3×3 neighborhood window centered on the coordinates pointed to by the motion vector;

[0089] The argmin operation selects the pixel with the smallest depth value in the neighborhood.

[0090] Additionally, it includes the loss function, which aims to enhance the anti-aliasing capability of the neural network model, while improving the robustness of the results in the time domain and reducing distortion problems such as ghosting and flickering. Therefore, the loss function consists of these parts: spatial loss L s and time loss L t .

[0091] Space loss L s Loss L from anti-aliasing results rgb And the predicted SSIM loss L used to guide the anti-aliasing network ssim To prevent the trained image results from being too smooth and to preserve the details of both the SSIM and RGB images, an L1 loss function was sampled to avoid a few extreme pixels dominating the optimization process. Specifically, the L1 loss function is used to evaluate the spatial loss. rgb and L ssim The loss function can be written as:

[0092]

[0093] in:

[0094] and To calculate the SSIM values ​​for the predicted image and the ground truth image;

[0095] Let i be the predicted image of the i-th frame;

[0096] The image is the actual image of the i-th frame;

[0097] N is the total number of pixels in the image;

[0098] i is the pixel index.

[0099] Time loss L tThe aim is to enhance the model's temporal consistency, improve the robustness of results in the time domain, and reduce distortion problems such as ghosting and flickering. The specific loss function L... t as follows:

[0100]

[0101] M d =|Δdepth|<0.01;

[0102] M p =max(|ΔRGB|)<f(B,D);

[0103] F t , For frame t and frame t-1 of the reprojection.

[0104] The formula for the loss function of the combination is:

[0105] L total =αL ssim +βL rgb +γL t

[0106] in:

[0107] α=0.04, β=0.16, γ=0.8;

[0108] L ssim Predict L1 loss for SSIM;

[0109] L rgb To reconstruct the L1 loss;

[0110] L rgb This results in a loss of time.

[0111] The hyperparameters α, β, and γ were tested and found to be 0.04, 0.16, and 0.8, respectively. These parameters take into account that the value of IFSSIM is much smaller than the value of L1 loss, allowing the model to be trained well.

[0112] An anti-aliasing system based on structural similarity guidance, comprising:

[0113] Input processing module: used to acquire relevant image information of the current frame and the previous frame, and form an 11-channel input image after reprojection;

[0114] Decoupler module: used to perform multi-scale feature extraction on 11-channel input images, separating features for SSIM prediction and kernel prediction;

[0115] SSIM prediction module: used to process the separated features and output a single-channel SSIM value image;

[0116] Kernel predictor module: used to combine SSIM prediction results and separated features to generate a 25-channel 5×5 convolution kernel matrix;

[0117] Temporal accumulator module: It is used to align and fuse the current frame with the previous frame based on motion vector and depth information, perform convolution operation using the generated convolution kernel, and output 3-channel anti-aliasing result and 5-channel feature map.

[0118] A feedback loop is implemented through a recurrent neural network, which concatenates the output 3-channel anti-aliasing result image and 5-channel feature map into an 8-channel image, which is then used as context information for the next frame.

[0119] Experimental results and analysis:

[0120] 1. Training dataset

[0121] like Figure 2 As shown, the dataset creation process in this embodiment uses game scenes obtained from the Unity Asset Store and UE's Asset Store, recording the camera trajectories within the scenes. Each scene was rendered twice: once to generate a 1920*1080 resolution image sequence, including low dynamic range RGB images, depth maps, and motion vector maps; and once to generate a 3840*2160 resolution image sequence. These images were adjusted to 1920*1080 resolution using a Blackmann-Harris filter to create anti-aliasing reference images as the baseline for the dataset. SSIM images were calculated for the corresponding anti-aliased and un-aliased images, and downsampled to 960*540 resolution using max pooling. Max pooling was used instead of other downsampling algorithms to highlight the worst-case scenario for each 2*2 pixel window. For the test dataset, the scenes were additionally rendered using UE5's TAA and TSR anti-aliasing algorithms to generate TAA and TSR anti-aliased images for subsequent comparative experiments.

[0122] The renderer used a Halton2,3 sequence loop with eight different subpixel jitter offsets, so the training set used eight frames per batch to ensure all jitter was accounted for during training. In total, 800 data points were generated from three scenes and divided into training and validation sets in an 80:20 ratio. An additional 300 data points were generated from six other scenes as the test dataset.

[0123] 2. Experimental Data and Evaluation Criteria

[0124] For quantitative analysis of the experimental results, this embodiment uses common image analysis metrics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as evaluation criteria. The selected benchmark image is a low-resolution anti-aliased image obtained by downsampling a high-resolution image using Blackmann-Harris filtering.

[0125] To further analyze the stability of the algorithm over time, the coefficient of variation (CV) is introduced as a supplementary metric:

[0126]

[0127] Where, σ PSNR The standard PSNR value within the scene, μ PENR The value is the mean. A larger CV value indicates a worse time stability of the algorithm in that scenario (e.g., flickering, inter-frame abrupt changes).

[0128] 3. Experimental setup

[0129] This experiment was conducted using the PyTorch deep learning framework. All algorithm models were trained on a single NVIDIA GeForce RTX 3090 24GB graphics card, with a batch size of 4. The Adam algorithm was used to optimize the loss function, with an initial learning rate of 1e-4. Training was terminated after 200 model iterations. A complete training cycle took approximately 15 hours.

[0130] 4. Analysis of Experimental Results

[0131] To verify the anti-aliasing capability of the model proposed in this embodiment, the algorithm of this embodiment was compared with the learning-based anti-aliasing algorithm MCNN and the traditional non-learning algorithms TAA and TSR. For a fair comparison, MCNN used the officially released code and was retrained using the same training dataset as this embodiment, while TAA and TSR algorithms used the official anti-aliasing algorithms implemented in UE5. The mean PSNR and mean SSIM of various algorithms were tested in different scenarios, and the image performance of the algorithm of this embodiment was compared with other algorithms.

[0132] (1) Quantitative comparative analysis, as shown in Tables 1, 2 and 3. Table 1 is a comparison of PSNR performance of different algorithms in different scenarios (unit: dB); Table 2 is a comparison of SSIM performance of different algorithms in different scenarios; Table 3 is a comparison of performance of different algorithms in different scenarios (unit: %).

[0133] Table 1

[0134] algorithm Scene 1 Scene 2 Scene 3 Scene 4 Scene 5 average SGNN 38.923 34.347 32.967 31.138 32.116 33.898 MCNN 31.038 33.650 33.392 30.674 31.679 32.087 TAA 31.167 31.745 28.841 27.030 30.380 29.833 TSR 31.299 31.722 29.264 27.390 30.297 29.994

[0135] Table 2

[0136] algorithm Scene 1 Scene 2 Scene 3 Scene 4 Scene 5 average SGNN 0.983 0.958 0.954 0.934 0.983 0.962 MCNN 0.975 0.950 0.949 0.923 0.968 0.953 TAA 0.934 0.919 0.910 0.844 0.965 0.914 TSR 0.937 0.919 0.914 0.861 0.960 0.918

[0137] Table 3

[0138] algorithm ART example FirstFloor village Warehouse average SGNN 0.268 0.923 0.789 1.411 0.149 0.709 MCNN 0.159 1.759 4.405 0.890 0.229 1.782 TAA 2.140 1.083 1.308 9.409 3.156 3.419 TSR 0.637 1.083 1.421 2.868 1.145 1.431

[0139] Tables 1 and 2 show the PSNR and SSIM test results for six scenarios, respectively. As can be seen from Table 1, the algorithm in this embodiment achieves optimal or near-optimal performance in multiple scenarios, improving the PSNR by 2.113 dB compared to the second-ranked MCNN, and the SSIM by 0.011 dB compared to the second-ranked MCNN. These results demonstrate that the algorithm in this embodiment performs better than the compared algorithms.

[0140] Figure 3 The PSNR fluctuations of all algorithms in a given scenario were compared across frame sequences. SGNN showed the most stable fluctuations, while TAA gradually increased its PSNR value over time. Table 3 compares the average coefficient of variation (COV) of all algorithms across all scenarios. The magnitude of the values ​​reflects the temporal stability of the image sequence. The algorithm in this embodiment exhibits lower COV values ​​than other algorithms in all scenarios, demonstrating significantly superior temporal stability compared to other algorithms.

[0141] (2) Image comparison

[0142] To better verify the effectiveness of the algorithm in anti-aliasing under various scenarios, Figure 4 The anti-aliasing details of different algorithms were compared in multiple scenarios. In the first image (text) and the fourth image (canned goods) (clear surface), the TAA and TSR algorithms are significantly blurrier than other algorithms, failing to restore the original details. The MCNN algorithm exhibits artifacts at some edges; in the second image, the grapes show obvious gray areas, which are artifact residues from the previous frame. The algorithm in this embodiment has better clarity than other algorithms while minimizing artifact interference, and its display effect is closest to the reference image from an image perspective.

[0143] Through comparative experiments, the anti-aliasing capability of this embodiment is improved compared with other common algorithms, both in terms of subjective indicators and objective visual effects.

[0144] 5. Ablation test

[0145] Ablation experiments are commonly used for effectiveness analysis of complex neural networks. For each module that makes up the network, the impact of removing some modules on the network performance is verified. In this embodiment, to verify the impact of the SSIM prediction module on the anti-aliasing network, the following ablation experiment was designed. Network0 represents the original network model, and Network1 represents the network model with the SSIM prediction module removed. The experimental results are shown in Table 4.

[0146] Table 4 compares the performance of the two networks on six test datasets. After removing the SSIM prediction module, the mean PSNR and mean SSIM of the model both decreased, with the mean PSNR decreasing by 1.853 dB and the mean SSIM decreasing by 0.013 dB. Figure 5 The performance of the two networks in Scenario 1 is shown. From the local images, it can be seen that after removing the SSIM prediction module, the model's performance in handling jagged anti-aliasing for high-frequency diagonal lines decreased. For images with large color changes, Network1's colors are darker and do not match the original image, which shows that the model's sensitivity to changes has decreased.

[0147] Table 5 shows the average coefficient of variation (COP) of the two networks in various scenarios. Network0 has a lower COP in all scenarios. Overall, compared to the model with the SSIM prediction module removed, the original SGNN model has a 0.218% lower COP, indicating that the model's temporal stability decreases after removing the SSIM prediction module. The ablation experiments demonstrate the effectiveness of this method. Based on the previous experimental analysis, the SSIM prediction module can effectively improve the model's anti-aliasing performance while maintaining higher temporal stability. Table 4 shows the comparison of PSNR (dB) and SSIM performance of different algorithms in different scenarios.

[0148] Table 4

[0149]

[0150] Table 5 shows the average coefficient of variation (in %) for Network0 and Network1 under different scenarios.

[0151] Table 5

[0152] algorithm Scene 1 Scene 2 Scene 3 Scene 4 Scene 5 average Network0 0.268 1.077 3.814 2.603 0.183 1.589 Network1 0.544 1.209 6.366 2.337 0.403 2.172

[0153] Image rendering suffers from artifacts, flickering, and other aliasing issues due to sampling rate limitations. This embodiment employs a structural similarity-guided anti-aliasing method. By constructing multi-scale encoding and decoding modules and using predicted structural similarity as guidance, it effectively corrects image aliasing problems. This embodiment uses quantization to analyze image aliasing in both the spatial and temporal domains and demonstrates the model's performance in different scenarios through experiments. By analyzing and comparing the performance of various anti-aliasing algorithms, including non-learning and learning-based algorithms, in these scenarios, experimental results show that the model in this embodiment has significant advantages in correcting image aliasing. It achieves the best anti-aliasing effect while preserving the original sharpness to the greatest extent possible, and it effectively handles flickering, ghosting, and other aliasing problems, exhibiting good temporal robustness.

[0154] Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-mentioned technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-mentioned technical features or their equivalent features without departing from the inventive concept. For example, technical solutions formed by substituting the above-mentioned features with technical features disclosed in this application (but not limited to) that have similar functions.

Claims

1. An anti-aliasing method based on structural similarity guidance, characterized in that, Includes the following steps: S1: Obtain the image data of the current frame and the context information of the previous frame. The image data of the current frame includes an RGB image, a depth map, and a motion vector map. The context information of the previous frame includes an RGB image and a feature map. Perform reprojection processing on the context information of the previous frame to obtain the reprojection result. S2: Fuse the reprojection result with the RGB image of the current frame to generate the input image; S3: Input the input image into the neural network model, which adopts an encoder-decoder architecture, including a decoupler, an SSIM predictor, a kernel predictor, and a time accumulator; The time accumulator projects the context information of the previous frame onto the coordinate space of the current frame using motion vectors. It uses depth information as a constraint and selects the point closest to the camera within a 3×3 neighborhood window as the sampling coordinates to achieve image alignment. Its reprojection function is: ； in: These are the pixel coordinates of the current frame; and These are the components of the motion vector in the x and y directions; Represents a 3×3 neighborhood window centered on the coordinates pointed to by the motion vector; Calculate the position of the pixel with the smallest depth value in the neighborhood, and then select the motion vector at the position of the smallest pixel. S4: The decoupler performs multi-scale feature extraction on the input image and outputs a first feature image and a second feature image. The first feature image is used for the SSIM predictor, and the second feature image is used for the kernel predictor. S5: The SSIM predictor performs structural similarity index prediction on the first feature image and generates a single-channel SSIM value image; S6: The kernel predictor performs convolution kernel prediction on the SSIM value image and the second feature image, generating dynamic convolution kernels for the current frame and the previous frame respectively; S7: The time accumulator, based on the depth map and motion vector map, and combined with the two dynamic convolution kernels, performs convolution operations and fusion on the RGB image of the current frame and the reprojected RGB image to obtain the anti-aliasing result and feature map of the current frame. The feature map serves as the context information for the next frame, realizing a feedback loop.

2. The method according to claim 1, characterized in that, The decoupler achieves multi-scale feature extraction through a three-scale downsampling network. Each scale network uses three layers of residual blocks connected together and uses max pooling to downsample the image. From the first scale to the third scale, the output channels of each layer are set to 16, 32, and 64, respectively.

3. The method according to claim 1, characterized in that, The SSIM predictor is a three-layer network. The input is a 32-channel image feature that has been downsampled by 2×2, and the output is a single-channel SSIM value image. Each pixel in this image corresponds to the SSIM value of a 2×2 pixel region in the input image.

4. The method according to claim 1, characterized in that, The kernel predictor obtains the features of the SSIM predicted image through a three-layer network, and extracts the image features from the decoupler. The two features are fused and then processed by the three-layer network to generate a 25-channel convolution kernel matrix, where the 25 channels correspond to the convolution kernels of a 5×5 window of pixels.

5. The method according to claim 1, characterized in that, It also includes a loss function used to train the neural network model, the loss function being: ； in: α=0.04, β=0.16, γ=0.8; Predict L1 loss for SSIM; To reconstruct the L1 loss; This results in a loss of time.

6. The method according to claim 5, characterized in that, The expression for SSIM predicting L1 loss is as follows: ； ； in: and To calculate the SSIM values ​​for the predicted image and the ground truth image; Let i be the predicted image of the i-th frame; The image is the actual image of the i-th frame; N is the total number of pixels in the image; i is the pixel index; ； ＜0.01； ＜ ； For frame t and frame t-1 of the reprojection.

7. The method according to claim 1, characterized in that, The recurrent neural network of the time accumulator implements a feedback loop, which concatenates the output anti-aliasing result image and the feature map to obtain a channel image, which is then used as the context information for the next frame.

8. An anti-aliasing system based on structural similarity guidance, used to implement the method described in any one of claims 1-7, characterized in that, include: Input processing module: used to acquire the image data of the current frame and the context information of the previous frame. The image data of the current frame includes an RGB image, a depth map and a motion vector map. The context information of the previous frame includes an RGB image and a feature map. The module also performs reprojection processing on the context information of the previous frame to obtain the reprojection result. The reprojection result is fused with the RGB image of the current frame to generate the input image; Decoupler module: used to input the input image into the neural network model, which adopts an encoder-decoder architecture and includes a decoupler, an SSIM predictor, a kernel predictor, and a time accumulator; the decoupler performs multi-scale feature extraction on the input image and outputs a first feature image and a second feature image, the first feature image being used by the SSIM predictor and the second feature image being used by the kernel predictor; The time accumulator projects the context information of the previous frame onto the coordinate space of the current frame using motion vectors. It uses depth information as a constraint and selects the point closest to the camera within a 3×3 neighborhood window as the sampling coordinates to achieve image alignment. Its reprojection function is: ； in: These are the pixel coordinates of the current frame; and These are the components of the motion vector in the x and y directions; Represents a 3×3 neighborhood window centered on the coordinates pointed to by the motion vector; Calculate the position of the pixel with the smallest depth value in the neighborhood, and then select the motion vector at the position of the smallest pixel. SSIM prediction module: used to predict the structural similarity index of the first feature image and generate a single-channel SSIM value image; Kernel predictor module: used to combine SSIM prediction results and separated features to generate a 5×5 convolution kernel matrix; The temporal accumulator module is used to perform convolution operations and fuse the RGB image of the current frame and the reprojected RGB image based on the depth map and motion vector map, combined with the two dynamic convolution kernels, to obtain the anti-aliasing result and feature map of the current frame. The feature map serves as the context information for the next frame, realizing a feedback loop.

9. The system according to claim 8, characterized in that, The feedback loop implemented by the recurrent neural network of the time accumulator module in the above system is to stitch the output anti-aliasing result image and the feature map to obtain the channel image, which is then used as the context information for the next frame.

Images