A GPU-based super-resolution and high dynamic range processing method and storage medium

Abstract

The application relates to the technical field of image processing, and discloses a GPU-based super-resolution and high dynamic range processing method and a storage medium, the method comprising the following steps: applying ACES input color conversion to materials; analyzing the geometric edges of the materials by using a GPU; calculating the weight function of a filter by writing a custom GPU calculation shader and according to a weight function calculation formula; completing an image spatial domain filter reconstruction process based on an image upsampling calculation formula; performing bidirectional projection before and after the time domain, and accumulating and calculating the mean and variance adjustment fusion coefficients of the colors and edges of video frames and realizing time domain filtering according to a time sequence fusion formula; and performing color gamut conversion and tone mapping on the result of the time domain filtering, wherein the method uses a GPU to complete all calculation processes such as motion estimation, edge extraction, pixel upsampling, color space conversion, brightness statistics and tone mapping, the processing efficiency of the algorithm is high, the CPU load is reduced, and the real-time processing requirement is met.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a GPU-based super-resolution and high dynamic range processing method and storage medium. Background Technology

[0002] Super-resolution refers to the reconstruction / restoration of a low-resolution image into a corresponding high-resolution image using specific algorithms. Specifically, super-resolution technology utilizes knowledge from computer vision, image processing, and computer graphics to reconstruct a high-resolution image from a given low-resolution image using specific algorithms and processing procedures. It aims to solve problems such as blurry images, low image quality, and indistinct regions of interest. Super-resolution reconstruction technology has wide applications in image compression, medical imaging, remote sensing and mapping, video perception, security monitoring, and many other fields.

[0003] Based on time and effectiveness, super-resolution reconstruction algorithms can be broadly categorized into traditional algorithms and deep learning algorithms. Traditional super-resolution reconstruction algorithms primarily rely on digital image processing techniques, using spatial linear filtering interpolation or quality degradation models for reconstruction, often resulting in blurry or jagged images. Convolutional neural networks, on the other hand, can adaptively learn deep features from training data, leading to significantly improved super-resolution reconstruction performance compared to traditional methods. However, deep learning methods typically use fixed downsampling methods to obtain low-resolution images and then learn network model parameters by optimizing pixel-by-pixel loss, which can easily cause over-blurring of the reconstructed image. This makes them difficult to adapt to complex real-world scenes. Furthermore, deep learning methods depend on large amounts of data, and the complexity of convolutional network models results in higher computational costs compared to traditional methods, making them unable to meet real-time requirements. The higher the output resolution, the greater the computational overhead.

[0004] High dynamic range (HDR) is a technique for acquiring, processing, rendering, and displaying images. It realistically presents the colors and ranges of the physical world under natural lighting by increasing the detail of shadows and highlights in a scene. In computer graphics and film editing, it can achieve a wider brightness range, a broader color gamut, a greater bit depth, different gamma corrections, and photoelectric conversion functions than ordinary images, presenting the human eye with a richer visual experience, clearer details, and stronger contrast.

[0005] When editing, processing, displaying, or compositing, High Dynamic Range (HDR) images often require tone mapping to ensure compatibility with Standard Dynamic Range (SDR) images and various display output devices. Tone mapping is responsible for transforming scene brightness to a processable, displayable, and compositing range while preserving image details and colors. However, under conditions of significant contrast degradation, most existing tone mapping algorithms perform color mapping based on brightness or contrast, without considering color space and color perception preservation. This often results in overly sharp and distorted images that fail to retain important details and color information of the original scene.

[0006] In the current era of the Internet of Everything, there is a wide variety of connected devices, but hardware resources and computing power are relatively limited. How to make full use of the computing power of edge devices to support real-time processing, rendering and presentation of super-resolution and high dynamic range visual effects is also a major challenge in technical engineering applications. Summary of the Invention

[0007] The purpose of this application is to overcome the shortcomings of the prior art and provide a GPU-based super-resolution and high dynamic range processing method and storage medium.

[0008] Firstly, a GPU-based super-resolution and high dynamic range processing method is provided, including:

[0009] Apply ACES input color transformation to the source material;

[0010] Utilize GPUs to analyze the geometric edges of the material;

[0011] By writing a custom GPU compute shader, the weight function of the filter is calculated according to the weight function calculation formula;

[0012] The image spatial domain filtering and reconstruction process is completed based on the image upsampling calculation formula;

[0013] Perform bidirectional projection in the temporal domain and accumulate the mean and variance of the color and edges of the video frames;

[0014] Adjust the fusion coefficients and implement temporal filtering according to the temporal fusion formula;

[0015] Perform gamut conversion and tone mapping on the results of temporal filtering, and output the reconstruction results.

[0016] Furthermore, the geometric edges of the analyzed material include:

[0017] Calculate the brightness of the current image color;

[0018] Sampling of pixels surrounding a 2x2 sub-block at the center of a 4x4 region;

[0019] Based on the brightness change of the central pixel in the region, calculate the brightness gradient intensity of the four local edge directions at 0°, 45°, 90° and 135°.

[0020] Extract the geometric contours and edge information of the material.

[0021] Furthermore, the formula for calculating the weighting function is as follows:

[0022] x r = y*sinθ+x*cosθ

[0023] y r = y*cosθ - x*sinθ

[0024] K(x,y)=k1Lut(Dist(x,y))+k2Lum(x,y)+k3Feat(x r y r (1)

[0025] Where K(x,y) is the filter weight function, (x r ,y r Dist(x,y) is the coordinates rotated by the local edge direction θ, Lut(d) is the sampling center distance, Lut(d) is the Lanczos filter coefficient lookup table function, Lum(x,y) is the scaling center brightness, and Feat(x) is the sampling center distance. r ,y r ) is the rotated coordinate (x) r ,y r The local gradient intensity of ).

[0026] Furthermore, the image upsampling calculation formula is as follows:

[0027]

[0028] Where S(x,y) is the input low-resolution image, H(x,y) is the upsampled output image, and k(x,y) is the filtering weight function.

[0029] Furthermore, the bidirectional projection in the time domain includes the following steps:

[0030] The material was jittered and rendered using Halton low-difference sequences and by adjusting the projection matrix parameters according to formula (3);

[0031] Based on the projection matrix and jitter sequence parameters of adjacent frames, find the corresponding samples of the previous and next frames by bidirectional inverse projection forward and backward;

[0032] Formula (3) is as follows:

[0033]

[0034] Here, Halton(·,N) represents a Halton low-discrepancy sequence of length N, which is a pseudo-random function uniformly distributed in a high-dimensional space.

[0035] Furthermore, the bidirectional projection in the time domain also includes:

[0036] Based on the edge direction θ of the sample, detection and calculation are performed in the 7x7 neighborhood of the sample center. If the color brightness change and edge direction change of the corresponding sample in the previous and next frames of the video are within the pre-set interval, the sample is considered to be effectively aligned. If the color brightness change and edge direction change of the corresponding sample in the previous and next frames of the video are not within the pre-set interval, the sample is considered to be invalidally aligned.

[0037] Furthermore, the cumulative calculation of the mean and variance of the color and edges of the video frames includes the following steps:

[0038] When the number of valid samples in a 7x7 neighborhood exceeds a pre-set threshold, the mean value of sample color brightness Mean(x,y) and the range of variation Range(x,y) under N jitter renderings are calculated.

[0039] Furthermore, the time-series fusion formula is as follows:

[0040]

[0041] in, H represents the output image of the time-series filtering at time t. t (x,y) represents the spatial filtering result image at time t, and α represents the fusion coefficient.

[0042] Furthermore, the color gamut conversion and tone mapping of the time-domain filtering result includes the following steps:

[0043] Based on the input materials, display devices, and scene requirements, and guided by the principle of preserving brightness perception in the human visual system, the results of temporal filtering are subjected to color gamut conversion and tone mapping. The minimum and peak brightness of the scene colors obtained through the reconstruction process are reused. Using the minimum and peak brightness of the displayed output as references, appropriate intermediate grayscale brightness is selected. Then, two piecewise curve coefficients—minimum brightness - intermediate grayscale and intermediate grayscale - peak brightness—are fitted according to the second-order Bézier curve formula. The specific formula is as follows:

[0044] B(t)=(1-t 2 )P0+2(1-t)tP1+t 2 P2 0≤t≤1 (5)

[0045] Where t is the distance parameter of the Bézier curve, and P0, P1, and P2 represent the control points of the second-order Bézier curve. For the lowest brightness-intermediate grayscale curve, P0 and P2 correspond to the lowest brightness and intermediate grayscale, respectively; for the intermediate grayscale-peak brightness curve, P0 and P2 correspond to the intermediate grayscale and peak brightness, respectively.

[0046] In a second aspect, a computer-readable storage medium is provided that stores program code for execution by a device, the program code including steps for performing a method as described in any implementation of the first aspect.

[0047] This application offers the following advantages: It can dynamically configure and adjust the scaling resolution based on input materials, output display devices, and scene requirements, maintaining consistent algorithm processing and rendering performance. Simultaneously, it utilizes brightness perception preservation to achieve a high dynamic range tone mapping process, resulting in better-looking and more physically realistic visual effects and impact. Starting from the human eye's perception system, it comprehensively considers factors such as image resolution, brightness range, color gamut, and color space, combining super-resolution with high dynamic range. The tone mapping process is dynamically adjusted based on video / image geometry, display output devices, and color perception, resulting in a richer color, clearer details, and stronger contrast visual experience. Furthermore, it fully leverages the efficient and powerful parallel pipeline processing capabilities of modern graphics processors, utilizing the GPU to complete all calculation processes, including motion estimation, edge extraction, pixel upsampling, color space conversion, brightness statistics, and tone mapping. The algorithm boasts high processing efficiency, reduces CPU load, and meets real-time processing requirements. Attached Figure Description

[0048] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of a GPU-based super-resolution and high dynamic range processing method according to Embodiment 1 of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1

[0053] This application's embodiment 1 discloses a GPU-based super-resolution and high dynamic range processing method, comprising: applying ACES input color transformation to the source material; analyzing the geometric edges of the source material using a GPU; calculating the weight function of the filter by writing a custom GPU computing shader and using a weight function calculation formula; completing the image spatial domain filtering and reconstruction process based on the image upsampling calculation formula; performing bidirectional projection in the temporal domain and accumulating the mean and variance of the color and edges of the video frames; adjusting the fusion coefficients and implementing temporal filtering according to the temporal fusion formula; performing color gamut conversion and tone mapping on the temporal filtering results, and outputting the reconstruction results. This method can dynamically configure and adjust the scaling resolution according to the input source material, output display device, and scene requirements, while maintaining consistent algorithm processing and rendering performance. By using luminance perception preservation to achieve high dynamic range tone mapping, a better-looking and more physically realistic visual effect and impact can be obtained. Starting from the human eye perception system, it comprehensively considers factors such as image resolution, luminance range, color gamut, and color space, combining super-resolution with high dynamic range. The tone mapping process is dynamically adjusted according to the video / image geometry, display output device, and color perception, which can present a visual experience with richer colors, clearer details, and stronger contrast. It can also make full use of the efficient and powerful parallel pipeline processing capabilities of modern graphics processors, using the GPU to complete all calculation processes such as motion estimation, edge extraction, pixel upsampling, color space conversion, luminance statistics, and tone mapping. The algorithm has high processing efficiency, reduces CPU load, and meets real-time processing requirements.

[0054] Because deep learning-based super-resolution algorithms suffer from problems such as dataset dependence, poor computational performance, and difficulty in adjusting output resolution, GPU-based super-resolution and high dynamic range algorithms abandon convolutional neural networks to avoid their inherent defects and adopt a method that combines temporal and spatial domains.

[0055] According to the Nyquist sampling theorem, super-resolution reconstruction can be viewed as an image / video upsampling process. For signals with finite bandwidth, although the Sinc kernel function is theoretically the optimal reconstruction filter, its theoretical assumptions do not perfectly match the real physical world. In practical applications, Lanczos resampling yields better reconstruction results. The definition of the Lanczos kernel function is:

[0056]

[0057] Where x is the position from the center of the Lanczos kernel function window, and a represents the size of the Lanczos kernel function window. a is generally a positive integer value, such as 1, 2 or 3.

[0058] Assuming the input low-resolution image is S(x,y) and the upsampled output image is H(x,y), super-resolution reconstruction is performed based on spatial domain filtering and using the image upsampling calculation formula:

[0059]

[0060] Where S(x,y) is the input low-resolution image, H(x,y) is the upsampled output image, and K(x,y) is the filter weight function, which is determined by parameters such as the filter kernel function, spatial distance, color brightness, and region edges.

[0061] Temporal filtering, by utilizing the temporal information of consecutive frames, can leverage motion vectors, temporal accumulation, regional mean, and luminance variance for video and animation. It optimizes super-resolution reconstruction using the Exponential Moving Average method, with the temporal fusion formula as follows:

[0062]

[0063] in, H represents the output image of the time-series filtering at time t. t (x,y) represents the spatial filtering result image at time t, and α represents the fusion coefficient.

[0064] Combining temporal and spatial GPU super-resolution reconstruction algorithms, the input video / image is first mapped to a unified standard color space. The GPU then analyzes the image content, calculating pixel brightness, regional gradients, and temporal changes to obtain the image content outlines and geometric edges. This analysis reveals spatial edge information, motion trajectories, and temporal changes. Next, a computational shader is written to dynamically adjust the filtering coefficients and pixel fusion intensity based on the image region edge contours and historical cumulative motion vectors. This achieves adaptive geometric edges and dynamic sharpening during scaling. Finally, based on the input material, display device, and scene requirements, and guided by the preservation of brightness perception in the human visual system, the video / image tone mapping curve parameters are calculated to achieve high dynamic range color space conversion and tone mapping. This results in a visually rich, detailed, and high-contrast experience on the device.

[0065] Specifically, Figure 1A flowchart of the GPU-based super-resolution and high dynamic range processing method in Embodiment 1 of the application is shown, including:

[0066] S101. Apply ACES input color transformation to the material;

[0067] Specifically, the material refers to images or videos. Referring to the ACES color standard, the ACES input transformation is applied to the image / video material to map the input colors to a unified standard color space ACEScg. That is, the image / video content generated by different shooting / capture devices is mapped to a unified color standard, removing color characteristics or high dynamic range transfer functions, etc.

[0068] S102. Utilize GPU to analyze the geometric edges of the material;

[0069] In a further embodiment, the geometric edges of the analyzed material include:

[0070] Calculate the brightness of the current image color;

[0071] Sampling of pixels surrounding a 2x2 sub-block at the center of a 4x4 region;

[0072] Based on the brightness change of the central pixel in the region, calculate the brightness gradient intensity of the four local edge directions at 0°, 45°, 90° and 135°.

[0073] Extract the geometric contours and edge information of the material.

[0074] Specifically, leveraging the efficient and powerful parallel processing capabilities of modern GPUs, a custom computation shader is written to implement the process of spatial domain analysis of video / image content. Pixels surrounding a 2x2 sub-block are sampled at the center of a 4x4 region. Based on the brightness changes of the pixels at the center of the region, the brightness gradient intensities of four local edge directions (0°, 45°, 90°, and 135°) are calculated.

[0075] S103. Calculate the filter weight function by writing a custom GPU computing shader and using the weight function calculation formula.

[0076] Specifically, to reduce the complexity of calculations such as the Lanczos kernel function, engineering approximation and table lookup methods were used to calculate the Lanczos filter coefficients. The various filter coefficients were adjusted based on the scaling center color brightness Lum, pixel distance Dist, spatial local edge direction θ, and gradient intensity Feat. A custom GPU computing shader was written to calculate the filter weight function according to the weight function calculation formula, thus obtaining the image filter coefficients. The weight function calculation formula is as follows:

[0077] x r = y*sinθ+x*cosθ

[0078] y r = y*cosθ - x*sinθ

[0079] K(x,y)=k1Lut(Dist(x,y))+k2Lum(x,y)+k3Feat(x r ,y r )

[0080] Where K(x,y) is the filter weight function, (x r ,y r Dist(x,y) is the coordinates rotated by the local edge direction θ, Lut(d) is the sampling center distance, Lut(d) is the Lanczos filter coefficient lookup table function, Lum(x,y) is the scaling center brightness, and Feat(x) is the sampling center distance. r ,y r ) is the rotated coordinate (x) r ,y r The local gradient intensity of ).

[0081] S104. Complete the image spatial domain filtering and reconstruction process based on the image upsampling calculation formula;

[0082] Specifically, the image upsampling calculation formula is as follows:

[0083]

[0084] Where S(x,y) is the input low-resolution image, H(x,y) is the upsampled output image, and k(x,y) is the filtering weight function.

[0085] S105. Perform bidirectional projection in the temporal domain and accumulate the mean and variance of the color and edges of the video frames;

[0086] Specifically, for videos and animations, Halton low-difference sequences are used and the projection matrix parameters are adjusted according to formula (3) for jitter rendering. Then, based on the projection matrix and jitter sequence parameters of adjacent frames, samples corresponding to the previous and next frames are found by bidirectional inverse projection. In order to improve the effectiveness and robustness of sample alignment in the case of light changes, object occlusion, and fast movement in videos and animations, the edge direction of the samples obtained in step S102 is considered. The 7x7 neighborhood of the sample center is used for detection and calculation. If the color brightness change and edge direction change of the corresponding samples of the previous and next frames of the video are within the preset range, they are considered to be effectively aligned samples. Otherwise, they are considered to be unaligned samples.

[0087] Formula (3) is as follows:

[0088]

[0089] Here, Halton(·,N) represents a Halton low-discrepancy sequence of length N, which is a pseudo-random function uniformly distributed in a high-dimensional space and has fast convergence properties.

[0090] When the number of valid samples in a 7x7 neighborhood exceeds a pre-set threshold, the average color brightness and variation range of the samples under N jitter rendering.

[0091] S106. Adjust the fusion coefficients and implement time-domain filtering according to the time-series fusion formula;

[0092] Specifically, the resampled color is calculated according to the temporal fusion formula. If the difference between the two values ​​is within the range of local center variation, it is directly output. Otherwise, the difference is limited to ensure that the brightness of the color after temporal fusion does not change abruptly.

[0093] The timing fusion formula is as follows:

[0094]

[0095] in, H represents the output image of the time-series filtering at time t. t (x,y) represents the spatial filtering result image at time t, and α represents the fusion coefficient.

[0096] S107. Perform gamut conversion and tone mapping on the result of temporal filtering, and output the reconstruction result.

[0097] Specifically, the color gamut conversion and tone mapping of the time-domain filtering result includes the following steps:

[0098] Based on the display device and scene requirements, the output color dynamic range is defined, the tone mapping piecewise spline curve adjustment parameters are calculated, and the ACES color system is referenced. The output transformation is performed based on the maximum brightness of the display device and the maximum brightness of the scene content. That is, the color gamut conversion and tone mapping are performed on the super-resolution reconstruction results. Specifically, if the device supports high dynamic range display or needs to output HDR video / images, the ACES reference rendering transformation is performed on the super-resolution reconstruction results to stretch and map the scene content colors to the high dynamic range; conversely, if the device only supports standard dynamic range display or needs to output SDR video / images, the ACES inverse reference rendering transformation is performed to compress the scene content colors to the standard dynamic range.

[0099] To preserve important details and color information of the original scene to the greatest extent possible, guided by the principle of preserving color perception in the human visual system, color gamut conversion and tone mapping were performed on the super-resolution reconstruction results. The minimum and peak brightness of the scene colors obtained during the reconstruction process were reused, and the minimum and peak brightness of the display output were used as references. 15 nits (candela per square meter, cd / m²) was selected as the intermediate grayscale brightness. Two piecewise curve coefficients, minimum brightness - intermediate grayscale and intermediate grayscale - peak brightness, were fitted using the second-order Bézier curve formula. The specific formula is as follows:

[0100] B(t)=(1-t 2 )P0+2(1-t)tP1+t 2 P2 0≤t≤1

[0101] Where t is the distance parameter of the Bézier curve, and P0, P1, and P2 represent the control points of the second-order Bézier curve. For the lowest brightness-intermediate grayscale curve, P0 and P2 correspond to the lowest brightness and intermediate grayscale, respectively; for the intermediate grayscale-peak brightness curve, P0 and P2 correspond to the intermediate grayscale and peak brightness, respectively.

[0102] Before outputting the reconstruction results, color compensation and correction can be performed. Specifically, according to color theory, human color perception is non-linear, and the visual perception system exhibits the Stevens effect and the Hunt effect. To reduce color perception shift during brightness changes, color compensation and correction are performed with reference to ITU-R BT.2446-1 report. The specific formula is as follows:

[0103] C o =C i +Hue c *Sat c *Cst c (1-C i )

[0104] Among them, C i and C o These represent the color values ​​before and after compensation / correction, respectively. c Sat c and Cst c These represent predefined constant coefficients for adjusting hue, saturation, and contrast, which can generally be obtained through the color calibration process.

[0105] Depending on the display device and specific requirements, select a target output color space such as sRGB, scRGB, BT.2100, Adobe RGB, Display-P3, or DCI-P3. Convert different color sources through XYZ space and apply Gamma, PQ (Perceptual quantizer), or HLG (Hybrid log–gamma) transfer functions according to the specific color space to convert the colors to the display device space.

[0106] To reduce CPU load, the above color conversion process is implemented by writing a custom pixel shader, so that the function runs entirely in the GPU, making full use of the efficient and powerful parallel pipeline processing capabilities of modern graphics processors to meet the real-time processing requirements of the end device.

[0107] It should be noted that a shader refers to the functionality required by an algorithm program implemented through a GPU-specific programming language. This code runs entirely within the GPU. Computation shaders and pixel shaders are programmable functions provided to developers by the GPU in different pipelines and are general concepts in GPU development. Here, a custom pixel shader refers to writing functional code to implement color gamut mapping and color conversion for each pixel of an image / video.

[0108] Finally, the reconstruction results are output, either by displaying the reconstruction results on a display device or by encoding the result data as video / images and saving it to a file.

[0109] It should be noted that the temporal and spatial domain combined GPU super-resolution reconstruction and high dynamic range algorithm utilizes the features of modern graphics processing unit (GPU) programmable pipelines and powerful parallel data processing capabilities to analyze video / image content in real time on the GPU, performing super-resolution reconstruction and color dynamic range conversion. This significantly improves the algorithm's processing efficiency and greatly reduces CPU computational load, enabling real-time processing on mobile devices. To maximize rendering efficiency and the algorithm's versatility, this method first selects the optimal GPU programming language based on the specific GPU's capabilities and different operating systems. For example, it uses Direct3D / Vulkan / CUDA / OpenCL on Microsoft Windows systems, Vulkan / OpenGL / OpenCL / CUDA on Linux systems, Vulkan / OpenGL ES on Google Android systems, and Metal / OpenGL / OpenGL ES / OpenCL on Apple iOS and macOS X systems. Then, the specific data processing steps of the algorithm are implemented using the corresponding GPU shader programming language.

[0110] The following example demonstrates a concrete implementation of real-time super-resolution reconstruction and high dynamic range methods, using video editing as an example:

[0111] 1. Apply photoelectric transfer function and ACES input color transformation to the input image or video, convert it to the ACEScg linear color space, and remove the color characteristics of the material resources;

[0112] 2. Utilize the GPU to analyze video content in real time, calculate the color and brightness of the current frame image, sample the surrounding pixels of a 2x2 sub-block in the center of a 4x4 region, and extract the geometric contours and edge information of the image content;

[0113] 3. By writing a custom GPU computing shader and calculating the weight function of the filter according to the weight function calculation formula, the image filtering coefficients are obtained, and the spatial domain reconstruction process is completed by using the image upsampling calculation formula;

[0114] 4. Using a forward and backward bidirectional projection method, with a 7x7 neighborhood as the block unit, the mean and variance of the color and edge of adjacent video frames are accumulated and calculated, the fusion coefficients are adjusted, and temporal filtering is achieved according to the temporal fusion formula;

[0115] 5. Reuse the scene brightness obtained from the super-resolution reconstruction process, calculate the tone mapping piecewise spline curve adjustment parameters according to the display device or specific requirements, and write a custom GPU pixel shader to implement the color conversion process;

[0116] 6. Present the reconstruction results to a display device or save the results data to a file.

[0117] Example 2

[0118] The second embodiment of this application relates to a computer-readable storage medium that stores program code for execution by a device, the program code including steps for performing the method in any implementation of the first embodiment of this application;

[0119] The computer-readable storage medium may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM); the computer-readable storage medium may store program code, and when the program stored in the computer-readable storage medium is executed by a processor, the processor is used to perform the steps of the method in any of the implementations of Embodiment 1 of this application.

[0120] The above are merely preferred embodiments of this application; however, the scope of protection of this application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in this application, based on the technical solution and its improved concept, should be covered within the scope of protection of this application.

Claims

1. A GPU-based super-resolution and high dynamic range processing method, characterized in that, include: Apply ACES input color transformation to the source material; Utilize GPUs to analyze the geometric edges of the material; By writing a custom GPU compute shader, the weight function of the filter is calculated according to the weight function calculation formula; The image spatial domain filtering and reconstruction process is completed based on the image upsampling calculation formula; Perform bidirectional projection in the temporal domain and accumulate the mean and variance of the color and edges of the video frames; Adjust the fusion coefficients and implement temporal filtering according to the temporal fusion formula; Perform gamut conversion and tone mapping on the results of temporal filtering, and output the reconstruction results.

2. The GPU-based super-resolution and high dynamic range processing method according to claim 1, characterized in that, The geometric edges of the analyzed material include: Calculate the brightness of the current image color; Sampling of pixels surrounding a 2x2 sub-block at the center of a 4x4 region; Based on the brightness change of the central pixel in the region, calculate the brightness gradient intensity of the four local edge directions at 0°, 45°, 90° and 135°. Extract the geometric contours and edge information of the material.

3. The GPU-based super-resolution and high dynamic range processing method according to claim 1, characterized in that, The formula for calculating the weighting function is as follows: x r =y*sinθ+x*cosθ and r =y*sinθ-x*sinθ K(x, y)=k1Lut(Dist(x, y))+k2Lum(x, y)+k3Feat(xr, yr) (1) Where K(x,y) is the filter weight function, (x r ,y r Dist(x,y) is the coordinates rotated by the local edge direction θ, Lut(d) is the sampling center distance, Lut(d) is the Lanczos filter coefficient lookup table function, Lum(x,y) is the scaling center brightness, and Feat(x) is the sampling center distance. r ,y r ) is the rotated coordinate (x) r ,y r The local gradient intensity of ).

4. The GPU-based super-resolution and high dynamic range processing method according to claim 1, characterized in that, The formula for calculating image upsampling is: Where S(x,y) is the input low-resolution image, H(x,y) is the upsampled output image, and K(x,y) is the filtering weight function.

5. The GPU-based super-resolution and high dynamic range processing method according to claim 3, characterized in that, The bidirectional projection in the time domain includes the following steps: The material was jittered and rendered using Halton low-difference sequences and by adjusting the projection matrix parameters according to formula (3); Based on the projection matrix and jitter sequence parameters of adjacent frames, find the corresponding samples of the previous and next frames by bidirectional inverse projection forward and backward; Formula (3) is as follows: Here, Halton(·,N) represents a Halton low-discrepancy sequence of length N, which is a pseudo-random function uniformly distributed in a high-dimensional space.

6. The GPU-based super-resolution and high dynamic range processing method according to claim 5, characterized in that, The bidirectional projection in the time domain further includes: Based on the edge direction θ of the sample, detection and calculation are performed in the 7x7 neighborhood of the sample center. If the color brightness change and edge direction change of the corresponding sample in the previous and next frames of the video are within the pre-set interval, the sample is considered to be effectively aligned. If the color brightness change and edge direction change of the corresponding sample in the previous and next frames of the video are not within the pre-set interval, the sample is considered to be invalidally aligned.

7. The GPU-based super-resolution and high dynamic range processing method according to claim 1, characterized in that, The cumulative calculation of the mean and variance of the color and edges of the video frames includes the following steps: When the number of valid samples in a 7x7 neighborhood exceeds a pre-set threshold, the mean value of sample color brightness Mean(x,y) and the range of variation Range(x,y) under N jitter renderings are calculated.

8. The GPU-based super-resolution and high dynamic range processing method according to claim 1, characterized in that, The timing fusion formula is as follows: in, H represents the output image of the time-series filtering at time t. t (x,y) represents the spatial filtering result image at time t, and α represents the fusion coefficient.

9. The GPU-based super-resolution and high dynamic range processing method according to claim 1, characterized in that, The color gamut conversion and tone mapping of the time-domain filtering result includes the following steps: Based on the input materials, display devices, and scene requirements, and guided by the principle of preserving brightness perception in the human visual system, the results of temporal filtering are subjected to color gamut conversion and tone mapping. The minimum and peak brightness of the scene colors obtained through the reconstruction process are reused. Using the minimum and peak brightness of the displayed output as references, appropriate intermediate grayscale brightness is selected. Then, two piecewise curve coefficients—minimum brightness - intermediate grayscale and intermediate grayscale - peak brightness—are fitted according to the second-order Bézier curve formula. The specific formula is as follows: B(t)=(1-t 2 )P0+2(1-t)tP1+t 2 P2 0≤t≤1 (5) Where t is the distance parameter of the Bézier curve, and P0, P1, and P2 represent the control points of the second-order Bézier curve. For the lowest brightness-intermediate grayscale curve, P0 and P2 correspond to the lowest brightness and intermediate grayscale, respectively; for the intermediate grayscale-peak brightness curve, P0 and P2 correspond to the intermediate grayscale and peak brightness, respectively.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is for program code executed by the device, the program code including steps for performing the method as described in any one of claims 1-9.

Images