Image quality enhancement method and system based on cloud game

Abstract

The invention provides an image quality enhancement method and system based on a cloud game, and the method comprises the following steps: firstly, a cloud end renders a game picture and generates an initial video stream; then, performing preliminary optimization on the video stream, and reducing jumping and frame loss through motion compensation of an inter-frame motion vector; the optimized video stream is transmitted to a user side, after the user side receives the video stream, image quality enhancement processing is carried out, specifically, denoising processing is carried out, and random noise and compression noise are eliminated by combining an inter-frame difference method and spatial domain bilateral filtering; sharpening enhancement: enhancing image edges and texture details by using a Laplacian operator; artifact removal: removing compression artifacts through frequency domain filtering; the dynamic contrast is enhanced, and gray level distribution is adjusted through histogram equalization; finally, the video stream after enhancement processing is presented to a user in real time, the image quality of the cloud game can be remarkably improved by gradually updating the processing result of each pixel point, the video stream is presented in real time in a high-quality mode, and the user experience is optimized.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method and system for enhancing image quality based on cloud gaming. Background Art

[0002] Cloud gaming is an emerging technology that has developed rapidly in recent years. It uses cloud computing and streaming technology to allow players to experience high-quality game content without high-performance hardware. However, since the rendering of the game screen is completed in the cloud and transmitted to the user end in the form of video streaming, it is limited by bandwidth, latency and compression algorithms, and the image quality is often affected to a certain extent, such as resolution reduction, loss of details and compression artifacts.

[0003] Therefore, it is necessary to design a cloud gaming-based image quality enhancement method and system to solve the above technical problems. Summary of the invention

[0004] The present invention provides a method and system for enhancing image quality based on cloud gaming, which are used to solve the technical problems mentioned in the above background technology.

[0005] To solve the above problems, the present invention provides the following technical solution: a method for enhancing image quality based on cloud gaming, comprising the following steps:

[0006] S100, cloud-based rendering of game images and generation of initial video streams;

[0007] S200, performing preliminary optimization on the generated video stream, including motion compensation based on inter-frame motion vectors, to reduce inter-frame jitter and frame loss;

[0008] S300, transmitting the optimized video stream to the user end;

[0009] S400, after receiving the video stream, the user end enhances the image quality of the video stream, including:

[0010] S401, denoising processing, combining inter-frame difference method and spatial domain bilateral filtering method to eliminate random noise and compression noise;

[0011] S402, sharpening enhancement, sharpens the image through the Laplacian operator to enhance edge and texture details;

[0012] S403, artifact removal, eliminating compression artifacts based on frequency domain filtering, and applying regional interpolation smoothing to macroblock boundaries;

[0013] S404, dynamic contrast enhancement, adjusts the image grayscale distribution through histogram equalization, and combines gamma correction to improve the image brightness and color performance;

[0014] S500, presenting the enhanced video stream to the user in real time.

[0015] Furthermore, in step S401, it includes:

[0016] S4011, time domain denoising:

[0017] By calculating the pixel value difference between the differential images of consecutive frames, the random noise in the time series can be eliminated;

[0018] If the difference value is less than the specified threshold, the noise pixels are smoothed by weighted averaging;

[0019] S4012, spatial denoising:

[0020] Use bilateral filtering algorithm to filter the current frame image;

[0021] In bilateral filtering, spatial Gaussian kernel and intensity Gaussian kernel are used to control the distance weight and grayscale difference weight respectively, so as to retain edge details and remove noise at the same time.

[0022] Furthermore, in step S402, the sharpening enhancement step is implemented based on the following formula:

[0023] I'(x, y) = I(x, y) + λ * ∇²I(x, y)

[0024] Among them, I(x, y) represents the pixel value of the original image, ∇²I(x, y) represents the Laplacian operator of the image, and λ is the sharpening strength coefficient, which is used to control the degree of edge enhancement.

[0025] Furthermore, in step S403, the artifact removal step includes:

[0026] S4031, frequency domain filtering:

[0027] Perform discrete cosine transform DCT on the image to extract the frequency domain components of the image;

[0028] Threshold filtering is performed on high-frequency components to remove compression artifacts;

[0029] Perform inverse discrete cosine transform IDCT on the filtered frequency domain data to restore the optimized image;

[0030] S4032, Blockiness Fix:

[0031] In view of the macroblock boundary effect generated during image transmission, a regional interpolation algorithm is used to smooth the boundary area.

[0032] Reduce the impact of blocking effects on image visual quality.

[0033] Furthermore, in step S404, the dynamic contrast enhancement step includes:

[0034] S4041, histogram equalization:

[0035] Calculate the grayscale histogram distribution and cumulative distribution function of the image;

[0036] The original grayscale value is mapped to the enhanced grayscale range using the cumulative distribution function to expand the dynamic range of the image;

[0037] S4042, Gamma Correction:

[0038] For the enhanced image, the pixel value is adjusted according to the following formula:

[0039] I'(x, y) = I(x, y)^γ

[0040] Among them, I(x, y) is the original pixel value; γ is the gamma value, which is used to adjust the brightness or dark details of the image.

[0041] Furthermore, in step S200, the motion compensation uses a block matching algorithm (BMA) to estimate the motion vector, which specifically includes:

[0042] S201, dividing the current frame and the reference frame into a number of blocks of fixed sizes;

[0043] S202, for each block, find the best matching block within the search range of the reference frame and calculate the matching error MAD:

[0044] MAD = (1 / N²) Σ |B_t(x, y) - B_(t-1)(x + dx, y + dy)|

[0045] Where N is the size of the block; B_t(x, y) and B_(t-1)(x + dx, y + dy) represent the pixel values ​​of the corresponding blocks in the current frame and the reference frame respectively; dx, dy are the displacements of the blocks in the reference frame;

[0046] S203, generating a motion vector according to the matching result, and performing motion compensation processing on the current frame.

[0047] The present invention also provides a cloud gaming-based image quality enhancement system, comprising:

[0048] Cloud rendering module, used to render game images and generate initial video streams;

[0049] The video optimization module is set up in the cloud and is used to perform preliminary optimization on the generated video stream, including motion compensation based on inter-frame motion vectors to reduce inter-frame jitter and frame loss;

[0050] A transmission module, used for transmitting the optimized video stream to the user end;

[0051] The user-side image quality enhancement module is provided at the user-side and is used to enhance the image quality of the received video stream, including:

[0052] De-noising processing unit, combining inter-frame difference method and spatial domain bilateral filtering method, is used to eliminate random noise and compression noise;

[0053] The sharpening enhancement unit is based on the Laplacian operator and is used to sharpen the image and enhance the edge and texture details;

[0054] Artifact removal unit, based on frequency domain filtering, is used to remove compression artifacts and use regional interpolation smoothing on macroblock boundaries;

[0055] Dynamic contrast enhancement unit, which adjusts the grayscale distribution of the image through histogram equalization and combines gamma correction to improve the brightness and color performance of the picture;

[0056] The video presentation module is used to present the enhanced video stream to the user in real time.

[0057] Compared with the prior art, the present invention has at least the following beneficial effects:

[0058] Through processes such as denoising, sharpening, artifact removal and dynamic contrast enhancement, the present invention will gradually update the processing results of each pixel point to the image frame, and finally output an enhanced high-quality video stream and present it to the user end in real time, significantly improving the image quality performance of cloud games. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 is a flowchart of the steps of a method for enhancing image quality based on cloud gaming in an embodiment of the present invention;

[0061] Figure 2 It is a structural diagram of a cloud gaming-based image quality enhancement system in an embodiment of the invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention. It is understood that the drawings are only provided for reference and illustration, and are not used to limit the present invention. The connection relationship shown in the drawings is only for the convenience of clear description and does not limit the connection method.

[0063] The present invention discloses a method for enhancing image quality based on cloud gaming, which significantly improves the image quality of video streams by combining cloud-side optimization with user-side enhancement processing.

[0064] See also Figure 1 , specifically including the following steps:

[0065] In step S100, the cloud server uses a high-performance GPU to render the game screen, generates 60 frames per second, and compresses it into a video stream through a video codec (such as H.264 or H.265). Taking 1080p resolution as an example, the initial size of each frame is about 1920 × 1080 pixels, and the target bit rate of the video stream is set to 10 Mbps.

[0066] Step S200, performing preliminary optimization processing on the generated video stream, using motion compensation technology based on inter-frame motion vectors to reduce inter-frame jitter and frame loss problems.

[0067] In the cloud optimization stage, this method uses motion compensation technology to reduce frame jumps and frame drops. The specific implementation steps include:

[0068] Use the block matching algorithm BMA to estimate the inter-frame motion vector. The specific steps are as follows:

[0069] Step S201, dividing the current frame and the reference frame into a number of blocks of fixed size;

[0070] In this embodiment, the current frame and the reference frame are divided into blocks of a fixed size of 16×16 pixels.

[0071] Step S202, for each block, find the best matching block within the search range of the reference frame and calculate the matching error MAD:

[0072] MAD = (1 / N²) Σ |B_t(x, y) - B_(t-1)(x + dx, y + dy)|

[0073] Where N is the size of the block; B_t(x, y) and B_(t-1)(x + dx, y + dy) represent the pixel values ​​of the corresponding blocks in the current frame and the reference frame respectively; dx, dy are the displacements of the blocks in the reference frame;

[0074] In this embodiment, the block size is 16 × 16 pixels, so N=16;

[0075] The pixel value range of a block B_t in the current frame is [120, 128, 130, ...];

[0076] The pixel value range of the corresponding block of the reference frame is [122, 126, 129, ...];

[0077] Assume dx=1,dy=1;

[0078] Calculate the MAD value:

[0079] MAD = (1 / 16²) Σ |B_t(x, y) - B_(t-1)(x + dx, y + dy)|=2.8;

[0080] Step S203: Generate a motion vector according to the matching result, and perform motion compensation processing on the current frame.

[0081] The best matching block is selected according to the minimum MAD value, and a motion vector (dx, dy) is generated. Based on this, motion compensation is performed on the current frame to reduce jitter.

[0082] Step S300, transmitting the optimized video stream to the user end;

[0083] In this embodiment, the optimized video stream can be transmitted to the user end at 60 FPS (frame rate) and 10 Mbps bit rate.

[0084] Step S400: After receiving the video stream, the user terminal enhances the image quality through the following four steps.

[0085] S401, denoising processing combines time domain denoising and spatial domain denoising to eliminate random noise and compression noise.

[0086] S4011, time domain denoising: use frame difference method combined with weighted average method to smooth noise.

[0087] Assume that two consecutive frames of images are F_t(x, y) and F_(t-1)(x, y), then the inter-frame difference image D_t(x, y) can be expressed as:

[0088] D_t(x, y) = |F_t(x, y) - F_(t-1)(x, y)|

[0089] If the difference value D_t(x, y) is less than a certain threshold T_d, the pixel is considered to be noise and is smoothed using the weighted average method:

[0090] F'_t(x, y) = α * F_t(x, y) + (1 - α) * F_(t-1)(x, y)

[0091] Here, α is the smoothing weight parameter.

[0092] In this embodiment, assuming that the current frame pixel value F_t(x, y) =125, F_(t-1)(x, y)=124, then D_t(x, y) =1.

[0093] If D_t(x, y)<5 (threshold), weighted average smoothing is used:

[0094] F'_t(x, y) =0.8* F_t(x, y) + 0.2* F_(t-1)(x, y)

[0095] Substitute the data: F'_t(x, y) =124.8

[0096] The calculated F'_t(x, y) =124.8 replaces the value of the pixel in the current frame and updates the denoised frame.

[0097] S4012, spatial denoising:

[0098] Bilateral filtering is used to remove noise while retaining edge details. The calculation formula for bilateral filtering is:

[0099] I'(x, y) = (1 / W_p) * Σ_(i,j) G_s(||p - q||) * G_r(|I(p) - I(q)|) *I(q)

[0100] in:

[0101] I(p) represents the gray value of pixel p; G_s is the spatial Gaussian kernel, which is used to control the distance weight; G_r is the intensity Gaussian kernel, which is used to control the gray difference weight; W_p is the normalization factor.

[0102] In this embodiment, I'(x, y) ≈123.5 is calculated and replaced with the original pixel value to complete the spatial domain denoising process.

[0103] S402, the sharpening enhancement step is implemented based on the following formula:

[0104] I'(x, y) = I(x, y) + λ * ∇²I(x, y)

[0105] Among them, I(x, y) represents the pixel value of the original image, ∇²I(x, y) represents the Laplacian operator of the image, and λ is the sharpening strength coefficient, which is used to control the degree of edge enhancement.

[0106] In this embodiment, λ=0.5, I(x, y)=128, ∇²I(x, y)=-4;

[0107] Substitute the formula: I'(x, y)=126, and replace the original pixel value I(x, y) with the calculated value 126 to complete the sharpening process.

[0108] S403, artifact removal, eliminating compression artifacts based on frequency domain filtering, and applying regional interpolation smoothing to macroblock boundaries;

[0109] S4031, frequency domain filtering:

[0110] Perform discrete cosine transform DCT on the image to extract the frequency domain components of the image;

[0111] Threshold filtering is performed on high-frequency components to remove compression artifacts;

[0112] Perform inverse discrete cosine transform IDCT on the filtered frequency domain data to restore the optimized image;

[0113] Assuming the high-frequency component threshold is 20 and the original high-frequency coefficient is 30, the high-frequency coefficient after filtering is 30 - 20 = 10; the inverse transform (IDCT) is restored to the spatial domain, and the updated high-frequency component is applied to image restoration.

[0114] S4032, Blockiness Fix:

[0115] In view of the macroblock boundary effect generated during image transmission, a regional interpolation algorithm is used to smooth the boundary area.

[0116] Reduce the impact of blocking effects on image visual quality.

[0117] In this embodiment, it is assumed that the block boundary pixel values ​​I1(x, y)=120, I2(x, y)=124,

[0118] Boundary pixel value I'(x, y)=[I1(x, y)+I2(x, y)] / 2=122

[0119] Therefore, the boundary pixel value is updated to 122 to smooth out the blocking effect.

[0120] Step S404, the dynamic contrast enhancement step includes:

[0121] S4041, histogram equalization:

[0122] Calculate the grayscale histogram distribution and cumulative distribution function of the image;

[0123] The original grayscale value is mapped to the enhanced grayscale range using the cumulative distribution function to expand the dynamic range of the image;

[0124] For example: the original grayscale distribution range is [50, 150], which is mapped to [0, 255];

[0125] S4042, Gamma Correction:

[0126] For the enhanced image, the pixel value is adjusted according to the following formula:

[0127] I'(x, y) = [I(x, y) / 255]^γ* 255

[0128] Where I(x, y) is the original pixel value; γ is the gamma value, which is used to adjust the brightness or dark details of the image;

[0129] Assume γ = 2.2, the original pixel value I (x, y) = 128, and substitute it into the formula to calculate:

[0130] I'(x, y) = 64, therefore, replace the original pixel value I(x, y) = 128 with $I'(x, y) = 64 to adjust the brightness and contrast.

[0131] Step S500: presenting the enhanced video stream to the user in real time.

[0132] Each of the above-mentioned enhanced image frames is combined into a video stream and presented to the user in real time at a target resolution (such as 1080p) and frame rate (such as 60 FPS).

[0133] The value obtained by the calculation formula is directly applied to the adjustment of pixel values, thereby completing the processes of denoising, sharpening, artifact removal and dynamic contrast enhancement. The processing results of each pixel will be gradually updated to the image frame, and finally the enhanced high-quality video stream will be output and presented to the user end in real time, significantly improving the image quality of cloud games.

[0134] Through experimental comparison, the above embodiments can achieve:

[0135] Denoising effect: Peak signal-to-noise ratio (PSNR) is improved by 5 dB.

[0136] Sharpening effect: Increases edge strength by 15%.

[0137] Artifact Removal: Structural Similarity (SSIM) improved to 0.96.

[0138] Contrast Enhancement: Expands the image dynamic range up to 3 times.

[0139] like Figure 2 As shown, the present invention also discloses a system for enhancing image quality based on cloud gaming, which is used to execute the method steps described above, including:

[0140] A cloud rendering module 11 is used to render the game screen and generate an initial video stream;

[0141] The video optimization module 12 is arranged in the cloud and is used to perform preliminary optimization on the generated video stream, including motion compensation based on inter-frame motion vectors to reduce inter-frame jitter and frame loss;

[0142] The transmission module 13 is used to transmit the optimized video stream to the user end;

[0143] The user-side image quality enhancement module 14 is provided at the user-side and is used to enhance the image quality of the received video stream, including:

[0144] A denoising processing unit 141 is used to eliminate random noise and compression noise by combining an inter-frame difference method and a spatial domain bilateral filtering method;

[0145] A sharpening enhancement unit 142, based on a Laplacian operator, is used to sharpen the image and enhance edge and texture details;

[0146] An artifact removal unit 143 is used to remove compression artifacts based on frequency domain filtering and to smooth the macroblock boundaries using regional interpolation;

[0147] A dynamic contrast enhancement unit 144 adjusts the grayscale distribution of the image through histogram equalization and improves the brightness and color performance of the image in combination with gamma correction;

[0148] The video presentation module 15 is used to present the enhanced video stream to the user in real time.

[0149] In the specification and claims of the present application, the words "include / comprise" and the words "have / include" and their variations are used to specify the existence of stated features, values, steps or components, but do not exclude the existence or addition of one or more other features, values, steps, components or combinations thereof.

[0150] Some features of the present invention are described in different embodiments for clarity of description, however, these features may also be described in combination in a single embodiment. Conversely, some features of the present invention are described in only a single embodiment for brevity, however, these features may also be described in different embodiments alone or in any suitable combination.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for enhancing image quality based on cloud gaming, characterized in that: The following steps are involved: S100, cloud-based rendering of game images and generation of initial video streams; S200, performing preliminary optimization on the generated video stream, including motion compensation based on inter-frame motion vectors, to reduce inter-frame jitter and frame loss; S300, transmitting the optimized video stream to the user end; S400, after receiving the video stream, the user end enhances the image quality of the video stream, including: S401, denoising processing, combining inter-frame difference method and spatial domain bilateral filtering method to eliminate random noise and compression noise; S402, sharpening enhancement, sharpens the image through the Laplacian operator to enhance edge and texture details; S403, artifact removal, eliminating compression artifacts based on frequency domain filtering, and applying regional interpolation smoothing to macroblock boundaries; S404, dynamic contrast enhancement, adjusts the image grayscale distribution through histogram equalization, and combines gamma correction to improve the image brightness and color performance; S500, presenting the enhanced video stream to the user in real time.

2. The method for enhancing image quality based on cloud gaming according to claim 1, characterized in that: In step S401, it includes: S4011, time domain denoising: By calculating the pixel value difference between the differential images of consecutive frames, the random noise in the time series can be eliminated; If the difference value is less than the specified threshold, the noise pixels are smoothed by weighted averaging; S4012, spatial denoising: Use bilateral filtering algorithm to filter the current frame image; In bilateral filtering, spatial Gaussian kernel and intensity Gaussian kernel are used to control the distance weight and grayscale difference weight respectively, so as to retain edge details and remove noise at the same time.

3. The method for enhancing image quality based on cloud gaming according to claim 2, characterized in that: In step S402, the sharpening enhancement step is implemented based on the following formula: I'(x, y) = I(x, y) + λ * ∇²I(x, y) Among them, I(x, y) represents the pixel value of the original image, ∇²I(x, y) represents the Laplacian operator of the image, and λ is the sharpening strength coefficient, which is used to control the degree of edge enhancement.

4. The method for enhancing image quality based on cloud gaming according to claim 3, characterized in that: In step S403, the artifact removal step includes: S4031, frequency domain filtering: Perform discrete cosine transform DCT on the image to extract the frequency domain components of the image; Threshold filtering is performed on high-frequency components to remove compression artifacts; Perform inverse discrete cosine transform IDCT on the filtered frequency domain data to restore the optimized image; S4032, Blockiness Fix: In view of the macroblock boundary effect generated during image transmission, a regional interpolation algorithm is used to smooth the boundary area. Reduce the impact of blocking effects on image visual quality.

5. The method for enhancing image quality based on cloud gaming according to claim 4, characterized in that: In step S404, the dynamic contrast enhancement step includes: S4041, histogram equalization: Calculate the grayscale histogram distribution and cumulative distribution function of the image; The original grayscale value is mapped to the enhanced grayscale range using the cumulative distribution function to expand the dynamic range of the image; S4042, Gamma Correction: For the enhanced image, the pixel value is adjusted according to the following formula: I'(x, y) = I(x, y)^γ Among them, I(x, y) is the original pixel value; γ is the gamma value, which is used to adjust the brightness or dark details of the image.

6. The method for enhancing image quality based on cloud gaming according to any one of claims 1 to 5, characterized in that: In step S200, the motion compensation uses a block matching algorithm (BMA) to estimate a motion vector, specifically including: S201, dividing the current frame and the reference frame into a number of blocks of fixed sizes; S202, for each block, find the best matching block within the search range of the reference frame and calculate the matching error MAD: MAD = (1 / N²) Σ |B_t(x, y) - B_(t-1)(x + dx, y + dy)| Where N is the size of the block; B_t(x, y) and B_(t-1)(x + dx, y + dy) represent the pixel values ​​of the corresponding blocks in the current frame and the reference frame respectively; dx, dy are the displacements of the blocks in the reference frame; S203, generating a motion vector according to the matching result, and performing motion compensation processing on the current frame.

7. A cloud gaming-based image quality enhancement system, characterized in that: include: Cloud rendering module, used to render game images and generate initial video streams; The video optimization module is set up in the cloud and is used to perform preliminary optimization on the generated video stream, including motion compensation based on inter-frame motion vectors to reduce inter-frame jitter and frame loss; A transmission module, used for transmitting the optimized video stream to the user end; The user-side image quality enhancement module is provided at the user-side and is used to enhance the image quality of the received video stream, including: De-noising processing unit, combining inter-frame difference method and spatial domain bilateral filtering method, is used to eliminate random noise and compression noise; The sharpening enhancement unit is based on the Laplacian operator and is used to sharpen the image and enhance the edge and texture details; The artifact removal unit is based on frequency domain filtering to eliminate compression artifacts and uses regional interpolation smoothing on macroblock boundaries; Dynamic contrast enhancement unit, which adjusts the grayscale distribution of the image through histogram equalization and combines gamma correction to improve the brightness and color performance of the picture; The video presentation module is used to present the enhanced video stream to the user in real time.

Images