Terminal image quality enhancement method and device, and computer readable storage medium

Abstract

A terminal image quality enhancement method and device, and a computer readable storage medium. The terminal image quality enhancement method comprises: acquiring image quality enhancement parameters of a user; according to the image quality enhancement parameters of the user, determining the number of cycles J for performing image quality enhancement by using an image quality enhancement model; and receiving an image to be enhanced, and cyclically using the image quality enhancement model to perform image quality enhancement on said image for J times, wherein the input image of the image quality enhancement model during the first cycle is said image, the input image of the image quality enhancement model during the j-th cycle is the output image of the image quality enhancement model during the (j-1)-th cycle, internal parameters used by the image quality enhancement model in the J cycles are the same, and J≥j>1.

Description

Terminal image quality enhancement method and device, and computer-readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on November 17, 2022, with application number 202211460861.9 and invention name “Terminal image quality enhancement method and device, computer-readable storage medium”, the content of which should be understood as incorporated into this application by reference. Technical Field

[0002] The embodiments of the present disclosure relate to, but are not limited to, the field of image quality enhancement technology, and in particular to a method and apparatus for enhancing terminal image quality, and a computer-readable storage medium. Background Art

[0003] Image quality refers to the quality of the image. In real-world playback scenarios, it's often necessary to improve the image quality of devices like TVs. For example, when playing standard-definition video on a 2K-resolution TV, or when playing standard-definition or 2K-resolution video on a 4K-resolution TV, the mismatch between the source resolution and the screen's display resolution can lead to poor display quality, such as blurry images and jagged edges. However, due to the limited computing power and storage space of devices like TVs, current methods for enhancing image quality often fail to meet the lightweight requirements of real-world scenarios.

[0004] Summary of the Invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The present disclosure provides a method for enhancing terminal image quality, including:

[0007] Get the user's image quality enhancement parameters;

[0008] Determine the number of cycles J for image quality enhancement using the image quality enhancement model based on the user's image quality enhancement parameters;

[0009] An image to be enhanced is received, and the image quality enhancement model is used cyclically to enhance the image quality of the image to be enhanced J times, wherein the input image of the image quality enhancement model in the first cycle is the image to be enhanced, and the input image of the image quality enhancement model in the j-th cycle is the output image of the image quality enhancement model in the j-1th cycle, and the internal parameters of the image quality enhancement model used during the J cycles are the same, and J≥j>1.

[0010] An embodiment of the present disclosure also provides a terminal image quality enhancement device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the terminal image quality enhancement method described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0011] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the terminal image quality enhancement method described in any embodiment of the present disclosure.

[0012] Other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are intended to provide a further understanding of the technical solutions of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation of the technical solutions of the present disclosure. The shapes and sizes of the components in the drawings do not reflect the actual scale and are intended only to illustrate the contents of the present disclosure.

[0014] FIG1 is a flow chart of a method for enhancing terminal image quality according to an exemplary embodiment of the present disclosure;

[0015] FIG2 is a schematic diagram of a solution for image quality enhancement using an image super-resolution (SR) model provided by an exemplary embodiment of the present disclosure;

[0016] FIG3 is a schematic diagram of the structure of an image super-resolution model provided by an exemplary embodiment of the present disclosure;

[0017] FIG4 is a schematic diagram of a solution for image quality enhancement using a definition enhancement (BP) model provided by an exemplary embodiment of the present disclosure;

[0018] FIG5 is a schematic structural diagram of a clarity enhancement model provided by an exemplary embodiment of the present disclosure;

[0019] FIG6 is a schematic diagram of a training method for an image super-resolution model provided by an exemplary embodiment of the present disclosure;

[0020] FIG7 is a schematic diagram of a process of first performing Float32 precision training and then using Int8 quantization training, provided by an exemplary embodiment of the present disclosure;

[0021] FIG8 is a schematic diagram of a method for training a clarity enhancement model provided by an exemplary embodiment of the present disclosure;

[0022] FIG9 is a schematic diagram of a television terminal image quality enhancement module provided by an exemplary embodiment of the present disclosure;

[0023] FIG10 is a schematic diagram of a method for setting front-end parameters of a terminal image quality enhancement module provided by an exemplary embodiment of the present disclosure;

[0024] FIG11 is a schematic structural diagram of a terminal image quality enhancement device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other in any manner.

[0026] Unless otherwise defined, the technical or scientific terms used in the embodiments of the present disclosure should have the ordinary meaning understood by people with ordinary skills in the field to which the present disclosure belongs. The words "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. The words "include" or "comprising" and similar words mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0027] As shown in FIG1 , an embodiment of the present disclosure provides a method for enhancing terminal image quality, including the following steps:

[0028] Step 101: Obtain the user's image quality enhancement parameters;

[0029] Step 102: Determine the number of cycles J for image quality enhancement using the image quality enhancement model according to the user's image quality enhancement parameters;

[0030] Step 103: Receive the image to be enhanced, and cyclically use the image quality enhancement model to enhance the image to be enhanced J times. The input image of the image quality enhancement model in the first cycle is the image to be enhanced, and the input image of the image quality enhancement model in the j-th cycle is the output image of the image quality enhancement model in the j-1-th cycle. The internal parameters of the image quality enhancement model used in the J cycles are the same, and J≥j>1.

[0031] The terminal image quality enhancement method of the embodiment of the present disclosure determines the number of cycles J for image quality enhancement using the image quality enhancement model based on the user's image quality enhancement parameters, and then cyclically uses the image quality enhancement model to enhance the image quality of the image to be enhanced J times, and the internal parameters of the image quality enhancement model used during the J cycles (the internal parameters of the image quality enhancement model here refer to the optimal parameters obtained through training samples in model training) are the same. When the model is initialized, no matter how many cycles are performed, the memory occupied by the model is only the memory occupied by a single model, which can greatly reduce the memory consumption of the hardware, make the model lightweight, and meet the low computing power requirements of the terminal.

[0032] In some exemplary embodiments, as shown in FIG2 , the image quality enhancement model may be an image super resolution (SR) model, and the image quality enhancement parameter may be a first image quality enhancement parameter s, where J=s, x=2 s, x is the resolution enhancement ratio between the output image of the image super-resolution model and the image to be enhanced in the J-th cycle. In the embodiment of the present disclosure, the first image quality enhancement parameter s can be understood as the number of 2x super-resolution cycles. For example, no super-resolution (s=0), 2x super-resolution (s=1), 4x super-resolution (s=2), 8x super-resolution (s=3), 16x super-resolution (s=4), 32x super-resolution (s=5), etc. can be selected. The SR in Figure 2 can be regarded as a 2x super-resolution module, and the internal parameters of the image super-resolution model used in multiple cycles (the internal parameters of the image super-resolution model here refer to the optimal parameters obtained through training samples in model training) are the same.

[0033] In some exemplary embodiments, as shown in FIG3 , the image super-resolution model includes a first feature extraction layer FE1, a first truncation layer Clip1, m serially connected spatial attention layers SA, a channel adjustment layer CC, a first upsampling layer Piex1 Shuffle1, and a second truncation layer Clip2, where m is a natural number between 2 and 5.

[0034] A first feature extraction layer FE1 is configured to extract a feature map of an input image of an image super-resolution model;

[0035] The first truncation layer Clip1 is configured to limit the value of the feature map output by the first feature extraction layer FE1 to a preset range;

[0036] m serially connected spatial attention layers SA are configured to perform feature filtering and enhancement on the feature map output by the first clipping layer Clip1;

[0037] The channel adjustment layer CC is configured to adjust the number of channels of the feature map;

[0038] The first upsampling layer Piex1 Shuffle1 is configured to convert the feature map output by the channel adjustment layer CC into a high-resolution image;

[0039] The second truncation layer Clip2 is configured to limit the value of the high-resolution image output by the first upsampling layer Piex1 Shuffle1 to a preset range.

[0040] In actual use, m can be set according to the hardware conditions of the terminal. For example, m can be set to 2 or 3.

[0041] In some exemplary embodiments, as shown in FIG3 , each spatial attention layer SA includes a second feature extraction layer FE2, a third truncation layer Clip3, a weighted feature mapping layer WFM, a third feature extraction layer FE3, and a fourth truncation layer Clip4, wherein:

[0042] The second feature extraction layer FE2 is configured to perform feature refinement on the feature map input by the current spatial attention layer SA;

[0043] The third truncation layer Clip3 is configured to limit the value of the feature map output by the second feature extraction layer FE2 to a preset range;

[0044] The weighted feature map layer WFM is configured to calculate the spatial attention weight and multiply the calculated spatial attention weight with the feature map output by the third truncation layer Clip3 to obtain a weighted feature map;

[0045] The third feature extraction layer FE3 is configured to perform feature refinement on the weighted feature map;

[0046] The fourth truncation layer Clip4 is configured to limit the value of the feature map output by the third feature extraction layer FE3 to a preset range.

[0047] In some exemplary embodiments, as shown in Figure 3, the weighted feature mapping layer WFM may include a first branch, a second branch and a multiplier, the first branch and the second branch are both arranged between the third truncation layer Clip3 and the multiplier, and the first branch includes a first convolutional layer Conv, a seventh truncation layer Clip7 and an activation function layer Sigmoid connected in series in sequence.

[0048] In some exemplary embodiments, the first convolutional layer Conv may be a convolutional layer with a convolution kernel size of 1*1 or 3*3, however, the embodiments of the present disclosure are not limited thereto. Using a convolution kernel size of 1*1 for the first convolutional layer Conv can further reduce the computing power requirements of the terminal.

[0049] In some exemplary embodiments, the first feature extraction layer FE1 may be a convolution layer with a convolution kernel size of 3*3, however, the embodiments of the present disclosure are not limited thereto.

[0050] In some exemplary embodiments, the second feature extraction layer FE2 may be a convolution layer with a convolution kernel size of 3*3, however, the embodiments of the present disclosure are not limited to this.

[0051] In some exemplary embodiments, the third feature extraction layer FE3 may be a convolution layer with a convolution kernel size of 1*1 or 3*3, however, the present disclosure is not limited to this. The third feature extraction layer FE3 uses a convolution layer with a convolution kernel size of 1*1, which can further reduce the computing power requirements of the terminal.

[0052] In some exemplary embodiments, the channel adjustment layer CC may be a convolution layer with a convolution kernel size of 1*1, however, the embodiments of the present disclosure are not limited thereto.

[0053] In some exemplary embodiments, the clipping layer Clip (which can be any one of the first clipping layer Clip1, the second clipping layer Clip2, the third clipping layer Clip3, the fourth clipping layer Clip4, and the seventh clipping layer Clip7, or any one of the fifth clipping layer Clip5 and the sixth clipping layer Clip6 described later) functions as a clipping operation. The clipping value can be between 0 and 1 (when the model output value is a normalized value) or between 0 and 255 (when the model output value is not normalized). It is expressed by the following formula:

[0054] Or;

[0055] The function of the clipping layer Clip is to ensure that the value output by each convolutional function is between 0 - 1 (or 0 - 255), which is beneficial for subsequent model quantization training and reduces quantization loss. In the image super-resolution model of the embodiments of the present disclosure, the clipping layer Clip is adopted after each convolutional layer, which is beneficial for maximizing the preservation of the model's precision performance during Int8 quantization training.

[0056] In some exemplary embodiments, the image quality enhancement model can be a sharpness enhancement model, and the image quality enhancement parameter is the second image quality enhancement parameter w, where 0 < w ≤ 100%.

[0057] As shown in FIG. 4, BP can be regarded as a sharpness enhancement module, and the internal parameters of the sharpness enhancement model used in multiple loop processes (here, the internal parameters of the sharpness enhancement model refer to the optimal parameters obtained through training samples during model training) are the same. In the embodiments of the present disclosure, the sharpness enhancement model is a back-projection (BackProjection) structure, which includes a downsampling layer (PiexlUnShuffle or others) and an upsampling layer (PiexlShuffle or others). The input of the BP structure is an image, and the output is also an image. Its structure is a residual structure, that is, the substantial content obtained by the sharpness enhancement module is the residual between the enhanced image and the input image, that is, the enhanced detailed part. The second image quality enhancement parameter w is the intensity coefficient for the current user to adjust the enhancement effect, and the value of w is between 0 and 100%.

[0058] In some exemplary embodiments, according to the user's image quality enhancement parameter, determining the number of loops J for performing image quality enhancement using the image quality enhancement model includes:

[0059] Determining the maximum number of loops I, and dividing 0 to 100% into I equal parts. The enhancement percentage corresponding to the i-th equal part is A * 100%, where (i - 1) / I < A ≤ i / I, and i is a natural number between 1 and I;

[0060] The number of equal portions corresponding to the second image quality enhancement parameter w is determined to be the number of cycles J of performing image quality enhancement using the definition enhancement model.

[0061] When the definition enhancement model of the embodiment of the present disclosure is actually used to enhance image quality, the user can control the enhancement effect by adjusting the second image quality enhancement parameter w. For example, assuming that the maximum number of cycles I = 5, the enhancement degree can be divided into the following five levels (the number of levels is the maximum number of cycles):

[0062] 0%——20%——40%——60%——80%——100%

[0063] When the user adjusts the slider of the enhancement effect, if the second image quality enhancement parameter w is 0%, no enhancement is performed, that is, the definition enhancement model is not used for image quality enhancement;

[0064] When the second image quality enhancement parameter w is between (0%, 20%], the definition enhancement model is used to perform image quality enhancement once: Pout = BP 1 (P) = BP(P);

[0065] When the second image quality enhancement parameter w is between (20%, 40%], the definition enhancement model is used cyclically to perform image quality enhancement twice: Pout = BP 2 (P) = BP (BP 1 (P));

[0066] When the second image quality enhancement parameter w is between (40%, 60%], the clarity enhancement model is used cyclically for image quality enhancement 3 times (can be set to the default configuration): Pout = BP 3 (P) = BP (BP 2 (P));

[0067] When the second image quality enhancement parameter w is between (60%, 80%], the definition enhancement model is used cyclically to perform image quality enhancement 4 times: Pout = BP 4 (P) = BP (BP 3 (P));

[0068] When the second image quality enhancement parameter w is between (80%, 100%), the definition enhancement model is used cyclically for image quality enhancement 5 times: Pout = BP 5 (P) = BP (BP 4 (P)).

[0069] In some exemplary embodiments, when the second image quality enhancement parameter w is not equal to an integer multiple of 1 / 1, the terminal image quality enhancement method further includes:

[0070] The difference coefficient w′ is calculated according to the following formula: w′=w*I-J+1;

[0071] Determine the output image as BP J-1 (P)+w′*(BP J (P)-BP J-1 (P)), where BP J-1 (P) is the output image of the clarity enhancement model at the J-1th cycle, BP J (P) is the output image of the clarity enhancement model in the Jth cycle.

[0072] In the embodiment of the present disclosure, in order to enable the user to obtain a continuous clarity change process between each gear, when the second image quality enhancement parameter w falls into one of the above gears, for example, assuming that the second image quality enhancement parameter w adjusted by the user is w=35%, which falls into the gear of (20%, 40%], the clarity enhancement model is used cyclically to enhance the image quality twice, and then 35% is linearly mapped in the interval of (20%, 40%] by 0-1 w′=(35%-20%) / (40%-20%)=0.75 (i.e. w′=w*I-J+1=35%*5-2+1=0.75), then the enhanced picture is Pout=BP 1 (P)+0.75*(BP 2 (P)-BP 1 (P)).

[0073] For example, if the user adjusts the image quality enhancement parameter w = 65%, it falls into the (60%, 80%] gear. At this time, the clarity enhancement model is used cyclically for image quality enhancement 4 times. According to the above calculation, we can get: w′ = (65% - 60%) / (80% - 60%) = 0.25 (that is, w′ = w*I-J+1 = 65%*5-4+1 = 0.25). Then the enhanced picture is Pout = BP 3 (P)+0.25*(BP 4 (P)-BP 3 (P)).

[0074] In this way, users can get a continuous enhancement effect by sliding the bar. When using it, the default configuration can be set to loop 3 times, and the program supports setting loops 4 or 5 times. This can ensure that when the original video to be enhanced is particularly blurry, more times of clarity enhancement are needed to achieve a more ideal effect, thereby providing users with a wider range of enhancement effects.

[0075] In some exemplary embodiments, determining the maximum number of cycles I comprises:

[0076] Detect the computing power space M that the current terminal can provide to the image quality enhancement algorithm;

[0077] Determine whether M is greater than or equal to I2*M bp , where I2 is the maximum number of cycles that the terminal supports to use the definition enhancement model in an ideal situation, M bp The computing power required to use the definition enhancement model to enhance the image quality.

[0078] When M is greater than or equal to I2*M bp When setting the maximum number of cycles I=I2;

[0079] When M is less than I2*M bp When setting the maximum number of cycles in, Indicates M / M bp Round down.

[0080] Ideally, we believe that the computing power of devices such as TVs or mobile terminals can stably and reliably support the cyclic use of the definition enhancement model for image quality enhancement I2 (for example, I2=5) times. However, when the TV simultaneously starts many other modules or the mobile terminal runs many other applications in the background, occasional computing power occupation may not be able to provide sufficient computing power for the definition enhancement model. In this case, the terminal image quality enhancement method of the embodiment of the present disclosure can realize online real-time adjustment of the number of times the definition enhancement model is used for image quality enhancement to complete the enhancement process, as follows:

[0081] Assume that the computing power required for image quality enhancement using the clarity enhancement model is M bp , after online detection, the computing power space that the terminal device can provide to the image quality enhancement algorithm is M. When M≥I2*M bp When M is set to the maximum number of cycles I = I2, and then the number of cycles is determined according to the aforementioned logic to enhance the number of cycles. <I2*M bp When the maximum number of cycles that can be run is recalculated in, Indicates M / M bp Round down (disregarding the situation where even one loop fails to run). Based on the recalculated maximum number of loops, 0-100% is divided into I equal parts, and then the number of loops for enhancement is determined according to the aforementioned logic. Resetting the maximum number of loops I by this method enables adaptive online, real-time control of the number of loops used to enhance the image to be enhanced using the clarity enhancement model based on hardware performance.

[0082] For example, when I=5, the enhancement degree can be divided into the following 5 levels:

[0083] 0%——20%——40%——60%——80%——100%;

[0084] When I=4, the enhancement degree can be divided into the following 4 levels:

[0085] 0%——25%——50%——75%——100%;

[0086] When I=3, the enhancement degree can be divided into the following three levels:

[0087] 0%——33%——66%——100%;

[0088] When I=2, the enhancement degree can be divided into the following two levels:

[0089] 0%——50%——100%;

[0090] When I=1, the enhancement degree can be divided into the following 1 levels:

[0091] At 1 BP, the area is divided into: 0% - 100%.

[0092] The clarity enhancement model of the embodiment of the present disclosure is a backprojection structure, which is a residual structure as shown in Figure 5, including a downsampling layer (PiexlUnShuffle or other) and an upsampling layer (PiexlShuffle or other).

[0093] In some exemplary embodiments, as shown in FIG5 , the definition enhancement model includes a first residual block RB1, the first residual block RB1 including a first main branch, a first shortcut branch, and a first adder, the first main branch and the first shortcut branch are both arranged between the input image and the first adder, and the first residual block RB1 is configured to sum the input image with the input image processed by the first main branch and output the sum; the first main branch includes a downsampling layer Piex1 UnShuffle, a second residual block RB2, a fifth truncation layer Clip5, and a second upsampling layer Piex1 Shuffle2 connected in sequence, wherein:

[0094] The downsampling layer Piexl UnShuffle is configured to downsample the input image to obtain a downsampled image;

[0095] The second residual block RB2 includes a second main branch, a second shortcut branch and a second adder, the second main branch and the second shortcut branch are both arranged between the downsampling layer Piex1 UnShuffle and the second adder, the second main branch includes a plurality of second convolutional layers Conv' and a sixth truncation layer Clip6 arranged between the plurality of second convolutional layers Conv';

[0096] The fifth truncation layer Clip5 is configured to limit the values of the feature map output by the second residual block RB2 within a preset range;

[0097] The second upsampling layer Piexl Shuffle2 is configured to convert the feature map output by the fifth truncation layer Clip5 into a high-resolution image.

[0098] In some exemplary embodiments, the second convolutional layer Conv' can be a convolutional layer with a kernel size of 3*3. However, the embodiments of the present disclosure do not limit this.

[0099] In some exemplary embodiments, the number of the second convolutional layer Conv' in the second main branch can be between 1 and 3. Exemplarily, the number of the second convolutional layer Conv' in the second main branch can be 2.

[0100] In the embodiments of the present disclosure, the role of the truncation layer Clip (any one of the fifth truncation layer Clip5 and the sixth truncation layer Clip6) is to ensure that the values output by each convolutional function are between 0 and 1 (or 0 and 255), which is beneficial to subsequent model quantization training and reduces quantization loss. The clarity enhancement model of the embodiments of the present disclosure adopts the truncation layer Clip after each convolutional layer, which is beneficial to maintaining the accuracy performance of the model to the greatest extent during Int8 quantization training.

[0101] In some exemplary embodiments, the training process of the image quality enhancement model includes:

[0102] Training using C tandem image quality enhancement models with the same structure and the same internal parameters, and simultaneously calculating the losses of using 1 to C image quality enhancement models. Floating-point precision is used during training;

[0103] Quantize the trained floating-point image quality enhancement model into an integer image quality enhancement model.

[0104] In some exemplary embodiments, the training process of the image super-resolution model includes:

[0105] Training using y tandem image super-resolution models with the same structure and the same internal parameters, where 1 < y ≤ I1, and I1 is the maximum number of loops that the terminal supports for recycling the image super-resolution model, and simultaneously calculating the losses of using 1 to y image super-resolution models. Floating-point precision is used during training;

[0106] Quantize the trained floating-point image super-resolution model into an integer image super-resolution model.

[0107] In the embodiments of the present disclosure, training the image super-resolution model includes two steps:

[0108] 1) Floating-point training: As shown in Figure 6, when training an image super-resolution model in floating-point, the internal parameters of multiple image super-resolution models are shared (assuming y = 3, that is, the above 3 image super-resolution models are all the same model with the same set of internal parameters, and the image super-resolution model loops three times). At the same time, 2x / 4x / 8x super-resolution is trained and the losses of 2x / 4x / 8x are calculated simultaneously. The loss function can use the absolute loss function (L1Loss) or the mean square error (MSE) loss function. During training, the model can be trained with Float32 precision;

[0109] 2) As shown in Figure 7, after completing floating-point training, load the Float32 precision model for quantization-aware training (QAT), which can maintain the quality and accuracy of the model while greatly reducing the inference cost. This step of training the model can be trained with Int8 precision.

[0110] In some exemplary embodiments, the training process of the sharpness enhancement model includes:

[0111] Use z concatenated sharpness enhancement models with the same structure and the same internal parameters for training, where 1 < z ≤ I2, and I2 is the maximum number of loops that the terminal supports for recycling the sharpness enhancement model. At the same time, calculate the losses of using 1 to z sharpness enhancement models, and use floating-point precision during training;

[0112] Quantize the trained sharpness enhancement model into an integer sharpness enhancement model.

[0113] In the embodiments of the present disclosure, training the sharpness enhancement model includes two steps:

[0114] 1) Floating-point training: As shown in Figure 8, when training the sharpness enhancement model in floating-point, the internal parameters of multiple sharpness enhancement models are shared (assuming z = 3, that is, the above 3 sharpness enhancement models are all the same model with the same set of internal parameters, and the sharpness enhancement model loops three times). At the same time, enhance 1 time / 2 times / 3 times and calculate the losses of enhancing 1 time / 2 times / 3 times simultaneously. The loss function can use the absolute loss function or the mean square error loss function. During training, the model can be trained with Float32 precision;

[0115] 2) As shown in Figure 7, after completing floating-point training, load the Float32 precision model for quantization-aware training, which can maintain the quality and accuracy of the model while greatly reducing the inference cost. This step of training the model can be trained with Int8 precision.

[0116] In some exemplary embodiments, the image quality enhancement model includes an image super-resolution model and a sharpness enhancement model, and the image quality enhancement parameters include a first image quality enhancement parameter s and a second image quality enhancement parameter w;

[0117] According to the user's image quality enhancement parameters, determine the number of loops J for image quality enhancement using the image quality enhancement model, including: determining the number of loops J1 = s for image quality enhancement using the image super-resolution model; determining the maximum number of loops I for image quality enhancement using the sharpness enhancement model, and dividing 0 to 100% into I equal parts, the enhancement percentage corresponding to the i-th part is A * 100%, where (i - 1) / I < A ≤ i / I, and i is a natural number between 1 and I; determining the number of equal parts corresponding to the second image quality enhancement parameter w, which is the number of loops J2 for image quality enhancement using the sharpness enhancement model;

[0118] Loop and use the image quality enhancement model to perform image quality enhancement on the image to be enhanced J times, including: first, loop and use the sharpness enhancement model to perform image quality enhancement on the image to be enhanced J2 times, and output the first image quality enhanced picture; then, loop and use the image super-resolution model to perform image quality enhancement on the first image quality enhanced picture J1 times, and output the second image quality enhanced picture.

[0119] The structures, usage methods, and training methods of the image super-resolution model and the sharpness enhancement model can be referred to the foregoing, and the embodiments of the present disclosure will not be elaborated herein.

[0120] The terminal image quality enhancement method of the embodiments of the present disclosure can be applied to terminals such as televisions / one-in-all devices / mobile terminals, etc., taking a television terminal as an example, as shown in FIG. 9, a picture quality enhancement module is provided in the television terminal, and the picture quality enhancement module is divided into two parts, a super-resolution (i.e., image super-resolution) module and a sharpness enhancement module: start the above modules after turning on the picture quality enhancement module.

[0121] As shown in FIG. 10, the super-resolution module can achieve resolution enhancement by selecting magnification factors of 1x (s = 0), 2x (s = 1), 4x (s = 2), 8x (s = 3) through the first image quality enhancement parameter s, where 1x represents not performing super-resolution operation.

[0122] The sharpness enhancement module sets a slider mode, and the second image quality enhancement parameter w is set between [0, 100%], and the specific rules can be referred to the loop number setting logic of the foregoing sharpness enhancement model.

[0123] After the setting is completed, transfer the first image quality enhancement parameter s to the super-resolution module and transfer the second image quality enhancement parameter w to the sharpness enhancement module, then an image with enhanced image quality can be obtained.

[0124] An embodiment of the present disclosure also provides a terminal image quality enhancement device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the terminal image quality enhancement method as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0125] As shown in Figure 11, in one example, the terminal image quality enhancement device may include: a processor 1110, a memory 1120, a bus system 1130 and a transceiver 1140, wherein the processor 1110, the memory 1120 and the transceiver 1140 are connected through the bus system 1130, the memory 1120 is used to store instructions and image quality enhancement models, and the processor 1110 is used to execute the instructions stored in the memory 1120 to control the transceiver 1140 to send and receive signals. Specifically, the transceiver 1140 can obtain the user's image quality enhancement parameters and receive the image to be enhanced under the control of the processor 1110. The processor 1110 determines the number of cycles J for image quality enhancement using the image quality enhancement model based on the user's image quality enhancement parameters; the image quality enhancement model is cyclically used to enhance the image to be enhanced J times, wherein the input image of the image quality enhancement model in the first cycle is the image to be enhanced, and the input image of the image quality enhancement model in the j-1 cycle is the output image of the image quality enhancement model in the j-1 cycle. The internal parameters of the image quality enhancement model used during the J cycles are the same, J≥j>1, and the obtained output image is output to the display interface of the terminal through the transceiver 1140.

[0126] It should be understood that the processor 1110 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0127] The memory 1120 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1110. A portion of the memory 1120 may also include a non-volatile random access memory. For example, the memory 1120 may also store information about the device type.

[0128] In addition to the data bus, the bus system 1130 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the bus system 1130 in FIG.

[0129] During the implementation process, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the processor 1110 or the instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1120, and the processor 1110 reads the information in the memory 1120 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0130] The present disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the terminal image quality enhancement method described in any of the embodiments of the present disclosure. The method for driving prognostic analysis by executing executable instructions is substantially the same as the terminal image quality enhancement method provided in the aforementioned embodiments of the present disclosure and is not further described here.

[0131] In some possible implementations, various aspects of the terminal image quality enhancement method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the terminal image quality enhancement method according to various exemplary implementations of the present application described above in this specification. For example, the computer device can execute the terminal image quality enhancement method recorded in the embodiments of the present application.

[0132] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0133] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0134] Although the embodiments disclosed in this disclosure are as described above, the contents described are merely embodiments adopted to facilitate understanding of the disclosure and are not intended to limit the present invention. Any person skilled in the art may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope of the disclosure. However, the scope of patent protection of the present invention shall still be based on the scope defined by the appended claims.

Claims

1. A terminal image quality enhancement method, including: obtaining the image quality enhancement parameters of the user; determining the number of loops J for using the image quality enhancement model to enhance the image quality according to the image quality enhancement parameters of the user; receiving the image to be enhanced, and repeatedly using the image quality enhancement model to enhance the image to be enhanced J times. Among them, the input image of the image quality enhancement model in the first loop is the image to be enhanced, and the input image of the image quality enhancement model in the j-th loop is the output image of the image quality enhancement model in the (j - 1)-th loop. The internal parameters of the image quality enhancement model used during the J loops are the same, where J ≥ j > 1.

2. The terminal image quality enhancement method according to claim 1, wherein, The image quality enhancement model is an image super-resolution model, and the image quality enhancement parameter is the first image quality enhancement parameter s, where J = s and x = 2 s , and x is the resolution enhancement ratio between the output image of the image super-resolution model and the image to be enhanced at the J-th iteration.

3. The terminal image quality enhancement method according to claim 2, wherein, the image super-resolution model includes a first feature extraction layer, a first truncation layer, m serially connected spatial attention layers, a channel adjustment layer, a first upsampling layer, and a second truncation layer connected in sequence. m is a natural number between 2 and 5, and among them: the first feature extraction layer is configured to extract the feature map of the input image of the image super-resolution model; the first truncation layer is configured to limit the value of the feature map output by the first feature extraction layer within a preset range; the m serially connected spatial attention layers are configured to perform feature screening and enhancement on the feature map output by the first truncation layer; the channel adjustment layer is configured to adjust the number of channels of the feature map; the first upsampling layer is configured to convert the feature map output by the channel adjustment layer into a high resolution image; the second truncation layer is configured to limit the value of the high-resolution image output by the first upsampling layer within a preset range.

4. The terminal image quality enhancement method according to claim 3, wherein, each of the spatial attention layers includes a second feature extraction layer, a third truncation layer, a weighted feature mapping layer, a third feature extraction layer, and a fourth truncation layer, and among them: the second feature extraction layer is configured to refine the features of the feature map input to the current spatial attention layer; the third truncation layer is configured to limit the value of the feature map output by the second feature extraction layer within a preset range; the weighted feature mapping layer is configured to calculate the spatial attention weight, and multiply the calculated spatial attention weight by the feature map output by the third truncation layer to obtain a weighted feature map; the third feature extraction layer is configured to refine the features of the weighted feature map; the fourth truncation layer is configured to limit the value of the feature map output by the third feature extraction layer within a preset range.

5. The terminal image quality enhancement method according to claim 2, wherein, the training process of the image super-resolution model includes: using y serially connected image super-resolution models with the same structure and the same internal parameters for training, where 1 < y ≤ I1, and I1 is the maximum number of loops that the terminal supports for repeatedly using the image super-resolution model. At the same time, calculate the losses of using 1 to y image super-resolution models, and use floating-point precision during training; Quantize the trained floating-point image super-resolution model into an integer image super-resolution model.

6. The terminal image quality enhancement method according to claim 1, wherein, the image quality enhancement model is a sharpness enhancement model, and the image quality enhancement parameter is the second image quality enhancement parameter w, where 0 < w ≤ 100%; determining the number of loops J for performing image quality enhancement using the image quality enhancement model according to the user's image quality enhancement parameter includes: including: determine the maximum number of loops I that the terminal supports for repeatedly using the sharpness enhancement model, and divide 0 to 100% into I equal parts. The enhancement percentage corresponding to the i-th equal part is A * 100%, where (i - 1) / I < A ≤ i / I, and i is a natural number between 1 and I; Determine that the number of equal parts corresponding to the second image quality enhancement parameter w is the number of loops J for performing image quality enhancement using the sharpness enhancement model.

7. The terminal image quality enhancement method according to claim 6, wherein, determining the maximum number of loops I includes: detect the computing power space M that the current terminal can provide for the image quality enhancement algorithm; Determine whether M is greater than or equal to I2 * M bp , where I2 is the maximum number of cycles that the terminal supports for recycling the clarity enhancement model in an ideal situation, and M bp is the computing power space required to perform one-time image quality enhancement on the image to be enhanced using the clarity enhancement model; When M is greater than or equal to I2 * M bp set the maximum number of loops I = I2; When M is less than I2 * M bp set the maximum number of loops Among them, Indicates rounding down the value of M / M bp to the nearest whole number.

8. The terminal image quality enhancement method according to claim 6, wherein, when w is not an integer multiple of 1 / I, the method further includes: calculate the difference coefficient w' according to the following formula: w' = w * I - J + 1; Determine that the output image is BP J-1 (P) + w′ * (BP J (P) - BP J-1 (P)), where BP J-1 (P) is the output image of the sharpness enhancement model in the (J - 1)-th iteration, and BP J (P) is the output image of the sharpness enhancement model in the J-th iteration.

9. The terminal image quality enhancement method according to claim 6, wherein, the sharpness enhancement model includes a first residual block, the first residual block includes a first main branch, a first shortcut branch and a first adder. The first main branch and the first shortcut branch are both arranged between the input image and the first adder. The first residual block is configured to sum the input image and the input image after being processed by the main branch and then output; the first main branch includes a downsampling layer, a second residual block, a fifth truncation layer and a second upsampling layer connected in sequence, where: the downsampling layer is configured to downsample the input image to obtain a downsampled image; the second residual block includes a second main branch, a second shortcut branch and a second adder. The second main branch and the second shortcut branch are both arranged between the downsampling layer and the second adder. The second main branch includes a plurality of second convolutional layers and a sixth truncation layer arranged between the plurality of second convolutional layers; the fifth truncation layer is configured to limit the value of the feature map output by the second residual block within a preset range; the second upsampling layer is configured to convert the feature map output by the fifth truncation layer into a high-resolution image.

10. The terminal image quality enhancement method according to claim 6, wherein, the training process of the sharpness enhancement model includes: using z cascaded sharpness enhancement models with the same structure and the same internal parameters for training, 1 < z ≤ I, and simultaneously calculating the losses of using 1 to z sharpness enhancement models. The floating-point precision is used during training; Quantize the trained sharpness enhancement model into an integer sharpness enhancement model.

11. The terminal image quality enhancement method according to claim 1, wherein, The image quality enhancement model includes an image super-resolution model and a sharpness enhancement model, and the image quality enhancement parameters include a first image quality enhancement parameter s and a second image quality enhancement parameter w; Determining the number of loops J for performing image quality enhancement using the image quality enhancement model according to the user's image quality enhancement parameters, including: determining the number of loops J1 = s for performing image quality enhancement using the image super-resolution model; determining the maximum number of loops I for performing image quality enhancement using the sharpness enhancement model, and dividing 0 to 100% into I equal parts, the enhancement percentage corresponding to the i-th equal part is A * 100%, where, (i - 1) / I < A ≤ i / I, and i is a natural number between 1 and I; determining the number of equal parts corresponding to the second image quality enhancement parameter w, which is the number of loops J2 for performing image quality enhancement using the sharpness enhancement model; Looping to perform image quality enhancement on the image to be enhanced J times using the image quality enhancement model, including: first looping to perform image quality enhancement on the image to be enhanced J2 times using the sharpness enhancement model, and outputting a first image quality enhanced picture; then looping to perform image quality enhancement on the first image quality enhanced picture J1 times using the image super-resolution model, and outputting a second image quality enhanced picture.

12. A terminal image quality enhancement device, including a memory; and a processor connected to the memory, the memory is used to store instructions, and the processor is configured to execute the steps of the terminal image quality enhancement method according to any one of claims 1 to 11 based on the instructions stored in the memory.

13. A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the terminal image quality enhancement method according to any one of claims 1 to 11.

Images