Terminal image quality enhancement method and apparatus, computer-readable storage medium

Abstract

A terminal image quality enhancement method and device, and a computer readable storage medium, the terminal image quality enhancement method comprising: obtaining an image quality enhancement parameter of a user; determining a number of cycles J of using an image quality enhancement model for image quality enhancement according to the image quality enhancement parameter of the user; receiving an image to be enhanced; and using the image quality enhancement model to perform image quality enhancement on the image to be enhanced J times, wherein an input image of the image quality enhancement model in the first cycle is the image to be enhanced, an input image of the image quality enhancement model in the jth cycle is an output image of the image quality enhancement model in the (j-1)th cycle, internal parameters of the image quality enhancement model used in the J cycles are the same, and J≥j>1.

Description

TECHNICAL FIELD

[0001] The embodiment of the present disclosure relates to, but is not limited to, the technical field of image quality enhancement, and in particular to a terminal image quality enhancement method and device and a computer readable storage medium. BACKGROUND

[0002] Image quality refers to picture quality. In actual playing scenarios, the image quality of a terminal such as a television often needs to be improved. For example, when playing a standard definition video on a 2K resolution television or playing a standard definition or 2K resolution video on a 4K resolution television, the display effect is poor due to mismatch between the resolution of the video source and the screen display resolution, such as blurred and jagged pictures. However, due to the limited computing power and storage space of the terminal such as a television, the current terminal image quality enhancement method often cannot meet the lightweight requirements of actual scenarios. SUMMARY

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiment of the present disclosure provides a terminal image quality enhancement method, comprising:

[0005] obtaining an image quality enhancement parameter of a user;

[0006] determining, according to the image quality enhancement parameter of the user, a number of cycles J of using an image quality enhancement model for image quality enhancement;

[0007] receiving an image to be enhanced, and using the image quality enhancement model to perform image quality enhancement on the image to be enhanced for J times, wherein an input image of the image quality enhancement model in the first cycle is the image to be enhanced, an input image of the image quality enhancement model in the jth cycle is an output image of the image quality enhancement model in the (j-1)th cycle, the internal parameters of the image quality enhancement model used in the J cycles are the same, and J≥j>1.

[0008] The embodiment of the present disclosure also provides a terminal image quality enhancement device, comprising a memory and a processor connected to the memory, the memory is used to store instructions, and the processor is configured to execute the steps of the terminal image quality enhancement method according to any one of the embodiments of the present disclosure based on the instructions stored in the memory.

[0009] The embodiment of the present disclosure also provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the terminal image quality enhancement method according to any one of the embodiments of the present disclosure.

[0010] Other aspects can become apparent after reading the following detailed description and understanding the attached drawings. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are used to provide a further understanding of the technical solutions of the present disclosure, and constitute a part of the specification, and are used to explain the technical solutions of the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions of the present disclosure. The shapes and sizes of the components in the drawings do not reflect true proportions, and the purpose is only to schematically illustrate the present disclosure.

[0012] Figure 1 A flowchart of a terminal image quality enhancement method provided for an exemplary embodiment of the present disclosure is shown in FIG. 6.

[0013] Figure 2 A scheme diagram of performing image super-resolution (SR) model-based image quality enhancement provided for an exemplary embodiment of the present disclosure is shown in FIG. 7.

[0014] Figure 3 A structure diagram of an image super-resolution model provided for an exemplary embodiment of the present disclosure is shown in FIG. 8.

[0015] Figure 4 A scheme diagram of performing sharpness enhancement (BP) model-based image quality enhancement provided for an exemplary embodiment of the present disclosure is shown in FIG. 9.

[0016] Figure 5 A structure diagram of a sharpness enhancement model provided for an exemplary embodiment of the present disclosure is shown in FIG. 10.

[0017] Figure 6 A training method diagram of an image super-resolution model provided for an exemplary embodiment of the present disclosure is shown in FIG. 11.

[0018] Figure 7 A process diagram of first performing Float32 precision training and then using Int8 quantization training provided for an exemplary embodiment of the present disclosure is shown in FIG. 12.

[0019] Figure 8 A training method diagram of a sharpness enhancement model provided for an exemplary embodiment of the present disclosure is shown in FIG. 13.

[0020] Figure 9 A television terminal image quality enhancement module provided for an exemplary embodiment of the present disclosure is shown in FIG. 14.

[0021] Figure 10 A terminal image quality enhancement module front-end parameter setting method provided for an exemplary embodiment of the present disclosure is shown in FIG. 15.

[0022] Figure 11 A structure diagram of a terminal image quality enhancement device provided for an exemplary embodiment of the present disclosure is shown in FIG. 16. DETAILED DESCRIPTION

[0023] In order to make the objects, technical solutions and advantages of the present disclosure clearer, the embodiments of the present disclosure will be described in detail below with reference to the drawings. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other as long as there is no conflict.

[0024] Unless otherwise defined, technical terms or scientific terms used in the embodiments of the present disclosure shall have the common meaning understood by one of ordinary skill in the art to which the present disclosure belongs. The terms "first", "second", and similar terms used in the embodiments of the present disclosure do not denote any order, quantity, or importance, but are used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects.

[0025] As shown in Figure 1 The embodiments of the present disclosure provide a terminal image quality enhancement method, comprising the following steps:

[0026] Step 101, obtaining an image quality enhancement parameter of a user;

[0027] Step 102, determining a cycle number J of using an image quality enhancement model for image quality enhancement according to the image quality enhancement parameter of the user;

[0028] Step 103, receiving an image to be enhanced, and cyclically using the image quality enhancement model to perform image quality enhancement on the image to be enhanced for J times, wherein the input image of the image quality enhancement model in the first cycle is the image to be enhanced, the input image of the image quality enhancement model in the jth cycle is the output image of the image quality enhancement model in the (j-1)th cycle, the internal parameters of the image quality enhancement model used in the J cycles are the same, and J≥j>1.

[0029] The terminal image quality enhancement method of the embodiments of the present disclosure determines the cycle number J of using the image quality enhancement model for image quality enhancement according to the image quality enhancement parameter of the user, and then cyclically uses the image quality enhancement model to perform image quality enhancement on the image to be enhanced for J times and the internal parameters of the image quality enhancement model used in the J cycles are the same. When the model is initialized, the memory occupied by the model is only the memory occupied by a single model, which can greatly reduce the memory consumption of the hardware, make the model lightweight, and meet the low computing power requirement of the terminal.

[0030] In some example embodiments, as shown in Figure 2 The image quality enhancement model can be an image super resolution (SR) model, and the image quality enhancement parameter is a first image quality enhancement parameter s, where J=s, x=2 s, x is the resolution enhancement ratio between the output image of the image super-resolution model in the Jth cycle and the image to be enhanced. In the embodiments of the present disclosure, the first image quality enhancement parameter s can be understood as the number of times of 2x super-resolution cycles. For example, no super-resolution (s=0), 2x super-resolution (s=1), 4x super-resolution (s=2), 8x super-resolution (s=3), 16x super-resolution (s=4), 32x super-resolution (s=5), etc. can be selected. Figure 2 The SR in the above formula can be regarded as a 2x super-resolution module, and the internal parameters of the image super-resolution model used in the multiple cycles are the same (here, the internal parameters of the image super-resolution model refer to the optimal parameters obtained through training samples in model training).

[0031] In some example embodiments, as shown in Figure 3 The image super-resolution model includes a first feature extraction layer FE1, a first clipping layer Clip1, m serial spatial attention layers SA, a channel adjustment layer CC, a first upsampling layer Piexl Shuffle1, and a second clipping layer Clip2 connected in sequence, and m is a natural number between 2 and 5, wherein:

[0032] The first feature extraction layer FE1 is configured to extract a feature map of an input image of the image super-resolution model;

[0033] The first clipping layer Clip1 is configured to limit the numerical value of the feature map output by the first feature extraction layer FE1 within a preset range;

[0034] The m serial spatial attention layers SA are configured to perform feature screening and enhancement on the feature map output by the first clipping layer Clip1;

[0035] The channel adjustment layer CC is configured to adjust the number of channels of the feature map;

[0036] The first upsampling layer Piexl Shuffle1 is configured to convert the feature map output by the channel adjustment layer CC into a high-resolution image;

[0037] The second clipping layer Clip2 is configured to limit the numerical value of the high-resolution image output by the first upsampling layer Piexl Shuffle1 within a preset range.

[0038] In actual use, m can be set according to the hardware conditions of the terminal. For example, m can be set to 2 or 3.

[0039] In some example embodiments, as shown in Figure 3 Each spatial attention layer SA includes a second feature extraction layer FE2, a third clipping layer Clip3, a weighted feature mapping layer WFM, a third feature extraction layer FE3, and a fourth clipping layer Clip4, wherein:

[0040] The second feature extraction layer FE2 is configured to refine the feature map input to the current spatial attention layer SA.

[0041] The third truncation layer Clip3 is configured to limit the value of the feature map output by the second feature extraction layer FE2 to a preset range;

[0042] The weighted feature map layer (WFM) is configured to compute spatial attention weights and multiply the computed spatial attention weights by the feature map output by the third truncation layer (Clip3) to obtain the weighted feature map.

[0043] The third feature extraction layer, FE3, is configured to refine the features of the weighted feature map.

[0044] The fourth truncation layer, Clip4, is configured to limit the values ​​of the feature maps output by the third feature extraction layer, FE3, to a preset range.

[0045] In some exemplary implementations, such as Figure 3 As shown, the weighted feature map layer WFM can include a first branch, a second branch, and a multiplier. Both the first and second branches are set between the third truncated layer Clip3 and the multiplier. The first branch includes a first convolutional layer Conv, a seventh truncated layer Clip7, and an activation function layer Sigmoid, which are connected in series.

[0046] In some exemplary embodiments, the first convolutional layer Conv can be a convolutional layer with a kernel size of 1*1 or 3*3; however, this disclosure does not limit this. Using a convolutional layer with a kernel size of 1*1 in the first convolutional layer Conv can further reduce the computing power requirements of the terminal.

[0047] In some exemplary embodiments, the first feature extraction layer FE1 may be a convolutional layer with a kernel size of 3*3; however, this disclosure does not limit this.

[0048] In some exemplary embodiments, the second feature extraction layer FE2 can be a convolutional layer with a kernel size of 3*3; however, this disclosure does not limit this.

[0049] In some exemplary embodiments, the third feature extraction layer FE3 can be a convolutional layer with a kernel size of 1*1 or 3*3; however, this disclosure does not limit this. Using a convolutional layer with a kernel size of 1*1 for the third feature extraction layer FE3 can further reduce the computing power requirements of the terminal.

[0050] In some exemplary embodiments, the channel adjustment layer CC can be a convolutional layer with a kernel size of 1*1; however, this disclosure does not limit this.

[0051] In some exemplary embodiments, the clipping layer (which can be any one of the first clipping layer Clip1, the second clipping layer Clip2, the third clipping layer Clip3, the third clipping layer Clip4, and the seventh clipping layer Clip7, or any one of the fifth clipping layer Clip5 and the sixth clipping layer Clip6 described later) serves to truncate the model. The clipping value can be between 0 and 1 (when the model output value is a normalized value) or between 0 and 255 (when the model output value is not normalized), as expressed by the following formula:

[0052]

[0053] or;

[0054]

[0055] The purpose of the clipping layer is to ensure that the output value of each convolutional function is between 0 and 1 (or 0 and 255), which is beneficial for subsequent model quantization training and reduces quantization loss. The image super-resolution model of this disclosure uses a clipping layer after each convolutional layer, which helps to maintain the model's accuracy and performance to the greatest extent during Int8 quantization training.

[0056] In some exemplary embodiments, the image quality enhancement model can be a sharpness enhancement model, and the image quality enhancement parameter is a second image quality enhancement parameter w, where 0 < w ≤ 100%.

[0057] like Figure 4 As shown, BP can be considered as a single sharpness enhancement module. The internal parameters of the sharpness enhancement model used in multiple iterations (here, the internal parameters of the sharpness enhancement model refer to the optimal parameters obtained through training samples during model training) are the same. In this embodiment, the sharpness enhancement model is a back projection structure, containing a downsampling layer (Piexl UnShuffle or others) and an upsampling layer (Piexl Shuffle or others). The input to the BP structure is an image, and the output is also an image. Its structure is a residual structure, meaning that the sharpness enhancement module essentially obtains the residual between the enhanced image and the input image, i.e., the enhanced details. The second image quality enhancement parameter w is the intensity coefficient of the current user's adjustment of the enhancement effect, and the value of w is between 0 and 100%.

[0058] In some exemplary embodiments, determining the number of loops J for image enhancement using the image enhancement model based on the user's image enhancement parameters includes:

[0059] determining a maximum number of cycles I and dividing 0-100% into I equal parts, the i-th equal part corresponding to an enhancement percentage of A*100%, where (i-1) / I

[0060] The equal part corresponding to the second quality enhancement parameter w is the number of cycles J of quality enhancement using the sharpness enhancement model.

[0061] In actual use of the sharpness enhancement model of the embodiments of the present disclosure for quality enhancement, the user can control the enhancement effect by adjusting the second quality enhancement parameter w. For example, assuming that the maximum number of cycles I = 5, the enhancement degree can be divided into the following 5 levels (the number of levels is the maximum number of cycles):

[0062] 0% - 20% - 40% - 60% - 80% - 100%

[0063] When the user adjusts the sliding bar of the enhancement effect, if the second quality enhancement parameter w is 0%, no enhancement is performed, i.e., no quality enhancement using the sharpness enhancement model is performed.

[0064] When the second quality enhancement parameter w is between (0%, 20%], quality enhancement is performed once using the sharpness enhancement model: Pout = BP 1 (P) = BP(P).

[0065] When the second quality enhancement parameter w is between (20%, 40%], quality enhancement is performed twice using the sharpness enhancement model in cycles: Pout = BP 2 (P) = BP(BP 1 (P)).

[0066] When the second quality enhancement parameter w is between (40%, 60%], quality enhancement is performed three times using the sharpness enhancement model in cycles (which can be set as a default configuration): Pout = BP 3 (P) = BP(BP 2 (P)).

[0067] When the second quality enhancement parameter w is between (60%, 80%], quality enhancement is performed four times using the sharpness enhancement model in cycles: Pout = BP 4 (P) = BP(BP 3 (P)).

[0068] When the second quality enhancement parameter w is between (80%, 100%], quality enhancement is performed five times using the sharpness enhancement model in cycles: Pout = BP 5 (P) = BP(BP 4 (P)).

[0069] In some example embodiments, when the second quality enhancement parameter w is not equal to an integer multiple of 1 / I, the terminal quality enhancement method further comprises:

[0070] The difference coefficient w' is calculated according to the following formula: w' = w*I-J+1;

[0071] The output image is determined as BP J-1 (P) + w'*(BP J (P) - BP J-1 (P). J-1 (P) is the output image of the definition enhancement model at the J-1th cycle, BP J (P) is the output image of the definition enhancement model at the Jth cycle.

[0072] In the embodiments of the present disclosure, in order to enable the user to obtain a continuous definition change process between each gear, when the second quality enhancement parameter w falls into a certain gear, for example, assuming that the user adjusts the second quality enhancement parameter w = 35%, which falls into the (20%, 40%] gear, at this time, the definition enhancement model is used for quality enhancement 2 times, and then 35% is linearly mapped to 0-1 in the (20%, 40%] interval w' = (35%-20%) / (40%-20%) = 0.75 (i.e. w' = w*I-J+1 = 35%*5-2+1 = 0.75), at this time, the enhanced picture is Pout = BP 1 (P) + 0.75*(BP 2 (P) - BP 1 (P).

[0073] For another example, assuming that the user adjusts the quality enhancement parameter w = 65%, at this time, it falls into the (60%, 80%] gear, at this time, the definition enhancement model is used for quality enhancement 4 times, according to the above calculation, w' = (65%-60%) / (80%-60%) = 0.25 (i.e. w' = w*I-J+1 = 65%*5-4+1 = 0.25), at this time, the enhanced picture is Pout = BP 3 (P) + 0.25*(BP 4 (P) - BP 3 (P).

[0074] In this way, the user can obtain a continuous enhancement effect by sliding the bar. When used, the cycle can be configured to 3 times by default, and the program supports setting the cycle to 4 times or 5 times, which can ensure that when the original video picture to be enhanced is particularly blurred, more definition enhancement is needed to achieve a more ideal effect, thereby providing the user with a wider space for enhancement effect.

[0075] In some example embodiments, the maximum number of iterations I is determined, comprising:

[0076] detecting a computing power space M that the current terminal can provide to the picture quality enhancement algorithm;

[0077] determining whether M is greater than or equal to I2*M bp , where I2 is the maximum number of iterations that the terminal supports for the iterative use of the definition enhancement model, and M bp is the computing power space required for one picture quality enhancement of the definition enhancement model to the image to be enhanced;

[0078] when M is greater than or equal to I2*M bp , the maximum number of iterations I is set to I2;

[0079] when M is less than I2*M bp , the maximum number of iterations I is set to , where represents the floor of M / M bp .

[0080] In an ideal case, we believe that the computing power of a device such as a television or a mobile terminal can stably and reliably support picture quality enhancement for I2 times (for example, I2 = 5) using the definition enhancement model iteratively, but when the television starts many other modules at the same time or the mobile terminal runs many other applications in the background at the same time, there may be occasional computing power occupation that cannot provide enough computing power for the definition enhancement model. At this time, the terminal picture quality enhancement method of the embodiments of the present disclosure can realize online real-time adjustment of the number of times of picture quality enhancement using the definition enhancement model to complete the enhancement processing, as follows:

[0081] Suppose that the computing power required for one picture quality enhancement using the definition enhancement model is M bp , and the computing power space that the terminal device can provide to the picture quality enhancement algorithm is M obtained through online detection. When M ≥ I2*M bp , the maximum number of iterations I is set to I2, and then the number of iterations is determined according to the foregoing logic to perform enhancement. When M < I2*M bp , the maximum number of iterations that can be run is recalculated as , where represents the floor of M / M bp . According to the maximum number of iterations recalculated, 0-100% is divided into I equal parts, and then the number of iterations is determined according to the foregoing logic to perform enhancement. According to this method, the maximum number of iterations I is reset, which can adaptively control the number of times of iterative enhancement of the definition enhancement model to the image to be enhanced in real time according to the hardware performance.

[0082] For example, when I = 5, the enhancement degree can be divided into the following 5 levels:

[0083] 0% - 20% - 40% - 60% - 80% - 100%;

[0084] When I = 4, the enhancement degree can be divided into the following 4 levels:

[0085] 0% - 25% - 50% - 75% - 100%;

[0086] When I = 3, the enhancement degree can be divided into the following 3 levels:

[0087] 0% - 33% - 66% - 100%;

[0088] When I = 2, the enhancement degree can be divided into the following 2 levels:

[0089] 0% - 50% - 100%;

[0090] When I = 1, the enhancement degree can be divided into the following 1 level:

[0091] When there is 1 BP, the region is divided into: 0% - 100%.

[0092] The definition enhancement model of the embodiment of the present disclosure is a back projection structure, which is a residual structure as shown in Figure 5 , and includes a down-sampling layer (Piexl UnShuffle or other) and an up-sampling layer (Piexl Shuffle or other).

[0093] In some example embodiments, as shown in Figure 5 , the definition enhancement model includes a first residual block RB1, the first residual block RB1 includes a first main branch, a first shortcut branch, and a first adder, the first main branch and the first shortcut branch are both arranged between an input image and the first adder, and the first residual block RB1 is configured to output the input image after summing the input image with the input image processed by the first main branch; the first main branch includes a down-sampling layer Piexl UnShuffle, a second residual block RB2, a fifth clipping layer Clip5, and a second up-sampling layer Piexl Shuffle2 connected in sequence, wherein:

[0094] The down-sampling layer Piexl UnShuffle is configured to down-sample the input image to obtain a down-sampled image;

[0095] The second residual block RB2 includes a second main branch, a second shortcut branch, and a second adder, the second main branch and the second shortcut branch are both arranged between the down-sampling layer Pixel UnShuffle and the second adder, the second main branch includes a plurality of second convolutional layers Conv' and a sixth clipping layer Clip6 arranged between the plurality of second convolutional layers Conv';

[0096] The fifth clipping layer Clip5 is configured to limit the values of the feature map output by the second residual block RB2 within a preset range;

[0097] The second up-sampling layer Pixel Shuffle2 is configured to convert the feature map output by the fifth clipping layer Clip5 into a high-resolution image.

[0098] In some example embodiments, the second convolutional layer Conv' can be a convolutional layer with a convolution kernel size of 3*3, however, the present disclosure is not limited thereto.

[0099] In some example embodiments, the number of the second convolutional layers Conv' in the second main branch can be between 1 and 3. For example, the number of the second convolutional layers Conv' in the second main branch can be 2.

[0100] In the present disclosure, the clipping layer Clip (any one of the fifth clipping layer Clip5 and the sixth clipping layer Clip6) can ensure that the values output by each convolutional function are between 0-1 (or 0-255), which is beneficial for subsequent model quantization training and reduces quantization loss. The clarity enhancement model of the present disclosure adopts a clipping layer Clip after each convolutional layer, which is beneficial for maintaining the accuracy performance of the model to the greatest extent during Int8 quantization training.

[0101] In some example embodiments, the training process of the image quality enhancement model includes:

[0102] The C image quality enhancement models with the same structure and internal parameters are used for training, and the loss of using 1 to C image quality enhancement models is calculated at the same time, and floating-point precision is used during training;

[0103] The trained floating-point image quality enhancement model is quantized into an integer image quality enhancement model.

[0104] In some example embodiments, the training process of the image super-resolution model includes:

[0105] using y serially connected image super-resolution models with the same structure and the same internal parameters, 1 < y < I1, I1 being the maximum number of cycles of the terminal supporting the cyclic use of the image super-resolution model, and simultaneously calculating the loss of using 1 to y image super-resolution models, and using floating-point precision during training;

[0106] quantizing the trained floating-point image super-resolution model into an integer image super-resolution model.

[0107] In the embodiments of the present disclosure, training the image super-resolution model includes two steps:

[0108] 1) floating-point training: as shown in Figure 6 When training the image super-resolution model in floating-point, the internal parameters of the plurality of image super-resolution models are shared (assuming y = 3, i.e., the above three image super-resolution models are the same model with the same set of internal parameters, and the image super-resolution model is cycled three times), and 2x / 4x / 8x super-resolution is trained simultaneously and the loss of 2x / 4x / 8x is calculated simultaneously. The loss function can use the absolute loss function (L1 Loss) or the mean square error (MSE) loss function. During training, the model can be trained using Float32 precision.

[0109] 2) as shown in Figure 7 After completing the floating-point training, the Float32 precision model is loaded for quantization-aware training (QAT), which can greatly reduce the inference cost while maintaining the quality and accuracy of the model. The model trained in this step can be trained using Int8 precision.

[0110] In some example embodiments, the training process of the sharpness enhancement model includes:

[0111] using z serially connected sharpness enhancement models with the same structure and the same internal parameters, 1 < z < I2, I2 being the maximum number of cycles of the terminal supporting the cyclic use of the sharpness enhancement model, and simultaneously calculating the loss of using 1 to z sharpness enhancement models, and using floating-point precision during training;

[0112] quantizing the trained sharpness enhancement model into an integer sharpness enhancement model.

[0113] In the embodiments of the present disclosure, training the sharpness enhancement model includes two steps:

[0114] 1) floating-point training: as shown in Figure 8As shown, when training the floating-point clarity enhancement model, the internal parameters of multiple clarity enhancement models are shared (assuming z = 3, i.e. the above three clarity enhancement models are the same model with the same set of internal parameters, and the clarity enhancement model is trained three times), and the 1st / 2nd / 3rd enhancement is trained and the loss of the 1st / 2nd / 3rd enhancement is calculated at the same time. The loss function can use the absolute loss function or the mean square error loss function. During training, the model can be trained with Float32 precision.

[0115] 2) as shown in Figure 7 After completing the floating-point training, the Float32 precision model is loaded for perceptual quantization training, which can greatly reduce the inference cost while maintaining the quality and accuracy of the model. The model trained in this step can be trained with Int8 precision.

[0116] In some example embodiments, the image quality enhancement model includes an image super-resolution model and a clarity enhancement model, and the image quality enhancement parameter includes a first image quality enhancement parameter s and a second image quality enhancement parameter w.

[0117] According to the image quality enhancement parameter of the user, the number of cycles J of using the image quality enhancement model for image quality enhancement is determined, including: determining the number of cycles J1 of using the image super-resolution model for image quality enhancement; determining the maximum number of cycles I of using the clarity enhancement model for image quality enhancement, and dividing 0 to 100% into I equal parts, the enhancement percentage corresponding to the i-th equal part is A*100%, where (i-1) / I < A ≤ i / I, i is a natural number between 1 and I; the equal part corresponding to the second image quality enhancement parameter w is the number of cycles J2 of using the clarity enhancement model for image quality enhancement.

[0118] The image quality enhancement model is used to enhance the image quality of the to-be-enhanced image J times, including: first, the clarity enhancement model is used to enhance the image quality of the to-be-enhanced image J2 times, and a first image quality enhanced picture is output; then, the image super-resolution model is used to enhance the image quality of the first image quality enhanced picture J1 times, and a second image quality enhanced picture is output.

[0119] The structure, use method and training method of the image super-resolution model and the clarity enhancement model can be referred to the foregoing description, and will not be described here again.

[0120] The terminal image quality enhancement method of the embodiments of the present disclosure can be applied to terminals such as televisions, all-in-one machines and mobile terminals. Taking a television terminal as an example, as shown in Figure 9 The image quality enhancement module is provided in the television terminal, which is divided into two parts, a super-resolution (i.e. image super-resolution) module and a clarity enhancement module: the above modules are started after the image quality enhancement module is turned on.

[0121] AsFigure 10 As shown, the super-resolution module can select 1x (s=0), 2x (s=1), 4x (s=2), and 8x (s=3) for resolution enhancement through the first image quality enhancement parameter s, where 1x represents not performing super-resolution operation.

[0122] The sharpness enhancement module is set to a slider mode, and the second image quality enhancement parameter w is set between [0, 100%]. For specific rules, please refer to the loop number setting logic of the aforementioned sharpness enhancement model.

[0123] Once the settings are complete, the first image quality enhancement parameter s is passed to the super-resolution module, and the second image quality enhancement parameter w is passed to the sharpness enhancement module, and the image with enhanced image quality can be obtained.

[0124] This disclosure also provides a terminal image quality enhancement device, including a memory; and a processor connected to the memory, the memory being used to store instructions, the processor being configured to execute the steps of the terminal image quality enhancement method as described in any embodiment of this disclosure based on the instructions stored in the memory.

[0125] like Figure 11 As shown, in one example, the terminal image enhancement device may include: a processor 1110, a memory 1120, a bus system 1130, and a transceiver 1140, wherein the processor 1110, the memory 1120, and the transceiver 1140 are connected via the bus system 1130, the memory 1120 is used to store instructions and an image enhancement model, and the processor 1110 is used to execute the instructions stored in the memory 1120 to control the transceiver 1140 to send and receive signals. Specifically, transceiver 1140 can acquire the user's image enhancement parameters and receive the image to be enhanced under the control of processor 1110. Processor 1110 determines the number of loops J for using the image enhancement model to enhance the image quality based on the user's image enhancement parameters. The image enhancement model is used to enhance the image quality of the image to be enhanced J times in a loop. In the first loop, the input image of the image enhancement model is the image to be enhanced. In the j-th loop, the input image of the image enhancement model is the output image of the image enhancement model in the (j-1)-th loop. The internal parameters of the image enhancement model used in the J loops are the same, J≥j>1. The obtained output image is output to the terminal's display interface through transceiver 1140.

[0126] It is understood that the processor 1110 can be a central processing unit (CPU), and the processor 1110 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), programmable logic devices (PLD), discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0127] The memory 1120 can include a read-only memory and a random access memory, and provide instructions and data for the processor 1110. A part of the memory 1120 can also include a non-volatile random access memory. For example, the memory 1120 can also store device type information.

[0128] The bus system 1130 can include not only a data bus, but also a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, all the buses are marked as the bus system 1130 in the Figure 11

[0129] In the implementation process, the processing performed by the processing device can be completed by the integrated logic circuit of the hardware in the processor 1110 or the instructions in the form of software. That is, the method steps of the embodiments of the present disclosure can be embodied as being completed by a hardware processor or being completed by a combination of hardware and software modules in the processor. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, or the like storage medium. The storage medium is located in the memory 1120, and the processor 1110 reads the information in the memory 1120 and completes the steps of the above method in combination with the hardware. To avoid repetition, it will not be described in detail here.

[0130] The embodiments of the present disclosure also provide a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the terminal image quality enhancement method according to any of the embodiments of the present disclosure. The method of driving prognosis analysis by executing executable instructions is basically the same as the terminal image quality enhancement method provided by the above embodiments of the present disclosure, and will not be described here.

[0131] In some possible implementation manners, each aspect of the terminal image quality enhancement method provided by the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to perform the steps of the terminal image quality enhancement method according to various exemplary embodiments of the present application described above in the specification when the program product runs on the computer device, for example, the computer device can execute the terminal image quality enhancement method described in the embodiments of the present application. ​

[0132] The program product can employ any combination of one or more computer readable media. The computer readable media can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] Those of ordinary skill in the art will appreciate that all or certain steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division of the functional modules / units referred to in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Certain components or all components can be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Further, it should be appreciated by those of ordinary skill in the art that computer readable media generally include computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The program product can be implemented in a variety of ways. For example, the program product can be implemented as a method, apparatus, or article of manufacture (e.g., computer program product) including a computer readable medium. Each module of the program product can be implemented as software, firmware, hardware or any suitable combination thereof. The computer readable medium can be a computer readable storage medium or a computer readable signal medium.

[0134] Although the embodiments disclosed by the present disclosure are as above, the content described is only the embodiments adopted for the convenience of understanding the present disclosure, and is not intended to limit the present disclosure. Any person skilled in the art can make any modification and change in the form and details without departing from the spirit and scope of the present disclosure, but the patent protection scope of the present disclosure shall be subject to the scope defined by the appended claims.

Claims

1. A terminal image quality enhancement method, characterized by, The method comprises: obtaining a user's image quality enhancement parameter; determining the number of cycles J of using an image quality enhancement model for image quality enhancement according to the user's image quality enhancement parameter; receiving an image to be enhanced, and using the image quality enhancement model to enhance the image to be enhanced for J times, wherein the input image of the image quality enhancement model in the first cycle is the image to be enhanced, the input image of the image quality enhancement model in the jth cycle is the output image of the image quality enhancement model in the (j-1)th cycle, the internal parameters of the image quality enhancement model used in the J cycles are the same, and J > j > 1; wherein the image quality enhancement model is a definition enhancement model, and the image quality enhancement parameter is a second image quality enhancement parameter w, wherein 0 < w ≤ 100%; determining the number of cycles J of using an image quality enhancement model for image quality enhancement according to the user's image quality enhancement parameter comprises: determining the maximum number of cycles I of using the definition enhancement model for image quality enhancement supported by the terminal, and dividing 0 to 100% into I equal parts, wherein the enhancement percentage corresponding to the ith part is A*100%, wherein (i-1) / I < A ≤ i / I, and i is a natural number between 1 and I; determining the number of cycles J of using the definition enhancement model for image quality enhancement corresponding to the equal part of the second image quality enhancement parameter w.

2. The method of claim 1, wherein The image quality enhancement model is an image super-resolution model, and the image quality enhancement parameter is a first image quality enhancement parameter s, wherein J = s, x is a resolution enhancement ratio between an output image of the image super-resolution model in the Jth cycle and the image to be enhanced.

3. The method of claim 2, wherein The image super-resolution model comprises a first feature extraction layer, a first truncation layer, m serial spatial attention layers, a channel adjustment layer, a first up-sampling layer, and a second truncation layer connected in sequence, and m is a natural number between 2 and 5, wherein: the first feature extraction layer is configured to extract a feature map of an input image of the image super-resolution model; the first truncation layer is configured to limit the numerical value of the feature map output by the first feature extraction layer within a preset range; the m serial spatial attention layers are configured to perform feature screening and enhancement on the feature map output by the first truncation layer; the channel adjustment layer is configured to adjust the channel number of the feature map; the first up-sampling layer is configured to convert the feature map output by the channel adjustment layer into a high-resolution image; the second truncation layer is configured to limit the numerical value of the high-resolution image output by the first up-sampling layer within a preset range.

4. The method of claim 3, wherein the terminal image quality enhancement method is characterized by, Each spatial attention layer comprises a second feature extraction layer, a third truncation layer, a weighted feature mapping layer, a third feature extraction layer, and a fourth truncation layer, wherein: the second feature extraction layer is configured to perform feature refinement on the feature map input by the current spatial attention layer; the third truncation layer is configured to limit the numerical value of the feature map output by the second feature extraction layer within a preset range; the weighted feature mapping layer is configured to calculate a spatial attention weight, and multiply the calculated spatial attention weight with the feature map output by the third truncation layer to obtain a weighted feature map; the third feature extraction layer is configured to perform feature refinement on the weighted feature map; the fourth truncation layer is configured to limit the numerical value of the feature map output by the third feature extraction layer within a preset range.

5. The method of claim 2, wherein the terminal image quality enhancement method is characterized by, The training process of the image super-resolution model comprises: training using y image super-resolution models with the same structure and the same internal parameters in series, 1 quantizing the trained floating-point image super-resolution model into an integer image super-resolution model.

6. The method of claim 5, wherein the terminal image quality enhancement method is characterized by, The determination of the maximum cycle number I comprises: detecting the computing power space M that the current terminal can provide to the image quality enhancement algorithm; determining whether M is greater than or equal to I2 wherein I2 is a maximum number of cycles that the terminal supports in a cycle usage resolution enhancement model in an ideal case, a computing resource space required for performing one time image quality enhancement on the image to be enhanced using the resolution enhancement model. When M is greater than or equal to I2 I = I2; When M is less than I2 , the maximum number of iterations I is set to , where represents the floor of the value of .

7. The method of claim 5, wherein the terminal image quality enhancement method is characterized by, when w is not an integer multiple of 1 / I, the method further comprises: The difference coefficient is calculated according to the following formula : ; determining an output image as (P) + (P) (P) - (P) (P) - (P) (P) + (P), wherein (P) is an output image of the definition enhancement model at the J-1th cycle, (P) is an output image of the definition enhancement model at the Jth cycle.

8. The method of claim 5, wherein the terminal image quality enhancement method is characterized by, The definition of the image quality enhancement model comprises a first residual block, the first residual block comprises a first main branch, a first shortcut branch and a first adder, the first main branch and the first shortcut branch are both arranged between an input image and the first adder, and the first residual block is configured to output the sum of the input image and the input image processed by the main branch; the first main branch comprises a down-sampling layer, a second residual block, a fifth truncation layer and a second up-sampling layer connected in series, wherein: The down-sampling layer is configured to down-sample the input image to obtain a down-sampled image; The second residual block comprises a second main branch, a second shortcut branch and a second adder, the second main branch and the second shortcut branch are both arranged between the down-sampling layer and the second adder, and the second main branch comprises a plurality of second convolution layers and a sixth truncation layer arranged between the plurality of second convolution layers; The fifth truncation layer is configured to limit the values of the feature maps output by the second residual block within a preset range; The second up-sampling layer is configured to convert the feature maps output by the fifth truncation layer into a high-resolution image.

9. The method of claim 5, wherein the terminal image quality enhancement method is characterized by, The training process of the definition enhancement model comprises: training using z definition enhancement models with the same structure and the same internal parameters in series, 1 quantizing the trained definition enhancement model into an integer definition enhancement model.

10. The method of claim 1, wherein the terminal image quality enhancement method is characterized by, The image quality enhancement model comprises an image super-resolution model and a definition enhancement model, and the image quality enhancement parameter comprises a first image quality enhancement parameter s and a second image quality enhancement parameter w; According to the image quality enhancement parameter of the user, the cycle number J of using the image quality enhancement model for image quality enhancement is determined, comprising: determining the cycle number J1=s of using the image super-resolution model for image quality enhancement; determining the maximum cycle number I of using the definition enhancement model for image quality enhancement, and dividing 0 to 100% into I equal parts, and the enhancement percentage corresponding to the i-th equal part is A*100%, wherein (i-1) / I <A ≤i / I, i is a natural number between 1 and I; the equal part corresponding to the second image quality enhancement parameter w is the cycle number J2 of using the definition enhancement model for image quality enhancement. The picture quality enhancement model is cyclically used for picture quality enhancement of the to-be-enhanced image for J times, including: the definition enhancement model is cyclically used for picture quality enhancement of the to-be-enhanced image for J2 times, and a first picture quality enhanced picture is output; and the image super-resolution model is cyclically used for picture quality enhancement of the first picture quality enhanced picture for J1 times, and a second picture quality enhanced picture is output.

11. A terminal image quality enhancement apparatus, characterized by comprising: A terminal picture quality enhancement method includes a memory; and a processor connected to the memory, the memory being configured to store instructions, and the processor being configured to execute steps of the terminal picture quality enhancement method according to any one of claims 1 to 10 based on the instructions stored in the memory.

12. A computer-readable storage medium, characterized in that, A computer program is stored on the computer program, and the program is executed by the processor to implement the terminal picture quality enhancement method according to any one of claims 1 to 10.

Images