Method and device for enhancing image quality, equipment and medium

Abstract

The invention provides a method, a device, equipment and a medium for enhancing image quality. In one method, an image sequence is acquired, the head and the tail of the image sequence respectively comprising a head image frame and a tail image frame, and other locations in the image sequence other than the head and the tail respectively comprising a set of intermediate image frames. For an intermediate image frame of the set of intermediate image frames, a mapping frame of the intermediate image frame is determined from the optical flow estimate based on at least any one of the head image frame and the tail image frame. The intermediate image frame is adjusted with the mapping frame to determine an enhanced image frame of the intermediate image frame. With exemplary implementations of the present disclosure, image quality of an intermediate image frame at an intermediate position may be enhanced using a tail image frame and / or data with higher image quality in the tail image frame based on an optical flow estimation technique.

Description

Technical Field

[0001] Exemplary implementations of the present disclosure relate generally to image processing, and more particularly to methods, devices, apparatuses, and computer-readable storage media for enhancing image quality in variable resolution decoding. Background Art

[0002] With the development of video processing technology, variable resolution encoding and decoding technology has been proposed. In the encoding process, the original resolution of some frames in the original video can be retained at predetermined intervals, and other frames can be compressed. In the decoding process, these image frames with original resolution can be used to restore the compressed video frames so that the decoded video can be better restored to the quality of the original video. However, the encoding and decoding process will damage the quality of the original video, and may cause the restored video to not present the original quality, but to have problems such as blur. At this time, how to perform the video processing process in a more effective way to enhance the image quality in the video processing process. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for enhancing image quality is provided. In the method, an image sequence is acquired, the head and the tail of the image sequence respectively include a head image frame and a tail image frame, and other positions other than the head and the tail in the image sequence respectively include a group of intermediate image frames. For an intermediate image frame in a group of intermediate image frames, a mapping frame of the intermediate image frame is determined according to optical flow estimation based on at least any one of the head image frame and the tail image frame. The intermediate image frame is adjusted using the mapping frame to determine an enhanced image frame of the intermediate image frame.

[0004] In a second aspect of the present disclosure, a device for enhancing image quality is provided. The device includes: an acquisition module configured to acquire an image sequence, wherein the head and tail of the image sequence include a head image frame and a tail image frame respectively, and other positions other than the head and the tail in the image sequence include a group of intermediate image frames respectively; a determination module configured to determine a mapping frame of the intermediate image frame according to optical flow estimation based on at least any one of the head image frame and the tail image frame for an intermediate image frame in the group of intermediate image frames; and an adjustment module configured to adjust the intermediate image frame using the mapping frame to determine an enhanced image frame of the intermediate image frame.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar figure labels represent the same or similar elements, wherein:

[0009] Figure 1 A block diagram showing an application environment for enhancing image quality according to an exemplary implementation of the present disclosure is shown;

[0010] Figure 2 A block diagram for enhancing image quality according to some implementations of the present disclosure is shown;

[0011] Figure 3 A block diagram for enhancing image quality based on an enhancement model according to some implementations of the present disclosure is shown;

[0012] Figure 4 A block diagram of an enhancement model according to some implementations of the present disclosure is shown;

[0013] Figure 5 A block diagram showing various networks in an enhanced model according to some implementations of the present disclosure;

[0014] Figure 6 A block diagram for generating an enhanced image sequence according to some implementations of the present disclosure is shown;

[0015] Figure 7 A flowchart of a method for enhancing image quality according to some implementations of the present disclosure is shown;

[0016] Figure 8 A block diagram showing an apparatus for enhancing image quality according to some implementations of the present disclosure; and

[0017] Fig. 9 A block diagram of a device capable of implementing multiple implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0018] The implementation of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the implementations described herein. On the contrary, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of the implementation methods of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly related to the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; while in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.

[0026] Example Environment

[0027] With the development of video processing technology, variable resolution encoding and decoding technology has been proposed. Figure 1 Describes an application environment according to an example implementation of the present disclosure. Figure 1 A block diagram 100 of an application environment for enhancing image quality according to an exemplary implementation of the present disclosure is shown. Figure 1 As shown, the image sequence 110 may represent an original image sequence to be processed, and the image sequence may include multiple image frames, and each image frame has an original higher resolution. During the encoding 140 process, the original resolution of some frames in the image sequence 110 may be retained at predetermined intervals.

[0028] For example, 16 frames (or other numbers) may be used as a predetermined interval. The original video may be divided into a plurality of groups every 16 frames. In each group, the image frames at the head position and the tail position may be kept at the original resolution, and compression processing may be performed on the intermediate image frames at other positions other than the head position and the tail position (e.g., performing a downsampling operation of a predetermined coefficient (e.g., 2), etc.). For ease of description, the original resolution of the original video may be referred to as a first resolution (or high resolution), and the resolution of the compressed intermediate image frames may be referred to as a second resolution (or low resolution). At this point, the amount of data of the encoded image sequence 120 will be greatly reduced, and thus it is more suitable for subsequent processing such as storage and / or transmission.

[0029] During the decoding 150 process, the decoder can be used to process the image sequence 120. The decoder can use the reference image frames with high resolution at the head position and the tail position in the image sequence 120 to restore the compressed intermediate image frames so that the decoded video can be better restored to the quality of the original video. However, the encoding and decoding process will damage the quality of the original video, and may cause the restored video to not present the original quality, but to be blurred. At this time, how to perform the video processing process in a more effective way to enhance the image quality in the video processing process.

[0030] Summary of image quality enhancement

[0031] In order to at least partially solve the deficiencies in the prior art, according to an exemplary implementation of the present disclosure, a method for enhancing image quality is proposed. In summary, during decoding 150, the image frames with high image quality at the head position and the tail position in the image sequence 130 can be used to enhance the intermediate image frames with low image quality at the middle position. For example, an optical flow estimation algorithm can be used to determine the mapping frame associated with the intermediate image frame, and then the quality of the intermediate image frame can be enhanced based on the mapping frame.

[0032] See also Figure 2 Describes an overview of an exemplary implementation according to the present disclosure, Figure 2 A block diagram 200 for enhancing image quality according to some implementations of the present disclosure is shown. Figure 2 As shown, an image sequence 130 may be acquired. Here, the head position of the image sequence 130 may include a head image frame 210 with high image quality, and the tail position may include a tail image frame 212 also with high image quality. Further, other positions other than the head and tail in the image sequence 120 respectively include a group of intermediate image frames with low image quality.

[0033] It should be understood that, although each image frame in the image sequence 130 has the original resolution of the video, the intermediate image frames undergo a compression operation during the encoding process, and thus the image quality of the corresponding image frames after decoding may not reach the original image quality before compression. In this case, it is necessary to perform an image enhancement operation, and use the head image frame and / or the tail image frame to improve the quality of the intermediate image frame.

[0034] According to an exemplary implementation of the present disclosure, the image sequence 120 may include a predetermined number of multiple image frames (for example, represented by a positive integer n). At this time, the first image frame and the nth image frame in the image sequence 130 have high image quality, and the second image frame to the n-1th image frame in the image sequence 120 have low image quality. For the intermediate image frame 220 in a group of intermediate image frames, the mapping frame of the intermediate image frame 220 may be determined according to the optical flow estimation based on at least one of the head image frame 210 and the tail image frame 212.

[0035] It should be understood that optical flow refers to the movement of target pixels in an image due to the movement of an object in the image or the movement of a camera in two consecutive frames of images. The motion pattern of the surface and edge of an object in a visual scene caused by the relative motion between the observer and the scene. According to an example implementation of the present disclosure, the mapping frame between the intermediate image frame 220 and the head image frame 210 (and / or the tail image frame 212) can be determined based on the optical flow estimation technology, that is, the motion pattern of the object in the two frames of images is determined, and then the motion pattern and the high-definition image in the head image frame 210 (and / or the tail image frame 212) with high image quality are used to enhance the image quality in the intermediate image frame 220.

[0036] According to an example implementation of the present disclosure, the mapping frame may include a forward mapping frame 230 and / or a backward mapping frame 232. Here, the mapping frame 230 is determined based on the head image frame 210 and the intermediate image frame 220, and the mapping frame 232 is determined based on the tail image frame 212 and the intermediate image frame 220. Specifically, at least one of the mapping frames 230 and 232 may be determined, and the intermediate image frame 220 may be adjusted using the determined mapping frame, thereby improving the image quality. In other words, an enhanced image frame 240 of the intermediate image frame may be determined based on the mapping frame.

[0037] At this time, the enhanced image frame 240 is an image enhanced by the head image frame 210 and / or the tail image frame 212, and thus has better image quality than the intermediate image frame 220. That is, the intermediate data frame 220 has an original high-resolution restored clear image. Using the exemplary implementation of the present disclosure, the quality of each image frame in the image sequence can be improved without increasing the size of the image sequence. In this way, the image quality can be improved while ensuring the video compression rate, thereby improving the user's visual experience.

[0038] Detailed process of enhancing image quality

[0039] The outline of an exemplary implementation of the present disclosure has been described. Hereinafter, the symbolic representation of each term involved in the processing is first described. According to an exemplary implementation of the present disclosure, after the original video enters the encoder, 16 frames (or other numbers) can be used as a group of pictures (abbreviated as GoP) for downsampling. That is, except for the head frame and the tail frame, each intermediate image frame Downsampling is performed according to the spatial downsampling parameters set by the variable resolution coding, as shown in Formula 1:

[0040]

[0041] In the above formula, F represents the image frame in the video; the subscript n represents the temporal order of each frame (that is, the number of the position of the frame), slice represents the original high resolution, L represents the low resolution after downsampling, and Indicates the intermediate image frame at positions other than the head and tail positions. Represents the intermediate image frames in the encoded video that are located outside the head and tail. Indicates the intermediate image frames in the video before encoding, which are located outside the head and tail. (·)↓ d Indicates a downsampling operation with a coefficient of d (for example, d=2 means that the width and height after downsampling are half of the original values, respectively. This coefficient can be set to other values). D represents the noise introduced by the downsampling process. x represents an integer. The encoder is used for image frames with high image quality. and image frames with low image quality Encoding is performed, and compression loss ε is introduced (H，L) :

[0042]

[0043] In the above formula, Represents the decoded image frames at the head and tail positions, Represents the decoded image frame located at the middle position other than the head and tail positions. Represents the intermediate image frames in the encoded video that are located outside the head and tail. represents the intermediate image frame in the video before encoding, which is located outside the head and tail, and ε (H，L) Indicates compression loss.

[0044] During the decoding process, the decoder interpolates and upsamples the reconstructed low-quality frames to restore them to the original resolution. The noise introduced by interpolation is denoted as ε U , the upsampling process can be expressed as:

[0045]

[0046] In the above formula, represents the intermediate image frame restored to the original resolution after upsampling the decoded image, and ε U Indicates the noise introduced during operation. The original resolution is restored, however the image quality of these image frames is lost during the encoding and decoding process.

[0047] According to an exemplary implementation of the present disclosure, the first and last high-quality frames in each GoP can be used to enhance the quality of the intermediate image frames, and a temporal bidirectional guidance enhancement strategy can be adopted. Specifically, the head image frame or the tail image frame in the GoP can be used to perform optical flow estimation with the intermediate image frame to be enhanced, respectively, to determine the corresponding forward or backward mapping frame.

[0048] According to an example implementation of the present disclosure, a global motion aggregation (GMA) algorithm may be used to determine a mapping frame between two image frames. It should be understood that the GMA algorithm is only a specific example of optical flow estimation, and alternatively and / or additionally, a mapping frame between two image frames may be determined based on other optical flow estimation algorithms currently known and / or to be developed in the future.

[0049] According to an example implementation of the present disclosure, in the process of determining the mapping frame of the intermediate image frame, the forward mapping frame of the intermediate image frame can be determined based on the head image frame according to the optical flow estimation (as shown in Formula 4.1 below). Further, the backward mapping frame of the intermediate image frame can be determined based on the tail image frame according to the optical flow estimation (as shown in Formula 4.2 below).

[0050]

[0051]

[0052] In the above formula, represents the decoded head image frame located at the head position, represents the intermediate image frame currently to be processed, represents the tail image frame at the tail position, represents a forward mapping frame determined based on the head image frame and the intermediate image frame, and Represents a backward mapping frame determined based on the tail image frame and the intermediate image frame.

[0053] It should be understood that, although Formulas 4.1 and 4.2 are provided above, in a specific application environment, at least any one of Formulas 4.1 and 4.2 may be used alone. For example, forward optical flow estimation or backward optical flow estimation may be used alone, and a high-precision frame in the forward or backward direction may be used to achieve the effect of enhancing the image quality of the intermediate image frame. Alternatively and / or additionally, Formulas 4.1 and 4.2 may also be used in combination. At this time, the forward optical flow estimation and the backward optical flow estimation may be used in combination, thereby making full use of the high-quality images on both sides of the intermediate image frame to enhance the image quality of the intermediate image frame in a more effective manner. In this way, accurate and reliable optical flow estimation technology may be fully utilized to improve the quality of each image frame in the decoded video, thereby improving the visualization effect of the entire video.

[0054] According to an example implementation of the present disclosure, machine learning techniques may be used to determine an enhanced image frame of an intermediate image frame. Figure 3 A block diagram 300 for enhancing image quality based on an enhancement model according to some implementations of the present disclosure is shown. Figure 3 As shown, the enhancement model 310 can be used to determine the enhancement residual 320 of the intermediate image frame. In the context of the present disclosure, the enhancement residual 320 can represent the amount of change used to modify the intermediate image frame 220, that is, the enhanced image frame 240 can be determined based on the enhancement residual 320 and the intermediate image frame 220.

[0055] like Figure 3 As shown, the enhancement model 310 may be trained based on reference data, and the enhancement model 310 may describe the association relationship between a first reference image frame, a second reference image frame corresponding to the first reference image frame, and a reference mapping frame of the second reference image frame. Here, the first reference image frame may have a high image quality, and the second reference image frame may have a second low image quality.

[0056] According to an exemplary implementation of the present disclosure, multiple machine learning models currently known and / or to be developed in the future may be used to construct the enhancement model 310. Further, the enhancement model 310 may be continuously updated using reference data so that the enhancement model 310 may accurately describe the association relationship between the intermediate image frames of the compressed video obtained from the variable resolution encoder and the corresponding original intermediate image frames. Further, the quality of the intermediate image frames may be enhanced based on the association relationship, thereby improving the visual experience of the video viewer.

[0057] See also Figure 4 Describe the process of obtaining the enhanced model 310. Figure 4 A block diagram 400 of an enhanced model according to some implementations of the present disclosure is shown. Figure 4As shown, reference data can be obtained from an image sequence 440, which can be obtained by performing variable resolution encoding and decoding on a known original image sequence (each image frame has high resolution and high quality). The image sequence 440 may include multiple image frames, the image frames at the head position and the tail position may have high quality, and other intermediate image frames other than the head position and the tail position may have low quality.

[0058] At this time, an intermediate image frame 430 may be selected from the image sequence 440, and the forward mapping frame 410 and the backward mapping frame 420 of the intermediate image frame may be determined using the formulas 4.1 and 4.2 described above. Further, the forward mapping frame 410, the intermediate image frame 430, and the backward mapping frame 420 may be input to the enhancement model 310, and a prediction value 450 of an enhanced residual for enhancing the intermediate image frame 430 may be determined from the enhancement model 310. Further, the enhancement model 310 may be updated based on the prediction value 450 and the original high-quality reference image frame.

[0059] By using the example implementation of the present disclosure, the corresponding relationship between the known original image sequence and the image frames in the encoded-decoded image sequence can be fully utilized to generate corresponding reference data. Further, these reference data can be used to update the enhancement model 310 in a direction that continuously reduces the difference between the predicted value and the image residual determined based on the true value. In this way, the enhancement model 310 can more accurately describe the association relationship between the high-quality first reference image frame, the low-quality second reference image frame corresponding to the first reference image frame, and the reference mapping frame of the second reference image frame.

[0060] According to an exemplary implementation of the present disclosure, an intermediate image frame located outside the head and tail of the uncompressed original image sequence (also referred to as a reference image sequence) of the image sequence 440 may be selected as the first reference image frame. Further, an intermediate image frame located outside the head and tail of the uncompressed original image sequence (also referred to as a reference image sequence) may be selected as the first reference image frame. Figure 4 An intermediate image frame corresponding to the first reference image frame is selected from the image sequence 440 shown in FIG. 4 as the second reference image frame.

[0061] According to an example implementation of the present disclosure, a certain position may be specified and a first reference image frame and a second reference image frame may be selected from the original image sequence and the encoded-decoded image sequence, respectively. For example, an intermediate image frame at the ith position (when the GoP includes 16 image frames, 2≤i≤15) may be selected. Specifically, the intermediate image frame at the ith position may be selected from the original image sequence as the first reference image frame, and the intermediate image frame at the ith position may be selected from the encoded-decoded image sequence as the second reference image frame. Further, the formula described above may be used to determine the corresponding forward mapping frame and / or backward mapping frame, thereby determining the loss function for updating the response of the enhancement model 310.

[0062] According to an example implementation of the present disclosure, see Figure 5 Further details of the enhancement model 310 are described. Figure 5 A block diagram 500 of each network in the enhancement model 310 according to some implementations of the present disclosure is shown. The enhancement model 310 may include multiple networks: a bidirectional optical flow network 510, a feature extraction network 520, an aggregated attention network 530, and a quality enhancement network 540. Each network may be used to perform corresponding processing to determine the final enhancement residual.

[0063] First, the bidirectional optical flow network 510 is described. The head image frame 512, the middle image frame 514, and the tail image frame 516 determined based on the above method can be input to the bidirectional optical flow network 510. Further, corresponding processing can be performed on each input image frame to provide input data to the subsequent feature extraction network 520. Specifically, Figure 5 The mapping module (eg, warp module) shown may be configured to perform a mapping operation. For example, the mapping operation may be performed based on the following formula.

[0064]

[0065]

[0066] In the above formula, represents the forward mapping frame after mapping, represents the forward mapping frame determined based on the above formula, represents the head image frame, represents the noise introduced during the forward mapping process. Similarly, Represents the mapped backward mapping frame, represents the backward mapping frame determined based on the above formula, represents the tail image frame, represents the noise introduced in the backward mapping process. Further, the data and Input to the subsequent feature extraction network 520.

[0067] It should be understood that although Figure 5 An example of using a bidirectional optical flow network 510 to process a head image frame 512, an intermediate data frame 514, and a tail image frame 5116 is shown. Alternatively and / or additionally, the function of the bidirectional optical flow network 510, or a portion of the function thereof, can be completed outside the enhanced model 310. For example, the intermediate image frame and the corresponding forward mapping frame and backward mapping frame can be directly obtained outside the enhanced model. At this time, the intermediate image frame and the corresponding forward mapping frame and backward mapping frame can be directly input to the feature extraction module 520 for subsequent processing.

[0068] According to an exemplary implementation of the present disclosure, the feature extraction network 520 in the enhancement model 310 can be used to extract features from the intermediate image frame and the mapping frame. Figure 5 As shown, the feature extraction network 520 may include a connection module (such as Figure 5 ), and a subsequent convolutional network. Eight (or other numbers) of convolutional layers with ReLU can be used to extract features 522. Specifically, the feature extraction network 520 can extract features based on the following formula:

[0069]

[0070] In the above formula, F represents the feature extracted by the feature extraction network 520, which may include three components: F t , F 1 , F 2 , Conv() represents the convolution operation, and They respectively represent the data output by the upstream bidirectional optical flow network 510.

[0071] Furthermore, the fusion attention network 520 in the enhanced model 310 can be used to process the features 522 output by the upstream feature extraction network 520 to further determine the fusion features 538 of the intermediate image frame. Figure 5 As shown, the aggregated attention network 530 may include a temporal attention network 532 and a spatial attention network 534. At this time, the temporal attention network 532 in the fused attention network 530 may be used to determine a preliminary fused feature 536 of the intermediate image frame based on the feature 522. Further, the spatial attention network 534 in the fused attention network 530 may be used to determine a final fused feature 538 of the intermediate image frame based on the preliminary fused feature 536.

[0072] like Figure 5As shown, the temporal attention network 532 may include a convolutional module (e.g., Figure 5 conv module in ), product modules (e.g., Figure 5 ) and activation modules (e.g., Figure 5 These modules can generate preliminary fusion features 536 based on the following formula, that is, determine the associated attention weights of the intermediate image frames.

[0073] ω 1 =sigmoid(θ(F t )⊙φ(F 1 )) Formula 7

[0075] In the above formula, F t 、F 1 Respectively represent feature components from the feature extraction network, θ and φ respectively represent parameters of the convolutional network, ⊙ represents a multiplication operation, and sigmoid() represents a corresponding activation function. According to an example implementation of the present disclosure, the specific values ​​of each parameter can be determined based on a variety of functional modes currently known and / or to be developed in the future, and thus will not be described in detail. Further, the fusion function can be used to process the preliminary fusion feature 536 so as to perform subsequent spatial attention-related processing.

[0076]

[0077] According to an exemplary implementation of the present disclosure, similar processing may be performed on other components to obtain corresponding results. Furthermore, each initially fused feature obtained may be further processed based on the following formula.

[0078]

[0079] In the above formula, F fusion represents the result after fusion processing, FusionConv() represents the fusion function, and Respectively represent the output results of the time domain attention network 532. At this time, in the spatial domain attention network 534, the attention operation within the spatial domain can be further processed to obtain the final fused feature 533. Specifically, the features obtained by the initial fusion are down-sampled twice by convolution, and then the spatial attention is obtained by up-sampling and summing from top to bottom. Finally, a fused feature map is generated by element-wise multiplication. According to an example implementation of the present disclosure, for example, the fused feature 538 can be determined based on the following formula.

[0080] F′=F fusion ↓ 2 , F″=F′↓2 Formula 10

[0082]

[0084]

[0085] In the above formula, F′ and F″ represent the processing results after two downsampling operations, ↓ 2 Indicates downsampling operation, ↑ 2 represents an upsampling operation, "+" represents an element-wise addition operation, represents the intermediate result of the addition operation, and ⊙ represents the product operation performed in an element-wise manner. In this way, the process of extracting fusion features can be converted into mathematical operations, so that the fusion feature 538 can increase the accurate description of the management relationship between each image frame, thereby improving the accuracy of the entire enhancement network.

[0086] Further, the aggregate attention network 530 can output the fused features 538 to the downstream quality enhancement network 540, and further utilize the quality enhancement network in the enhancement model to determine the enhanced residual based on the fused features. Figure 5 As shown, the quality enhancement network 540 may include a residual network, and an enhanced residual R may be obtained through a convolutional layer. QE Then, after the bidirectional guidance module and the residual, the enhanced residual for enhancing the intermediate image frame 514 can be generated using the following formula.

[0087]

[0088] In the above formula, R QE represents the enhanced residual, represents the processing function of the residual network, and It represents the fusion feature 538 generated according to the above formula. Further, the following formula can be used to perform enhancement processing on the intermediate image frame to obtain an enhanced image frame.

[0089]

[0090] In the above formula, represents an enhanced image frame with higher quality. represents the intermediate image frame 514, and R QE Indicates that it is used for The enhanced residual after the enhancement process is performed. Using the example implementation of the present disclosure, the enhancement model can be used to fully learn the association relationship between each image frame in the reference data, so as to enhance the image quality in a more accurate manner.

[0091] According to an exemplary implementation of the present disclosure, based on Figure 5 The specific network structure shown is used to process each reference data, thereby obtaining the corresponding predicted value of the enhanced residual. Further, the enhanced model can be updated in a manner that minimizes the difference between the predicted value and the true value of the enhanced residual based on the following loss function.

[0092]

[0093] In the above formula, represents the loss function, The predicted value of the enhanced image frame after enhancement, and Represents the true value of the intermediate image frame in the original image sequence used as reference data. Using the example implementation of the present disclosure, a large number of image sequences can be used to construct reference samples for updating the enhancement model. In this way, the enhancement model can continuously output corresponding enhanced residuals and enhanced image frames in a more accurate direction.

[0094] According to an example implementation of the present disclosure, the enhanced model may be continuously updated in an iterative manner until the enhanced model 310 satisfies a predetermined stop condition. For example, training may be stopped when the loss function reaches a predetermined range, training may be stopped when the training process reaches a predetermined round, and so on.

[0095] According to an exemplary implementation of the present disclosure, when a trained enhancement model 310 has been obtained, the enhancement model 310 can be used to process each intermediate image frame in the decoded video one by one. For example, a group of enhanced image frames of a group of intermediate image frames can be obtained respectively. Further, the head image frame, the group of enhanced image frames, and the tail image frame can be used to generate an enhanced image sequence corresponding to the image sequence. Figure 6 Describes more details about the enhancement process.

[0096] Figure 6 A block diagram 600 for generating an enhanced image sequence according to some implementations of the present disclosure is shown. Figure 6 As shown, the image sequence 120 represents a compressed video, and the compressed video can be obtained by performing an encoding operation on the original video using a variable resolution encoder. Here, the original video has a higher first resolution. In the image sequence 120, the image frames at the head position and the tail position have a first resolution, and the intermediate image frames at other positions have a lower second resolution. A decoding 150 operation can be performed on the compressed video to obtain a decoded image sequence 130. At this time, each image frame in the image sequence 130 has a higher resolution, but the image quality of the intermediate image frames at the middle position is lower.

[0097] The process described above can be used to process each intermediate image frame one by one, thereby generating corresponding enhanced image frames 610, 612, ..., 614, etc. Further, these enhanced image frames 610, 612, ..., 614 and the extracted reference image frames with higher image quality at the head position and the tail position can be used to generate an enhanced image sequence 620. At this time, each image frame in the enhanced image sequence has higher image quality, thereby improving the visual experience of the viewer.

[0098] According to an example implementation of the present disclosure, by combining variable resolution coding, high-resolution reconstructed frames can be used to guide the quality enhancement of low-resolution reconstructed frames, and the bidirectional optical flow network can greatly improve the accuracy of motion information modeling. Further, in the process of variable resolution coding, several video frames are grouped together, and the first and last two frames are coded with high resolution, which can effectively assist video enhancement while saving bit rate.

[0099] It should be understood that the method described above can be used to process videos including different scenes. For example, character videos, landscape videos, animation scenes, game videos, etc. can be processed. At present, a large number of video contents in game scenes such as live video and recorded video have appeared. Generally speaking, compared with real scenes, animation scenes and game scenes are relatively simple, and the difference between the contents of each frame in the video may be small. At this time, according to an example implementation of the present disclosure, it is more helpful to use the high-definition reference image frames at the head position and the tail position of the GoP to enhance the visual effect of the intermediate image frame.

[0100] By using the example implementation of the present disclosure, the proposed method can improve the reconstruction quality of animation and / or game video variable resolution compression, and reduce the encoding delay of animation transmission, game live broadcast, and cloud game system. Compared with traditional fixed resolution encoding or variable resolution encoding, the proposed method can bring significant improvement in subjective quality and objective quality, while reducing the storage and transmission overhead of game videos to a certain extent.

[0101] According to an example implementation of the present disclosure, for a specific animation scene and / or game scene, a pre-recorded video can be used to generate corresponding reference samples according to the method described above. Further, these reference samples can be used to train enhancement models dedicated to different animation scenes and / or game scenes. In this way, the accuracy and performance of the enhancement model can be further improved, thereby improving the visual experience of the audience.

[0102] Example Process

[0103] Figure 7A flow chart of a method 700 for enhancing image quality according to some implementations of the present disclosure is shown. At box 710, an image sequence is acquired, the head and tail of the image sequence include a head image frame and a tail image frame, respectively, and other positions other than the head and the tail in the image sequence include a group of intermediate image frames. At box 720, for an intermediate image frame in a group of intermediate image frames, a mapping frame of the intermediate image frame is determined according to optical flow estimation based on at least any one of the head image frame and the tail image frame. At box 730, the intermediate image frame is adjusted using the mapping frame to determine an enhanced image frame of the intermediate image frame.

[0104] According to an example implementation of the present disclosure, determining a mapping frame of an intermediate image frame includes: determining a forward mapping frame of the intermediate image frame according to optical flow estimation based on the head image frame.

[0105] According to an example implementation of the present disclosure, determining the mapping frame includes: determining a backward mapping frame of the intermediate image frame according to optical flow estimation based on the tail image frame.

[0106] According to an example implementation of the present disclosure, determining an enhanced image frame of an intermediate image frame includes: determining an enhanced residual of the intermediate image frame using an enhancement model, the enhancement model describing an association relationship between a first reference image frame, a second reference image frame corresponding to the first reference image frame, and a reference mapping frame of the second reference image frame; and determining the enhanced image frame based on the enhanced residual and the intermediate image frame.

[0107] According to an example implementation of the present disclosure, the enhancement model is obtained based on: based on a reference mapping frame, using the enhancement model to determine a prediction value of an enhancement residual corresponding to a first reference image frame; and determining the enhancement model based on the prediction value and the first reference image frame.

[0108] According to an exemplary implementation of the present disclosure, the first reference image frame is an intermediate reference image frame selected from a reference image sequence and located at a position other than the head and the tail.

[0109] According to an example implementation of the present disclosure, the second reference image frame is an intermediate image frame corresponding to the first reference image frame selected from an encoded image sequence of the reference image sequence.

[0110] According to an example implementation of the present disclosure, using an enhancement model to determine an enhanced residual of an intermediate image frame includes: using a feature extraction network in the enhancement model to extract features from the intermediate image frame and a mapping frame; using a fusion attention network in the enhancement model to determine fused features of the intermediate image frame based on the features; and using a quality enhancement network in the enhancement model to determine an enhanced residual based on the fused features.

[0111] According to an example implementation of the present disclosure, the fusion features include: utilizing the temporal attention network in the fusion attention network to determine preliminary fusion features of the intermediate image frame based on the features; and utilizing the spatial attention network in the fusion attention network to determine the fusion features of the intermediate image frame based on the preliminary fusion features.

[0112] According to an example implementation of the present disclosure, determining an enhanced residual of an intermediate image frame includes: utilizing a residual network in a quality enhancement network to determine an enhanced residual of the intermediate image frame based on fused features.

[0113] According to an example implementation of the present disclosure, the method further includes: acquiring a group of enhanced image frames of a group of intermediate image frames respectively; and generating an enhanced image sequence corresponding to the image sequence using the head image frame, the group of enhanced image frames, and the tail image frame.

[0114] According to an exemplary implementation of the present disclosure, the image sequence is obtained by decoding a compressed video using a variable resolution decoder, and the compressed video is obtained by encoding an original video using a variable resolution encoder.

[0115] According to an exemplary implementation of the present disclosure, the image quality of the head image frame and the tail image frame is higher than the image quality of a group of middle image frames.

[0116] Example devices and equipment

[0117] Figure 8 A block diagram of an apparatus 800 for enhancing image quality according to some implementations of the present disclosure is shown. The apparatus 800 includes: an acquisition module 810 configured to acquire an image sequence, wherein the head and tail of the image sequence include a head image frame and a tail image frame respectively, and other positions other than the head and the tail in the image sequence include a group of intermediate image frames respectively; a determination module 820 configured to determine, for an intermediate image frame in the group of intermediate image frames, a mapping frame of the intermediate image frame based on at least any one of the head image frame and the tail image frame according to optical flow estimation; and an adjustment module 830 configured to adjust the intermediate image frame using the mapping frame to determine an enhanced image frame of the intermediate image frame.

[0118] According to an example implementation of the present disclosure, the determination module 820 includes: a first determination module configured to determine a forward mapping frame of the intermediate image frame according to optical flow estimation based on the head image frame.

[0119] According to an example implementation of the present disclosure, the determination module 820 includes: a second determination module configured to determine the mapping frame includes: determining a backward mapping frame of the intermediate image frame according to optical flow estimation based on the tail image frame.

[0120] According to an example implementation of the present disclosure, the determination module 820 includes: a calling module, configured to determine an enhanced residual of an intermediate image frame using an enhancement model, the enhancement model describing an association relationship between a first reference image frame, a second reference image frame corresponding to the first reference image frame, and a reference mapping frame of the second reference image frame; and an enhancement module, configured to determine an enhanced image frame based on the enhanced residual and the intermediate image frame.

[0121] According to an example implementation of the present disclosure, the enhancement model is obtained based on: based on a reference mapping frame, using the enhancement model to determine a prediction value of an enhancement residual corresponding to a first reference image frame; and determining the enhancement model based on the prediction value and the first reference image frame.

[0122] According to an exemplary implementation of the present disclosure, the first reference image frame is an intermediate reference image frame selected from a reference image sequence and located at a position other than the head and the tail.

[0123] According to an example implementation of the present disclosure, the second reference image frame is an intermediate image frame corresponding to the first reference image frame selected from an encoded image sequence of the reference image sequence.

[0124] According to an example implementation of the present disclosure, the calling module includes: an extraction module configured to extract features from the intermediate image frame and the mapping frame using a feature extraction network in the enhancement model; a fusion module configured to determine fused features of the intermediate image frame based on the features using a fusion attention network in the enhancement model; and a residual determination module configured to determine an enhanced residual based on the fused features using a quality enhancement network in the enhancement model.

[0125] According to an example implementation of the present disclosure, a fusion module configured to determine fusion features includes: a time domain module configured to determine preliminary fusion features of the intermediate image frame based on features using the time domain attention network in the fusion attention network; and a spatial domain module configured to determine the fusion features of the intermediate image frame based on the preliminary fusion features using the spatial domain attention network in the fusion attention network.

[0126] According to an example implementation of the present disclosure, the residual determination module: the enhanced residual determination module is configured to utilize the residual network in the quality enhancement network to determine the enhanced residual of the intermediate image frame based on the fusion features.

[0127] According to an example implementation of the present disclosure, the apparatus further includes: a video processing module configured to respectively acquire a group of enhanced image frames of a group of intermediate image frames; and a video generation module configured to generate an enhanced image sequence corresponding to the image sequence using the head image frame, the group of enhanced image frames, and the tail image frame.

[0128] According to an exemplary implementation of the present disclosure, the image sequence is obtained by decoding a compressed video using a variable resolution decoder, and the compressed video is obtained by encoding an original video using a variable resolution encoder.

[0129] According to an exemplary implementation of the present disclosure, the image quality of the head image frame and the tail image frame is higher than the image quality of a group of middle image frames.

[0130] Fig. 9 9 is a block diagram of a device 900 capable of implementing multiple implementations of the present disclosure. It should be understood that Fig. 9 The computing device 900 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. Fig. 9 The computing device 900 shown may be used to implement the methods described above.

[0131] like Fig. 9 As shown, computing device 900 is in the form of a general-purpose computing device. Components of computing device 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processing unit 910 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 920. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of computing device 900.

[0132] The computing device 900 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 920 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 930 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device 900.

[0133] The computing device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig. 9 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 920 may include a computer program product 925 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0134] The communication unit 940 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 900 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 900 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0135] Input device 950 may be one or more input devices, such as a mouse, keyboard, tracking ball, etc. Output device 960 may be one or more output devices, such as a display, a speaker, a printer, etc. Computing device 900 may also communicate with one or more external devices (not shown) through communication unit 940 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 900, or communicate with any device that allows computing device 900 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0136] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0137] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0138] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0139] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0140] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0141] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for enhancing image quality, comprising: Acquire an image sequence, wherein the head and the tail of the image sequence respectively include a head image frame and a tail image frame, and other positions other than the head and the tail in the image sequence respectively include a group of intermediate image frames; For an intermediate image frame in the group of intermediate image frames, determining a mapping frame of the intermediate image frame according to optical flow estimation based on at least any one of the head image frame and the tail image frame; as well as The intermediate image frame is adjusted using the mapping frame to determine an enhanced image frame of the intermediate image frame.

2. The method of claim 1 , wherein determining the mapping frame of the intermediate image frame comprises: Based on the head image frame, a forward mapping frame of the intermediate image frame is determined according to the optical flow estimation.

3. The method of claim 1 , wherein determining the mapping frame of the intermediate image frame comprises: Based on the tail image frame, a backward mapping frame of the intermediate image frame is determined according to the optical flow estimation.

4. The method of claim 1 , wherein determining the enhanced image frame of the intermediate image frame comprises: Determining an enhanced residual of the intermediate image frame using an enhancement model, wherein the enhancement model describes an association relationship between a first reference image frame, a second reference image frame corresponding to the first reference image frame, and a reference mapping frame of the second reference image frame; as well as Based on the enhanced residual and the intermediate image frame, the enhanced image frame is determined.

5. The method according to claim 4, wherein the enhanced model is obtained based on: Determining, based on the reference mapping frame, a prediction value of an enhancement residual corresponding to the first reference image frame using the enhancement model; and The enhancement model is determined based on the prediction value and the first reference image frame. 6 . The method according to claim 5 , wherein the first reference image frame is an intermediate reference image frame selected from a reference image sequence and located at a position other than a head and a tail.

7. The method according to claim 6, wherein the second reference image frame is an intermediate image frame corresponding to the first reference image frame selected from an encoded image sequence of the reference image sequence.

8. The method of claim 4, wherein determining the enhanced residual of the intermediate image frame using the enhanced model comprises: Extracting features from the intermediate image frame and the mapping frame using a feature extraction network in the enhancement model; Determining fusion features of the intermediate image frame based on the features using a fusion attention network in the enhancement model; as well as The enhancement residual is determined based on the fused features using a quality enhancement network in the enhancement model.

9. The method of claim 8, wherein determining the fusion feature comprises: Determine preliminary fusion features of the intermediate image frame based on the features using a temporal attention network in the fusion attention network; as well as The spatial attention network in the fusion attention network is used to determine the fusion features of the intermediate image frame based on the preliminary fusion features.

10. The method of claim 8, wherein determining the enhanced residual of the intermediate image frame comprises: The enhanced residual of the intermediate image frame is determined based on the fused features using a residual network in the quality enhancement network.

11. The method according to claim 1, further comprising: Respectively acquiring a group of enhanced image frames of the group of intermediate image frames; as well as An enhanced image sequence corresponding to the image sequence is generated using the head image frame, the group of enhanced image frames, and the tail image frame.

12. The method according to claim 1, wherein the image sequence is obtained by decoding a compressed video using a variable resolution decoder, and the compressed video is obtained by encoding an original video using a variable resolution encoder.

13. The method according to claim 1, wherein the image quality of the head image frame and the tail image frame is higher than the image quality of the group of intermediate image frames.

14. A device for enhancing image quality, comprising: an acquisition module configured to acquire an image sequence, wherein the head and the tail of the image sequence respectively include a head image frame and a tail image frame, and other positions other than the head and the tail in the image sequence respectively include a group of intermediate image frames; a determination module configured to determine, for an intermediate image frame in the group of intermediate image frames, a mapping frame of the intermediate image frame based on at least one of the head image frame and the tail image frame according to optical flow estimation; as well as An adjustment module is configured to adjust the intermediate image frame using the mapping frame to determine an enhanced image frame of the intermediate image frame.

15. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 13 when executed by the at least one processing unit.

16. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 13.

Images