Method and apparatus for enhancing image quality, device, and medium

Abstract

Provided are a method and apparatus for enhancing image quality, a device, and a medium. The method comprises: acquiring an image sequence, wherein the head and the tail of the image sequence respectively comprise a head image frame and a tail image frame, and a group of intermediate image frames is separately comprised at positions of the image sequence other than the head and the tail; for an intermediate image frame in a group of intermediate image frames, determining a mapping frame of the intermediate image frame on the basis of at least one of the head image frame and the tail image frame by means of optical flow estimation; and adjusting the intermediate image frame by means of the mapping frame to determine an enhanced image frame of the intermediate image frame.

Description

Method, apparatus, device and medium for enhancing image quality

[0001] This application claims priority to the Chinese invention patent application entitled “Methods, devices, apparatus and media for enhancing image quality” filed on August 31, 2023, with application number 202311116996.8, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] Exemplary implementations of the present disclosure relate generally to image processing, and more particularly to methods, devices, apparatuses, and computer-readable storage media for enhancing image quality in variable resolution decoding. Background Art

[0003] With the development of video processing technology, variable resolution encoding and decoding technology has been proposed. During the encoding process, the original resolution of certain frames in the original video can be retained at predetermined intervals, and other frames can be compressed. During the decoding process, these image frames with original resolution can be used to restore the compressed video frames so that the decoded video can better restore the quality of the original video. However, the encoding and decoding process will damage the quality of the original video and may cause the restored video to not present the original quality, but instead appear blurred. At this time, how to perform the video processing process in a more efficient way to enhance the image quality during the video processing process.

[0004] Summary of the Invention

[0005] In a first aspect of the present disclosure, a method for enhancing image quality is provided. In this method, an image sequence is acquired, wherein the head and tail of the image sequence include a head image frame and a tail image frame, respectively, and each position in the image sequence other than the head and tail includes a set of intermediate image frames. For an intermediate image frame in the set of intermediate image frames, a mapping frame for the intermediate image frame is determined based on at least one of the head image frame and the tail image frame according to optical flow estimation. The mapping frame is used to adjust the intermediate image frame to determine an enhanced image frame for the intermediate image frame.

[0006] In a second aspect of the present disclosure, a device for enhancing image quality is provided. The device includes: an acquisition module configured to acquire an image sequence, wherein the head and tail of the image sequence respectively include a head image frame and a tail image frame, and positions other than the head and tail in the image sequence respectively include a set of intermediate image frames; a determination module configured to determine, for an intermediate image frame in the set of intermediate image frames, a mapping frame of the intermediate image frame based on at least one of the head image frame and the tail image frame according to optical flow estimation; and an adjustment module configured to adjust the intermediate image frame using the mapping frame to determine an enhanced image frame of the intermediate image frame.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0009] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent hereinafter with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] FIG1 shows a block diagram of an application environment for enhancing image quality according to an exemplary implementation of the present disclosure;

[0012] FIG2 shows a block diagram for enhancing image quality according to some implementations of the present disclosure;

[0013] FIG3 shows a block diagram for enhancing image quality based on an enhancement model according to some implementations of the present disclosure;

[0014] FIG4 illustrates a block diagram of an enhancement model according to some implementations of the present disclosure;

[0015] FIG5 illustrates a block diagram of various networks in an enhanced model according to some implementations of the present disclosure;

[0016] FIG6 illustrates a block diagram for generating an enhanced image sequence according to some implementations of the present disclosure;

[0017] FIG7 shows a flowchart of a method for enhancing image quality according to some implementations of the present disclosure;

[0018] FIG8 shows a block diagram of an apparatus for enhancing image quality according to some implementations of the present disclosure; and

[0019] FIG9 illustrates a block diagram of a device capable of implementing various implementations of the present disclosure. DETAILED DESCRIPTION

[0020] The following describes implementations of the present disclosure in more detail with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the implementations described herein. Rather, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of the implementation of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0022] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0024] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0025] As an optional but non-limiting implementation, in response to receiving a user's active request, a prompt message may be sent to the user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0027] As used herein, the term "in response to" refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is satisfied. For example, in some cases, a subsequent action may be executed immediately upon the occurrence of the event or the satisfaction of the condition; in other cases, the subsequent action may be executed some time after the occurrence of the event or the satisfaction of the condition.

[0028] Sample Environment

[0029] With the development of video processing technology, variable resolution encoding and decoding technologies have been proposed. Referring to FIG. 1 , an application environment according to an example implementation of the present disclosure is described. FIG. 1 shows a block diagram 100 of an application environment for enhancing image quality according to an example implementation of the present disclosure. As shown in FIG. 1 , an image sequence 110 may represent an original image sequence to be processed. The image sequence may include multiple image frames, each of which has an original, higher resolution. During encoding 140 , the original resolution of certain frames in the image sequence 110 may be retained at predetermined intervals.

[0030] For example, 16 frames (or other numbers) can be used as a predetermined interval. The original video can be divided into multiple groups every 16 frames. In each group, the image frames located at the head position and the tail position can maintain the original resolution, and the intermediate image frames at other positions other than the head position and the tail position can be compressed (for example, performing a downsampling operation of a predetermined coefficient (for example, 2), etc.). For ease of description, the original resolution of the original video can be referred to as the first resolution (or high resolution), and the resolution of the compressed intermediate image frames can be referred to as the second resolution (or low resolution). At this time, the amount of data of the encoded image sequence 120 will be greatly reduced, and it is therefore more suitable for subsequent processing such as storage and / or transmission.

[0031] During decoding 150, a decoder can be used to process the image sequence 120. The decoder can use high-resolution reference image frames at the beginning and end of the image sequence 120 to restore the compressed intermediate image frames, thereby enabling the decoded video to better restore the quality of the original video. However, the encoding and decoding processes will damage the quality of the original video and may cause the restored video to not present the original quality, but to be blurry, etc. At this time, how to perform the video processing process in a more efficient manner to enhance the image quality during the video processing process.

[0032] Summary of image quality enhancement

[0033] To at least partially address the shortcomings of the prior art, an exemplary implementation of the present disclosure provides a method for enhancing image quality. In summary, during decoding 150, high-quality image frames at the beginning and end of an image sequence 130 can be used to enhance intermediate image frames with lower image quality. For example, an optical flow estimation algorithm can be used to determine the mapping frames associated with the intermediate image frames, and the quality of the intermediate image frames can then be enhanced based on the mapping frames.

[0034] An overview of an exemplary implementation of the present disclosure is provided with reference to FIG2 , which illustrates a block diagram 200 for enhancing image quality according to some implementations of the present disclosure. As shown in FIG2 , an image sequence 130 may be acquired. Here, the beginning of the image sequence 130 may include a leading image frame 210 having high image quality, and the end of the image sequence 130 may include a trailing image frame 212 also having high image quality. Furthermore, positions other than the beginning and end of the image sequence 120 may each include a set of intermediate image frames having low image quality.

[0035] It should be understood that although each image frame in the image sequence 130 has the original resolution of the video, the intermediate image frames undergo compression during the encoding process. Therefore, the image quality of the corresponding image frames after decoding may not reach the original image quality before compression. In this case, it is necessary to perform image enhancement and utilize the leading image frame and / or the trailing image frame to improve the quality of the intermediate image frames.

[0036] According to an exemplary implementation of the present disclosure, the image sequence 120 may include a predetermined number of image frames (e.g., represented by a positive integer n). In this case, the first and nth image frames in the image sequence 130 have high image quality, and the second through n-1th image frames in the image sequence 120 have low image quality. For an intermediate image frame 220 in a set of intermediate image frames, a mapping frame for the intermediate image frame 220 may be determined based on at least one of the head image frame 210 and the tail image frame 212 according to optical flow estimation.

[0037] It should be understood that optical flow refers to the movement of target pixels in an image due to the movement of an object in the image or the movement of a camera in two consecutive frames of images. The motion pattern of the surface and edges of objects in a visual scene caused by the relative motion between the observer and the scene. According to an example implementation of the present disclosure, the mapping frame between the intermediate image frame 220 and the head image frame 210 (and / or the tail image frame 212) can be determined based on the optical flow estimation technology, that is, the motion pattern of the objects in the two frames of images can be determined, and then the motion pattern and the high-definition image in the head image frame 210 (and / or the tail image frame 212) with high image quality can be used to enhance the image quality in the intermediate image frame 220.

[0038] According to an exemplary implementation of the present disclosure, the mapping frame may include a forward mapping frame 230 and / or a backward mapping frame 232. Here, the mapping frame 230 is determined based on the head image frame 210 and the intermediate image frame 220, and the mapping frame 232 is determined based on the tail image frame 212 and the intermediate image frame 220. Specifically, at least one of the mapping frames 230 and 232 may be determined, and the determined mapping frame may be used to adjust the intermediate image frame 220, thereby improving image quality. In other words, an enhanced image frame 240 of the intermediate image frame may be determined based on the mapping frame.

[0039] In this case, the enhanced image frame 240 is an image enhanced using the head image frame 210 and / or the tail image frame 212, and thus has better image quality than the intermediate image frame 220. That is, the intermediate data frame 220 has the original high-resolution, restored, clear image. Using the exemplary implementations of the present disclosure, the quality of each image frame in an image sequence can be improved without increasing the size of the image sequence. In this way, image quality can be improved while ensuring video compression rate, thereby enhancing the user's visual experience.

[0040] Detailed process of enhancing image quality

[0041] Having described the outline of an exemplary implementation of the present disclosure, the following first describes the symbolic representation of the various terms involved in the processing. According to an exemplary implementation of the present disclosure, after the original video enters the encoder, it can be downsampled by taking 16 frames (or other numbers) as a group of pictures (GoP). That is, except for the header frame and the tail frame, each intermediate image frame Downsampling is performed according to the spatial downsampling parameters set by the variable resolution encoding, as shown in Formula 1:

[0042] In the above formula, F represents the image frame in the video; the subscript n represents the temporal order of each frame (that is, the number of the position of the frame), H represents the original high resolution, L represents the low resolution after downsampling, and Indicates the intermediate image frame at positions other than the head and tail positions. Represents the intermediate image frames in the encoded video that are located outside the head and tail. Indicates the intermediate image frames in the video before encoding, which are located outside the head and tail. (·)↓ d Indicates a downsampling operation with a coefficient of d (for example, d=2 means that the width and height after downsampling are half of the original values, respectively. This coefficient can be set to other values). D Represents the noise introduced by the downsampling process. x represents an integer. The encoder is used for high-quality image frames. and image frames with low image quality Encoding is performed, and compression loss ε is introduced (H,L) :

[0043] In the above formula, Represents the decoded image frames at the head and tail positions, Represents the decoded image frame at the middle position other than the head and tail positions. Represents the intermediate image frames in the encoded video that are located outside the head and tail. represents the intermediate image frame in the video before encoding, which is located outside the head and tail, and ε (H,L) Indicates compression loss.

[0044] During the decoding process, the decoder interpolates and upsamples the reconstructed low-quality frames to restore them to their original resolution. The noise introduced by interpolation is denoted as ε U , the upsampling process can be expressed as:

[0045] In the above formula, represents the intermediate image frame restored to the original resolution after performing upsampling on the decoded image, and ε U Indicates the noise introduced during operation. The original resolution is restored, however the image quality of these frames is lost during the encoding and decoding process.

[0046] According to an exemplary implementation of the present disclosure, the quality of intermediate image frames can be enhanced using the first and last high-quality frames in each GoP, employing a temporal bidirectional guidance pair enhancement strategy. Specifically, the first or last image frame in a GoP can be used to perform optical flow estimation with the intermediate image frame to be enhanced, thereby determining the corresponding forward or backward mapping frame.

[0047] According to an example implementation of the present disclosure, a Global Motion Aggregation (GMA) algorithm may be used to determine a mapping frame between two image frames. It should be understood that the GMA algorithm is merely a specific example of optical flow estimation. Alternatively and / or additionally, the mapping frame between two image frames may be determined based on other optical flow estimation algorithms currently known and / or to be developed in the future.

[0048] According to an example implementation of the present disclosure, in determining the mapping frame of the intermediate image frame, a forward mapping frame of the intermediate image frame can be determined based on the head image frame and optical flow estimation (as shown in Formula 4.1 below). Furthermore, a backward mapping frame of the intermediate image frame can be determined based on the tail image frame and optical flow estimation (as shown in Formula 4.2 below).

[0049] In the above formula, Represents the decoded head image frame located at the head position, Indicates the intermediate image frame currently to be processed, represents the tail image frame at the tail position, represents a forward mapping frame determined based on the head image frame and the intermediate image frame, and Represents a backward mapping frame determined based on the tail image frame and the intermediate image frame.

[0050] It should be understood that although formulas 4.1 and 4.2 are provided above, in a specific application environment, at least any one of formulas 4.1 and 4.2 can be used alone. For example, forward optical flow estimation or backward optical flow estimation can be used alone, and a high-precision frame in the forward or backward direction can be used to enhance the image quality of the intermediate image frame. Alternatively and / or additionally, formulas 4.1 and 4.2 can also be used in combination. At this time, the forward optical flow estimation and the backward optical flow estimation can be used in combination, thereby making full use of the high-quality images on both sides of the intermediate image frame to enhance the image quality of the intermediate image frame in a more effective manner. In this way, accurate and reliable optical flow estimation technology can be fully utilized to improve the quality of each image frame in the decoded video, thereby improving the visualization effect of the entire video.

[0051] According to an example implementation of the present disclosure, machine learning techniques can be used to determine an enhanced image frame for an intermediate image frame. FIG3 illustrates a block diagram 300 for enhancing image quality based on an enhancement model, according to some implementations of the present disclosure. As shown in FIG3 , an enhancement model 310 can be used to determine an enhancement residual 320 for an intermediate image frame. In the context of the present disclosure, enhancement residual 320 can represent a change used to modify intermediate image frame 220. That is, enhanced image frame 240 can be determined based on enhancement residual 320 and intermediate image frame 220.

[0052] As shown in FIG3 , an enhancement model 310 may be trained based on reference data and may describe an association relationship between a first reference image frame, a second reference image frame corresponding to the first reference image frame, and a reference mapping frame of the second reference image frame. Here, the first reference image frame may have a high image quality, and the second reference image frame may have a second low image quality.

[0053] According to an exemplary implementation of the present disclosure, various machine learning models currently known and / or to be developed in the future can be utilized to construct an enhancement model 310. Furthermore, the enhancement model 310 can be continuously updated using reference data so that the enhancement model 310 can accurately describe the association between intermediate image frames of a compressed video obtained from a variable-resolution encoder and the corresponding original intermediate image frames. Furthermore, based on this association, the quality of the intermediate image frames can be enhanced, thereby improving the visual experience of video viewers.

[0054] The process of obtaining the enhancement model 310 is described with reference to FIG4 , which shows a block diagram 400 of the enhancement model according to some implementations of the present disclosure. As shown in FIG4 , reference data can be obtained from an image sequence 440. This image sequence 440 can be obtained, for example, by performing variable-resolution encoding and decoding on a known original image sequence (each image frame having high resolution and high quality). This image sequence 440 can include multiple image frames, where the image frames at the head and tail positions can have high quality, and the intermediate image frames other than the head and tail positions can have low quality.

[0055] At this time, an intermediate image frame 430 can be selected from the image sequence 440, and the forward mapping frame 410 and the backward mapping frame 420 of the intermediate image frame can be determined using the formulas 4.1 and 4.2 described above. Further, the forward mapping frame 410, the intermediate image frame 430, and the backward mapping frame 420 can be input to the enhancement model 310, and the prediction value 450 of the enhancement residual for enhancing the intermediate image frame 430 can be determined from the enhancement model 310. Further, the enhancement model 310 can be updated based on the prediction value 450 and the original high-quality reference image frame.

[0056] Using the example implementation of the present disclosure, the correspondence between the known original image sequence and the image frames in the encoded-decoded image sequence can be fully utilized to generate corresponding reference data. Furthermore, this reference data can be used to update the enhancement model 310 in a direction that continuously reduces the difference between the predicted value and the image residual determined based on the true value. In this way, the enhancement model 310 can more accurately describe the association between the high-quality first reference image frame, the low-quality second reference image frame corresponding to the first reference image frame, and the reference mapping frame of the second reference image frame.

[0057] According to an exemplary implementation of the present disclosure, an intermediate image frame located outside the head and tail of an uncompressed original image sequence (also referred to as a reference image sequence) of image sequence 440 may be selected as the first reference image frame. Furthermore, an intermediate image frame corresponding to the first reference image frame may be selected from an encoded and decoded image sequence of the original image sequence (e.g., image sequence 440 shown in FIG. 4 ) to serve as the second reference image frame.

[0058] According to an example implementation of the present disclosure, a certain position can be specified and a first reference image frame and a second reference image frame can be selected from the original image sequence and the encoded-decoded image sequence, respectively. For example, an intermediate image frame at the i-th position (when the GoP includes 16 image frames, 2≤i≤15) can be selected. Specifically, the intermediate image frame at the i-th position can be selected from the original image sequence as the first reference image frame, and the intermediate image frame at the i-th position can be selected from the encoded-decoded image sequence as the second reference image frame. Further, the formula described above can be used to determine the corresponding forward mapping frame and / or backward mapping frame, and then determine the loss function for updating the response of the enhancement model 310.

[0059] According to an example implementation of the present disclosure, further details of the enhancement model 310 are described with reference to FIG5 . FIG5 shows a block diagram 500 of various networks in the enhancement model 310 according to some implementations of the present disclosure. The enhancement model 310 may include multiple networks: a bidirectional optical flow network 510, a feature extraction network 520, an aggregated attention network 530, and a quality enhancement network 540. Each network may be utilized to perform corresponding processing to determine the final enhancement residual.

[0060] First, a bidirectional optical flow network 510 is described. The head image frame 512, the middle image frame 514, and the tail image frame 516 determined based on the above method can be input to the bidirectional optical flow network 510. Furthermore, corresponding processing can be performed on each input image frame to provide input data to the subsequent feature extraction network 520. Specifically, the mapping module (e.g., the warp module) shown in FIG5 can be configured to perform a mapping operation. For example, the mapping operation can be performed based on the following formula.

[0061] In the above formula, Represents the forward mapping frame after mapping, represents the forward mapping frame determined based on the above formula, represents the head image frame, represents the noise introduced during the forward mapping process. Similarly, Represents the backward mapping frame after mapping, represents the backward mapping frame determined based on the above formula, represents the tail image frame, represents the noise introduced in the backward mapping process. and Input to the subsequent feature extraction network 520.

[0062] It should be understood that although FIG5 illustrates an example of using a bidirectional optical flow network 510 to process a head image frame 512, an intermediate data frame 514, and a tail image frame 516, alternatively and / or additionally, the functionality of the bidirectional optical flow network 510, or a portion thereof, can be performed outside of the enhancement model 310. For example, the intermediate image frames and the corresponding forward and backward mapping frames can be directly obtained outside of the enhancement model. In this case, the intermediate image frames and the corresponding forward and backward mapping frames can be directly input to the feature extraction module 520 for subsequent processing.

[0063] According to an example implementation of the present disclosure, a feature extraction network 520 in the enhancement model 310 can be used to extract features from the intermediate image frame and the mapping frame. As shown in FIG5 , the feature extraction network 520 may include a connection module (such as the concat module shown in FIG5 ) and a subsequent convolutional network. Eight (or other) layers of convolutional layers with ReLUs can be used to extract features 522. Specifically, the feature extraction network 520 may extract features based on the following formula:

[0064] In the above formula, F represents the feature extracted by the feature extraction network 520, which can include three components: F t ,F1,F2,Conv() represents the convolution operation, and They respectively represent the data output by the upstream bidirectional optical flow network 510.

[0065] Furthermore, the aggregated attention network 530 in the enhanced model 310 can be used to process the features 522 output by the upstream feature extraction network 520 to further determine the fused features 538 of the intermediate image frame. As shown in FIG5 , the aggregated attention network 530 can include a temporal attention network 532 and a spatial attention network 534. At this point, the temporal attention network 532 in the aggregated attention network 530 can be used to determine preliminary fused features 536 of the intermediate image frame based on the features 522. Furthermore, the spatial attention network 534 in the aggregated attention network 530 can be used to determine the final fused features 538 of the intermediate image frame based on the preliminary fused features 536.

[0066] As shown in FIG5 , the temporal attention network 532 may include a convolution module (e.g., the conv module in FIG5 ), a product module (e.g., the element-wise product module in FIG5 ), and an activation module (e.g., the sigmoid module in FIG5 ). These modules may generate preliminary fusion features 536 based on the following formula, that is, determine the associated attention weights of the intermediate image frames. ω1=sigmoid(θ(F t )⊙φ(F1)) Formula 7

[0067] In the above formula, F t , F1 respectively represent feature components from the feature extraction network, θ and φ respectively represent parameters of the convolutional network, ⊙ represents a multiplication operation, and sigmoid() represents a corresponding activation function. According to an exemplary implementation of the present disclosure, the specific values ​​of each parameter can be determined based on a variety of functional methods currently known and / or to be developed in the future, and thus will not be described in detail. Furthermore, a fusion function can be used to process the preliminary fused features 536 so that subsequent spatial attention-related processing can be performed.

[0068] According to an exemplary implementation of the present disclosure, similar processing can be performed on other components to obtain corresponding results. Furthermore, each initially fused feature obtained can be further processed based on the following formula.

[0069] In the above formula, F fusion Represents the result after fusion processing, FusionConv() represents the fusion function, and Respectively represent the output results of the time domain attention network 532. At this time, in the spatial domain attention network 534, the attention operation within the spatial domain can be further processed to obtain the final fusion feature 533. Specifically, the features obtained by the initial fusion are down-sampled twice by convolution, and then the spatial attention is obtained by up-sampling and summing from top to bottom. Finally, the fusion feature map is generated by element multiplication. According to an example implementation of the present disclosure, for example, the fusion feature 538 can be determined based on the following formula. F′=F fusion ↓2,F″=F′↓2 Formula 10

[0070] In the above formula, F′ and F″ represent the processing results after two downsampling operations, ↓2 represents the downsampling operation, ↑2 represents the upsampling operation, and “+” represents the addition operation performed element-wise. Represents the intermediate result of the sum operation, and ⊙ represents the product operation performed element-wise. In this way, the process of extracting fusion features can be converted into mathematical operations, so that the fusion feature 538 can accurately describe the management relationship between each image frame, thereby improving the accuracy of the entire enhancement network.

[0071] Furthermore, the aggregated attention network 530 can output the fused features 538 to the downstream quality enhancement network 540, and further utilize the quality enhancement network in the enhancement model to determine the enhanced residual based on the fused features. As shown in FIG5 , the quality enhancement network 540 can include a residual network, and can obtain the enhanced residual R through the convolution layer. QE Then, after the bidirectional guidance module and the residual, the enhanced residual for enhancing the intermediate image frame 514 can be generated using the following formula.

[0072] In the above formula, R QE represents the enhanced residual, represents the processing function of the residual network, and It represents the fusion feature 538 generated according to the above formula. Further, the following formula can be used to perform enhancement processing on the intermediate image frame to obtain an enhanced image frame.

[0073] In the above formula, represents an enhanced image frame with higher quality. represents the intermediate image frame 514, and R QE Indicates that it is used for Performing enhancement processing on the enhanced residual. Using the example implementation of the present disclosure, the enhancement model can be used to fully learn the association relationship between each image frame in the reference data, thereby enhancing the image quality in a more accurate manner.

[0074] According to an exemplary implementation of the present disclosure, each reference data is processed based on the specific network structure shown in Figure 5 to obtain a corresponding predicted value of the enhanced residual. Furthermore, the enhanced model can be updated in a manner that minimizes the difference between the predicted value and the true value of the enhanced residual based on the following loss function.

[0075] In the above formula, represents the loss function, The predicted value of the enhanced image frame after enhancement, and Represents the true value of an intermediate image frame in the original image sequence used as reference data. Using the exemplary implementations of this disclosure, a large number of image sequences can be used to construct reference samples for updating the enhancement model. In this way, the enhancement model can continuously output corresponding enhanced residuals and enhanced image frames in a more accurate direction.

[0076] According to an exemplary implementation of the present disclosure, the enhanced model may be continuously updated in an iterative manner until the enhanced model 310 satisfies a predetermined stopping condition. For example, training may be stopped when the loss function reaches a predetermined range, when the training process reaches a predetermined number of rounds, and so on.

[0077] According to an exemplary implementation of the present disclosure, once a trained enhancement model 310 has been obtained, it can be used to process each intermediate image frame in the decoded video one by one. For example, a set of enhanced image frames can be obtained for each set of intermediate image frames. Furthermore, an enhanced image sequence corresponding to the image sequence can be generated using the head image frame, the set of enhanced image frames, and the tail image frame. See FIG6 for more details on the enhancement process.

[0078] FIG6 illustrates a block diagram 600 for generating an enhanced image sequence according to some implementations of the present disclosure. As shown in FIG6 , an image sequence 120 represents compressed video, which may be obtained by encoding an original video using a variable resolution encoder. The original video has a higher first resolution. In image sequence 120, image frames at the beginning and end of the image sequence have a first resolution, while intermediate image frames at other locations have a lower second resolution. A decoding operation 150 may be performed on the compressed video to obtain a decoded image sequence 130. In this case, each image frame in image sequence 130 has a higher resolution, but intermediate image frames at intermediate locations have lower image quality.

[0079] The above-described process can be used to process each intermediate image frame one by one, thereby generating corresponding enhanced image frames 610, 612, ..., 614, and so on. Furthermore, these enhanced image frames 610, 612, ..., 614 and the extracted reference image frames with higher image quality at the head and tail positions can be used to generate an enhanced image sequence 620. In this case, each image frame in the enhanced image sequence has higher image quality, thereby improving the viewer's visual experience.

[0080] According to an exemplary implementation of the present disclosure, by combining variable-resolution encoding, high-resolution reconstructed frames can be used to guide quality enhancement of low-resolution reconstructed frames, and a bidirectional optical flow network can significantly improve the accuracy of motion information modeling. Furthermore, in the variable-resolution encoding process, several video frames are grouped together, with the first and last frames encoded at high resolution. This effectively aids video enhancement while saving bitrate.

[0081] It should be understood that the method described above can be used to process videos containing different scenes. For example, it can process videos of people, landscapes, animation scenes, game videos, and so on. Currently, a large amount of video content in game scenes, such as live video broadcasts and recorded broadcasts, has emerged. Generally speaking, compared to real scenes, animation scenes and game scenes are relatively simple, and the differences between the content of each frame in the video may be small. In this case, according to an example implementation of the present disclosure, it is more helpful to use high-definition reference image frames at the head and tail positions of the GoP to enhance the visual effects of the intermediate image frames.

[0082] Using the example implementations of this disclosure, the proposed method can improve the reconstruction quality of variable-resolution compression for animation and / or game videos, reducing encoding latency for animation transmission, game live streaming, and cloud gaming systems. Compared to traditional fixed-resolution or variable-resolution encoding, the proposed method can significantly improve both subjective and objective quality while also reducing the storage and transmission overhead of game videos to a certain extent.

[0083] According to an exemplary implementation of the present disclosure, for a specific animation scene and / or game scene, a pre-recorded video can be used to generate corresponding reference samples according to the method described above. Furthermore, these reference samples can be used to train enhancement models specifically for different animation scenes and / or game scenes. In this way, the accuracy and performance of the enhancement models can be further improved, thereby enhancing the audience's visual experience.

[0084] Example Process

[0085] FIG7 illustrates a flow chart of a method 700 for enhancing image quality according to some implementations of the present disclosure. At block 710, an image sequence is acquired, wherein the head and tail of the image sequence include a head image frame and a tail image frame, respectively, and each position in the image sequence other than the head and tail includes a set of intermediate image frames. At block 720, for an intermediate image frame in the set of intermediate image frames, a mapping frame is determined based on at least one of the head image frame and the tail image frame according to optical flow estimation. At block 730, the mapping frame is used to adjust the intermediate image frame to determine an enhanced image frame of the intermediate image frame.

[0086] According to an example implementation of the present disclosure, determining a mapping frame of the intermediate image frame includes: determining a forward mapping frame of the intermediate image frame according to optical flow estimation based on the head image frame.

[0087] According to an example implementation of the present disclosure, determining the mapping frame includes: determining a backward mapping frame of the intermediate image frame according to optical flow estimation based on the tail image frame.

[0088] According to an example implementation of the present disclosure, determining an enhanced image frame of an intermediate image frame includes: determining an enhanced residual of the intermediate image frame using an enhancement model, the enhancement model describing an association relationship between a first reference image frame, a second reference image frame corresponding to the first reference image frame, and a reference mapping frame of the second reference image frame; and determining the enhanced image frame based on the enhanced residual and the intermediate image frame.

[0089] According to an example implementation of the present disclosure, the enhancement model is obtained based on: determining a prediction value of the enhancement residual corresponding to the first reference image frame using the enhancement model based on the reference mapping frame; and determining the enhancement model based on the prediction value and the first reference image frame.

[0090] According to an exemplary implementation of the present disclosure, the first reference image frame is an intermediate reference image frame selected from a reference image sequence and located at a position other than the head and the tail.

[0091] According to an exemplary implementation of the present disclosure, the second reference image frame is an intermediate image frame corresponding to the first reference image frame selected from an encoded image sequence of the reference image sequence.

[0092] According to an example implementation of the present disclosure, using an enhancement model to determine an enhanced residual of an intermediate image frame includes: using a feature extraction network in the enhancement model to extract features from the intermediate image frame and a mapping frame; using a fusion attention network in the enhancement model to determine fused features of the intermediate image frame based on the features; and using a quality enhancement network in the enhancement model to determine an enhanced residual based on the fused features.

[0093] According to an example implementation of the present disclosure, the fusion features include: utilizing the temporal attention network in the fusion attention network to determine the preliminary fusion features of the intermediate image frame based on the features; and utilizing the spatial attention network in the fusion attention network to determine the fusion features of the intermediate image frame based on the preliminary fusion features.

[0094] According to an example implementation of the present disclosure, determining the enhanced residual of the intermediate image frame includes: utilizing a residual network in a quality enhancement network to determine the enhanced residual of the intermediate image frame based on fused features.

[0095] According to an example implementation of the present disclosure, the method further includes: acquiring a set of enhanced image frames of a set of intermediate image frames respectively; and generating an enhanced image sequence corresponding to the image sequence using the head image frame, the set of enhanced image frames, and the tail image frame.

[0096] According to an exemplary implementation of the present disclosure, an image sequence is obtained by decoding a compressed video using a variable resolution decoder, and the compressed video is obtained by encoding an original video using a variable resolution encoder.

[0097] According to an exemplary implementation of the present disclosure, the image quality of the head image frame and the tail image frame is higher than the image quality of a group of middle image frames.

[0098] Example devices and equipment

[0099] FIG8 shows a block diagram of an apparatus 800 for enhancing image quality according to some implementations of the present disclosure. The apparatus 800 includes: an acquisition module 810 configured to acquire an image sequence, wherein the head and tail of the image sequence include a head image frame and a tail image frame, respectively, and positions other than the head and tail of the image sequence include a set of intermediate image frames; a determination module 820 configured to determine, for an intermediate image frame in the set of intermediate image frames, a mapping frame of the intermediate image frame based on at least one of the head image frame and the tail image frame according to optical flow estimation; and an adjustment module 830 configured to adjust the intermediate image frame using the mapping frame to determine an enhanced image frame of the intermediate image frame.

[0100] According to an exemplary implementation of the present disclosure, the determination module 820 includes: a first determination module configured to determine a forward mapping frame of the intermediate image frame according to optical flow estimation based on the head image frame.

[0101] According to an example implementation of the present disclosure, the determination module 820 includes: a second determination module configured to determine the mapping frame, including: determining a backward mapping frame of the intermediate image frame according to optical flow estimation based on the tail image frame.

[0102] According to an example implementation of the present disclosure, the determination module 820 includes: a calling module, configured to determine an enhanced residual of the intermediate image frame using an enhancement model, the enhancement model describing the association relationship between a first reference image frame, a second reference image frame corresponding to the first reference image frame, and a reference mapping frame of the second reference image frame; and an enhancement module, configured to determine an enhanced image frame based on the enhanced residual and the intermediate image frame.

[0103] According to an example implementation of the present disclosure, the enhancement model is obtained based on: determining a prediction value of the enhancement residual corresponding to the first reference image frame using the enhancement model based on the reference mapping frame; and determining the enhancement model based on the prediction value and the first reference image frame.

[0104] According to an exemplary implementation of the present disclosure, the first reference image frame is an intermediate reference image frame selected from a reference image sequence and located at a position other than the head and the tail.

[0105] According to an exemplary implementation of the present disclosure, the second reference image frame is an intermediate image frame corresponding to the first reference image frame selected from an encoded image sequence of the reference image sequence.

[0106] According to an example implementation of the present disclosure, the calling module includes: an extraction module configured to extract features from the intermediate image frame and the mapping frame using a feature extraction network in the enhancement model; a fusion module configured to determine the fused features of the intermediate image frame based on the features using a fusion attention network in the enhancement model; and a residual determination module configured to determine the enhanced residual based on the fused features using a quality enhancement network in the enhancement model.

[0107] According to an example implementation of the present disclosure, a fusion module configured to determine fusion features includes: a time domain module configured to determine preliminary fusion features of the intermediate image frame based on features using a time domain attention network in a fusion attention network; and a spatial domain module configured to determine the fusion features of the intermediate image frame based on the preliminary fusion features using a spatial domain attention network in the fusion attention network.

[0108] According to an exemplary implementation of the present disclosure, the residual determination module: the enhanced residual determination module is configured to utilize the residual network in the quality enhancement network to determine the enhanced residual of the intermediate image frame based on the fusion features.

[0109] According to an example implementation of the present disclosure, the apparatus further includes: a video processing module configured to respectively acquire a set of enhanced image frames of a set of intermediate image frames; and a video generation module configured to generate an enhanced image sequence corresponding to the image sequence using the head image frame, the set of enhanced image frames, and the tail image frame.

[0110] According to an exemplary implementation of the present disclosure, an image sequence is obtained by decoding a compressed video using a variable resolution decoder, and the compressed video is obtained by encoding an original video using a variable resolution encoder.

[0111] According to an exemplary implementation of the present disclosure, the image quality of the head image frame and the tail image frame is higher than the image quality of a group of middle image frames.

[0112] FIG9 shows a block diagram of a device 900 capable of implementing various implementations of the present disclosure. It should be understood that the computing device 900 shown in FIG9 is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. The computing device 900 shown in FIG9 can be used to implement the methods described above.

[0113] As shown in FIG9 , computing device 900 is in the form of a general-purpose computing device. Components of computing device 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage devices 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processing unit 910 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 920. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 900.

[0114] The computing device 900 typically includes a plurality of computer storage media. Such media can be any available media accessible to the computing device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 920 can be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 930 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the computing device 900.

[0115] The computing device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 9 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 920 may include a computer program product 925 having one or more program modules configured to perform various methods or actions of various implementations of the present disclosure.

[0116] The communication unit 940 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 900 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 900 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0117] Input device 950 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 960 may be one or more output devices, such as a display, a speaker, or a printer. Computing device 900 may also communicate with one or more external devices (not shown) via communication unit 940, as needed, such as storage devices, display devices, or the like, with one or more devices that allow a user to interact with computing device 900, or with any device that allows computing device 900 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0118] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0119] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0120] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0121] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0122] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented using a dedicated hardware-based system that performs the function or action of the specification, or can be implemented using a combination of dedicated hardware and computer instructions.

[0123] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for enhancing image quality, comprising: Acquire an image sequence, wherein the head and the tail of the image sequence respectively include a head image frame and a tail image frame, and other positions other than the head and the tail in the image sequence respectively include a group of intermediate image frames; For an intermediate image frame in the group of intermediate image frames, determining a mapping frame of the intermediate image frame according to optical flow estimation based on at least any one of the head image frame and the tail image frame; as well as The intermediate image frame is adjusted using the mapping frame to determine an enhanced image frame of the intermediate image frame.

2. The method of claim 1 , wherein determining the mapping frame of the intermediate image frame comprises: Based on the head image frame, a forward mapping frame of the intermediate image frame is determined according to the optical flow estimation.

3. The method of claim 1 , wherein determining the mapping frame of the intermediate image frame comprises: Based on the tail image frame, a backward mapping frame of the intermediate image frame is determined according to the optical flow estimation.

4. The method of claim 1 , wherein determining the enhanced image frame of the intermediate image frame comprises: Determining an enhanced residual of the intermediate image frame using an enhancement model, wherein the enhancement model describes an association relationship between a first reference image frame, a second reference image frame corresponding to the first reference image frame, and a reference mapping frame of the second reference image frame; as well as Based on the enhanced residual and the intermediate image frame, the enhanced image frame is determined.

5. The method according to claim 4, wherein the enhanced model is obtained based on: Determining, based on the reference mapping frame, a prediction value of an enhancement residual corresponding to the first reference image frame using the enhancement model; and The enhancement model is determined based on the prediction value and the first reference image frame. 6 . The method according to claim 5 , wherein the first reference image frame is an intermediate reference image frame selected from a reference image sequence and located at a position other than a head and a tail.

7. The method according to claim 6, wherein the second reference image frame is an intermediate image frame corresponding to the first reference image frame selected from an encoded image sequence of the reference image sequence.

8. The method of claim 4, wherein determining the enhanced residual of the intermediate image frame using the enhanced model comprises: Extracting features from the intermediate image frame and the mapping frame using a feature extraction network in the enhancement model; Determining fusion features of the intermediate image frame based on the features using a fusion attention network in the enhancement model; as well as The enhancement residual is determined based on the fused features using a quality enhancement network in the enhancement model.

9. The method of claim 8, wherein determining the fusion feature comprises: Determine preliminary fusion features of the intermediate image frame based on the features using a temporal attention network in the fusion attention network; as well as The spatial attention network in the fusion attention network is used to determine the fusion features of the intermediate image frame based on the preliminary fusion features.

10. The method of claim 8, wherein determining the enhanced residual of the intermediate image frame comprises: The enhanced residual of the intermediate image frame is determined based on the fused features using a residual network in the quality enhancement network.

11. The method according to claim 1, further comprising: Respectively acquiring a group of enhanced image frames of the group of intermediate image frames; as well as An enhanced image sequence corresponding to the image sequence is generated using the head image frame, the group of enhanced image frames, and the tail image frame.

12. The method according to claim 1, wherein the image sequence is obtained by decoding a compressed video using a variable resolution decoder, and the compressed video is obtained by encoding an original video using a variable resolution encoder.

13. The method according to claim 1, wherein the image quality of the head image frame and the tail image frame is higher than the image quality of the group of intermediate image frames.

14. A device for enhancing image quality, comprising: an acquisition module configured to acquire an image sequence, wherein the head and the tail of the image sequence respectively include a head image frame and a tail image frame, and other positions other than the head and the tail in the image sequence respectively include a group of intermediate image frames; a determination module configured to determine, for an intermediate image frame in the group of intermediate image frames, a mapping frame of the intermediate image frame based on at least one of the head image frame and the tail image frame according to optical flow estimation; as well as An adjustment module is configured to adjust the intermediate image frame using the mapping frame to determine an enhanced image frame of the intermediate image frame.

15. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 13 when executed by the at least one processing unit.

16. A computer-readable storage medium having a computer program stored thereon, the computer program causing a processor to implement the method according to any one of claims 1 to 13 when the computer program is executed by the processor.

Images