Image resolution enhancement method and device, storage medium and electronic device

By acquiring the optical flow of the target frame and reference frame, performing feature extraction and video super-segment model processing, combined with upsampling technology, the problem of insufficient image resolution improvement in the existing technology is solved, and a higher quality image amplification meeting is achieved , improve image resolution and edge details.

CN114418845BActive Publication Date: 2024-11-15BEIJING OPPO TELECOMM CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111630872.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-28
Publication Date
2024-11-15
Estimated Expiration
2041-12-28

AI Technical Summary

Technical Problem

In the prior art, the image resolution enhancement method has poor accuracy and cannot effectively maintain the edges and details of the image, resulting in poor quality of the enlarged image.

Method used

By obtaining the optical flow between the target frame and the reference frame, feature extraction is performed, the resolution is improved using the pre-trained video super-score model, and combined with upsampling technology, the target super-score image is generated, making full use of time information to improve image resolution Rate.

Benefits of technology

The accuracy of the image resolution enhancement method is significantly improved, the edges and details of the image are maintained, the results are closer to the original high-resolution image, and the calculation complexity is relatively low.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114418845B_ABST
    Figure CN114418845B_ABST
Patent Text Reader

Abstract

The present disclosure relates to the field of image processing technology, and specifically to an image resolution enhancement method and device, a computer-readable storage medium, and an electronic device, the method comprising: obtaining a target frame and at least one reference frame, and determining the optical flow between the target frame and the reference frame; extracting features from the target frame and the reference frame to obtain a target feature image and a reference feature image; using a pre-trained video super-resolution model to enhance the resolution of the target feature image according to the target feature image, the reference feature image, and the optical flow to obtain a target intermediate super-resolution image; upsampling the target feature image by a first preset multiple to obtain a first reference image, and obtaining a target super-resolution image according to the first reference image and the intermediate super-resolution image. The technical solution of the embodiment of the present disclosure improves the accuracy of the image resolution enhancement method.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] With the development of computer and image processing technology, super-resolution technology of video images provides a good solution for enhancing and enlarging low-resolution videos. It can better enlarge low-resolution videos so that they can be displayed on high-resolution display devices.

[0003] Video image super-resolution is a very promising technology. Compared with traditional image interpolation methods, super-resolution methods can provide better image and video magnification effects. It can maintain the edges and details of the image during image magnification, thereby obtaining a result that is closer to the original high-resolution image. However, the accuracy of image resolution enhancement methods in the existing technology is poor.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide an image resolution enhancement method, an image resolution enhancement device, a computer-readable medium and an electronic device, thereby improving the accuracy of the image resolution enhancement method at least to a certain extent.

[0006] According to a first aspect of the present disclosure, there is provided an image resolution enhancement method, comprising: acquiring a target frame and at least one reference frame, and determining an optical flow between the target frame and the reference frame; performing feature extraction on the target frame and the reference frame to obtain a target feature image and a reference feature image; performing resolution enhancement on the target feature image using a pre-trained video super-resolution model according to the target feature image, the reference feature image and the optical flow to obtain a target intermediate super-resolution image; performing upsampling on the target feature image by a first preset multiple to obtain a first reference image, and obtaining a target super-resolution image according to the first reference image and the intermediate super-resolution image.

[0007] According to a second aspect of the present disclosure, there is provided an image resolution enhancement device, comprising: an image acquisition module, configured to acquire a target frame and at least one reference frame, and determine an optical flow between the target frame and the reference frame; a feature extraction module, configured to perform feature extraction on the target frame and the reference frame to obtain a target feature image and a reference feature image; an image super-resolution module, configured to perform resolution enhancement on the target feature image according to the target feature image, the reference feature image and the optical flow using a pre-trained video super-resolution model to obtain a target intermediate super-resolution image; and an image fusion module, configured to upsample the target frame to obtain a first reference image, and obtain a target super-resolution image according to the first reference image and the intermediate super-resolution image.

[0008] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which enables the one or more processors to implement the above method when the one or more programs are executed by the one or more processors.

[0010] An image resolution enhancement method provided by an embodiment of the present disclosure obtains a target frame and at least one reference frame, and determines the optical flow between the target frame and the reference frame; performs feature extraction on the target frame and the reference frame to obtain a target feature image and a reference feature image; performs resolution enhancement on the target feature image and the reference frame according to the target feature image, the reference feature image and the optical flow using a pre-trained video super-resolution model to obtain a target intermediate super-resolution image; performs upsampling on the target feature image by a first preset multiple to obtain a first reference image, and obtains a target super-resolution image based on the first reference image and the intermediate super-resolution image. Compared with the prior art, the target feature image and the reference feature image are enhanced in resolution based on the target feature image, the reference feature image and the optical flow using a pre-trained video super-resolution model to obtain an intermediate super-resolution image, which can make better use of time information, has stronger expressiveness and improves accuracy. The target feature image is further upsampled by a first preset multiple to obtain a first reference image, and the target super-resolution image is obtained based on the first reference image and the intermediate super-resolution image. The target feature image is used as a reference to further improve the accuracy of the image resolution enhancement method.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0013] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0014] Figure 2 A flowchart of a method for improving image resolution in an exemplary embodiment of the present disclosure is schematically shown;

[0015] Figure 3 The overall data flow diagram of an image resolution enhancement method in an exemplary embodiment of the present disclosure is schematically shown;

[0016] Figure 4 A schematic diagram showing a structure of a video super-resolution model in an image resolution enhancement method in an exemplary embodiment of the present disclosure;

[0017] Figure 5 A data flow diagram of a video super-resolution model in a second direction optical flow in an exemplary embodiment of the present disclosure is schematically shown;

[0018] Figure 6 A data flow diagram of a video super-resolution model in a first direction optical flow in an exemplary embodiment of the present disclosure is schematically shown;

[0019] Figure 7 A schematic diagram of the structure of a transformer layer in an exemplary embodiment of the present disclosure is schematically shown;

[0020] Figure 8 A diagram schematically shows a result of an ablation experiment in an exemplary embodiment of the present disclosure;

[0021] Fig. 9 A diagram schematically showing a comparison between an image resolution enhancement method and other methods in an exemplary embodiment of the present disclosure;

[0022] Fig.10 A data diagram schematically showing a comparison between an image resolution enhancement method and other methods in an exemplary embodiment of the present disclosure;

[0023] Fig.11 A schematic diagram schematically shows the composition of an image resolution improvement device in an exemplary embodiment of the present disclosure;

[0024] Fig.12 A schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0027] Figure 1 1 shows a schematic diagram of a system architecture, which may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, etc. The server 120 generally refers to a background system that provides XXX related services in this exemplary embodiment, and may be a server or a cluster formed by multiple servers. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to perform data exchange.

[0028] In one implementation, the above-mentioned image resolution enhancement method may be performed by the terminal 110. For example, after a user uses the terminal 110 to shoot an image or the user selects a target frame and at least one reference frame in an album of the terminal 110, the terminal 110 enhances the image resolution and outputs a target super-resolution image.

[0029] In one implementation, the image resolution enhancement method may be performed by the server 120. For example, after a user uses the terminal 110 to shoot an image or the user selects a target frame and at least one reference frame in the photo album of the terminal 110, the terminal 110 uploads the image to the server 120, and the server 120 enhances the image resolution and returns the target super-resolution image to the terminal 110.

[0030] As can be seen from the above, the execution subject of the image resolution enhancement method in this exemplary embodiment may be the terminal 110 or the server 120, and the present disclosure does not limit this.

[0031] The exemplary embodiments of the present disclosure also provide an electronic device for executing the above-mentioned image resolution enhancement method, and the electronic device may be the above-mentioned terminal 110 or the server 120. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned image resolution enhancement method by executing the executable instructions.

[0032] In related technologies, according to the model scale, there are three modes: upsampling, downsampling and mixed sampling. Direct upsampling to a high dimension and then further extracting feature expressions will have a high computational complexity; downsampling generally performs feature calculations at a low scale and finally upsamples to a high dimension, which reduces the complexity, but the model selection at the low-dimensional stage has a great impact on the results.

[0033] From another perspective, it can also be divided into single-frame super-resolution and multi-frame super-resolution. The former is to obtain a high-resolution image based on the information of a picture, and the latter is to use multiple frames of time-related pictures to obtain a high-resolution image of a certain frame. In the time series model, the alignment mechanism is very important, otherwise artifacts are likely to appear. Feature expression has a very important impact on the results, but generally speaking, good feature expression requires more CNN layers, which also means greater complexity. The introduction of time series information plus a suitable alignment mechanism is beneficial to the super-resolution results. However, the introduction of time series information will also increase the computational complexity.

[0034] Combine the following Figure 2 The image resolution enhancement method in this exemplary embodiment is described. Figure 2 An exemplary process of the method for improving image resolution is shown, which may include:

[0035] Step S210, acquiring a target frame and at least one reference frame, and determining the optical flow between the target frame and the reference frame;

[0036] Step S220, extracting features from the target frame and the reference frame to obtain a target feature image and a reference feature image;

[0037] Step S230, according to the target feature image, the reference feature image and the optical flow, using a pre-trained video super-resolution model to improve the resolution of the target feature image to obtain a target intermediate super-resolution image;

[0038] Step S240 , upsampling the target feature image by a first preset multiple to obtain a first reference image, and obtaining a target super-resolution image according to the first reference image and the intermediate super-resolution image.

[0039] Based on the above method, compared with the existing technology, the target feature image and the intermediate super-resolution image are obtained by improving the resolution of the target feature image using a pre-trained video super-resolution model according to the target feature image, the reference feature image and the optical flow, which can make better use of time information, have stronger expressiveness and improve accuracy. The target feature image is further up-sampled by a first preset multiple to obtain a first reference image, and the target super-resolution image is obtained based on the first reference image and the intermediate super-resolution image. The target feature image is used as a reference to further improve the accuracy of the image resolution improvement method.

[0040] Below Figure 2 Each step in the procedure is described in detail.

[0041] refer to Figure 2 In step S210, a target frame and at least one reference frame are acquired, and an optical flow between the target frame and the reference frame is determined.

[0042] In an exemplary embodiment, in an exemplary embodiment, a target frame and at least one reference frame may be obtained as input data, wherein the input data may include multiple frames of continuous images, that is, the reference image includes multiple, for example, three frames of continuous images, four frames of continuous graphics or more frames of continuous images, etc., and the use of continuous images can achieve a better effect of improving the video resolution. In another exemplary embodiment, the input data may also include multiple frames of non-continuous images, and the multiple frames of images are not specifically limited in this exemplary embodiment.

[0043] In this example implementation, when the output data is three consecutive frames of images, the reference frame may be the previous frame and the next frame of the target frame, that is, the target frame is the middle frame of the input data. If the input data is an even-numbered frame, the middle frame may include two frames, wherein the target frame may be any one of the two frames, or the average of the two frames. The other frames are all reference frames.

[0044] In this exemplary embodiment, the input data includes 3 frames of continuous images for detailed description, that is, the reference frame includes the previous frame of the target frame and the next frame of the target frame for description.

[0045] In step S220, feature extraction is performed on the target frame and the reference frame to obtain a target feature image and a reference feature image.

[0046] In this example implementation, the server may first divide the target frame and the reference frame according to a preset rule, and may divide the target frame and the reference frame into 20, 50 or 69 parts, etc., or may customize the parts according to user needs, which is not specifically limited in this example implementation.

[0047] It should be noted that the target frame and the reference frame are segmented the same number of times, and the segmented target frames correspond one-to-one to the reference frames.

[0048] In this example implementation, refer to Figure 3 As shown, after segmentation, a feature pre-extraction module can be used to extract features from the segmented target frame and the reference frame to obtain a target feature image and a reference feature image. j-1 ,x j ,andx j+1 Represents 3 consecutive low-resolution images, F0,j-1 ,F 0,j ,andF 0,j+1 Represents from x j-1 ,x j ,andx j+1 The target feature image and the reference feature image are obtained after the feature pre-extraction module 310. These pre-extracted features are sent to the SR module, i.e., the video super-resolution module.

[0049] In this example implementation, the feature pre-extraction module 310 may be composed of 5 residual blocks, which can extract shallow features of low-resolution images. The specific structure of the feature pre-extraction module 310 can also be customized according to user needs, and is not specifically limited in this example implementation.

[0050] In step S230, the target feature image is subjected to resolution enhancement using a pre-trained video super-resolution model according to the target feature image, the reference feature image and the optical flow to obtain a target intermediate super-resolution image.

[0051] In this example implementation, the target feature image can be subjected to multi-stage resolution enhancement using a pre-trained video super-resolution model 330 according to the target feature image, the reference feature image, and the optical flow to obtain an intermediate super-resolution image, wherein the above-mentioned stages can be 4, 5, 6, etc., and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0052] In this example embodiment, the above-mentioned optical flow may include a first-directional optical flow and a second-directional optical flow, wherein the first-directional optical flow may be a forward time optical flow, and the second-directional optical flow may be a reverse time optical flow, that is, the first-directional optical flow is the optical flow from the previous frame to the next frame, and the second-directional optical flow is the optical flow from the next frame to the previous frame.

[0053] In this example, refer to Figure 4 As shown, in the resolution improvement process of the above multiple stages, if the stage is an odd-numbered stage, the optical flow is the second-direction optical flow, and if the stage is an even-numbered stage, the optical flow is the first-direction optical flow. The hybrid two-way propagation mechanism is adopted, which can make fuller use of time information and has stronger expressiveness.

[0054] It should be noted that in Figure 4 In the above example, directly using the reference frame features to integrate into the target frame will lead to deviations. Using optical flow to warp the reference frame, that is, Figure 4 Flow i,j ;p i (·) represents the i-th forward propagation node, Figure 4 where i = [1, 2, 3, 4]. i,jRepresents the output feature of the target feature image after passing through the i-th forward propagation function. j Represents the intermediate super-resolution image of the target feature image frame. Then, there is the following formula:

[0055] F i,j =p i (x j ),F j =SR(x j )

[0056] Low resolution image x j The SR module processing process can be described as

[0057] F j =SR(F 0,j-1 ,F 0,j ,F 0,j+1 ,flow j+1,j ,flow i-1,j )

[0058] SR(·) represents the SR module of super-resolution, i.e., the video super-resolution model, flow j+1,j Represents x j+1 to x j The optical flow is the second direction optical flow. SR is composed of multiple forward propagations and backward propagations, specifically:

[0059] The forward propagation node is F i,j =P i (F 0,j+1 ,F 0,j ,F i-1,j ,F i,j-1 ,flow j-1,j )

[0060] The back propagation node is F i,j =P i (F 0,j-1 ,F 0,j ,F i-1,j ,F i,j+1 ,flow j+1,j )

[0061] After the output of the last layer, it is fused with the output of all previous layers to obtain

[0062] F h =C(f 1,j ,f 2,j ,F 3,j ,F 4,j )

[0063] C stands for concatenation operation, that is, fusion operation.

[0064] In this example implementation, refer to Figure 5 As shown, in response to the optical flow being a first direction optical flow, the target feature image, the reference feature image corresponding to the next frame of the target frame, the feature image corresponding to the reference frame of the previous frame of the target frame at the current stage, and the reference intermediate super-resolution image of the previous stage can be input into the pre-trained video super-resolution model to enhance the resolution of the target feature image to obtain the reference intermediate super-resolution image of the current stage.

[0065] Specifically, a three-dimensional convolutional layer 520 is used to perform a convolution operation on the target feature image, the reference feature image corresponding to the next frame of the target frame, and the feature image corresponding to the reference frame of the previous frame of the target frame in the current stage to obtain a first feature image; the feature image and the optical flow corresponding to the reference frame of the previous frame of the target frame in the current stage are changed to obtain a second feature image; a fusion operation, i.e., a concatenation operation, is performed on the second feature image, the reference intermediate super-resolution image of the previous stage, and the first feature image to obtain a third feature image; at least one transformer layer 520 is used to perform feature transformation on the third feature image to obtain a first super-resolution image to be output; and the first super-resolution image to be output is upsampled by a second preset multiple to obtain the reference intermediate super-resolution image of the current stage.

[0066] Or, refer to Figure 6 As shown, the response optical flow is the second direction optical flow, and the target feature image, the reference feature image corresponding to the previous frame of the target frame, the feature image corresponding to the reference frame of the next frame of the target frame in the current stage, and the reference intermediate super-resolution image in the previous stage are input into the pre-trained video super-resolution model to enhance the resolution of the target feature image to obtain the reference intermediate super-resolution image in the current stage.

[0067] Specifically, a three-dimensional convolutional layer 510 is used to perform a convolution operation on the target feature image, the reference feature image corresponding to the previous frame of the target frame, and the feature image corresponding to the reference frame of the next frame of the target frame in the current stage to obtain a fourth feature image; the feature image and the optical flow corresponding to the reference frame of the previous frame of the target frame in the current stage are changed to obtain a fifth feature image; a fusion operation, i.e., a concatenation operation, is performed on the fourth feature image, the reference intermediate super-resolution image of the previous stage, and the fifth feature image to obtain a sixth feature image; at least one transformer layer 520 is used to perform feature transformation on the sixth feature image to obtain a second super-resolution image to be output; and the second super-resolution image to be output is upsampled by a second preset multiple to obtain the reference intermediate super-resolution image of the current stage.

[0068] In this example implementation, refer to Figure 7As shown, when at least one transformer layer 520 is used to perform feature transformation on the third feature image to obtain the first super-resolution image to be output, the self-attention mechanism layer ( Figure 7 Attentionceng1) in the Q matrix, V matrix and K matrix, and then use the self-attention mechanism and feedforward neural network ( Figure 7 Feed Forward in ( ) obtains the first super-resolution image to be output.

[0069] Similarly, when using at least one transformer layer to perform feature transformation on the sixth feature image to obtain a second super-resolution image to be output, the Q matrix, V matrix and K matrix in the self-attention mechanism layer in the transformer layer can be first calculated based on the above-mentioned sixth feature image, and then the self-attention mechanism and the feedforward neural network can be used to obtain the above-mentioned second super-resolution image to be output.

[0070] In this example embodiment, after obtaining the above-mentioned first super-resolved image to be output and the second super-resolved image to be output, the above-mentioned first super-resolved image to be output can be upsampled by a second preset multiple to obtain a reference intermediate feature image of the current stage, or the above-mentioned second super-resolved image to be output can be upsampled by a second preset multiple to obtain a reference intermediate feature image of the current stage.

[0071] The second preset multiple may be 2 times, 4 times, etc., and may also be customized according to user needs, which is not specifically limited in this example implementation.

[0072] In this example implementation, at least one upsampling convolution layer may be determined according to the second preset multiple, and the upsampling process may be completed using the at least one upsampling convolution layer pair.

[0073] Specifically, at least one upsampling convolution layer is used to increase the number of channels of the first super-resolution image to be output to obtain the first channel image to be merged, and the first channel image to be merged can be transferred to the height and width of the first feature image to be output to obtain the reference intermediate super-resolution image of the current stage. Specifically, pixel_shuffle can be used to 'transfer' the first channel image to be merged to the width and height of the first feature image to be output. If the preset multiple is 2, the height and width of the obtained target image are both twice that of the first feature image to be output.

[0074] Similarly, at least one upsampling convolution layer is used to increase the number of channels of the second super-resolution image to be output to obtain the second channel image to be merged, and the second channel image to be merged can be transferred to the height and width of the second feature image to be output to obtain the reference intermediate super-resolution image of the current stage. Specifically, pixel_shuffle can be used to 'transfer' the second channel image to be merged to the width and height of the second feature image to be output. If the preset multiple is 2, the height and width of the target image are twice that of the second feature image to be output.

[0075] In step S240, the target feature image is upsampled by a first preset multiple to obtain a first reference image, and a target super-resolution image is obtained according to the first reference image and the intermediate super-resolution image.

[0076] In this example implementation, the above-mentioned first reference image can be obtained by the upsampling module 320 of the above-mentioned target feature image at a first preset multiple, wherein the above-mentioned first preset multiple can be 2 times, four times, etc., and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0077] After obtaining the first reference image, the first reference image and the target intermediate super-resolution image may be added to obtain the target super-resolution image.

[0078] In this example implementation, the technical effect of the image resolution enhancement method proposed in the present disclosure can be described in detail through ablation experiments. Figure 8 As shown, Base is a simplified version of basicVSR, and MBP stands for Figure 4 The propagation structure of LGM represents the proposed 3Dresblocks that collects multiple information, LVT is the transformer module, and MDF is the reconstruction module. PSNR and SSIM are two indicators to measure the super-resolution effect. Each proposed module has a gain on the final result. Base is a simplified version of basicVSR[2], which reduces the number of resblocks from 30 to 20. On this basis, we further reduce the number of 3D-Res-Blocks layers in the LGM module, which can be reduced to a complexity similar to that of basicVSR. At this time, PSNR is still higher than basicVSR. Fig. 9 Shown Fig.10 As shown in the figure, the image resolution enhancement method proposed in this paper, namely MBT, is compared with other methods. It can be seen that our proposed method has a significant improvement on multiple datasets including REDS4 and VID4, and is also very competitive on vimeo-90K, with the number of parameters and computational complexity 67% lower than SOTA. FLOPs / frame indicates the FLOPs corresponding to each frame output.

[0079] To sum up, in this exemplary embodiment, compared with the prior art, the target feature image is imaged and the resolution is improved by using a pre-trained video super-resolution model according to the target feature image, the reference feature image and the optical flow to obtain an intermediate super-resolution image, which can make better use of time information, have stronger expressiveness and improve accuracy. The target feature image is further up-sampled by a first preset multiple to obtain a first reference image, and the target super-resolution image is obtained based on the first reference image and the intermediate super-resolution image. The target feature image is used as a reference to further improve the accuracy of the image resolution improvement method. The hybrid bidirectional optical flow propagation mechanism is further adopted to make better use of time information and have stronger expressiveness. Furthermore, the feature conversion performed by the transformer layer can more accurately capture the global information of the image and improve the accuracy of the resolution improvement.

[0080] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0081] For further reference, Fig.11 As shown, in the implementation of this example, an image resolution enhancement device 1100 is also provided, including an image acquisition module 1110, a feature extraction module 1120, an image super-resolution module 1130 and an image fusion module 1140. Among them:

[0082] The image acquisition module 1110 can be used to acquire a target frame and at least one reference frame, and determine the optical flow between the target frame and the reference frame. The optical flow includes a first direction optical flow and a second direction optical flow; wherein the first direction optical flow is a forward time optical flow; and the second direction optical flow is a reverse time optical flow. The reference frame includes a previous frame of the target frame and a subsequent frame of the target frame.

[0083] The feature extraction module 1120 can be used to extract features from the target frame and the reference frame to obtain a target feature image and a reference feature image. Specifically, the target frame is segmented according to a preset rule and feature extraction is performed to obtain a plurality of target feature images; the reference frame is segmented according to a preset rule and feature extraction is performed to obtain a plurality of reference feature images.

[0084] The image super-resolution module 1130 may be configured to improve the resolution of the target feature image, the reference feature image, and the corresponding optical flow using a pre-trained video super-resolution model to obtain an intermediate super-resolution image.

[0085] Specifically, according to the target feature image, the reference feature image and the optical flow, a pre-trained video super-resolution model is used to perform multi-stage resolution enhancement on the target feature image to obtain a target intermediate super-resolution image.

[0086] The image super-resolution module 1130 performs the following steps for each stage:

[0087] In response to the optical flow being a first direction optical flow, the target feature image, the reference feature image corresponding to the next frame of the target frame, the feature image corresponding to the reference frame of the previous frame of the target frame in the current stage, and the reference intermediate super-resolution image in the previous stage are input into the pre-trained video super-resolution model to enhance the resolution of the target feature image to obtain the reference intermediate super-resolution image in the current stage; the reference intermediate super-resolution images of each stage are fused to obtain the target intermediate super-resolution image, wherein the intermediate super-resolution image of the 0th stage is the target feature image.

[0088] Specifically, if the stage is an odd-numbered stage, the optical flow is a second-direction optical flow; if the stage is an even-numbered stage, the optical flow is a first-direction optical flow. A three-dimensional convolutional layer is used to perform convolution operations on the target feature image, the reference feature image corresponding to the next frame of the target frame, and the feature image corresponding to the reference frame of the previous frame of the target frame in the current stage to obtain a first feature image; the feature image corresponding to the reference frame of the previous frame of the target frame in the current stage and the optical flow are changed to obtain a second feature image; the second feature image, the reference intermediate super-resolution image of the previous stage, and the first feature image are combined to obtain a third feature image; at least one transformer layer is used to perform feature conversion on the third feature image to obtain a first super-resolution image to be output; the first super-resolution image to be output is upsampled by a second preset multiple to obtain the reference intermediate super-resolution image of the current stage. Or

[0089] The image super-resolution module 1130 responds to the optical flow being the second direction optical flow, and inputs the target feature image, the reference feature image corresponding to the previous frame of the target frame, the feature image corresponding to the reference frame of the next frame of the target frame in the current stage, and the reference intermediate super-resolution image of the previous stage into the pre-trained video super-resolution model to enhance the resolution of the target feature image to obtain the reference intermediate super-resolution image of the current stage; and fuses the reference intermediate super-resolution images of each stage to obtain the target intermediate super-resolution image.

[0090] Specifically, a three-dimensional convolutional layer is used to perform a convolution operation on the target feature image, the reference feature image corresponding to the previous frame of the target frame, and the feature image corresponding to the reference frame of the next frame of the target frame in the current stage to obtain a fourth feature image; the feature image corresponding to the reference frame of the previous frame of the target frame in the current stage and the optical flow are changed to obtain a fifth feature image; the fourth feature image, the reference intermediate super-resolution image of the previous stage and the fifth feature image are combined to obtain a sixth feature image; at least one transformer layer is used to perform feature transformation on the sixth feature image to obtain a second super-resolution image to be output; the second super-resolution image to be output is upsampled by a second preset multiple to obtain the reference intermediate super-resolution image of the current stage.

[0091] The image fusion module 1140 may be used to upsample the target feature image to obtain a first reference image, and obtain a target super-resolution image based on the first reference image and the intermediate super-resolution image. Specifically, the first reference image and the target intermediate super-resolution image are added to obtain the target super-resolution image.

[0092] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0093] Below Fig.12 The mobile terminal 1200 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Fig.12 The construction in can also be applied to fixed type equipment.

[0094] like Fig.12 As shown, the mobile terminal 1200 may specifically include: a processor 1201, a memory 1202, a bus 1203, a mobile communication module 1204, an antenna 1, a wireless communication module 1205, an antenna 2, a display screen 1206, a camera module 1207, an audio module 1208, a power module 1209 and a sensor module 1210.

[0095] The processor 1201 may include one or more processing units, for example, the processor 1210 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. The image resolution enhancement method in this exemplary embodiment may be executed by an AP, a GPU or a DSP, and when the method involves processing related to a neural network, it may be executed by an NPU.

[0096] The encoder can encode (i.e. compress) an image or video, for example, the target image can be encoded into a specific format to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of the image or video to restore the image or video data, such as reading the encoded data of the target image, decoding it through the decoder to restore the data of the target image, and then perform relevant processing on the data to improve the resolution. The mobile terminal 1200 can support one or more encoders and decoders. In this way, the mobile terminal 1200 can process images or videos in multiple encoding formats, such as: JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap) and other image formats, MPEG (Moving Picture Experts Group) 1, MPEG12, H.1263, H.1264, HEVC (High Efficiency Video Coding) and other video formats.

[0097] The processor 1201 may be connected to the memory 1202 or other components via a bus 1203 .

[0098] The memory 1202 may be used to store computer executable program codes, which include instructions. The processor 1201 executes various functional applications and data processing of the mobile terminal 1200 by running the instructions stored in the memory 1202. The memory 1202 may also store application data, such as images, videos and other files.

[0099] The communication function of the mobile terminal 1200 can be implemented by the mobile communication module 1204, the antenna 1, the wireless communication module 1205, the antenna 12, the modulation and demodulation processor, and the baseband processor. The antenna 1 and the antenna 12 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1204 can provide 12G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1200. The wireless communication module 1205 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 1200.

[0100] The display screen 1206 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 1207 is used to implement shooting functions, such as shooting images, videos, etc. The audio module 1208 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 1209 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 1210 may include a depth sensor 12101, a pressure sensor 12102, a gyroscope sensor 12103, an air pressure sensor 12104, etc., to implement corresponding sensing detection functions.

[0101] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0102] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification.

[0103] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. Examples of computer-readable storage media based on the body may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0104] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0105] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0106] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0107] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for improving image resolution, characterized in that: include: Acquire a target frame and at least one reference frame, and determine the optical flow between the target frame and the reference frame; The optical flow includes a first direction optical flow and a second direction optical flow; wherein the first direction optical flow is a forward time optical flow; and the second direction optical flow is a reverse time optical flow; Performing feature extraction on the target frame and the reference frame to obtain a target feature image and a reference feature image; According to the target feature image, the reference feature image and the optical flow, the resolution of the target feature image is improved by using a pre-trained video super-resolution model to obtain a target intermediate super-resolution image; The target feature image is upsampled by a first preset multiple to obtain a first reference image, and the first reference image is added to the target intermediate super-resolution image to obtain a target super-resolution image.

2. The method according to claim 1, characterized in that The step of improving the resolution of the target feature image by using a pre-trained video super-resolution model according to the target feature image, the reference feature image and the optical flow to obtain a target intermediate super-resolution image comprises: According to the target feature image, the reference feature image and the optical flow, a pre-trained video super-resolution model is used to perform multi-stage resolution enhancement on the target feature image to obtain a target intermediate super-resolution image.

3. The method according to claim 2, characterized in that The reference frame includes a previous frame of the target frame and a subsequent frame of the target frame, and the step of performing multi-stage resolution enhancement on the target feature image using a pre-trained video super-resolution model according to the target feature image, the reference feature image, and the optical flow to obtain a target intermediate super-resolution image includes: For each stage, perform the following steps: In response to the optical flow being a first direction optical flow, the target feature image, the reference feature image corresponding to the next frame of the target frame, the feature image corresponding to the reference frame of the previous frame of the target frame at the current stage, and the reference intermediate super-resolution image at the previous stage are input into the pre-trained video super-resolution model to perform resolution enhancement on the target feature image to obtain the reference intermediate super-resolution image at the current stage; or In response to the optical flow being the second direction optical flow, the target feature image, the reference feature image corresponding to the previous frame of the target frame, the feature image corresponding to the reference frame of the next frame of the target frame at the current stage, and the reference intermediate super-resolution image at the previous stage are input into the pre-trained video super-resolution model to improve the resolution of the target feature image, so as to obtain the reference intermediate super-resolution image at the current stage; The reference intermediate super-resolution images of each stage are fused to obtain the target intermediate super-resolution image; Among them, the intermediate super-resolution image in the 0th stage is the target feature image.

4. The method according to claim 3, characterized in that If the stage is an odd-numbered stage, the optical flow is the second-directional optical flow, and if the stage is an even-numbered stage, the optical flow is the first-directional optical flow.

5. The method according to claim 3, characterized in that: The step of inputting the target feature image, the reference feature image corresponding to the next frame of the target frame, the feature image corresponding to the reference frame of the next frame of the target frame at the current stage, and the reference intermediate super-resolution image at the previous stage into the pre-trained video super-resolution model to improve the resolution of the target feature image to obtain the reference intermediate super-resolution image at the current stage includes: Using a three-dimensional convolution layer, a convolution operation is performed on the target feature image, a reference feature image corresponding to a frame after the target frame, and a feature image corresponding to a reference frame before the target frame at the current stage to obtain a first feature image; The feature image corresponding to the reference frame of the previous frame of the target frame in the current stage and the optical flow are changed to obtain a second feature image; Performing a fusion operation on the second feature image, the reference intermediate super-resolution image of the previous stage, and the first feature image to obtain a third feature image; Using at least one transformer layer to perform feature transformation on the third feature image to obtain a first super-resolved image to be output; The first to-be-output super-resolution image is up-sampled by a second preset multiple to obtain the reference intermediate super-resolution image of the current stage.

6. The method according to claim 3, characterized in that: The target feature image, the reference feature image corresponding to the previous frame of the target frame, the feature image corresponding to the reference frame of the next frame of the target frame at the current stage, and the reference intermediate super-resolution image at the previous stage are input into the pre-trained video super-resolution model to improve the resolution of the target feature image to obtain the reference intermediate super-resolution image at the current stage, including: Using a three-dimensional convolution layer, a convolution operation is performed on the target feature image, a reference feature image corresponding to a previous frame of the target frame, and a feature image corresponding to a reference frame of a subsequent frame of the target frame in the current stage to obtain a fourth feature image; The fifth feature image is obtained by changing the feature image and the optical flow corresponding to the reference frame one frame after the target frame in the current stage; Performing a fusion operation on the fourth feature image, the reference intermediate super-resolution image of the previous stage, and the fifth feature image to obtain a sixth feature image; Using at least one transformer layer to perform feature transformation on the sixth feature image to obtain a second super-resolved image to be output; The second to-be-output super-resolution image is up-sampled by a second preset multiple to obtain the reference intermediate super-resolution image of the current stage.

7. The method according to claim 1, characterized in that The extracting features from the target frame and the reference frame to obtain a target feature image and a reference feature image comprises: The target frame is segmented according to a preset rule and features are extracted to obtain a plurality of target feature images; The reference frame is segmented according to a preset rule and features are extracted to obtain a plurality of reference feature images.

8. An image resolution enhancement device, characterized in that: include: An image acquisition module, used to acquire a target frame and at least one reference frame, and determine the optical flow between the target frame and the reference frame; The optical flow includes a first direction optical flow and a second direction optical flow; wherein the first direction optical flow is a forward time optical flow; and the second direction optical flow is a reverse time optical flow; A feature extraction module, used for extracting features from the target frame and the reference frame to obtain a target feature image and a reference feature image; An image super-resolution module is used to improve the resolution of the target feature image by using a pre-trained video super-resolution model according to the target feature image, the reference feature image and the corresponding optical flow to obtain a target intermediate super-resolution image; The image fusion module is used to upsample the target feature image to obtain a first reference image, and add the first reference image to the target intermediate super-resolution image to obtain a target super-resolution image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image resolution improvement method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the image resolution enhancement method as described in any one of claims 1 to 7.