Video resolution enhancement method and device, storage medium and electronic device

Abstract

The present disclosure relates to the field of image processing technology, and specifically to a method and device for improving video resolution, a computer-readable storage medium, and an electronic device, the method comprising: obtaining a target frame and at least one reference frame; merging the target frame and at least one reference frame into one frame to obtain a first feature image, the number of channels of the first feature image being the same as the number of channels of the target frame; extracting features from the first feature image to obtain a second feature image including edge features; obtaining an image to be output according to the target frame and the second feature image; and upsampling the image to be output by a preset multiple to obtain the target image. The technical solution of the embodiment of the present disclosure reduces the complexity of improving video resolution.

Description

Background Art

[0002] With the development of computer and image processing technology, super-resolution technology of video images provides a good solution for enhancing and enlarging low-resolution videos. It can better enlarge low-resolution videos so that they can be displayed on high-resolution display devices.

[0003] Video image super-resolution is a very promising technology. Compared with traditional image interpolation methods, super-resolution methods can provide better image and video magnification effects. It can maintain the edges and details of the image during image magnification, thereby obtaining a result closer to the original high-resolution image. However, current super-resolution models are generally heavy models that cannot achieve real-time processing, and have high computational complexity and cannot be deployed on the mobile terminal side.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a video resolution enhancement method, a video resolution enhancement device, a computer-readable medium and an electronic device, thereby reducing the computational complexity of video resolution enhancement at least to a certain extent.

[0006] According to a first aspect of the present disclosure, a video resolution enhancement method is provided, comprising: acquiring a target frame and at least one reference frame; merging the target frame and at least one reference frame into one frame to obtain a first feature image, wherein the number of channels of the first feature image is the same as the number of channels of the target frame; performing feature extraction on the first feature image to obtain a second feature image including edge features; obtaining an image to be output based on the target frame and the second feature image; and upsampling the image to be output by a preset multiple to obtain the target image.

[0007] According to a second aspect of the present disclosure, a video resolution improvement device is provided, comprising: 10. A video resolution improvement device, characterized in that it comprises: an acquisition module, used to acquire a target frame and at least one reference frame; a fusion module, used to use a first convolutional layer to merge the target frame and at least one reference frame into one frame to obtain a first feature image, and the number of channels of the first feature image is the same as the number of channels of the target frame; a feature extraction module, used to extract features from the first feature image to obtain a second feature image including edge features; a residual module, used to obtain an image to be output based on the target frame and the second feature image; a sampling module, used to upsample the image to be output by a preset multiple to obtain the target image.

[0008] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which enables the one or more processors to implement the above method when the one or more programs are executed by the one or more processors.

[0010] A video resolution enhancement method provided by an embodiment of the present disclosure obtains a target frame and at least one reference frame; merges the target frame and at least one reference frame into one frame to obtain a first feature image, the number of channels of the first feature image is the same as the number of channels of the target frame; extracts features from the first feature image to obtain a second feature image including edge features; obtains an image to be output based on the target frame and the second feature image; and performs upsampling of the image to be output by a preset multiple to obtain the target image. Compared with the prior art, the target frame and the reference frame are firstly fused, and the fusion of multi-frame information can ensure the accuracy of resolution enhancement. At the same time, the first feature map obtained has the same number of channels as the above-mentioned target frame, which reduces the feature dimension and reduces the calculation. Further, an edge feature map is obtained to be fused with the target frame to obtain an output image. There is no downsampling process in the process, and the height and width of the image are not reduced, which reduces the amount of calculation in the subsequent upsampling process.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0013] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0014] Figure 2 A flowchart of a method for improving video resolution in an exemplary embodiment of the present disclosure is schematically shown;

[0015] Figure 3 A schematic diagram schematically illustrates a convolution operation of a first convolution layer in an exemplary embodiment of the present disclosure;

[0016] Figure 4 Schematically showing a flow chart of acquiring a second feature image in an exemplary embodiment of the present disclosure;

[0017] Figure 5 A diagram schematically showing the relationship between the floating point calculation amount and the model channel dimension obtained in an exemplary embodiment of the present disclosure;

[0018] Figure 6 A diagram schematically showing the relationship between peak signal-to-noise ratio and model channel dimension in an exemplary embodiment of the present disclosure;

[0019] Figure 7 A data flow diagram schematically illustrating a method for improving video resolution in an exemplary embodiment of the present disclosure;

[0020] Figure 8 A schematic diagram schematically shows the composition of a video resolution enhancement device in an exemplary embodiment of the present disclosure;

[0021] Fig. 9 A schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0022] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0023] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0024] Figure 1 1 shows a schematic diagram of a system architecture, which may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, etc. The server 120 generally refers to a background system that provides services related to video resolution enhancement in this exemplary embodiment, and may be a server or a cluster formed by multiple servers. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to perform data exchange.

[0025] In one implementation, the video resolution enhancement method may be performed by the terminal 110. For example, after a user uses the terminal 110 to shoot a video or the user selects a video from a storage repository of the terminal 110, the terminal 110 enhances the video resolution of the image and outputs the video with enhanced resolution.

[0026] In one implementation, the video resolution enhancement method may be performed by the server 120. For example, a user uses the terminal 110 to obtain the target frame and at least one reference frame, and the terminal 110 uploads the target frame and at least one reference frame to the server 120, and the server 120 enhances the video resolution of the image and returns the target image to the terminal 110.

[0027] As can be seen from the above, the execution subject of the video resolution enhancement method in this exemplary embodiment may be the above-mentioned terminal 110 or server 120, and the present disclosure does not limit this.

[0028] The exemplary embodiments of the present disclosure also provide an electronic device for executing the above-mentioned video resolution enhancement method, and the electronic device may be the above-mentioned terminal 110 or the server 120. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned image video resolution enhancement method by executing the executable instructions.

[0029] In related technologies, current super-resolution models are generally heavy models, such as vision transformer[2]. Even lightweight models such as Basic vsr[3] and Basicr vsr++[4] have hundreds of GB of floating-point calculations and parameters. As a result, super-resolution models, especially multi-frame super-resolution models, cannot be processed in real time.

[0030] At present, without considering the computational complexity, the peak signal-to-noise ratio of the EDVR model is the highest. For example, when 5 frames of images are input and mid-feature=64, when the input size=(1,5,3,720,1280), FLOPs=646.7G, where FLOPs represents the floating point calculation amount, and the complexity is very high. This seriously affects its application and makes it difficult to deploy on the mobile terminal side.

[0031] Combine the following Figure 2 The video resolution enhancement method in this exemplary embodiment is described. Figure 2 An exemplary process of the method for improving video resolution is shown, which may include:

[0032] Step S210, acquiring a target frame and at least one reference frame;

[0033] Step S220, merging the target frame and at least one of the reference frames into one frame to obtain a first feature image, wherein the number of channels of the first feature image is the same as the number of channels of the target frame;

[0034] Step S230, performing feature extraction on the first feature image to obtain a second feature image including edge features;

[0035] Step S240, obtaining an image to be output according to the target frame and the second feature image;

[0036] Step S250 , upsampling the image to be output by a preset multiple to obtain the target image.

[0037] Based on the above method, the target frame and the reference frame are first fused. The fusion of multi-frame information can ensure the accuracy of resolution improvement. At the same time, the first feature map obtained has the same number of channels as the target frame, which reduces the feature dimension and reduces the calculation. Further, an edge feature map is obtained to be fused with the target frame to obtain an output image. There is no downsampling process in the process, and the height and width of the image are not reduced, which reduces the amount of calculation in the subsequent upsampling process.

[0038] Below Figure 2 Each step in the procedure is described in detail.

[0039] refer to Figure 2 , in step S210, a target frame and at least one reference frame are obtained.

[0040] In an exemplary embodiment, a target frame and at least one reference frame may be obtained as input data, wherein the input data may include multiple frames of continuous images, that is, the reference image includes multiple, for example, three frames of continuous images, four frames of continuous graphics or more frames of continuous images, etc., and the use of continuous images can achieve a better effect of improving the video resolution. In another exemplary embodiment, the input data may also include multiple frames of non-continuous images, and the multiple frames of images are not specifically limited in this exemplary embodiment.

[0041] In this example implementation, when the output data is three consecutive frames of images, the reference frame may be the previous frame and the next frame of the target frame, that is, the target frame is the middle frame of the input data. If the input data is an even-numbered frame, the middle frame may include two frames, wherein the target frame may be any one of the two frames, or the average of the two frames. The other frames are all reference frames.

[0042] In this example implementation, the characteristic parameters of the input data can be expressed as (B, T, H, W, C); they respectively include the height H (hight), width W (width) and time T (time) of the image, and the concurrent number B (batch) and the number of channels C (channel) of the input data. Each frame of the image corresponds to a time, and the concurrent number of the input data indicates the number of input data input at the same time. For example, when the input data includes three consecutive frames of images, each frame of the image corresponds to a height, width and time; when three groups of three consecutive frames of images are input at the same time, the concurrent number at this time is three.

[0043] In this exemplary embodiment, the input data includes 3 frames of continuous images for detailed description, that is, the reference frame includes the previous frame of the target frame and the next frame of the target frame for description.

[0044] In this example implementation, the input data includes three frames of continuous images. At this time, the time T is 3, only one set of three frames of continuous data is input, the concurrency quantity B is 1, and the input image can be an RGB image, and the number of channels is 3.

[0045] In step S220, the target frame and at least one of the reference frames are merged into one frame to obtain a first feature image, and the number of channels of the first feature image is the same as the number of channels of the target frame.

[0046] In this example implementation, the target frame and the multiple reference frames may first be combined into a first frame image. Specifically, the target frame and the reference frame may be combined into one frame image by using a first convolutional layer to obtain a first feature image, and the number of channels in the first feature image obtained is the same as the number of channels of the target frame.

[0047] For example, the input is three consecutive RGB images. The features of the image obtained after preliminary fusion can be (B, 9, H, W), where 9 is the number of channels. Then, after the first convolution layer, the feature dimension is reduced from (B, 9, H, W) to (B, 3, H, W), and the fusion effect is achieved at the same time. The first convolution layer can include three groups of convolution kernels, each group of convolution kernels can obtain an image of one channel, and each group of convolution kernels is used to perform convolution operations on the image after preliminary fusion to obtain the first feature image.

[0048] In this example implementation, the number of convolution kernels in each group can be 9, or can be one, three, etc., and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0049] In this example implementation, refer to Figure 3As shown, after the target frame and the reference frame are initially fused, a preliminary fused image 310 with 9 channels is obtained, and then the three different groups of convolution kernels in the first convolution layer 320 can be used to perform convolution operations on the above image. Each group of convolution kernels can calculate an image of one channel, and the above three groups of convolution kernels are used to perform convolution operations respectively, and the first feature image 330 with 3 channels can be obtained. The size of each group of convolution kernels can be 1*1, 2*2, etc., and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0050] In step S230, feature extraction is performed on the first feature image to obtain a second feature image including edge features.

[0051] In the real-time mode of this example, after obtaining the first feature image, the first feature image is subjected to industry feature extraction to obtain a second feature image of edge features.

[0052] Specifically, refer to Figure 4 As shown, it may include steps S410 to S420.

[0053] In step S410, a feature extraction model of a preset dimension is obtained.

[0054] In this example implementation, a target peak signal-to-noise ratio may be first determined, and then the feature extraction model of the preset dimension may be determined based on the target signal-to-noise ratio, wherein the feature extraction model may be obtained by compressing the MobileEDVRV3* model including an inverted residual network, wherein MobileEDVRV3* is a model obtained by replacing the PCD module, the pre-deblur module and the reconstruction module with ordinary convolutional layers on the basis of MobileEDVRV3, and adjusting the resdiual (residual) BlockNoBN module to an invertedResidual (inverted residual) BlockNoBN module, but keeping the dimension of each layer unchanged, and reducing the input from 5 frames to 3 frames. Among them, the MobileEDVRV3 model is a model obtained by replacing the convolutional layer in EDVR with a lightweight structure Depthwise Separable Convolutions. Among them, the MobileEDVRV3* model is a feature extraction model in the prior art and is not specifically stated in this example implementation.

[0055] In this example implementation, the feature extraction model can be subjected to channel compression according to the target peak signal-to-noise ratio to obtain a feature extraction model of preset dimensions, that is, while ensuring that the peak signal-to-noise ratio of the model is greater than or equal to the target peak signal-to-noise ratio, a model with the smallest number of channels is obtained as a feature extraction model of preset row dimensions, wherein the target peak signal-to-noise ratio can be 34.6 or customized according to user needs, which is not specifically limited in this example implementation.

[0056] The peak signal-to-noise ratio and floating-point calculation amount corresponding to the feature extraction model of each dimension can be determined through experiments. For details, please refer to Table 1.

[0057] Table 1

[0058] dim PSNR FLOPs / G 64 35.631 83 56 35.67 64 48 35.62 47 40 35.12 33 32 34.69 21.4 24 34.73 12.3 16 34.623 5.7

[0059] The relationship between the floating-point calculation amount and the model channel dimension can be obtained from the data in Table 1. The relationship between the peak signal-to-noise ratio and the model channel dimension can be obtained from the data in Table 1. For details, please refer to Figure 5 and Figure 6 As shown in the figure, as the model channels are compressed more, the floating-point calculation amount corresponding to the feature extraction model decreases a lot, but at the same time, the peak signal-to-noise ratio decreases less.

[0060] In step S420, the feature extraction model of the preset dimension is used to extract features from the first feature image to obtain the second feature image including the edge features.

[0061] When the target peak signal-to-noise ratio is 34.6, a feature extraction model with 16 channels can be selected according to the data in Table 1 to perform feature extraction on the first feature image to obtain the second feature image.

[0062] In step S240, an image to be output is obtained according to the target frame and the second feature image.

[0063] In this example implementation, after obtaining the second characteristic image, the image to be output may be obtained according to the target frame and the second characteristic image. Specifically, the image to be output may be obtained by adding the target frame to the second characteristic image.

[0064] Step S250 , upsampling the image to be output by a preset multiple to obtain the target image.

[0065] In this example implementation, after the image to be output is obtained, the image to be output may be upsampled by a preset multiple to obtain the target image.

[0066] Specifically, the preset multiple may be determined first, and the preset multiple may be 2, 3, 4, etc., or may be customized according to user needs, which is not specifically limited in this example implementation.

[0067] In this example implementation, the second convolution layer can be first determined according to the above preset multiple, and then the number of channels of the above output image can be increased by using the above second convolution layer, and then the channel image to be merged can be obtained, and then the above target image can be obtained according to the above image to be output. Specifically, the above channel image to be merged can be transferred to the height and width of the above image to be output to obtain the above target image, and specifically, pixel_shuffle can be used to 'transfer' the channel image to be merged to the width and height of the image. If the above preset multiple is 2, the height and width of the obtained target image are both twice that of the above image to be output.

[0068] In another example implementation, multiple third convolutional layers can be determined according to the above-mentioned AND number of devices to upsample the above-mentioned image to be output, wherein, if the above-mentioned AND multiple is 4, two third convolutional layers with a magnification of 2 or one second convolutional layer with a magnification of 4 can be used, which is not specifically limited in the real-time mode of this example.

[0069] In this example implementation, refer to Figure 7 The data flow in the video resolution enhancement method disclosed in the present invention is explained. First, the target frame 720 and the reference frame 710 can be obtained, and then the above target frame and the reference frame are input into the first convolutional layer 730 to obtain a first feature image, and then the first feature image output value feature extraction model 740 is used to obtain a second feature image, and then the second feature image is summed with the above target frame to obtain an image to be output, that is, the second feature image is summed with the above target frame using the residual structure 750, and finally the above image to be output can be output to the upsampling layer 760 to obtain the target image.

[0070] In this example implementation, it is known through experiments that the computational complexity of the video resolution enhancement method disclosed in the present invention is relatively small, as shown in Table 2.

[0071] Table 2

[0072]

[0073]

[0074] It can be seen from Table 2 that compared with the prior art, the present disclosure significantly reduces the amount of floating-point calculations while ensuring that the peak signal-to-noise ratio meets the conditions, that is, the accuracy is met, thereby reducing the amount of calculations in the process of improving video resolution.

[0075] In summary, in this exemplary embodiment, the target frame is first fused with the reference frame. The fusion of multiple frames can ensure the accuracy of resolution improvement. At the same time, the first feature map obtained has the same number of channels as the target frame, which reduces the feature dimension and reduces the calculation. Further, an edge feature map is obtained to be fused with the target frame to obtain an output image. There is no downsampling process in the process, and the height and width of the image are not reduced, which reduces the amount of calculation in the subsequent upsampling process. Furthermore, when extracting features from the first feature image to obtain the second feature image, a feature extraction model with low channel dimensions is used, which further reduces the amount of calculation in the process of video resolution improvement while ensuring accuracy.

[0076] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0077] For further reference, Figure 8 As shown, in the implementation of this example, a video resolution enhancement device 800 is also provided, including an acquisition module 810, a fusion module 820, a feature extraction module 830, a residual module 840 and a sampling module 850. Among them:

[0078] The acquisition module 810 may be used to acquire a target frame and at least one reference frame, wherein the reference frame includes a previous frame of the target frame and a subsequent frame of the target frame.

[0079] The fusion module 820 can be used to merge the target frame and at least one reference frame into one frame to obtain a first feature image, and the number of channels of the first feature image is the same as the number of channels of the target frame. Specifically, the target frame and at least one reference frame can be merged into one frame using the first convolutional layer to obtain the first feature image.

[0080] The feature extraction module 830 can be used to perform feature extraction on the first feature image to obtain a second feature image including edge features. Specifically, a feature extraction model of a preset dimension is obtained; and the feature extraction model of the preset dimension is used to perform feature extraction on the first feature image to obtain the second feature image including edge features. When obtaining the feature extraction model of the preset dimension, a target peak signal-to-noise ratio can be first determined; and the feature extraction model of the preset dimension can be determined according to the target peak signal-to-noise ratio. When the feature extraction model of the preset dimension is used to perform feature extraction on the first feature image to obtain the second feature image including edge features, a feature extraction model of an inverse residual network structure can be used to perform feature extraction on the first feature image to obtain the second feature image including edge features.

[0081] The residual module 840 may be configured to obtain the image to be output according to the target frame and the second feature image. Specifically, the image to be output may be obtained by adding the second feature image to the target frame.

[0082] The sampling module 850 may be configured to upsample the image to be output by a preset multiple to obtain the target image.

[0083] In an example implementation, the sampling module 850 may first determine a second convolutional layer according to the preset multiple; then use the second convolutional layer to increase the number of channels of the image to be output to obtain a channel image to be merged; and finally obtain the target image according to the image to be output and the channel image to be merged.

[0084] In another example implementation, the sampling module 850 may first determine a plurality of third convolutional layers according to the preset multiple; and then use the plurality of third convolutional layers to upsample the image to be output to obtain the target image.

[0085] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0086] Below Fig. 9 The mobile terminal 900 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Fig. 9 The construction in can also be applied to fixed type equipment.

[0087] like Fig. 9As shown, the mobile terminal 900 may specifically include: a processor 901, a memory 902, a bus 903, a mobile communication module 904, an antenna 1, a wireless communication module 905, an antenna 9, a display screen 906, a camera module 907, an audio module 908, a power module 909 and a sensor module 910.

[0088] The processor 901 may include one or more processing units, for example, the processor 910 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. The video resolution enhancement method in this exemplary embodiment may be executed by an AP, a GPU or a DSP, and when the method involves processing related to a neural network, it may be executed by an NPU.

[0089] The encoder can encode (i.e. compress) an image or video, for example, the target image can be encoded into a specific format to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of the image or video to restore the image or video data, such as reading the encoded data of the target image, decoding it through the decoder to restore the data of the target image, and then perform relevant processing on the data to improve the video resolution. The mobile terminal 900 can support one or more encoders and decoders. In this way, the mobile terminal 900 can process images or videos in multiple encoding formats, such as: JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap) and other image formats, MPEG (Moving Picture Experts Group) 1, MPEG9, H.963, H.964, HEVC (High Efficiency Video Coding) and other video formats.

[0090] The processor 901 may be connected to the memory 902 or other components via a bus 903 .

[0091] The memory 902 may be used to store computer executable program codes, which include instructions. The processor 901 executes various functional applications and data processing of the mobile terminal 900 by running the instructions stored in the memory 902. The memory 902 may also store application data, such as images, videos, and other files.

[0092] The communication function of the mobile terminal 900 can be implemented through the mobile communication module 904, antenna 1, wireless communication module 905, antenna 2, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 904 can provide 9G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 900. The wireless communication module 905 can provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. applied to the mobile terminal 900.

[0093] The display screen 906 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 907 is used to implement shooting functions, such as shooting images, videos, etc. The audio module 908 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 909 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 910 may include a depth sensor 9101, a pressure sensor 9102, a gyroscope sensor 9103, an air pressure sensor 9104, etc., to implement corresponding sensing detection functions.

[0094] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0095] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification.

[0096] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0097] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0098] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0099] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0100] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for improving video resolution, characterized in that: include: Acquire a target frame and at least one reference frame; Merging the target frame and at least one of the reference frames into one frame to obtain a first feature image, wherein the number of channels of the first feature image is the same as the number of channels of the target frame; Performing feature extraction on the first feature image to obtain a second feature image including edge features; Obtaining an image to be output according to the target frame and the second feature image; Upsampling the image to be output by a preset multiple to obtain a target image; The step of extracting features from the first feature image to obtain a second feature image including edge features comprises: Determine target peak signal-to-noise ratio; Determine a feature extraction model of preset dimensions according to the target peak signal-to-noise ratio to ensure that the number of channels is minimized when the peak signal-to-noise ratio of the feature extraction model is greater than the target peak signal-to-noise ratio; The feature extraction model of the preset dimension is used to extract features from the first feature image to obtain the second feature image including edge features; wherein the number of channels corresponding to the feature extraction model depends on the target peak signal-to-noise ratio.

2. The method according to claim 1, characterized in that The target frame and at least one of the reference frames are merged into one frame to obtain a first feature image, comprising: The target frame and at least one reference frame are merged into one frame using the first convolution layer to obtain a first feature image.

3. The method according to claim 1, characterized in that The reference frame includes a previous frame of the target frame and a subsequent frame of the target frame.

4. The method according to claim 1, characterized in that The extracting features of the first feature image by using the feature extraction model of the preset dimension to obtain the second feature image including the edge feature comprises: The feature extraction model of the inverse residual network structure is used to extract features from the first feature image to obtain the second feature image including the edge features.

5. The method according to claim 1, characterized in that Obtaining the image to be output according to the target frame and the second feature image includes: The image to be output is obtained by adding the second feature image to the target frame.

6. The method according to claim 1, characterized in that The step of upsampling the image to be output by a preset multiple to obtain the target image comprises: Determine the second convolutional layer according to the preset multiple; Using the second convolutional layer to increase the number of channels of the image to be output, so as to obtain a channel image to be merged; The target image is obtained according to the image to be output and the channel image to be merged.

7. The method according to claim 1, characterized in that The target image is obtained by upsampling the image to be output by a preset multiple, Determine a plurality of third convolutional layers according to the preset multiple; The image to be output is upsampled using a plurality of the third convolutional layers to obtain the target image.

8. A video resolution enhancement device, characterized in that: include: An acquisition module, used for acquiring a target frame and at least one reference frame; A fusion module, configured to use a first convolutional layer to merge the target frame and at least one of the reference frames into one frame to obtain a first feature image, wherein the number of channels of the first feature image is the same as the number of channels of the target frame; A feature extraction module, used for extracting features from the first feature image to obtain a second feature image including edge features; A residual module, used for obtaining an image to be output according to the target frame and the second feature image; A sampling module, used for upsampling the image to be output by a preset multiple to obtain a target image; The feature extraction module extracts features from the first feature image to obtain a second feature image including edge features, including: Determine target peak signal-to-noise ratio; Determine a feature extraction model of preset dimensions according to the target peak signal-to-noise ratio to ensure that the number of channels is minimized when the peak signal-to-noise ratio of the feature extraction model is greater than the target peak signal-to-noise ratio; The feature extraction model of the preset dimension is used to extract features from the first feature image to obtain the second feature image including edge features; wherein the number of channels corresponding to the feature extraction model depends on the target peak signal-to-noise ratio.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video resolution enhancement method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the video resolution enhancement method as described in any one of claims 1 to 7.

Images