Image processing method and device

Abstract

The present invention discloses an image processing method and apparatus for reducing the time required for video super-resolution. The method comprises: a decoding device processes image information of a target frame to be super-resolved and image information of adjacent frames of the target frame, combining motion vector information in an encoded bitstream, to generate a high-resolution reconstructed frame.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to an image processing method and apparatus. Background Art

[0002] Traditional video super-resolution algorithms use techniques like linear interpolation to reconstruct a set of consecutive low-resolution frames, ultimately fusing them into a single high-resolution image. With the rapid development of deep learning technology in recent years, researchers have begun using deep learning methods to improve the effectiveness of super-resolution technology, reconstructing low-resolution video sequences into temporally and spatially correlated high-resolution video sequences.

[0003] Existing video super-resolution technology usually uses the combining local and global-television (CLG-TV) optical flow model algorithm to calculate the optical flow velocity vector between all low-resolution video image sequences and the current frame video image, that is, to perform motion estimation. According to the optical flow velocity vector, the motion-compensated 2T frame low-resolution video image and the current frame low-resolution video image are obtained. Then, a deep residual network is used to sequentially perform the initial stage, the cascaded convolution layer calculation stage and the residual block calculation stage on the 2T frame low-resolution video image and the current frame low-resolution video image, and then gradually reconstruct the high-resolution video image using deconvolution and convolution operations.

[0004] Existing video super-resolution algorithms require motion estimation, which consumes a lot of computing resources. Summary of the Invention

[0005] An embodiment of the present application provides an image processing method for processing super-resolution frames into high-resolution reconstructed frames by using motion vector information in a coded bitstream, thereby avoiding the large amount of computational requirements required for motion estimation and significantly reducing the time required for video super-resolution.

[0006] A first aspect of an embodiment of the present application provides an image processing method, comprising: a decoding device acquiring motion vector information of a target frame, image information of the target frame, and image information of adjacent frames in a coded code stream, wherein the target frame and the adjacent frames are images of a first resolution, the target frame is an image that needs to be super-resolution processed, and the adjacent frames include images within a preset period before or after the target frame; the decoding device generates a reconstructed frame based on the motion vector information, the image information of the target frame, and the image information of the adjacent frames, wherein the reconstructed frame is an image of a second resolution, the second resolution is greater than the first resolution, and the motion vector information is used to indicate adaptive local sampling of the image information of the adjacent frames and the image information of the target frame.

[0007] In the first aspect mentioned above, the encoded code stream can be a code stream generated by an image compression coding technology including predictive coding based on motion estimation and compensation. The motion estimation and motion compensation algorithm is used to remove time domain redundant information, that is, the image compression coding technology can apply a motion estimation algorithm to determine motion vector information.

[0008] In an embodiment of the present application, a decoding device can directly extract motion vector information from a coded bitstream and decode the coded bitstream to obtain decoded image information. The image information can include image information of a target frame for super-resolution processing and image information of adjacent frames within a preset period before or after the target frame. Exemplarily, the adjacent frames can include T frames before and T frames after the target frame. The target frame and the adjacent frames are both images of a first resolution, i.e., a low resolution. After decoding the image information of the target frame and the image information of the adjacent frames, the decoding device can process the image information of the target frame and the image information of the adjacent frames in combination with the motion vector information to generate a reconstructed frame of a second resolution. The reconstructed frame is an image of a high resolution, i.e., the second resolution is greater than the first resolution. The motion vector information is used to indicate adaptive local sampling of the image information of the adjacent frames and the image information of the target frame. In an embodiment of the present application, the decoding device does not need to perform a computationally expensive motion estimation process when performing super-resolution, which can significantly reduce the time required for video super-resolution.

[0009] In a possible implementation of the first aspect, the decoding device performs adaptive local sampling on each position corresponding to the image information of the target frame in the image information of the adjacent frames based on motion vector information; the decoding device generates a reconstructed frame based on the image information of the target frame and the image information of the adjacent frames after adaptive local sampling.

[0010] In this possible implementation, the decoding device can adaptively select and sample highly similar image points from multiple image points at corresponding local locations in the image information of adjacent frames based on the local locations of the target frame's image information and the motion vector information. The decoding device can then reconstruct the corresponding image based on the image information of the adjacent frames after this adaptive local sampling to generate a reconstructed frame, thereby reducing noise in the motion vector information and improving robustness.

[0011] In a possible implementation of the first aspect, the decoding device in the above-mentioned step adaptively locally samples each position corresponding to the image information of the target frame in the image information of the adjacent frames based on the motion vector information, including: the decoding device generates a target feature pyramid based on the image information of the target frame, and generates an adjacent feature pyramid based on the image information of each adjacent frame, the target feature pyramid includes target features of multiple scales, and each adjacent feature pyramid includes adjacent features of multiple scales; the decoding device adaptively locally samples the adjacent features at the position corresponding to each target feature in each adjacent feature pyramid based on the motion vector information and the position of each target feature.

[0012] In this possible implementation, the feature pyramid of an image is a set of features arranged in a pyramid shape. The feature pyramid is obtained by downsampling an original feature in stages, so that the size decreases layer by layer. The decoding device can extract the target features and adjacent features from the image information of the target frame and the image information of the adjacent frames through a feature extraction function, and then generate target features and adjacent features at multiple scales through downsampling to form a corresponding feature pyramid. Based on the scale invariance of the feature pyramid, the decoding device can use the position of each target feature at each scale as a reference and perform adaptive local sampling of the adjacent features at the corresponding position of each adjacent feature pyramid according to the motion vector information, that is, further refine the search for better matching features within the adjacent features at the mapped location to improve the feature quality of each adjacent feature.

[0013] In a possible implementation of the first aspect, the decoding device generates a reconstructed frame based on the image information of the target frame and the image information of the adjacent frames after adaptive local sampling, including: the decoding device fuses the target feature pyramid and each adjacent feature pyramid after adaptive local sampling to generate a fused feature pyramid, the fused feature pyramid includes fused features at multiple scales; and the decoding device processes the fused feature pyramid to generate the reconstructed frame.

[0014] In this possible implementation, the decoding device stacks the adjacent feature pyramids after adaptive local sampling and the target feature pyramid, convolving them into a fused feature pyramid. This fused feature pyramid is then reconstructed into a high-resolution image. Stacking (concat) combines the number of feature channels, meaning that the number of features (channels) describing the image itself increases, while the information contained in each feature remains unchanged.

[0015] In a possible implementation of the first aspect, the decoding device adaptively locally samples the adjacent features corresponding to each target feature in each adjacent feature pyramid based on the motion vector information and the position of each target feature, including: for each adjacent feature pyramid, the decoding device searches for the second local feature block based on the coordinates of the first local feature block in the target feature and the mapping relationship between the first local feature block contained in the motion vector information and the second local feature block in the adjacent feature; the decoding device performs feature matching on the first local feature block and the second local feature block through a fully connected layer to determine a set of related attention coefficients, the set of related attention coefficients including multiple related attention coefficients, wherein each related attention coefficient indicates the similarity between a feature point in the first local feature block and a corresponding feature point in the second local feature block; the decoding device performs weighted averaging on multiple feature points in the second local feature block based on the set of related attention coefficients to determine the adjacent feature pyramid after adaptive local sampling.

[0016] In this possible implementation, the attention coefficient is the degree of attention, that is, more attention is paid to the feature points in the adjacent features that are more similar to the target features. In the embodiment of the present application, the decoding device extracts a first local feature block at one scale of the target feature pyramid, and based on the mapping relationship indicated by the motion vector information, extracts a second local feature block from the corresponding coordinates of an adjacent feature pyramid for the coordinates of the first local feature block, and then determines the attention coefficient of each feature point in the second local feature block through a two-layer fully connected layer, and then attaches the degree of attention corresponding to the above-mentioned attention coefficient to each feature point in the second local feature block, that is, obtains the second local feature block after adaptive local sampling. After processing the feature blocks extracted from the adjacent features of all adjacent feature pyramids corresponding to all scales of the target feature pyramid, the adjacent feature pyramid after adaptive local sampling can be obtained.

[0017] In a possible implementation of the first aspect, the above-mentioned step of the decoding device fusing the target feature pyramid and the adjacent feature pyramid after adaptive local sampling to generate a fused feature pyramid includes: for each adjacent feature pyramid, the decoding device calculates an attention map based on the target feature and the adjacent features after adaptive local sampling, and the attention map is used to represent the similarity between the adjacent features after adaptive local sampling and the target feature; the decoding device performs feature enhancement processing on the adjacent features after adaptive local sampling and the attention map; the decoding device stacks and performs convolution calculations on all the adjacent features after feature enhancement processing and the target feature to generate a fused feature, and determines the fused feature pyramid.

[0018] In this possible implementation, the decoding device can generate an attention map in the time domain based on the alignment quality of the features of adjacent frames, and dynamically fuse the features of the adjacent frames and the target frame after adaptive local sampling by increasing the weights of the local areas of high-quality adaptive local sampling and reducing the weights of the local areas of low-quality adaptive local sampling. The alignment quality can be represented by calculating the feature inner product of the adjacent frame features and the target features after adaptive local sampling at each coordinate point. The feature inner product can characterize the similarity between the adjacent features after adaptive local sampling and the target features at that point. Then, each feature area is weighted, for example, the feature is multiplied point by point with the above-mentioned attention map. The adjacent features after feature enhancement are stacked with the target features as described above, and convolved to generate a fused feature. After the decoding device performs a feature fusion, it is necessary to detect whether there are adjacent features that have not been enhanced, until all adjacent features are fused with the target features to generate a fused feature pyramid.

[0019] In a possible implementation of the first aspect, the above-mentioned step of the decoding device generating a target feature pyramid based on the image information of the target frame includes: the decoding device convolving the image information of each adjacent frame, and then processing the first cascade residual block to generate an adjacent feature corresponding to the image information of each adjacent frame; the decoding device generates adjacent features of multiple scales through bilinear interpolation of the adjacent features, and constructs an adjacent feature pyramid.

[0020] In this possible implementation, the residual blocks (rasidual blocks) use skip links to improve the accuracy by increasing the depth considerably, wherein the skip links are for the residual blocks to directly bypass the received input information to the output, thereby protecting the integrity of the information. The scale represents the number of pixels in the image. In an embodiment of the present application, the decoding device extracts a target feature from the image information of the target frame through a feature extraction function, and the feature extraction function includes convolution processing and cascaded residual blocks. The decoding device can reduce the image size of the target feature to different degrees by bilinear interpolation downsampling, so as to obtain target features of different scales, and then arrange the target features according to the scale size to generate the above-mentioned target feature pyramid. The number of pixels in each layer of the target feature pyramid is continuously reduced from the bottom to the top, which can greatly reduce the amount of calculation.

[0021] In a possible implementation of the first aspect, the decoding device generates an adjacent feature pyramid based on the image information of each adjacent frame, including: the decoding device convolves the image information of each adjacent frame, and then processes the first cascade residual block to generate an adjacent feature corresponding to the image information of each adjacent frame; the decoding device generates an adjacent feature pyramid by bilinear interpolation of adjacent features at multiple scales.

[0022] In this possible implementation, the decoding device simultaneously extracts features from image information of adjacent frames through multiple feature extraction functions that share weights with the above-mentioned feature extraction function to obtain adjacent features, and then performs image reduction to different degrees through bilinear interpolation downsampling to obtain adjacent features of different scales, and then arranges the adjacent features according to the scale to generate an adjacent feature pyramid.

[0023] In a possible implementation of the first aspect, the decoding device processes the fused feature pyramid to generate a reconstructed frame, including: the decoding device calculates the fused feature pyramid through a second cascade residual block to generate an optimized feature pyramid; the decoding device enlarges the size and convolves the optimized feature pyramid to generate a reconstructed residual signal; the decoding device adds the reconstructed residual signal to the image magnification result to obtain a reconstructed frame, where the image magnification result is generated by bilinear interpolation of image information of the target frame.

[0024] In this possible implementation, the decoding device exchanges information between the features of each scale level in the above-generated fusion feature pyramid through the residual block of the second cascade. For example, the features of each scale level can be upsampled or downsampled, and interacted at the same scale to optimize the fusion features. The optimized fusion features are then amplified and convolved, and added to the image information of the target frame amplified by bilinear interpolation to obtain a high-resolution reconstructed frame.

[0025] A second aspect of the present application provides a decoding device that implements the method of the first aspect or any possible implementation of the first aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functionality, such as a receiving unit and a processing unit.

[0026] A third aspect of an embodiment of the present application provides a computer device, which includes at least one processor, a storage system, an input / output (I / O) interface, and computer execution instructions stored in the storage system and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes the method of the first aspect or any possible implementation of the first aspect.

[0027] The fourth aspect of an embodiment of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0028] The fifth aspect of the embodiments of the present application provides a computer program product that stores one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0029] A sixth aspect of the present application provides a chip system, comprising at least one processor configured to support a decoding device in implementing the functions described in the first aspect or any one of the possible implementations of the first aspect. In one possible design, the chip system may further include a memory configured to store program instructions and data necessary for the decoding device. The chip system may consist of a chip or may include a chip and other discrete components.

[0030] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can be referred to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.

[0031] The solution provided in the embodiment of the present application is that the decoding device generates high-resolution reconstructed frames by performing neural network processing on the motion vector information, target feature pyramid and adjacent feature pyramid obtained from the encoded bitstream. The encoded bitstream contains certain motion information, and the computational cost of extracting the motion information in the bitstream is negligible, so the time for video super-resolution can be greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an application scenario of the embodiment of the present application;

[0033] Figure 2 is a schematic block diagram of a video encoding and decoding system in an embodiment of the present application;

[0034] Figure 3 This is a flowchart of an embodiment of the image processing method in the embodiment of the present application;

[0035] Figure 4 This is a schematic diagram of extracting local feature blocks in an embodiment of the present application;

[0036] Figure 5 This is a schematic diagram of the architecture of a decoding device in an embodiment of the present application;

[0037] Figure 6 This is an image processing flow chart of the feature extraction module in an embodiment of the present application;

[0038] Figure 7 This is a flowchart of the image processing of the flexible alignment module in an embodiment of the present application;

[0039] Figure 8This is an image processing flow chart of the multi-frame feature fusion module in an embodiment of the present application;

[0040] Figure 9 This is an image processing flow chart of the feature super-resolution reconstruction module in an embodiment of the present application;

[0041] Figure 10 A comparison chart of the super-resolution results of the embodiment of the present application and the super-resolution results of the prior art;

[0042] Figure 11 This is a structural diagram of a decoding device in an embodiment of the present application;

[0043] Figure 12 This is another structural diagram of the decoding device in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The embodiments of the present application provide an image processing method and a decoding device for reducing the time required for video super-resolution.

[0045] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. In the following description, reference is made to the drawings that form a part of this disclosure and illustrate specific aspects of the embodiments of the present application or specific aspects of the embodiments of the present application that can be used. It should be understood that the embodiments of the present application can be used in other aspects and may include structural or logical changes not depicted in the drawings. Therefore, the following detailed description should not be understood in a restrictive sense, and the scope of the present application is defined by the appended claims. For example, it should be understood that the disclosure in conjunction with the described method can also be applied to the corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units such as functional units to perform the one or more method steps described (for example, one unit performs one or more steps, or multiple units, each of which performs one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific device is described based on one or more units such as functional units, the corresponding method may include a step to perform the functionality of the one or more units (e.g., a step to perform the functionality of the one or more units, or multiple steps, each of which performs the functionality of one or more of the multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Further, it should be understood that unless otherwise explicitly stated, the features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0046] Video encoding generally refers to the processing of a sequence of images to form a video or video sequence. In the field of video encoding, the terms "picture", "frame" or "image" can be used as synonyms. Video encoding is performed on the source side and generally involves processing (e.g., by compression) the original video image to reduce the amount of data required to represent the video image so that it can be stored and / or transmitted more efficiently. Video decoding is performed on the destination side and generally involves inverse processing relative to the encoder to reconstruct the video image. The embodiments involving "encoding" of video images should be understood as involving "encoding" or "decoding" of video sequences. The combination of the encoding part and the decoding part is also called codec (encoding and decoding).

[0047] This embodiment can be applied to Figure 1 In the application scenario shown, the terminal 11, the server 12, the set-top box 13 and the TV 14 are connected via a wireless or wired network. The terminal 11 can remotely control the display 14 through a locally installed application software (APP). For example, the user can operate on the operation interface of the terminal 11 to output a video source for TV playback. The terminal 11 encodes the video source through the server 12 and then forwards it to the set-top box 13. The set-top box 13 decodes the encoded video source and transmits it to the display 14, and then the display 14 can play based on the decoded video source.

[0048] The following describes the system architecture used in the embodiments of this application. Figure 2 , Figure 2 A schematic block diagram of a video encoding and decoding system used in an embodiment of the present application is given as an example. Figure 2As shown, a video encoding and decoding system may include an encoding device 21 and a decoding device 22, where the encoding device 21 generates encoded video data. The decoding device 22 may decode the encoded video data generated by the encoding device 21. Various implementations of the encoding device 21, the decoding device 22, or both may include one or more processors and a memory coupled to the one or more processors. The memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or any other medium that can be used to store desired program code in the form of computer-accessible instructions or data structures, as described herein. The encoding device 21 and the decoding device 22 may include various devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, wireless communication devices, or the like.

[0049] Although Figure 2 The encoding device 21 and the decoding device 22 may be separate devices, or may include both the encoding device 21 and the decoding device 22 or the functionality of both, that is, the encoding device 21 or the corresponding functionality and the decoding device 22 or the corresponding functionality. In such embodiments, the encoding device 21 or the corresponding functionality and the decoding device 22 or the corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0050] Encoding device 21 and decoding device 22 may be communicatively connected via link 23, and decoding device 22 may receive encoded video data from encoding device 21 via link 23. Link 23 may include one or more media or devices capable of moving encoded video data from encoding device 21 to decoding device 22. In one example, link 23 may include one or more communication media that enable encoding device 21 to transmit encoded video data directly to decoding device 22 in real time. In this example, encoding device 21 may modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated video data to decoding device 22. The one or more communication media may include wireless and / or wired communication media, such as radio frequency spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from encoding device 21 to decoding device 22.

[0051] The encoding device 21 includes an encoder 211. Optionally, the encoding device 21 may also include an image preprocessor 212 and a first communication interface 213. In a specific implementation, the encoder 211, the image preprocessor 212, and the first communication interface 213 may be hardware components in the encoding device 21 or software programs in the encoding device 21.

[0052] They are described as follows:

[0053] The image preprocessor 212 is configured to receive raw image data 214 transmitted from an external terminal and perform preprocessing on the raw image data 214 to obtain preprocessed image data 215 or preprocessed image data 215. For example, the preprocessing performed by the image preprocessor 212 may include retouch, color format conversion (e.g., conversion from a three-primary color (RGB) format to a luminance and color difference signal (YUV, where Y represents luminance and UV represents chrominance) format), color grading, or denoising.

[0054] An image can be considered a two-dimensional array or matrix of pixels (picture elements). The pixels in the array can also be referred to as samples. The number of samples in the array or image in the horizontal and vertical directions (or axes) defines the image's size and / or resolution. To represent color, three color components are typically used, meaning an image can be represented as or contain three sample arrays. For example, in the RBG format or color space, an image includes corresponding arrays of red, green, and blue samples. However, in video coding, each pixel is typically represented in a luma / chroma format or color space. For example, an image in the YUV format includes a luma component designated by Y (sometimes designated by L) and two chroma components designated by U and V. The luma component Y represents the brightness or grayscale intensity (for example, in grayscale images, both are the same), while the two chroma components U and V represent the chroma or color information components. Accordingly, a YUV format image includes a luma sample array of luma sample values ​​(Y) and two chroma sample arrays of chroma values ​​(U and V). An image in RGB format can be converted or transformed into YUV format and vice versa, a process also known as color conversion or transformation. If the image is black and white, the image may include only an array of luminance samples.

[0055] The encoder 211 (or video encoder 211 ) is configured to receive pre-processed image data 215 , and process the pre-processed image data 215 using a relevant prediction mode (such as the prediction mode in various embodiments herein) to provide encoded image data 216 .

[0056] The first communication interface 213 may be configured to receive encoded image data 216 and transmit the encoded image data 216 to the decoding device 22 or any other device (e.g., a memory) via the link 23 for storage or direct reconstruction. The other device may be any device configured for decoding or storage. The first communication interface 213 may be configured to encapsulate the encoded image data 216 into a suitable format, such as a data packet, for transmission over the link 23.

[0057] The decoding device 22 includes a decoder 221. Optionally, the decoding device 22 may also include a second communication interface 222 and an image post-processor 223. They are described as follows:

[0058] Second communication interface 222 may be configured to receive encoded image data 216 from encoding device 21 or any other source, such as a storage device, such as an encoded image data storage device. Second communication interface 222 may be configured to transmit or receive encoded image data 216 via link 23 between encoding device 21 and decoding device 22, such as a direct wired or wireless connection, or via any type of network, such as a wired or wireless network, any combination thereof, or any type of private or public network, or any combination thereof. Second communication interface 222 may be configured to, for example, decapsulate data packets transmitted by first communication interface 213 to obtain encoded image data 216.

[0059] The second communication interface 222 and the first communication interface 213 can both be configured as unidirectional communication interfaces or bidirectional communication interfaces, and can be used to, for example, send and receive messages to establish connections, confirm and exchange any other information related to communication links and / or data transmission, such as encoded image data transmission.

[0060] The decoder 221 (or decoder 221) is configured to receive the encoded image data 216 and provide decoded image data 224 or a decoded image 224. In some embodiments, the decoder 221 may be configured to execute various embodiments described below to implement the image processing method described herein on the decoding side.

[0061] Image post-processor 223 is configured to perform post-processing on decoded image data 224 (also referred to as reconstructed image data) to obtain post-processed image data 225. The post-processing performed by image post-processor 223 may include color format conversion (e.g., from YUV to RGB), color grading, retouch or resampling, or any other processing. Image post-processor 223 may also be configured to transmit the post-processed image data to an external display device for playback. The display device may be or include any type of display for presenting the reconstructed image, such as an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0062] Although, Figure 2The encoding device 21 and the decoding device 22 are shown as separate devices, but device embodiments may also include both the encoding device 21 and the decoding device 22 or the functionality of both, i.e., the encoding device 21 or the corresponding functionality and the decoding device 22 or the corresponding functionality. In such embodiments, the encoding device 21 or the corresponding functionality and the decoding device 22 or the corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0063] It is obvious to those skilled in the art based on the description that the functionality or Figure 2 The presence and (exact) division of the functionality of the encoding device 21 and / or decoding device 22 shown may vary depending on the actual device and application. The encoding device 21 and the decoding device 22 may include any of a variety of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a video camera, a desktop computer, a set-top box, a television, a camera, a car device, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may not use or use any type of operating system.

[0064] The encoder 211 and the decoder 221 can be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology is partially implemented in software, the device may store the software instructions in a suitable non-transitory computer-readable storage medium and may use one or more processors to execute the instructions in hardware to perform the technology of the present disclosure. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) can be considered one or more processors.

[0065] In some cases, Figure 2The video encoding and decoding systems shown in the foregoing are merely examples, and the techniques of the present application can be applied to video encoding arrangements (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding and decoding devices. In other instances, data can be retrieved from local storage, streamed over a network, etc. A video encoding device can encode data and store the data in a memory, and / or a video decoding device can retrieve data from a memory and decode the data. In some instances, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve data from a memory and decode data.

[0066] Currently, in the case of lossless video coding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (for example, there is no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed by, for example, quantization to reduce the amount of data required to represent the video image. The super-resolution algorithm used in the process of reconstructing the video image on the decoder side requires motion estimation, and motion estimation requires a large amount of computing resources. In order to reduce the time of video super-resolution, an embodiment of the present application provides a corresponding image processing method, which includes a decoding device obtaining motion vector information of a target frame in a coded stream, image information of the target frame, and image information of adjacent frames, the target frame and the adjacent frames being images of a first resolution, the target frame being an image to be super-resolution processed, and the adjacent frames including images within a preset period before or after the target frame; the decoding device generates a reconstructed frame based on the motion vector information, the image information of the target frame, and the image information of the adjacent frames, the reconstructed frame being an image of a second resolution, the second resolution being greater than the first resolution, and the motion vector information being used to indicate that the image information of the adjacent frames is adaptively locally sampled with the image information of the target frame. In this way, the present application uses the motion vector information of the target frame in the coded stream to improve the resolution of the reconstructed frame, saving resources for re-estimating the motion vector information.

[0067] Based on the above-mentioned step of the decoding device generating a reconstructed frame based on motion vector information, image information of the target frame, and image information of adjacent frames, an implementation method of the embodiment of the present application may also be as follows: the decoding device adaptively performs local sampling on each position in the image information of the adjacent frames corresponding to the image information of the target frame based on the motion vector information; and the decoding device generates a reconstructed frame based on the image information of the target frame and the image information of the adjacent frames after adaptive local sampling. Through adaptive local sampling, the embodiment of the present application can select image points with high similarity for sampling, reduce the influence of noise in the motion vector information, and improve the robustness of super-resolution.

[0068] Next, based on the above application scenarios and system architecture, combined with Figure 3 The image processing method in the embodiment of the present application is described.

[0069] See also Figure 3 , an embodiment of the image processing method of the present application includes:

[0070] 301. A decoding device obtains image information of a target frame and image information of adjacent frames in a coded bitstream.

[0071] In this embodiment, after receiving the encoded code stream sent by the server, the decoder can decode the encoded video to obtain image information of the target frame and image information of adjacent frames.

[0072] The target frame is the image that needs to be super-resolution processed in this embodiment. Super-resolution is to improve the resolution of the original image by hardware or software methods, that is, the process of obtaining a high-resolution image by a series of low-resolution images is super-resolution reconstruction.

[0073] When the images within a preset period before and after the target frame have completed decoding—for example, when the 2T adjacent frames before and after the target frame have also completed decoding—the decoding device can obtain the image information of these adjacent frames. The period T can be preset or modified as needed. For boundary conditions in a sequence, such as the first or last frame, the operation can be repeated for existing adjacent frames to ensure that the input meets the network's requirements.

[0074] The encoded code stream may be a code stream generated by an image compression coding technology including predictive coding based on motion estimation and compensation. The motion estimation and motion compensation algorithm is used to remove time domain redundant information, that is, the image compression coding technology may apply a motion estimation algorithm to determine motion vector information.

[0075] In this embodiment, the target frame and the adjacent frames are images of a first resolution, and the resolution indicated by the first resolution specifically refers to the low resolution to which the image after decoding the coded code stream belongs.

[0076] 302. The decoding device extracts target features from the image information of the target frame and extracts adjacent features from the image information of the adjacent frame.

[0077] In this embodiment, the decoding device includes multiple feature extraction functions with shared weights. After the decoding device receives the image information of the target frame and the image information of the adjacent frames sent by the decoder, it can use each feature extraction function to extract image features for the image information of the target frame and the image information of each adjacent frame, so as to generate a target feature corresponding to the image information of the target frame and an adjacent feature corresponding to the image information of each adjacent frame.

[0078] Optionally, the above-mentioned image feature may be image texture information of the image.

[0079] The above feature extraction function consists of a convolutional layer and several cascaded residual blocks.

[0080] 303. The decoding device constructs a target feature pyramid based on the target features and constructs an adjacent feature pyramid based on the adjacent features.

[0081] In this embodiment, the decoding device continuously reduces the image size of the above-mentioned target features through filtering and bilinear interpolation downsampling to obtain target features of different scales, and then arranges the target features of different scales according to the scale size to generate a target feature pyramid. The image at the bottom layer of the target feature pyramid corresponds to the original target feature. By averaging every 2*2=4 pixels, a second-level target feature can be formed. And so on, a multi-level target feature pyramid is formed. Optionally, the pyramid structure can be a Gaussian pyramid, a Laplacian pyramid, or a wavelet pyramid, etc., which is not limited here.

[0082] The decoding device continuously reduces the image size of the adjacent features corresponding to the image information of each adjacent frame by filtering and bilinear interpolation downsampling to obtain adjacent features of different scales, and then arranges the adjacent features of different scales according to the scale size to generate an adjacent feature pyramid. For example, based on the 2T adjacent frames before and after the target frame, the embodiment of the present application can generate 2T adjacent feature pyramids accordingly.

[0083] A feature pyramid is a fundamental component of multi-scale object detection systems. An image's feature pyramid is a collection of features arranged in a pyramidal shape. It is obtained by sequentially downsampling an original feature, reducing its size layer by layer. This feature pyramid exhibits a certain degree of scale invariance, enabling the decoding device of this embodiment to detect images of a wide range of scales.

[0084] 304. The decoding device determines a position mapping relationship between the target feature and adjacent features according to the motion vector information.

[0085] In this embodiment, the decoding device can directly determine the motion vector information from the encoded code stream, and determine the position mapping relationship between the target feature and the adjacent feature indicated by the motion vector information. The position mapping relationship is that for the above-mentioned target feature pyramid and adjacent feature pyramid, for each coordinate in the target feature of each scale in the target feature pyramid, there is a corresponding coordinate in the adjacent feature of the same scale in the adjacent feature pyramid.

[0086] 305. The decoding device searches for the second local feature block of the adjacent feature according to the coordinates of the first local feature block of the target feature and the position mapping relationship.

[0087] In this embodiment, the same soft alignment operation is performed for each scale in the adjacent feature pyramid. Taking a certain scale as an example, that is, the features on a certain pixel layer of the feature pyramid, the specific operation is as follows: Figure 4 As shown in the schematic diagram of local feature block extraction, under the guidance of motion vector information, that is, position mapping relationship 41, for each coordinate of target feature 42 in the target feature pyramid, there is a corresponding coordinate in the adjacent feature 223 of any adjacent feature pyramid. The decoding device extracts a first local feature block 421 corresponding to the target feature and a second local feature block corresponding to the adjacent feature from these two corresponding coordinates.

[0088] 306. The decoding device performs feature matching on the first local feature block and the second local feature block through a fully connected layer to determine a set of related attention coefficients.

[0089] In this embodiment, the decoding device permutes the first local feature block of a target feature and the second local feature block of an adjacent feature to form two one-dimensional feature vectors. The two one-dimensional feature vectors are then combined in series and input into a two-layer fully connected layer to generate an attention vector with the same length as the number of pixel values ​​contained in the local feature blocks. The decoding device permutes the attention vector to obtain a set of related attention coefficients between the two local feature blocks.

[0090] The above-mentioned relevant attention coefficient set includes multiple relevant attention coefficients, wherein each relevant attention coefficient can indicate the similarity between a feature point in the first local feature block and a corresponding feature point in the second local feature block;

[0091] 307. The decoding device performs weighted averaging on multiple feature points in the second local feature block based on the relevant attention coefficient set to determine an adjacent feature pyramid after adaptive local sampling.

[0092] In this embodiment, due to the design logic of the encoding framework itself, the position mapping relationship provided by the above-mentioned motion vector information is not necessarily completely real object movement and may contain coding noise. The relevant attention coefficients in the decoding device can allow the network to further refine the search for better matching features within the local neighborhood of the mapping. In this embodiment, after the decoding device obtains the relevant attention coefficients, the multiple relevant attention coefficients can be combined with the local feature blocks of adjacent features for point-by-point multiplication and summation to achieve sampling processing of the second local feature block.

[0093] The relevant attention coefficient is the importance weight of each feature in the local feature block indicating the adjacent features, which is used to improve the feature quality of the adjacent features after adaptive local sampling.

[0094] The same steps as above need to be performed for the local feature block of each coordinate on the target feature on the target feature pyramid.

[0095] When the decoding device performs adaptive local sampling, it only performs adaptive local sampling on one local feature block in a target feature and one local feature block in an adjacent feature. For the local feature blocks of the adjacent features corresponding to the local feature blocks of other coordinates in the other adjacent feature pyramids of the 2T adjacent feature pyramids and the target feature, it is also necessary to perform adaptive local sampling on them in the same steps to perform adaptive local sampling on all adjacent feature pyramids.

[0096] 308. The decoding device calculates an attention map based on the target feature and the adjacent features after adaptive local sampling.

[0097] After the decoding device performs adaptive local sampling on the pyramid of adjacent features, it can generate an attention map in the time domain based on the adaptive local sampling quality of the features of adjacent frames.

[0098] For example, the decoding device can calculate the inner product of the adjacent frame features and the target features after adaptive local sampling at each coordinate point. This inner product can represent the similarity between the adjacent features after adaptive local sampling and the target features at that point. The similarity also indicates the quality of the adaptive local sampling at that point to a certain extent. The decoding device can obtain an attention map of the same size as the feature size based on the above adaptive local sampling quality.

[0099] 309. The decoding device performs feature enhancement processing on the adjacent features after adaptive local sampling and the attention map.

[0100] After determining the above attention map, the decoding device can dynamically fuse the features of the adjacent frames and the current frame after adaptive local sampling by increasing the weights of high-quality local areas and reducing the weights of low-quality local areas.

[0101] Exemplarily, the decoding device multiplies the adjacent features in the adjacent feature pyramid after adaptive local sampling with the above-mentioned attention map point by point, so that those areas on the adjacent features that are more similar to the target features are adaptively assigned higher attention, that is, the possible contribution to the super-resolution result has a higher weight, the required features are enhanced, and possible interference such as mismatching is suppressed.

[0102] 310. The decoding device stacks and convolves all adjacent features and target features after feature enhancement to generate fused features, and determines a fused feature pyramid.

[0103] After determining that all adjacent features have been enhanced, the decoding device can stack the adjacent features after feature enhancement and the target features, that is, superimpose the features on the adjacent features after feature enhancement on the target features, and then obtain the fused features through a convolution layer. When all adjacent features after feature enhancement are stacked on the target features in the target feature pyramid and convolved, a fused feature pyramid can be obtained.

[0104] After the decoding device performs a feature fusion, it is necessary to detect whether there are adjacent features that have not been enhanced. If there are adjacent features that have not been enhanced, the adjacent features and the target features need to be enhanced as described above until all adjacent features are enhanced and fused with the target features, and a complete fusion feature pyramid is determined.

[0105] 311. The decoding device generates an optimized feature pyramid by calculating the fused feature pyramid through the second cascade residual block.

[0106] The decoding device uses cascaded residual blocks with scale fusion to reconstruct features of the fused feature pyramid. Unlike the residual blocks that generally only process a single scale, the second cascaded residual block mentioned above will add additional upsampling or downsampling operations at the end of the jump connection according to the different levels of the features in the pyramid. This allows the reconstructed residuals of features of different scales to fully exchange information, enhance the quality of the reconstructed features, obtain optimized fusion features, and determine the optimized feature pyramid.

[0107] 312. The decoding device performs size expansion and convolution on the optimized feature pyramid to generate a reconstructed residual signal.

[0108] Since downsampling in the decoding device reduces the size of the feature image, it is necessary to enlarge the size of the above-mentioned optimized feature pyramid through a sub-pixel convolution layer, and then generate a high-resolution reconstructed residual signal through a convolution layer.

[0109] 313. The decoding device adds the reconstructed residual signal to the image magnification result to obtain a reconstructed frame.

[0110] In this embodiment, the decoding device may upscale the image information of the target frame through a bilinear interpolation upsampling operation. The decoding device may add the reconstructed residual signal to the upsampled image information of the target frame to obtain image information of the reconstructed frame and determine the reconstructed frame. The reconstructed frame is the target frame after super-resolution.

[0111] In the technical solution of the embodiment of the present application, the decoding device generates a high-resolution reconstructed frame by performing neural network processing on the motion vector information, target feature pyramid and adjacent feature pyramid obtained from the encoded bitstream. The encoded bitstream contains certain motion information, and the computational cost of extracting the motion information in the bitstream is negligible, so the time for video super-resolution can be greatly reduced.

[0112] The super-resolution process in the image processing of the embodiment of the present application can be implemented by a pre-trained network model. For example, see Figure 5 , a schematic diagram of the architecture of a decoding device according to an embodiment of the present application, wherein the decoding device 22 may include a decoder 221, a network model 222, a graphics processing unit (GPU) memory 223, and an output buffer 224. They are described as follows:

[0113] The decoder 221 is a device that performs a decompression operation on an encoded code stream. Optionally, the decoder 221 can be a video decoder that supports codecs such as H.264 / high-efficiency video coding (HEVC) / versatile video coding (VVC), such as an HEVC decoder. The decoder 221 of the embodiment of the present application adds a motion vector information output interface.

[0114] The network model 222 is implemented in the form of a product in an embodiment of the present application. It is a program code contained in a machine learning and deep learning platform software and deployed on a decoding device. The program code of the embodiment of the present application exists outside the existing decoder 221. The network model 222 can be generated by supervised training using a machine learning method for data of low-resolution decoded video and its unencoded high-resolution video. The network model 222 in this embodiment of the present application is designed with a feature extraction module 2221, a flexible alignment module 2222, a multi-frame feature fusion module 2223 and a feature super-resolution reconstruction module 2224.

[0115] The network model 222 first includes a feature extraction module 2221. Since image features have important physical meanings in deep learning methods, this module is designed to convert the input decoded frame from the pixel domain to the feature domain.

[0116] Flexible alignment module 2222, which receives the motion vectors in the extracted code stream from the decoder 221, and uses this as a guide to design a local attention mechanism at multiple scales to achieve flexible alignment of adjacent frames at the feature level.

[0117] Multi-frame feature fusion module 2223, this module receives the aligned adjacent frame features and the current frame features, uses the attention mechanism in the time domain, and completes the feature fusion operation.

[0118] The feature super-resolution reconstruction module 2224 receives the fused image features and uses cascaded multi-scale fusion residual blocks and sub-pixel convolution to complete the super-resolution reconstruction of the decoded video and generate a reconstructed frame.

[0119] The GPU memory 223 is used to support the execution of program codes for the calculations of each module in the network model 222 .

[0120] The output buffer 224 receives and stores the reconstructed frames output by the network model 222 .

[0121] The embodiments of the present application implement program code for HEVC standard video super-resolution decoding in an open-source Python-first deep learning framework (PyTorch) machine learning platform and run on a decoding unit equipped with an NVIDIA GPU card. The NVIDIA GPU card provides computing acceleration capabilities through the Compute Unified Device Architecture (CUDA) programming interface. In this embodiment, the network model inference process in the distributed PyTorch machine learning platform can be accelerated, and the trained model can be directly reconstructed end-to-end from decoded video with compression noise.

[0122] Based on the above architecture, the image processing method in the embodiment of the present application is described below:

[0123] See also Figure 6 , Figure 6 The image processing flow chart of the feature extraction module shown in the figure, the feature extraction module converts the target frame and adjacent frames output by the decoder from the pixel domain to the feature domain, as follows:

[0124] 601. A feature extraction module obtains image information of a target frame and image information of adjacent frames from a decoder.

[0125] In this embodiment, after the decoder receives the encoded code stream sent by the source device, it can decode the encoded video to obtain the image information of the target frame and the image information of the adjacent frames, and the feature extraction module can receive the image information of the target frame and the image information of the adjacent frames transmitted by the decoder.

[0126] The target frame is the image that needs to be super-resolution processed in this embodiment. Super-resolution is to improve the resolution of the original image by hardware or software methods, that is, the process of obtaining a high-resolution image by a series of low-resolution images is super-resolution reconstruction.

[0127] When the images within a preset period before and after the target frame have completed decoding—for example, when the 2T adjacent frames before and after the target frame have also completed decoding—the decoder outputs the image information of these adjacent frames. The period T can be preset or modified as needed. For edge cases in sequences, such as the first or last frame, the operation can be repeated for existing adjacent frames to ensure that the input meets the network's requirements.

[0128] The encoded code stream may be a code stream generated by an image compression coding technology including predictive coding based on motion estimation and compensation. The motion estimation and motion compensation algorithm is used to remove time domain redundant information, that is, the image compression coding technology may apply a motion estimation algorithm to determine motion vector information.

[0129] In this embodiment, the target frame and the adjacent frames are images of a first resolution, and the resolution indicated by the first resolution specifically refers to the low resolution to which the image after decoding the coded code stream belongs.

[0130] 602. The feature extraction module extracts target features from the image information of the target frame and extracts adjacent features from the image information of the adjacent frames.

[0131] 603. The feature extraction module constructs a target feature pyramid based on the target features and constructs an adjacent feature pyramid based on the adjacent features.

[0132] Steps 602-603 and Figure 3 Steps 302-303 of the image processing method shown are similar and will not be described in detail here.

[0133] See also Figure 7 , Figure 7 The image processing flow chart of the flexible alignment module shown in FIG. 1 is a flowchart of the flexible alignment module. The flexible alignment module implements flexible alignment of adjacent frames based on the target feature pyramid and adjacent feature pyramid output by the feature extraction module. For each adjacent feature pyramid, the specific steps are as follows:

[0134] 701 The flexible alignment module receives the motion vector information from the decoder, and the target feature pyramid and the adjacent feature pyramid from the feature extraction module.

[0135] In this embodiment, the encoded bitstream may be generated using an image compression coding technique that includes predictive coding based on motion estimation and compensation. The motion estimation and compensation algorithms are used to remove temporal redundant information. That is, the image compression coding technique may use a motion estimation algorithm to determine motion vector information. The decoder may extract this motion vector information from the encoded bitstream and send it to the flexible alignment module.

[0136] The flexible alignment module can also receive the feature extraction module Figure 3 The target feature pyramid and the adjacent feature pyramid.

[0137] 702. The flexible alignment module determines a position mapping relationship between the target feature and adjacent features according to the motion vector information.

[0138] 703. The flexible alignment module searches for the second local feature block of the adjacent feature according to the coordinates of the first local feature block of the target feature and the position mapping relationship.

[0139] 704. The flexible alignment module performs feature matching on the first local feature block and the second local feature block through a fully connected layer to determine a set of related attention coefficients.

[0140] 705. The flexible alignment module performs weighted averaging on the second local feature block based on the relevant attention coefficient set to determine an adjacent feature pyramid after adaptive local sampling.

[0141] Steps 702-705 and Figure 3 Steps 304-307 in the image processing method shown are similar and will not be described in detail here.

[0142] See also Figure 8 , Figure 8 The image processing flow chart of the multi-frame feature fusion module shown in the figure is as follows:

[0143] 801. The multi-frame feature fusion module receives the adjacent feature pyramid after adaptive local sampling from the flexible alignment module.

[0144] After the flexible alignment module performs adaptive local sampling on the adjacent feature pyramids, the adaptive local sampled adjacent feature pyramids are sent to the multi-frame feature fusion module.

[0145] 802. The multi-frame feature fusion module calculates the attention map based on the target feature and the adjacent features after adaptive local sampling.

[0146] 803. The multi-frame feature fusion module performs feature enhancement processing on the adjacent features after adaptive local sampling and the attention map.

[0147] 804. The multi-frame feature fusion module stacks and performs convolution calculations on all adjacent features and target features after feature enhancement to generate fused features, and determines a fused feature pyramid.

[0148] Steps 802-804 and Figure 3 Steps 308-310 in the image processing method shown are similar and will not be described in detail here.

[0149] See also Figure 9 , Figure 9 The image processing flow chart of the feature super-resolution reconstruction module shown in the figure is as follows:

[0150] 901. The feature super-resolution reconstruction module receives the fused feature pyramid from the multi-frame feature fusion module.

[0151] 902. The feature super-resolution reconstruction module generates an optimized feature pyramid by calculating the fused feature pyramid through the second cascade residual block.

[0152] 903. The feature super-resolution reconstruction module performs size expansion and convolution on the optimized feature pyramid to generate a reconstructed residual signal.

[0153] 904. The feature super-resolution reconstruction module adds the reconstructed residual signal and the image magnification result to obtain a reconstructed frame.

[0154] Steps 902-904 and Figure 3 Steps 311-313 in the image processing method shown are similar and will not be described in detail here.

[0155] Based on the technical solutions of the embodiments of this application, Figure 10The super-resolution results of the embodiment of the present application are compared with the super-resolution results of the prior art. The vertical axis is the peak signal to noise ratio (PSNR) in decibels (dB), and the horizontal axis is the test time of each frame in milliseconds (ms). The super-resolution results of the embodiment of the present application Ours are compared with the compression noise removal technology and lightweight super-resolution technology of the prior art. The two-step method, such as: variable-filter-size residue-learning convolutional neural networks (VRCNN) + video efficient sub-pixel convolution network (VESPCN), multi-frame quality enhancement method (MFQE) + VESPCN, deep convolutional neural networks-based auto decoder (DCAD) + video super-resolution method based on optical flow super-resolution (super-resolving optical flow for video) Super-resolution (SOFVSR) and MFQE+SOFVSR, etc., have greatly improved the peak signal-to-noise ratio. Compared with the end-to-end (compression noise removal + video super-resolution) methods such as the progressive fusion video super-resolution network via exploiting non-Localspatio-temporal correlations (PFNL), the reasonable use of the characteristics of motion vectors has significantly reduced the average processing time per frame. For example, Figure 10 The super-resolution test time of the embodiment of the present application is shown as 280ms compared to the test time of 850ms used by PFNL.

[0156] The residual learning convolutional neural network with variable filter size is used for lightweight compression noise removal network, which consists of 4 convolutional layers.

[0157] The multi-frame quality enhancement method is used to perform an end-to-end network for compressed video noise removal using motion estimation and motion compensation, as well as the idea of ​​"good frames compensate bad frames".

[0158] An automatic decoder based on a deep convolutional neural network is used to compress the noise removal network, which consists of 10 convolutional layers.

[0159] Efficient sub-pixel convolutional networks for video are used to exploit motion estimation and motion compensation to perform video super-resolution by aligning adjacent frames while taking into account temporal correlations.

[0160] Video super-resolution methods based on optical flow super-resolution utilize motion estimation and motion compensation to align adjacent frames and account for temporal correlations, thereby achieving video super-resolution. Compared to efficient sub-pixel convolutional networks for video, which only predict low-resolution optical flow, video super-resolution methods based on optical flow super-resolution predict more accurate high-resolution optical flow.

[0161] Progressive Fusion Video Super-Resolution with Non-local Spatio-temporal Correlations The network is used to perform end-to-end video super-resolution by computing non-local attention and the proposed progressive fusion module.

[0162] The image processing method is described above. The decoding device of an embodiment of the present application is introduced below with reference to the accompanying drawings.

[0163] Figure 11 Schematic diagram of an embodiment of the decoding device 110 in the embodiment of the present application.

[0164] like Figure 11 As shown, an embodiment of the present application provides a decoding device, which includes:

[0165] An acquisition unit 1101 is configured to acquire motion vector information of a target frame in a coded bitstream, image information of the target frame, and image information of adjacent frames, wherein the target frame and the adjacent frames are images of a first resolution, the target frame is an image to be super-resolution processed, and the adjacent frames include images located within a preset period before or after the target frame;

[0166] A generation unit 1102 is used to generate a reconstructed frame based on the motion vector information, the image information of the target frame and the image information of the adjacent frame, wherein the reconstructed frame is an image of a second resolution, the second resolution is greater than the first resolution, and the motion vector information is used to indicate adaptive local sampling of the image information of the adjacent frames and the image information of the target frame.

[0167] The solution provided in the embodiment of the present application is that the decoding device generates high-resolution reconstructed frames by performing neural network processing on the motion vector information, target feature pyramid and adjacent feature pyramid obtained from the encoded bitstream. The encoded bitstream contains certain motion information, and the computational cost of extracting the motion information in the bitstream is negligible, so the time for video super-resolution can be greatly reduced.

[0168] Optionally, the generation unit 1102 is specifically used to generate a target feature pyramid based on the image information of the target frame, and generate an adjacent feature pyramid based on the image information of each adjacent frame, where the target feature pyramid includes target features of multiple scales, and each adjacent feature pyramid includes adjacent features of multiple scales; based on the motion vector information and taking the position of each target feature as a reference, the adjacent features at the position corresponding to each target feature in each adjacent feature pyramid are adaptively locally sampled; the target feature pyramid and each adjacent feature pyramid after the adaptive local sampling are fused to generate a fused feature pyramid, where the fused feature pyramid includes fused features of multiple scales; and the fused feature pyramid is processed to generate a reconstructed frame.

[0169] Optionally, the generation unit 1102 is also used to search for the second local feature block for each adjacent feature pyramid based on the coordinates of the first local feature block in the target feature and the mapping relationship between the first local feature block contained in the motion vector information and the second local feature block in the adjacent feature; perform feature matching on the first local feature block and the second local feature block through a fully connected layer to determine a set of relevant attention coefficients, the set of relevant attention coefficients including multiple relevant attention coefficients, wherein each relevant attention coefficient indicates the similarity between a feature point in the first local feature block and a corresponding feature point in the second local feature block; perform weighted averaging on multiple feature points in the second local feature block based on the set of relevant attention coefficients to determine the adjacent feature pyramid after adaptive local sampling.

[0170] Optionally, the generation unit 1102 is further used to calculate an attention map for each adjacent feature pyramid based on the target feature and the adjacent features after adaptive local sampling, where the attention map is used to represent the similarity between the adjacent features after adaptive local sampling and the target feature; perform feature enhancement processing on the adjacent features after adaptive local sampling and the attention map; stack and convolve all the enhanced adjacent features and the target feature to generate a fused feature, and determine the fused feature pyramid.

[0171] Optionally, the generation unit 1102 is further configured to perform convolution processing on the image information of the target frame, and then perform processing on the first cascade residual block to generate target features of multiple scales; and generate a target feature pyramid by performing bilinear interpolation on the target features of multiple scales.

[0172] Optionally, the generation unit 1102 is further configured to perform convolution processing on the image information of each adjacent frame, and then perform processing on the first cascade residual block to generate adjacent features of multiple scales; and generate an adjacent feature pyramid by bilinear interpolation of the adjacent features of multiple scales.

[0173] Optionally, the generating unit 1102 is further configured to generate an optimized feature pyramid by calculating the fused feature pyramid through the second cascade residual block; perform size expansion and convolution on the optimized feature pyramid to generate a reconstructed residual signal; add the reconstructed residual signal to the image magnification result to obtain a reconstructed frame, where the image magnification result is generated by bilinear interpolation of the image information of the target frame.

[0174] The decoding device described above can be understood by referring to the corresponding content of the aforementioned method embodiment part, and will not be described in detail here.

[0175] Figure 12 1 is a schematic diagram of a UP structure provided in an embodiment of the present application. The decoding device 1200 may include one or more central processing units (CPU) 1201 and a memory 1205. The memory 1205 stores one or more applications or data.

[0176] Memory 1205 may be volatile or persistent storage. The program stored in memory 1205 may include one or more modules, each of which may include a series of instruction operations on the service control unit. Furthermore, central processing unit 1201 may be configured to communicate with memory 1205 and execute the series of instruction operations in memory 1205 on decoding device 1200.

[0177] The decoding device 1200 may also include one or more power supplies 1202, one or more wired or wireless network interfaces 1203, one or more input and output interfaces 1204, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0178] The decoding device 1200 can perform the aforementioned Figures 3 to 9 The operations performed by the decoding device in the illustrated embodiment will not be described in detail here.

[0179] In another embodiment of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device executes the above-mentioned Figures 3 to 9 The steps of the image processing method executed by the processor.

[0180] In another embodiment of the present application, a computer program product is further provided. The computer program product includes computer-executable instructions stored in a computer-readable storage medium. When the processor of the device executes the computer-executable instructions, the device executes the above-mentioned Figures 3 to 9 The steps of the image processing method executed by the processor.

[0181] In another embodiment of the present application, a chip system is further provided. The chip system includes at least one processor, which is used to support the decoding device to implement the above Figures 3 to 9 The steps of the image processing method executed by the processor in the chip. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for the decoding device. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0182] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0183] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0185] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0186] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0187] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0188] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this.

Claims

1. An image processing method, characterized in that: include: The decoding device obtains motion vector information of a target frame in an encoded code stream, image information of the target frame, and image information of adjacent frames, wherein the target frame and the adjacent frames are images of a first resolution, the target frame is an image to be super-resolution processed, and the adjacent frames include one or more images located within a preset period before or after the target frame; The decoding device generates a target feature pyramid based on the image information of the target frame, and generates a neighboring feature pyramid based on the image information of each adjacent frame, wherein the target feature pyramid includes target features at multiple scales, and each neighboring feature pyramid includes neighboring features at multiple scales; The decoding device performs adaptive local sampling on adjacent features at positions corresponding to each target feature in each adjacent feature pyramid based on the motion vector information and taking the position of each target feature as a reference; The decoding device generates a reconstructed frame based on the image information of the target frame and the image information of the adjacent frame after adaptive local sampling. The reconstructed frame is an image of a second resolution, which is greater than the first resolution. The motion vector information is used for adaptive local sampling of the image information of the adjacent frames and the image information of the target frame.

2. The image processing method according to claim 1, wherein: The decoding device generates the reconstructed frame according to the image information of the target frame and the image information of the adjacent frame after adaptive local sampling, including: The decoding device fuses the target feature pyramid with each of the adjacent feature pyramids after adaptive local sampling to generate a fused feature pyramid, where the fused feature pyramid includes fused features at multiple scales; The decoding device processes the fused feature pyramid to generate the reconstructed frame.

3. The image processing method according to claim 1, wherein: The decoding device adaptively locally samples adjacent features at positions corresponding to each target feature in each adjacent feature pyramid based on the motion vector information and taking the position of each target feature as a reference, including: for each adjacent feature pyramid, The decoding device searches for the second local feature block in the target feature according to the coordinates of the first local feature block and the mapping relationship between the first local feature block and the second local feature block in the adjacent feature included in the motion vector information; The decoding device performs feature matching on the first local feature block and the second local feature block through a fully connected layer to determine a set of relevant attention coefficients, where the set of relevant attention coefficients includes a plurality of relevant attention coefficients, wherein each relevant attention coefficient indicates a similarity between a feature point in the first local feature block and a corresponding feature point in the second local feature block; The decoding device performs weighted averaging on multiple feature points in the second local feature block based on the relevant attention coefficient set to determine the adjacent feature pyramid after adaptive local sampling.

4. The image processing method according to claim 2, wherein: The decoding device fuses the target feature pyramid and the adjacent feature pyramid after adaptive local sampling to generate a fused feature pyramid, including: for each adjacent feature pyramid, The decoding device calculates an attention map based on the target feature and the adjacent features after the adaptive local sampling, wherein the attention map is used to represent the similarity between the adjacent features after the adaptive local sampling and the target feature; The decoding device performs feature enhancement processing on the adjacent features after the adaptive local sampling and the attention map; The decoding device stacks and performs convolution calculation on all adjacent features after feature enhancement processing and the target feature to generate the fused feature, and determines the fused feature pyramid.

5. The image processing method according to any one of claims 1 to 4, characterized in that: The decoding device generating a target feature pyramid according to image information of the target frame includes: The decoding device performs convolution processing on the image information of the target frame, and then performs processing on the first cascade residual block to generate target features; The decoding device generates target features of multiple scales by bilinear interpolation from the target features, and constructs the target feature pyramid.

6. The image processing method according to any one of claims 1 to 4, characterized in that: The decoding device generates an adjacent feature pyramid according to the image information of each adjacent frame, including: The decoding device performs convolution processing on the image information of each of the adjacent frames, and then performs processing on the first cascade residual block to generate an adjacent feature corresponding to the image information of each of the adjacent frames; The decoding device generates adjacent features of multiple scales by bilinear interpolation from the adjacent features, and constructs the adjacent feature pyramid.

7. The image processing method according to claim 2 or 4, characterized in that: The decoding device processes the fused feature pyramid to generate the reconstructed frame, including: The decoding device calculates the fused feature pyramid through a second cascade residual block to generate an optimized feature pyramid; The decoding device performs size expansion and convolution on the optimized feature pyramid to generate a reconstructed residual signal; The decoding device adds the reconstructed residual signal to an image magnification result to obtain the reconstructed frame, where the image magnification result is generated by bilinear interpolation of image information of the target frame.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, and when the computer executes the program, the method according to any one of claims 1 to 7 is performed.

9. A computing device, characterized in that comprising a processor and a computer-readable storage medium storing a computer program; The processor is coupled to the computer-readable storage medium, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that When the computer program product is executed on a computer, the computer performs the method according to any one of claims 1 to 7.

11. A chip system, characterized in that: The method comprises a processor, wherein the processor is called to execute the method according to any one of claims 1 to 7.

Images