Image processing method and device
The method addresses the lack of high-frequency information in super-resolution images by aligning and accumulating residual frames using motion vectors, enhancing image quality through neural network processing.
Patent Information
- Application Number
- CN201910840356.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-09-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-09-06
AI Technical Summary
Existing super-resolution technologies struggle to recover high-frequency information in low-resolution images, leading to poor image quality and user experience.
An image processing method that utilizes motion vector information and residual frames from video encoding and decoding processes to align and accumulate residuals, inputting these aligned and residual frames into a neural network to enhance high-frequency information and improve image resolution.
Enhances image quality by compensating for low resolution due to lacking high-frequency information without increasing hardware costs, improving the viewing experience.
Smart Images

Figure CN112465698B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device for generating a clear image during image super-resolution. Background Art
[0002] With the increasing popularity of mobile phones, tablets, smart TVs and other devices and the improvement of mobile communication technology, short videos, streaming videos and real-time video calls have sprung up like mushrooms after a rain, and more and more of them occupy the terminal screen. In order to enable users to experience these video services, not only can they meet the smoothness of video playback under uncertain network conditions, but also reduce the cost of edge storage. Therefore, the video resolution transmitted to users is often not high, and the picture quality is blurred, noisy, blocky, and has negative effects such as jagged edges. In order to remove these negative effects, super-resolution technology can generally be used to improve the picture quality without increasing the edge storage and the amount of data transmitted. The super-resolution technology mentioned above can also be called "super-resolution" technology.
[0003] Super-resolution technology mainly refers to an important computer vision and image processing method for restoring low-resolution images to high-resolution images. It is especially used in related fields such as medical imaging, security monitoring and television entertainment. The difficulty of super-resolution technology lies in the fact that low-resolution images lose the amount of information related to the high-frequency part, and at the same time add negative effects caused by the recovery ability of the camera's photosensitive element, compression coding loss and transmission channel packet loss.
[0004] Traditional super-resolution algorithms based on image features generally perform restoration from the perspectives of denoising, filtering, edge contour analysis, extraction and fitting, and wavelet transform to restore frequency domain information. With the development of deep learning related fields, more and more work has turned to using convolutional neural network models for super-resolution, and has achieved breakthrough results. Figure 1 As shown in the figure, the current super-resolution work based on deep learning can be mainly divided into the following three directions: single image super-resolution (SISR), sequence super-resolution (SSR) and reference super-resolution (RSR).
[0005] Among them, SISR only sends a single frame image to the neural network for super-resolution processing; SSR sends multiple consecutive frames of images in the video stream to the neural network for super-resolution processing; RSR uses previously super-resolved images, or pre-placed image templates with special textures, and inputs them together with the input image / frame sequence into the neural network for super-resolution processing. The three research directions mentioned above all have a common problem, that is, in the super-resolution process, the low-resolution frame image lacks high-frequency information, and the lost high-frequency information is almost impossible to recover through a single image. As a result, due to the lack of information on the high-frequency edge of the image, the resolution of the super-resolved image is low, affecting the user's viewing experience. Summary of the invention
[0006] The embodiment of the present application discloses an image processing method for solving the problem of low resolution of the super-resolved image due to the lack of high-frequency information in the image generated during super-resolution.
[0007] In order to solve this technical problem, this application discloses the following technical solutions:
[0008] In a first aspect, an embodiment of the present application provides an image processing method, which can be applied to a receiving device, such as a destination device or a decoder, and specifically, the method includes:
[0009] A video code stream is obtained, wherein the video code stream includes a first frame image, a second frame image, and a third frame image that are adjacent in time sequence; the video code stream is decoded to obtain a first aligned frame, a second aligned frame, and at least one residual between the first frame image, the second frame image, and the third frame image, and at least one residual frame is generated according to the at least one residual; and the second frame image is super-resolved according to the at least one residual frame, the first aligned frame, and the second aligned frame to obtain a second frame image after super-resolvation.
[0010] The first aligned frame is generated after the first frame image is moved to the second frame image by pixel blocks according to the first motion vector, the second aligned frame is generated after the third frame image is moved to the second frame image by pixel blocks according to the second motion vector, and the residual is the pixel difference between each macroblock of the previous frame image and the next frame image after motion compensation is performed on the previous frame image according to the motion vector;
[0011] The method provided in this aspect uses the motion vector information and residual information in the video encoding and decoding process to align and accumulate residuals of video frames, and uses the aligned frames and residual frames as the input of the neural network. Since the accumulated frames output high-frequency information after passing through the neural network, the high-frequency information is fitted back into the brightness channel to enhance edge details, thereby compensating for the problem of low image resolution caused by the lack of high-frequency information in the low-resolution video frame during the super-resolution process. This method improves the quality of video image quality without increasing hardware costs.
[0012] In combination with the first aspect, in a possible implementation of the first aspect, generating at least one residual frame according to the at least one residual includes: generating a first residual frame according to the first residual;
[0013] The first residual and the first aligned frame satisfy the following relationship:
[0014]
[0015] in, represents the second frame image, represents the first alignment frame, represents the first residual, i represents a macroblock of the first frame image, iT i (t2) Indicates that the macroblock i is based on its corresponding motion vector T i (t2) For the macroblock after the move, t1 represents the time when the first frame image is generated, and t2 represents the time when the second frame image is generated.
[0016] In combination with the first aspect, in another possible implementation of the first aspect, the second frame image is super-resolved according to the at least one residual frame, the first aligned frame and the second aligned frame to obtain a super-resolved second frame image, including: inputting the at least one residual frame into a neural network for feature extraction to obtain at least one first feature image; inputting the first aligned frame, the second aligned frame and the second frame image into the neural network for feature extraction to obtain at least one second feature image; inputting the at least one first feature image into a first super-resolved network for processing to generate high-frequency information; inputting the at least one second feature map into a second super-resolved network for processing to generate a brightness channel; and fusing the high-frequency information with the brightness channel to generate the super-resolved second frame image.
[0017] In this implementation, at least one residual frame and at least one aligned frame are input into the same neural network for super-resolution processing, thereby improving parameter utilization, reducing the amount of model parameters, and improving super-resolution processing efficiency.
[0018] In combination with the first aspect, in another possible implementation of the first aspect, before inputting the at least one residual frame into the neural network for feature extraction, it also includes: determining the macroblock of the area of interest in the first residual frame, the macroblock of the area of interest being the macroblock in which the sum of all pixel values contained in the current macroblock exceeds a preset value; and determining the area of interest of the residual frames remaining in the at least one residual frame except the first residual frame based on the macroblock of the area of interest in the first residual frame.
[0019] Correspondingly, the at least one residual frame is input into a neural network for feature extraction, including: inputting macroblocks of all regions of interest of the at least one residual frame into the neural network for feature extraction, the at least one residual frame including the first residual frame and the remaining residual frames.
[0020] In this implementation, each frame of the image is divided into an area of interest and an area of non-interest, that is, the residual accumulated texture details and the motion characteristics of the previous and next frames of each frame of the image are analyzed, and only the macroblocks in the area of interest are super-resolution processed, while the super-resolution processing result of the previous frame can be directly used for the area of non-interest, thereby avoiding super-resolution processing of the entire image, reducing the amount of calculation, reducing power consumption, delay and memory overhead, and improving the super-resolution efficiency of a single frame image, thereby achieving the beneficial effect of obtaining a super-resolution image in real time in a short time.
[0021] In combination with the first aspect, in another possible implementation of the first aspect, inputting the first aligned frame, the second aligned frame, and the second frame image into the neural network for feature extraction includes: inputting macroblocks of regions of interest in the first aligned frame and the second aligned frame, and the second frame image into the neural network for feature extraction. The regions of interest in the first aligned frame and the second aligned frame are the same as the regions of interest in the first residual frame.
[0022] In combination with the first aspect, in another possible implementation of the first aspect,
[0023] The first frame image, the second frame image and the third frame image are three frame images in a first image group;
[0024] Alternatively, the first frame of image is the last frame of image in the first image group, and the second frame of image and the third frame of image are the first two frames of image in the second image group;
[0025] Alternatively, the first frame image and the second frame image are the last two frame images in the first image group, and the third frame image is the first frame image in the second image group.
[0026] In the second aspect, an embodiment of the present application further provides an image processing device, which includes a unit for executing the steps in the first aspect and each implementation method of the first aspect, such as an acquisition unit, a processing unit, and a sending unit, and may also include a storage unit.
[0027] Optionally, the device is a communication device or a chip.
[0028] Optionally, the apparatus may be integrated in an encoder, a decoder, or in a video decoding system. In addition, the apparatus may also be a source device or a destination device. This embodiment does not limit the specific form of the image processing apparatus.
[0029] In the third aspect, an embodiment of the present application further provides an electronic device or electronic apparatus, such as a video encoding device, which includes a processor, a transceiver and a memory, wherein the memory is coupled to the processor and is used to store computer program instructions necessary for the video encoding device. When the processor calls the computer program instructions stored in the memory, the device can execute the methods in the aforementioned first aspect and various implementation methods of the first aspect.
[0030] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores instructions, and when the instructions are executed on a computer or a processor, the instructions are used to execute the methods in the aforementioned first aspect and various implementations of the first aspect.
[0031] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes computer instructions. When the instructions are executed by a computer or a processor, the method in the aforementioned first aspect and various implementation methods of the first aspect can be implemented.
[0032] In the sixth aspect, an embodiment of the present application also provides a chip system, which includes a processor and an interface circuit, wherein the interface circuit is coupled to the processor, and the processor is used to execute computer programs or instructions to implement the methods in the aforementioned first aspect and various implementation methods of the first aspect; wherein the interface circuit is used to communicate with other modules outside the chip system.
[0033] The image processing method provided in the embodiment of the present application uses the motion vector information and residual information in the video encoding and decoding process to perform alignment and residual accumulation calculations on the video frames, and uses the aligned frames and accumulated residual frames as the input of the neural network module. Since the residual accumulation frames output high-frequency information after passing through the neural network model, the high-frequency information is fitted back into the brightness channel, thereby enhancing the edge details and compensating for the problem of low image resolution caused by the lack of high-frequency information in the low-resolution video frames during the super-resolution process. This method improves the quality of the video image without increasing the hardware cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 A schematic diagram of the classification of a super-resolution algorithm provided in this application;
[0035] Figure 2A A schematic block diagram of a video encoding and decoding system provided in an embodiment of the present application;
[0036] Figure 2B An exemplary structural diagram of a video decoding system provided in an embodiment of the present application;
[0037] Figure 3A A schematic diagram of a video encoding and decoding method provided in an embodiment of the present application;
[0038] Figure 3B A schematic diagram of a flow chart of another video encoding and decoding method provided in an embodiment of the present application;
[0039] Figure 4A An exemplary structural diagram of an encoder 20 provided in an embodiment of the present application;
[0040] Figure 4B An exemplary structural diagram of a decoder 30 provided in an embodiment of the present application;
[0041] Figure 5 A flowchart of an image processing method provided in an embodiment of the present application;
[0042] Figure 6 A flowchart of using a neural network to process and generate a super-resolution image is provided in an embodiment of the present application;
[0043] Fig. 7A A flowchart of another image processing method provided in an embodiment of the present application;
[0044] Figure 7B A schematic diagram of a super-resolution image generated by combining super-resolution effects provided in an embodiment of the present application;
[0045] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;
[0046] Fig. 9 A schematic diagram of the structure of a video decoder device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0048] Before describing the technical solution of the embodiments of the present application, the technical scenarios and related technical terms of the present application are first introduced in conjunction with the accompanying drawings.
[0049] The technical solution of this embodiment is applied to the technical field of image processing, mainly for super-resolution processing of a series of continuous frames in a video. The video can be understood as a number of frames of images (also described as images in this field) played in a certain order and frame rate. The process of processing the video code stream includes video encoding and video decoding.
[0050] Further, video encoding is the process of performing an encoding operation on each frame of the video to obtain the encoding information of each frame. Video encoding is performed on the source side. Video decoding is the process of reconstructing each frame of the image based on the encoding information of each frame of the image. Video decoding is performed on the destination side. The combination of video encoding operation and video decoding operation can be called video coding and decoding (encoding and decoding).
[0051] Existing video codecs operate according to video codec standards (e.g., the high-efficiency video codec H.265 standard) and follow the high-efficiency video coding standard (HEVC) test model. Alternatively, video codecs operate according to other proprietary or industry standards, such as standards including ITU-TH.261, ISO / IECMPEG-1Visual, ITU-TH.262 or ISO / IECMPEG-2Visual, ITU-TH.263, ISO / IECMPEG-4Visual, ITU-TH.264 (or ISO / IECMPEG-4AVC), or the standards may also include hierarchical video codecs and multi-view video codec extensions. It should be understood that the technology of the present application is not limited to any particular codec standard or technology.
[0052] Generally, encoding and decoding operations are performed in units of coding units (CU). Specifically, during the encoding process, the image is divided into multiple CUs, and then the pixel data in these CUs is encoded to obtain the coding information of each CU. During the decoding process, the image is divided into multiple CUs, and then each CU is reconstructed according to the coding information corresponding to each CU to obtain a reconstructed block of each CU. Alternatively, the image can also be divided into a grid of coding tree blocks. In some examples, the coding tree block is also referred to as a "tree block", "largest coding unit" (LCU) or "coding tree unit". Optionally, the coding tree block can be further divided into multiple CUs.
[0053] See also Figure 2A , Figure 2A A schematic block diagram of a video encoding and decoding system 10 used in the present application is given as an example. Figure 2A As shown, the system 10 includes a source device 12 and a destination device 14, wherein the source device 12 generates encoded video data, and therefore the source device 12 is also called a video encoding device. The destination device 14 decodes the encoded video data generated by the source device 12, and therefore the destination device 14 is also called a video decoding device.
[0054] The source device 12 and the destination device 14 include one or more processors and a memory coupled to the one or more processors. The memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory, or any other medium that can be used to store desired program codes in the form of instructions or data structures accessed by a computer.
[0055] The source device 12 and the destination device 14 include various devices, such as desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, televisions, cameras, display devices, digital media players, video game consoles, vehicle-mounted computers, wireless communication devices, artificial intelligence devices, virtual reality / mixed reality / augmented reality devices, autonomous driving systems or other devices. The embodiments of the present application do not limit the structures and specific forms of the above-mentioned devices.
[0056] like Figure 2AAs shown, the source device 12 and the destination device 14 are connected via a link 13, and the destination device 14 receives the encoded video data from the source device 12 via the link 13. Among them, the link 13 includes one or more media or devices. In a possible implementation, the link 13 includes one or more communication media that enable the source device 12 to transmit the encoded video data directly to the destination device 14 in real time. In one example, the source device 12 modulates the video data according to a communication standard (e.g., a wireless communication protocol) and transmits the modulated video data to the destination device 14. The one or more communication media include wireless or wired communication media, such as a radio frequency (RF) spectrum or at least one physical transmission line. The one or more communication media may form part of a packet-based network, which may be a local area network, a wide area network, or a global network (e.g., the Internet), etc. The one or more communication media include routers, switches, base stations, or other devices that facilitate communication from the source device 12 to the destination device 14.
[0057] The source device 12 includes an image source 16, an image preprocessor 18, an encoder 20, and a communication interface 22. In a specific implementation, the encoder 20, the image source 16, the image preprocessor 18, and the communication interface 22 may be hardware components in the source device 12, or may be software programs in the source device 12.
[0058] A more specific description is as follows:
[0059] The image source 16 may include any type of image capture device for capturing real-world images or comments, where the comments refer to encoding screen content, some text on the screen. The image capture device is used to acquire and / or provide real-world images, computer animation images, such as screen content, virtual reality (VR) images, augmented reality (AR) images, etc. The image source 16 may be a camera for capturing images or a memory for storing images. The image source 16 may also include any type of (internal or external) interface for storing previously captured or generated images and / or acquiring or receiving images.
[0060] When the image source 16 is a camera, the image source 16 may be a local or integrated camera integrated in the source device; when the image source 16 is a memory, the image source 16 may be a local or integrated memory integrated in the source device. When the image source 16 includes an interface, the interface may be an external interface for receiving images from an external video source, the external video source is an external image capture device such as a camera, an external memory, or an external image generating device, and the external image generating device is an external computer graphics processor, a computer, or a server. The interface may be any type of interface according to any proprietary or standardized interface protocol, such as a wired or wireless interface, an optical interface.
[0061] The image stored in the image source 16 can be regarded as a two-dimensional array or matrix of picture elements. The pixels in the array can also be called sampling points. The number of sampling points in the array or image in the horizontal and vertical directions (or axes) defines the size and / or resolution of the image. In order to represent color, three color components are usually used, that is, the image can be represented as including three sampling arrays. For example, in the RBG format or color space, the image includes corresponding red (R), green (G) and blue (B) sampling arrays. However, in video encoding, each pixel is usually represented in a brightness / chrominance format or color space, such as for an image in the YUV format, including a brightness component indicated by Y (sometimes also indicated by L) and two chrominance components indicated by U and V. The brightness (luma) component Y represents the brightness or grayscale level intensity, for example, the two are the same in a grayscale image; and the two chrominance (chroma) components U and V represent the chrominance or color information components. Accordingly, the image in the YUV format includes a brightness sampling array of brightness sampling values (Y) and two chrominance sampling arrays of chrominance values (U and V). An image in RGB format can be converted or transformed into YUV format, and vice versa, which process is also called color conversion or transformation. If the image is black and white, the image can only include a brightness sampling array. In the embodiment of the present application, the image transmitted from the image source 16 to the image preprocessor 18 can also be referred to as raw image data 17.
[0062] The image preprocessor 18 is used to receive the raw image data 17 and perform preprocessing on the raw image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by the image preprocessor 18 may include refurbishment, color format conversion (e.g., from RGB format to YUV format), color grading, or denoising.
[0063] The encoder 20 or video encoder 20 is used to receive the pre-processed image data 19, and process the pre-processed image data 19 using a prediction mode, thereby providing encoded image data 21 (or video code stream). In some embodiments, the encoder 20 can be used to execute the embodiments of the various video encoding methods described below to implement the image generation method described in this application.
[0064] The communication interface 22 may be used to receive the encoded image data 21 and transmit the encoded image data 21 to the destination device 14 via the link 13. The communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format, such as a data packet, for transmission over the link 13.
[0065] The destination device 14 includes a communication interface 28, a decoder 30, an image post-processor 32 and a display device 34. The following describes each component or device included in the destination device 14 one by one, as follows:
[0066] The communication interface 28 is used to receive the encoded image data 21 from the source device 12. In addition, the communication interface 28 is also used to receive the encoded image data 21 via the link 13 between the source device 12 and the destination device 14, where the link 13 is a direct wired or wireless connection, any type of network such as a wired or wireless network or any combination thereof, or any type of private network and public network, or any combination thereof. The communication interface 28 can also be used to decapsulate the data packets transmitted by the communication interface 22 to obtain the encoded image data 21.
[0067] It should be noted that both the communication interface 28 and the communication interface 22 can be unidirectional communication interfaces or bidirectional communication interfaces, and can be used to send and receive messages, and / or to establish a communication link and transmit image data such as encoded image data transmission through the link.
[0068] The decoder 30 (or video decoder 30) is used to receive the encoded image data 21 and provide decoded image data 31 or decoded image 31. In some embodiments, the decoder 30 can be used to execute the embodiments of various video decoding methods described below to implement the image generation method described in this application.
[0069] The image post-processor 32 is used to perform post-processing on the decoded image data 31 to obtain post-processed image data 33. The post-processing performed by the image post-processor 32 may include: color format conversion (for example, from YUV format to RGB format), color adjustment, repair or resampling, or any other processing, and can also be used to transmit the post-processed image data 33 to a display device 34.
[0070] A display device 34 is provided for receiving the post-processed image data 33 for displaying the image to a user or viewer. The display device 34 includes any type of display for presenting the reconstructed image, such as an integrated or external display or monitor. Further, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.
[0071] It should be understood that Figure 2A The source device 12 and the destination device 14 shown may be separate devices or may be integrated into the same device, that is, the integrated device includes the functionality of both the source device 12 and the destination device 14. In a possible implementation, the source device 12 or the corresponding functionality and the destination device 14 or the corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.
[0072] In addition, based on the above description, it can be seen that the functionality or Figure 2A The existence and (accurate) division of the functionality of the source device 12 and / or the destination device 14 shown may vary depending on the actual device and application. The source device 12 and the destination device 14 may include any of a variety of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a camera, a vehicle-mounted device, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content distribution server), a broadcast receiver device, a broadcast transmitter device, etc. The embodiment of the present application does not limit the specific structure and implementation form of the source device 12 and the destination device 14.
[0073] The encoder 20 and the decoder 30 may be any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), discrete logic, hardware, or any combination thereof. If the technology is partially implemented in software, the device may store the instructions of the software in a suitable computer-readable storage medium, and may use one or more processors to execute the computer program instructions to perform the image generation method described in the present application.
[0074] In one example, Figure 2A The video encoding and decoding system 10 shown is only an example, and the technical solution of the embodiment of the present application can be applied to video encoding settings that do not necessarily include any data communication between encoding and decoding devices, such as video encoding or video decoding. In other examples, data can be retrieved from a local memory, streamed over a network, etc. A video encoding device can encode data and store the data in a memory, and / or a video decoding device can retrieve data from a memory and decode the data.
[0075] See also Figure 2B , is a schematic diagram of a video decoding system 40 including an encoder 20 and a decoder 30 according to an exemplary embodiment. The video decoding system 40 can implement various method steps of the embodiments of the present application. In the various embodiments, the video decoding system 40 can include an imaging device 41, an encoder 20, a decoder 30 (and / or a video encoder / decoder implemented by a processing unit 46), an antenna 42, a processor 43, a memory 44, and a display device 45.
[0076] like Figure 2B As shown, the imaging device 41, antenna 42, processing unit 46, encoder 20, decoder 30, processor 43, memory 44 and display device 45 can communicate with each other. And the processing unit 46 may only include encoder 20 or only include decoder 30.
[0077] In one example, antenna 42 is used to transmit or receive a video code stream, or a coded bit stream of video data. In addition, display device 45 can also be used to present video data. In one implementation, processing unit 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general processor, etc. Video decoding system 40 can also include processor 43, which can similarly include ASIC logic, a graphics processor, a general processor, etc. Further, processing unit 46 can be implemented by hardware, such as video encoding dedicated hardware, etc., and processor 43 can be implemented by general software, operating system, etc.
[0078] The memory 44 may be any type of memory, such as a volatile memory, for example, a static random access memory (SRAM), a dynamic random access memory (DRAM), etc. or a non-volatile memory (e.g., flash memory), etc. The processing unit 46 may access the memory 44, for example, to implement an image buffer. In addition, the processing unit 46 may also include a memory, for example, a cache, etc.
[0079] In one implementation, the encoder 20 implemented by the logic circuit includes an image buffer and a graphics processing unit, the image buffer can be implemented by the processing unit 46 or the memory 44; the graphics processing unit can be implemented by the processing unit 46. In addition, one possible scenario is to couple the graphics processing unit to the image buffer. In one example, the graphics processing unit includes the encoder 20 implemented by the processing unit 46.
[0080] The decoder 30 may be implemented in a similar manner by means of the processing unit 46. In one example, the decoder 30 includes an image buffer and a graphics processing unit. The graphics processing unit may be coupled to the image buffer. In one example, the graphics processing unit includes the decoder 30 implemented by the processing unit 46.
[0081] Antenna 42 is used to receive a video code stream or a coded bit stream of video data. Specifically, the coded bit stream includes data related to the coded video frame, indicators, index values, mode selection data, etc., such as data related to the coded segmentation, such as transform coefficients or quantized transform coefficients, optional indicators, and / or data defining the coded segmentation. In addition, the video decoding system 40 may also include a decoder 30 coupled to the antenna 42 and used to decode the coded bit stream. The display device 45 is used to display the image frame.
[0082] It should be understood that in the functional description of the encoder 20 in the embodiment of the present application, the decoder 30 can be used to perform the opposite function of the encoder 20. The decoder 30 can be used to receive and decode the relevant video data. It should be noted that the decoding method described in the present application is mainly used for the decoding process, which exists in both the encoder 20 and the decoder 30.
[0083] See also Figure 3A , is a flow chart of a video encoding and decoding method provided in an embodiment of the present application, which can be applied to the aforementioned Figure 2A and 2B Specifically, the method can be summarized into the following five steps, namely: video input 110, video encoding 120, video code stream transmission 130, video decoding 140 and video output 150.
[0084] Among them, in step "input video 110", the lossless video or image captured by the acquisition device, such as the camera, is input to the encoder; in step "video encoding 120", the acquired video or image is compressed and encoded through the H.264 or H.265 codec to generate an encoded video stream; then in step "video stream transmission 130", the video stream is uploaded to the cloud server, and the user downloads the video stream from the cloud server. Step "video decoding 140" includes the process of the terminal device decoding the video stream downloaded from the cloud through the decoder, and finally outputting and displaying the decoded video image in step "output video 150".
[0085] Furthermore, if Figure 3B As shown, in Figure 3A The encoding and decoding process shown also includes steps "cross-GOP motion vector calculation 1201" and "video quality improvement 1401". The cross-GOP motion vector calculation refers to the motion vector calculation between two adjacent GOPs. Step 1201 cross-GOP motion vector calculation mainly acts on the video encoding 120 process; step 1401 video quality improvement process mainly acts after video decoding 140 and before outputting video 150. Specifically, the implementation process of these two steps will be described in detail in subsequent embodiments.
[0086] Optionally, the steps of "cross-GOP motion vector calculation 1201" and "video quality improvement 1401" can be implemented through program code or through a corresponding neural network model, such as implementing the above steps 1201 and 1401 through a newly added unit module, or through an existing processing unit (including an encoder and a decoder).
[0087] The following is a brief introduction to the video coding layer (VCL) codec structure principle of the video codec standard H.264. Figure 4A As shown, it is a schematic diagram of the structure of an encoder 20. Specifically, the execution function of the encoder 20 includes two paths, one is a forward path, and the other is a reconstruction path. For the forward path, the input frame is processed by the encoder for intra-frame (Intra) or inter-frame (Inter) encoding in units of macroblocks or sub-blocks. If it is intra-frame encoding, its prediction value is obtained by predicting the pixels in the current image; if it is inter-frame encoding, its prediction value is obtained by motion compensation of the reference image, and the reference image can be selected from the frames that are encoded, decoded, reconstructed and filtered in the past or future (in the display order).
[0088] Among them, after the prediction value is subtracted from the current block, a residual block is generated. The residual block is transformed and quantized to produce a set of quantized transform coefficients, which are then entropy encoded and together with some side information required for decoding (such as prediction mode, quantization parameter, motion vector information, etc.) to form a compressed video code stream, which is finally handed over to the network abstraction layer (NAL) for transmission and storage.
[0089] In the reconstruction path, in order to provide a reference image for prediction, the encoder needs to have the function of reconstructing the image. Therefore, the transform coefficient image must be dequantized, and the residual obtained after the inverse transformation is added to the prediction value to obtain the unfiltered image (reconstructed block). Finally, the reconstructed block is filtered through a filter to obtain a reconstructed reference image, that is, a reconstructed frame.
[0090] See also Figure 4B , which is a schematic diagram of the structure of a decoder 30. The decoder 30 receives a video bitstream from a NAL unit, obtains a transform coefficient after entropy decoding of the video bitstream, and then obtains a residual after inverse quantization and inverse transformation, uses the header information obtained by decoding the bitstream, and the prediction value generated by the decoder, adds the prediction value to the residual, and finally obtains a decoded image after filtering.
[0091] When encoding, video frames are divided into two encoding modes: intra and inter. First, each video frame is divided into blocks (or macroblocks) so that the frame processing can be performed at the block level. Then, the spatial redundancy in the video frame is used to predict the pixels of the block through adjacent pixels, and the video block is transformed between different domains. From the time domain to another domain, the transformation coefficients are concentrated on a few points; or the temporal redundancy between consecutive video frames is used to search for a video block that matches the current block in the reference frame through motion vector estimation, and then the difference between the two blocks is calculated and the difference is changed. Finally, the transformation coefficients and motion vectors are entropy coded.
[0092] This embodiment is mainly applied to encoding and decoding during inter-frame prediction. Inter-frame prediction and various inter-frame prediction technologies are introduced below.
[0093] Inter-frame prediction and coding mainly utilize the temporal correlation of video signals to remove the temporal redundant information of video signals through motion estimation and compensation, thereby achieving the purpose of compressing video data. Since the temporal correlation of video signals is much greater than its spatial correlation, the encoding bit rate can be further reduced by using inter-frame prediction and coding.
[0094] In inter-frame coding, a frame group or group of pictures (GOP) mainly includes two frame modes: I frame (Intra Frame) and P frame (Predict Frame), and motion estimation uses positive pixel motion vectors. Among them, I frame is also called key frame, which only performs predictive coding of macroblocks within the frame and can retain more information. P frame is also called predicted frame. Furthermore, the P frame is a forward predicted frame, which can be understood as P frame needs to use the previous frame to perform motion estimation of macroblocks between frames and calculate motion vectors.
[0095] Various inter-frame coding techniques are introduced below.
[0096] (1) Block division
[0097] When performing motion estimation, the size of the block used has a great influence on the effect of motion estimation, so using smaller blocks can make the motion estimation result more accurate, thereby producing smaller residuals and achieving the effect of reducing the bit rate. H.264 uses different macroblock segmentation methods when performing motion estimation. For example, a 16×16 macroblock can be divided into one 16×16, two 16×8 or 8×16, or four 8×8 block sizes. For an 8×8 block, it can be divided into one 8×8, two 8×4 or 4×8, or four 4×4 blocks. Among them, the chrominance component of the macroblock uses the same segmentation mode as the luminance block, but the size is halved in the horizontal and vertical directions.
[0098] Each block has a motion vector MV, each MV is encoded and transmitted, and the segmentation selection is also encoded and compressed into the bit stream. For large segmentation sizes, the selection of MV and the segmentation type only require fewer bits, but the energy of the motion compensation residual in the detailed area is high. For small segmentation sizes, the energy of the motion compensation residual is low, but more bits are required to represent the MV and segmentation selection, so overall, large segmentation sizes are suitable for flat areas, and small segmentation sizes are suitable for detailed areas.
[0099] (2) Motion Vector MV Prediction
[0100] Since H.264 supports multiple macroblock and sub-macroblock segmentation, if the graphics have more details, the size of the divided blocks is small. If the MV of each block is encoded independently, a considerable number of bits are required. The MV of a block has a strong correlation with the adjacent blocks, so the MV can be predicted by the adjacent encoded segmented blocks, that is, the difference between the motion vector prediction value (MV Predict, MVP) of the adjacent encoded segmented blocks and the MV of the current macroblock can be obtained, and the difference can be encoded and transmitted.
[0101] The technical solution of the embodiment of the present application is described in detail below.
[0102] The technical solution provided in the embodiments of the present application can be applied to scenarios of real-time video services, such as displaying videos on mobile terminals and large-screen devices, and improving video quality by combining video encoding and decoding technology to achieve high resolution of video frames and low negative effects.
[0103] Embodiment 1
[0104] The technical solution of the embodiment of the present application is based on the motion vectors of each block (or macroblock) obtained by the decoder, and directly performs pixel alignment on multiple consecutive video frames according to each motion vector to obtain an aligned frame, and obtains the residual between the video frames to obtain a residual frame, and sends the aligned frame and the residual frame to a neural network to perform super-resolution on the image to be processed to obtain a super-resolution video frame.
[0105] Specifically, Figure 5 As shown, this embodiment provides an image processing method, which can be applied to the decoder 30, and the method includes:
[0106] Step 101: Acquire a video code stream, wherein the video code stream includes a first frame image, a second frame image, and a third frame image that are adjacent in time sequence.
[0107] The video code stream is a code stream or bit stream output after the input video is encoded and compressed by an encoder, and the video code stream includes two or more frames of images. The temporally adjacent refers to frames that are continuously shot (or generated) in time.
[0108] In one example, during the encoding stage, the encoder divides the video code stream into at least one group of pictures (GOP), each GOP including an I frame and a number of subsequent P frames. For example, it includes the first GOP and the second GOP, wherein the first GOP includes an I frame and 3 P frames, namely {I1, P1, P2, P3}; the second GOP also includes an I frame and 3 P frames, namely {I2, P4, P5, P6}, and then the frames in the video code stream include: I1, P1, P2, P3, I2, P4, P5, P6.
[0109] Step 102: Decode the video code stream to obtain a first aligned frame, a second aligned frame, and at least one residual between the first frame image, the second frame image, and the third frame image.
[0110] The decoder receives the video code stream transmitted by the encoder, and decodes the video code stream to obtain video information, wherein the video information includes: all frame images constituting the video code stream, motion vectors of two adjacent frames, and residuals.
[0111] In this embodiment, taking super-resolution of the second frame image as an example, the first motion vector, the second motion vector, the first aligned frame and the second aligned frame, and at least one residual are obtained after parsing the video code stream. Specifically, the first aligned frame is generated after the first frame image is moved to the second frame image by pixel blocks according to the first motion vector, and the second aligned frame is generated after the third frame image is moved to the second frame image by pixel blocks according to the second motion vector.
[0112] For example, in the first GOP of a video code stream, the first frame image is I1, the second frame image is P1, and the third frame image is P2. The second frame image P1 is used as the target frame (P1 is used as the super-resolved frame). The process of generating the first aligned frame and the second aligned frame includes: the same position of the first frame image I1 and the second frame image P1 is pre-divided into multiple macroblocks, for example, 9 macroblocks of 3×3. Each macroblock includes multiple pixel points, each pixel point corresponds to a pixel value, and the relative displacement between the pixel value of the first macroblock of the I1 frame and the best matching macroblock in the target frame P1 is the motion vector of the first macroblock of the I1 frame, which can be expressed as MV 11 For the 9 macroblocks of the I1 frame, matching macroblocks are searched in the P1 frame respectively, and then the 9 motion vectors of the I1 frame can be obtained. These 9 motion vectors are collectively referred to as "first motion vectors" and can be represented by a matrix MV1. For example,
[0113] The first motion vector,
[0114] One possible situation is that if a macroblock in the I1 frame has no matching macroblock in the target frame P1, the motion vector of the macroblock is (0,0), indicating that the macroblock does not move, and the matching position is the original position of the macroblock. The image formed by moving each macroblock in the first frame image I1 according to the first motion vector MV1 is the first alignment frame. For the specific process, please refer to the relevant description of "image motion compensation alignment" in the standard, which will not be described in detail in this embodiment.
[0115] Similarly, the third frame image P2 generates a second aligned frame after pixel blocks are moved toward the target frame P1 according to the second motion vector MV2. The specific implementation process is similar to the process of generating the first aligned frame, which will not be repeated here.
[0116] The residual is the pixel difference between each macroblock of the previous frame image after motion compensation is performed on the next frame image according to the motion vector. Specifically, the residual (also called the first residual) between the first frame image I1 and the second frame image P1 can be defined as the pixel difference between each macroblock of the first aligned frame and the second frame image P1 after the first frame image I1 is motion compensated and aligned to the second frame image P1 according to the first motion vector MV1. If the first aligned frame includes 9 macroblocks of 3×3, there will be 9 pixel differences between the first aligned frame and the 9 macroblocks of the second frame image P1, and each of the pixel differences can be expressed as For example, the first frame image I1 is generated at time t1, and the second frame image P1 is generated at time t2. represents the residual between the first macroblock of the first aligned frame and the first macroblock of the P1 frame. It can be understood that the first residual can be represented by a matrix, for example,
[0117] The first residual is, Wherein i is a positive integer, and 1≤i≤9.
[0118] Similarly, the method further includes obtaining a second residual, which is a pixel difference of each macroblock between the second aligned frame and the second frame image P2, wherein the second aligned frame is generated after the second frame image P1 is aligned to the third frame image P2 by motion compensation according to the second motion vector MV2. For example, when the third frame image P2 is generated at time t3, the second residual is expressed as
[0119] In addition, the process also includes obtaining the residual of the first frame image P1. Since the first frame image P1 is the first frame of the entire video code stream, there is no frame image before it, so the residual of the first frame image P1 is 0.
[0120] Therefore, obtaining at least one residual between the first frame image I1, the second frame image P1 and the third frame image P2 through step 102 includes: obtaining the first residual and the second residual
[0121] Step 103: Generate at least one residual frame according to the at least one residual.
[0122] Specifically, step 103 includes:
[0123] According to the first residual Generate a first residual frame, and, according to the first residual and the second residual A second residual frame is generated.
[0124] The first residual and the first aligned frame satisfy the following relationship:
[0125]
[0126] in, represents the second frame image P1, represents the first alignment frame, represents the first residual, i represents a macroblock of the first frame image, iT i (t2) Indicates that the macroblock i is based on its corresponding motion vector T i (t2) The macroblock after the move, t1 represents the generation time of the first frame image I1, and t2 represents the generation time of the second frame image P1.
[0127] The first residual The pixel values (RGB values) of each macroblock represented are restored to obtain the first residual frame.
[0128] Similarly, the second residual and the second aligned frame satisfy the following relationship:
[0129]
[0130] in, represents the third frame image P2, represents the second alignment frame, represents the second residual, i represents a macroblock of the second frame image, iT i (t3) Indicates that the macroblock i is based on its corresponding motion vector T i (t3) The macroblock after the move, t3 represents the generation time of the third frame image P2.
[0131] Substituting the above formula 2 into the formula 1, the corresponding relationship between the third frame image P2 and the first frame image I1 is obtained as follows:
[0132]
[0133] in, is the cumulative residual from the first frame image I1 to the third frame image P2, that is, the cumulative residual is the first residual With the second residual The sum.
[0134] According to the first residual and the second residual Generating a second residual frame comprises: The pixel values (RGB values) of each macroblock represented are restored to obtain the second residual frame.
[0135] In this embodiment, the corresponding relationship between the first frame image and the third frame image is obtained through the above formula 1 and formula 2, that is, two interval frames are realized, and the third frame image P2 is expressed by the key frame I1.
[0136] It should be understood that according to the corresponding relationship of the above formula 1, if the fourth frame image P3 or the fifth frame image P4, etc. is also included, the corresponding relationship between the frame and the key frame I1 can also be obtained through the cumulative relationship, that is, the expression of the fourth frame image P3 or the fifth frame image P4 and the key frame I1, as well as the cumulative residual, can be obtained through the deformation of the above formulas 1 and 2.
[0137] Step 104: super-resolution the second frame image according to the at least one residual frame, the first aligned frame and the second aligned frame to obtain a super-resolution second frame image.
[0138] Specifically, Figure 6 As shown, step 104 includes:
[0139] Step 1041: Input the at least one residual frame into a neural network for feature extraction to obtain at least one first feature image.
[0140] The neural network includes multiple layers with different functions. The functions of each layer in the multiple layers include but are not limited to convolution, pooling, activation and other operations. After processing in these layers, the input image data can be feature extracted, analyzed and integrated, and finally the super-resolved image can be output.
[0141] In one example, the first residual frame and the second residual frame are input into the neural network, wherein the first residual frame is input into the neural network through the first residual frame. The second residual frame is generated by the first residual and the second residual Generate (i.e., accumulate residuals ).
[0142] Optionally, in one implementation, the neural network may include a feature extraction network and a super-resolution network, and the super-resolution network includes a first super-resolution network and a second super-resolution network. Figure 6As shown, step 1041 specifically includes: inputting the first residual frame and the second residual frame into the feature extraction network to obtain at least one first feature map.
[0143] Step 1042: Input the first aligned frame, the second aligned frame and the second frame image into the neural network for feature extraction to obtain at least one second feature image.
[0144] Optionally, one implementation method is to input the first aligned frame, the second aligned frame and the second frame image into the feature extraction network, and then obtain at least one second feature image through feature extraction.
[0145] Step 1043: Input the at least one first feature image into the first super-resolution network for processing to generate high-frequency information.
[0146] Step 1044: Input the at least one second feature map into the second super-resolution network for processing to generate a brightness channel.
[0147] The first super-resolution network is different from the second super-resolution network, for example, the two super-resolution networks have different network structures, complexity, and weight parameters.
[0148] Step 1045: Merge the high-frequency information with the brightness channel to generate the second frame image after super-resolution.
[0149] In this embodiment, at least one residual frame and at least one aligned frame are input into the same neural network for super-resolution processing, thereby improving parameter utilization, reducing the amount of model parameters, and improving super-resolution processing efficiency.
[0150] An embodiment of the present application provides a joint video encoding and decoding solution, which uses motion vector information and residual information in the video encoding and decoding process to align and accumulate residuals of video frames, and uses aligned frames and residual frames as inputs of a neural network. Since the accumulated frames output high-frequency information after passing through the neural network, the high-frequency information is fitted back into the brightness channel to enhance edge details, thereby compensating for the problem of low image resolution caused by the lack of high-frequency information in low-resolution video frames during the super-resolution process.
[0151] In addition, in a possible implementation manner, before sending the at least one residual frame and the at least one aligned frame to the neural network for super-resolution processing in the above step 104, the method further includes:
[0152] The at least one residual frame and aligned frame are divided into regions of interest, wherein the regions of interest are macroblocks in the residual frame or aligned frame that need to be super-resolution processed; further, the regions of interest can be determined by the size between the pixel value of the macroblock divided in the residual frame or aligned frame and a preset value.
[0153] For example, taking the first residual frame as an example, the macroblocks of the area of interest in the first residual frame are determined, and the macroblocks of the area of interest are macroblocks in which the sum of all pixel values contained in the current macroblock exceeds the first preset value; that is, the first residual frame is divided into multiple macroblocks, such as 9 macroblocks of 3×3, and the pixel sum of all pixels in each macroblock is calculated, and then the pixel sum of each macroblock is compared with the size of the first preset value, and all macroblocks with a pixel sum greater than or equal to the first preset value are screened out as the area of interest, and super-resolution processing is performed on this part of the area.
[0154] Correspondingly, pixels and macroblocks smaller than the first preset value are set as non-interest areas. The macroblocks in the non-interest areas are low-texture areas, so the super-resolution results of the previous frame processing can be directly used for this part of the area without the need for super-resolution processing, thereby saving the amount of calculation and improving the efficiency of super-resolution processing.
[0155] In addition, in this embodiment, the method further includes: dividing the regions of interest for all residual frames other than the first residual frame (for example, the second residual frame).
[0156] Specifically, a possible implementation method is, in the same way as the aforementioned method of dividing the first residual frame into regions of interest, comparing the pixel value of each macroblock with the size of the first preset value to determine whether each macroblock is a region of interest. The detailed determination process is similar to the aforementioned process of determining the region of interest in the first residual frame, and will not be repeated here.
[0157] Another possible implementation is to use the macroblocks at the same position as the region of interest determined in the first residual frame as the region of interest of the second residual frame. For example, when the macroblocks of the region of interest of the first residual frame are determined to be macroblocks 4 to 8 (a total of 9 macroblocks), correspondingly, the macroblocks of the region of interest in the second residual frame are also determined to be macroblocks 4 to 8.
[0158] In another possible implementation, the region of interest of the first residual frame may also be determined by using the same region as the aligned frame, that is, firstly dividing at least one aligned frame into regions of interest and regions of no interest, and then using the region of interest determined in the aligned frame as the region of interest of the first residual frame.
[0159] Specifically, dividing the at least one aligned frame into regions of interest includes: a possible division method is to compare the pixel value of each macroblock of the first aligned frame with a second preset value, and set all macroblocks greater than or equal to the second preset value as regions of interest, and set macroblocks less than the second preset value as regions of no interest. Similarly, determining the region of interest of the second aligned frame can be the same as the region of interest of the first aligned frame.
[0160] After determining the regions of interest of all residual frames and all aligned frames, all macroblocks of the regions of interest are input into the neural network for feature extraction. For the remaining macroblocks of non-regions of interest, the super-resolution result of the previous frame image can be used, combined with the super-resolution result obtained by the super-resolution processing of the region of interest this time, and finally fused into a super-resolution image.
[0161] In the present embodiment, each frame of the image is divided into an area of interest and an area of non-interest, that is, the residual accumulated texture details and the motion characteristics of the previous and next frames of each frame of the image are analyzed, and only the macroblocks in the area of interest are super-resolution processed, while the super-resolution processing result of the previous frame can be directly used for the area of non-interest, thereby avoiding super-resolution processing of the entire image, reducing the amount of calculation, reducing power consumption, delay and memory overhead, and improving the super-resolution efficiency of a single frame image, thereby achieving the beneficial effect of obtaining a super-resolution image in real time in a short time.
[0162] It should be noted that the above embodiment cites the first three frames of a GOP, including but not limited to these three frames, and may also be other frames, or located in different GOPs. For example, one possible scenario is that the first frame is the last frame in the first GOP, and the second and third frames are the first two frames in the second GOP. Another possible scenario is that the first and second frames are the last two frames in the first GOP, and the third frame is the first frame in the second GOP. The first GOP and the second GOP are adjacent in time sequence. Furthermore, the target frame to be super-resolved may be any one of the first, second and third frames, which is not limited in this embodiment.
[0163] Embodiment 2
[0164] The following is an example in which the first frame image is the last frame of the first GOP, and the second frame image and the third frame image are the first two frames in the second GOP, to introduce the image processing method provided by the present application. The method is applied to the decoder 30, or the video decoding system 40, or the destination device 14 described in the above embodiment.
[0165] Furthermore, if Fig. 7A As shown, the method includes:
[0166] Step 701: Obtain a video source, wherein the video source may be from a short video or video call APP platform, and the short video or video stream may be downloaded and obtained from the cloud.
[0167] Step 702: The encoder encodes the video source to generate a video code stream.
[0168] In one example, during the encoding stage, the encoder divides the video bitstream into two GOPs, namely the first GOP and the second GOP, wherein the first GOP includes one I frame and three P frames, namely {I1, P1, P2, P3}; the second GOP also includes one I frame and three P frames, namely {I2, P4, P5, P6}, so the frames in the video bitstream include: I1, P1, P2, P3, I2, P4, P5, P6. Any one of these frames is selected as the target frame and is super-resolved.
[0169] Wherein, step 702 in the video encoding stage further includes: 7021: the encoder obtains a cross-GOP motion vector, and performs inter-frame prediction based on the cross-GOP motion vector.
[0170] The "cross-GOP motion vector" refers to the motion vector between the last frame image of the previous GOP and the first frame image of the next GOP. For example, the cross-GOP motion vector between the first GOP and the second GOP refers to the motion vector between the P3 frame of the first GOP and the I2 frame of the second GOP. The cross-GOP motion vector is used to align the target frame to generate the aligned frame, thereby ensuring the accuracy of the "alignment" between the cross-GOP frames.
[0171] In one example, when the target frame is an I2 frame (the first frame of the second GOP), the alignment processing refers to moving pixel blocks of the P3 frame of the first GOP toward the I2 frame according to the cross-GOP motion vector, and moving pixel blocks of the P4 frame of the second GOP toward the I2 frame according to the motion vector, thereby generating two aligned frames.
[0172] Among them, when using motion vectors for motion estimation, the motion vectors of pixel or sub-pixel inter-frame pixel blocks can be used for estimation according to the accuracy requirements, and motion compensation information, such as residuals, can be obtained based on the motion vector calculations. Specifically, the corresponding relationship between the motion vector, the aligned frame, and the residual is described in the relevant description of each formula in the first embodiment, which will not be repeated in this embodiment.
[0173] During the encoding process, this embodiment adds motion vectors between temporally adjacent GOPs (i.e., cross-GOP motion vectors) to perform motion estimation and generate a video bitstream, providing continuous motion vectors for the frame alignment operation in the subsequent decoding stage, thereby ensuring the continuity of inter-frame prediction.
[0174] Step 703: The decoder obtains the video stream. Specifically, one implementation is that the encoder transmits the video stream to the cloud, and the decoder downloads and obtains the video stream from the cloud. Alternatively, another implementation is that the encoder directly transmits the encoded video stream to the decoder.
[0175] Step 704: The decoder decrypts the video code stream to obtain at least one video frame, motion vectors and residuals between adjacent frames, etc. The motion vectors include motion vectors between two adjacent frames in the same GOP, and also include cross-GOP motion vectors. In addition, it also includes residuals between all two adjacent video frames.
[0176] Step 705: The decoder pre-processes the super-resolved image according to the video frame, motion vector, residual and other information to generate at least one alignment frame and at least one residual frame, specifically including:
[0177] Step 7051: Align the previous frame and the next frame of the target frame to the target frame according to their respective motion vectors to generate a first aligned frame and a second aligned frame.
[0178] In the process of generating the alignment frame, in order to improve the accuracy of the super-resolution image, the accuracy of the motion vector can be divided into pixel and sub-pixel. For the motion vector with pixel accuracy, the alignment frame can be generated according to the normal alignment operation.
[0179] For sub-pixel precision motion vectors, the video frame needs to be enlarged and aligned first, and then the aligned image is restored to the original size. Specifically, it includes: extracting multiple adjacent continuous video frames, and selecting an image to be super-resolved as the target frame, and the remaining frames are front and back auxiliary frames. The front and back auxiliary frames use formula 4 to move the pixel points to the specified position according to the motion vector between the target frame and the image.
[0180] (x′,y′)=(x+mv x ,y+mv y ), (Formula 4)
[0181] Among them, (x′, y′) represents the position of the pixel after it is moved; (x, y) represents the original position of the pixel; (mv x ,mv y ) represents the motion vector.
[0182] In addition, when aligning sub-pixel precision motion vectors, first enlarge and align the continuous video frames, and then restore the aligned frames to the original size. Because when the codec calculates the sub-pixel precision motion vector, the current target frame is first enlarged by the multiple of the corresponding pixel precision, and then the motion vector is calculated with the previous auxiliary frame. In this case, the sub-pixel precision motion vector obtained is a non-integer (i.e., a decimal). In the alignment process, firstly, an image cache corresponding to the magnification factor is established for multiple consecutive video frames, where the image to be super-resolved is the target frame and the remaining frames are auxiliary frames. Then, the auxiliary frame uses Formula 5 to move the pixel point to the specified position according to the motion vector and super-resolution magnification factor between the auxiliary frame and the current target frame.
[0183] (x′,y′)=(x+mv x ×factor,y+mv y ×factor), (Formula 5)
[0184] Among them, factor represents the super-resolution magnification, (x′, y′) represents the position of the pixel after it is moved; (x, y) represents the original position of the pixel; (mv x ,mv y ) represents the motion vector.
[0185] In this embodiment, the video frame is enlarged and moved so that only The pixels of the image have pixel values, and the rest are 0. In order to reduce the amount of data input to the subsequent super-resolution model, the enlarged image cache is weighted and sampled to the original image frame size to complete the alignment. And when using sub-pixel precision for macroblock alignment, the changed area can be more accurately placed on the pixel points in the macroblock, thereby improving the accuracy of alignment.
[0186] Step 7052: According to the residual between the two adjacent frames, residual accumulation is performed according to key frames to generate at least one residual frame about the target frame.
[0187] The key frame is the first frame of each GOP. In step 7052, the residuals may be accumulated using a backtracking technique to generate at least one residual frame.
[0188] In the backtracking technology, the residuals are accumulated, that is, the corresponding relationship between each auxiliary frame and the key frame (first frame) in each GOP is obtained. It can be understood that each auxiliary frame is expressed by the key frame and the accumulated residuals.
[0189] In one example, the P3 frame, the I2 frame, and the P4 frame in the two GOPs are the first frame, the second frame, and the third frame, and the I2 frame is the target frame to be super-resolved. The process of generating the at least one residual frame includes:
[0190] (1) For the residual frame of the P3 frame, the relationship between the P3 frame and the I1 frame of the first GOP is expressed as follows according to formulas 1 to 3 of the first embodiment:
[0191]
[0192] in, represents the P3 frame, It indicates that the P3 frame is aligned to the I2 frame according to the cross-GOP motion vector, t1 is the generation time of the I1 frame image, t2 is the generation time of the P1 frame image, t3 is the generation time of the P2 frame image, and t4 is the generation time of the P3 frame image. represents the residual between the I1 frame and the P1 frame, represents the residual between the P1 frame and the P2 frame, represents the residual between the P2 frame and the P3 frame, represents the accumulated residual between the I1 frame and the P3 frame, i represents a macroblock of the image, iT i (t2) -T i (t3) -T i (t4) It represents the macroblock after the macroblock i is moved according to its corresponding motion vector.
[0193] According to the accumulated residual between the I1 frame and the P3 frame A first residual frame is generated.
[0194] (2) For the residual frame of the I2 frame, since the I2 frame is the first frame of the second GOP, the residual of the I2 frame is.
[0195] (3) For the residual frame of the P4 frame, the P4 frame is the second frame of the second GOP, so according to the above formula 2, the corresponding relationship between the P4 frame and the I2 frame is obtained as follows:
[0196]
[0197] in, represents the P4 frame, It indicates that the P4 frame is aligned with the I2 frame according to the motion vector and the generated aligned frame, t5 is the generation time of the I2 frame image, t6 is the generation time of the P4 frame image, represents the residual between the I2 frame and the P4 frame, i represents a macroblock of the image, iT i (t6) Indicates that the macroblock i is based on its corresponding motion vector T i (t6) The macroblock after the move.
[0198] According to the accumulated residual between the I2 frame and the P4 frame A second residual frame is generated.
[0199] In this embodiment, the correspondence between the first frame of each GOP and any subsequent frame can be calculated, and no longer depends on the relationship expression between two adjacent frames. Instead, the correspondence between the key frame I1 generated at time t1 and each image generated at t2, t3, and t4 is directly established, thereby breaking the mutual dependence between two adjacent P frames and P frames, avoiding the storage of the correspondence between two consecutive frames, and the method saves storage space. In addition, the generated cumulative residual frame contains relevant redundant information in the time sequence, providing more original information for the subsequent recovery of the details of the moving subject.
[0200] Step 706: Determine whether the image to be generated is the first frame of the video code stream, where the first frame is the first video frame played in the video code stream, that is, the I1 frame of the first GOP.
[0201] If it is not the first frame, execute steps 707 to 709; if it is the first frame, execute step 710.
[0202] Step 707: For non-I1 frames, the at least one residual frame and the aligned frame are divided into a region of interest and a region of no interest.
[0203] Specifically, a possible screening process includes: using motion vectors and residual information to select a super-resolution region of interest. If there is little texture information in the region in the residual accumulation map, the region is a non-interest region, and the super-resolution result of the region in the previous frame can be used directly, thereby reducing the amount of super-resolution calculation for the frame; otherwise, it is an interest region and super-resolution processing is required. Further, the interest region can be determined by formula 6 and formula 7.
[0204]
[0205]
[0206] Among them, I SR represents the set of macroblocks of all regions of interest, K represents the total number of pixels in each macroblock, and I residual Indicates the current image macroblock, i k represents the pixel value of the kth pixel, sign(x) is the sign function, which can be defined by formula 7: I represents the number of macroblocks in the target frame, and U function represents the union of macroblocks. The threshold value (Threshold) is a constant and can be preset. Formula 7 indicates that if the sum of all pixel values in a macroblock is less than the preset threshold value, then sign(x) = 0, and the macroblock is a macroblock of a non-interested area; if sign(x) = 1, then i k The macroblock corresponding to the pixel value is the region of interest and needs to be super-resolution processed. Then, the union of all macroblocks in the region of interest is taken as the multi-texture region that needs to be super-resolution processed.
[0207] Step 7081: The macroblocks in the non-interest area adopt the processing results of the previous frame super-resolution. For the macroblocks in the non-interest area, extract the image information obtained by super-resolution of all macroblocks in the non-interest area in the previous frame, and the image information includes the super-resolution results of the macroblocks at the corresponding position of the previous frame image.
[0208] Step 7082: Send the macroblocks of the region of interest to the neural network for super-resolution processing, wherein the macroblocks of the region of interest are the region of interest in the following frame.
[0209] In one example, the macroblocks of the regions of interest of the first aligned frame and the second aligned frame generated by the above step 7051, and the target frame I2 are input into the neural network, and at the same time, the macroblocks of the regions of interest of the first residual frame and the second residual frame generated by step 7052 are input into the neural network to obtain the brightness channel and high-frequency information of the target frame I2 respectively.
[0210] Step 709: Fusing the brightness channel with the high frequency information to obtain a super-resolved image. Figure 7B shown.
[0211] Wherein, the neural network includes: a feature extraction network and a super-resolution network. Furthermore, the feature extraction network is used to extract features from at least one aligned frame and at least one residual frame input, and obtain a feature map of the aligned frame and a feature map of the residual frame respectively. Wherein, the feature extraction network can be a shared network, that is, shared with the aligned frame and the residual frame for feature extraction, so as to improve parameter utilization and reduce the number of parameters of the neural network. Then, the extracted feature maps are sent to different super-resolution networks respectively, and the feature map obtained by the aligned frame is output as a brightness channel after passing through the first super-resolution network; the feature map obtained by the residual frame is output as high-frequency information after passing through the second super-resolution network, and finally the high-frequency information is pasted back into the brightness channel to enhance the edge details and high-frequency information of the image to be super-resolution, and finally a high-quality super-resolution image is obtained.
[0212] Step 710: In the above step 706, if the image to be super-resolved is the first frame, the region of interest is not divided, and the generated aligned frame and residual frame are directly input into the neural model for super-resolution processing to obtain the super-resolved image. Because the first frame does not have the previous frame and the super-resolution result of the previous frame, it is not necessary to divide the region of interest and the non-region of interest, and the whole image needs to be super-resolved.
[0213] Otherwise, it is necessary to splice the output result with the super-resolution result of the previous frame corresponding to the non-interested region in step 7081 to obtain the complete super-resolution result of the current frame, such as Figure 7BAs shown, the three current frame super-blocks, namely the brim, the nose and the hat tassel, are fused with the previous frame super-block to obtain the super-resolved image.
[0214] In this embodiment, when the frame to be super-resolution is not the first frame of the video code stream, it is divided into an area of interest and an area of non-interest, and only the macroblocks in the area of interest are super-resolution processed, while the macroblocks in the area of non-interest are super-resolutioned using the super-resolution results of the previous frame, that is, the areas where pixels move and where information exists in the accumulated residuals are super-resolutioned, and the modules where pixels do not move and the amount of accumulated residual information is small use the results of the super-resolution of the previous frame or directly interpolate and amplify, thereby avoiding super-resolution processing of all macroblocks of the entire image. This method saves the amount of calculation and calculation time input to the neural network, and improves the super-resolution efficiency.
[0215] In addition, the macroblocks at the same position of the aligned frame complement each other with the same details, and this redundancy provides a larger amount of information. The residual is actually the difference compensation of the pixel matching block, which contains information such as the edge of the moving subject. This embodiment obtains the residual frame by accumulating the residual, and uses the residual to restore the high-frequency edge of the image, so that the subjective effect is better. At the same time, the amount of calculation allocated to different chips can be dynamically adjusted according to the load capacity of the terminal device chip to reduce delay.
[0216] In addition, the methods described in the above embodiments of the present application can also implement corresponding functions through software modules. Figure 2B As shown, the video decoding system 40 includes a processing unit 46, or, as shown in Figure 8 An image processing device is shown, which includes: an acquisition unit 810, a processing unit 820 and a sending unit 830, and may also include other functional modules or units, such as a storage unit.
[0217] In one embodiment, the image processing apparatus may be used to perform the above Figure 5 , Figure 6 and Fig. 7A For example, the acquisition unit 810 is used to acquire a video code stream; the video code stream includes a first frame image, a second frame image, and a third frame image that are adjacent in time sequence; the processing unit 820 is used to decode the video code stream to obtain a first aligned frame, a second aligned frame, and at least one residual between the first frame image, the second frame image, and the third frame image; the processing unit 820 is also used to generate at least one residual frame according to the at least one residual, and perform super-resolution processing on the second frame image according to the at least one residual frame, the first aligned frame, and the second aligned frame to obtain a second frame image after super-resolution.
[0218] The sending unit 830 is used to output the super-resolved second frame image to a display screen for display.
[0219] The first aligned frame is generated after the first frame image is moved toward the second frame image by pixel blocks according to the first motion vector, the second aligned frame is generated after the third frame image is moved toward the second frame image by pixel blocks according to the second motion vector, and the residual is the pixel difference between each macroblock of the previous frame image and the subsequent frame image after motion compensation is performed toward the subsequent frame image by the motion vector.
[0220] Optionally, the first frame image, the second frame image and the third frame image are three frames of images in a first image group;
[0221] Alternatively, the first frame of image is the last frame of image in the first image group, and the second frame of image and the third frame of image are the first two frames of image in the second image group;
[0222] Alternatively, the first frame image and the second frame image are the last two frame images in the first image group, and the third frame image is the first frame image in the second image group.
[0223] The embodiment of the present application does not restrict the three frames of images selected in the video code stream, and does not restrict which frame is selected as the image to be super-resolved.
[0224] In a possible implementation manner of this embodiment, the processing unit 820 is specifically configured to generate a first residual frame, wherein the first residual and the first aligned frame satisfy the following relationship:
[0225]
[0226] in, represents the second frame image, represents the first alignment frame, represents the first residual, i represents a macroblock of the first frame image, iT i (t2) Indicates that the macroblock i is based on its corresponding motion vector T i (t2) For the macroblock after the move, t1 represents the time when the first frame image is generated, and t2 represents the time when the second frame image is generated.
[0227] In another possible implementation of this embodiment, the processing unit 820 is specifically configured to:
[0228] Inputting the at least one residual frame into a neural network for feature extraction to obtain at least one first feature image;
[0229] Inputting the first aligned frame, the second aligned frame and the second frame image into the neural network for feature extraction to obtain at least one second feature image;
[0230] Inputting the at least one first feature image into a first super-resolution network for processing to generate high-frequency information;
[0231] Inputting the at least one second feature map into a second super-resolution network for processing to generate a brightness channel;
[0232] And, the high-frequency information is merged with the brightness channel to generate the second frame image after super-resolution.
[0233] In another possible implementation of the present embodiment, the processing unit 820 is further used to determine the macroblocks of the area of interest in the first residual frame, and determine the areas of interest of the remaining residual frames in the at least one residual frame except the first residual frame based on the macroblocks of the area of interest in the first residual frame; the macroblocks of the area of interest are macroblocks in which the sum of all pixel values contained in the current macroblock exceeds a preset value.
[0234] In addition, the processing unit 820 is specifically used to input the macroblocks of all regions of interest of the at least one residual frame into the neural network for feature extraction, and the at least one residual frame includes the first residual frame and the remaining residual frames.
[0235] In another possible implementation of the present embodiment, the processing unit 820 is specifically used to input the macroblocks of the region of interest in the first aligned frame and the second aligned frame, and the second frame image into the neural network for feature extraction; wherein the region of interest in the first aligned frame and the second aligned frame is the same as the region of interest in the first residual frame.
[0236] Fig. 9 900 is a schematic diagram showing another possible structure of the video encoding device involved in the above embodiment. The video encoding device includes a processor 901, a transceiver 902 and a memory 903. Fig. 9 As shown, the memory 903 is used to couple with the processor 902, and stores the computer programs necessary for the video encoding device.
[0237] For example, in one embodiment, the transceiver 901 is configured to send encoding information to the decoder 30. The processor 902 is configured for encoding operations or functions of a video encoding device.
[0238] Furthermore, the transceiver 902 is used to obtain a video code stream, and the processor 901 can be used to decode the video code stream to obtain a first aligned frame, a second aligned frame, and at least one residual between the first frame image, the second frame image and the third frame image; generate at least one residual frame based on the at least one residual, and perform super-resolution processing on the second frame image based on the at least one residual frame, the first aligned frame and the second aligned frame to obtain a super-resolved second frame image.
[0239] In a specific implementation, the processor 901 is also used to input the at least one residual frame into a neural network for feature extraction to obtain at least one first feature image; input the first aligned frame, the second aligned frame and the second frame image into the neural network for feature extraction to obtain at least one second feature image; input the at least one first feature image into a first super-resolution network for processing to generate high-frequency information; input the at least one second feature map into a second super-resolution network for processing to generate a brightness channel; and merge the high-frequency information with the brightness channel to generate the super-resolution second frame image.
[0240] In another specific implementation, the processor 901 can also be used to determine the macroblocks of the area of interest in the first residual frame, the macroblocks of the area of interest being the macroblocks in which the sum of all pixel values contained in the current macroblock exceeds a preset value; determine the areas of interest of the remaining residual frames in the at least one residual frame except the first residual frame based on the macroblocks of the area of interest in the first residual frame; and input the macroblocks of all the areas of interest of the at least one residual frame into the neural network for feature extraction, the at least one residual frame including the first residual frame and the remaining residual frames.
[0241] In another specific implementation, the processor 901 is further configured to input the macroblocks of the region of interest in the first aligned frame and the second aligned frame, and the second frame image into the neural network for feature extraction, wherein the region of interest in the first aligned frame and the second aligned frame is the same as the region of interest in the first residual frame.
[0242] For details on the implementation process of the processor 901, see the above-mentioned Embodiment 1 and Embodiment 2 and the attached Figure 5 , Figure 6 and Fig. 7A The description of this embodiment will not be repeated here.
[0243] The video decoding device provided in this embodiment also includes a computer storage medium, which can store computer program instructions. When the program instructions are executed, all steps of the image processing method described in the above embodiments of the present application can be implemented. The computer storage medium includes a disk, an optical disk, a read-only storage memory ROM or a random access memory RAM, etc.
[0244] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product, which is not limited in this embodiment.
[0245] The present application also provides a computer program product, which includes one or more computer program instructions. When a computer loads and executes the computer program instructions, the process or function described in the above embodiments of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.
[0246] The computer program instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one network node, computer, server or data center to another site, computer or server via wired or wireless means.
[0247] Furthermore, when the program is executed, the following steps may be implemented: FIG. 2A to FIG. 7A Some or all of the steps in the embodiments of the video encoding method and the video decoding method are provided. The storage medium in any device can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).
[0248] In the embodiment of the present application, the above-mentioned processor may be a central processing unit (CPU), a network processor (NP) or a combination of a CPU and a NP. The processor may further include a hardware chip. The above-mentioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The memory may include a volatile memory (volatile memory), such as a random-access memory (RAM); the memory may also include a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); the memory may also include a combination of the above-mentioned types of memory.
[0249] It is also known to those skilled in the art that the various illustrative logical blocks and steps listed in this application can be implemented by electronic hardware, computer software, or a combination of the two. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art can use various methods to implement the functions described for each specific application, but such implementation should not be understood as exceeding the scope of protection of this application.
[0250] The various illustrative logic units and circuits described in this application can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration.
[0251] The steps of the method or algorithm described in this application can be directly embedded in hardware, a software unit executed by a processor, or a combination of the two. The software unit can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a UE. Optionally, the processor and the storage medium can also be arranged in different components in the UE.
[0252] It should be understood that in various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the present application.
[0253] In addition, the terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0254] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.
[0255] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the network device / node or apparatus, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.
[0256] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.
Claims
1. An image processing method, characterized in that: The method comprises: Acquire a video code stream, wherein the video code stream includes a first frame image, a second frame image, and a third frame image that are adjacent in time sequence; Decoding the video code stream to obtain a first aligned frame, a second aligned frame, and at least one residual between the first frame image, the second frame image, and the third frame image, wherein the first aligned frame is generated after the first frame image is moved to the second frame image by pixel blocks according to a first motion vector, the second aligned frame is generated after the third frame image is moved to the second frame image by pixel blocks according to a second motion vector, and the residual is a pixel difference between each macroblock of a previous frame image and a subsequent frame image after motion compensation is performed on the previous frame image according to the motion vector; generating at least one residual frame based on the at least one residual; Performing super-resolution processing on the second frame image according to the at least one residual frame, the first aligned frame, and the second aligned frame to obtain a super-resolution second frame image; The super-resolution processing of the second frame image according to the at least one residual frame, the first aligned frame and the second aligned frame to obtain the super-resolution second frame image includes: generating high frequency information based on the at least one residual frame; Generate a brightness channel based on the first aligned frame, the second aligned frame and the second frame image; The high-frequency information is merged with the brightness channel to generate the second frame image after super-resolution.
2. The method according to claim 1, characterized in that: The step of generating at least one residual frame according to the at least one residual comprises: generating a first residual frame according to the first residual; The first residual and the first aligned frame satisfy the following relationship: in, represents the second frame image, represents the first alignment frame, represents the first residual, i represents a macroblock of the first frame image, Indicates that the macroblock i is based on its corresponding motion vector For the macroblock after the move, t1 represents the time when the first frame image is generated, and t2 represents the time when the second frame image is generated.
3. The method according to claim 1 or 2, characterized in that: The generating high frequency information based on the at least one residual frame comprises: Inputting the at least one residual frame into a neural network for feature extraction to obtain at least one first feature image; Inputting the first aligned frame, the second aligned frame and the second frame image into the neural network for feature extraction to obtain at least one second feature image; Inputting the at least one first feature image into a first super-resolution network for processing to generate high-frequency information; The generating of the brightness channel comprises: The at least one second feature map is input into a second super-resolution network for processing to generate a brightness channel.
4. The method according to claim 3, characterized in that Before inputting the at least one residual frame into the neural network for feature extraction, the method further includes: Determine a macroblock of an area of interest in the first residual frame, wherein the macroblock of the area of interest is a macroblock whose sum of all pixel values included in the current macroblock exceeds a preset value; Determine the region of interest of the remaining residual frames in the at least one residual frame except the first residual frame according to the macroblock of the region of interest in the first residual frame; Inputting the at least one residual frame into a neural network for feature extraction includes: The macroblocks of all regions of interest of the at least one residual frame are input into the neural network for feature extraction, and the at least one residual frame includes the first residual frame and the remaining residual frames.
5. The method according to claim 4, characterized in that Inputting the first aligned frame, the second aligned frame and the second frame image into the neural network for feature extraction, comprising: Inputting the macroblocks of the region of interest in the first aligned frame and the second aligned frame, and the second frame image into the neural network for feature extraction; The regions of interest in the first aligned frame and the second aligned frame are the same as the region of interest in the first residual frame.
6. The method according to claim 1 or 2, characterized in that: The first frame image, the second frame image and the third frame image are three frame images in a first image group; Alternatively, the first frame of image is the last frame of image in the first image group, and the second frame of image and the third frame of image are the first two frames of image in the second image group; Alternatively, the first frame image and the second frame image are the last two frame images in the first image group, and the third frame image is the first frame image in the second image group.
7. An image processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring a video code stream, wherein the video code stream includes a first frame image, a second frame image and a third frame image which are adjacent in time sequence; a processing unit, configured to decode the video code stream to obtain a first aligned frame, a second aligned frame, and at least one residual between the first frame image, the second frame image, and the third frame image, wherein the first aligned frame is generated after the first frame image is moved to the second frame image by a first motion vector, the second aligned frame is generated after the third frame image is moved to the second frame image by a second motion vector, and the residual is a pixel difference between each macroblock of a previous frame image and a subsequent frame image after motion compensation is performed on the previous frame image by a motion vector; The processing unit is further configured to generate at least one residual frame according to the at least one residual, and to perform super-resolution processing on the second frame image according to the at least one residual frame, the first aligned frame, and the second aligned frame to obtain a second frame image after super-resolution; Among them, the processing unit is specifically used to generate high-frequency information based on the at least one residual frame; generate a brightness channel based on the first aligned frame, the second aligned frame and the second frame image; and fuse the high-frequency information with the brightness channel to generate the super-resolved second frame image.
8. The device according to claim 7, characterized in that The processing unit is specifically configured to generate a first residual frame; The first residual and the first aligned frame satisfy the following relationship: in, represents the second frame image, represents the first alignment frame, represents the first residual, i represents a macroblock of the first frame image, Indicates that the macroblock i is based on its corresponding motion vector For the macroblock after the move, t1 represents the time when the first frame image is generated, and t2 represents the time when the second frame image is generated.
9. The device according to claim 7 or 8, characterized in that The processing unit is specifically configured to input the at least one residual frame into a neural network for feature extraction to obtain at least one first feature image; input the first aligned frame, the second aligned frame and the second frame image into the neural network for feature extraction to obtain at least one second feature image; The at least one first feature image is input into a first super-resolution network for processing to generate high-frequency information; the at least one second feature image is input into a second super-resolution network for processing to generate a brightness channel.
10. The device according to claim 9, characterized in that The processing unit is further configured to determine a macroblock of an area of interest in the first residual frame, and determine an area of interest of the remaining residual frames in the at least one residual frame except the first residual frame according to the macroblock of the area of interest in the first residual frame; the macroblock of the area of interest is a macroblock whose sum of all pixel values contained in the current macroblock exceeds a preset value; The processing unit is specifically used to input macroblocks of all regions of interest of the at least one residual frame into the neural network for feature extraction, and the at least one residual frame includes the first residual frame and the remaining residual frames.
11. The device according to claim 10, characterized in that The processing unit is specifically used to input the macroblocks of the region of interest in the first aligned frame and the second aligned frame, and the second frame image into the neural network for feature extraction; The regions of interest in the first aligned frame and the second aligned frame are the same as the region of interest in the first residual frame.
12. The device according to claim 7 or 8, characterized in that The first frame image, the second frame image and the third frame image are three frame images in a first image group; Alternatively, the first frame of image is the last frame of image in the first image group, and the second frame of image and the third frame of image are the first two frames of image in the second image group; Alternatively, the first frame image and the second frame image are the last two frame images in the first image group, and the third frame image is the first frame image in the second image group.
13. A communication device, comprising a processor, wherein the processor is coupled to a memory, wherein: The memory is used to store instructions; The processor is configured to execute instructions in the memory so that the communication device performs the method according to any one of claims 1 to 6.
14. A computer-readable storage medium, wherein instructions are stored in the storage medium, characterized in that: When the instructions are executed, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Super-resolution image reconstruction method and device, medium and computing equipment
CN108109109A
Video compression-based continuous picture management method and device, equipment and medium
CN108769690A
Video image processing method and device
CN110062164A
