Image quality enhancement method and apparatus for video layer data rate

Abstract

Embodiments of the present disclosure provide an image quality enhancement method and apparatus for a video layer data rate. The method comprises: performing video encoding to a first resolution pixel to obtain a video layer data rate, a second resolution pixel, and predicted information of the first resolution pixel; upsampling the second resolution pixel to obtain a second original resolution pixel; performing calculation to a first original resolution pixel and the second original resolution pixel to obtain an initial enhancement layer pixel and eliminating redundancy; compressing the redundancy-eliminated enhancement layer pixel, the predicted information and matching information to obtain an enhancement layer data rate; and sending the video layer data rate and the enhancement layer data rate to a decoding end, so as to enhance the image quality of the video layer data rate on the basis of the enhancement layer data rate. Narrow-band high-definition in a cloud desktop / video conference desktop sharing scenario is realized, computing resources of an encoding end and a decoding end and resource occupancy of network transmission are reduced, and the problem in the related art that the super resolution enhancement scheme is not suitable for a cloud desktop / video conference desktop sharing scenario with complex and diversified characteristics is solved.

Description

Method and device for enhancing image quality of video layer code stream

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present disclosure is based on Chinese patent application CN 202310810047.3 filed on June 30, 2023, entitled “Image quality method and device for video layer code stream”, and claims the priority of the patent application, and all the disclosed contents are incorporated into the present disclosure by reference. Technical Field

[0003] The present disclosure relates to the field of communications, and more particularly, to a method and apparatus for improving the image quality of a video layer stream. Background Art

[0004] With the popularity and widespread use of high-definition screens, users have gradually raised demands for high resolution in cloud desktop / video conferencing desktop sharing scenarios. How to ensure that users can smoothly and clearly realize high-resolution cloud desktops for office work, design, or video conferencing desktop sharing on the basis of smaller encoding and decoding resources and bandwidth resources has become a difficult problem that cloud desktops and video conferencing urgently need to overcome.

[0005] Currently, there are few high-resolution super-resolution enhancement solutions based on cloud desktop / video conferencing desktop sharing scenarios. Basically, they are super-resolution solutions for ordinary video scenarios such as live video. The technical means used are generally divided into four types. Traditional super-resolution reconstruction methods include pixel interpolation-based reconstruction methods, degradation model-based reconstruction methods, machine learning-based reconstruction methods, and the most commonly used deep learning-based convolutional neural network-based reconstruction methods, such as super-resolution reconstruction convolutional neural network (SRCNN) and super-resolution generative adversarial network (SRGAN).

[0006] However, these solutions are not suitable for cloud desktop / video conferencing desktop sharing scenarios that require high definition, high real-time performance, no distortion or noise, and complex and diverse scenarios.

[0007] Summary of the Invention

[0008] The embodiments of the present disclosure provide a method and device for enhancing the image quality of a video layer code stream, so as to at least solve the problem that the super-resolution enhancement solution in the related art is not suitable for cloud desktop / video conference desktop sharing scenarios with high definition, high real-time performance, no distortion and noise, and complex and diverse scenes.

[0009] According to an embodiment of the present disclosure, a method for enhancing the image quality of a video layer stream is provided, which is applied to an encoding end. The method includes:

[0010] Downsampling the first original resolution pixels of the screen to obtain first resolution pixels;

[0011] Obtaining matching information between the first resolution pixels and the historical resolution pixels of the screen;

[0012] Performing video encoding on the first-resolution pixels to obtain a video layer code stream corresponding to the first-resolution pixels, second-resolution pixels, and prediction information of the first-resolution pixels; the second-resolution pixels are restored pixels of the first-resolution pixels;

[0013] Upsampling the second-resolution pixels to obtain second-original-resolution pixels;

[0014] Performing difference calculation on the first original resolution pixels and the second original resolution pixels to obtain initial enhancement layer pixels, and eliminating redundancy of the initial enhancement layer pixels;

[0015] Compressing the enhanced layer pixels after redundancy elimination, the prediction information of the first resolution pixels, and the matching information to obtain an enhanced layer code stream;

[0016] The video layer code stream and the enhancement layer code stream are sent to a decoding end, so that the decoding end performs image quality enhancement on the video layer code stream based on the enhancement layer code stream.

[0017] According to another embodiment of the present disclosure, a method for enhancing the image quality of a video layer stream is provided, which is applied to a decoding end, and the method includes:

[0018] Receive the enhancement layer stream and video layer stream sent by the encoder;

[0019] Decoding the video layer code stream to obtain pixels at a second resolution, where the pixels at the second resolution are restored pixels of the pixels at the first resolution;

[0020] Upsampling the second-resolution pixels to obtain second-original-resolution pixels;

[0021] Decoding the enhancement layer code stream to obtain enhancement layer pixels, prediction information of the first resolution pixels, and matching information between the first resolution pixels and historical resolution pixels of the screen;

[0022] synthesizing the second original resolution pixel and the enhancement layer pixel to obtain a third original resolution pixel; the third original resolution pixel is a pixel after the second original resolution pixel is enhanced;

[0023] The third original resolution pixel is enhanced according to the prediction information or the matching information of the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0024] According to another embodiment of the present disclosure, a device for enhancing the image quality of a video layer stream is provided, which is applied to an encoding end and includes:

[0025] a downsampling module configured to downsample the first original resolution pixels of the screen to obtain first resolution pixels;

[0026] an acquisition module, configured to acquire matching information between the first resolution pixels and the historical resolution pixels of the screen;

[0027] a video encoding module configured to perform video encoding on the first-resolution pixels to obtain a video layer code stream corresponding to the first-resolution pixels, second-resolution pixels, and prediction information of the first-resolution pixels; the second-resolution pixels are restored pixels of the first-resolution pixels;

[0028] a first upsampling module, configured to upsample the second resolution pixels to obtain second original resolution pixels;

[0029] a difference calculation module configured to perform difference calculation on pixels of the first original resolution and pixels of the second original resolution to obtain pixels of an initial enhancement layer, and eliminate redundancy of pixels of the initial enhancement layer;

[0030] a compression module configured to compress the enhanced layer pixels after redundancy elimination, the prediction information of the first resolution pixels, and the matching information to obtain an enhanced layer code stream;

[0031] The sending module is configured to send the video layer code stream and the enhancement layer code stream to a decoding end, so as to assist the decoding end in enhancing the image quality of the video layer code stream based on the enhancement layer code stream.

[0032] According to another embodiment of the present disclosure, a device for enhancing the image quality of a video layer stream is provided, which is applied to a decoding end and includes:

[0033] A receiving module configured to receive the enhancement layer stream and the video layer stream sent by the encoding end;

[0034] a first decoding module configured to decode the video layer code stream to obtain pixels of a second resolution, where the pixels of the second resolution are restored pixels of the pixels of the first resolution;

[0035] a second upsampling module, configured to upsample the second resolution pixels to obtain second original resolution pixels;

[0036] a second decoding module configured to decode the enhancement layer code stream to obtain prediction information of the enhancement layer pixels of the first resolution and matching information between the first resolution pixels and historical resolution pixels of the screen;

[0037] a synthesis module configured to synthesize the second original resolution pixel and the enhancement layer pixel to obtain a third original resolution pixel; the third original resolution pixel is a pixel after the second original resolution pixel is enhanced;

[0038] The pixel enhancement module is configured to enhance the third original resolution pixels according to the prediction information of the first resolution pixels or the matching information to obtain fourth original resolution pixels with enhanced image quality.

[0039] According to another embodiment of the present disclosure, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when running.

[0040] According to another embodiment of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1 is a hardware structure block diagram of a mobile terminal according to a method for improving the image quality of a video layer code stream according to an embodiment of the present disclosure;

[0042] FIG2 is a schematic diagram of a network architecture of an encoding terminal according to an embodiment of the present disclosure;

[0043] FIG3 is a flow chart of the image quality of a video layer code stream according to an embodiment of the present disclosure;

[0044] FIG4 is a schematic diagram of a network architecture of a decoding terminal according to an embodiment of the present disclosure;

[0045] FIG5 is a flowchart of the image quality of a video layer code stream according to another embodiment of the present disclosure;

[0046] FIG6 is a structural block diagram of a device for enhancing image quality of a video layer stream according to an embodiment of the present disclosure;

[0047] FIG7 is a structural block diagram of a device for enhancing image quality of a video layer stream according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings and in conjunction with embodiments.

[0049] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0050] The super-resolution enhancement schemes in related technologies generally include reconstruction methods based on pixel interpolation, reconstruction methods based on degradation models, reconstruction methods based on machine learning, and reconstruction methods based on deep learning of convolutional neural networks.

[0051] However, reconstruction based on typical interpolation algorithms such as pixel interpolation, such as neighborhood interpolation, bilinear interpolation, and bicubic interpolation, has blurring and aliasing problems, and is not suitable for high-definition scenarios such as cloud desktops / video conferencing desktop sharing; super-resolution reconstruction based on degradation models is generally suitable for specific scenarios such as low-resolution images introduced by camera blur, distortion, noise characteristics, etc., and is not suitable for cloud desktops / video conferencing desktop sharing, where the original high-definition downsampling introduces image quality degradation; machine learning methods such as those based on image self-similarity and sparse representation rely on trained samples and learning dictionaries, and their wide applicability in different scenarios remains to be tested; and the deep learning methods based on convolutional neural networks, which are currently being studied more, can achieve better enhancement effects based on online training and feedback mechanisms, but cannot meet the real-time requirements of cloud desktops / video conferencing desktop sharing scenarios.

[0052] Low Complexity Enhancement Video Coding (LCEVC), a video compression algorithm for high-resolution shared desktops in cloud desktops and video conferencing, uses a double-downsampling algorithm to upsample the low-resolution reconstructed frame, calculate parameters with the original image layer, and then encode the residual. This is then transmitted to the decoder as an enhanced bitstream to assist in differential reconstruction of the downsampled bitstream, resulting in excellent real-time performance and image enhancement. However, LCEVC does not analyze the feature information and prediction of the original pixels, and treats all pixels equally, resulting in high bandwidth and CPU encoding consumption.

[0053] The technical concept of the embodiments of the present disclosure is: since cloud desktop / video conference desktop sharing has the characteristics of high definition, high real-time performance, no distortion and noise, and complex and diverse scenes, based on the analysis of the original resolution pixel characteristics of cloud desktop / video conference desktop sharing, a de-redundant enhancement layer code stream is added to the low-resolution code stream, thereby achieving a super-resolution enhancement effect at the decoding end.

[0054] The method embodiments provided in the embodiments of the present disclosure can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal according to a method for image quality of a video layer code stream in accordance with an embodiment of the present disclosure. As shown in Figure 1, the mobile terminal may include one or more (only one is shown in Figure 1) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that the structure shown in Figure 1 is only for illustration and does not limit the structure of the above-mentioned mobile terminal. For example, the mobile terminal may also include more or fewer components than those shown in Figure 1, or have a configuration different from that shown in Figure 1.

[0055] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image quality method of the video layer code stream in the embodiment of the present disclosure. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0056] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by the mobile terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0057] FIG2 is a schematic diagram of the network architecture of the encoding end according to an embodiment of the present disclosure. The embodiment of the present disclosure can run on the network architecture shown in FIG2. As shown in FIG2, the network architecture includes: a downsampling module, a video encoding module, an upsampling module, a feature information calculation module, an enhancement information calculation module, a redundant information elimination module, and a self-developed encoding module. The functions of each module are as follows:

[0058] Downsampling module: The input data is set to the original resolution size of the cloud desktop / video conference desktop sharing, which has been transcoded into the video encoding data format. The input data can be downsampled according to actual needs to obtain output data with a smaller resolution after downsampling.

[0059] Video encoding module: connected to the downsampling module, configured to perform video encoding on the downsampled output data according to the data format, and output the video layer code stream, the restored frame corresponding to the output data, and the prediction information corresponding to the output data.

[0060] Upsampling module: connected to the video encoding module, configured to perform upsampling operations on the restored frames. Upsampling is an operation opposite to downsampling, which is an operation that restores low-resolution data to the original high-resolution through a certain method.

[0061] Feature information calculation module: connected to the downsampling module, set to take the downsampled pixel data as input, analyze the pixel features, and output the feature information of the downsampled pixel data. Among them, the pixel features may include unique identifiers for pixel gradient, pixel color distribution, pixel hash value, etc. These pixel features can be used to enhance the sparse processing of pixels. In addition, the feature information calculation module can divide the pixel area type according to the pixel gradient and pixel color distribution, for example, it can be divided into text-like pixel areas and image pixel areas; it can perform hash matching from the historical downsampled screen pixels according to the pixel hash value to obtain matching information with the historical downsampled screen pixels.

[0062] Enhancement information calculation module: connects the upsampling module and the original resolution pixel input, and is set to perform difference calculation on the original resolution pixels and the upsampled restored data to obtain the initial enhancement layer pixels.

[0063] The redundant information elimination module is connected to the video encoding module, the feature information module, and the enhancement information calculation module respectively, and is configured to eliminate the spatial redundancy, temporal redundancy, and visual redundancy in the initial enhancement layer pixels based on the prediction information output by the video encoding module and the feature information output by the feature information module, thereby obtaining a sparse enhancement layer pixel of the original resolution size.

[0064] The self-developed encoding module is connected to the redundant information elimination module and is configured to compress the enhanced layer pixels after the redundant information is removed and encode and compress the matching information and the prediction information.

[0065] In this embodiment, a method for enhancing the image quality of a video layer stream running on a network architecture at an encoding end is provided. FIG3 is a flowchart of the image quality enhancement of a video layer stream according to an embodiment of the present disclosure. As shown in FIG3 , the process includes the following steps:

[0066] Step S301 : down-sample the first original resolution pixels of the screen to obtain first resolution pixels.

[0067] As an example, downsampling the first original resolution pixel may obtain a first resolution pixel lower than the first original resolution pixel, that is, a first low-resolution pixel.

[0068] For example, a changing screen image can be captured in a cloud desktop / video conference desktop sharing system, and the captured screen image can be converted into pixels of a first original resolution size corresponding to video encoding, and then downsampled based on the pixels of the first original resolution size to obtain pixels of a first low resolution.

[0069] It should be noted that downsampling only changes the size of the pixel set, but does not change the format of the pixel set.

[0070] Step S302: Acquire matching information between the first resolution pixels and the historical resolution pixels of the screen.

[0071] As an example, the encoding end may obtain matching information between the first resolution pixels and the historical resolution pixels of the screen.

[0072] As an example, the matching information between the first resolution pixels and the historical resolution pixels of the screen may include a matching relationship between the first resolution pixel area and the historical resolution pixel area.

[0073] In an optional embodiment, obtaining matching information between the first resolution pixel and the historical resolution pixel of the screen includes: extracting feature information of the first resolution pixel, the feature information including a pixel hash value; obtaining a hash value of the historical resolution pixel; and obtaining matching information between the first resolution pixel and the historical resolution pixel of the screen based on the pixel hash value of the first resolution pixel and the hash value of the historical resolution pixel.

[0074] As an example, feature information of the first resolution pixel can be extracted, and the pixel hash value in the feature information of the first resolution pixel can be matched with the hash value of the historical resolution pixel to obtain matching information between the first resolution pixel and the historical resolution pixel.

[0075] As an example, the matching relationship between the first resolution pixel region and the historical resolution pixel region can be matched based on the pixel hash value of the first resolution pixel region and the hash value of the historical resolution pixel region.

[0076] For example, the hash feature value of the first resolution pixel area can be calculated based on the feature information of the first resolution pixel, and hash matching can be performed from the historically downsampled screen pixels, or pixel matching can be performed based on the cloud desktop / video conference desktop sharing operation information such as keyboard and mouse, so as to obtain the matching relationship between the first resolution pixel area and the historical resolution pixel area of ​​the screen.

[0077] In an optional embodiment, the feature information also includes pixel gradient and pixel color distribution. Before eliminating the redundancy of the initial enhancement layer pixels, it also includes: classifying the first resolution pixel area according to the pixel gradient of the first resolution pixel and the pixel color distribution of the first resolution pixel to obtain classification information of the first resolution pixel area.

[0078] As an example, the feature information may also include pixel gradients and pixel color distributions. Before eliminating the redundancy of the initial enhancement layer pixels, the pixel gradients and pixel color distributions of the first resolution pixels may be pre-extracted. The first resolution pixel areas may be classified according to the pixel gradients and pixel color distributions of the first resolution pixels to obtain classification information of the first resolution pixel areas.

[0079] For example, the first resolution pixel region may be divided into a text-like pixel region and an image pixel region according to the pixel gradient and pixel color distribution of the first resolution pixels.

[0080] Step S303 , performing video encoding on the first resolution pixels to obtain a video layer code stream corresponding to the first resolution pixels, second resolution pixels, and prediction information of the first resolution pixels; the second resolution pixels are restored pixels of the first resolution pixels.

[0081] As an example, video encoding can be performed on first-resolution pixels to obtain a video layer stream corresponding to the first-resolution pixels, second-resolution pixels, and prediction information of the first-resolution pixels, where the second-resolution pixels are restored pixels of the first-resolution pixels.

[0082] As an example, based on the first resolution pixels, conventional video encoding can be performed. The specific encoding specification can be determined according to actual conditions, and the embodiments of the present disclosure do not limit this.

[0083] Step S304, upsampling the second resolution pixels to obtain second original resolution pixels;

[0084] As an example, the second-resolution pixel is a restored pixel of the first-resolution pixel, and is also a low-resolution pixel.

[0085] As an example, an upsampling operation corresponding to downsampling may be performed based on the second-resolution pixels, and the low-resolution second-resolution pixels may be upsampled to pixels of the second original resolution size.

[0086] Step S305 , performing difference calculation on the first original resolution pixels and the second original resolution pixels to obtain initial enhancement layer pixels, and eliminating redundancy of the initial enhancement layer pixels.

[0087] As an example, a difference calculation may be performed between the second original resolution pixels and the first original resolution pixels before downsampling to obtain initial enhancement layer pixels.

[0088] As an example, redundancy of pixels in the initial enhancement layer may be eliminated to reduce resource usage at the encoding end.

[0089] As an example, redundancy of enhancement layer pixels may be eliminated based on spatial correlation (pixel references at different positions in the same frame) and temporal correlation (pixel references at the same position in different frames) between pixels.

[0090] Methods for eliminating the redundancy of pixels in the initial enhancement layer may include the following:

[0091] In an optional embodiment, redundancy of the initial enhancement layer pixels is eliminated based on prediction information of the first resolution pixels.

[0092] In an optional embodiment, redundancy of the initial enhancement layer pixels is eliminated based on matching information between the first resolution pixels and the historical resolution pixels of the screen.

[0093] In an optional embodiment, redundancy of the initial enhancement layer pixels is eliminated based on classification information of the first resolution pixel area.

[0094] As an example, the redundancy of enhancement layer pixels may include, but is not limited to, temporal redundancy, spatial redundancy, and visual redundancy. In response to different types of redundancy in enhancement layer pixels, the disclosed embodiments provide multiple methods for eliminating the redundancy of initial enhancement layer pixels. These methods can be freely combined based on actual circumstances to more comprehensively eliminate the redundancy of initial enhancement layer pixels.

[0095] The embodiments of the present disclosure do not limit the combination of multiple methods for eliminating redundancy of initial enhancement layer pixels.

[0096] For example, the redundancy of the initial enhancement layer pixels can be eliminated based on the prediction information of the first resolution pixels or the matching information between the first resolution pixels and the historical resolution pixels of the screen or the classification information of the first resolution pixel area; the redundancy of the initial enhancement layer pixels can also be eliminated based on the prediction information of the first resolution pixels, and then the redundancy of the initial enhancement layer pixels can be further eliminated based on the classification information of the first resolution pixel area; the redundancy of the initial enhancement layer pixels can also be eliminated based on the matching information between the first resolution pixels and the historical resolution pixels of the screen, and then the redundancy of the initial enhancement layer pixels can be further eliminated based on the classification information of the first resolution pixel area; the redundancy of the initial enhancement layer pixels can also be eliminated based on the prediction information of the first resolution pixels, and the redundancy of the initial enhancement layer pixels can be eliminated based on the matching information between the first resolution pixels and the historical resolution pixels of the screen, and then the redundancy of the initial enhancement layer pixels can be further eliminated based on the classification information of the first resolution pixel area.

[0097] It should be noted that the embodiment of the present disclosure does not limit the order of implementing each method of eliminating the redundancy of the initial enhancement layer pixels in the above-mentioned combination of examples.

[0098] As can be seen from the above optional embodiments, there are multiple ways to eliminate the redundancy of the initial enhancement layer pixels. To facilitate understanding by those skilled in the art, the following describes the specific implementation methods of eliminating the redundancy of the initial enhancement layer pixels one by one:

[0099] In an optional embodiment, the prediction information of the first resolution includes reference frame information, prediction mode information, and motion vector information, and eliminating redundancy of the initial enhancement layer pixels based on the prediction information of the first resolution pixels includes:

[0100] Determining, based on the reference frame information, the prediction mode information, and the motion vector information, pixel regions having prediction residuals of 0 at different positions within the same reference frame and pixel regions having prediction residuals of 0 at the same position within different reference frames;

[0101] The initial enhanced pixels of the pixel areas with zero prediction residuals at different positions in the same reference frame and the pixel areas with zero prediction residuals at the same position in different reference frames are set to 0 to obtain second enhanced layer pixels that eliminate temporal redundancy and spatial redundancy.

[0102] As an example, the pixel areas with prediction residuals of 0 at different positions in the same reference frame and the pixel areas with prediction residuals of 0 at the same position in different reference frames can be determined based on the reference frame information, prediction mode information and motion vector information in the prediction information of the first resolution.

[0103] As an example, all the initial enhancement layer pixels of the pixel area with a prediction residual of 0 at different positions in the same reference frame can be cleared to eliminate the temporal redundancy of the initial enhancement pixels; the initial enhancement layer pixels of the pixel area with a prediction residual of 0 at the same position in different reference frames can be set to 0 to eliminate the spatial redundancy of the initial enhancement pixels.

[0104] In an optional embodiment, eliminating redundancy of the initial enhancement layer pixels according to matching information between the first resolution pixels and the historical resolution pixels of the screen includes:

[0105] Determining a target pixel area of ​​the first resolution pixels that matches the historical resolution pixels based on matching information between the first resolution pixels and the historical resolution pixels of the screen;

[0106] The initial enhanced pixels corresponding to the target pixel area are set to 0 to obtain the third enhanced layer pixels that eliminate temporal redundancy.

[0107] As an example, the matching information between the first resolution pixel and the historical resolution pixel of the screen may include the matching relationship between the first resolution pixel area and the historical resolution pixel area, and the target pixel area of ​​the first resolution pixel that matches the historical resolution pixel can be determined based on the matching relationship between the first resolution pixel area and the historical resolution pixel area.

[0108] As an example, the initial enhanced pixels corresponding to the target pixel area may be set to 0 to eliminate the temporal redundancy of the initial enhanced pixels.

[0109] In an optional embodiment, the classification information of the first-resolution pixel region includes a text-like pixel region and an image pixel region, and eliminating redundancy of the initial enhancement layer pixels according to the classification information of the first-resolution pixel region includes:

[0110] Obtaining enhancement layer pixels of the text-like pixel area, and if the enhancement layer pixels of the text-like pixel area are less than a first preset enhancement layer pixel threshold, setting the initial enhancement layer pixels, the second enhancement layer pixels, or the third enhancement layer pixels corresponding to the text-like pixel area to 0, to obtain fourth enhancement layer pixels that eliminate visual redundancy;

[0111] Obtain the enhancement layer pixels of the image pixel area, and when the enhancement layer pixels of the image pixel area are less than the second preset enhancement layer pixel threshold, set the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the image pixel area to 0 to obtain the fourth enhancement layer pixels that eliminate visual redundancy.

[0112] As an example, the first-resolution pixel area may be divided into a text-like pixel area and an image pixel area.

[0113] As an example, since text-like areas contain more contour details and are highly sensitive to pixel loss, while natural image areas have a low perception of pixel loss except for internal details and textures, different pixel loss thresholds can be set in advance for the two types of areas to eliminate insensitive enhancement information that is less than the pixel threshold. Generally, the pixel loss threshold of text-like areas is set smaller than the pixel loss threshold of image areas.

[0114] As an example, the pixel loss threshold for the text-like region can be set to a first preset enhancement layer pixel threshold, and the pixel loss threshold for the image region can be set to a second preset enhancement layer pixel threshold. After respectively obtaining the enhancement layer pixels of the text-like pixel region and the enhancement layer pixels of the image pixel region, the enhancement layer pixels of the text-like pixel region can be compared with the first preset enhancement layer pixel threshold, and the enhancement layer pixels of the image pixel region can be compared with the second preset enhancement layer pixel threshold.

[0115] As an example, when the enhancement layer pixels in the text-like pixel area are smaller than the first preset enhancement layer pixel threshold, the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the text-like pixel area can be set to 0 to obtain the fourth enhancement layer pixels that eliminate visual redundancy; when the enhancement layer pixels in the image pixel area are smaller than the second preset enhancement layer pixel threshold, the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the image pixel area can be set to 0 to obtain the fourth enhancement layer pixels that eliminate visual redundancy.

[0116] Step S306 : compressing the enhanced layer pixels after redundancy elimination, the prediction information of the first resolution pixels, and the matching information to obtain an enhanced layer code stream.

[0117] As an example, the enhanced layer pixels after redundancy elimination, the prediction information of the first resolution pixels, and the matching information between the first resolution pixels and the screen historical resolution pixels may be compressed to obtain an enhanced layer code stream.

[0118] In an optional embodiment, the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels and the compression to obtain the enhanced layer code stream include: compressing the second enhancement layer pixels or the third enhancement layer pixels or the fourth enhancement layer pixels, the prediction information of the first resolution pixels and the matching information to obtain the enhanced layer code stream.

[0119] As an example, since the various methods of eliminating the redundancy of the initial enhancement layer pixels in the embodiments of the present disclosure can be freely combined according to actual conditions, if the second enhancement layer pixels are obtained after the combination, the prediction information of the second enhancement layer pixels and the first resolution pixels, and the matching information of the first resolution pixels and the historical resolution pixels of the screen can be compressed to obtain the enhancement layer code stream; if the third enhancement layer pixels are obtained after the combination, the prediction information of the third enhancement layer pixels and the first resolution pixels, and the matching information of the first resolution pixels and the historical resolution pixels of the screen can be compressed to obtain the enhancement layer code stream; if the fourth enhancement layer pixels are obtained after the combination, the prediction information of the fourth enhancement layer pixels and the first resolution pixels, and the matching information of the first resolution pixels and the historical resolution pixels of the screen can be compressed to obtain the enhancement layer code stream.

[0120] Step S307: Send the video layer code stream and the enhancement layer code stream to a decoding end, so that the decoding end performs image quality enhancement on the video layer code stream based on the enhancement layer code stream.

[0121] As an example, the encoder may send the video layer stream and the enhancement layer stream to the decoder, so that the decoder may enhance the image quality of the video layer stream based on the enhancement layer stream.

[0122] In the disclosed embodiment, the disclosed embodiment performs video encoding on the first-resolution pixels after downsampling the first original-resolution pixels of the screen to obtain the video layer code stream corresponding to the first-resolution pixels, the restored pixels of the first-resolution pixels, and the prediction information of the first-resolution pixels. At the same time, the matching information between the first-resolution pixels and the historical-resolution pixels of the screen is obtained. The restored pixels of the first-resolution pixels are upsampled to obtain the second original-resolution pixels, and the difference between the first original-resolution pixels and the second original-resolution pixels is calculated to obtain the initial enhancement layer pixels. The redundancy of the initial enhancement layer pixels is then eliminated. The enhanced layer pixels, prediction information, and matching information after eliminating redundancy are compressed and encoded and transmitted to the decoding end to assist in enhancing the image quality of the corresponding video layer code stream. This achieves a narrowband high-definition effect in cloud desktop / video conference desktop sharing scenarios, saves computing resources and network transmission resources on the encoding and decoding ends, and solves the problem in the related art that the super-resolution enhancement solution is not suitable for cloud desktop / video conference desktop sharing scenarios with high definition, high real-time performance, no distortion noise, and complex and diverse scenes.

[0123] FIG4 is a schematic diagram of a network architecture of a decoding end according to an embodiment of the present disclosure. The embodiment of the present disclosure can be run on the network architecture shown in FIG4 . As shown in FIG4 , the network architecture includes: a video decoding module, an upsampling module, a self-developed decoding module, and a synthesis and restoration module. The functions of each module are as follows:

[0124] Video decoding module: configured to receive the video layer code stream of low-resolution pixels sent by the encoding end, and decode or decompress the video layer code stream to obtain a low-resolution restored pixel.

[0125] Upsampling module: connected to the video decoding module, configured to receive the decoded low-resolution restored pixels, upsample the decoded low-resolution restored pixels, and restore the low-resolution restored pixels to pixels of the original resolution size.

[0126] Self-developed decoding module: This module is configured to receive the enhancement layer stream transmitted by the encoder and decode the enhancement layer stream according to the algorithm corresponding to the encoder to obtain the enhancement layer pixels at the original resolution, the prediction information of the first resolution pixels, and the matching information between the first resolution pixels and the historical resolution pixels of the screen;

[0127] Synthesis and restoration module: It is connected to the upsampling module and the self-developed decoding module respectively, and is configured to fill the enhancement layer code stream of the original resolution into the pixels of the original resolution size after upsampling to obtain an enhanced frame. At the same time, it combines the prediction information of the first resolution pixels or the matching information of the first resolution pixels and the historical resolution pixels of the screen, and further fills it into the current enhancement frame to obtain restored pixels with enhanced image quality.

[0128] In this embodiment, a method for enhancing the image quality of a video layer stream in a network architecture running on a decoding end is provided. FIG5 is a flowchart of enhancing the image quality of a video layer stream according to another embodiment of the present disclosure. As shown in FIG5 , the process includes the following steps:

[0129] Step S501: receiving the enhancement layer stream and the video layer stream sent by the encoder.

[0130] In the embodiment of the present disclosure, the decoding end can receive the enhancement layer code stream and the video layer code stream sent by the encoding end.

[0131] Step S502: Decode the video layer code stream to obtain second-resolution pixels, where the second-resolution pixels are restored pixels of the first-resolution pixels.

[0132] As an example, the video layer code stream may be decoded or decompressed. This process corresponds to the encoding process at the encoding end, and restored pixels of the first resolution pixels are obtained after decoding.

[0133] The first resolution pixel is a low-resolution pixel lower than the original resolution of the screen, and the second resolution pixel is a restored pixel of the first resolution pixel, which is a low-resolution restored pixel lower than the original resolution of the screen.

[0134] Step S503: up-sample the second-resolution pixels to obtain second-original-resolution pixels.

[0135] As an example, the second resolution pixels may be upsampled to restore the second resolution pixels to the original screen resolution size, thereby obtaining second original resolution pixels, where the second original resolution pixels may be restored pixels of the first original resolution.

[0136] Step S504: Decode the enhancement layer code stream to obtain enhancement layer pixels, prediction information of the first resolution pixels, and matching information between the first resolution pixels and the historical resolution pixels of the screen.

[0137] As an example, the enhancement layer code stream may be decoded to obtain enhancement layer pixels, prediction information of first resolution pixels, and matching information between the first resolution pixels and historical resolution pixels of the screen.

[0138] Step S505 : synthesize the second original resolution pixel and the enhancement layer pixel to obtain a third original resolution pixel; the third original resolution pixel is the pixel after the second original resolution pixel is enhanced.

[0139] As an example, the second original resolution pixels and the enhancement layer pixels may be synthesized to obtain enhanced third original resolution pixels.

[0140] Step S506 : enhancing the third original resolution pixel according to the prediction information or the matching information of the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0141] As an example, the embodiment of the present disclosure may enhance the third original resolution pixel based on the prediction information of the first resolution pixel to obtain the fourth original resolution pixel with enhanced image quality; it may also enhance the third original resolution pixel based on the matching information between the first resolution pixel and the historical resolution pixel of the screen to obtain the fourth original resolution pixel with enhanced image quality; it may also enhance the third original resolution pixel based on both the prediction information of the first resolution pixel and the matching information between the first resolution pixel and the historical resolution pixel of the screen to obtain the fourth original resolution pixel with enhanced image quality.

[0142] It should be noted that, in the embodiment of the present disclosure, when enhancing the third original resolution pixel based on the prediction information of the first resolution pixel and the matching information between the first resolution pixel and the historical resolution pixel of the screen, the order of enhancement is not restricted. The third original resolution pixel can be enhanced first based on the prediction information of the first resolution pixel, and then enhanced based on the matching information between the first resolution pixel and the historical resolution pixel of the screen; or the third original resolution pixel can be enhanced first based on the matching information between the first resolution pixel and the historical resolution pixel of the screen, and then enhanced based on the prediction information of the first resolution pixel.

[0143] The following is a detailed description of the enhancement process of the third original resolution pixels:

[0144] In an optional embodiment, the prediction information includes reference frame information, prediction mode information, and motion vector information, and enhancing the third original resolution pixel based on the prediction information of the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality includes:

[0145] Calculating enhanced pixels in an area where the prediction residual is 0 pixels based on the reference frame information, the prediction mode information, and the motion vector information in the prediction information;

[0146] Based on the enhanced pixels, the third original resolution pixels are enhanced to obtain fourth original resolution pixels with enhanced image quality.

[0147] As an example, the enhanced pixels in the area where the prediction residual is 0 pixels can be calculated based on the reference frame information, prediction method information and motion vector information in the prediction information of the first resolution pixels. The third original resolution pixels can be enhanced based on the enhanced pixels in the area where the prediction residual is 0 pixels to obtain fourth original resolution pixels with enhanced image quality.

[0148] In an optional embodiment, enhancing the third original resolution pixels according to the matching information to obtain fourth original resolution pixels with enhanced image quality includes:

[0149] Determining, based on matching information between the first-resolution pixel and the historical-resolution pixel of the screen, a pixel region of the historical-resolution pixel that matches the first-resolution pixel;

[0150] The third original resolution pixel is enhanced based on a pixel region of historical resolution pixels that matches the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0151] As an example, the matching information between the first resolution pixel and the historical resolution pixel of the screen may include the matching relationship between the first resolution pixel area and the historical resolution pixel area, and the pixel area of ​​the historical resolution pixel that matches the first resolution pixel can be determined based on the matching relationship between the first resolution pixel area and the historical resolution pixel area.

[0152] As an example, the third original resolution pixel may be enhanced based on a pixel region of a historical resolution pixel that matches the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0153] In this embodiment, the decoder receives the enhancement layer stream and video layer stream sent by the encoder, decodes each of them, and synthesizes the second original resolution pixels obtained by decoding and upsampling the video layer stream with the enhancement layer pixels obtained by decoding the enhancement layer stream to obtain third original resolution pixels. The third original resolution pixels are then further enhanced based on prediction information or matching information obtained by decoding the enhancement layer stream to obtain fourth enhancement layer pixels with enhanced image quality. This achieves narrowband high-definition effects in cloud desktop / video conferencing desktop sharing scenarios, conserves computing resources and network transmission resources at the encoder and decoder, and addresses the problem in related technologies where super-resolution enhancement solutions are unsuitable for cloud desktop / video conferencing desktop sharing scenarios that require high definition, high real-time performance, no distortion or noise, and complex and diverse scenes.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0155] This embodiment also provides a device for enhancing the image quality of a video layer stream. This device is used to implement the above-mentioned embodiments and preferred implementations. Details that have already been described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0156] FIG6 is a structural block diagram of a device for enhancing the image quality of a video layer stream according to an embodiment of the present disclosure, which is applied to an encoding end. As shown in FIG6 , the device includes:

[0157] A downsampling module 601 is configured to downsample the first original resolution pixels of the screen to obtain first resolution pixels;

[0158] An acquisition module 602 is configured to acquire matching information between pixels of the first resolution and pixels of a historical resolution of the screen;

[0159] The video encoding module 603 is configured to perform video encoding on the first resolution pixels to obtain a video layer stream corresponding to the first resolution pixels, second resolution pixels, and prediction information of the first resolution pixels; the second resolution pixels are restored pixels of the first resolution pixels;

[0160] A first upsampling module 604 is configured to upsample the second resolution pixels to obtain second original resolution pixels;

[0161] a difference calculation module 605 configured to perform difference calculation on pixels of the first original resolution and pixels of the second original resolution to obtain pixels of an initial enhancement layer, and eliminate redundancy of pixels of the initial enhancement layer;

[0162] A compression module 606 is configured to compress the enhanced layer pixels after redundancy elimination, the prediction information of the first resolution pixels, and the matching information to obtain an enhanced layer code stream;

[0163] The sending module 607 is configured to send the video layer code stream and the enhancement layer code stream to a decoding end, so as to assist the decoding end in enhancing the image quality of the video layer code stream based on the enhancement layer code stream.

[0164] In an optional embodiment, the difference calculation module 605 includes:

[0165] The first redundancy elimination submodule is configured to eliminate redundancy of the initial enhancement layer pixels according to the prediction information of the first resolution pixels.

[0166] In an optional embodiment, the prediction information of the first resolution includes reference frame information, prediction mode information, and motion vector information, and the first redundancy elimination submodule includes:

[0167] a first determining unit configured to determine, based on the reference frame information, the prediction mode information, and the motion vector information, a pixel region with a prediction residual of 0 and a pixel region with a prediction residual of 0 at the same position in different reference frames;

[0168] The first redundancy elimination unit is configured to set the initial enhanced pixels of the pixel area with zero prediction residuals at different positions of the same reference frame and the pixel area with zero prediction residuals at the same position of different reference frames to 0, thereby obtaining the second enhanced layer pixels that eliminate temporal redundancy and spatial redundancy.

[0169] In an optional embodiment, the difference calculation module 605 further includes:

[0170] The second redundancy elimination submodule is configured to eliminate redundancy of the initial enhancement layer pixels according to matching information between the first resolution pixels and the historical resolution pixels of the screen.

[0171] In an optional embodiment, the second redundancy elimination submodule includes:

[0172] a second determining unit configured to determine, based on matching information between the first resolution pixels and the historical resolution pixels of the screen, a target pixel area of ​​the first resolution pixels that matches the historical resolution pixels;

[0173] The second redundancy elimination unit is configured to set the initial enhanced pixels corresponding to the target pixel area to 0 to obtain the third enhanced layer pixels with the temporal redundancy eliminated.

[0174] In an optional embodiment, the acquisition module 602 includes:

[0175] A feature extraction submodule, configured to extract feature information of pixels at the first resolution, wherein the feature information includes a pixel hash value;

[0176] A first acquisition submodule is configured to obtain a hash value of the historical resolution pixel;

[0177] The second acquisition submodule is configured to acquire matching information between the first resolution pixel and the historical resolution pixel of the screen according to the pixel hash value of the first resolution pixel and the hash value of the historical resolution pixel.

[0178] In an optional embodiment, the feature information further includes pixel gradient and pixel color distribution, and the apparatus further includes:

[0179] The classification module is configured to classify the pixel areas of the first-resolution pixels according to the pixel gradients of the first-resolution pixels and the pixel color distribution of the first-resolution pixels before eliminating the redundancy of the initial enhancement layer pixels, so as to obtain classification information of the first-resolution pixel areas.

[0180] In an optional embodiment, the difference calculation module 605 further includes:

[0181] The third redundancy elimination submodule is configured to eliminate redundancy of pixels in the initial enhancement layer according to classification information of the pixel region of the first resolution.

[0182] In an optional embodiment, the classification information of the first-resolution pixel region includes text-like pixel regions and image pixel regions, and the third redundancy elimination submodule includes:

[0183] a first acquisition unit configured to acquire enhancement layer pixels of the text-like pixel area, and, if the enhancement layer pixels of the text-like pixel area are less than a first preset enhancement layer pixel threshold, set the initial enhancement layer pixels, the second enhancement layer pixels, or the third enhancement layer pixels corresponding to the text-like pixel area to 0, to obtain fourth enhancement layer pixels that eliminate visual redundancy;

[0184] The second acquisition unit is configured to acquire the enhancement layer pixels of the image pixel area, and when the enhancement layer pixels of the image pixel area are less than the second preset enhancement layer pixel threshold, the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the image pixel area are set to 0 to obtain the fourth enhancement layer pixels that eliminate visual redundancy.

[0185] In an optional embodiment, the compression module 606 includes:

[0186] The compression submodule is configured to compress the second enhancement layer pixel or the third enhancement layer pixel or the fourth enhancement layer pixel, the prediction information of the first resolution pixel and the matching information to obtain an enhancement layer code stream.

[0187] FIG7 is a structural block diagram of a device for enhancing the image quality of a video layer stream according to another embodiment of the present disclosure, which is applied to a decoding end. As shown in FIG7 , the device includes:

[0188] The receiving module 701 is configured to receive the enhancement layer stream and the video layer stream sent by the encoding end;

[0189] A first decoding module 702 is configured to decode the video layer code stream to obtain pixels of a second resolution, where the pixels of the second resolution are restored pixels of the pixels of the first resolution;

[0190] A second upsampling module 703 is configured to upsample the second resolution pixels to obtain second original resolution pixels;

[0191] A second decoding module 704 is configured to decode the enhancement layer code stream to obtain prediction information of the enhancement layer pixels of the first resolution and matching information between the first resolution pixels and the historical resolution pixels of the screen;

[0192] A synthesis module 705 is configured to synthesize the second original resolution pixel and the enhancement layer pixel to obtain a third original resolution pixel; the third original resolution pixel is a pixel after the second original resolution pixel is enhanced;

[0193] The pixel enhancement module 706 is configured to enhance the third original resolution pixels according to the prediction information or the matching information of the first resolution pixels to obtain fourth original resolution pixels with enhanced image quality.

[0194] In an optional embodiment, the prediction information includes reference frame information, prediction mode information, and motion vector information. The pixel enhancement module 706 includes:

[0195] a calculation submodule configured to calculate enhanced pixels in an area where the prediction residual is 0 pixels based on the reference frame information, the prediction mode information, and the motion vector information in the prediction information;

[0196] The first pixel enhancement submodule is configured to enhance the third original resolution pixel based on the enhanced pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0197] In an optional embodiment, the pixel enhancement module 706 includes:

[0198] a determination submodule configured to determine a pixel area of ​​historical resolution pixels that matches the first resolution pixels based on matching information between the first resolution pixels and historical resolution pixels of the screen;

[0199] The second pixel enhancement submodule is configured to enhance the third original resolution pixel based on a pixel area of ​​a historical resolution pixel that matches the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0200] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; the above modules are located in different processors in any combination.

[0201] An embodiment of the present disclosure further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0202] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0203] An embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0204] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0205] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0206] Obviously, those skilled in the art should understand that the modules or steps of the present disclosure described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present disclosure is not limited to any particular combination of hardware and software.

[0207] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations of the present disclosure are possible. Any modifications, equivalent substitutions, or improvements made within the principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A method for enhancing the image quality of a video layer code stream, applied to an encoding end, the method comprising: Downsampling the first original resolution pixels of the screen to obtain first resolution pixels; Acquire matching information between the first resolution pixels and the historical resolution pixels of the screen; Performing video encoding on the first-resolution pixel to obtain a video layer code stream corresponding to the first-resolution pixel, a second-resolution pixel, and prediction information of the first-resolution pixel; The second resolution pixel is a restored pixel of the first resolution pixel; Upsampling the second resolution pixels to obtain second original resolution pixels; Performing difference calculation on the first original resolution pixel and the second original resolution pixel to obtain initial enhancement layer pixels, and eliminating redundancy of the initial enhancement layer pixels; Compressing the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels and the matching information to obtain an enhanced layer code stream; The video layer code stream and the enhancement layer code stream are sent to a decoding end, so that the decoding end performs image quality enhancement on the video layer code stream based on the enhancement layer code stream.

2. The method according to claim 1, wherein: The eliminating redundancy of pixels in the initial enhancement layer comprises: Eliminate redundancy of the initial enhancement layer pixels according to the prediction information of the first resolution pixels.

3. The method according to claim 2, wherein: The prediction information of the first resolution includes reference frame information, prediction mode information and motion vector information, and the eliminating redundancy of the initial enhancement layer pixels according to the prediction information of the first resolution pixels includes: Determine, according to the reference frame information, the prediction mode information and the motion vector information, pixel regions where the prediction residuals are 0 at different positions of the same reference frame and pixel regions where the prediction residuals are 0 at the same position of different reference frames; The initial enhanced pixels of the pixel regions with prediction residuals of 0 at different positions of the same reference frame and the pixel regions with prediction residuals of 0 at the same position of different reference frames are set to 0 to obtain the second enhanced layer pixels that eliminate temporal redundancy and spatial redundancy.

4. The method according to claim 2, wherein: The eliminating redundancy of the initial enhancement layer pixels further includes: The redundancy of the initial enhancement layer pixels is eliminated according to the matching information between the first resolution pixels and the historical resolution pixels of the screen.

5. The method according to claim 4, wherein: Eliminating redundancy of pixels of the initial enhancement layer according to matching information between pixels of the first resolution and pixels of the historical resolution of the screen includes: Determining a target pixel area of ​​the first resolution pixel that matches the historical resolution pixel according to matching information between the first resolution pixel and the historical resolution pixel of the screen; The initial enhanced pixels corresponding to the target pixel area are set to 0 to obtain the third enhanced layer pixels that eliminate temporal redundancy.

6. The method according to claim 4, wherein: The obtaining matching information between the first resolution pixel and the historical resolution pixel of the screen includes: Extracting feature information of pixels of the first resolution, the feature information comprising a pixel hash value; Obtaining a hash value of the historical resolution pixel; and obtaining matching information between the first resolution pixel and the historical resolution pixel of the screen according to the pixel hash value of the first resolution pixel and the hash value of the historical resolution pixel.

7. The method according to claim 6, wherein: The feature information also includes pixel gradient and pixel color distribution, and before eliminating the redundancy of the initial enhancement layer pixels, further includes: According to the pixel gradient of the first-resolution pixel and the pixel color distribution of the first-resolution pixel, the pixel area of ​​the first-resolution pixel is classified to obtain classification information of the first-resolution pixel area.

8. The method according to claim 7, wherein: The eliminating redundancy of the initial enhancement layer pixels further includes: Eliminate redundancy of the initial enhancement layer pixels according to the classification information of the first resolution pixel area.

9. The method according to claim 8, wherein: The classification information of the first resolution pixel area includes a text-like pixel area and an image pixel area, and eliminating the redundancy of the initial enhancement layer pixels according to the classification information of the first resolution pixel area includes: Acquire enhancement layer pixels of the text-like pixel area, and when the enhancement layer pixels of the text-like pixel area are less than a first preset enhancement layer pixel threshold, set the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the text-like pixel area to 0, so as to obtain fourth enhancement layer pixels that eliminate visual redundancy; The enhancement layer pixels of the image pixel area are obtained, and when the enhancement layer pixels of the image pixel area are less than the second preset enhancement layer pixel threshold, the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the image pixel area are set to 0 to obtain the fourth enhancement layer pixels that eliminate visual redundancy.

10. The method according to claim 9, wherein: The step of compressing the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels, and the matching information to obtain an enhanced layer code stream includes: The second enhancement layer pixel or the third enhancement layer pixel or the fourth enhancement layer pixel, the prediction information of the first resolution pixel and the matching information are compressed to obtain an enhancement layer code stream.

11. A method for enhancing the image quality of a video layer code stream, applied to a decoding end, the method comprising: Receiving the enhancement layer code stream and the video layer code stream sent by the encoding end; Decoding the video layer code stream to obtain second resolution pixels, where the second resolution pixels are restored pixels of the first resolution pixels; Upsampling the second resolution pixels to obtain second original resolution pixels; Decoding the enhancement layer bitstream to obtain enhancement layer pixels, prediction information of the first resolution pixels, and matching information of the first resolution pixels and historical resolution pixels of the screen; The second original resolution pixel and the enhancement layer pixel are synthesized to obtain a third original resolution pixel; the third original resolution pixel is a pixel after the second original resolution pixel is enhanced; The third original resolution pixel is enhanced according to the prediction information or the matching information of the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

12. The method according to claim 11, wherein: The prediction information includes reference frame information, prediction mode information and motion vector information, and the enhancing the third original resolution pixel according to the prediction information of the first resolution pixel to obtain the fourth original resolution pixel with enhanced image quality includes: Calculate and obtain enhanced pixels in an area where the prediction residual is 0 pixels according to the reference frame information, the prediction mode information and the motion vector information in the prediction information; Based on the enhanced pixels, the third original resolution pixels are enhanced to obtain fourth original resolution pixels with enhanced image quality.

13. The method according to claim 11, wherein: The step of enhancing the third original resolution pixel according to the matching information to obtain a fourth original resolution pixel with enhanced image quality comprises: Determine, according to matching information between the first resolution pixel and the historical resolution pixel of the screen, a pixel area of ​​the historical resolution pixel that matches the first resolution pixel; The third original resolution pixel is enhanced based on a pixel region of a historical resolution pixel matching the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

14. A device for enhancing the image quality of a video layer code stream, applied to an encoding end, the device comprising: A downsampling module, configured to downsample the first original resolution pixels of the screen to obtain first resolution pixels; An acquisition module, configured to acquire matching information between the first resolution pixels and the historical resolution pixels of the screen; a video encoding module, configured to perform video encoding on the first resolution pixel to obtain a video layer code stream corresponding to the first resolution pixel, a second resolution pixel, and prediction information of the first resolution pixel; The second resolution pixel is a restored pixel of the first resolution pixel; A first upsampling module is configured to upsample the second resolution pixels to obtain second original resolution pixels; a difference calculation module, configured to perform difference calculation on pixels of the first original resolution and pixels of the second original resolution to obtain pixels of an initial enhancement layer, and eliminate redundancy of pixels of the initial enhancement layer; A compression module, configured to compress the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels and the matching information to obtain an enhanced layer code stream; The sending module is configured to send the video layer code stream and the enhancement layer code stream to a decoding end, so as to assist the decoding end in enhancing the image quality of the video layer code stream based on the enhancement layer code stream.

15. A device for enhancing the image quality of a video layer code stream, applied to a decoding end, the device comprising: A receiving module, configured to receive an enhancement layer code stream and a video layer code stream sent by an encoding end; A first decoding module is configured to decode the video layer code stream to obtain a second resolution pixel, where the second resolution pixel is a restored pixel of the first resolution pixel; A second upsampling module is configured to upsample the second resolution pixels to obtain second original resolution pixels; A second decoding module is configured to decode the enhancement layer code stream to obtain enhancement layer pixels, prediction information of the first resolution pixels, and matching information of the first resolution pixels and historical resolution pixels of the screen; a synthesis module, configured to synthesize the second original resolution pixel and the enhancement layer pixel to obtain a third original resolution pixel; The third original resolution pixel is a pixel after the second original resolution pixel is enhanced; The pixel enhancement module is configured to enhance the third original resolution pixel according to the prediction information of the first resolution pixel or the matching information to obtain a fourth original resolution pixel with enhanced image quality.

16. A computer-readable storage medium having a computer program stored therein, wherein: When the computer program is executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 10, or implements the steps of the method described in any one of claims 11 to 13.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of claims 1 to 10, or implements the steps of the method described in any one of claims 11 to 13 when executing the computer program.

Images