Image quality enhancement method and device of video layer code stream

Abstract

The embodiment of the invention provides an image quality enhancement method and device for a video layer code stream, and the method comprises the steps: carrying out the video coding of a first-resolution pixel, and obtaining the prediction information of the video layer code stream, a second-resolution pixel, and the first-resolution pixel; performing up-sampling on the second resolution pixel to obtain a second original resolution pixel; calculating the first original resolution pixel and the second original resolution pixel to obtain an initial enhancement layer pixel and eliminating redundancy; compressing the enhancement layer pixels, the prediction information and the matching information of which the redundancy is eliminated to obtain an enhancement layer code stream; and sending the video layer code stream and the enhancement layer code stream to a decoding end so as to enhance the image quality of the video layer code stream based on the enhancement layer code stream. The narrow-band high definition in the cloud desktop / video conference desktop sharing scene is realized, the computing resources of the coding end and the decoding end and the resource occupation of network transmission are saved, and the problem that the super-resolution enhancement scheme in the related technology is not suitable for the cloud desktop / video conference desktop sharing scene with complex and diversified scene characteristics is solved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a method and device for improving the image quality of a video layer code stream. Background Art

[0002] With the popularity and widespread use of high-definition screens, users have gradually put forward demands for high resolution in cloud desktop / video conferencing desktop sharing scenarios. How to ensure that users can smoothly and clearly use high-resolution cloud desktops for office work, design, or video conferencing desktop sharing on the basis of smaller encoding and decoding resources and bandwidth resources has become a difficult problem that cloud desktops and video conferencing urgently need to overcome.

[0003] At present, there are few high-resolution super-resolution enhancement solutions based on cloud desktop / video conference desktop sharing scenarios. Basically, they are super-resolution solutions for ordinary videos such as live video. The technical means used are generally divided into four types. The traditional super-resolution reconstruction methods include pixel interpolation-based reconstruction methods, degradation model-based reconstruction methods, machine learning-based reconstruction methods, and the most commonly used deep learning-based convolutional neural network-based reconstruction methods such as super-resolution reconstruction convolutional neural network (SRCNN) and super-resolution generative adversarial network (SRGAN).

[0004] However, these solutions are not suitable for cloud desktop / video conference desktop sharing scenarios with high definition, high real-time performance, no distortion or noise, and complex and diverse scenarios. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for enhancing the image quality of a video layer code stream, so as to at least solve the problem that the super-resolution enhancement scheme in the related art is not suitable for cloud desktop / video conference desktop sharing scenarios with high definition, high real-time performance, no distortion noise, and complex and diverse scenes.

[0006] According to an embodiment of the present invention, a method for enhancing the image quality of a video layer code stream is provided, which is applied to an encoding end, and the method includes:

[0007] Downsampling the first original resolution pixels of the screen to obtain first resolution pixels;

[0008] Obtaining matching information between the first resolution pixels and the historical resolution pixels of the screen;

[0009] Performing video encoding on the first resolution pixel to obtain a video layer code stream corresponding to the first resolution pixel, a second resolution pixel, and prediction information of the first resolution pixel; the second resolution pixel is a restored pixel of the first resolution pixel;

[0010] Upsampling the second resolution pixels to obtain second original resolution pixels;

[0011] Performing difference calculation on the first original resolution pixel and the second original resolution pixel to obtain initial enhancement layer pixels, and eliminating redundancy of the initial enhancement layer pixels;

[0012] Compressing the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels and the matching information to obtain an enhanced layer code stream;

[0013] The video layer code stream and the enhancement layer code stream are sent to a decoding end, so that the decoding end performs image quality enhancement on the video layer code stream based on the enhancement layer code stream.

[0014] According to another embodiment of the present invention, a method for enhancing the image quality of a video layer code stream is provided, which is applied to a decoding end, and the method includes:

[0015] Receiving the enhancement layer code stream and the video layer code stream sent by the encoding end;

[0016] Decoding the video layer code stream to obtain second resolution pixels, where the second resolution pixels are restored pixels of the first resolution pixels;

[0017] Upsampling the second resolution pixels to obtain second original resolution pixels;

[0018] Decoding the enhancement layer bitstream to obtain enhancement layer pixels, prediction information of the first resolution pixels, and matching information of the first resolution pixels and historical resolution pixels of the screen;

[0019] The second original resolution pixel and the enhancement layer pixel are synthesized to obtain a third original resolution pixel; the third original resolution pixel is a pixel after the second original resolution pixel is enhanced;

[0020] The third original resolution pixel is enhanced according to the prediction information or the matching information of the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0021] According to another embodiment of the present invention, a video layer code stream quality enhancement device is provided, which is applied to an encoding end and includes:

[0022] A downsampling module, used for downsampling the first original resolution pixels of the screen to obtain first resolution pixels;

[0023] An acquisition module, used to acquire matching information between the first resolution pixels and the historical resolution pixels of the screen;

[0024] a video encoding module, configured to perform video encoding on the first resolution pixel to obtain a video layer code stream corresponding to the first resolution pixel, a second resolution pixel, and prediction information of the first resolution pixel; the second resolution pixel is a restored pixel of the first resolution pixel;

[0025] A first up-sampling module, configured to up-sample the second resolution pixels to obtain second original resolution pixels;

[0026] a difference calculation module, configured to perform difference calculation on pixels of the first original resolution and pixels of the second original resolution to obtain pixels of an initial enhancement layer, and eliminate redundancy of pixels of the initial enhancement layer;

[0027] A compression module, used for compressing the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels and the matching information to obtain an enhanced layer code stream;

[0028] The sending module is used to send the video layer code stream and the enhancement layer code stream to the decoding end, so as to assist the decoding end to enhance the image quality of the video layer code stream based on the enhancement layer code stream.

[0029] According to another embodiment of the present invention, a video layer code stream quality enhancement device is provided, which is applied to a decoding end and includes:

[0030] A receiving module, used for receiving the enhancement layer code stream and the video layer code stream sent by the encoding end;

[0031] A first decoding module, configured to decode the video layer code stream to obtain a second resolution pixel, where the second resolution pixel is a restored pixel of the first resolution pixel;

[0032] A second upsampling module, configured to upsample the second resolution pixels to obtain second original resolution pixels;

[0033] A second decoding module is used to decode the enhancement layer code stream to obtain prediction information of the enhancement layer pixels of the first resolution pixels and matching information of the first resolution pixels and historical resolution pixels of the screen;

[0034] A synthesis module, used for synthesizing the second original resolution pixel and the enhancement layer pixel to obtain a third original resolution pixel; the third original resolution pixel is a pixel after the second original resolution pixel is enhanced;

[0035] The pixel enhancement module is used to enhance the third original resolution pixel according to the prediction information of the first resolution pixel or the matching information to obtain a fourth original resolution pixel with enhanced image quality.

[0036] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0037] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0038] The embodiment of the present invention performs video encoding on the first resolution pixels after downsampling the first original resolution pixels of the screen, obtains the video layer code stream corresponding to the first resolution pixels, the restored pixels of the first resolution pixels, and the prediction information of the first resolution pixels, and simultaneously obtains the matching information between the first resolution pixels and the historical resolution pixels of the screen, upsamples the restored pixels of the first resolution pixels to obtain the second original resolution pixels, and calculates the difference between the first original resolution pixels and the second original resolution pixels to obtain the initial enhancement layer pixels, and then eliminates the redundancy of the initial enhancement layer pixels, and compresses and encodes the enhanced layer pixels, prediction information, and matching information after eliminating the redundancy and transmits them to the decoding end to assist the corresponding video layer code stream in image quality enhancement. Thus, the narrowband high-definition effect in the cloud desktop / video conference desktop sharing scenario is achieved, the computing resources of the encoding end and the decoding end and the resource occupation of the network transmission are saved, and the problem that the super-resolution enhancement scheme in the related art is not suitable for the cloud desktop / video conference desktop sharing scenario with high definition, high real-time performance, no distortion noise, and complex and diverse scenes is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a hardware structure block diagram of a mobile terminal according to a method for improving the image quality of a video layer code stream according to an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of a network architecture of an encoding end according to an embodiment of the present invention;

[0041] Figure 3 is a flow chart of the image quality of a video layer code stream according to an embodiment of the present invention;

[0042] Figure 4 is a schematic diagram of a network architecture of a decoding end according to an embodiment of the present invention;

[0043] Figure 5 is a flowchart of the image quality of a video layer code stream according to another embodiment of the present invention;

[0044] Figure 6 is a structural block diagram of a device for enhancing the image quality of a video layer code stream according to an embodiment of the present invention;

[0045] Figure 7 It is a structural block diagram of a device for enhancing the image quality of a video layer code stream according to another embodiment of the present invention. DETAILED DESCRIPTION

[0046] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0048] The super-resolution enhancement schemes in the related art generally include reconstruction methods based on pixel interpolation, reconstruction methods based on degradation models, reconstruction methods based on machine learning, and reconstruction methods based on deep learning of convolutional neural networks.

[0049] However, reconstruction based on typical interpolation algorithms such as neighborhood interpolation, bilinear interpolation, and bicubic interpolation has blurring and aliasing problems, and is not suitable for high-definition scenarios such as cloud desktops / video conference desktop sharing; super-resolution reconstruction based on degradation models is generally suitable for specific scenarios such as low-resolution images introduced by camera blur, distortion, noise characteristics, etc., but is not suitable for cloud desktops / video conference desktop sharing, where the original high-definition downsampling introduces image quality degradation; machine learning based on methods such as image self-similarity and sparse representation rely on trained samples and learning dictionaries, and its wide applicability in scenarios remains to be tested; and the deep learning methods based on convolutional neural networks, which are currently being studied more, can achieve better enhancement effects based on online training and feedback mechanisms, but cannot meet the real-time requirements of cloud desktops / video conference desktop sharing scenarios.

[0050] The video compression algorithm for cloud desktop / video conference desktop sharing high-resolution scenes - Low Complexity Enhancement Video Coding (LCEVC) is based on two downsampling to upsample the low-resolution reconstructed frame and calculate the parameters with the original image of the layer, and then encode the residual, which is passed to the decoding end as an enhanced bitstream to assist the downsampled bitstream for differential reconstruction, with good real-time performance and image enhancement effect. However, LCEVC does not analyze the feature information and prediction of the original pixels, and the same treatment of all pixels will result in large bandwidth and CPU encoding consumption.

[0051] The technical concept of the embodiment of the present invention is: since cloud desktop / video conference desktop sharing has the characteristics of high definition, high real-time performance, no distortion noise, complex and diverse scenes, etc., based on the analysis of the original resolution pixel characteristics of cloud desktop / video conference desktop sharing, a de-redundant enhancement layer code stream is added to the low-resolution code stream, thereby achieving the effect of super-resolution enhancement at the decoding end.

[0052] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal according to a method for improving the image quality of a video layer code stream according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0053] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image quality method of the video layer code stream in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0054] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0055] Figure 2 is a schematic diagram of the network architecture of the encoding end according to an embodiment of the present invention. The embodiment of the present application can be run on Figure 2 In the network architecture shown in Figure 2 As shown in the figure, the network architecture includes: downsampling module, video encoding module, upsampling module, feature information calculation module, enhancement information calculation module, redundant information elimination module, self-developed encoding module. The functions of each module are as follows:

[0056] Downsampling module: It is used to take the data of the original resolution size of the cloud desktop / video conference desktop sharing, which has been transcoded into the video encoding data format, as input data. The input data can be downsampled according to actual needs to obtain output data with a smaller resolution after downsampling.

[0057] Video encoding module: connected to the downsampling module, used to perform video encoding on the above-mentioned down-sampled output data according to the data format, and output the video layer code stream, the restored frame corresponding to the above-mentioned output data, and the prediction information corresponding to the above-mentioned output data.

[0058] Upsampling module: connected to the video encoding module, used to perform upsampling operation on the above-mentioned restored frames, wherein upsampling is an operation opposite to downsampling, which is an operation of restoring low-resolution data to the original high-resolution through a certain method.

[0059] Feature information calculation module: connected to the downsampling module, used to take the downsampled pixel data as input, analyze the pixel features, and output the feature information of the downsampled pixel data. Among them, the pixel features may include unique identifiers for pixel gradient, pixel color distribution, pixel hash value, etc. These pixel features can be used to enhance the sparse processing of pixels. In addition, the feature information calculation module can divide the pixel area type according to the pixel gradient and pixel color distribution, for example, it can be divided into text-like pixel areas and image pixel areas; hash matching can be performed from historical downsampled screen pixels according to the pixel hash value to obtain matching information with the historical downsampled screen pixels.

[0060] Enhanced information calculation module: connected to the upsampling module and the original resolution pixel input, used to perform difference calculation on the original resolution pixels and the restored data after upsampling to obtain the initial enhanced layer pixels.

[0061] The redundant information elimination module is respectively connected to the video encoding module, the feature information module, and the enhancement information calculation module, and is used to eliminate the spatial redundancy, temporal redundancy and visual redundancy in the initial enhancement layer pixels according to the prediction information output by the video encoding module and the feature information output by the feature information module, so as to obtain a sparse enhancement layer pixel of the original resolution size.

[0062] The self-developed encoding module is connected to the redundant information elimination module, and is used to compress the enhanced layer pixels after the redundant information is removed, and to encode and compress the matching information and the prediction information.

[0063] In this embodiment, a method for enhancing the image quality of a video layer bit stream running on a network architecture of an encoding end is provided. Figure 3 is a flowchart of image quality enhancement of a video layer code stream according to an embodiment of the present invention, such as Figure 3 As shown, the process includes the following steps:

[0064] Step S301 , down-sampling the first original resolution pixels of the screen to obtain first resolution pixels.

[0065] As an example, downsampling the first original resolution pixel may obtain a first resolution pixel lower than the first original resolution pixel, that is, a first low-resolution pixel.

[0066] For example, a changing screen image can be captured in a cloud desktop / video conference desktop sharing system, the captured screen image can be converted into pixels of a first original resolution size corresponding to video encoding, and then downsampled based on the pixels of the first original resolution size to obtain pixels of a first low resolution.

[0067] It should be noted that downsampling only changes the size of the pixel set, but does not change the format of the pixel set.

[0068] Step S302: Obtain matching information between the first resolution pixels and the historical resolution pixels of the screen.

[0069] As an example, the encoding end may obtain matching information between the first resolution pixels and the historical resolution pixels of the screen.

[0070] As an example, the matching information between the first resolution pixel and the historical resolution pixel of the screen may include a matching relationship between the first resolution pixel area and the historical resolution pixel area.

[0071] In an optional embodiment, obtaining matching information between the first resolution pixel and the historical resolution pixel of the screen includes: extracting feature information of the first resolution pixel, the feature information including a pixel hash value; obtaining a hash value of the historical resolution pixel; and obtaining matching information between the first resolution pixel and the historical resolution pixel of the screen based on the pixel hash value of the first resolution pixel and the hash value of the historical resolution pixel.

[0072] As an example, feature information of a first resolution pixel may be extracted, and a pixel hash value in the feature information of the first resolution pixel may be matched with a hash value of a historical resolution pixel to obtain matching information between the first resolution pixel and the historical resolution pixel.

[0073] As an example, the matching relationship between the first resolution pixel region and the historical resolution pixel region may be matched according to a pixel hash value of the first resolution pixel region and a hash value of the historical resolution pixel region.

[0074] For example, the hash feature value of the first resolution pixel area can be calculated based on the feature information of the first resolution pixel, and hash matching can be performed from the historically downsampled screen pixels, or pixel matching can be performed based on the cloud desktop / video conference desktop sharing operation information such as keyboard and mouse, so as to obtain the matching relationship between the first resolution pixel area and the historical resolution pixel area of ​​the screen.

[0075] In an optional embodiment, the feature information also includes pixel gradient and pixel color distribution. Before eliminating the redundancy of the initial enhancement layer pixels, it also includes: classifying the first resolution pixel area according to the pixel gradient of the first resolution pixel and the pixel color distribution of the first resolution pixel to obtain classification information of the first resolution pixel area.

[0076] As an example, the feature information may also include pixel gradient and pixel color distribution. Before eliminating the redundancy of the initial enhancement layer pixels, the pixel gradient and pixel color distribution of the first resolution pixels may be pre-extracted. The first resolution pixel area may be classified according to the pixel gradient and pixel color distribution of the first resolution pixels to obtain classification information of the first resolution pixel area.

[0077] For example, the first resolution pixel region may be divided into a text-like pixel region and an image pixel region according to the pixel gradient and pixel color distribution of the first resolution pixel.

[0078] Step S303, performing video encoding on the first resolution pixel to obtain a video layer code stream corresponding to the first resolution pixel, a second resolution pixel, and prediction information of the first resolution pixel; the second resolution pixel is a restored pixel of the first resolution pixel.

[0079] As an example, the first resolution pixel may be video encoded to obtain the video layer code stream corresponding to the first resolution pixel, the second resolution pixel and the prediction information of the first resolution pixel, wherein the second resolution pixel is the restored pixel of the first resolution pixel.

[0080] As an example, based on the first resolution pixels, conventional video encoding may be performed, and the specific encoding specification may be determined according to actual conditions, and the embodiment of the present invention does not impose any limitation thereto.

[0081] Step S304, upsampling the second resolution pixels to obtain second original resolution pixels;

[0082] As an example, the second resolution pixel is a restored pixel of the first resolution pixel, and is also a low-resolution pixel.

[0083] As an example, an upsampling operation corresponding to downsampling may be performed based on the second resolution pixels, and the low-resolution second resolution pixels may be upsampled to pixels of the second original resolution size.

[0084] Step S305 , performing difference calculation on the first original resolution pixel and the second original resolution pixel to obtain initial enhancement layer pixels, and eliminating redundancy of the initial enhancement layer pixels.

[0085] As an example, a difference calculation may be performed between the second original resolution pixel and the first original resolution pixel before downsampling to obtain the initial enhancement layer pixel.

[0086] As an example, redundancy of pixels in the initial enhancement layer may be eliminated to reduce resource usage at the encoding end.

[0087] As an example, redundancy of enhancement layer pixels may be eliminated based on spatial correlation (pixel references at different positions in the same frame) and temporal correlation (pixel references at the same position in different frames) between pixels.

[0088] Ways to eliminate the redundancy of pixels in the initial enhancement layer may include the following:

[0089] In an optional embodiment, redundancy of the initial enhancement layer pixels is eliminated based on prediction information of the first resolution pixels.

[0090] In an optional embodiment, redundancy of the initial enhancement layer pixels is eliminated based on matching information between the first resolution pixels and historical resolution pixels of the screen.

[0091] In an optional embodiment, redundancy of the initial enhancement layer pixels is eliminated according to classification information of the first resolution pixel area.

[0092] As an example, the redundancy of the enhancement layer pixels may include but is not limited to temporal redundancy, spatial redundancy, and visual redundancy. In view of different types of redundancy of the enhancement layer pixels, the embodiments of the present invention provide multiple ways to eliminate the redundancy of the initial enhancement layer pixels. The multiple ways to eliminate the redundancy of the initial enhancement layer pixels can be freely combined according to actual conditions to more comprehensively eliminate the redundancy of the initial enhancement layer pixels.

[0093] The embodiment of the present invention does not limit the combination of multiple methods for eliminating redundancy of initial enhancement layer pixels.

[0094] For example, the redundancy of the initial enhancement layer pixels may be eliminated based on the prediction information of the first resolution pixels, or the matching information between the first resolution pixels and the historical resolution pixels of the screen, or the classification information of the first resolution pixel area. The redundancy of the initial enhancement layer pixels may be eliminated based on the prediction information of the first resolution pixels, and the redundancy of the initial enhancement layer pixels may be further eliminated based on the classification information of the first resolution pixel area. The redundancy of the initial enhancement layer pixels may be eliminated based on the matching information between the first resolution pixels and the historical resolution pixels of the screen, and the redundancy of the initial enhancement layer pixels may be further eliminated based on the classification information of the first resolution pixel area. The redundancy of the initial enhancement layer pixels may be eliminated based on the prediction information of the first resolution pixels, and the redundancy of the initial enhancement layer pixels may be eliminated based on the matching information between the first resolution pixels and the historical resolution pixels of the screen, and the redundancy of the initial enhancement layer pixels may be further eliminated based on the classification information of the first resolution pixel area.

[0095] It should be noted that the embodiment of the present invention does not limit the order of implementing each method of eliminating the redundancy of the initial enhancement layer pixels in the above-mentioned combination method.

[0096] It can be seen from the above optional embodiments that there are multiple ways to eliminate the redundancy of the initial enhancement layer pixels. To facilitate the understanding of those skilled in the art, the following describes the above specific implementation methods of eliminating the redundancy of the initial enhancement layer pixels one by one:

[0097] In an optional embodiment, the prediction information of the first resolution includes reference frame information, prediction mode information and motion vector information, and eliminating redundancy of the initial enhancement layer pixels according to the prediction information of the first resolution pixels includes:

[0098] Determine, according to the reference frame information, the prediction mode information and the motion vector information, pixel regions where the prediction residuals are 0 at different positions of the same reference frame and pixel regions where the prediction residuals are 0 at the same position of different reference frames;

[0099] The initial enhanced pixels of the pixel regions with prediction residuals of 0 at different positions of the same reference frame and the pixel regions with prediction residuals of 0 at the same position of different reference frames are set to 0 to obtain the second enhanced layer pixels that eliminate temporal redundancy and spatial redundancy.

[0100] As an example, pixel areas with prediction residuals of 0 at different positions of the same reference frame and pixel areas with prediction residuals of 0 at the same position of different reference frames can be determined based on reference frame information, prediction mode information and motion vector information in the prediction information of the first resolution.

[0101] As an example, the initial enhancement layer pixels of the pixel area with a prediction residual of 0 at different positions in the same reference frame can be cleared to eliminate the temporal redundancy of the initial enhancement pixels; the initial enhancement layer pixels of the pixel area with a prediction residual of 0 at the same position in different reference frames can be set to 0 to eliminate the spatial redundancy of the initial enhancement pixels.

[0102] In an optional embodiment, eliminating redundancy of pixels of the initial enhancement layer according to matching information between pixels of the first resolution and pixels of the historical resolution of the screen includes:

[0103] Determining a target pixel area of ​​the first resolution pixel that matches the historical resolution pixel according to matching information between the first resolution pixel and the historical resolution pixel of the screen;

[0104] The initial enhanced pixels corresponding to the target pixel area are set to 0 to obtain the third enhanced layer pixels that eliminate temporal redundancy.

[0105] As an example, the matching information between the first resolution pixel and the historical resolution pixel of the screen may include the matching relationship between the first resolution pixel area and the historical resolution pixel area, and the target pixel area of ​​the first resolution pixel that matches the historical resolution pixel can be determined based on the matching relationship between the first resolution pixel area and the historical resolution pixel area.

[0106] As an example, the initial enhanced pixels corresponding to the target pixel area may be set to 0 to eliminate the time redundancy of the initial enhanced pixels.

[0107] In an optional embodiment, the classification information of the first-resolution pixel region includes a text-like pixel region and an image pixel region, and eliminating redundancy of the initial enhancement layer pixels according to the classification information of the first-resolution pixel region includes:

[0108] Acquire enhancement layer pixels of the text-like pixel area, and when the enhancement layer pixels of the text-like pixel area are less than a first preset enhancement layer pixel threshold, set the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the text-like pixel area to 0, so as to obtain fourth enhancement layer pixels that eliminate visual redundancy;

[0109] The enhancement layer pixels of the image pixel area are obtained, and when the enhancement layer pixels of the image pixel area are less than the second preset enhancement layer pixel threshold, the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the image pixel area are set to 0 to obtain the fourth enhancement layer pixels that eliminate visual redundancy.

[0110] As an example, the first resolution pixel area may be divided into a text-like pixel area and an image pixel area.

[0111] As an example, since text-like areas contain more contour details and are highly sensitive to pixel loss, while natural image areas have a low perception of pixel loss except for internal detail textures, different pixel loss thresholds can be set in advance for the two types of areas to eliminate insensitive enhancement information that is less than the pixel threshold. Generally, the pixel loss threshold of text-like areas is set smaller than the pixel loss threshold of image areas.

[0112] As an example, the pixel loss threshold of the text-like area can be set as the first preset enhancement layer pixel threshold, and the pixel loss threshold of the image area can be set as the second preset enhancement layer pixel threshold. After respectively acquiring the enhancement layer pixels of the text-like pixel area and the enhancement layer pixels of the image pixel area, the enhancement layer pixels of the text-like pixel area can be compared with the first preset enhancement layer pixel threshold, and the enhancement layer pixels of the image pixel area can be compared with the second preset enhancement layer pixel threshold.

[0113] As an example, when the enhancement layer pixels in the text-like pixel area are smaller than the first preset enhancement layer pixel threshold, the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the text-like pixel area may be set to 0 to obtain the fourth enhancement layer pixels that eliminate visual redundancy; when the enhancement layer pixels in the image pixel area are smaller than the second preset enhancement layer pixel threshold, the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the image pixel area may be set to 0 to obtain the fourth enhancement layer pixels that eliminate visual redundancy.

[0114] Step S306: compressing the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels, and the matching information to obtain an enhanced layer code stream.

[0115] As an example, the enhanced layer pixels after redundancy elimination, the prediction information of the first resolution pixels, and the matching information between the first resolution pixels and the screen historical resolution pixels may be compressed to obtain an enhanced layer code stream.

[0116] In an optional embodiment, the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels and the compression to obtain the enhanced layer code stream include: compressing the second enhancement layer pixels or the third enhancement layer pixels or the fourth enhancement layer pixels, the prediction information of the first resolution pixels and the matching information to obtain the enhanced layer code stream.

[0117] As an example, since the various methods of eliminating the redundancy of the initial enhancement layer pixels in the embodiments of the present invention can be freely combined according to actual conditions, if the second enhancement layer pixels are obtained after the combination, the prediction information of the second enhancement layer pixels and the first resolution pixels, and the matching information of the first resolution pixels and the historical resolution pixels of the screen can be compressed to obtain the enhancement layer code stream; if the third enhancement layer pixels are obtained after the combination, the prediction information of the third enhancement layer pixels and the first resolution pixels, and the matching information of the first resolution pixels and the historical resolution pixels of the screen can be compressed to obtain the enhancement layer code stream; if the fourth enhancement layer pixels are obtained after the combination, the prediction information of the fourth enhancement layer pixels and the first resolution pixels, and the matching information of the first resolution pixels and the historical resolution pixels of the screen can be compressed to obtain the enhancement layer code stream.

[0118] Step S307: sending the video layer code stream and the enhancement layer code stream to a decoding end, so that the decoding end performs image quality enhancement on the video layer code stream based on the enhancement layer code stream.

[0119] As an example, the encoder may send the video layer code stream and the enhancement layer code stream to the decoder, so that the decoder enhances the image quality of the video layer code stream based on the enhancement layer code stream.

[0120] In an embodiment of the present invention, the embodiment of the present invention performs video encoding on the first resolution pixels after downsampling the first original resolution pixels of the screen, obtains the video layer code stream corresponding to the first resolution pixels, the restored pixels of the first resolution pixels, and the prediction information of the first resolution pixels, and simultaneously obtains the matching information between the first resolution pixels and the historical resolution pixels of the screen, upsamples the restored pixels of the first resolution pixels to obtain the second original resolution pixels, and calculates the difference between the first original resolution pixels and the second original resolution pixels to obtain the initial enhancement layer pixels, and then eliminates the redundancy of the initial enhancement layer pixels, and compresses and encodes the enhanced layer pixels, prediction information, and matching information after eliminating the redundancy and transmits them to the decoding end to assist the corresponding video layer code stream in image quality enhancement. Thus, a narrowband high-definition effect in the cloud desktop / video conference desktop sharing scenario is achieved, the computing resources of the encoding end and the decoding end and the resource occupation of the network transmission are saved, and the problem that the super-resolution enhancement scheme in the related art is not suitable for the cloud desktop / video conference desktop sharing scenario with high definition, high real-time performance, no distortion noise, and complex and diverse scenes is solved.

[0121] Figure 4 is a schematic diagram of the network architecture of the decoding end according to an embodiment of the present invention. The embodiment of the present application can be operated on Figure 4 In the network architecture shown in Figure 4 As shown in the figure, the network architecture includes: video decoding module, upsampling module, self-developed decoding module, synthesis restoration module, and the functions of each module are as follows:

[0122] Video decoding module: used to receive the video layer code stream of low-resolution pixels sent by the encoding end, and decode or decompress the video layer code stream to obtain a low-resolution restored pixel.

[0123] Up-sampling module: connected to the video decoding module, used to receive the decoded low-resolution restored pixels, up-sample the decoded low-resolution restored pixels, and restore the low-resolution restored pixels to pixels of the original resolution size.

[0124] Self-developed decoding module: used to receive the enhancement layer code stream transmitted by the encoder, and decode the enhancement layer code stream according to the algorithm corresponding to the encoder to obtain the enhancement layer pixels of the original resolution size, the prediction information of the first resolution pixels, and the matching information between the first resolution pixels and the historical resolution pixels of the screen;

[0125] Synthesis and restoration module: It is connected to the upsampling module and the self-developed decoding module respectively, and fills the enhancement layer code stream of the original resolution into the pixels of the original resolution size after upsampling to obtain an enhanced frame. At the same time, it combines the prediction information of the first resolution pixels or the matching information of the first resolution pixels and the historical resolution pixels of the screen, and further fills them into the current enhancement frame to obtain restored pixels with enhanced image quality.

[0126] In this embodiment, a method for enhancing the image quality of a video layer code stream running on a network architecture of a decoding end is provided. Figure 5 is a flowchart of enhancing the image quality of a video layer code stream according to another embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0127] Step S501: receiving an enhancement layer bitstream and a video layer bitstream sent by an encoding end.

[0128] The decoding end in the embodiment of the present invention can receive the enhancement layer code stream and the video layer code stream sent by the encoding end.

[0129] Step S502: decode the video layer code stream to obtain second resolution pixels, where the second resolution pixels are restored pixels of the first resolution pixels.

[0130] As an example, the video layer code stream may be decoded or decompressed, which process corresponds to the encoding process at the encoding end, and restored pixels of the first resolution pixels are obtained after decoding.

[0131] The first resolution pixel is a low-resolution pixel lower than the original resolution of the screen, and the second resolution pixel is a restored pixel of the first resolution pixel, which is a low-resolution restored pixel lower than the original resolution of the screen.

[0132] Step S503: up-sample the second resolution pixels to obtain second original resolution pixels.

[0133] As an example, the second resolution pixels may be upsampled to restore the second resolution pixels to the original screen resolution size to obtain second original resolution pixels, where the second original resolution pixels may be restored pixels of the first original resolution.

[0134] Step S504: decode the enhancement layer code stream to obtain enhancement layer pixels, prediction information of the first resolution pixels, and matching information between the first resolution pixels and historical resolution pixels of the screen.

[0135] As an example, the enhancement layer code stream may be decoded to obtain enhancement layer pixels, prediction information of first resolution pixels, and matching information of first resolution pixels and historical resolution pixels of the screen.

[0136] Step S505: synthesize the second original resolution pixel and the enhancement layer pixel to obtain a third original resolution pixel; the third original resolution pixel is a pixel after the second original resolution pixel is enhanced.

[0137] As an example, the second original resolution pixel and the enhancement layer pixel may be synthesized to obtain an enhanced third original resolution pixel.

[0138] Step S506: enhancing the third original resolution pixel according to the prediction information or the matching information of the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0139] As an example, an embodiment of the present invention may enhance the third original resolution pixel according to the prediction information of the first resolution pixel to obtain the fourth original resolution pixel with enhanced image quality; may also enhance the third original resolution pixel according to the matching information of the first resolution pixel and the historical resolution pixel of the screen to obtain the fourth original resolution pixel with enhanced image quality; may also enhance the third original resolution pixel according to both the prediction information of the first resolution pixel and the matching information of the first resolution pixel and the historical resolution pixel of the screen to obtain the fourth original resolution pixel with enhanced image quality.

[0140] It should be noted that, when the embodiment of the present invention enhances the third original resolution pixel according to the prediction information of the first resolution pixel and the matching information between the first resolution pixel and the historical resolution pixel of the screen, the order of enhancement is not limited. The third original resolution pixel may be enhanced according to the prediction information of the first resolution pixel first, and then the third original resolution pixel may be enhanced according to the matching information between the first resolution pixel and the historical resolution pixel of the screen; or the third original resolution pixel may be enhanced according to the matching information between the first resolution pixel and the historical resolution pixel of the screen, and then the third original resolution pixel may be enhanced according to the prediction information of the first resolution pixel.

[0141] The enhancement process of the third original resolution pixel is described in detail below:

[0142] In an optional embodiment, the prediction information includes reference frame information, prediction mode information and motion vector information, and the enhancing the third original resolution pixel according to the prediction information of the first resolution pixel to obtain the fourth original resolution pixel with enhanced image quality includes:

[0143] Calculate and obtain enhanced pixels in an area where the prediction residual is 0 pixels according to the reference frame information, the prediction mode information and the motion vector information in the prediction information;

[0144] Based on the enhanced pixels, the third original resolution pixels are enhanced to obtain fourth original resolution pixels with enhanced image quality.

[0145] As an example, the enhanced pixels in the area where the prediction residual is 0 pixels can be calculated based on the reference frame information, prediction method information and motion vector information in the prediction information of the first resolution pixels. The third original resolution pixels can be enhanced based on the enhanced pixels in the area where the prediction residual is 0 pixels to obtain fourth original resolution pixels with enhanced image quality.

[0146] In an optional embodiment, enhancing the third original resolution pixel according to the matching information to obtain a fourth original resolution pixel with enhanced image quality includes:

[0147] Determine, according to matching information between the first resolution pixel and the historical resolution pixel of the screen, a pixel area of ​​the historical resolution pixel that matches the first resolution pixel;

[0148] The third original resolution pixel is enhanced based on a pixel region of a historical resolution pixel matching the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0149] As an example, the matching information between the first resolution pixel and the historical resolution pixel of the screen may include the matching relationship between the first resolution pixel area and the historical resolution pixel area, and the pixel area of ​​the historical resolution pixel that matches the first resolution pixel can be determined based on the matching relationship between the first resolution pixel area and the historical resolution pixel area.

[0150] As an example, the third original resolution pixel may be enhanced based on a pixel region of a historical resolution pixel that matches the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0151] In this embodiment, the decoding end receives the enhancement layer code stream and the video layer code stream sent by the encoding end, and decodes the enhancement layer code stream and the video layer code stream respectively, synthesizes the second original resolution pixels obtained by decoding and upsampling the video layer code stream and the enhancement layer pixels obtained by decoding the enhancement layer code stream, to obtain the third original resolution pixels, and then further enhances the third original resolution pixels based on the prediction information or matching information obtained by decoding the enhancement layer code stream, to obtain the fourth enhancement layer pixels with enhanced image quality. Thus, the narrowband high-definition effect in the cloud desktop / video conference desktop sharing scenario is achieved, the computing resources of the encoding end and the decoding end and the resource occupation of the network transmission are saved, and the problem that the super-resolution enhancement scheme in the related technology is not suitable for the cloud desktop / video conference desktop sharing scenario with high definition, high real-time performance, no distortion noise, and complex and diverse scenes is solved.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0153] In this embodiment, a video layer code stream image quality enhancement device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0154] Figure 6is a structural block diagram of a video layer code stream image quality enhancement device according to an embodiment of the present invention, which is applied to an encoding end, such as Figure 6 As shown, the device comprises:

[0155] A downsampling module 601 is used to downsample the first original resolution pixels of the screen to obtain first resolution pixels;

[0156] An acquisition module 602 is used to acquire matching information between the first resolution pixels and the historical resolution pixels of the screen;

[0157] The video encoding module 603 is used to perform video encoding on the first resolution pixel to obtain a video layer code stream corresponding to the first resolution pixel, a second resolution pixel, and prediction information of the first resolution pixel; the second resolution pixel is a restored pixel of the first resolution pixel;

[0158] A first up-sampling module 604, configured to up-sample the second resolution pixels to obtain second original resolution pixels;

[0159] A difference calculation module 605 is used to perform difference calculation on the pixels of the first original resolution and the pixels of the second original resolution to obtain initial enhancement layer pixels, and eliminate redundancy of the initial enhancement layer pixels;

[0160] A compression module 606, configured to compress the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels and the matching information to obtain an enhanced layer bitstream;

[0161] The sending module 607 is used to send the video layer code stream and the enhancement layer code stream to the decoding end, so as to assist the decoding end to enhance the image quality of the video layer code stream based on the enhancement layer code stream.

[0162] In an optional embodiment, the difference calculation module 605 includes:

[0163] The first redundancy elimination submodule is used to eliminate the redundancy of the initial enhancement layer pixels according to the prediction information of the first resolution pixels.

[0164] In an optional embodiment, the prediction information of the first resolution includes reference frame information, prediction mode information and motion vector information, and the first redundancy elimination submodule includes:

[0165] A first determining unit, configured to determine, according to the reference frame information, the prediction mode information and the motion vector information, a pixel region where the prediction residual is 0 and a pixel region where the prediction residual is 0 at the same position of different reference frames;

[0166] The first redundancy elimination unit is used to set the initial enhanced pixels of the pixel area with zero prediction residual at different positions of the same reference frame and the pixel area with zero prediction residual at the same position of different reference frames to 0, so as to obtain the second enhanced layer pixels that eliminate temporal redundancy and spatial redundancy.

[0167] In an optional embodiment, the difference calculation module 605 further includes:

[0168] The second redundancy elimination submodule eliminates redundancy of the initial enhancement layer pixels according to matching information between the first resolution pixels and the historical resolution pixels of the screen.

[0169] In an optional embodiment, the second redundancy elimination submodule includes:

[0170] A second determining unit, configured to determine a target pixel area of ​​first resolution pixels matching the historical resolution pixels according to matching information between the first resolution pixels and the historical resolution pixels of the screen;

[0171] The second redundancy elimination unit is used to set the initial enhanced pixels corresponding to the target pixel area to 0, so as to obtain the third enhanced layer pixels with the temporal redundancy eliminated.

[0172] In an optional embodiment, the acquisition module 602 includes:

[0173] A feature extraction submodule, configured to extract feature information of pixels of the first resolution, wherein the feature information includes a pixel hash value;

[0174] A first acquisition submodule, used to acquire the hash value of the historical resolution pixel;

[0175] The second acquisition submodule is used to acquire matching information between the first resolution pixel and the historical resolution pixel of the screen according to the pixel hash value of the first resolution pixel and the hash value of the historical resolution pixel.

[0176] In an optional embodiment, the feature information further includes pixel gradient and pixel color distribution, and the device further includes:

[0177] A classification module is used to classify the pixel areas of the first-resolution pixels according to the pixel gradients of the first-resolution pixels and the pixel color distribution of the first-resolution pixels before eliminating the redundancy of the initial enhancement layer pixels, so as to obtain classification information of the first-resolution pixel areas.

[0178] In an optional embodiment, the difference calculation module 605 further includes:

[0179] The third redundancy elimination submodule is used to eliminate the redundancy of the initial enhancement layer pixels according to the classification information of the first resolution pixel area.

[0180] In an optional embodiment, the classification information of the first resolution pixel area includes a text-like pixel area and an image pixel area, and the third redundancy elimination submodule includes:

[0181] A first acquisition unit is used to acquire enhancement layer pixels of the text-like pixel area, and when the enhancement layer pixels of the text-like pixel area are less than a first preset enhancement layer pixel threshold, set the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the text-like pixel area to 0, so as to obtain fourth enhancement layer pixels that eliminate visual redundancy;

[0182] The second acquisition unit is used to acquire the enhancement layer pixels of the image pixel area, and when the enhancement layer pixels of the image pixel area are less than the second preset enhancement layer pixel threshold, the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the image pixel area are set to 0 to obtain the fourth enhancement layer pixels that eliminate visual redundancy.

[0183] In an optional embodiment, the compression module 606 includes:

[0184] The compression submodule is used to compress the prediction information of the second enhancement layer pixel or the third enhancement layer pixel or the fourth enhancement layer pixel, the first resolution pixel and the matching information to obtain an enhancement layer code stream.

[0185] Figure 7 is a structural block diagram of a video layer code stream image quality enhancement device according to another embodiment of the present invention, which is applied to a decoding end, such as Figure 7 As shown, the device comprises:

[0186] The receiving module 701 is used to receive the enhancement layer code stream and the video layer code stream sent by the encoding end;

[0187] A first decoding module 702 is used to decode the video layer code stream to obtain a second resolution pixel, where the second resolution pixel is a restored pixel of the first resolution pixel;

[0188] A second up-sampling module 703, configured to up-sample the second resolution pixels to obtain second original resolution pixels;

[0189] A second decoding module 704 is used to decode the enhancement layer code stream to obtain prediction information of the enhancement layer pixels of the first resolution pixels and matching information of the first resolution pixels and historical resolution pixels of the screen;

[0190] A synthesis module 705 is used to synthesize the second original resolution pixel and the enhancement layer pixel to obtain a third original resolution pixel; the third original resolution pixel is a pixel after the second original resolution pixel is enhanced;

[0191] The pixel enhancement module 706 is used to enhance the third original resolution pixel according to the prediction information or the matching information of the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0192] In an optional embodiment, the prediction information includes reference frame information, prediction mode information and motion vector information, and the pixel enhancement module 706 includes:

[0193] A calculation submodule, configured to calculate enhanced pixels in an area where the prediction residual is 0 pixels according to the reference frame information, the prediction mode information and the motion vector information in the prediction information;

[0194] The first pixel enhancement submodule is used to enhance the third original resolution pixel based on the enhanced pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0195] In an optional embodiment, the pixel enhancement module 706 includes:

[0196] A determination submodule, configured to determine a pixel area of ​​historical resolution pixels matching the first resolution pixels according to matching information between the first resolution pixels and historical resolution pixels of the screen;

[0197] The second pixel enhancement submodule is used to enhance the third original resolution pixel based on the pixel area of ​​the historical resolution pixel matching the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

[0198] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; the above modules are located in different processors in any combination.

[0199] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0200] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0201] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0202] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0203] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0204] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0205] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for enhancing the image quality of a video layer code stream, characterized in that: Applied to the encoding end, the method includes: Downsampling the first original resolution pixels of the screen to obtain first resolution pixels; Obtaining matching information between the first resolution pixels and the historical resolution pixels of the screen; Performing video encoding on the first resolution pixel to obtain a video layer code stream corresponding to the first resolution pixel, a second resolution pixel, and prediction information of the first resolution pixel; the second resolution pixel is a restored pixel of the first resolution pixel; Upsampling the second resolution pixels to obtain second original resolution pixels; Performing difference calculation on the first original resolution pixel and the second original resolution pixel to obtain initial enhancement layer pixels, and eliminating redundancy of the initial enhancement layer pixels; Compressing the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels and the matching information to obtain an enhanced layer bitstream; The video layer code stream and the enhancement layer code stream are sent to a decoding end, so that the decoding end performs image quality enhancement on the video layer code stream based on the enhancement layer code stream.

2. The method according to claim 1, characterized in that The eliminating redundancy of pixels in the initial enhancement layer comprises: Eliminate redundancy of the initial enhancement layer pixels according to the prediction information of the first resolution pixels.

3. The method according to claim 2, characterized in that The prediction information of the first resolution includes reference frame information, prediction mode information and motion vector information, and the eliminating redundancy of the initial enhancement layer pixels according to the prediction information of the first resolution pixels includes: Determine, according to the reference frame information, the prediction mode information and the motion vector information, pixel regions where the prediction residuals are 0 at different positions of the same reference frame and pixel regions where the prediction residuals are 0 at the same position of different reference frames; The initial enhanced pixels of the pixel regions with prediction residuals of 0 at different positions of the same reference frame and the pixel regions with prediction residuals of 0 at the same position of different reference frames are set to 0 to obtain the second enhanced layer pixels that eliminate temporal redundancy and spatial redundancy.

4. The method according to claim 2, characterized in that: The eliminating redundancy of pixels in the initial enhancement layer further includes: The redundancy of the initial enhancement layer pixels is eliminated according to the matching information between the first resolution pixels and the historical resolution pixels of the screen.

5. The method according to claim 4, characterized in that Eliminating redundancy of pixels of the initial enhancement layer according to matching information between pixels of the first resolution and pixels of the historical resolution of the screen includes: Determining a target pixel area of ​​the first resolution pixel that matches the historical resolution pixel according to matching information between the first resolution pixel and the historical resolution pixel of the screen; The initial enhanced pixels corresponding to the target pixel area are set to 0 to obtain the third enhanced layer pixels that eliminate temporal redundancy.

6. The method according to claim 4, characterized in that The obtaining matching information between the first resolution pixel and the historical resolution pixel of the screen includes: Extracting feature information of pixels of the first resolution, the feature information comprising a pixel hash value; Obtaining a hash value of the historical resolution pixel; and obtaining matching information between the first resolution pixel and the historical resolution pixel of the screen according to the pixel hash value of the first resolution pixel and the hash value of the historical resolution pixel.

7. The method according to claim 6, characterized in that The feature information also includes pixel gradient and pixel color distribution, and before eliminating the redundancy of the initial enhancement layer pixels, further includes: According to the pixel gradient of the first-resolution pixel and the pixel color distribution of the first-resolution pixel, the pixel area of ​​the first-resolution pixel is classified to obtain classification information of the first-resolution pixel area.

8. The method according to claim 7, characterized in that The eliminating redundancy of pixels in the initial enhancement layer further includes: Eliminate redundancy of the initial enhancement layer pixels according to the classification information of the first resolution pixel area.

9. The method according to claim 8, characterized in that The classification information of the first resolution pixel area includes a text-like pixel area and an image pixel area, and eliminating the redundancy of the initial enhancement layer pixels according to the classification information of the first resolution pixel area includes: Acquire enhancement layer pixels of the text-like pixel area, and when the enhancement layer pixels of the text-like pixel area are less than a first preset enhancement layer pixel threshold, set the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the text-like pixel area to 0, so as to obtain fourth enhancement layer pixels that eliminate visual redundancy; The enhancement layer pixels of the image pixel area are obtained, and when the enhancement layer pixels of the image pixel area are less than the second preset enhancement layer pixel threshold, the initial enhancement layer pixels or the second enhancement layer pixels or the third enhancement layer pixels corresponding to the image pixel area are set to 0 to obtain the fourth enhancement layer pixels that eliminate visual redundancy.

10. The method according to claim 9, characterized in that The step of compressing the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels, and the matching information to obtain an enhanced layer code stream includes: The second enhancement layer pixel or the third enhancement layer pixel or the fourth enhancement layer pixel, the prediction information of the first resolution pixel and the matching information are compressed to obtain an enhancement layer code stream.

11. A method for enhancing the image quality of a video layer code stream, characterized in that: Applied to a decoding end, the method comprises: Receiving the enhancement layer code stream and the video layer code stream sent by the encoding end; Decoding the video layer code stream to obtain second resolution pixels, where the second resolution pixels are restored pixels of the first resolution pixels; Upsampling the second resolution pixels to obtain second original resolution pixels; Decoding the enhancement layer bitstream to obtain enhancement layer pixels, prediction information of the first resolution pixels, and matching information of the first resolution pixels and historical resolution pixels of the screen; The second original resolution pixel and the enhancement layer pixel are synthesized to obtain a third original resolution pixel; the third original resolution pixel is a pixel after the second original resolution pixel is enhanced; The third original resolution pixel is enhanced according to the prediction information or the matching information of the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

12. The method according to claim 11, characterized in that The prediction information includes reference frame information, prediction mode information and motion vector information, and the enhancing the third original resolution pixel according to the prediction information of the first resolution pixel to obtain the fourth original resolution pixel with enhanced image quality includes: Calculate and obtain enhanced pixels in an area where the prediction residual is 0 pixels according to the reference frame information, the prediction mode information and the motion vector information in the prediction information; Based on the enhanced pixels, the third original resolution pixels are enhanced to obtain fourth original resolution pixels with enhanced image quality.

13. The method according to claim 11, characterized in that The step of enhancing the third original resolution pixel according to the matching information to obtain a fourth original resolution pixel with enhanced image quality comprises: Determine, according to matching information between the first resolution pixel and the historical resolution pixel of the screen, a pixel area of ​​the historical resolution pixel that matches the first resolution pixel; The third original resolution pixel is enhanced based on a pixel region of a historical resolution pixel matching the first resolution pixel to obtain a fourth original resolution pixel with enhanced image quality.

14. A device for enhancing the image quality of a video layer code stream, characterized in that: Applied to the encoding end, the device includes: A downsampling module, used for downsampling the first original resolution pixels of the screen to obtain first resolution pixels; An acquisition module, used to acquire matching information between the first resolution pixels and the historical resolution pixels of the screen; a video encoding module, configured to perform video encoding on the first resolution pixel to obtain a video layer code stream corresponding to the first resolution pixel, a second resolution pixel, and prediction information of the first resolution pixel; the second resolution pixel is a restored pixel of the first resolution pixel; A first up-sampling module, configured to up-sample the second resolution pixels to obtain second original resolution pixels; a difference calculation module, configured to perform difference calculation on pixels of the first original resolution and pixels of the second original resolution to obtain pixels of an initial enhancement layer, and eliminate redundancy of pixels of the initial enhancement layer; A compression module, used for compressing the enhanced layer pixels after eliminating redundancy, the prediction information of the first resolution pixels and the matching information to obtain an enhanced layer code stream; The sending module is used to send the video layer code stream and the enhancement layer code stream to the decoding end, so as to assist the decoding end to enhance the image quality of the video layer code stream based on the enhancement layer code stream.

15. A device for enhancing the image quality of a video layer code stream, characterized in that: Applied to a decoding end, the device comprises: A receiving module, used for receiving the enhancement layer code stream and the video layer code stream sent by the encoding end; A first decoding module, configured to decode the video layer code stream to obtain a second resolution pixel, where the second resolution pixel is a restored pixel of the first resolution pixel; A second upsampling module, configured to upsample the second resolution pixels to obtain second original resolution pixels; A second decoding module is used to decode the enhancement layer code stream to obtain prediction information of enhancement layer pixels, first resolution pixels, and matching information of the first resolution pixels and historical resolution pixels of the screen; A synthesis module, used for synthesizing the second original resolution pixel and the enhancement layer pixel to obtain a third original resolution pixel; the third original resolution pixel is a pixel after the second original resolution pixel is enhanced; The pixel enhancement module is used to enhance the third original resolution pixel according to the prediction information of the first resolution pixel or the matching information to obtain a fourth original resolution pixel with enhanced image quality.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 10 are implemented, or the steps of the method described in any one of claims 11 to 13 are implemented.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements the steps of the method described in any one of claims 1 to 10, or implements the steps of the method described in any one of claims 11 to 13.

Images