Image coding / decoding method and apparatus

Abstract

An image coding / decoding method and apparatus, which relate to the technical field of video processing. According to feature values of sub-images of an image to be coded, a coding device classifies these sub-images. Since the feature values of different sub-images are different, after the sub-images are matched with coded sub-images in a cached queue, matched sub-images and unmatched sub-images can be determined. Therefore, the matched sub-images of the image to be coded can reuse image coded data, in a previous image frame, of the matched sub-images in the image to be coded, without needing to have the same location and the same content as in the previous image frame. Therefore, during the coding process of the image to be coded, coding processing (such as standard coding) is performed on the unmatched sub-images, and information of coded sub-images, which match the matched sub-images, is generated for the matched sub-images, so as to avoid the problem of effective pixels in the matched sub-images being repeatedly coded, thereby reducing a coding bandwidth of video coding, and improving the coding efficiency.

Description

Image encoding and decoding method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 20, 2023, with application number 202310898245.X and application name “A Method and Device for Image Coding and Decoding”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of video processing technology, and in particular to an image encoding and decoding method and device. Background Art

[0003] With the development of big data, artificial intelligence (AI), and cloud computing, vast amounts of data are constantly being generated. To reduce data storage costs, compression devices can use hardware encoding algorithms to compress image data. This involves reconstructing the image data using a specific method, thereby reducing data redundancy and shrinking the storage capacity occupied by the image data.

[0004] During video encoding, the compression device compares the current frame with the adjacent previous frame, encoding the image areas that differ between the current and previous frames and not encoding image areas with the same location and content. In other words, the current frame reuses the image encoding data from the adjacent previous frame with the same location and content. However, for scenarios such as page scrolling and page turning, which are common in desktop scenarios, video encoding requires a large bandwidth and low encoding efficiency.

[0005] Summary of the Invention

[0006] The present application provides an image encoding and decoding method and device, which solves the problem of large encoding bandwidth and low encoding efficiency of video encoding in desktop scenarios.

[0007] In a first aspect, the present application provides an image encoding method. The image encoding method is applied to a codec system and is executed by an encoding device included in the codec system. The image encoding method includes: the encoding device obtains a first image to be encoded and divides the first image into multiple sub-images. The encoding device matches the multiple sub-images with a cached queue to determine a first category of sub-images and a second category of sub-images. The queue includes feature values ​​of one or more encoded sub-images, and the image to which the one or more encoded sub-images belong belongs to the same stream as the first image. The first category of sub-images matches the feature values ​​of the encoded sub-images in the queue, and the queue does not record feature values ​​of the second category of sub-images. Furthermore, the encoding device generates first information for the first category of sub-images, encodes the second category of sub-images to obtain second information, and then outputs encoded data for the first image. The first information indicates the encoded sub-images in the queue that match the first category of sub-images. The output encoded data includes the aforementioned first and second information.

[0008] Compared to the current frame multiplexing the image coding data with the same position and content in the previous frame, in this application, the encoding device classifies the sub-images of the image to be encoded based on the feature values ​​of the sub-images of the image to be encoded. Since the feature values ​​of different sub-images are different, after matching the sub-images with the encoded sub-images in the cached queue, it is possible to determine the matched sub-images (such as the first category of sub-images) and the unmatched sub-images (such as the second category of sub-images). Therefore, the matched sub-images of the image to be encoded do not need to be in the same position and have the same content as in the previous frame image in order to reuse the image coding data of the matched sub-images in the previous frame image. Furthermore, during the encoding process of the image to be encoded, the unmatched sub-images are encoded (such as standard encoding), and information (first information) of the matched encoded sub-images is generated for the matched sub-images, avoiding the problem of repeated encoding of valid pixels in the matched sub-images and reducing the encoding bandwidth of the video encoding. This improves the encoding efficiency.

[0009] In combination with the image encoding method provided in the first aspect, in an optional implementation, the characteristic values ​​of the encoded sub-images stored in the queue may include any one or a combination of the following items: image size, position, identification and matching status of the encoded sub-image, and the matching status indicates whether the encoded sub-image matches the sub-image to be encoded.

[0010] As an optional implementation, the identifier of the encoded sub-image is: a portion of the original pixel values ​​of the encoded sub-image, or a value obtained by processing the portion of the original pixel values ​​according to a reference method. The reference method includes at least one of a hash algorithm and a cyclic redundancy check algorithm.

[0011] In the present application, the encoding device stores the characteristic values ​​of the encoded sub-image (such as the image size, position, identification and matching status mentioned above) in an independent queue, and distinguishes the sub-images of the image to be encoded based on the relevant information stored in the queue, so that the encoding device can reuse the image encoding data of the sub-images in the previous multiple frames of images, reducing the repeated encoding process of valid pixels in sub-images with the same content, which is conducive to reducing the encoding bandwidth of video encoding and improving encoding efficiency. Moreover, the queue does not need to store the complete image encoding data of the encoded sub-image, that is, when the encoded code stream stores the initial encoding data of the encoded sub-image, the queue occupies a smaller amount of data than the image encoding data obtained by repeatedly encoding sub-images with the same content. The encoding device only needs to provide a small amount of storage space to implement the relevant functions of the queue, which is conducive to reducing the encoding bandwidth of video encoding and improving encoding efficiency.

[0012] In conjunction with the image encoding method provided in the first aspect, in an optional implementation, an encoding device generates first information of a first-category sub-image, including: the encoding device obtains a characteristic value of the first-category sub-image, and fills reference pixels for the first-category sub-image, and encodes the filled first-category sub-image to obtain first data; the encoding device obtains the first information based on the first data and the characteristic value of the first-category sub-image. Exemplarily, the reference pixel may refer to a preset fixed pixel value, and the fixed pixel value occupies a small encoding bandwidth, such as when all pixels in the filled first-category sub-image are the same, or when the pixels in the filled first-category sub-image conform to a certain pattern, and the certain pattern results in a small amount of encoded data. Furthermore, the amount of data output by the encoding device after encoding the video containing the image to be encoded is reduced, thereby reducing the encoding bandwidth of the video encoding, which is beneficial to improving encoding efficiency and reducing the bandwidth occupied by data transmission.

[0013] As an optional implementation, the first information includes: characteristic values ​​of the first-category sub-images, and second data obtained by encoding the first-category sub-images using a first bitrate. The first bitrate is less than or equal to a bitrate threshold. For the matched sub-images (first-category sub-images), the encoding device encodes the matched sub-images at a lower bitrate, resulting in a smaller amount of encoded data for the portion of the image area, thereby reducing the encoding bandwidth of the video encoding and improving encoding efficiency.

[0014] In conjunction with the image encoding method provided in the first aspect, in an optional implementation, after the encoding device determines the first category of sub-images and the second category of sub-images, the image encoding method provided in this application further includes: the encoding device generates a feature value for each sub-image in the second category of sub-images, and writes the feature value of each sub-image in the second category of sub-images into a queue. It is worth noting that the encoding device writes the feature value of the unmatched sub-image encoded for the first time into the queue. When the subsequent frame image of the image in the video also has other sub-images with the same content as the unmatched sub-image, the other sub-images can reuse the image encoding data of the sub-image whose feature value has been added to the queue, thereby reducing the encoding bandwidth of multiple frames of images in the video and improving encoding efficiency.

[0015] In conjunction with the image encoding method provided in the first aspect, in an optional implementation, the multiple sub-images have the same image size. Different frames in a video are divided in the same manner, so that the encoded sub-images and the sub-image to be encoded only need to be compared based on their characteristic values. Image encoding data for encoded sub-images with the same characteristic values ​​can then be reused, which helps reduce the encoding bandwidth of video encoding and improve encoding efficiency.

[0016] In conjunction with the image encoding method provided in the first aspect, in an optional implementation, an encoding device divides a first image into multiple sub-images, including: the encoding device obtains multiple image features of the first image and selects at least one division method from a plurality of preset division methods based on the multiple image features; the at least one image feature is used to describe image texture information of the first image, and the at least one division method matches the image texture information. Furthermore, the encoding device divides the first image according to the at least one division method to obtain multiple sub-images. The encoding device adaptively selects different division methods for different frames based on the image texture information of different frames, so that each divided sub-image matches the image texture, which is beneficial for improving the encoding effect of the image content contained in the sub-image and thereby improving the image encoding accuracy. Furthermore, when the same content appears repeatedly in different frames of a video, the matching image content and image texture makes it easier for the sub-image to be encoded to match the feature values ​​of the already encoded sub-image, thereby reusing the image encoding data of the already encoded sub-image, thereby reducing the encoding bandwidth of the video encoding and beneficial for improving the encoding efficiency of the video encoding.

[0017] In conjunction with the image encoding method provided in the first aspect, in an optional implementation, the encoding device outputting encoded data of a first image includes: the encoding device sending the encoded data of the first image. Exemplarily, the encoding device may store the encoded data of the first image in a memory of a codec system, or send the encoded data of the first image to a decoding device, so that the decoding device reconstructs and restores the image based on the encoded data.

[0018] In combination with the image encoding method provided in the first aspect, in an optional implementation method, the image encoding method provided by the present application further includes: the encoding device sends the data stored in the queue, and the data stored in the queue is carried in a code stream parameter set, a transport protocol header, a real-time transport protocol (RTP) payload or a file format. During the data communication process, in addition to transmitting the encoded encoded data (such as a code stream), the encoding device also transmits the data stored in the introduced queue, so that the decoding device can quickly reconstruct and restore the image based on the data stored in the queue, thereby improving the decoding efficiency. Moreover, since the amount of data stored in the queue is small, it can be carried in one of the above optional methods, so that the bandwidth required for communication between the encoding device and the decoding device is reduced, which is beneficial to improving the communication efficiency of the encoding and decoding system and reducing bandwidth consumption.

[0019] In a second aspect, the present application provides an image decoding method. This image encoding method is applied to a codec system and is executed by a decoding device included in the codec system. The image decoding method includes: the decoding device obtains a code stream and decodes the encoded data in the code stream to obtain a first image; the code stream includes the encoded data of at least one frame of image and data stored in a queue used for the encoded image. Furthermore, the decoding device matches the data stored in the queue in the code stream with the first image, determines a first-category sub-image and a second-category sub-image in the first image, and then, based on feature values ​​matching the first-category sub-image in the queue, restores the image region corresponding to the first-category sub-image in the first image to obtain a reconstructed image. The queue includes feature values ​​of one or more decoded sub-images, the one or more decoded sub-images belonging to an image that belongs to the same stream as the first image, the first-category sub-image matches the feature values ​​of the decoded sub-images in the queue, and no feature values ​​of the second-category sub-image are recorded in the queue.

[0020] Compared to the current frame multiplexing the image decoding data of the same position and content in the previous frame, in this application, the decoding device classifies the sub-images of the first image based on the characteristic values ​​of these sub-images. Since the characteristic values ​​of different sub-images are different, after matching the sub-images with the decoded sub-images in the cached queue, it is possible to determine the matched sub-images (such as the first category sub-images) and the unmatched sub-images (such as the second category sub-images). Therefore, the matched sub-images of the first image do not need to be in the same position and have the same content as in the previous frame image in order to reuse the image decoding data of the matched sub-images in the previous frame image with the image to be decoded. Furthermore, during the decoding process of the first image, the unmatched sub-images are decoded (such as standard decoding), and the information of the matched decoded sub-images is multiplexed for the matched sub-images, avoiding the problem of repeated decoding of valid pixels in the matched sub-images and reducing the decoding bandwidth of the video decoding. This improves decoding efficiency.

[0021] In combination with the image decoding method provided in the second aspect, in an optional implementation, the characteristic value of the decoded sub-image includes any one or a combination of the following items: image size, position, identification and matching status of the decoded sub-image, and the matching status indicates whether the decoded sub-image matches the sub-image to be decoded.

[0022] In combination with the image decoding method provided in the second aspect, in an optional implementation, the identifier of the decoded sub-image is: part of the original pixel value of the decoded sub-image, or, a value obtained by processing part of the original pixel value according to a reference method; the reference method includes at least one of a hash algorithm and a cyclic redundancy check algorithm.

[0023] In combination with the image decoding method provided in the second aspect, in an optional implementation manner, the image sizes of the multiple sub-images are consistent.

[0024] In combination with the image decoding method provided in the second aspect, in an optional implementation, the data stored in the aforementioned queue is carried in a code stream parameter set, a transport protocol header, a real-time transport protocol RTP payload or a file format.

[0025] In a third aspect, the present application provides a coding and decoding apparatus, which includes modules for executing the first aspect or any possible design of the method of the first aspect, and modules for executing the second aspect or any possible design of the method of the second aspect.

[0026] In a fourth aspect, the present application provides an encoding device. The encoding device includes at least one processor and a memory, wherein the memory is used to store a computer program, so that when the computer program is executed by the at least one processor, it implements the method described in the first aspect or any possible design of the first aspect.

[0027] In a fifth aspect, the present application provides a decoding device. The decoding device includes at least one processor and a memory, wherein the memory is used to store a computer program, so that when the computer program is executed by the at least one processor, it implements the method described in the second aspect or any possible design of the second aspect.

[0028] In a sixth aspect, the present application provides a coding and decoding system, which includes the coding device described in the fourth aspect and the decoding device described in the fifth aspect.

[0029] In a seventh aspect, the present application provides a chip. The chip includes: a processor and a power supply circuit. The power supply circuit is configured to supply power to the processor; the processor is configured to execute the steps of the method according to the first aspect or any possible implementation of the first aspect, and to execute the steps of the method according to the second aspect or any possible implementation of the second aspect.

[0030] In an eighth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes computer software instructions. When the computer software instructions are executed on a computer, the computer performs the steps of the method described in the first aspect or any possible implementation of the first aspect, as well as the steps of the method described in the second aspect or any possible implementation of the second aspect. For example, the computer is the aforementioned encoding device or decoding device.

[0031] In a ninth aspect, the present application provides a computer program product. When the computer program product is executed on a computer, the computer program product causes the computing device to perform the steps of the method described in the first aspect or any possible implementation of the first aspect, as well as the steps of the method described in the second aspect or any possible implementation of the second aspect. For example, the computer is the aforementioned encoding device or decoding device.

[0032] The beneficial effects of the third to ninth aspects above can be referred to the description of any implementation in the first or second aspects, and will not be repeated here. Based on the implementation provided by the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a schematic diagram of a video transmission system provided by the present application;

[0034] FIG2 is a schematic diagram of a video encoding and decoding system provided by the present application;

[0035] FIG3 is a flowchart diagram of an image encoding method provided by the present application;

[0036] FIG4 is a second flow chart of an image encoding method provided by the present application;

[0037] FIG5 is a third flow chart of an image encoding method provided by the present application;

[0038] FIG6 is a schematic diagram of the flow of the image encoding and decoding method provided by this application;

[0039] FIG7 is a schematic diagram of an image encoding and decoding method using a fixed block division method provided by the present application;

[0040] FIG8 is a schematic diagram of an image encoding and decoding method using an adaptive partitioning method provided by the present application;

[0041] FIG9 is a schematic structural diagram of a coding and decoding device provided by the present application;

[0042] FIG10 is a schematic structural diagram of an image processing system provided in this application. DETAILED DESCRIPTION

[0043] The technical solutions involved in this application may not only be used in combination with video coding standards (such as H.264, HEVC and other standards), but may also be used in combination with future video coding standards (such as H.266 standards). The terms used in the implementation methods of this application are only used to explain the specific embodiments of this application and are not intended to limit this application. The following is a brief introduction to some concepts that may be involved in this application.

[0044] Video contains multiple continuous images. When the continuous images change at more than 24 frames per second, the human eye cannot distinguish a single static image according to the principle of visual persistence. Then the multiple images that appear smooth and continuous are called video.

[0045] Video coding refers to the processing of a sequence of images to form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. Video coding as used in this article means video encoding or video decoding. Video encoding is performed on the source side and generally includes processing (for example, compressing) the original video picture to reduce the amount of data required to represent the video picture under the condition that a certain image quality is met, so as to store and / or transmit the video picture more efficiently. Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video picture. The "encoding" of the video picture involved in the embodiment should be understood as involving the "encoding" or "decoding" of the video sequence. The combination of the encoding part and the decoding part is also called codec (encoding and decoding). Video coding can also be called image coding (image coding) or image compression (image compression). Image decoding (image decoding) refers to the inverse process of image encoding.

[0046] A video sequence consists of a series of images, which are further divided into slices, and slices are further divided into blocks. Video coding is performed in blocks, and some new video coding standards have further expanded the concept of blocks. For example, the H.264 standard includes macroblocks (MBs), which can be further divided into multiple prediction blocks (partitions) for predictive coding. The High-Efficiency Video Coding (HEVC) standard uses basic concepts such as coding units (CUs), prediction units (PUs), and transform units (TUs), functionally classifying various block units and describing them using a new tree-based structure. For example, a CU can be divided into smaller CUs using a quadtree, and smaller CUs can be further divided, forming a quadtree structure. The CU is the basic unit for partitioning and encoding the coded image. A similar tree structure exists for PUs and TUs. A PU corresponds to a prediction block and is the basic unit for predictive coding. A CU is further divided into multiple PUs based on the partitioning pattern. A TU may correspond to a transform block and is a basic unit for transforming a prediction residual. However, whether CU, PU or TU, they all essentially belong to the concept of a block (or coding unit).

[0047] For example, in HEVC, a CTU is split into multiple CUs using a quadtree structure represented as a coding tree. A decision is made at the CU level whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode a picture area. Each CU can be further split into one, two or four PUs according to the PU split type. The same prediction process is applied within a PU, and relevant information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU split type, the CU can be partitioned into transform units (TUs) according to other quadtree structures similar to the coding tree for the CU. In the latest developments in video compression technology, quadtree and binary tree (QTBT) are used to partition frames to partition coding blocks. In the QTBT block structure, the CU can be square or rectangular in shape.

[0048] Lossless video coding means that the original video picture can be reconstructed, that is, the reconstructed video picture has the same quality as the original video picture (assuming there is no transmission loss or other data loss during storage or transmission).

[0049] Lossy video coding refers to performing further compression, such as quantization, to reduce the number of bits required to represent a video picture, but the decoder side cannot fully reconstruct the video picture, that is, the quality of the reconstructed video picture is lower or worse than that of the original video picture.

[0050] A code stream is a binary stream generated by encoding an image or video. It is also called a bit stream.

[0051] Bitrate: Video bitrate is the amount of data per unit time obtained by compressing the video during encoding. It can also be considered the amount of data transmitted per unit time during transmission. Generally speaking, for the same video source, a higher bitrate yields higher-quality images, while a lower bitrate yields lower-quality images.

[0052] Rate control, referred to as bitrate control below, adjusts the bitrate during the encoding and decoding process. Rate control modes include constant bitrate (CBR) and variable bitrate (VBR). Constant bitrate ensures a stable bitrate within the bitrate statistics period; variable bitrate allows bitrate fluctuations within the bitrate statistics period, thereby ensuring consistent encoded image quality.

[0053] Quantization refers to the process of mapping the continuous value of a signal into multiple discrete amplitudes.

[0054] The quantization parameter (QP) is used to quantize the residual values ​​generated by the prediction operation or the coefficients generated by the transform operation during the encoding process. During the decoding process, the syntax elements are dequantized to obtain the residual values ​​or coefficients. The quantization parameter is used in the quantization process. Generally, a larger quantization parameter value indicates a greater degree of quantization, resulting in lower reconstructed image quality and a lower bitrate. Conversely, a smaller quantization parameter value indicates better reconstructed image quality and a higher bitrate.

[0055] I-frame (Intra frame, IDR frame): In the field of video coding, the coding prediction method of all blocks in a frame is intra-frame prediction. It can be decoded independently without relying on other frames. In the bitstream, this frame is marked as an I-frame type frame and is generally referred to as "I-frame".

[0056] Predictive-frame (P-frame): In the field of video coding, the encoding of a frame requires reference to a previous frame, and the frame is marked as a P-frame type frame in the bitstream, generally referred to as a "P-frame".

[0057] Bi-directional interpolated prediction frame (B-frame): In the field of video coding, the encoding of a frame requires reference to the previous frame and the next frame, and the frame is marked as a B-frame type frame in the bitstream, generally referred to as a "B-frame".

[0058] In the field of coding and decoding, the above I frames, P frames and B frames can be collectively referred to as IPB frames.

[0059] Reference frame: The frame that the IPB frame needs to refer to during predictive coding is its reference frame.

[0060] Reconstructed frame: A video frame generated by an encoder after a complete encoding process is a reconstructed frame, which is usually used as a reference frame for subsequent frames. The encoder's reconstructed frame is equivalent to the frame generated by the decoder after decoding.

[0061] Parameter Set: A parameter set contains global information about the corresponding coded image. A Video Parameter Set (VPS) contains video classification information, which facilitates compatibility with standard extensions for scalable video coding or multi-view video coding. A Sequence Parameter Set (SPS) contains parameters for a continuously coded video sequence. A Picture Parameter Set (PPS) corresponds to a single image or several images in a sequence.

[0062] The following describes the implementation of the image encoding and decoding method provided by this application with reference to the accompanying drawings.

[0063] Figure 1 is a schematic diagram of a video transmission system provided by the present application. The video processing process includes video acquisition, video encoding, video transmission, video decoding and display processes. The video transmission system includes multiple terminal devices (such as terminal devices 111 to terminal devices 115 shown in Figure 1) and a network, wherein the network can realize the function of video transmission. The network may include one or more network devices, which may be routers or switches, etc.

[0064] The terminal device shown in FIG1 may be, but is not limited to, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The terminal device may be a mobile phone (such as the terminal device 114 shown in FIG1 ), a tablet computer, a computer with wireless transceiver function (such as the terminal device 115 shown in FIG1 ), a virtual reality (VR) terminal device (such as the terminal device 113 shown in FIG1 ), an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.

[0065] As shown in FIG1 , in different video processing processes, the terminal devices are different.

[0066] For example, during the video capture process, the terminal device 111 may be a camera device for recording screen or desktop content, a mobile phone, tablet computer or personal computer with a video recording function, etc.

[0067] For another example, during the video encoding process, the terminal device 112 may be a server or a data center, which may include one or more physical devices with encoding capabilities, such as servers, mobile phones, tablet computers, or other encoding devices.

[0068] For example, during the video decoding and display process, terminal device 113 may be an all-in-one computer, and the user may control the displayed image range by turning the device. Terminal device 114 may be a mobile phone, and the user may control the image range displayed on mobile phone 114 through touch or air operations. Terminal device 115 may be a personal computer, and the user may control the image range displayed on the display screen through input devices such as a mouse or keyboard.

[0069] It can be understood that video is a general term, and a video includes an image sequence of multiple consecutive frames, where one frame corresponds to one image.

[0070] Figure 1 is only a schematic diagram, and the video transmission system may also include other devices, which are not shown in Figure 1. The embodiments of the present application do not limit the number and type of terminal devices included in the system.

[0071] Based on the video transmission system shown in Figure 1, Figure 2 is a schematic diagram of a video encoding and decoding system provided by this application. The video encoding and decoding system 200 includes an encoding device 210 and a decoding device 220. The encoding device 210 establishes a communication connection with the decoding device 220 through a communication channel 230.

[0072] The encoding device 210 described above can implement the function of video encoding. As shown in FIG1 , the encoding device 210 can be the terminal device 112 . The encoding device 210 can also be a data center with video encoding capabilities, for example, the data center includes multiple servers.

[0073] The encoding device 210 may include a data source 211 , a pre-processing module 212 , an encoder 213 , a communication interface 214 , and a context-aware deduplication module 215 .

[0074] Data source 211 may include or may be any type of electronic device for capturing video and / or any type of source video generating device, such as a computer graphics processor for generating computer animation scenes or any type of device for acquiring and / or providing source video or computer-generated source video. Data source 211 may be any type of memory or storage for storing the aforementioned source video. The aforementioned source video may include multiple video streams or images captured by multiple video capture devices (e.g., cameras).

[0075] An image can be viewed as a two-dimensional array or matrix of pixels (picture elements). The pixels in the array are also referred to as samples. The number of samples in the array or image in the horizontal and vertical directions (or axes) defines the image's size and / or resolution. To represent color, three color components are typically used; that is, an image can be represented as or contain three sample arrays. For example, in the RBG format or color space, an image includes corresponding arrays of red, green, and blue samples. However, in video coding, each pixel is typically represented in a luma / chroma format or color space. For example, an image in the YUV format includes a luma component designated by Y (sometimes designated by L) and two chroma components designated by U and V. The luma component Y represents the brightness or grayscale intensity (for example, in grayscale images, both are the same), while the two chroma components U and V represent the chroma or color information components. Accordingly, a YUV format image includes a luma sample array of luma sample values ​​(Y) and two chroma sample arrays of chroma values ​​(U and V). An image in RGB format can be converted or transformed into YUV format, and vice versa, a process also referred to as color conversion or transformation. If the image is black and white, the image may include only the array of luminance samples. In this application, the image transmitted by the data source 211 to the encoder 213 may also be referred to as raw image data or source image.

[0076] The preprocessing module 212 is configured to receive a source video and preprocess the source video to obtain a preprocessed image, such as a panoramic video or a multi-frame image. For example, the preprocessing performed by the preprocessing module 212 may include color format conversion (e.g., from RGB to YCbCr), octree structuring, video splicing, etc.

[0077] The encoder 213 is used to receive the pre-processed image and encode the pre-processed image to obtain encoded data (such as a code stream). For example, the encoder 213 may include a code control unit 2131 and an encoding unit 2132. The code control unit 2131 is used to determine the quantization parameter used to encode each coding unit in the current frame, so that the encoding unit 2132 predicts, quantizes, and encodes the pre-processed image according to the quantization parameter to obtain a code stream. For example, the encoder 213 matches the feature value of the sub-image with the feature value of the encoded sub-image stored in the queue, and if the feature values ​​are the same, generates instruction information for the sub-image, which is used to indicate the image encoding data of the encoded sub-image.

[0078] The communication interface 214 in the encoding device 210 can be used to receive a code stream and send the code stream (or a version of the code stream after any other processing) to another device such as the decoding device 220 or any other device through the communication channel 230 to store, display or directly reconstruct the original image.

[0079] The context-aware deduplication module 215 is used to identify and match sub-images in the image to be encoded that have the same content as the encoded sub-images, and generate image instruction information (or feature values) for the matched sub-images, package the image instruction information and the low-bitrate encoded data of the matched sub-image, and transmit them to the communication interface 214.

[0080] Optionally, the encoding device 210 includes a storage area, which is used to store the aforementioned queue, in which characteristic values ​​of one or more encoded sub-images are stored.

[0081] The above-mentioned decoding device 220 can implement the functions of image decoding or video decoding. As shown in FIG1 , the decoding device 220 can be any one of the terminal devices 113 to 115 shown in FIG1 .

[0082] The decoding device 220 may include a display device 221 , a post-processing module 222 , a decoder 223 , a communication interface 224 , and a context-aware deduplication module 225 .

[0083] The communication interface 224 in the decoding device 220 is used to receive a code stream (or a version of the code stream after any other processing has been performed on the code stream) from the encoding device 210 or from any other encoding device such as a storage device.

[0084] The communication interface 214 and the communication interface 224 can be used to send or receive code streams through a direct communication link between the encoding device 210 and the decoding device 220, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0085] The communication interface 224 corresponds to the communication interface 214 , and can be used, for example, to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain a code stream.

[0086] Both the communication interface 224 and the communication interface 214 can be configured as a unidirectional communication interface as indicated by the arrow pointing from the encoding device 210 to the corresponding communication channel 230 of the decoding device 220 in Figure 2, or a bidirectional communication interface, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link, or data transmission, such as the transmission of encoded compressed data (such as a code stream), etc.

[0087] The decoder 223 is used to receive the encoded data and decode the encoded data to obtain decoded data (image or video, etc.). Exemplarily, the decoder 223 performs entropy decoding, dequantization and predictive reconstruction on the code stream to obtain a reconstructed image. The decoder 223 may include a code control unit 2231 and a decoding unit 2232. The code control unit 2231 is used to determine the quantization parameter used to decode each coding unit in the current frame, so that the decoding unit 2232 can decode, dequantize, and predictively reconstruct the code stream according to the quantization parameter to obtain the reconstructed image. The decoder 223 can determine the target number of bits of the coding unit according to the image content of the coding unit, the number of lossy bits and the buffer fullness of the bitstream buffer, and decode the bitstream of the coding unit according to the quantization parameter determined by the target number of bits.

[0088] The context-aware deduplication module 225 is used to identify and match sub-images in the image to be reconstructed that have the same content as the decoded sub-images. Based on the image instruction information (or feature value) of the matched sub-image, it decodes and fills the decoded sub-image indicated by the image instruction information into the corresponding position of the matched sub-image in the image to be reconstructed, and transmits the result to the post-processing module 222. On the encoding side, the context-aware deduplication module is also called the context-aware deduplication coding (CADC) module; on the decoding side, the context-aware deduplication module is also called the context-aware deduplication decoding (CADC) module.

[0089] The post-processing module 222 is configured to perform post-processing on the decoded data to obtain post-processed data (e.g., a reconstructed image to be displayed). The post-processing performed by the post-processing module 222 may include, for example, color format conversion (e.g., from YCbCr to RGB), octree reconstruction, video splitting and fusion, or any other processing for generating data for display on the display device 221.

[0090] Display device 221 is configured to receive the post-processed data for display to a user or viewer. Display device 221 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0091] As an optional implementation, the encoding device 210 and the decoding device 220 may transmit the encoded data via a data forwarding device, such as a router or a switch.

[0092] The functions of the above context-aware deduplication modules are to reuse the data of encoded sub-images or decoded sub-images, thereby reducing the amount of processing required by the encoder or decoder and improving the processing efficiency of the encoding and decoding process; this application does not limit the name of the context-aware deduplication module.

[0093] The structure of the above-mentioned codec system is only a schematic illustration. In some possible implementations, the codec system may also include other devices, for example, the codec system may also include an end-side device or a cloud-side device. After the acquisition device (such as the terminal device 111 in Figure 1) obtains the original image, it preprocesses the original image to obtain a preprocessed image; and then transmits the preprocessed image to the end-side device or cloud-side device (such as the terminal device 112 in Figure 1), which then performs the encoding and decoding function on the preprocessed image.

[0094] It is worth noting that the standard video encoder was originally designed to compress natural videos captured by cameras, and has obvious shortcomings in some specific application scenarios (such as screen content sharing). Although the standard encoder later introduced the screen content coding (SCC) tool, the SCC tool is only implemented in software, with overhead bottlenecks and high energy consumption. For example, in the desktop sharing scenario of real-time video communication (RTC), due to specifications and latency requirements, a standard hardware video encoder is often used. However, since the standard video encoder does not include screen content coding tools such as SCC, there are problems of low compression rate and high bandwidth overhead in such scenarios. Therefore, the solution provided in this application can be used to solve the problems of large encoding bandwidth and low compression rate of the standard video encoder in screen content scenarios.

[0095] The image encoding and decoding method provided in this application is applied to encoding devices and decoding devices. The image encoding method provided in this application is first described below in conjunction with the accompanying drawings. As shown in Figure 3, Figure 3 is a flowchart of an image encoding method provided in this application. The image encoding method is performed by encoding device 310, which can refer to encoding device 210 in Figure 2, or encoder 213 in Figure 2. The image encoding method provided in this embodiment includes the following steps S310 to S340.

[0096] S310 : The encoding device 310 obtains a first image to be encoded, and divides the first image into a plurality of sub-images.

[0097] The first image may refer to any frame image in a video to be transmitted, and the video may be generated in a desktop sharing scenario.

[0098] In one optional implementation, the first image is divided into blocks of a fixed size (fixed block division), i.e., the sub-images have the same image size. Exemplarily, the preset block division methods include, but are not limited to, fixed-size rectangles, triangles, and the like. The identifier of each sub-image after division is calculated according to preset rules, including, but not limited to, pixel value calculation and feature calculation.

[0099] For example, in this embodiment, the size of the sub-image after segmentation can be an enumeration of several fixed shape types, or variable values ​​such as length, width and height, or a method for obtaining the size by specifying a rule for calculating the size (such as the difference in length, width and height with the previous sub-image).

[0100] In this embodiment, different frame images in the video are divided in the same way, so that each encoded sub-image and the sub-image to be encoded only need to be compared based on the characteristic value, and then the image encoding data of the encoded sub-image with the same characteristic value can be reused, which is beneficial to reducing the encoding bandwidth of video encoding and improving encoding efficiency.

[0101] In another optional implementation, the first image is divided into blocks adaptively (adaptively or dynamically) based on different image content. Exemplarily, the encoding device obtains multiple image features of the first image and selects at least one division method from a plurality of preset division methods based on the multiple image features; the at least one image feature is used to describe image texture information of the first image, and the selected at least one division method matches the image texture information. Furthermore, the encoding device divides the first image according to the selected at least one division method to obtain multiple sub-images.

[0102] For example, the preset division method (block method) includes but is not limited to rectangles, triangles, etc. of variable size. The identification of each sub-image after division is calculated according to the preset rules. The preset rules include but are not limited to pixel value calculation, feature calculation, etc.

[0103] As shown in FIG3 , the first image is divided into sub-image 1, sub-image 2, sub-image 3, sub-image 4, and other sub-images, etc. Sub-images 1 to 3 are of the same size, and sub-image 4 is different from the other sub-images.

[0104] In this embodiment, the encoding device adaptively selects different division methods for images of different frames based on the image texture information of different frame images, so that each sub-image after division matches the image texture, which is conducive to improving the encoding effect of the image content contained in the sub-image, and thus improving the encoding accuracy of the image.

[0105] S320 : The encoding device 310 matches the plurality of sub-images with the cached queue to determine the first category of sub-images and the second category of sub-images.

[0106] As a feasible example, the storage space of the queue is provided by a memory included in the encoding device.

[0107] As another feasible example, the storage space of the queue is provided by a storage device that communicates with the encoding device. In this case, the queue is also called an external buffer queue in the video encoding process.

[0108] The above memory and storage devices may include but are not limited to the following types of storage media: dynamic random access memory (DRAM) or storage class memory (SCM). DRAM is a semiconductor memory, and like most random access memories (RAM), it is a volatile memory device. SCM is a composite storage technology that combines the characteristics of traditional storage devices and memories. Storage class memory can provide faster read and write speeds than hard disks, but its access speed is slower than DRAM and its cost is cheaper than DRAM. However, DRAM and SCM are only exemplary in this embodiment. Memory may also include other random access memories, such as static random access memory (SRAM). Alternatively, it may be a storage medium such as solid-state storage particles, mechanical hard disks, magnetic tape disks, memory or cache, etc., which is not limited in this application.

[0109] The queue includes characteristic values ​​of one or more encoded sub-images, and the images to which the one or more encoded sub-images belong belong to the same stream (video stream or other multimedia stream containing images) as the first image.

[0110] For example, the encoded sub-image may be a sub-image included in a previous frame image of the first image in the video stream.

[0111] For another example, the encoded sub-image may be a sub-image included in a subsequent frame image of a first image in a video stream.

[0112] Therefore, the characteristic values ​​of the sub-images stored in the external cache queue (or queue) can be sub-images of several frames before the first image, or sub-images of images after the first image, so that when the encoding device matches the sub-images of the first image with the data stored in the queue, the range of image encoding data that can be used for the multiplexed sub-images is increased, and the encoding bandwidth can be reduced to a greater extent.

[0113] The first type of sub-image determined in S320 matches the characteristic value of the encoded sub-image in the queue, and the characteristic value of the second type of sub-image is not recorded in the queue.

[0114] In an optional implementation, the characteristic value of the encoded sub-image may include any one or a combination of the following: image size, position, identification and matching status of the encoded sub-image, and the matching status indicates whether the encoded sub-image matches the sub-image to be encoded.

[0115] The image size of the encoded sub-image can be an enumeration of several fixed shape types, or variable values ​​such as length, width and height, or a sub-image size determined by a method that can obtain the size, such as a specified rule for calculating the size (such as the difference in length, width and height with the previous sub-image block).

[0116] The position of the encoded sub-image can be the absolute value of the vertex coordinates of the encoded sub-image, or the position of the upper, lower, left, and right sides, or the sub-image position can be determined by a method that can obtain the position, such as a rule for calculating the position (such as an offset value from the position of the previous sub-image).

[0117] The identification (ID) value of the encoded sub-image may be a portion of the original pixel values ​​of the encoded sub-image, or a value obtained by processing the portion of the original pixel values ​​according to a reference method, and the value uniquely represents the sub-image. The reference method includes at least one of a hash algorithm and a cyclic redundancy check (CRC) algorithm.

[0118] The matching status can be marked with a specific identifier, such as a binary value of 0 or 1: 0 for no match, 1 for match; or 0 for match, 1 for no match. Optionally, other possible status type strings can also be used for marking, which is not limited by this application.

[0119] In this embodiment, the encoding device stores the characteristic values ​​of the encoded sub-image (such as the image size, position, identification and matching status mentioned above) in an independent queue, and distinguishes the sub-images of the image to be encoded based on the relevant information stored in the queue, so that the encoding device can reuse the image encoding data of the sub-images in the previous multiple frames of images, reducing the repeated encoding process of valid pixels in sub-images with the same content, which is conducive to reducing the encoding bandwidth of video encoding and improving encoding efficiency.

[0120] Moreover, the queue does not need to store the complete image coding data of the encoded sub-image, that is, when the encoded code stream stores the initial coding data of the encoded sub-image, compared with the image coding data obtained by repeatedly encoding the sub-image with the same content, the amount of data occupied by the queue is smaller, and the encoding device only needs to provide a small amount of storage space to implement the relevant functions of the queue, which is conducive to reducing the coding bandwidth of video encoding and improving coding efficiency.

[0121] S330 : The encoding device 310 generates first information of the first type of sub-image, and encodes the second type of sub-image to obtain second information.

[0122] Exemplarily, the encoding device 310 divides the first image into multiple sub-images, inputs the sub-images into an encoder, and outputs the first information and the second information. The second information includes: image encoding data obtained by encoding the second type of sub-images, and the encoding method may be a standard encoder.

[0123] The first information indicates: the encoded sub-images in the queue that match the first type of sub-images. Regarding the implementation of the encoding device generating the first information, two feasible examples are provided below in conjunction with FIG4 and FIG5.

[0124] In a first feasible example, the encoding device obtains the first information of the first-category sub-image by matching block pixel padding. As shown in Figure 4 , Figure 4 is a second flow chart diagram of an image encoding method provided by this application. The process of encoding device 310 generating the first information of the first-category sub-image may include the following steps S410 to S430.

[0125] S410 : The encoding device 310 obtains feature values ​​of a first type of sub-image.

[0126] Exemplarily, the encoding device 310 may calculate the feature value according to the pixel values ​​included in the first type of sub-image.

[0127] As another example, the encoding device 310 may calculate the feature value according to the image features of the first type of sub-image.

[0128] The above two examples are merely optional implementations provided in this embodiment. The encoding device 310 may also calculate the feature value according to other contents of the first type of sub-image.

[0129] S420: The encoding device 310 fills reference pixels into the first-category sub-image, and encodes the filled first-category sub-image to obtain first data.

[0130] The pixel filling method may include, but is not limited to, a fixed color filling method or a background color filling method, etc. For example, the reference pixel may be a preset pixel value, such as 0 or other specified values.

[0131] In this embodiment, the encoding method adopted by the encoding device 310 for the first type of padded sub-image is consistent with the encoding method adopted for the second type of sub-image, which can avoid deploying multiple different types of encoders in the encoding device 310, resulting in blurring in different areas of the reconstructed image, and is beneficial to improving the image quality of image decoding.

[0132] S430: The encoding device 310 obtains first information according to the first data and the feature values ​​of the first type of sub-image.

[0133] Exemplarily, the first information includes the image coding data (first data) and instruction information (characteristic value) of the first type of sub-image after encoding and filling. During the decoding process, the decoding device queries the content of the decoded sub-image according to the instruction information, and fills the content of the decoded sub-image into the corresponding position of the first type of sub-image.

[0134] In a second feasible example, the encoding device encodes the first type of sub-image using a low bit rate control method to obtain the first information. As shown in Figure 5 , Figure 5 is a third flow chart diagram of an image encoding method provided in this application. The process of encoding device 310 generating the first information of the first type of sub-image may include the following steps S510 to S530.

[0135] S510 : The encoding device 310 obtains feature values ​​of a first type of sub-image.

[0136] Regarding the implementation of S510, reference may be made to the relevant contents of S410 and will not be elaborated here.

[0137] S520: The encoding device 310 encodes the first type of sub-image using the first coding rate to obtain second data.

[0138] The first bit rate is less than or equal to a bit rate threshold.

[0139] Exemplarily, rate control parameters are set for the sub-image according to a preset rate control rule, which includes, but is not limited to, block-level QP setting. If a larger QP value (e.g., QP=50) is set for the first type of sub-image, the amount of encoded data obtained by encoding the first type of sub-image using this QP value is smaller, which is beneficial for reducing the encoding bandwidth of the video encoding.

[0140] S530: The encoding device 310 obtains first information according to the second data and the characteristic values ​​of the first type of sub-image.

[0141] In combination with the embodiments provided in Figures 4 and 5, when the same content appears repeatedly in different frames of a video, the sub-image to be encoded can more easily match the feature values ​​of the encoded sub-image due to the matching of the image content and the image texture, and then reuse the image encoding data of the encoded sub-image, thereby reducing the encoding bandwidth of the video encoding and improving the encoding efficiency of the video encoding.

[0142] Moreover, during the encoding process, whether filling reference pixels for the first type of sub-image or setting a larger QP value for the first type of sub-image, the encoding method of the first type of sub-image in the corresponding area of ​​the first image and the aforementioned second type of sub-image use the same encoder. Therefore, the encoding device only needs to provide a universal standard hardware video encoder (such as a standard encoder that complies with the video coding standard), which can avoid the problem of obvious boundaries or blurred boundary positions between sub-images caused by encoding different sub-images with multiple encoders.

[0143] Continuing to refer to FIG. 3 , the image encoding method provided in this embodiment further includes S340 .

[0144] S340 , the encoding device 310 outputs the encoded data of the first image.

[0145] The coded data of the first image includes the first information and the second information obtained in S330.

[0146] In one possible example, encoding device 310 may output the encoded data by sending the encoded data of the first image to another device. The other device may be a decoding device or a storage device for storing the video stream to which the first image belongs. Furthermore, the decoding device may reconstruct and restore the image based on the encoded data of the first image.

[0147] To improve image decoding efficiency on the decoding side, as an optional implementation, based on Figures 3 to 5, the image encoding method provided in this embodiment further includes: encoding device 310 sending data stored in a queue. Exemplarily, the data stored in the queue can be packaged with the video stream obtained after image encoding. This packaging method includes, but is not limited to, adding specific fields or splicing parameters into the header or tail of the bitstream. For example, the data stored in the queue is carried in a bitstream parameter set, a transport protocol header, an RTP payload, or a file format.

[0148] The code stream parameter set may include but is not limited to VPS, SPS, PPS and Supplemental Enhancement Information (SEI).

[0149] The transport protocol header may refer to an RTP extension header. For example, the extension header specifies that the extension type is image instruction information (characteristic value), and the corresponding value is the above-mentioned characteristic value.

[0150] The RTP payload is identified by the RTP version number or character syntax information to indicate that the RTP payload carries image instruction information (characteristic value).

[0151] The image instruction information (characteristic value) can also be unloaded in a file format, such as carried in the file format established by the standard ISO14496-12, or in the transport stream (MPEG-2 Transport Stream, MPEG2TS) file format. If carried in ISO14496-12, it can be carried in the track hearder, or a corresponding box can be added to independently present the image instruction information of each decoded frame in the box. If the image instruction information is carried in the MPEG2TS file, the information can be set in the adaptation_field_extension_flag in the Adaptation field, and the OutLoopPicOrderflag can be added to the adaptation_field_extension_flag to mark the use of the image instruction information (characteristic value). The outLoopPicParams contains the image instruction information. Through this parameter, the decoding side can obtain the corresponding image instruction information (characteristic value). The format is as follows:

[0152] "adaptation_field(){

[0153] adaptation_field_length

[0154] if(adaptation_field_length>0){

[0155] if(adaptation_field_extension_flag=='1'){

[0156] if(OutLoopPicOrderflag=='1'){

[0157] outLoopPicParams

[0158] }

[0159] }

[0160] }

[0161] }”

[0162] During the data communication process, in addition to transmitting the encoded data (such as the code stream), the encoding device also transmits the data stored in the introduced queue, so that the decoding device can quickly reconstruct and restore the image based on the data stored in the queue, thereby improving decoding efficiency.

[0163] Moreover, since the amount of data stored in the queue is small, it can be carried in one of the above optional methods, which reduces the bandwidth required for communication between the encoding device and the decoding device, which is beneficial to improving the communication efficiency of the encoding and decoding system and reducing bandwidth consumption.

[0164] In an optional implementation, this embodiment further updates the cached queue to improve the matching hit rate of sub-images in subsequent frames of the video within the queue, thereby reducing the encoding bandwidth of the video encoding. Exemplarily, after the encoding device 310 determines the first and second categories of sub-images, the image encoding method provided in this embodiment further includes: the encoding device 310 generating a feature value for each sub-image in the second category, and writing the feature value for each sub-image in the second category to the cached queue.

[0165] In this way, the encoding device writes the characteristic value of the unmatched sub-image (second type of sub-image) encoded for the first time into the queue. When the subsequent frame image of the currently encoded image in the video also has other sub-images with the same content as the unmatched sub-image, the image encoding data of the sub-image in the queue with reusable characteristic values ​​can be added to the sub-image in the queue, thereby reducing the encoding bandwidth of multiple frames of images in the video and improving encoding efficiency.

[0166] Based on Figure 2, the image encoding and decoding method provided by this application is described in general terms in conjunction with Figure 6. As shown in Figure 3, Figure 6 is a flow chart of the image encoding and decoding method provided by this application. The encoding method is performed by an encoding device, and the steps required to be performed by the encoding device include A1 to A6; the decoding method is performed by a decoding device, and the steps required to be performed by the decoding device include B1 to B6. The following provides exemplary descriptions of the steps required to be performed on both the encoding and decoding sides.

[0167] A1: The encoding device captures multiple frames of images included in the video.

[0168] The video may refer to a video of a video conference scene or other types of desktop content sharing (or screen content sharing) scenes.

[0169] A2: The encoding device divides the current frame image to be encoded into multiple sub-images, and searches and matches the multiple sub-images with the existing sub-images in the external cache queue to determine the matched sub-images and unmatched sub-images in the current frame image.

[0170] Furthermore, the encoding device outputs image instruction information regarding the segmentation and matching of the multiple sub-images, i.e., the aforementioned feature values, including: the position of each sub-image, the size of the sub-image, the sub-image identifier, and whether the sub-image can find a matching encoded sub-image in the external cache queue. The specific implementation of A2 can be found in the description of S310 and S320 above and is not detailed here.

[0171] A3: The encoding device performs pre-processing on the matched sub-image (matching sub-image) in the current frame image.

[0172] The pre-processing process includes at least one of filling reference pixels or setting a low bit rate. For the specific implementation of A3, please refer to the relevant description of the embodiments provided in Figures 4 and 5 above, which will not be repeated here.

[0173] A4: The encoding device inputs the pre-processed current frame image into a general video encoder and outputs an encoded video stream.

[0174] The encoded video stream includes image encoding data of unmatched sub-images and image encoding data of pre-processed matched sub-images.

[0175] A5: The encoding device packages the video stream output by A4 and the feature values ​​output after the search and matching in A2, and sends the packaged code stream to the decoding side.

[0176] The packaging method may include but is not limited to adding specific fields to the video stream output by A4, splicing parameters at the head or tail of the video stream, etc. The packaging method or transmission location of the feature value can be referred to the description of S340 and will not be repeated here.

[0177] It is worth noting that the above A2, A3 and A5 can be performed by a context-aware deduplication module included in the encoding device, such as the context-aware deduplication module 215 provided in Figure 2.

[0178] A6: The encoding device writes the feature value of the unmatched sub-image into the external cache queue to update the data stored in the external cache queue.

[0179] Following the packetized code stream sent by A5 on the encoding side, the decoding side executes the following B1 to B6.

[0180] B1: The decoding device receives the code stream sent by the encoding device.

[0181] The code stream includes: coded data of at least one frame of image and data (feature values) stored in a queue used for coding the image. The data stored in the queue can be referred to the related description of S320 above and will not be described in detail here.

[0182] B2: The decoding device separates the code stream to obtain the video stream and feature values ​​in the code stream.

[0183] The separation method is determined by the position of the characteristic value in the code stream, and this application does not limit this.

[0184] B3: The decoding device decodes the video stream to obtain a first image.

[0185] Exemplarily, the decoding device may use a universal standard decoder to decode the video stream, thereby outputting the first image.

[0186] B4: The decoding device matches the data stored in the queue in the codestream with the first image, and determines the matched sub-images (first category sub-images) and unmatched second category sub-images in the first image. The decoding device retrieves the feature value of the decoded sub-image corresponding to the matched sub-image from the external cache queue, and uses the feature value of the decoded sub-image to determine the image content of the decoded sub-image. Furthermore, the image area of ​​the matched sub-image in the first image is filled with the image content of the decoded sub-image, thereby restoring the reconstructed image of the first image.

[0187] The queue includes characteristic values ​​of one or more decoded sub-images, the image to which the one or more decoded sub-images belong belongs to the same stream (video stream) as the first image, the first type of sub-image matches the characteristic values ​​of the decoded sub-images in the queue, and the queue does not record the characteristic values ​​of the second type of sub-images.

[0188] The difference between B4 and the encoding side is that the decoding process is the reverse process of the encoding process. Therefore, the feature values ​​stored in the queue used in the matching process are the feature values ​​of the decoded sub-image.

[0189] B5: The decoding device displays the restored reconstructed image. Before displaying the reconstructed image, the decoding device may also perform image processing operations such as rendering or cropping on the reconstructed image, which is not limited in this application.

[0190] Compared with the current frame multiplexing the image decoding data with the same position and content in the previous frame, in the present application, the decoding device classifies the sub-images of the first image according to the characteristic values ​​of the sub-images. Since the characteristic values ​​of different sub-images are different, after matching the sub-images with the decoded sub-images in the cached queue, the matched sub-images (such as the first type of sub-images) and the unmatched sub-images (such as the second type of sub-images) can be determined. Therefore, the matched sub-image of the first image does not need to have the same position and content as in the previous frame image in order to reuse the image decoding data of the matched sub-image in the previous frame image and the image to be decoded.

[0191] Furthermore, during the decoding process of the first image, the unmatched sub-image is decoded (e.g., using standard decoding), and the information of the matched decoded sub-image is reused for the matched sub-image, thus avoiding the problem of repeated decoding of valid pixels in the matched sub-image, reducing the decoding bandwidth of the video decoding, and improving decoding efficiency.

[0192] B6: The decoding device writes the feature value of the unmatched sub-image into the external cache queue, thereby updating the data stored in the external cache queue.

[0193] Thus, the decoding device writes the characteristic value of the unmatched sub-image decoded for the first time into the queue. When the subsequent frame image of the image in the video also has other sub-images with the same content as the unmatched sub-image, the other sub-images can also reuse the image decoding data of the sub-image whose characteristic value has been added to the queue, thereby reducing the decoding bandwidth of multiple frames of images in the video and improving decoding efficiency.

[0194] It is worth noting that the sub-image division method and eigenvalue generation method adopted by the decoding side are consistent with those of the encoding side, and this application will not be described in detail here. To make the image encoding and decoding method provided by this application clearer, the image encoding and decoding method provided by this application is illustrated below using two sub-image division methods, fixed block division and adaptive division (dynamic division), as examples.

[0195] Example 1: The image is encoded and decoded in a fixed block division manner.

[0196] As shown in Figure 7, Figure 7 is a schematic diagram of an image encoding and decoding method using a fixed block partitioning method provided by this application. This image encoding and decoding method can be applied to the video transmission system shown in Figure 1 or the video encoding and decoding system shown in Figure 2. The following describes the method flow on the encoding side and the decoding side respectively.

[0197] On the encoding side, the encoding device divides the image to be encoded into multiple sub-images of fixed size and marks feature values ​​(including: identification, image size, position, etc.) for each sub-image, such as ID11 to ID44 in FIG7 .

[0198] In addition, the encoding device performs CDC search matching on multiple sub-images with the external cache queue to determine matched sub-images and unmatched sub-images. The matched sub-images include ID11, ID21, ID31, ID41, ID14, ID24, ID34 and ID44 in Figure 7, and the unmatched sub-images include ID12, ID22, ID32, ID42, ID13, ID23, ID33 and ID43 in Figure 7.

[0199] Afterwards, the encoding device outputs the feature value of the matched sub-image and fills the reference pixels in the area of ​​the matched sub-image, encodes the filled image to be encoded, and outputs a first code stream; the first code stream includes the complete image encoding data of the unmatched sub-image and the image encoding data after the matched sub-image is filled with reference pixels.

[0200] Finally, the encoding device packages the feature values ​​of the matched sub-image and the first code stream, and sends the packaged packages to the decoding device.

[0201] In addition, the encoding device also writes the feature values ​​of the unmatched sub-images into an external cache queue, so that subsequent frame images can reuse the image encoding data of these unmatched sub-images, thereby reducing the encoding bandwidth of the video encoding.

[0202] It is worth noting that the encoding device may also use a low bit rate to encode the image region of the unmatched sub-image, thereby reducing the data volume of the aforementioned first code stream.

[0203] The difference between the decoding and encoding sides is that after the decoding device identifies a matching sub-image in the first image and a decoded sub-image in the external cache queue, it fills the image area of ​​the matching sub-image with the image content of the decoded sub-image. The information required for the filling process, such as image size and position, is obtained from the external cache queue.

[0204] Example 2: An image is encoded and decoded in an adaptive partitioning manner.

[0205] As shown in Figure 8, Figure 8 is a schematic diagram of an image encoding and decoding method using an adaptive division method provided by the present application. The image encoding and decoding method can be applied to the video transmission system shown in Figure 1 or the video encoding and decoding system shown in Figure 2. The difference between Figure 8 and Figure 7 is that the images obtained by the division method are different. For example, the matched sub-images in Figure 8 include: ID11, ID31, ID411, ID412, ID413, ID414, ID141, ID142, ID143, and ID34; the other sub-images are unmatched sub-images. The other contents are consistent with those in Figure 7 and will not be repeated in this application.

[0206] In combination with the embodiments of Figures 2 to 8, the image coding and decoding method provided by the present application can be applied to the following coding and decoding scenarios: 1. Fixed block division + reference pixel filling + standard hardware video codec; 2. Adaptive division + reference pixel filling + standard hardware video codec; 3. Fixed block division + low bit rate encoding + standard hardware video codec; 4. Adaptive division + low bit rate encoding + standard hardware video codec, etc.

[0207] Compared with the encoding and decoding methods provided by conventional technologies in screen content scenarios such as video conferencing and desktop sharing, the image encoding and decoding method provided by this application performs repeated image matching outside the encoder on the encoding side (original image), establishes the matching results (image size, coordinates, identification, matching status, etc.) into an external cache queue, and sends it to the decoding side in the form of some instructions. The decoding side establishes an external cache queue in the same way, traverses all image information of the entire frame, and if it is a matching image, fills the previously decoded image from the external cache queue into the corresponding position; if it is an unmatched image, retains the original decoding result and updates the queue. In addition, the compression rate of the general video encoder in the screen content scenario is improved, the encoding bandwidth is reduced, and the encoding efficiency is improved.

[0208] It is worth noting that the image encoding and decoding method provided in this application can also be applied to screen content scenarios such as cloud conferencing, cloud desktop, cloud gaming, live video, and desktop extension (Xtended, XR).

[0209] It is understood that in order to implement the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. It should be readily apparent to those skilled in the art that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in hardware or in a manner driven by computer software depends on the specific application scenario and design constraints of the technical solution.

[0210] The image encoding and decoding method provided by this embodiment is described in detail above with reference to FIG. 1 to FIG. 8 . The encoding and decoding apparatus provided by this embodiment will be described below with reference to FIG. 9 .

[0211] Figure 9 is a schematic diagram of the structure of a codec device provided by this application. These codec devices can be used to implement the functions of the encoding device and decoding device in the above method embodiments, thereby also achieving the beneficial effects of the above method embodiments. In this embodiment, the codec device can be the encoding device 210 and decoding device 220 shown in Figure 2, or it can be a module (such as a chip) applied to a computing device.

[0212] As shown in Figure 9, the encoding and decoding apparatus 900 includes a communication module 910, a context-aware deduplication module 920, and an encoding and decoding module 930. The encoding and decoding apparatus 900 is used to implement the functions of the encoding device and the decoding device in the method embodiments shown in the figures.

[0213] When the encoding and decoding apparatus 900 is used to implement the functions of an encoding device, the specific functions of each module are as follows.

[0214] The communication module 910 is configured to obtain a first image to be encoded in a video.

[0215] The context-aware deduplication module 920 is used to divide the first image into multiple sub-images, and search and match the multiple sub-images with the feature values ​​(image instruction information) in the queue to determine the first category of sub-images and the second category of sub-images.

[0216] The encoding and decoding module 930 is configured to generate first information of a first type of sub-image and encode a second type of sub-image to obtain second information.

[0217] The communication module 910 is further configured to output encoded data including the first information and the second information.

[0218] When the encoding and decoding apparatus 900 is used to implement the functions of a decoding device, the specific functions of each module are as follows.

[0219] The communication module 910 is used to obtain a code stream, which includes: coded data of at least one frame of image and data (feature values) stored in a queue used for the coded image.

[0220] Context-aware deduplication module 920 is used to separate the data in the bitstream to obtain a video stream and feature values. After decoding the video stream by codec module 930, the first image is divided into multiple sub-images. These sub-images are then searched and matched with the feature values ​​(image instruction information) in the queue to determine the first and second categories of sub-images. Furthermore, context-aware deduplication module 920 fills the image content of the decoded sub-images in the queue into the image area corresponding to the first category of sub-images, thereby restoring the reconstructed image of the first image.

[0221] It should be understood that the codec device 900 of the embodiment of the present application can be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Alternatively, when the method shown in FIG. 3 is implemented using software, the various modules thereof can also be software modules, and the various modules of the codec device 900 can also be software modules.

[0222] According to the embodiment of the present application, the coding and decoding device 900 can correspond to the execution of the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of each unit in the coding and decoding device 900 are respectively for implementing the corresponding processes of each method in the aforementioned figures. For the sake of brevity, they are not repeated here.

[0223] Figure 10 is a schematic diagram of the structure of an image processing system provided in this application. This image processing system is described using a mobile phone as an example. The mobile phone or a chip system built into the mobile phone includes: memory 1010, processor 1020, sensor component 1030, multimedia component 1040, and input / output interface 1050. The following detailed description of the various components of the mobile phone or the chip system built into the mobile phone is provided in conjunction with Figure 10.

[0224] Memory 1010 can be used to store data, software programs, and modules. It primarily includes a program storage area and a data storage area. The program storage area can store software programs, including coded instructions, including but not limited to an operating system and applications required for at least one function, such as sound playback and image playback. The data storage area can store data generated based on the use of the mobile phone, such as audio data, image data, and a phone book. In embodiments of the present application, memory 1010 can be used to store, for example, the number of bits of data in a bitstream buffer. In some feasible embodiments, there may be one or more memory devices. Such memory devices may include floppy disks, hard disks such as internal hard disks and removable hard disks, magnetic disks, optical disks, magneto-optical disks such as CD-ROMs and DCD-ROMs, non-volatile storage devices such as RAM, ROM, PROM, EPROM, EEPROM, flash memory, or any other form of storage medium known in the art.

[0225] Processor 1020 is the control center of the mobile phone. It connects all parts of the device using various interfaces and circuits. By running or executing software programs and / or software modules stored in memory 1010 and accessing data stored in memory 1010, it performs various functions of the mobile phone and processes data, thereby providing overall monitoring of the mobile phone. In embodiments of the present application, processor 1020 can be used to perform one or more steps in the method embodiments of the present application. For example, processor 1020 can be used to perform one or more steps S420 to S470 in the following method embodiments. In some feasible embodiments, processor 1020 can have a single processor structure, a multi-processor structure, a single-threaded processor, a multi-threaded processor, etc. In some feasible embodiments, processor 1020 can include at least one of a central processing unit, a general-purpose processor, a digital signal processor, a neural network processor, an image processing unit, an image signal processor, a microcontroller, or a microprocessor. In addition, processor 1020 can further include other hardware circuits or accelerators, such as application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 1020 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like.

[0226] The sensor component 1030 includes one or more sensors for providing status assessments of various aspects of the mobile phone. The sensor component 1030 may include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications, i.e., as a component of a camera or a camera head. In an embodiment of the present application, the sensor component 1030 may be used to support the camera in the multimedia component 1040 in acquiring images, etc. In addition, the sensor component 1030 may also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor. The sensor component 1030 may detect the acceleration / deceleration, orientation, open / close status of the mobile phone, the relative positioning of components, or temperature changes of the mobile phone, etc.

[0227] The multimedia component 1040 provides a screen as an output interface between the mobile phone and the user. The screen can be a touch panel, and when the screen is a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In addition, the multimedia component 1040 also includes at least one camera. For example, the multimedia component 1040 includes a front camera and / or a rear camera. When the mobile phone is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can sense external multimedia signals, which are used to form image frames. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0228] The input / output interface 1050 provides an interface between the processor 1020 and a peripheral interface module. For example, the peripheral interface module may include a keyboard, a mouse, or a USB (Universal Serial Bus) device. In one possible implementation, the input / output interface 1050 may have only one input / output interface or multiple input / output interfaces.

[0229] Although not shown, the mobile phone may further include an audio component and a communication component, for example, the audio component includes a microphone, and the communication component includes a wireless fidelity (WiFi) module, a Bluetooth module, etc., which will not be described in detail in the embodiments of the present application.

[0230] The image processing system can be a general-purpose device or a dedicated device. For example, the image processing system can be an edge device (e.g., a box carrying a chip with processing capabilities). Alternatively, the image processing system can be a server or other device with computing capabilities. In some possible scenarios, the image processing system can also refer to the aforementioned video codec system.

[0231] It should be understood that the image processing system according to this embodiment may correspond to the codec device 900 in this embodiment, and may correspond to executing the corresponding subject according to any of the methods in the aforementioned figures, and the above-mentioned and other operations and / or functions of each module in the codec device 900 are respectively for implementing the corresponding processes of each method in the aforementioned figures. For the sake of brevity, they will not be repeated here.

[0232] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and storage medium can also exist as discrete components in a network device or a terminal device.

[0233] The present application also provides a chip system, which includes a processor for implementing the functions of the data processing unit in the above method. In one possible design, the chip system also includes a memory for storing program instructions and / or data. The chip system can be composed of a chip alone or can include a chip and other discrete devices.

[0234] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is performed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD).

[0235] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image coding method, characterized in that: The method comprises: Acquire a first image, wherein the first image includes a first type of sub-image and / or a second type of sub-image, a size of each sub-image of the first type of sub-image and / or the second type of sub-image is related to a matching condition of a feature value of an encoded sub-image, and an image to which the encoded sub-image belongs belongs to the same image sequence as the first image; The first type of sub-image matches the feature value of the encoded sub-image, and the second type of sub-image does not match the feature value of the encoded sub-image; Updating original pixel values ​​of the first type of sub-images; The updated first-category sub-image and the second-category sub-image are encoded to obtain a code stream.

2. The method according to claim 1, characterized in that The characteristic value of the encoded sub-image includes any one or a combination of the following: image size, position, identification and matching status of the encoded sub-image, and the matching status indicates whether the encoded sub-image matches the sub-image to be encoded.

3. The method according to claim 2, characterized in that The identifier of the encoded sub-image is: part of the original pixel values ​​of the encoded sub-image, or a value obtained by processing the part of the original pixel values ​​according to a reference method; the reference method includes at least one of a hash algorithm and a cyclic redundancy check algorithm.

4. The method according to any one of claims 1 to 3, characterized in that The updating of the original pixel values ​​of the first-type sub-images includes: Filling the first-category sub-image with reference pixels to obtain a filled first-category sub-image; The encoding of the updated sub-images of the first category and the sub-images of the second category to obtain a code stream includes: Encoding the padded first-type sub-image to obtain first information, where the first information includes a feature value of the first-type sub-image; encoding the second type of sub-image to obtain second information; The code stream is obtained according to the first information and the second information.

5. The method according to claim 4, characterized in that The first information also includes: second data obtained by encoding the first type of sub-image using a first code rate, where the first code rate is less than or equal to a code rate threshold.

6. The method according to any one of claims 1 to 5, characterized in that The first image includes the second type of sub-image, and after acquiring the first image, the method further includes: Obtaining a feature value of each sub-image in the second type of sub-image; The characteristic value of each sub-image in the second type of sub-image is cached.

7. The method according to claim 6, characterized in that The method further comprises: Send the cached characteristic value of each sub-image in the second type of sub-image, where the characteristic value of each sub-image in the second type of sub-image is carried in any one of the following: a code stream parameter set, a transport protocol header, a real-time transport protocol RTP payload, or a file format.

8. The method according to any one of claims 1 to 7, characterized in that The first type of sub-images and the second type of sub-images have the same image size.

9. The method according to any one of claims 1 to 8, characterized in that The size of each sub-image is determined according to image texture information of the first image.

10. The method according to any one of claims 1 to 9, characterized in that After obtaining the code stream, the method further includes: The code stream is sent.

11. An image decoding method, characterized in that: The method comprises: Acquire a code stream, wherein the code stream includes a code stream of at least one frame of image; Acquire a first image according to the bitstream, the first image comprising a first type of sub-image and / or a second type of sub-image, the size of each sub-image of the first type of sub-image and / or the second type of sub-image is related to a matching condition of a feature value of a decoded sub-image, and the image to which the decoded sub-image belongs belongs to the same image sequence as the first image; wherein the first type of sub-image matches the characteristic value of the decoded sub-image, and the second type of sub-image does not match the characteristic value of the decoded sub-image; An image region corresponding to the first type of sub-image in the first image is restored according to the characteristic value of the decoded sub-image to obtain a reconstructed image.

12. The method according to claim 11, characterized in that The characteristic value of the decoded sub-image includes any one or a combination of the following: image size, position, identification and matching status of the decoded sub-image, and the matching status indicates whether the decoded sub-image matches the sub-image to be decoded.

13. The method according to claim 12, characterized in that The identifier of the decoded sub-image is: part of the original pixel values ​​of the decoded sub-image, or a value obtained by processing the part of the original pixel values ​​according to a reference method; the reference method includes at least one of a hash algorithm and a cyclic redundancy check algorithm.

14. The method according to any one of claims 11 to 13, characterized in that The first type of sub-images and the second type of sub-images have the same image size.

15. The method according to any one of claims 11 to 14, characterized in that The code stream also includes a characteristic value of each sub-image in the second category of sub-images, and the characteristic value of each sub-image in the second category of sub-images is carried in any one of the following: a code stream parameter set, a transport protocol header, a real-time transport protocol RTP payload or a file format.

16. A coding and decoding device, characterized in that: The encoding and decoding device includes a device for executing the method according to any one of claims 1-15.

17. A coding device, characterized in that: The encoding device comprises at least one processor and a memory, wherein the memory is used to store a computer program, so that when the computer program is executed by the at least one processor, the method according to any one of claims 1 to 10 is implemented.

18. A decoding device, characterized in that: The decoding device comprises at least one processor and a memory, wherein the memory is used to store a computer program, so that when the computer program is executed by the at least one processor, the method according to any one of claims 11 to 15 is implemented.

19. A coding and decoding system, characterized in that: The encoding and decoding system includes an encoding device as described in claim 17 and a decoding device as described in claim 18, wherein the encoding device is used to perform the operating steps of the method described in any one of claims 1-10, and the decoding device is used to perform the operating steps of the method described in any one of claims 11-15.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer software instructions. When the computer software instructions are executed in a computer, the computer executes the method according to any one of claims 1 to 15.

Images