Image coding and decoding method and device

Abstract

The invention discloses an image encoding and decoding method and device, and relates to the technical field of video processing. The coding device classifies the sub-images according to the feature values of the sub-images of the image to be coded, and due to the fact that the feature values of different sub-images are different, after the sub-images are matched with the coded sub-images in the cached queue, the matched sub-images and the unmatched sub-images can be determined, and therefore the coding efficiency is improved. The image coding data of the matched sub-image in the to-be-coded image in the previous frame image can be multiplexed under the condition that the matched sub-image of the to-be-coded image is not the same as the matched sub-image in the previous frame image in position and content. Furthermore, in the coding process of the to-be-coded image, coding processing (such as standard coding) is performed on the unmatched sub-image, and information of the matched coded sub-image is generated for the matched sub-image, so that the problem of repeated coding of effective pixels in the matched sub-image is avoided, and the coding bandwidth of video coding is reduced. And the coding efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to an image encoding and decoding method and device. Background Art

[0002] With the development of big data, artificial intelligence (AI), and cloud computing technology, a large amount of data is constantly generated. In order to reduce the cost of data storage, compression devices can use hardware encoding algorithms to compress image data, that is, reconstruct image data according to a specific method, thereby reducing data redundancy and reducing the storage capacity occupied by image data.

[0003] During the video encoding process, the compression device compares the current frame image with the adjacent previous frame image, encodes the different image areas in the current frame and the previous frame, and does not encode the image areas with the same position and content, that is, the current frame reuses the image encoding data with the same position and content in the adjacent previous frame. However, for scenes such as page scrolling and page turning that often occur in desktop scenes, the encoding bandwidth of video encoding is large and the encoding efficiency is low. Summary of the invention

[0004] The present application provides an image encoding and decoding method and device, which solves the problem of large encoding bandwidth and low encoding efficiency of video encoding in desktop scenarios.

[0005] In a first aspect, the present application provides an image encoding method. The image encoding method is applied to a codec system, and the image encoding method is executed by an encoding device included in the codec system. The image encoding method includes: the encoding device obtains a first image to be encoded and divides the first image into multiple sub-images, the encoding device matches the multiple sub-images with a cached queue, and determines a first type of sub-image and a second type of sub-image; wherein the queue includes one or more feature values ​​of encoded sub-images, the image to which the one or more encoded sub-images belong belongs to the same stream as the first image, the first type of sub-image matches the feature value of the encoded sub-image in the queue, and the queue does not record the feature value of the second type of sub-image. And, the encoding device generates first information of the first type of sub-image, and after encoding the second type of sub-image to obtain the second information, outputs the encoded data of the first image. The first information indicates: the encoded sub-image in the queue that matches the first type of sub-image, the output encoded data includes the aforementioned first information and second information.

[0006] Compared with the current frame multiplexing the image coding data of the same position and content in the previous frame, in the present application, the coding device classifies the sub-images of the image to be coded according to the feature values ​​of the sub-images. Since the feature values ​​of different sub-images are different, after matching the sub-images with the coded sub-images in the cached queue, the matched sub-images (such as the first type of sub-images) and the unmatched sub-images (such as the second type of sub-images) can be determined. Therefore, the matched sub-images of the image to be coded do not need to be in the same position and content as in the previous frame image in order to reuse the image coding data of the matched sub-images in the previous frame image with the image to be coded. Furthermore, in the coding process of the image to be coded, the unmatched sub-images are coded (such as standard coding), and the information of the coded sub-images (first information) matching with the matched sub-images is generated for the matched sub-images, thereby avoiding the problem of repeated coding of valid pixels in the matched sub-images and reducing the coding bandwidth of the video coding. The coding efficiency is improved.

[0007] In combination with the image encoding method provided in the first aspect, in an optional implementation, the characteristic values ​​of the encoded sub-images stored in the queue may include any one or a combination of the following items: image size, position, identification and matching status of the encoded sub-images, the matching status indicating whether the encoded sub-image matches the sub-image to be encoded.

[0008] As an optional implementation, the identifier of the encoded sub-image is: part of the original pixel values ​​of the encoded sub-image, or a value obtained by processing part of the original pixel values ​​in a reference manner. The reference manner includes at least one of a hash algorithm and a cyclic redundancy check algorithm.

[0009] In the present application, the encoding device stores the characteristic values ​​of the encoded sub-image (such as the above-mentioned image size, position, identification and matching status) in an independent queue, and distinguishes the sub-images of the image to be encoded based on the relevant information stored in the queue, so that the encoding device can reuse the image encoding data of the sub-images in the previous multiple frames of images, reducing the repeated encoding process of valid pixels in sub-images with the same content, which is conducive to reducing the encoding bandwidth of video encoding and improving the encoding efficiency. Moreover, the queue does not need to store the complete image encoding data of the encoded sub-image, that is, when the first encoding data of the encoded sub-image is stored in the encoded bit stream, compared with the image encoding data obtained by repeatedly encoding the sub-image with the same content, the queue occupies a smaller amount of data, and the encoding device only needs to provide a small amount of storage space to implement the relevant functions of the queue, which is conducive to reducing the encoding bandwidth of video encoding and improving the encoding efficiency.

[0010] In combination with the image encoding method provided in the first aspect, in an optional implementation, the encoding device generates first information of a first-category sub-image, including: the encoding device obtains a characteristic value of the first-category sub-image, and fills reference pixels for the first-category sub-image, and encodes the filled first-category sub-image to obtain first data; the encoding device obtains the first information according to the first data and the characteristic value of the first-category sub-image. Exemplarily, the reference pixel may refer to a preset fixed pixel value, and the encoding bandwidth occupied by the fixed pixel value is small, such as all pixels in the filled first-category sub-image are the same, or the pixels in the filled first-category sub-image conform to a certain rule, and the certain rule makes the amount of encoded data small. Furthermore, the amount of data after the encoding device outputs the encoded video containing the image to be encoded is reduced, which reduces the encoding bandwidth of the video encoding, which is conducive to improving the encoding efficiency and reducing the bandwidth occupied by data transmission.

[0011] As an optional implementation, the first information includes: a feature value of a first type of sub-image, and second data obtained by encoding the first type of sub-image using a first bit rate. The first bit rate is less than or equal to a bit rate threshold. For a matched sub-image (first type of sub-image), the encoding device encodes the matched sub-image at a lower bit rate, so that the amount of encoded data in the partial image area is small, thereby reducing the encoding bandwidth of the video encoding and improving the encoding efficiency.

[0012] In combination with the image encoding method provided in the first aspect, in an optional implementation, after the encoding device determines the first category of sub-images and the second category of sub-images, the image encoding method provided by the present application further includes: the encoding device generates a feature value for each sub-image in the second category of sub-images, and writes the feature value for each sub-image in the second category of sub-images into a queue. It is worth noting that the encoding device writes the feature value of the unmatched sub-image encoded for the first time into the queue. When the subsequent frame image of the image in the video also has other sub-images with the same content as the unmatched sub-image, the other sub-images can reuse the image encoding data of the sub-image whose feature value has been added to the queue, thereby reducing the encoding bandwidth of multiple frames of images in the video and improving the encoding efficiency.

[0013] In combination with the image encoding method provided in the first aspect, in an optional implementation, the image sizes of the multiple sub-images are consistent. Different frame images in the video are divided in the same way, so that each encoded sub-image and the sub-image to be encoded only need to be compared based on the characteristic value, and then the image encoding data of the encoded sub-images with the same characteristic value are reused, which is conducive to reducing the encoding bandwidth of the video encoding and improving the encoding efficiency.

[0014] In combination with the image encoding method provided in the first aspect, in an optional implementation, the encoding device divides the first image into multiple sub-images, including: the encoding device obtains multiple image features of the first image, and selects at least one division method from a plurality of preset division methods according to the multiple image features; the aforementioned at least one image feature is used to describe the image texture information of the first image, and at least one division method matches the image texture information. And, the encoding device divides the first image according to at least one division method to obtain multiple sub-images. The encoding device adaptively selects different division methods for images of different frames according to the image texture information of different frame images, so that each divided sub-image matches the image texture, which is conducive to improving the encoding effect of the image content contained in the sub-image, thereby improving the encoding accuracy of the image. Moreover, when the same content appears repeatedly in different frames of video images, due to the matching of the image content and the image texture, the sub-image to be encoded can be more easily matched to the feature value of the encoded sub-image, and then reuse the image encoding data of the encoded sub-image, thereby reducing the encoding bandwidth of the video encoding, which is conducive to improving the encoding efficiency of the video encoding.

[0015] In conjunction with the image encoding method provided in the first aspect, in an optional implementation, the encoding device outputs the encoded data of the first image, including: the encoding device sends the encoded data of the first image. Exemplarily, the encoding device may store the encoded data of the first image in a memory of the encoding and decoding system, or send the encoded data of the first image to a decoding device, so that the decoding device reconstructs and restores the image according to the encoded data.

[0016] In combination with the image encoding method provided in the first aspect, in an optional implementation method, the image encoding method provided by the present application also includes: the encoding device sends the data stored in the queue, and the data stored in the queue is carried in a code stream parameter set, a transport protocol header, a real-time transport protocol (RTP) payload or a file format. During the data communication process, in addition to transmitting the encoded encoded data (such as a bit stream), the encoding device also transmits the data stored in the introduced queue, so that the decoding device can quickly reconstruct and restore the image according to the data stored in the queue, thereby improving the decoding efficiency. Moreover, since the amount of data stored in the queue is small, it can be carried in one of the above optional methods, so that the bandwidth required for communication between the encoding device and the decoding device is reduced, which is conducive to improving the communication efficiency of the encoding and decoding system and reducing bandwidth consumption.

[0017] In a second aspect, the present application provides an image decoding method. The image encoding method is applied to a coding and decoding system, and the image encoding method is executed by a decoding device included in the coding and decoding system. The image decoding method includes: the decoding device obtains a bit stream, and decodes the bit stream according to the encoded data to obtain a first image; the bit stream includes the encoded data of at least one frame of the image and the data stored in the queue used by the encoded image. And, the decoding device matches the data stored in the queue in the bit stream with the first image, determines the first type of sub-image and the second type of sub-image in the first image, and restores the image area corresponding to the first type of sub-image in the first image according to the feature value matching the first type of sub-image in the queue to obtain a reconstructed image. Wherein, the queue includes the feature values ​​of one or more decoded sub-images, the image to which the one or more decoded sub-images belong belongs to the same stream as the first image, the first type of sub-image matches the feature value of the decoded sub-image in the queue, and the feature value of the second type of sub-image is not recorded in the queue.

[0018] Compared with the current frame multiplexing the image decoding data of the same position and content in the previous frame, in the present application, the decoding device classifies the sub-images of the first image according to the characteristic values ​​of the sub-images. Since the characteristic values ​​of different sub-images are different, after matching the sub-images with the decoded sub-images in the cached queue, the matched sub-images (such as the first type of sub-images) and the unmatched sub-images (such as the second type of sub-images) can be determined. Therefore, the matched sub-images of the first image do not need to be in the same position and content as in the previous frame image in order to reuse the image decoding data of the matched sub-images in the previous frame image and the image to be decoded. Furthermore, in the decoding process of the first image, the unmatched sub-images are decoded (such as standard decoding), and the information of the matched decoded sub-images is multiplexed for the matched sub-images, thereby avoiding the problem of repeated decoding of valid pixels in the matched sub-images and reducing the decoding bandwidth of the video decoding. The decoding efficiency is improved.

[0019] In combination with the image decoding method provided in the second aspect, in an optional implementation, the characteristic value of the decoded sub-image includes any one or a combination of the following items: image size, position, identification and matching status of the decoded sub-image, and the matching status indicates whether the decoded sub-image matches the sub-image to be decoded.

[0020] In combination with the image decoding method provided in the second aspect, in an optional implementation, the identifier of the decoded sub-image is: part of the original pixel value of the decoded sub-image, or, a value obtained by processing part of the original pixel value according to a reference method; the reference method includes at least one of a hash algorithm and a cyclic redundancy check algorithm.

[0021] In combination with the image decoding method provided in the second aspect, in an optional implementation manner, the image sizes of the multiple sub-images are consistent.

[0022] In combination with the image decoding method provided in the second aspect, in an optional implementation manner, the data stored in the aforementioned queue is carried in a bitstream parameter set, a transport protocol header, a real-time transport protocol RTP payload or a file format.

[0023] In a third aspect, the present application provides a coding and decoding device, which includes modules for executing the first aspect or any possible design of the method of the first aspect, and modules for executing the second aspect or any possible design of the method of the second aspect.

[0024] In a fourth aspect, the present application provides a coding device. The coding device includes at least one processor and a memory, wherein the memory is used to store a computer program, so that when the computer program is executed by at least one processor, the method described in the first aspect or any possible design of the first aspect is implemented.

[0025] In a fifth aspect, the present application provides a decoding device. The decoding device includes at least one processor and a memory, wherein the memory is used to store a computer program, so that when the computer program is executed by at least one processor, the method described in the second aspect or any possible design of the second aspect is implemented.

[0026] In a sixth aspect, the present application provides a coding and decoding system, which includes the coding device described in the fourth aspect and the decoding device described in the fifth aspect.

[0027] In a seventh aspect, the present application provides a chip. The chip includes: a processor and a power supply circuit. The power supply circuit is used to supply power to the processor; the processor is used to execute the operation steps of the method in the first aspect or any possible implementation of the first aspect, and execute the operation steps of the method in the second aspect or any possible implementation of the second aspect.

[0028] In an eighth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes: computer software instructions. When the computer software instructions are executed in a computer, the computer executes the operation steps of the method in the first aspect or any possible implementation of the first aspect, and executes the operation steps of the method described in the second aspect or any possible implementation of the second aspect. For example, the computer is the aforementioned encoding device or decoding device.

[0029] In a ninth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computing device executes the operation steps of the method in the first aspect or any possible implementation of the first aspect, and executes the operation steps of the method described in the second aspect or any possible implementation of the second aspect. For example, the computer is the aforementioned encoding device or decoding device.

[0030] The beneficial effects of the third to ninth aspects above can be referred to the description of any implementation in the first or second aspect, and will not be repeated here. Based on the implementation provided by the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a video transmission system provided for this application;

[0032] Figure 2 A schematic diagram of a video encoding and decoding system provided in this application;

[0033] Figure 3 A schematic diagram of the process of an image encoding method provided in this application Figure 1 ;

[0034] Figure 4 A schematic diagram of the process of an image encoding method provided in this application Figure 2 ;

[0035] Figure 5 A schematic diagram of the process of an image encoding method provided in this application Figure 3 ;

[0036] Figure 6 A schematic diagram of the process flow of the image encoding and decoding method provided by this application;

[0037] Figure 7 A schematic diagram of an image encoding and decoding method using a fixed block division method provided by the present application;

[0038] Figure 8 A schematic diagram of an image encoding and decoding method using an adaptive partitioning method provided by the present application;

[0039] Fig. 9 A schematic diagram of the structure of a coding and decoding device provided by this application;

[0040] Fig.10 A schematic diagram of the structure of an image processing system provided in this application. DETAILED DESCRIPTION

[0041] The technical solutions involved in this application may not only be used in combination with video coding standards (such as H.264, HEVC and other standards), but may also be used in combination with future video coding standards (such as H.266 standards). The terms used in the implementation method section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application. The following is a brief introduction to some concepts that may be involved in this application.

[0042] Video contains multiple continuous images. When the continuous images change at more than 24 frames per second, according to the principle of visual persistence, the human eye cannot distinguish a single static image, and the multiple images that appear smooth and continuous are called video.

[0043] Video coding refers to the processing of a sequence of pictures to form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. Video coding used in this article means video encoding or video decoding. Video coding is performed on the source side, and generally includes processing (e.g., compressing) the original video picture to reduce the amount of data required to represent the video picture under the condition of meeting a certain image quality, so as to store and / or transmit it more efficiently. Video decoding is performed on the destination side, and generally includes inverse processing relative to the encoder to reconstruct the video picture. The "coding" of the video picture involved in the embodiment should be understood as "coding" or "decoding" involving a video sequence. The combination of the encoding part and the decoding part is also called codec (encoding and decoding). Video coding can also be called image coding (image coding) also known as image compression (image compression). Image decoding (image decoding) refers to the inverse process of image coding.

[0044] Video sequences include a series of pictures, which are further divided into slices, and slices are further divided into blocks. Video coding is performed in blocks. In some new video coding standards, the concept of blocks is further expanded. For example, in the H.264 standard, there are macroblocks (MB), which can be further divided into multiple prediction blocks (partitions) that can be used for predictive coding. In the high-performance video coding (HEVC) standard, basic concepts such as coding units (CU), prediction units (PU) and transform units (TU) are used to functionally divide various block units and describe them using a new tree-based structure. For example, CU can be divided into smaller CUs according to the quadtree, and the smaller CU can be further divided to form a quadtree structure. CU is the basic unit for dividing and encoding the coded image. There are similar tree structures for PU and TU. PU can correspond to prediction blocks and is the basic unit of predictive coding. CU is further divided into multiple PUs according to the division mode. TU may correspond to a transform block, which is a basic unit for transforming a prediction residual. However, whether CU, PU or TU, they all essentially belong to the concept of a block (or coding unit).

[0045] For example, in HEVC, a CTU is split into multiple CUs using a quadtree structure represented as a coding tree. A decision is made at the CU level whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode a picture area. Each CU can be further split into one, two or four PUs according to the PU split type. The same prediction process is applied within a PU, and relevant information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU split type, the CU can be divided into transform units (TUs) according to other quadtree structures similar to the coding tree for the CU. In the latest developments in video compression technology, quadtree and binarytree (QTBT) are used to split frames to split coding blocks. In the QTBT block structure, the CU can be square or rectangular in shape.

[0046] Lossless video coding means that the original video picture can be reconstructed, that is, the reconstructed video picture has the same quality as the original video picture (assuming there is no transmission loss or other data loss during storage or transmission).

[0047] Lossy video coding refers to reducing the number of bits required to represent a video picture by performing further compression, such as quantization, while the decoder side cannot fully reconstruct the video picture, that is, the quality of the reconstructed video picture is lower or worse than the quality of the original video picture.

[0048] Bitstream (Data Rate) refers to the binary stream generated after encoding an image or video. Bitstream is also called code stream or code rate, which is the number of bits transmitted per unit time. It is an important part of picture quality control in image coding. For images of the same resolution, the larger the bitstream of the image, the smaller the compression ratio and the better the picture quality.

[0049] Bit rate: Video bit rate is the amount of data per unit time obtained by compression during video encoding, and can also be considered as the amount of data transmitted per unit time during transmission. Generally speaking, when encoding the same video source, a high bit rate will result in a high-quality image, while a low bit rate will result in a low-quality image.

[0050] Bit rate control refers to the function of adjusting the bit rate during the encoding and decoding process, which is abbreviated as bit rate control below. Bit rate control modes include constant bit rate (CBR) and variable bit rate (VBR). Constant bit rate means ensuring a stable bit rate within the bit rate statistical time; dynamic bit rate means allowing bit rate fluctuations within the bit rate statistical time to ensure stable image quality after encoding.

[0051] Quantization refers to the process of mapping the continuous value of a signal into multiple discrete amplitudes.

[0052] The quantization parameter (QP) is used to quantize the residual value generated by the prediction operation or the coefficient generated by the transform operation during the encoding process. During the decoding process, the syntax elements are inversely quantized to obtain the residual value or coefficient. The quantization parameter is a parameter used in the quantization process. Generally, the larger the value of the quantization parameter, the more obvious the quantization degree, the worse the quality of the reconstructed image, and the lower the bit rate; conversely, the smaller the value of the quantization parameter, the better the quality of the reconstructed image, and the higher the bit rate.

[0053] I frame (Intra frame, IDR frame): In the field of video coding, the coding prediction method of all blocks in a frame is intra-frame prediction, which can be decoded independently without relying on other frames, and the frame is marked as an I frame type frame in the bitstream, generally referred to as "I frame".

[0054] Predictive-frame (P frame): In the field of video coding, the encoding of a frame needs to refer to the previous frame, and the frame is marked as a P frame type frame in the bitstream, generally referred to as a "P frame".

[0055] Bi-directional interpolated prediction frame (B frame): In the field of video coding, the encoding of a frame requires reference to a forward frame and a backward frame, and the frame is marked as a B frame type frame in the bitstream, generally referred to as a "B frame".

[0056] In the field of coding and decoding, the above I frames, P frames and B frames can be collectively referred to as IPB frames.

[0057] Reference frame: The frame that the IPB frame needs to refer to during predictive coding is its reference frame.

[0058] Reconstructed frame: The video frame generated by the encoder after the complete encoding process is a reconstructed frame, which is usually used as a reference frame for subsequent frames. The reconstructed frame of the encoder is equivalent to the frame generated by the decoder after decoding.

[0059] Parameter Set: The parameter set contains the global information of the corresponding coded image. The Video Parameter Set (VPS) contains the video classification information, which is conducive to the extension of compatible standards in scalable video coding or multi-view video coding. The Sequence Parameter Set (SPS) contains the parameters for a continuous coded video sequence. The Picture Parameter Set (PPS) corresponds to a certain image or several images in a sequence.

[0060] The implementation of the image encoding and decoding method provided by the present application is described below with reference to the accompanying drawings.

[0061] Figure 1 A schematic diagram of a video transmission system provided in the present application, wherein the video processing process includes video acquisition, video encoding, video transmission, video decoding and display process, and the video transmission system includes multiple terminal devices (such as Figure 1 The terminal devices 111 to 115 are shown and a network, wherein the network can realize the function of video transmission, and the network may include one or more network devices, which may be routers or switches, etc.

[0062] Figure 1 The terminal device shown may be, but is not limited to, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The terminal device may be a mobile phone (such as Figure 1 ), a tablet computer, a computer with a wireless transceiver function (such as a terminal device 114 shown in Figure 1 ), a virtual reality (VR) terminal device (such as the terminal device 115 shown in Figure 1 The terminal device 113 shown in the figure), augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self driving, wireless terminal in smart city, wireless terminal in smart home, etc.

[0063] like Figure 1 As shown, in different video processing processes, the terminal devices are different.

[0064] For example, during the video acquisition process, the terminal device 111 may be a camera device for recording screen or desktop content, a mobile phone, a tablet computer or a personal computer with a video recording function, etc.

[0065] For another example, during the video encoding process, the terminal device 112 may be a server or a data center, and the data center may include one or more physical devices with encoding functions, such as a server, a mobile phone, a tablet computer or other encoding devices.

[0066] For example, during the video decoding and display process, the terminal device 113 may be an all-in-one computer, and the user may control the image range displayed by turning. The terminal device 114 may be a mobile phone, and the user may control the image range displayed on the mobile phone 114 by touch operation or air operation, etc. The terminal device 115 may be a personal computer, and the user may control the image range displayed on the display screen by input devices such as a mouse or keyboard.

[0067] It can be understood that video is a general term, and a video includes an image sequence of multiple continuous frames, where one frame corresponds to one image.

[0068] Figure 1 This is just a schematic diagram. The video transmission system may also include other devices. Figure 1 The embodiments of the present application do not limit the number and type of terminal devices included in the system.

[0069] exist Figure 1 The video transmission system shown is based on Figure 2 This is a schematic diagram of a video coding and decoding system provided by the present application. The video coding and decoding system 200 includes an encoding device 210 and a decoding device 220 . The encoding device 210 establishes a communication connection with the decoding device 220 via a communication channel 230 .

[0070] The above-mentioned encoding device 210 can realize the function of video encoding, such as Figure 1 As shown, the encoding device 210 may be the terminal device 112, and the encoding device 210 may also be a data center with video encoding capabilities, for example, the data center includes multiple servers.

[0071] The encoding device 210 may include a data source 211 , a pre-processing module 212 , an encoder 213 , a communication interface 214 , and a context-aware deduplication module 215 .

[0072] The data source 211 may include or may be any type of electronic device for capturing video, and / or any type of source video generating device, such as a computer graphics processor for generating computer animation scenes or any type of device for acquiring and / or providing source video, or computer-generated source video. The data source 211 may be any type of memory or storage for storing the above-mentioned source video. The above-mentioned source video may include multiple video streams (bit streams) or images captured by multiple video capture devices (such as cameras).

[0073] An image can be considered as a two-dimensional array or matrix of pixels (picture elements). The pixels in the array can also be called sampling points. The number of sampling points in the array or image in the horizontal and vertical directions (or axes) defines the size and / or resolution of the image. In order to represent color, three color components are usually used, that is, the image can be represented as or include three sampling arrays. For example, in the RBG format or color space, the image includes corresponding red, green and blue sampling arrays. However, in video coding, each pixel is usually represented in a brightness / chrominance format or color space, such as for an image in the YUV format, including a brightness component indicated by Y (sometimes also indicated by L) and two chrominance components indicated by U and V. The brightness (luma) component Y represents the brightness or grayscale level intensity (for example, the two are the same in a grayscale image), while the two chroma (chroma) components U and V represent the chroma or color information components. Accordingly, an image in the YUV format includes a brightness sampling array of brightness sampling values ​​(Y) and two chroma sampling arrays of chrominance values ​​(U and V). An image in RGB format can be converted or transformed into YUV format, and vice versa, which process is also referred to as color conversion or conversion. If the image is black and white, the image can only include a brightness sampling array. In the present application, the image transmitted to the encoder 213 by the data source 211 can also be referred to as raw image data or source image.

[0074] The preprocessing module 212 is used to receive a source video and preprocess the source video to obtain a preprocessed image, such as a panoramic video or a multi-frame image. For example, the preprocessing performed by the preprocessing module 212 may include color format conversion (e.g., from RGB to YCbCr), octree structuring, video splicing, etc.

[0075] The encoder 213 is used to receive the preprocessed image and encode the preprocessed image to obtain encoded data (such as a bit stream). For example, the encoder 213 may include a code control unit 2131 and an encoding unit 2132, and the code control unit 2131 is used to determine the quantization parameter used to encode each encoding unit in the current frame, so that the encoding unit 2132 predicts, quantizes and encodes the preprocessed image according to the quantization parameter to obtain a bit stream. For example, the encoder 213 matches the feature value of the sub-image with the feature value of the encoded sub-image stored in the queue, and generates instruction information of the sub-image when the feature values ​​are the same, and the instruction information is used to indicate the image encoding data of the encoded sub-image.

[0076] The communication interface 214 in the encoding device 210 can be used to receive a bit stream and send the bit stream (or a version of the bit stream after any other processing) to another device such as the decoding device 220 or any other device through the communication channel 230 to store, display or directly reconstruct the original image, etc.

[0077] The context-aware deduplication module 215 is used to identify and match sub-images in the image to be encoded that have the same content as the encoded sub-images, and generate image instruction information (or feature values) for the matched sub-images, package the image instruction information and the low-bitrate encoding data of the matched sub-image, and transmit them to the communication interface 214.

[0078] Optionally, the encoding device 210 includes a storage area, which is used to store the aforementioned queue, in which characteristic values ​​of one or more encoded sub-images are stored.

[0079] The above-mentioned decoding device 220 can realize the functions of image decoding or video decoding, such as Figure 1 As shown, the decoding device 220 may be Figure 1 Any one of the terminal devices 113 to 115 is shown.

[0080] The decoding device 220 may include a display device 221 , a post-processing module 222 , a decoder 223 , a communication interface 224 , and a context-aware deduplication module 225 .

[0081] The communication interface 224 in the decoding device 220 is used to receive a bit stream (or a version of the bit stream after any other processing is performed on the bit stream) from the encoding device 210 or from any other encoding device such as a storage device.

[0082] The communication interface 214 and the communication interface 224 can be used to send or receive bit streams through a direct communication link between the encoding device 210 and the decoding device 220, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any type of combination thereof.

[0083] The communication interface 224 corresponds to the communication interface 214, for example, and may be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain a bit stream.

[0084] The communication interface 224 and the communication interface 214 can be configured as follows: Figure 2 A unidirectional communication interface, or a bidirectional communication interface, indicated by an arrow pointing from the encoding device 210 to the corresponding communication channel 230 of the decoding device 220, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link, or data transmission such as the transmission of encoded compressed data (such as a bit stream), etc.

[0085] The decoder 223 is used to receive the encoded data and decode the encoded data to obtain decoded data (image or video, etc.). Exemplarily, the decoder 223 performs entropy decoding, dequantization and predictive reconstruction on the bit stream to obtain a reconstructed image. Among them, the decoder 223 may include a code control unit 2231 and a decoding unit 2232, and the code control unit 2231 is used to determine the quantization parameter used for decoding each coding unit in the current frame, so that the decoding unit 2232 decodes, dequantizes, and predicts and reconstructs the bit stream according to the quantization parameter to obtain the reconstructed image. Among them, the decoder 223 can determine the target number of bits of the coding unit according to the image content of the coding unit, the number of lossy bits, and the buffer fullness of the bit stream buffer, and decode the bit stream of the coding unit according to the quantization parameter determined by the target number of bits.

[0086] The context-aware deduplication module 225 is used to identify and match the sub-images in the image to be reconstructed that have the same content as the decoded sub-images, and according to the image instruction information (or feature value) of the matched sub-image, the decoded sub-image indicated by the image instruction information is decoded and filled into the corresponding position of the matched sub-image in the image to be reconstructed, and transmitted to the post-processing module 222. On the encoding side, the context-aware deduplication module is also called the context-aware deduplication coding (CADC) module; on the decoding side, the context-aware deduplication module is also called the context-aware deduplication decoding (Context-Aware Deduplication Decoding) module.

[0087] The post-processing module 222 is used to post-process the decoded data to obtain post-processed data (such as a reconstructed image to be displayed). The post-processing performed by the post-processing module 222 may include, for example, color format conversion (such as from YCbCr to RGB), octree reconstruction, video splitting and fusion, etc., or any other processing for generating data for display by the display device 221, etc.

[0088] The display device 221 is used to receive the post-processing data for display to a user or viewer, etc. The display device 221 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or display. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0089] As an optional implementation, the encoding device 210 and the decoding device 220 may transmit the encoded data via a data forwarding device, such as a router or a switch.

[0090] The functions of the above context-aware deduplication modules are to reuse the data of encoded sub-images or decoded sub-images, thereby reducing the amount of processing required by the encoder or decoder and improving the processing efficiency of the encoding and decoding process; this application does not limit the name of the context-aware deduplication module.

[0091] The structure of the above codec system is only a schematic illustration. In some possible implementations, the codec system may further include other devices. For example, the codec system may further include a terminal-side device or a cloud-side device. Figure 1 After the terminal device 111 in the example obtains the original image, the original image is preprocessed to obtain a preprocessed image; and the preprocessed image is transmitted to the terminal side device or the cloud side device (such as Figure 1 In the terminal device 112), the terminal side device or the cloud side device implements the function of encoding and decoding the pre-processed image.

[0092] It is worth noting that the standard video encoder was originally designed to compress natural videos captured by the camera, and has obvious deficiencies in some specific application scenarios (such as screen content sharing). Although the standard encoder later introduced the screen content coding (SCC) tool, the SCC tool is only implemented in software, with bottlenecks in overhead and high energy consumption. For example, in the desktop sharing scenario of video real-time communication (RTC), due to specifications and latency requirements, a standard hardware video encoder is often used. However, since the standard video encoder does not include screen content coding tools such as SCC, there are problems of low compression rate and high bandwidth overhead in such scenarios. Therefore, the solution provided in the present application can be used to solve the problem of large encoding bandwidth and low compression rate of the standard video encoder in the screen content scenario.

[0093] The image encoding and decoding method provided by the present application is applied to an encoding device and a decoding device. The image encoding method provided by the present application is first described in conjunction with the accompanying drawings. Figure 3 As shown, Figure 3 A schematic diagram of the process of an image encoding method provided in this application Figure 1 The image encoding method is performed by an encoding device 310, which may refer to Figure 2 The encoding device 210, or the encoding device 310 may also refer to Figure 2 The image encoding method provided by this embodiment includes the following steps S310 to S340.

[0094] S310. The encoding device 310 obtains a first image to be encoded, and divides the first image into a plurality of sub-images.

[0095] The first image may refer to any frame image in a video to be transmitted, and the video may be generated in a desktop sharing scenario.

[0096] In an optional implementation, the first image is divided into blocks using a fixed-size image (fixed-block division), that is, the image sizes of the multiple sub-images are consistent. Exemplarily, the preset block division method includes but is not limited to fixed-size rectangles, triangles, etc., and the identifier of each sub-image after block division is calculated according to preset rules, and the preset rules include but are not limited to pixel value calculation, feature calculation, etc.

[0097] For example, in this embodiment, the size of the sub-image after segmentation can be an enumeration of several fixed shape types, or variable values ​​such as length, width and height, or a method for obtaining the size such as specifying a rule for calculating the size (such as the difference in length, width and height with the previous sub-image).

[0098] In this embodiment, different frame images in the video are divided in the same way, so that each encoded sub-image and the sub-image to be encoded only need to be compared based on the characteristic value, and then the image encoding data of the encoded sub-image with the same characteristic value can be reused, which is beneficial to reducing the encoding bandwidth of video encoding and improving encoding efficiency.

[0099] In another optional implementation, the first image is divided into blocks adaptively (adaptively or dynamically) according to different image contents. Exemplarily, the encoding device obtains multiple image features of the first image, and selects at least one division method from a plurality of preset division methods according to the multiple image features; wherein the at least one image feature is used to describe the image texture information of the first image, and the selected at least one division method matches the image texture information. And, the encoding device divides the first image according to the selected at least one division method to obtain multiple sub-images.

[0100] For example, the preset division method (block method) includes but is not limited to rectangles, triangles, etc. of variable size. The identification of each sub-image after block division is calculated according to preset rules. The preset rules include but are not limited to pixel value calculation, feature calculation, etc.

[0101] like Figure 3 As shown, the first image is divided into sub-image 1, sub-image 2, sub-image 3, sub-image 4 and other sub-images, etc. Among them, sub-images 1 to 3 are of the same size, and sub-image 4 is inconsistent with other sub-images in the plurality of sub-images.

[0102] In this embodiment, the encoding device adaptively selects different division methods for images of different frames according to the image texture information of different frame images, so that each sub-image after division matches the image texture, which is beneficial to improving the encoding effect of the image content contained in the sub-image, and thus improving the encoding accuracy of the image.

[0103] S320 , the encoding device 310 matches the multiple sub-images with the cached queue to determine the first category of sub-images and the second category of sub-images.

[0104] As a feasible example, the storage space of the queue is provided by a memory included in the encoding device.

[0105] As another feasible example, the storage space of the queue is provided by a storage device communicating with the encoding device. In this case, the queue is also called an external buffer queue in the video encoding process.

[0106] The above memory and storage device may include but are not limited to the following types of storage media: dynamic random access memory (DRAM) or storage class memory (SCM). DRAM is a semiconductor memory, and like most random access memories (RAM), it is a volatile memory device. SCM is a composite storage technology that combines the characteristics of traditional storage devices and memory. Storage class memory can provide faster read and write speeds than hard disks, but the access speed is slower than DRAM, and the cost is cheaper than DRAM. However, DRAM and SCM are only exemplary descriptions in this embodiment, and the memory may also include other random access memories, such as static random access memory (SRAM), etc. Alternatively, it may also be a storage medium such as solid-state storage particles, mechanical hard disks, magnetic tape disks, memory or cache, etc., which is not limited in this application.

[0107] The queue includes characteristic values ​​of one or more encoded sub-images, and the images to which the one or more encoded sub-images belong belong to the same stream (video stream or other multimedia stream containing images) as the first image.

[0108] For example, the encoded sub-image may be a sub-image included in a previous frame image of the first image in the video stream.

[0109] For another example, the encoded sub-image may be a sub-image included in a subsequent frame image of a first image in a video stream.

[0110] Therefore, the characteristic values ​​of the sub-images stored in the external cache queue (or queue) can be sub-images of several frames before the first image, or sub-images of images after the first image, so that when the encoding device matches the sub-images of the first image with the data stored in the queue, the range of image encoding data that can be used for the multiplexed sub-images is increased, and the encoding bandwidth can be reduced to a greater extent.

[0111] The first type of sub-image determined in S320 matches the characteristic value of the encoded sub-image in the queue, and the characteristic value of the second type of sub-image is not recorded in the queue.

[0112] In an optional implementation, the feature value of the encoded sub-image may include any one or a combination of the following: image size, position, identification and matching status of the encoded sub-image, and the matching status indicates whether the encoded sub-image matches the sub-image to be encoded.

[0113] The image size of the encoded sub-image can be an enumeration of several fixed shape types, or variable values ​​such as length, width and height, or a sub-image size that can be determined by a method that can obtain the size, such as a rule for calculating the size (such as the difference in length, width and height with the previous sub-image block).

[0114] The position of the encoded sub-image can be the absolute value of the vertex coordinates of the encoded sub-image, or the position of the upper, lower, left, and right sides, or the position of the sub-image can be determined by a method that can obtain the position, such as a rule for calculating the position (such as an offset value from the position of the previous sub-image).

[0115] The identification (ID) value of the encoded sub-image may be a portion of the original pixel values ​​of the encoded sub-image, or a value obtained by processing the portion of the original pixel values ​​in a reference manner, and the value may uniquely represent the sub-image. The reference manner includes at least one of a hash algorithm and a cyclic redundancy check algorithm (CRC).

[0116] The matching status can be marked by a specific identifier, such as a binary value of 0 and 1: 0 means no match, 1 means match; or 0 means match, 1 means no match. Optionally, other possible status type strings can also be used for marking, which is not limited in this application.

[0117] In this embodiment, the encoding device stores the characteristic values ​​of the encoded sub-image (such as the image size, position, identification and matching status mentioned above) in an independent queue, and distinguishes the sub-images of the image to be encoded based on the relevant information stored in the queue, so that the encoding device can reuse the image encoding data of the sub-images in the previous multiple frames of images, reducing the repeated encoding process of valid pixels in sub-images with the same content, which is beneficial to reducing the encoding bandwidth of video encoding and improving encoding efficiency.

[0118] Moreover, the queue does not need to store the complete image coding data of the encoded sub-image, that is, when the initial coding data of the encoded sub-image is stored in the encoded bit stream, compared with the image coding data obtained by repeatedly encoding the sub-image with the same content, the queue occupies a smaller amount of data, and the encoding device only needs to provide a small amount of storage space to implement the relevant functions of the queue, which is beneficial to reducing the encoding bandwidth of video encoding and improving encoding efficiency.

[0119] S330 , the encoding device 310 generates first information of the first type of sub-image, and encodes the second type of sub-image to obtain second information.

[0120] Exemplarily, the encoding device 310 divides the first image into multiple sub-images and inputs them into an encoder, and outputs the above first information and second information. The second information includes: image encoding data obtained by encoding the second type of sub-images, and the encoding method may be a standard encoder.

[0121] The first information indicates: the encoded sub-image in the queue that matches the first type of sub-image. Figure 4 , Figure 5 Two possible examples are provided.

[0122] In a first feasible example, the encoding device obtains the first information of the first type of sub-image by matching the block pixel filling method. Figure 4 As shown, Figure 4 A schematic diagram of the process of an image encoding method provided in this application Figure 2 The process of the encoding device 310 generating the first information of the first type of sub-image may include the following steps S410 to S430.

[0123] S410: The encoding device 310 obtains a feature value of a first type of sub-image.

[0124] Exemplarily, the encoding device 310 may calculate the characteristic value according to the pixel values ​​included in the first type of sub-image.

[0125] As another example, the encoding device 310 may calculate the feature value according to the image features of the first type of sub-image.

[0126] The above two examples are only optional implementations provided in this embodiment. The encoding device 310 may also calculate the feature value according to other contents of the first type of sub-image.

[0127] S420: The encoding device 310 fills reference pixels for the first-category sub-image, and encodes the filled first-category sub-image to obtain first data.

[0128] The pixel filling method may include, but is not limited to, a fixed color filling method or a background color filling method. Exemplarily, the reference pixel may be a preset pixel value, such as 0 or other specified values.

[0129] In this embodiment, the encoding method adopted by the encoding device 310 for the first type of sub-image after filling is consistent with the encoding method adopted for the second type of sub-image, which can avoid deploying multiple different types of encoders in the encoding device 310, resulting in blurring of different areas in the reconstructed image, which is beneficial to improving the image quality of image decoding.

[0130] S430: The encoding device 310 obtains first information according to the first data and the feature values ​​of the first type of sub-image.

[0131] Exemplarily, the first information includes image coding data (first data) and instruction information (characteristic value) of the first type of sub-image after encoding and filling. During the decoding process, the decoding device queries the content of the decoded sub-image according to the instruction information, and fills the content of the decoded sub-image into the corresponding position of the first type of sub-image.

[0132] In a second feasible example, the encoding device encodes the first type of sub-image by low bit rate control to obtain the first information. Figure 5 As shown, Figure 5 A schematic diagram of the process of an image encoding method provided in this application Figure 3 The process of the encoding device 310 generating the first information of the first-category sub-image may include the following steps S510 to S530.

[0133] S510: The encoding device 310 obtains a feature value of a first type of sub-image.

[0134] The implementation of S510 may refer to the relevant contents of S410 and will not be elaborated here.

[0135] S520: The encoding device 310 encodes the first type of sub-image by using the first code rate to obtain second data.

[0136] The first bit rate is less than or equal to a bit rate threshold.

[0137] Exemplarily, the rate control parameters are set for the sub-image according to a preset rate control rule, and the preset rate control rule includes but is not limited to block-level QP setting, etc. If the QP value set for the first type of sub-image is relatively large (such as QP=50), the amount of encoded data obtained by encoding the first type of sub-image using the QP value is relatively small, which is conducive to reducing the encoding bandwidth of the video encoding.

[0138] S530: The encoding device 310 obtains first information according to the second data and the feature value of the first type of sub-image.

[0139] Combination Figure 4 and Figure 5 In the provided embodiment, when the same content appears repeatedly in different frames of a video, the image content and image texture match, so that the sub-image to be encoded can be more easily matched to the characteristic value of the encoded sub-image, and then the image encoding data of the encoded sub-image can be reused, thereby reducing the encoding bandwidth of the video encoding, which is beneficial to improving the encoding efficiency of the video encoding.

[0140] Moreover, during the encoding process, whether filling reference pixels for the first type of sub-image or setting a larger QP value for the first type of sub-image, the encoding method of the first type of sub-image in the corresponding area of ​​the first image and the aforementioned second type of sub-image both use the same encoder. Therefore, the encoding device only needs to provide a universal standard hardware video encoder (such as a standard encoder that complies with the video encoding standard), which can avoid the problem of obvious boundaries or blurred boundary positions between sub-images caused by encoding different sub-images with multiple encoders.

[0141] Please continue to refer to Figure 3 , the image encoding method provided in this embodiment also includes S340.

[0142] S340. The encoding device 310 outputs the encoded data of the first image.

[0143] The encoded data of the first image includes the first information and the second information obtained in S330.

[0144] In a possible example, the encoding device 310 may output the encoded data by sending the encoded data of the first image to another device. The other device may be a decoding device or a storage device for storing a video stream to which the first image belongs. Furthermore, the decoding device may reconstruct and restore the image according to the encoded data of the first image.

[0145] In order to improve the image decoding efficiency at the decoding side, as an optional implementation method, Figures 3 to 6 Based on the above, the image encoding method provided in this embodiment also includes: the encoding device 310 sends the data stored in the queue. Exemplarily, the data stored in the queue can be packaged with the video stream obtained after encoding the image, wherein the packaging method includes but is not limited to adding a specific field, splicing parameters at the head or tail of the code stream (bit stream), etc. For example, the data stored in the queue is carried in a code stream parameter set, a transport protocol header, an RTP payload, or a file format.

[0146] The code stream parameter set may include but is not limited to VPS, SPS, PPS and Supplemental Enhancement Informatyion (SEI) and the like.

[0147] The transport protocol header may refer to an extended header of RTP. For example, the extended header specifies that the extended type is image instruction information (characteristic value), and the corresponding value is the above-mentioned characteristic value.

[0148] The RTP payload is identified by the RTP version number or character syntax information to indicate that the RTP payload carries image instruction information (characteristic value).

[0149] The image instruction information (characteristic value) can also be unloaded in the file format, such as carried in the file format specified by the standard ISO14496-12, or in the transport stream (MPEG-2Transport Stream, MPEG2TS) file format. If carried in ISO14496-12, it can be carried in the track header, or a corresponding box can be added to independently present the image instruction information of each decoded frame in the box. If the image instruction information is carried in the MPEG2TS file, the information can be set in the adaptation_field_extension_flag in the Adaptation field, and the OutLoopPicOrderflag can be added in the adaptation_field_extension_flag to mark the use of the image instruction information (characteristic value). The outLoopPicParams contains the image instruction information. Through this parameter, the decoding side can obtain the corresponding image instruction information (characteristic value). The format is as follows:

[0150] "adaptation_field(){

[0151] adaptation_field_length

[0152] if(adaptation_field_length>0){

[0153] if(adaptation_field_extension_flag=='1'){

[0154] if(OutLoopPicOrderflag=='1'){

[0155] outLoopPicParams

[0156] }

[0157] }

[0158] }

[0159] }”

[0160] During the data communication process, in addition to transmitting the encoded data (such as bit stream), the encoding device also transmits the data stored in the introduced queue, so that the decoding device can quickly reconstruct and restore the image according to the data stored in the queue, thereby improving the decoding efficiency.

[0161] Moreover, since the amount of data stored in the queue is small, it can be carried in one of the above optional methods, so that the bandwidth required for communication between the encoding device and the decoding device is reduced, which is beneficial to improving the communication efficiency of the encoding and decoding system and reducing bandwidth consumption.

[0162] In an optional implementation, the present embodiment further updates the cached queue to improve the matching hit rate of the sub-images of subsequent frames in the video in the queue, thereby reducing the encoding bandwidth of the video encoding. Exemplarily, after the encoding device 310 determines the first category of sub-images and the second category of sub-images, the image encoding method provided by the present embodiment further includes: the encoding device 310 generates a feature value of each sub-image in the second category of sub-images, and writes the feature value of each sub-image in the second category of sub-images into the cached queue.

[0163] In this way, the encoding device writes the characteristic value of the unmatched sub-image (second type of sub-image) encoded for the first time into the queue. When the subsequent frame image of the currently encoded image in the video also has other sub-images with the same content as the unmatched sub-image, the other sub-images can reuse the image encoding data of the sub-image whose characteristic values ​​have been added to the queue, thereby reducing the encoding bandwidth of multiple frames of images in the video and improving the encoding efficiency.

[0164] exist Figure 2 Based on the following Figure 6 The content of the present invention provides an overall description of the image encoding and decoding method provided by the present application, such as Figure 3 As shown, Figure 6 A flowchart of the image encoding and decoding method provided in this application. The encoding side method is executed by an encoding device, and the method steps required to be executed by the encoding device include A1 to A6; the decoding side method is executed by a decoding device, and the method steps required to be executed by the decoding device include B1 to B6. The following is an exemplary description of the method steps required to be executed on the encoding side and the decoding side, respectively.

[0165] A1: The encoding device captures multiple frames of images included in the video.

[0166] The video may refer to a video of a video conference scene or other types of desktop content sharing (or screen content sharing) scenes.

[0167] A2: The encoding device divides the current frame image to be encoded into multiple sub-images, and searches and matches the multiple sub-images with the existing sub-images in the external cache queue to determine the matched sub-images and the unmatched sub-images in the current frame image.

[0168] Furthermore, the encoding device also outputs image instruction information of the segmentation and matching of the multiple sub-images, that is, the above-mentioned feature values, including: the position of each sub-image, the image size of the sub-image, the identifier of the sub-image, and whether the sub-image can find a matching encoded sub-image in the external cache queue, etc. The specific implementation of A2 can refer to the related description of S310 and S320 above, which will not be repeated here.

[0169] A3: The encoding device performs pre-processing on the matched sub-image (matching sub-image) in the current frame image.

[0170] The pre-processing process includes at least one of filling reference pixels or setting a low bit rate. For the specific implementation of A3, please refer to the above Figure 4 and Figure 5 The relevant description of the provided embodiments will not be repeated here.

[0171] A4: The encoding device inputs the pre-processed current frame image into a general video encoder and outputs an encoded video stream.

[0172] The encoded video stream includes the image encoding data of the unmatched sub-image and the image encoding data of the matched sub-image after pre-processing.

[0173] A5: The encoding device packages the video stream output by A4 and the feature value output after searching for a match in A2, and sends the packaged bit stream to the decoding side.

[0174] The packaging method may include but is not limited to adding a specific field in the video stream output by A4, splicing parameters at the head or tail of the video stream, etc. The packaging method or transmission position of the feature value can be referred to the description of S340, which will not be repeated here.

[0175] It is worth noting that A2, A3 and A5 above can be performed by a context-aware deduplication module included in the encoding device, such as Figure 2 The context-aware deduplication module 215 provided in.

[0176] A6: The encoding device writes the feature value of the unmatched sub-image into the external cache queue to update the data stored in the external cache queue.

[0177] Following the packetized bit stream sent by A5 in the encoding side, the decoding side performs the following B1 to B6.

[0178] B1: The decoding device receives the bit stream sent by the encoding device.

[0179] The bit stream includes: coded data of at least one frame of image and data (feature value) stored in the queue used for coding the image. The data stored in the queue can be referred to the related description of S320 above, which will not be described in detail here.

[0180] B2: The decoding device separates the bit stream to obtain the video stream and feature values ​​in the bit stream.

[0181] The separation method is determined according to the position of the characteristic value in the bit stream, and this application is not limited to this.

[0182] B3: The decoding device decodes the video stream to obtain a first image.

[0183] Exemplarily, the decoding device may use a universal standard decoder to decode the video stream, thereby outputting the first image.

[0184] B4: The decoding device matches the data stored in the queue in the bitstream with the first image, and determines the matched sub-images (first category sub-images) and unmatched second category sub-images in the first image. The decoding device queries the feature value of the decoded sub-image corresponding to the matched sub-image from the external cache queue, and uses the feature value of the decoded sub-image to determine the image content of the decoded sub-image, and then fills the image area of ​​the matched sub-image in the first image with the image content of the decoded sub-image, and restores the reconstructed image of the first image.

[0185] The queue includes characteristic values ​​of one or more decoded sub-images, the image to which the one or more decoded sub-images belong belongs to the same stream (video stream) as the first image, the first type of sub-image matches the characteristic values ​​of the decoded sub-images in the queue, and the queue does not record the characteristic values ​​of the second type of sub-images.

[0186] The difference between B4 and the encoding side is that the decoding process is the reverse process of the encoding process. Therefore, the feature values ​​stored in the queue used in the matching process are the feature values ​​of the decoded sub-image.

[0187] B5: The decoding device displays the restored reconstructed image. Before displaying the reconstructed image, the decoding device may also perform image processing operations such as rendering or cropping on the reconstructed image, which is not limited in this application.

[0188] Compared with the current frame multiplexing the image decoding data with the same position and the same content in the previous frame, in the present application, the decoding device classifies the sub-images of the first image according to the feature values ​​of the sub-images. Since the feature values ​​of different sub-images are different, after matching the sub-images with the decoded sub-images in the cached queue, the matched sub-images (such as the first category of sub-images) and the unmatched sub-images (such as the second category of sub-images) can be determined. Therefore, the matched sub-image of the first image does not need to have the same position and the same content as in the previous frame image in order to reuse the image decoding data of the matched sub-image in the previous frame image and the image to be decoded.

[0189] Furthermore, in the decoding process of the first image, the unmatched sub-image is decoded (such as standard decoding), and the information of the matched decoded sub-image is reused for the matched sub-image, thereby avoiding the problem of repeated decoding of valid pixels in the matched sub-image, reducing the decoding bandwidth of the video decoding, and improving the decoding efficiency.

[0190] B6: The decoding device writes the feature value of the unmatched sub-image into the external cache queue to update the data stored in the external cache queue.

[0191] Thus, the decoding device writes the characteristic value of the unmatched sub-image decoded for the first time into the queue. When the subsequent frame image of the image in the video also has other sub-images with the same content as the unmatched sub-image, the other sub-image can also reuse the image decoding data of the sub-image whose characteristic value has been added to the queue, thereby reducing the decoding bandwidth of multiple frames of images in the video and improving decoding efficiency.

[0192] It is worth noting that the sub-image division method and the eigenvalue generation method adopted by the decoding side are consistent with those of the encoding side, and this application will not be repeated here. In order to make the image encoding and decoding method provided by this application clearer, the following uses two sub-image division methods, fixed block division and adaptive division (dynamic division), as examples to illustrate the image encoding and decoding method provided by this application.

[0193] Example 1: The image is encoded and decoded in a fixed block division manner.

[0194] like Figure 7 As shown, Figure 7 A schematic diagram of an image encoding and decoding method with a fixed block division method provided by the present application. The image encoding and decoding method can be applied to Figure 1 The video transmission system shown or Figure 2 The video encoding and decoding system shown in the figure will be described below with respect to the method flow on the encoding side and the decoding side respectively.

[0195] On the encoding side, the encoding device divides the image to be encoded into multiple sub-images of fixed size and marks feature values ​​(including: identification, image size, position, etc.) for each sub-image. Figure 7 ID11 to ID44 in.

[0196] And, the encoding device performs CADC search matching on the multiple sub-images and the external cache queue to determine the matched sub-images and the unmatched sub-images, wherein the matched sub-images include Figure 7 The unmatched sub-images include ID11, ID21, ID31, ID41, ID14, ID24, ID34, and ID44. Figure 7 ID12, ID22, ID32, ID42, ID13, ID23, ID33 and ID43.

[0197] Afterwards, the encoding device outputs the feature value of the matched sub-image and fills the reference pixels in the area of ​​the matched sub-image, encodes the filled image to be encoded, and outputs a first code stream; the first code stream includes the complete image encoding data of the unmatched sub-image and the image encoding data after the matched sub-image is filled with reference pixels.

[0198] Finally, the encoding device packages the feature value of the matched sub-image and the first code stream, and sends the package to the decoding device.

[0199] In addition, the encoding device also writes the feature values ​​of the unmatched sub-images into an external cache queue, so that the image encoding data of these unmatched sub-images can be reused in subsequent frame images, thereby reducing the encoding bandwidth of the video encoding.

[0200] It is worth noting that the encoding device may also use a low bit rate to encode the image region of the unmatched sub-image, thereby reducing the data volume of the aforementioned first code stream.

[0201] The difference between the decoding side and the encoding side is that after the decoding device identifies a matching sub-image in the first image and a decoded sub-image in the external cache queue, it fills the image content of the decoded sub-image into the image area of ​​the matching sub-image. Information such as image size and position required for the filling process can be obtained from the external cache queue.

[0202] Example 2: An image is encoded and decoded in an adaptive partitioning manner.

[0203] like Figure 8 As shown, Figure 8 A schematic diagram of an image encoding and decoding method in an adaptive partitioning manner provided by the present application. The image encoding and decoding method can be applied to Figure 1 The video transmission system shown or Figure 2 The video coding system shown, Figure 8 and Figure 7 The difference is that the images obtained by dividing the images are different. For example, Figure 8 The matched sub-images include: ID11, ID31, ID411, ID412, ID413, ID414, ID141, ID142, ID143, and ID34; the other sub-images are unmatched sub-images. Other content and Figure 7 The present application will not elaborate on this.

[0204] Combination Figures 2 to 8In the embodiments, the image coding and decoding method provided by the present application can be applied to the following coding and decoding scenarios: 1. Fixed block division + reference pixel filling + standard hardware video codec; 2. Adaptive division + reference pixel filling + standard hardware video codec; 3. Fixed block division + low bit rate encoding + standard hardware video codec; 4. Adaptive division + low bit rate encoding + standard hardware video codec, etc.

[0205] Compared with the encoding and decoding methods in screen content scenarios such as video conferencing and desktop sharing provided by conventional technologies, the image encoding and decoding method provided by this application performs repeated image matching outside the encoder on the encoding side (original image), establishes the matching results (image size, coordinates, identification, matching status, etc.) into an external cache queue, and sends it to the decoding side in the form of some instructions. The decoding side establishes an external cache queue in the same way, traverses all image information of the entire frame, and if it is a matching image, fills the previously decoded image from the external cache queue into the corresponding position; if it is an unmatched image, retains the original decoding result and updates the queue. In addition, the compression rate of the general video encoder in the screen content scenario is improved, the encoding bandwidth is reduced, and the encoding efficiency is improved.

[0206] It is worth noting that the image encoding and decoding method provided in this application can also be applied to screen content scenarios such as cloud conferencing, cloud desktop, cloud gaming, live video streaming, and desktop extension (Xtended, XR).

[0207] It is understandable that in order to implement the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0208] Combined with the above Figures 1 to 8 , describes in detail the image encoding and decoding method provided by this embodiment, and will now be combined with Fig. 9 , describing the encoding and decoding device provided according to this embodiment.

[0209] Fig. 9 The following is a schematic diagram of the structure of a coding and decoding device provided by the present application. These coding and decoding devices can be used to implement the functions of the coding device and the decoding device in the above method embodiments, and thus can also achieve the beneficial effects possessed by the above method embodiments. In this embodiment, the coding and decoding device can be as follows Figure 2 The encoding device 210 and decoding device 220 shown may also be modules (such as chips) applied to a computing device.

[0210] like Fig. 9 As shown, the coding and decoding apparatus 900 includes a communication module 910, a context-aware deduplication module 920 and a coding and decoding module 930. The coding and decoding apparatus 900 is used to implement the functions of the coding device and the decoding device in the method embodiments shown in the drawings.

[0211] When the encoding and decoding apparatus 900 is used to implement the function of the encoding device, the specific functions of each module are as follows.

[0212] The communication module 910 is used to obtain a first image to be encoded in a video.

[0213] The context-aware deduplication module 920 is used to divide the first image into multiple sub-images, and search and match the multiple sub-images with the feature values ​​(image instruction information) in the queue to determine the first type of sub-images and the second type of sub-images.

[0214] The encoding and decoding module 930 is used to generate first information of the first type of sub-image, and encode the second type of sub-image to obtain second information.

[0215] The communication module 910 is further configured to output encoded data including the first information and the second information.

[0216] When the encoding and decoding apparatus 900 is used to implement the function of a decoding device, the specific functions of each module are as follows.

[0217] The communication module 910 is used to obtain a bit stream, which includes: encoded data of at least one frame of image and data (feature values) stored in a queue used by the encoded image.

[0218] The context-aware deduplication module 920 is used to separate the data in the bitstream to obtain the video stream and the feature value; and after the first image obtained by decoding the video stream by the codec module 930 is divided into multiple sub-images, the multiple sub-images are searched and matched with the feature values ​​(image instruction information) in the queue to determine the first type of sub-image and the second type of sub-image. In addition, the context-aware deduplication module 920 fills the image content of the decoded sub-image in the queue into the image area corresponding to the first type of sub-image to restore the reconstructed image of the first image.

[0219] It should be understood that the encoding and decoding device 900 of the embodiment of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the above PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. It can also be implemented by software Figure 3 When the method is shown, each module thereof may also be a software module, and each module thereof of the coding and decoding device 900 may also be a software module.

[0220] According to the embodiment of the present application, the coding and decoding device 900 can correspond to the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of each unit in the coding and decoding device 900 are respectively for implementing the corresponding processes of each method in the aforementioned figures. For the sake of brevity, they will not be repeated here.

[0221] Fig.10 This is a schematic diagram of the structure of an image processing system provided by the present application. The image processing system is described by taking a mobile phone as an example. The mobile phone or a chip system built into the mobile phone includes: a memory 1010, a processor 1020, a sensor component 1030, a multimedia component 1040 and an input / output interface 1050. Fig.10 Provide a detailed introduction to each component of a mobile phone or a chip system built into a mobile phone.

[0222] The memory 1010 can be used to store data, software programs and modules; it mainly includes a program storage area and a data storage area, wherein the program storage area can store software programs, including instructions formed in code, including but not limited to an operating system, an application required for at least one function, such as a sound playback function, an image playback function, etc.; the data storage area can store data created according to the use of the mobile phone, such as audio data, image data, a phone book, etc. In the embodiment of the present application, the memory 1010 can be used to store the number of bits of data in the bit stream buffer, etc. In some feasible embodiments, there can be one memory or multiple memories; the memory can include a floppy disk, a hard disk such as a built-in hard disk and a mobile hard disk, a magnetic disk, an optical disk, a magneto-optical disk such as a CD_ROM, a DCD_ROM, a non-volatile storage device such as a RAM, a ROM, a PROM, an EPROM, an EEPROM, a flash memory, or any other form of storage medium known in the technical field.

[0223] The processor 1020 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire device. By running or executing the software programs and / or software modules stored in the memory 1010, and calling the data stored in the memory 1010, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. In the embodiment of the present application, the processor 1020 can be used to execute one or more steps in the method embodiment of the present application. For example, the processor 1020 can be used to execute one or more steps from S420 to S470 in the following method embodiment. In some feasible embodiments, the processor 1020 can be a single processor structure, a multi-processor structure, a single-thread processor, and a multi-thread processor; in some feasible embodiments, the processor 1020 can include at least one of a central processing unit, a general processor, a digital signal processor, a neural network processor, an image processing unit, an image signal processor, a microcontroller or a microprocessor. In addition, the processor 1020 may further include other hardware circuits or accelerators, such as application-specific integrated circuits, field programmable gate arrays or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 1020 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like.

[0224] The sensor component 1030 includes one or more sensors for providing status assessment of various aspects for the mobile phone. Among them, the sensor component 1030 may include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications, that is, a component of a camera or a camera. In an embodiment of the present application, the sensor component 1030 can be used to support the camera in the multimedia component 1040 to acquire images, etc. In addition, the sensor component 1030 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor. The acceleration / deceleration, orientation, open / closed state of the mobile phone, the relative positioning of the components, or the temperature change of the mobile phone can be detected by the sensor component 1030.

[0225] The multimedia component 1040 provides a screen of an output interface between the mobile phone and the user, and the screen can be a touch panel, and when the screen is a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In addition, the multimedia component 1040 also includes at least one camera, for example, the multimedia component 1040 includes a front camera and / or a rear camera. When the mobile phone is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can sense an external multimedia signal, which is used to form an image frame. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0226] The input / output interface 1050 provides an interface between the processor 1020 and a peripheral interface module. For example, the peripheral interface module may include a keyboard, a mouse, or a USB (Universal Serial Bus) device, etc. In a possible implementation, the input / output interface 1050 may have only one input / output interface or may have multiple input / output interfaces.

[0227] Although not shown, the mobile phone may also include an audio component and a communication component, for example, the audio component includes a microphone, and the communication component includes a wireless fidelity (WiFi) module, a Bluetooth module, etc., which will not be described in detail in the embodiments of the present application.

[0228] The above-mentioned image processing system can be a general-purpose device or a dedicated device. For example, the image processing system can be an edge device (e.g., a box carrying a chip with processing capabilities), etc. Optionally, the image processing system can also be a server or other device with computing capabilities. In some possible situations, the image processing system can also refer to the aforementioned video codec system, etc.

[0229] It should be understood that the image processing system according to this embodiment may correspond to the codec device 900 in this embodiment, and may correspond to executing the corresponding subject according to any method in the aforementioned figures, and the above-mentioned and other operations and / or functions of each module in the codec device 900 are respectively for implementing the corresponding processes of each method in the aforementioned figures, and for the sake of brevity, they will not be repeated here.

[0230] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist as discrete components in a network device or a terminal device.

[0231] The present application also provides a chip system, which includes a processor for implementing the functions of the data processing unit in the above method. In one possible design, the chip system also includes a memory for storing program instructions and / or data. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0232] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0233] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An image coding method, characterized in that: The method comprises: Acquire a first image to be encoded, and divide the first image into a plurality of sub-images; Matching the plurality of sub-images with the cached queue to determine a first category of sub-images and a second category of sub-images; The queue includes one or more feature values ​​of encoded sub-images, the image to which the one or more encoded sub-images belong belongs to the same stream as the first image, the first type of sub-images matches the feature values ​​of the encoded sub-images in the queue, and the queue does not record feature values ​​of the second type of sub-images; Generate first information of the first type of sub-image, and encode the second type of sub-image to obtain second information; the first information indicates: the encoded sub-image in the queue that matches the first type of sub-image; The encoded data of the first image is output, the encoded data including the first information and the second information.

2. The method according to claim 1, characterized in that The characteristic value includes any one or a combination of the following: image size, position, identification and matching status of the encoded sub-image, and the matching status indicates whether the encoded sub-image matches the sub-image to be encoded.

3. The method according to claim 2, characterized in that The identifier of the encoded sub-image is: part of the original pixel values ​​of the encoded sub-image, or a value obtained by processing the part of the original pixel values ​​according to a reference method; the reference method includes at least one of a hash algorithm and a cyclic redundancy check algorithm.

4. The method according to any one of claims 1 to 3, characterized in that The first information for generating the first type of sub-image includes: Obtaining feature values ​​of the first type of sub-images; Filling reference pixels for the first-category sub-image, and encoding the filled first-category sub-image to obtain first data; The first information is obtained according to the first data and the feature values ​​of the first type of sub-images.

5. The method according to any one of claims 1 to 3, characterized in that: The first information includes: characteristic values ​​of the first type of sub-images, and second data obtained by encoding the first type of sub-images using a first code rate; the first code rate is less than or equal to a code rate threshold.

6. The method according to any one of claims 1 to 5, characterized in that After determining the first type of sub-image and the second type of sub-image, the method further includes: generating a feature value for each sub-image in the second type of sub-image; The feature value of each sub-image in the second type of sub-image is written into the queue.

7. The method according to any one of claims 1 to 6, characterized in that The multiple sub-images have the same image size.

8. The method according to any one of claims 1 to 6, characterized in that The step of dividing the first image into a plurality of sub-images comprises: Acquire a plurality of image features of the first image, at least one image feature being used to describe image texture information of the first image; Selecting at least one division method from a plurality of preset division methods according to the plurality of image features, wherein the at least one division method matches the image texture information; The first image is divided according to the at least one division method to obtain the multiple sub-images.

9. The method according to any one of claims 1 to 8, characterized in that The outputting the encoded data of the first image comprises: The encoded data of the first image is transmitted.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: The data stored in the queue is sent, where the data stored in the queue is carried in a bitstream parameter set, a transport protocol header, a real-time transport protocol RTP payload or a file format.

11. An image decoding method, characterized in that: The method comprises: Acquire a bit stream, wherein the bit stream includes encoded data of at least one frame of an image and data stored in a queue used to encode the image; Decoding the encoded data in the bit stream to obtain a first image; Matching the data stored in the queue in the bit stream with the first image to determine a first type of sub-image and a second type of sub-image in the first image; The queue includes characteristic values ​​of one or more decoded sub-images, the image to which the one or more decoded sub-images belong belongs to the same stream as the first image, the first type of sub-images matches the characteristic values ​​of the decoded sub-images in the queue, and the queue does not record characteristic values ​​of the second type of sub-images; According to the feature values ​​matching the first type of sub-images in the queue, an image region corresponding to the first type of sub-images in the first image is restored to obtain a reconstructed image.

12. The method according to claim 11, characterized in that The characteristic value includes any one or a combination of the following: image size, position, identification and matching status of the decoded sub-image, and the matching status indicates whether the decoded sub-image matches the sub-image to be decoded.

13. The method according to claim 12, characterized in that The identifier of the decoded sub-image is: part of the original pixel values ​​of the decoded sub-image, or a value obtained by processing the part of the original pixel values ​​according to a reference method; the reference method includes at least one of a hash algorithm and a cyclic redundancy check algorithm.

14. The method according to any one of claims 11 to 13, characterized in that The multiple sub-images have the same image size.

15. The method according to any one of claims 11 to 14, characterized in that The data stored in the queue is carried in a bitstream parameter set, a transport protocol header, a real-time transport protocol RTP payload or a file format.

16. A coding and decoding device, characterized in that: The encoding and decoding device includes a device for executing the method according to any one of claims 1-15.

17. An encoding device, characterized in that: The encoding device comprises at least one processor and a memory, wherein the memory is used to store a computer program, so that when the computer program is executed by the at least one processor, the method according to any one of claims 1 to 10 is implemented.

18. A decoding device, characterized in that: The encoding device comprises at least one processor and a memory, wherein the memory is used to store a computer program, so that when the computer program is executed by the at least one processor, the method according to any one of claims 11 to 15 is implemented.

19. A coding and decoding system, characterized in that: The encoding and decoding system includes the encoding device as described in claim 17 and the decoding device as described in claim 18, wherein the encoding device is used to execute the operation steps of the method described in any one of claims 1-10, and the decoding device is used to execute the method described in any one of claims 11-15.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer software instructions. When the computer software instructions are executed in a computer, the computer executes the method according to any one of claims 1 to 15.

Images