Encoding method, decoding method, and electronic device

Abstract

Embodiments of the present application provide an encoding method, a decoding method, and an electronic device. The encoding method comprises: first, acquiring a first image and a second image, wherein the first image comprises an image of a first area in an original image and does not comprise an image of a second area in the original image, the second image comprises the image of the second area in the original image, the size of the first image, the size of the second image and the size of the original image are the same, and the image of the second area in the original image comprises user information; performing base layer encoding on the first image to obtain a base layer bit stream; and then performing enhance layer encoding on the second image to obtain an enhance layer bit stream, wherein the base layer bit stream and the enhance layer bit stream are interwoven into a bit stream. Thus, the encoding performance can be improved and the operation overhead can be reduced.

Description

Coding and decoding method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 16, 2023, with application number 202310731240.8 and application name “Encoding and Decoding Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of coding and decoding, and in particular to a coding and decoding method and an electronic device. Background Art

[0003] With the rapid development of the Internet and multimedia technology, as well as the continuous decline in the hardware cost and implementation cost of video surveillance, video surveillance technology has been widely used; for example, video surveillance technology is used to implement remote video surveillance of target areas to assist in public security management, accident warning, etc.

[0004] However, the collected video images may carry user information (such as faces, license plate numbers, etc.). If the video images are leaked, the user information will also be leaked, which may cause losses to the user; therefore, video privacy protection technology came into being.

[0005] In existing technologies, the encoding end feeds the privacy region image and the non-privacy region image of a video image into two separate encoders for encoding, generating a bitstream for the privacy region image and a bitstream for the non-privacy region image. The bitstream for the non-privacy region image and the encrypted bitstream for the privacy region image are then transmitted, thus achieving video privacy protection. However, the independent encoding of the privacy region image and the non-privacy region image reduces encoding performance to a certain extent. Furthermore, the system layer needs to synchronize the bitstreams of the privacy region image and the non-privacy region image to ensure that the decoding end can obtain the complete image, which incurs computational overhead.

[0006] Summary of the Invention

[0007] In order to solve the above technical problems, the present application provides an electronic device for a coding and decoding method, which can improve coding performance and reduce computational overhead.

[0008] In a first aspect, an embodiment of the present application provides an encoding method, which includes: first, obtaining a first image and a second image, the first image including an image of a first area in the original image and not including an image of a second area in the original image, the second image including an image of the second area in the original image, the size of the first image and the size of the second image being the same as the size of the original image, and the image of the second area in the original image containing user information; then, encoding the first image at a base layer to obtain a base layer bit stream; thereafter, encoding the second image at an enhancement layer to obtain an enhancement layer bit stream; wherein the base layer bit stream and the enhancement layer bit stream are interwoven into one bit stream.

[0009] Since the reference queue for encoding the enhancement layer includes the reconstructed image of the base layer (i.e., the reconstructed image of the first image (hereinafter referred to as the first reconstructed image)), that is, the reference queue for the second image includes the first reconstructed image; compared to the prior art in which the reference queue for the second image (i.e., the privacy image) only includes the reconstructed block of the second image of the current frame and the second reconstructed images of other frames (i.e., the reconstructed images of the second images of other frames), the present application encodes more reference images for the second image. In this way, the present application can obtain more accurate prediction blocks, thereby improving the encoding performance of the enhancement layer (i.e., improving encoding quality or reducing bitrate overhead), thereby improving the overall encoding performance. In addition, in the present application, the enhancement layer bitstream and the base layer bitstream are interwoven into one bitstream, which facilitates frame synchronization of the enhancement layer bitstream and the base layer bitstream, and can reduce computational overhead.

[0010] For example, the original image may refer to an image that is initially intended to be transmitted to a remote end. The original image may be generated in a variety of ways, such as an image captured by an image acquisition device, an image received from another terminal device, an image generated by AI (Artificial Intelligence), an image processed by an image processing tool, and the like, and this application does not impose any restrictions on this.

[0011] Exemplarily, the original image may include a first area and a second area, wherein the second area image of the original image may include user information, and user information may refer to information that directly or indirectly describes the identity of the user. For example, user information includes but is not limited to: user name, date of birth, ID number, address, telephone number, face, license plate number, movement posture, clothing, etc., and this application does not impose any restrictions on this. In some scenarios, user information may also be referred to as private information. Private information may refer to secrets that are not willing to be disclosed or known to others (people outside a certain range), and this secret is not related to other people or social interests.

[0012] It should be noted that the present application uses one encoder to encode the base layer and the enhancement layer, wherein one encoder outputs one bit stream; wherein the base layer (Base Layer, BL) bit stream and the enhancement layer (Enhance Layer, EL) bit stream can be interwoven into one bit stream, which can also be called the bit stream of the original image.

[0013] For example, for video data comprising n (n is a positive integer) frames of original images, the steps in the first aspect described above can be performed for each frame of original image, resulting in a bitstream of n frames of original images (hereinafter referred to as the n-frame bitstream). The first image and second image generated based on the i-th (i is an integer between 1 and n) frame of original image can be referred to as the i-th frame first image and the i-th frame second image. Thus, a bitstream output by the encoder can include n frame bitstreams, each frame of the bitstream comprising an EL bitstream and a BL bitstream.

[0014] It should be noted that the enhancement layer bitstream is output after the base layer bitstream, that is, after the encoder outputs the enhancement layer bitstream, the enhancement layer bitstream is connected to the base layer bitstream. At this time, the enhancement layer bitstream and the base layer bitstream are interleaved to form one bitstream (it can be understood that the base layer bitstream and the enhancement layer bitstream are interleaved and sorted in the bitstream output by the encoder); it is not that the encoder performs additional operations (such as interleaving operations) to make the enhancement layer bitstream and the base layer bitstream interleave to form one bitstream.

[0015] Based on the above, the present application can be regarded as a layered encoding of the original image, wherein, in the layered encoding process of the original image, the encoding of the base layer corresponds to the encoded image of the base layer as the first image, and the encoding of the enhancement layer corresponds to the encoded image of the enhancement layer as the second image.

[0016] For example, after obtaining the original image bitstream, the encoder can encrypt the enhancement layer bitstream in the original image bitstream to protect the user information in the original image during transmission to the remote end, thereby ensuring the security of the user information in the original image. The encryption operation can be performed by the system layer of the encoder.

[0017] According to the first aspect, the second image also includes the image of the first area in the original image; or the second image does not include the image of the first area in the original image. In other words, the second image only needs to include the image of the second area in the original image.

[0018] Exemplarily, the second image also includes an image of the first area in the original image, that is, the second image is the original image.

[0019] According to the first aspect, or any implementation method of the first aspect above, the enhancement layer of the second image is encoded to obtain an enhancement layer bit stream, including: performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the second image to obtain a predicted image of the second image; encoding the residual between the second image and the predicted image of the second image to obtain an enhancement layer bit stream.

[0020] For example, the reference image for inter-frame prediction of the second image may be a reconstructed image of the second image of another frame. The reference image for intra-frame prediction of the second image may be a reconstructed block of the second image of the current frame. The reference image for inter-layer prediction of the second image may be a reconstructed image of the first image of the current frame.

[0021] According to the first aspect, or any implementation method of the first aspect above, inter-frame prediction, intra-frame prediction or inter-layer prediction is performed on the second image to obtain a predicted image of the second image, including: dividing the second image to obtain multiple coding units (Coding Unit, CU); for the first CU among the multiple CUs, determining whether the first CU belongs to the second area; when the first CU belongs to the second area, performing inter-frame prediction or intra-frame prediction on the first CU to determine the prediction block of the first CU; when the first CU does not belong to the second area, performing inter-layer prediction on the first CU to determine the prediction block of the first CU; wherein the predicted image of the second image includes the prediction blocks of the multiple CUs obtained by dividing the second image.

[0022] Specifically, for the first CU belonging to the second region, the similarity with the reconstructed block in the second region in the reconstructed image of the second image of the other frame is higher, so a more accurate prediction block can be obtained, thereby improving encoding performance. When the second image is the original image, for the first CU not belonging to the second region, the similarity with the reconstructed block in the first region in the reconstructed image of the first image of the current frame is higher, so a more accurate prediction block can also be obtained, thereby improving encoding performance.

[0023] According to the first aspect, or any implementation of the first aspect above, inter-layer prediction is performed on the first CU to determine the prediction block of the first CU, including: using the reconstructed block in the first reconstructed image at the same position as the first CU as the prediction block of the first CU; wherein the first reconstructed image is a reconstructed image of the first image. In this way, the MV of the first CU is (0,0), and the number of bits required to encode the MV of the first CU is relatively small, which can reduce the bit rate overhead of the enhancement layer encoding, thereby reducing the overall bit rate overhead. In addition, the computational complexity of the motion search on the encoder side can be reduced, thereby reducing the encoding complexity.

[0024] According to the first aspect, or any implementation of the first aspect above, when the first CU does not belong to the second region, the residual between the prediction block of the first CU and the first CU is 0. The residual between the first CU and the prediction block of the first CU can be forcibly set to 0. In this way, the number of bits required to encode the residual between the first CU and the prediction block of the first CU is small, which can reduce the coding rate overhead of the enhancement layer, thereby reducing the overall coding rate overhead.

[0025] According to the first aspect, or any implementation of the first aspect above, inter-frame prediction, intra-frame prediction, or inter-layer prediction is performed on the second image to obtain a predicted image of the second image, including: dividing the second image to obtain multiple CUs; ​​performing inter-frame prediction, intra-frame prediction, or inter-layer prediction on each second CU in the multiple CUs to determine a prediction block for the second CU; wherein the predicted image of the second image includes the prediction blocks of the multiple CUs obtained by dividing the second image. In this way, a more accurate prediction block can be determined for each second CU, thereby improving the encoding quality of the second CU.

[0026] According to the first aspect, or any implementation of the first aspect above, any one of the multiple CUs is located within or outside the second region. In other words, the boundary line of the second region does not separate any CU; this can serve as a constraint condition for dividing the second image into the multiple CUs, thereby facilitating determination of whether the first CU belongs to the second region.

[0027] According to the first aspect, or any implementation of the first aspect above, the method also includes: generating a first reconstructed image; wherein the first reconstructed image is a reconstructed image of the first image, and the first reconstructed image is a reference image of the first image of other frames and / or a reference image of the second image; the first reconstructed image is an image after loop filtering, and the pixel value of the pixel point in the second area of ​​the first reconstructed image is a preset value.

[0028] According to the first aspect, or any implementation of the first aspect above, the method further includes: generating a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image for the second image of other frames, the image of the second region in the second reconstructed image is an image that has undergone loop filtering, and the image of the first region in the second reconstructed image is an image that has not undergone loop filtering. In this way, it is ensured that the image of the first region in the second reconstructed image is identical to the image of the first region in the first reconstructed image; in this case, after the decoder decodes and obtains the second reconstructed image, the second reconstructed image can be directly used as the reconstructed image of the original image, without the need to fuse the first reconstructed image and the second reconstructed image, thereby reducing computational overhead.

[0029] According to the first aspect, or any implementation of the first aspect above, when the second image does not include an image of the first region in the original image, the first reconstructed image and the second reconstructed image are fused to obtain an updated second reconstructed image. In this case, the updated second reconstructed image is a reconstructed image of the original image; the updated second reconstructed image can subsequently be stored in a decoded image buffer and used as a reference image for subsequent encoding of second images in other frames.

[0030] According to the first aspect, or any implementation of the first aspect above, the method further includes: obtaining position indication information, the position indication information being used to indicate the position of the second region; and adding the encoded position indication information to the base layer bitstream and / or the enhancement layer bitstream. In this way, the position indication information can be transmitted to a decoder, facilitating the decoder to constrain loop filtering for the first region and the second region, and to fuse the first reconstructed image with the second reconstructed image, based on the position indication information parsed from the base layer bitstream and / or the enhancement layer bitstream.

[0031] In one possible approach, when the second region is rectangular, the location indication information may include the coordinates of the upper left corner of the second region and the size of the second region. When the second region is circular, the location indication information may include the coordinates of the center of the second region and the radius of the second region. It should be understood that when the second region has other shapes, the location indication information may include other information, and this application does not limit this.

[0032] In one possible manner, the position indication information may be an image block identification map. Specifically, the original image may be divided into m (m is a positive integer) image blocks of a fixed size (which may be set as required, such as 2*2); then, a preset identifier is used for each image block to identify whether it is in the second area. For example, for an image block, when the image block is in the second area, the value of the preset identifier of the image block is set to the first identifier value; when the image block is not in the second area, the value of the preset identifier of the image block is set to the second identifier value; in this way, an image block identification map may be obtained; wherein the image block identification map may include m pixels, one pixel corresponding to an image block in the original image, and the pixel value of each pixel is the value of the preset identifier. The first identifier value is different from the second identifier value, and may be set specifically as required, and this application does not impose any restrictions on this. For example, the first identifier value is "1" and the second identifier value is "0"; at this time, among the m pixels contained in the image block identification map, the pixel values ​​of m1 pixels are "1" and the pixel values ​​of m2 pixels are "0". Wherein, m1+m2=m, m1 and m2 are integers greater than or equal to 0.

[0033] It should be understood that the image block identification map can also be called an image block identification matrix; the image block identification matrix may include m elements, one element corresponds to an image block in the original image, and the value of each element is a preset identification value.

[0034] In one possible manner, the above-mentioned image block may be a CU.

[0035] In a possible manner, the above-mentioned image block includes only one pixel. In this case, the position indication information may be a pixel identification map.

[0036] It should be noted that the location indication information can also be expressed in other forms, and this application does not limit this.

[0037] According to the first aspect, or any implementation of the first aspect above, the base layer bit stream is located before the enhancement layer bit stream.

[0038] According to the first aspect, or any implementation method of the first aspect above, the first area in the original image is the area in the original image excluding the second area; the second image is generated by replacing the pixel values ​​of the pixel points in the first area in the original image; the first image is generated by replacing the pixel values ​​of the pixel points in the second area in the original image; wherein the replacement method includes at least one of the following: solid color filling, mosaic or Gaussian blur.

[0039] It should be understood that the replacement method may also be other methods, and this application does not limit this.

[0040] First, user information detection can be performed on the original image to determine the area where the user information is located. For ease of explanation, the area in the original image containing user information (that is, the area where the user information is located) is referred to as the second area, and the area in the original image that does not contain user information is referred to as the first area; it should be understood that the first area is the area in the original image excluding the second area. The method for detecting user information on the original image can be, for example, an AI-based intelligent detection method, and this application does not limit this.

[0041] Next, a second image is generated. In one possible approach, the original image can be used as the second image; that is, the second image includes an image of the first region in the original image and an image of the second region in the original image. In another possible approach, the pixel values ​​of the pixels in the second region in the original image can be maintained unchanged, while the pixel values ​​of the pixels in the first region in the original image can be replaced. For example, the replacement can be performed using at least one of the following methods: solid color filling, mosaic, or Gaussian blur, etc., which is not limited in this application.

[0042] And, generating a first image. Exemplarily, the pixel values ​​of the pixels in the first area of ​​the original image can be kept unchanged, and the pixel values ​​of the pixels in the second area of ​​the original image can be replaced. For example, the replacement can be performed using at least one of the following methods: solid color filling, mosaic, or Gaussian blur, etc., which is not limited in this application.

[0043] It should be noted that the above steps of detecting user information, generating the first image and generating the second image can be performed by the encoder or by other modules, and this application does not impose any restrictions on this.

[0044] According to the first aspect, or any implementation of the first aspect above, the base layer bitstream and the enhancement layer bitstream include a reference frame identifier and a prediction mode. Thus, the reference frame identifier and the prediction mode are transmitted to a decoder, facilitating the decoder to determine a reference image and then perform prediction based on the reference image.

[0045] Exemplarily, the prediction mode may include but is not limited to an intra-frame prediction mode, an inter-frame prediction mode, an inter-layer prediction mode, and the like.

[0046] In a second aspect, an embodiment of the present application provides a decoding method, which includes: first, receiving a bit stream, the bit stream including a base layer bit stream and an enhancement layer bit stream; then, decoding the base layer bit stream at the base layer to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of the first image, the first image includes an image of the first area in the original image and does not include an image of the second area in the original image; thereafter, decoding the enhancement layer bit stream at the enhancement layer to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, the second image includes an image of the second area in the original image; wherein the first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

[0047] Exemplarily, decoding of the base layer and decoding of the enhancement layer may be performed by one decoder.

[0048] For example, after receiving the bitstream, the decoder can, on the one hand, input the base layer bitstream in the bitstream into the decoder; on the other hand, decrypt the encrypted enhancement layer bitstream in the bitstream and input it into the decoder; the decoder decodes the base layer and the enhancement layer. The decryption operation can be performed by the system layer of the decoder.

[0049] According to the second aspect, the second image further includes the image of the first area in the original image; or the second image does not include the image of the first area in the original image.

[0050] According to the second aspect, or any implementation of the second aspect above, the bitstream also includes a reference frame identifier and a prediction mode, and the enhancement layer bitstream is decoded to obtain a second reconstructed image, including: determining a reference image of the second image according to the reference frame identifier and the prediction mode; performing prediction based on the reference image of the second image to obtain a predicted image of the second image; and adding the predicted image of the second image to the residual of the second image parsed from the enhancement layer bitstream to obtain a second reconstructed image.

[0051] According to the second aspect, or any implementation method of the second aspect above, the second image includes multiple coding units CUs, the reference image of the second image includes reference images of multiple CUs, and prediction is performed based on the reference image of the second image to obtain a predicted image of the second image, including: for a first CU among multiple CUs, predicting the first CU based on the reference image of the first CU to determine the prediction block of the first CU; wherein the predicted image of the second image includes the prediction blocks of multiple CUs included in the second image.

[0052] According to the second aspect, or any implementation of the second aspect above, the first reconstructed image is an image after loop filtering, and the pixel values ​​of the pixels in the second area of ​​the first reconstructed image are preset values.

[0053] According to the second aspect, or any implementation of the second aspect, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

[0054] According to the second aspect, or any implementation of the second aspect above, the method further includes: when the second image does not include the image of the first area in the original image, fusing the first reconstructed image and the second reconstructed image to obtain an updated second reconstructed image.

[0055] According to the second aspect, or any implementation of the above second aspect, the bitstream also includes encoded position indication information, and the position indication information is used to indicate the position of the second area; the method also includes: determining the second area in the first reconstructed image based on the position indication information.

[0056] According to the second aspect, or any implementation of the second aspect above, the bitstream also includes a reference frame identifier and / or encoded position indication information, the position indication information is used to indicate the position of the second area, and the method also includes: determining the second area in the second reconstructed image based on the reference frame identifier and / or the position indication information.

[0057] For example, when the reference image identified by the reference frame identifier of the first CU is the first reconstructed image, it can be determined that the first CU does not belong to the second region; when the reference image identified by the reference frame identifier of the first CU is not the first reconstructed image, it can be determined that the first CU belongs to the second region. Then, the second region in the first reconstructed image can be determined based on the position of the first CU belonging to the second region.

[0058] According to the second aspect, or any implementation of the second aspect above, the bitstream also includes encoded position indication information, and the position indication information is used to indicate the position of the second area; the method also includes: determining the first area in the first reconstructed image and the second area in the second reconstructed image based on the position indication information; fusing the first reconstructed image and the second reconstructed image to obtain an updated second reconstructed image, including: fusing the image of the first area in the first reconstructed image and the image of the second area in the second reconstructed image to obtain an updated second reconstructed image.

[0059] For example, the pixel values ​​of pixels within the second region of the first reconstructed image may be replaced with the pixel values ​​of pixels at corresponding positions within the second region of the second reconstructed image. For another example, the pixel values ​​of pixels within the first region of the second reconstructed image may be replaced with the pixel values ​​of pixels at corresponding positions within the first region of the first reconstructed image.

[0060] According to the second aspect, or any implementation of the second aspect above, the method further includes: using the second reconstructed image as a reconstructed image of the original image.

[0061] Exemplarily, when the second image includes the image of the second region in the original image but does not include the image of the first region in the original image, the updated second reconstructed image may be used as the reconstructed image of the original image.

[0062] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0063] In a third aspect, an embodiment of the present application provides an encoding device, the device comprising:

[0064] an image acquisition module, configured to acquire a first image and a second image, wherein the first image includes an image of a first area in an original image and does not include an image of a second area in the original image, and the second image includes an image of the second area in the original image, the size of the first image and the size of the second image being the same as the size of the original image, and the image of the second area in the original image including user information;

[0065] A base layer encoding module, configured to encode the first image at a base layer to obtain a base layer bit stream;

[0066] The enhancement layer encoding module is used to encode the enhancement layer of the second image to obtain an enhancement layer bit stream; wherein the base layer bit stream and the enhancement layer bit stream are interwoven into one bit stream.

[0067] It should be understood that the encoding device of the third aspect can be used to execute the encoding method of the first aspect or any possible implementation of the first aspect.

[0068] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0069] In a fourth aspect, an embodiment of the present application provides a decoding device, the device comprising:

[0070] A code stream receiving module, configured to receive a bit stream, the bit stream including a base layer bit stream and an enhancement layer bit stream;

[0071] a base layer decoding module, configured to perform base layer decoding on a base layer bitstream to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of the first image, the first image includes an image of the first region in the original image and does not include an image of the second region in the original image;

[0072] An enhancement layer decoding module is used to decode the enhancement layer of the enhancement layer bit stream to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, and the second image includes an image of the second area in the original image; wherein the first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

[0073] It should be understood that the decoding device of the fourth aspect can be used to execute the decoding method of the second aspect or any possible implementation of the second aspect.

[0074] The fourth aspect and any implementation of the fourth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.

[0075] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0076] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0077] In the sixth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the decoding method in the second aspect or any possible implementation of the second aspect.

[0078] The sixth aspect and any implementation of the sixth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.

[0079] In the seventh aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the steps in the first aspect or any possible implementation of the first aspect are executed.

[0080] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0081] In an eighth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps in the second aspect or any possible implementation of the second aspect are executed.

[0082] The eighth aspect and any implementation of the eighth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.

[0083] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0084] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0085] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0086] The tenth aspect and any implementation of the tenth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.

[0087] In the eleventh aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0088] The eleventh aspect and any implementation of the eleventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.

[0089] In the twelfth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0090] The twelfth aspect and any implementation of the twelfth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] FIG1A is a schematic diagram illustrating an exemplary application scenario;

[0092] FIG1B is a schematic diagram of an exemplary compression frame;

[0093] FIG2A is a schematic diagram of an exemplary compression frame;

[0094] FIG2B is a schematic diagram showing an exemplary original image, a first image, and a second image;

[0095] FIG3A is a schematic diagram illustrating an exemplary encoding process;

[0096] FIG3B is a schematic diagram illustrating an exemplary bit stream structure;

[0097] FIG4A is a schematic diagram illustrating an exemplary decoding process;

[0098] FIG4B is a schematic diagram showing an exemplary first reconstructed image and a second reconstructed image;

[0099] FIG5A is a schematic diagram illustrating an exemplary encoding process;

[0100] FIG5B is a schematic diagram illustrating an exemplary encoding process;

[0101] FIG5C is a schematic diagram showing an exemplary prediction reference;

[0102] FIG6A is a schematic diagram illustrating an exemplary decoding process;

[0103] FIG6B is a schematic diagram illustrating an exemplary decoding process;

[0104] FIG7A is a schematic diagram illustrating an exemplary encoding process;

[0105] FIG7B is a schematic diagram showing an exemplary prediction reference;

[0106] FIG8 is a schematic diagram illustrating an exemplary decoding process;

[0107] FIG9A is a schematic diagram illustrating an exemplary compression process;

[0108] FIG9B is a schematic diagram illustrating an exemplary compression process;

[0109] FIG9C is a schematic diagram illustrating an exemplary compression process;

[0110] FIG10 is a schematic diagram of an encoding device shown as an example;

[0111] FIG11 is a schematic diagram of an exemplary decoding device;

[0112] FIG12 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION

[0113] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0114] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0115] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0116] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0117] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0118] Figure 1A is a schematic diagram of an exemplary application scenario. The application scenario shown in Figure 1A is a video surveillance scenario; it should be understood that the present application can also be applied to any scenario requiring video (or image) privacy protection, and the present application does not limit this.

[0119] 1A , illustratively, image acquisition devices can be deployed at locations such as intersections, shopping mall entrances and exits, and school gates, and the image acquisition devices can collect video data, and then transmit the collected video data to at least one terminal device through the network. For example, the video data collected by the image acquisition device deployed at the intersection can be transmitted to the smart screen, the video data collected by the image acquisition device deployed at the shopping mall entrance and exit can be transmitted to the smart screen and tablet computer, the video data collected by the image acquisition device deployed at the school gate can be transmitted to the smart screen and personal computer, and so on. After the terminal device receives the video data sent by the image acquisition device, it can store and display the video data (or the video data processed as required) to facilitate the user to monitor the monitored area.

[0120] It should be noted that in this application, the actions of the image acquisition device in collecting video data, transmitting video data, and the terminal device in storing, processing and displaying video data are all carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0121] For example, the image acquisition device can compress (or encode) the captured video data to obtain a bit stream (bit stream; where the bit stream can also be called a code stream or bit stream), and then transmit the bit stream to the terminal device to reduce the amount of transmitted data and reduce bandwidth requirements.

[0122] FIG. 1B is a schematic diagram of an exemplary compression frame.

[0123] Referring to Figure 1B, illustratively, the image acquisition device may include a camera, an encoding module (or encoder), and a sending module. Exemplarily, the encoding module may be a software module or a hardware module, and the embodiments of the present application are not limited to this. It should be understood that Figure 1B is only an example of an image acquisition device. The image acquisition device in other embodiments of the present application has more modules than those shown in Figure 1B, and the embodiments of the present application are not limited to this.

[0124] Referring to FIG1B , illustratively, the terminal device may include a display module, a decoding module (or decoder), and a receiving module. Exemplarily, the decoding module may be a software module or a hardware module, which is not limited in this embodiment of the present application. It should be understood that FIG1B is only an example of a terminal device, and terminal devices in other embodiments of the present application may have more modules than those shown in FIG1B , which is not limited in this embodiment of the present application.

[0125] Continuing with Figure 1B , the exemplary process of an image acquisition device sending raw video data captured by a camera to a terminal device for display is as follows: the camera can output the captured raw video data to an encoding module; the encoding module can then encode the raw video data to obtain a bitstream and output the encoded bitstream to a transmitting module; the transmitting module can then transmit the bitstream to the terminal device. Subsequently, a receiving module of the terminal device can receive the bitstream and then output the bitstream to a decoding module; the decoding module can then decode the bitstream to obtain reconstructed video data and output the reconstructed video data to a display module, which then displays the reconstructed video data.

[0126] For example, the image acquisition device in FIG1B may be referred to as an encoding end, and the terminal device in FIG1B may be referred to as a decoding end; the encoding process of the encoding end and the decoding process of the decoding end may refer to the description of subsequent embodiments.

[0127] Figure 2A is a schematic diagram of an exemplary compression framework, wherein the dotted lines in Figure 2A represent optional operations.

[0128] 2A , illustratively, before encoding at the encoding end, the original image may be preprocessed to obtain a first image and a second image.

[0129] For example, the original image may refer to an image that is initially intended to be transmitted to a remote end. The original image may be generated in a variety of ways, such as an image captured by an image acquisition device, an image received from another terminal device, an image generated by AI (Artificial Intelligence), an image processed by an image processing tool, and the like, and this application does not impose any restrictions on this.

[0130] Exemplarily, the original image may include user information; user information may refer to information that directly or indirectly describes the user's identity. For example, user information includes, but is not limited to, user name, date of birth, ID number, address, phone number, face, license plate number, movement posture, clothing, etc., which is not limited in this application. In some scenarios, user information may also be referred to as private information. Private information may refer to secrets that one does not want to be disclosed or known to others (people outside a certain range), and this secret is irrelevant to other people and social interests.

[0131] Specifically, the process of preprocessing the original image to obtain the first image and the second image can be as follows:

[0132] First, user information detection can be performed on the original image to determine the area where the user information is located. For ease of explanation, the area in the original image that contains user information (that is, the area where the user information is located) is referred to as the second area, and the area in the original image that does not contain user information is referred to as the first area. It should be understood that the first area in the original image is the area in the original image excluding the second area. The method for detecting user information on the original image can be, for example, an AI-based intelligent detection method, and this application does not limit this.

[0133] Exemplarily, the original image may be as shown in FIG2B(1), the second region may be as shown by the dotted box in FIG2B(1), and the first region is the region in the original image of FIG2B(1) excluding the second region.

[0134] Then, a second image is generated. In one possible way, the original image can be used as the second image; that is, the second image includes the image of the first area in the original image and the image of the second area in the original image. In one possible way, the pixel values ​​of the pixels in the second area in the original image can be kept unchanged, and the pixel values ​​of the pixels in the first area in the original image can be replaced. For example, the replacement can be performed in at least one of the following ways: solid color filling, mosaic or Gaussian blur, etc., and this application does not limit this. For example, if the replacement method is solid color filling, the second image can be as shown in Figure 2B (2); in Figure 2B (2), the pixel values ​​of the pixels in the second area in the original image remain unchanged, and the pixel values ​​of the pixels in the first area in the original image are filled with white (it should be understood that other colors can also be used for filling, and this application does not limit this).

[0135] And, generate a first image. Exemplarily, the pixel values ​​of the pixels in the first area of ​​the original image can be kept unchanged, and the pixel values ​​of the pixels in the second area of ​​the original image can be replaced. For example, the replacement can be performed in at least one of the following ways: solid color filling, mosaic or Gaussian blur, etc., and this application does not limit this. For example, if the replacement method is solid color filling, the first image can be as shown in Figure 2B (3); in Figure 2B (3), the pixel values ​​of the pixels in the first area of ​​the original image remain unchanged, and the pixel values ​​of the pixels in the second area of ​​the original image are filled with white (it should be understood that other colors can also be used for filling, and this application does not limit this).

[0136] That is, the first image includes the image of the first area in the original image and does not include the image of the second area in the original image, the second image includes the image of the second area in the original image, and the size of the first image and the size of the second image are the same as the size of the original image.

[0137] It should be noted that the preprocessing operation can be performed by the encoder or by other modules, and this application does not limit this. This application takes the example of preprocessing being performed by other modules for explanation.

[0138] Afterwards, the first image and the second image may be input into the same encoder for encoding, as described in the embodiment of FIG. 3A below.

[0139] FIG3A is a schematic diagram illustrating an exemplary encoding process, wherein the steps in the embodiment of FIG3A are performed by an encoder.

[0140] S301, acquiring a first image and a second image.

[0141] S302: Encode the base layer of the first image to obtain a base layer bit stream.

[0142] S303: Encode the enhancement layer of the second image to obtain an enhancement layer bit stream.

[0143] For example, first, the first image can be input as a base layer (BL) image to the encoder, which then performs base layer encoding on the first image to obtain a base layer bitstream (hereinafter referred to as the BL bitstream). Next, the second image can be input as an enhancement layer (EL) image to the encoder, which then performs enhancement layer encoding on the second image to obtain an enhancement layer bitstream (hereinafter referred to as the EL bitstream). In other words, the encoder first performs S302 and then S303.

[0144] It should be noted that the first image and the second image are encoded by the same encoder, wherein one encoder outputs one bit stream.

[0145] In this way, after the encoder completes encoding of the first image and the second image, the BL bit stream and the EL bit stream can be interleaved into a bit stream (this bit stream can also be called the bit stream of the original image). This bit stream can include the BL bit stream and the EL bit stream, with the BL bit stream being located before the EL bit stream.

[0146] For example, for video data comprising n (n is a positive integer) frames of original images, steps S301 to S303 can be performed for each frame of original image, resulting in a bitstream of n frames of original images (hereinafter referred to as the n-frame bitstream). The first image and second image generated based on the i-th (i is an integer between 1 and n) frame of original image can be referred to as the i-th frame first image and the i-th frame second image. Thus, a bitstream output by the encoder can include n frame bitstreams, each frame of which includes an EL bitstream and a BL bitstream, with the BL bitstream preceding the EL bitstream, as shown in FIG3B .

[0147] Figure 3B is a schematic diagram illustrating an exemplary bitstream structure. In Figure 3B , the encoder outputs a bitstream consisting of n frames: the first frame bitstream (i.e., the bitstream of the first original image frame) includes the BL_1 bitstream and the EL_1 bitstream, with the BL_1 bitstream preceding the EL_1 bitstream; the second frame bitstream (i.e., the bitstream of the second original image frame) includes the BL_2 bitstream and the EL_2 bitstream, with the BL_2 bitstream preceding the EL_2 bitstream; ...; the nth frame bitstream (i.e., the bitstream of the nth original image frame) includes the BL_n bitstream and the EL_n bitstream, with the BL_n bitstream preceding the EL_n bitstream.

[0148] Illustratively, the header of the BL bitstream output by the encoder includes a frame identifier (e.g., a picture order count (POC)) and a layer identifier (Layer ID); and the header of the EL bitstream includes a frame identifier and a layer identifier. The frame identifier is used to identify a frame bitstream, and the layer identifier is used to identify the BL bitstream and EL bitstream within each frame bitstream. It should be understood that the value of the frame identifier is different for each frame bitstream, and the value of the layer identifier is different for the BL bitstream and EL bitstream within each frame bitstream.

[0149] Illustratively, for the first image and the second image generated based on the same frame original image, the frame identifier values ​​in the BL bitstream and the EL bitstream are the same; therefore, the header of the EL bitstream may not include the frame identifier, and this application does not impose any restrictions on this.

[0150] It should be noted that the EL bit stream is output after the BL bit stream, that is, after the encoder outputs the EL bit stream, the EL bit stream is connected after the BL bit stream. At this time, the EL bit stream and the BL bit stream are interleaved to form a bit stream (it can be understood that the BL bit stream and the EL bit stream are interleaved and sorted in the bit stream output by the encoder); it does not mean that the encoder performs additional operations (such as interleaving operations) to interleave the EL bit stream and the BL bit stream to form a bit stream.

[0151] Based on the above, the present application can be regarded as a layered encoding of the original image, wherein, in the layered encoding process of the original image, the encoding of the base layer corresponds to the encoded image of the base layer as the first image, and the encoding of the enhancement layer corresponds to the encoded image of the enhancement layer as the second image.

[0152] Since the reference queue for encoding the enhancement layer includes the reconstructed image of the base layer (i.e., the reconstructed image of the first image (hereinafter referred to as the first reconstructed image)), that is, the reference queue for the second image includes the first reconstructed image; compared to the prior art in which the reference queue for the second image (i.e., the privacy image) only includes the reconstructed block of the second image of the current frame and the second reconstructed images of other frames (i.e., the reconstructed images of the second images of other frames), the present application encodes more reference images for the second image. In this way, the present application can obtain more accurate prediction blocks, thereby improving the encoding performance of the enhancement layer (improving encoding quality or reducing bitrate overhead), thereby improving the overall encoding performance. In addition, in the present application, the enhancement layer bitstream and the base layer bitstream are interwoven into one bitstream, which facilitates frame synchronization of the enhancement layer bitstream and the base layer bitstream, and can reduce computational overhead.

[0153] For example, after obtaining the original image bitstream, the encoder can encrypt the EL bitstream in the original image bitstream to protect the user information in the original image during transmission to the remote end, thereby ensuring the security of the user information in the original image. The encryption operation can be performed by the system layer of the encoder.

[0154] Exemplarily, after receiving the bitstream, the decoding end can, on the one hand, input the BL bitstream in the bitstream into the decoder; on the other hand, the encrypted EL bitstream in the bitstream can be decrypted and input into the decoder; wherein the decryption operation can be performed by the system layer of the decoding end. Then, referring to Figure 2A, the decoder decodes the BL bitstream and the EL bitstream to obtain a reconstructed image of the first image (hereinafter referred to as the first reconstructed image) and a reconstructed image of the second image (hereinafter referred to as the second reconstructed image). Thereafter, the post-processing module performs post-processing based on the first reconstructed image and the second reconstructed image to obtain a reconstructed image of the original image. The decoding process of the decoder can be shown in the embodiment of Figure 4A below.

[0155] FIG4A is a schematic diagram illustrating an exemplary decoding process.

[0156] S401: Receive a bit stream, where the bit stream includes a base layer bit stream and an enhancement layer bit stream.

[0157] For example, after receiving the bitstream, the decoder can read the frame identifier from the bitstream and determine a frame bitstream according to the frame identifier. Then, the decoder can read the layer identifier from the frame bitstream to determine the BL bitstream and the EL bitstream.

[0158] S402: Decode the base layer bit stream to obtain a first reconstructed image.

[0159] Next, the decoder may decode the base layer bitstream to obtain a first reconstructed image. For example, when the first image is as shown in FIG2B(3), the first reconstructed image may be as shown in FIG4B(1).

[0160] S403 , decoding the enhancement layer bitstream to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, and the second image includes an image of the second region in the original image.

[0161] Afterwards, the decoder can decode the enhancement layer bitstream to obtain a second reconstructed image. For example, when the second image is the original image (as shown in FIG2B(1)), the second reconstructed image can be shown in FIG4B(2). When the second image includes the image of the second region in the original image and does not include the image of the first region in the original image (as shown in FIG2B(2)), the second reconstructed image can be shown in FIG4B(3).

[0162] It should be noted that, when the encoder performs lossless encoding on the first image and the second image, the first reconstructed image in FIG4B(1) is the same as the first image in FIG2B(3), the second reconstructed image in FIG4B(2) is the same as the original image in FIG2B(1), and the second reconstructed image in FIG4B(3) is the same as the second image in FIG2B(2). When the encoder performs lossy encoding on the first image and the second image, the first reconstructed image in FIG4B(1) is different from the first image in FIG2B(3), the second reconstructed image in FIG4B(2) is different from the original image in FIG2B(1), and the second reconstructed image in FIG4B(3) is different from the second image in FIG2B(2).

[0163] It should be understood that FIG4B is merely a schematic diagram of the first reconstructed image and the second reconstructed image, and does not represent the actual first reconstructed image and the second reconstructed image obtained by encoding and decoding the first image and the second image in FIG2B using the encoding and decoding method of the present application.

[0164] Referring again to FIG. 2A , illustratively, when the second image includes an image of the second region in the original image and does not include an image of the first region in the original image, the first region in the first reconstructed image and the second region in the second reconstructed image can be determined; then, the image of the first region in the first reconstructed image and the image of the second region in the second reconstructed image are fused to obtain a reconstructed image of the original image. For example, the pixel values ​​of the pixels in the second region in the first reconstructed image can be replaced with the pixel values ​​of the pixels at corresponding positions in the second region in the second reconstructed image. For another example, the pixel values ​​of the pixels in the first region in the second reconstructed image can be replaced with the pixel values ​​of the pixels at corresponding positions in the first region in the first reconstructed image.

[0165] Exemplarily, when the second image is an original image, the second reconstructed image may be used as a reconstructed image of the original image.

[0166] The following describes the encoding process of the base layer and the encoding process of the enhancement layer.

[0167] Figure 5A is a schematic diagram illustrating an exemplary encoding process. In Figure 5A , during the enhancement layer encoding process, inter-frame prediction and / or intra-frame prediction are performed on the coding units (CUs) in the second image that belong to the second region, and inter-layer prediction is performed on the CUs in the second image that do not belong to the second region (i.e., belong to the first region).

[0168] S501, obtaining a first image and a second image; wherein the first image includes an image of a first area in the original image and does not include an image of a second area in the original image, the second image includes an image of the second area in the original image, the size of the first image and the size of the second image are the same as the size of the original image, and the image of the second area in the original image contains user information.

[0169] S502: Acquire location indication information, where the location indication information can be used to indicate the location of the second area.

[0170] In one possible approach, when the second region is rectangular, the location indication information may include the coordinates of the upper left corner of the second region and the size of the second region. When the second region is circular, the location indication information may include the coordinates of the center of the second region and the radius of the second region. It should be understood that when the second region has other shapes, the location indication information may include other information, and this application does not limit this.

[0171] In one possible manner, the position indication information may be an image block identification map. Specifically, the original image may be divided into m (m is a positive integer) image blocks of a fixed size (which may be set as required, such as 2*2); then, a preset identifier is used for each image block to identify whether it is in the second area; for example, for an image block, when the image block is in the second area, the value of the preset identifier of the image block is set to the first identifier value; when the image block is not in the second area, the value of the preset identifier of the image block is set to the second identifier value; in this way, an image block identification map may be obtained; wherein the image block identification map may include m pixels, one pixel corresponding to one image block, and the pixel value of each pixel is the value of the preset identifier. The first identifier value is different from the second identifier value, and may be set as required, and this application does not impose any restrictions on this. For example, the first identifier value is "1" and the second identifier value is "0"; at this time, among the m pixels contained in the image block identification map, the pixel values ​​of m1 pixels are "1" and the pixel values ​​of m2 pixels are "0". Wherein, m1+m2=m, m1 and m2 are integers greater than or equal to 0.

[0172] It should be understood that the image block identification map may also be referred to as an image block identification matrix; the image block identification matrix may include m elements, one element corresponds to one image block, and the value of each element is a preset identification value.

[0173] In one possible manner, the above-mentioned image block may be a CU.

[0174] In a possible manner, the above-mentioned image block includes only one pixel. In this case, the position indication information may be a pixel identification map.

[0175] It should be noted that the location indication information can also be expressed in other forms, and this application does not limit this.

[0176] S503 , performing inter-frame coding or intra-frame coding on the first image to obtain a BL bit stream.

[0177] FIG5B is a schematic diagram illustrating an exemplary encoding process.

[0178] Referring to FIG. 5B (1), illustratively, S503 may include the following steps:

[0179] S5031: Divide the first image into multiple CUs.

[0180] The following description is made by taking one CU (hereinafter referred to as the third CU) among the multiple CUs obtained by dividing the first image as an example, and reference may be made to S5032 to S5033:

[0181] S5032 : Perform inter-frame prediction or intra-frame prediction on a third CU among the multiple CUs obtained by dividing the first image, and determine a prediction block of the third CU.

[0182] S5033 , encoding a residual between the third CU and the prediction block of the third CU to obtain a bit stream of the third CU; wherein the bit streams of the multiple CUs obtained by dividing the first image constitute a BL bit stream.

[0183] Exemplarily, it is possible to determine whether to perform inter-frame prediction or intra-frame prediction on the third CU based on pre-configured coding parameters. When it is determined that intra-frame prediction is to be performed on the third CU, a search can be performed from the reconstructed block of the first image of the current frame to determine the prediction block of the third CU; then, the residual between the third CU and the prediction block of the third CU is determined. Afterwards, the residual is transformed, quantized, and entropy coded to obtain the bit stream of the third CU. Exemplarily, the residual is transformed to obtain a transform coefficient, which maps the spatial domain residual to the frequency domain coefficient to obtain the transform coefficient. The transform coefficient can then be quantized to obtain the residual coefficient; and then the size of the transform coefficient can be further reduced. Then, the residual coefficient is entropy coded to obtain the bit stream of the third CU.

[0184] For example, when it is determined that inter-frame prediction is performed on the third CU, motion search and motion estimation can be performed from the first reconstructed image of the other frame (i.e., the reconstructed image of the first image of the other frame) to determine the motion vector (MV) of the third CU and the prediction block of the third CU; then, the residual between the third CU and the prediction block of the third CU is determined. Thereafter, the residual is transformed, quantized, and entropy encoded, and the MV of the third CU is entropy encoded to obtain a bitstream of the third CU.

[0185] Exemplarily, inter-frame prediction and intra-frame prediction can be performed on the third CU respectively to determine two candidate prediction blocks of the third CU: candidate prediction block 1 and candidate prediction block 2, wherein candidate prediction block 1 is determined by inter-frame prediction and candidate prediction block 2 is determined by intra-frame prediction. Then, the prediction block of the third CU can be determined based on the two candidate prediction blocks of the third CU. In one possible approach, the optimal candidate prediction block is selected from candidate prediction block 1 and candidate prediction block 2 as the prediction block of the third CU. For example, based on candidate prediction block 1 and candidate prediction block 2, the rate-distortion loss of the first image is predicted respectively (it can also be other types of loss, which is not limited in this application), and the candidate prediction block with the smallest rate-distortion loss is selected as the optimal candidate prediction block. Afterwards, the residual between the third CU and the prediction block of the third CU is determined. Then, the residual is transformed, quantized, and entropy encoded, and the MV of the third CU is entropy encoded to obtain the bit stream of the third CU.

[0186] It should be noted that the operations performed in S5033 for encoding the residual between the third CU and the prediction block of the third CU include, but are not limited to, transformation, quantization, and entropy coding. For example, the residual between the third CU and the prediction block of the third CU may be transformed and quantized to obtain residual coefficients of the third CU; then, the residual coefficients of the third CU may be entropy coded to obtain a bitstream of the third CU.

[0187] Exemplarily, a reconstruction operation may be performed during encoding of the first image to generate a first reconstructed image. The first reconstructed image may serve as a reference image for first images of other frames and / or a reference image for second images. The first reconstructed image may also be referred to as a reconstructed image of a base layer.

[0188] Referring to FIG5B(1), illustratively, the residual coefficients of the third CU can be inversely quantized and inversely transformed to obtain the residual of the third CU (wherein, the residual obtained by inverse quantization and inverse transformation is different from the residual before transformation and quantization, and this application does not distinguish between the two in terms of name); then, the residual of the third CU is added to the prediction block of the third CU to obtain the reconstructed block of the third CU. Thereafter, the reconstructed block of the third CU is loop filtered to obtain the reconstructed block of the third CU after loop filtering. In this way, the reconstructed blocks of all CUs obtained by dividing the first image after loop filtering can constitute the first reconstructed image. It should be noted that the reconstructed blocks of all CUs obtained by dividing the first image are loop filtered, that is, the image of the first area and the image of the second image in the first reconstructed image are both loop filtered. Optionally, after obtaining the first reconstructed image, the pixel value of the pixel point in the second area of ​​the first reconstructed image can be set to a preset value, such as the dynamic range median (128).

[0189] Specifically, after obtaining the first reconstructed image, the pixel points in the second area of ​​the first reconstructed image can be determined according to the position indication information obtained above, and then the pixel values ​​of the pixel points in the second area of ​​the first reconstructed image can be set to preset values.

[0190] S504: Perform inter-frame coding, intra-frame coding, or inter-layer coding on the second image to obtain an EL bit stream.

[0191] Referring to FIG. 5B (2), illustratively, S504 may include the following steps:

[0192] S5041: Divide the second image into multiple coding units CU.

[0193] It should be noted that the constraint condition for dividing the second image into multiple CUs in S5041 is: any CU cannot be divided by the boundary line of the second area; that is, any CU among the multiple CUs obtained by dividing the second image is either located within the second area or outside the second area.

[0194] The following description is made by taking one CU (hereinafter referred to as the first CU) among the multiple CUs obtained by dividing the second image as an example, and reference may be made to S5042 to S5045:

[0195] S5042 : For a first CU among the multiple CUs obtained by dividing the second image, determine whether the first CU belongs to the second region.

[0196] Exemplarily, for the first CU, it can be determined whether the first CU belongs to the second region. Specifically, the encoder can determine whether the first CU belongs to the second region based on the position information of the first CU and the position indication information obtained above.

[0197] When the first CU belongs to the second region, S5043 may be executed; when the first CU does not belong to the second region, S5044 may be executed.

[0198] S5043 : Perform inter-frame prediction or intra-frame prediction on the first CU to determine a prediction block of the first CU.

[0199] Exemplarily, S5043 may refer to the description of the process of determining the prediction block of the third CU in the above S5032, which is not repeated here.

[0200] It should be noted that in S5043, the reference image for inter-frame prediction of the first CU is the second reconstructed image of other frames (that is, the reconstructed image of the second image of other frames); the reference block for intra-frame prediction of the first CU is the reconstructed block of the second image of the current frame.

[0201] S5044: Perform inter-layer prediction on the first CU to determine a prediction block of the first CU.

[0202] For example, the reconstructed image of the base layer, that is, the first reconstructed image, can be used as a reference image for inter-layer prediction to determine the prediction block of the first CU. Specifically, for the first CU, the reconstructed block at the same position as the first CU in the first reconstructed image can be used as the prediction block of the first CU. In this way, the MV of the first CU is (0,0), and the number of bits required to encode the MV of the first CU is relatively small, which can reduce the coding rate overhead of the enhancement layer, thereby reducing the overall coding rate overhead. In addition, the amount of computational effort in motion search on the encoding side can be reduced, thereby reducing the coding complexity.

[0203] FIG5C is a schematic diagram showing an exemplary prediction reference.

[0204] 5C , illustratively, assume that time k (k is a positive integer) is the current time, corresponding to the first image of the current frame and the second image of the current frame; time k-1 is the previous time, corresponding to the first reconstructed image of the previous frame and the second reconstructed image of the previous frame.

[0205] 5C , exemplarily, inter-frame prediction may be performed for the third CU in the first region of the first image of the current frame, as shown by R01 and R02; wherein R01 indicates that the inter-frame prediction refers to the reconstructed block in the first region of the first reconstructed image of the previous frame, and R02 indicates that the inter-frame prediction refers to the reconstructed block in the second region of the first reconstructed image of the previous frame.

[0206] 5C , exemplarily, intra-frame prediction may be performed for the third CU within the first region in the first image of the current frame, as shown in R03 and R04; wherein R03 indicates that intra-frame prediction refers to the reconstructed block within the first region in the first image of the current frame, and R04 indicates that intra-frame prediction refers to the reconstructed block within the second region in the first image of the current frame.

[0207] 5C , exemplarily, inter-frame prediction may be performed for the third CU in the second region in the first image of the current frame, as shown in R05 and R06; wherein R05 indicates that the inter-frame prediction refers to the reconstructed block in the first region in the first reconstructed image of the previous frame, and R06 indicates that the inter-frame prediction refers to the reconstructed block in the second region in the first reconstructed image of the previous frame.

[0208] 5C , exemplarily, intra-frame prediction may be performed for the third CU in the second region in the first image of the current frame, as shown in R07 and R08; wherein R07 indicates that intra-frame prediction refers to the reconstructed block in the first region in the first image of the current frame, and R08 indicates that intra-frame prediction refers to the reconstructed block in the second region in the first image of the current frame.

[0209] 5C , exemplarily, inter-layer prediction may be performed for the first CU in the first area of ​​the second image of the current frame, as shown in R09; wherein R09 indicates that the inter-layer prediction may refer to the reconstructed block of the first area in the first reconstructed image of the current frame (i.e., the reconstructed image of the base layer).

[0210] 5C , exemplarily, inter-frame prediction may be performed for the first CU in the second area in the second image of the current frame, as shown in R10 and R11; wherein R10 represents inter-frame prediction referring to the reconstructed block in the first area in the second reconstructed image of the previous frame (the reconstructed image of the enhancement layer of the previous frame), and R11 represents inter-frame prediction referring to the reconstructed block in the second area in the second reconstructed image of the previous frame.

[0211] 5C , exemplarily, intra-frame prediction may be performed for the first CU in the second region in the second image of the current frame, as shown in R12 and R13; wherein R12 indicates that intra-frame prediction refers to the reconstructed block in the first region in the second image of the current frame, and R13 indicates that intra-frame prediction refers to the reconstructed block in the second region in the second image of the current frame.

[0212] S5045 , encoding a residual between the first CU and the prediction block of the first CU to obtain a bit stream of the first CU; wherein the bit streams of the multiple CUs obtained by dividing the second image constitute an EL bit stream.

[0213] Exemplarily, when the first CU does not belong to the second region (i.e., the first CU belongs to the first region), the residual between the first CU and the prediction block of the first CU can be forcibly set to 0. In this way, the number of bits required to encode the residual between the first CU and the prediction block of the first CU is smaller, which can reduce the encoding bit rate overhead of the enhancement layer, thereby reducing the overall bit rate overhead.

[0214] It should be noted that the operations performed in S5045 for encoding the residual between the first CU and the prediction block of the first CU include, but are not limited to, transformation, quantization, and entropy coding. For example, the residual between the first CU and the prediction block of the first CU may be transformed and quantized to obtain residual coefficients of the first CU; then, the residual coefficients of the first CU may be entropy coded to obtain a bitstream of the first CU.

[0215] For example, a reconstruction operation may be performed during the encoding of the second image to obtain a second reconstructed image, which may be used as a reference image for subsequent encoding of other second images. The second reconstructed image may also be referred to as a reconstructed image of an enhancement layer.

[0216] 5B (2), exemplarily, when the first CU belongs to the second region, the residual coefficients of the first CU can be inversely quantized and inversely transformed to obtain the residual of the first CU; then, the residual of the first CU is added to the prediction block of the first CU to obtain the reconstructed block of the first CU. Afterwards, the reconstructed block of the first CU can be loop-filtered to obtain the reconstructed block of the first CU after loop filtering. When the first CU does not belong to the second region, the prediction block of the first CU can be used as the reconstructed block of the first CU, and there is no need to loop-filter the reconstructed block of the first CU. That is, the reconstructed blocks of the CUs located in the second region of all the CUs obtained by dividing the second image are loop-filtered, and the CUs located in the first region are not loop-filtered, that is, the image of the first region in the second reconstructed image of the current frame is the same as the image of the first region in the first reconstructed image of the current frame.

[0217] Specifically, whether the first CU belongs to the second region can be determined based on the position indication information obtained above. For details, please refer to the description of S5042 above, which will not be repeated here.

[0218] S505: Add the encoded position indication information to the BL bit stream and / or the BL bit stream.

[0219] It should be noted that S505 is an optional step.

[0220] Exemplarily, the encoded position indication information may be added to the front end or the end of the BL bit stream, and / or the encoded position indication information may be added to the end of the EL bit stream.

[0221] Exemplarily, after the encoded position indication information is added to the BL bitstream and / or the BL bitstream, the decoding end can constrain the loop filtering of the first area and the second area based on the position indication information parsed from the BL bitstream and / or the BL bitstream, and fuse the first reconstructed image and the second reconstructed image.

[0222] When the encoded position indication information is not added to the BL bitstream and / or the BL bitstream, the decoding end may derive the position information of the second region in other ways, which will be described in detail later.

[0223] It should be noted that the encoding end can also add encoding control parameters (such as prediction mode (which may include but is not limited to inter-frame prediction mode, intra-frame prediction mode and inter-layer prediction mode), division information (such as CU division method), transformation parameters and quantization parameters, etc.) to the bit stream of the original image, so that the decoding end can use the corresponding control parameters for decoding control and decode the reconstructed image of the original image.

[0224] Fig. 6A is a schematic diagram illustrating an exemplary decoding process, wherein the decoding process of Fig. 6A corresponds to the encoding process of Fig. 5A.

[0225] S601: Receive a bitstream, where the bitstream includes a base layer bitstream, an enhancement layer bitstream, and position indication information.

[0226] For example, after receiving the bit stream, the decoding end may parse the bit stream, determine each frame bit stream according to the frame identifier and layer identifier in the bit stream, and determine the BL bit stream and EL bit stream in each frame bit stream.

[0227] Exemplarily, in the embodiment of FIG. 5A , when the encoding end executes S505 , ie, adds the encoded position indication information to the BL bitstream and / or the BL bitstream, the decoding end may further parse the position indication information from the received bitstream.

[0228] Exemplarily, the decoding end may also parse out control parameters (such as prediction mode (inter-frame prediction or intra-frame prediction), partition information (such as CU partitioning method), transform parameters and quantization parameters, etc.) from the received bit stream.

[0229] The following describes decoding of the BL bit stream and the EL bit stream in a frame bit stream.

[0230] S602: Decode the base layer bit stream to obtain a first reconstructed image.

[0231] FIG6B is a schematic diagram illustrating an exemplary decoding process.

[0232] Exemplarily, the decoder may determine a division mode of the first image based on the division information parsed from the bitstream, and further determine the multiple CUs included in the first image; a decoding process for one CU (which may be referred to as a third CU) among the multiple CUs included in the first image may refer to S6021 to S6023:

[0233] S6021 : Determine a reference image of the third CU according to the reference frame identifier and prediction mode parsed from the BL bitstream.

[0234] S6022: Predict the third CU according to the reference image of the third CU to determine a prediction block of the third CU.

[0235] For example, when the decoder determines that the preset mode of the third CU is the inter-frame prediction mode based on the prediction mode parsed from the BL bitstream, the decoder can determine the reference image of the third CU from the first reconstructed image of another frame based on the reference frame identifier parsed from the BL bitstream. In this case, the decoder can also parse the MV of the third CU from the BL bitstream. Then, based on the MV of the third CU, the decoder determines the prediction block of the third CU from the reference image of the third CU.

[0236] Exemplarily, when the decoder determines that the prediction mode of the third CU is the intra-frame prediction mode based on the prediction mode parsed from the bit stream, the prediction block of the third CU can be determined from the reconstructed pixels adjacent to the third CU space based on the angle information parsed from the BL bit stream (or other information for determining the reference block or pre-agreed intra-frame prediction reference rules).

[0237] S6023: Add the residual of the third CU parsed from the BL bitstream and the prediction block of the third CU to obtain a reconstructed block of the third CU; wherein the reconstructed blocks of multiple CUs constitute a first reconstructed image.

[0238] Exemplarily, the decoder can also parse the residual of the third CU from the BL bit stream; then, the residual of the third CU and the prediction block of the third CU can be added to obtain the reconstructed block of the third CU; thereafter, the reconstructed block of the third CU can be loop filtered. For details, please refer to the loop filtering process for the reconstructed block of the third CU in the embodiment of Figure 5A above, which will not be repeated here.

[0239] It should be noted that, in the embodiment of FIG6A , the second area in the first reconstructed image can be determined based on the position indication information parsed from the bit stream by the decoder; and then the pixel values ​​of the pixel points in the second area in the first reconstructed image are set to preset values, such as the dynamic range median (128).

[0240] S603: Decode the enhancement layer on the enhancement layer bit stream to obtain a second reconstructed image.

[0241] Exemplarily, the decoder may determine a division mode of the second image based on the division information parsed from the bitstream, and further determine the multiple CUs included in the second image; a decoding process for one CU (which may be referred to as a first CU) among the multiple CUs included in the second image may refer to S6031 to S6035:

[0242] S6031 : Determine a reference image of the first CU according to the reference frame identifier and prediction mode parsed from the EL bitstream.

[0243] S6032: Predict the first CU according to the reference image of the first CU to determine a prediction block of the first CU.

[0244] S6033: Add the residual of the first CU parsed from the EL bit stream and the prediction block of the first CU to obtain a reconstructed block of the first CU; wherein the reconstructed blocks of multiple CUs constitute a second reconstructed image.

[0245] It should be noted that in S6033, when the residual of the first CU parsed from the EL bitstream is 0 and the MV of the first CU is (0,0), the reconstructed block at the same position as the first CU in the first reconstructed image can be used as the prediction block of the first CU.

[0246] Exemplarily, after obtaining the reconstructed block of the first CU, loop filtering may be performed on the reconstructed block of the first CU. For details, reference may be made to the description of loop filtering on the reconstructed block of the first CU in the embodiment of FIG. 5A , which will not be repeated here.

[0247] It should be noted that in the embodiment of FIG6A , when the decoder parses the position indication information from the bitstream, it can determine whether to perform loop filtering on the reconstructed block of the first CU based on the position indication information parsed from the bitstream. Specifically, it can determine whether the first CU belongs to the second region based on the position indication information. When the first CU belongs to the second region, loop filtering can be performed on the reconstructed block of the first CU; when the first CU does not belong to the second region, loop filtering can be omitted.

[0248] When the decoder does not parse the position indication information from the bitstream, after obtaining the reconstructed block of the first CU, if the reference image identified by the reference frame identifier of the first CU is not the reconstructed image of the base layer, it means that the first CU belongs to the second region, and at this time, loop filtering can be performed on the reconstructed block of the first CU. If the reference image identified by the reference frame identifier of the first CU is the reconstructed image of the base layer, it means that the first CU does not belong to the second region, and at this time, loop filtering does not need to be performed on the reconstructed block of the first CU.

[0249] S604: Use the second reconstructed image as a reconstructed image of the original image.

[0250] Exemplarily, since the image of the first area in the second reconstructed image is the same as the image of the first area in the first reconstructed image, and the image of the second area in the second reconstructed image is a reconstructed image of the image of the second area in the second image; therefore, the second reconstructed image can be directly used as the reconstructed image of the original image.

[0251] Figure 7A is a schematic diagram illustrating an exemplary encoding process. In the embodiment of Figure 7A , during the enhancement layer encoding process, at least one of inter-frame prediction, intra-frame prediction, and inter-layer prediction can be performed on the CU in the second image, regardless of whether it belongs to the second region or the CU belongs to the first region.

[0252] S701, obtaining a first image and a second image; wherein the first image includes an image of a first area in the original image and does not include an image of a second area in the original image, the second image includes an image of the second area in the original image, the size of the first image and the size of the second image are the same as the size of the original image, and the image of the second area in the original image contains user information.

[0253] S702: Acquire location indication information, where the location indication information can be used to indicate the location of the second area.

[0254] S703 , performing inter-frame prediction or intra-frame prediction on the first image to obtain a BL bit stream.

[0255] Exemplarily, S703 may include the following steps:

[0256] S7031: Divide the first image into multiple CUs.

[0257] S7032 : Perform inter-frame prediction or intra-frame prediction on a third CU among the multiple CUs obtained by dividing the first image, and determine a prediction block of the third CU.

[0258] S7033 , encode a residual between the third CU and the prediction block of the third CU to obtain a bit stream of the third CU; wherein the bit streams of the multiple CUs obtained by dividing the first image constitute a BL bit stream.

[0259] For example, S7031 to S7033 may refer to the description of S5031 to S5033 above, which will not be repeated here.

[0260] It should be noted that, unlike the embodiment of Figure 5A, in the embodiment of Figure 7A, after loop filtering is performed on all reconstructed blocks of the third CU to obtain the first reconstructed image, there is no need to set the pixel values ​​of the pixel points in the second area in the first reconstructed image to preset values.

[0261] S704 , perform inter-frame coding, intra-frame coding, or inter-layer coding on the second image to obtain an EL bit stream.

[0262] Exemplarily, S704 may include the following steps S7041 to S7043:

[0263] S7041: Divide the second image into multiple CUs.

[0264] It should be noted that, unlike S5041, in S7041, no constraint conditions may be set when dividing the second image into multiple CUs.

[0265] The following description is made by taking one CU (hereinafter referred to as the second CU) among the multiple CUs obtained by dividing the second image as an example, and reference may be made to S7042 to S7043:

[0266] S7042 , performing inter-frame prediction, intra-frame prediction, or inter-layer prediction on a second CU among the multiple CUs, and determining a prediction block of the second CU.

[0267] For example, the processes of inter-frame prediction, intra-frame prediction and inter-layer prediction of the second CU may be referred to the description in the embodiment of FIG5A and will not be repeated here.

[0268] For example, motion search and motion estimation can be performed from the first reconstructed image of the current frame to determine the motion vector (MV) of the second CU and the prediction block of the second CU. In other words, in S7042, no restriction is placed on the position of the reconstructed block referenced by the inter-layer prediction.

[0269] Compared with the embodiment of Figure 5A, in the embodiment of Figure 7A, during the encoding process of the enhancement layer, there are more reference images in the reference queue of the second image. In this way, a more accurate prediction block can be obtained, which can improve the encoding quality of the enhancement layer and thus improve the reconstruction quality of the reconstructed image of the original image.

[0270] S7043: Encode a residual between the second CU and the prediction block of the second CU.

[0271] Exemplarily, a reconstruction operation is also performed during the encoding of the second image to obtain a second reconstructed image; for details, please refer to the description of the embodiment in FIG5A . It should be noted that, unlike the embodiment in FIG5A , in which loop filtering is performed on the reconstructed blocks of the first CU, in the embodiment in FIG7A , loop filtering can be performed on all reconstructed blocks of the second CU. In other words, the reconstructed blocks of all CUs in the second reconstructed image are loop-filtered reconstructed blocks.

[0272] It should be noted that when the second image only includes the image of the second region of the original image and does not include the image of the first region of the original image, the image of the first region in the second reconstructed image after loop filtering is not a reconstructed image of the image of the first region of the original image. In this case, for reference in the subsequent encoding of the second image of other frames, the first reconstructed image and the second reconstructed image can be fused to obtain an updated second reconstructed image. In this case, the updated second reconstructed image is also a reconstructed image of the original image.

[0273] FIG. 7B is a schematic diagram showing an exemplary prediction reference.

[0274] 7B , illustratively, assume that time k (k is a positive integer) is the current time, corresponding to the first image of the current frame and the second image of the current frame; time k-1 is the previous time, corresponding to the first reconstructed image of the previous frame and the second reconstructed image of the previous frame.

[0275] Illustratively, during the prediction process, the reconstructed block referenced by the third CU of the first image in FIG. 7B is similar to the reconstructed block referenced by the third CU of the first image in FIG. 5C , and details thereof will not be repeated herein.

[0276] 7B , exemplarily, inter-layer prediction may be performed for the second CU in the second area in the second image of the current frame, as shown in R09 and R14; wherein R09 indicates that the inter-layer prediction may refer to the reconstructed block of the first area in the first reconstructed image of the current frame (i.e., the reconstructed image of the base layer), and R14 indicates that the inter-layer prediction may refer to the reconstructed block of the second area in the first reconstructed image of the current frame (i.e., the reconstructed image of the base layer).

[0277] 7B , exemplarily, inter-frame prediction may be performed for the second CU in the second area in the second image of the current frame, as shown in R10 and R11; wherein R10 represents inter-frame prediction referring to the reconstructed block in the first area in the second reconstructed image of the previous frame (the reconstructed image of the enhancement layer of the previous frame), and R11 represents inter-frame prediction referring to the reconstructed block in the second area in the second reconstructed image of the previous frame.

[0278] 7B , exemplarily, intra-frame prediction may be performed for the second CU in the second region in the second image of the current frame, as shown in R12 and R13; wherein R12 indicates that intra-frame prediction refers to the reconstructed block in the first region in the second image of the current frame, and R13 indicates that intra-frame prediction refers to the reconstructed block in the second region in the second image of the current frame.

[0279] Similarly, for the second CU in the first area of ​​the second image of the current frame, inter-frame prediction, intra-frame prediction and inter-layer prediction can be performed, as shown in R15 and R16, R17 and R18, R19 and R20, and will not be repeated here.

[0280] S705: Add the encoded position indication information to the BL bit stream and / or the BL bit stream.

[0281] Exemplarily, when the second image is an original image, S705 is an optional step.

[0282] Exemplarily, when the second image includes the image of the second area in the original image but does not include the image of the first area in the original image, S705 is a necessary step.

[0283] It should be noted that the encoding end may also encode encoding control parameters into the bit stream of the original image, so that the decoding end can use the corresponding control parameters to perform decoding control and decode a reconstructed image of the original image.

[0284] Fig. 8 is a schematic diagram illustrating an exemplary decoding process, which corresponds to the encoding process of Fig. 7A.

[0285] S801: Receive a bit stream, where the bit stream includes a base layer bit stream, an enhancement layer bit stream, and position indication information.

[0286] S802: Decode the base layer bit stream to obtain a first reconstructed image.

[0287] The decoding process for one CU (which may be referred to as a third CU) among the multiple CUs included in the first reconstructed image may refer to S8021 to S8023:

[0288] S8021 : Determine a reference image of the third CU according to the reference frame identifier and prediction mode parsed from the BL bitstream.

[0289] S8022: Predict the third CU according to the reference image of the third CU to determine a prediction block of the third CU.

[0290] S8023: Add the residual of the third CU parsed from the BL bitstream and the prediction block of the third CU to obtain a reconstructed block of the third CU; wherein the reconstructed blocks of multiple CUs constitute a first reconstructed image.

[0291] For example, S8021 to S8023 may refer to the description of S6021 to S6023 above, which will not be repeated here.

[0292] S803: Decode the enhancement layer bit stream to obtain a second reconstructed image.

[0293] Illustratively, S803 may refer to the description of S8031 to S8033 below.

[0294] S8031 : Determine a reference image of the second CU according to the reference frame identifier and prediction mode parsed from the EL bitstream.

[0295] S8032: Predict the second CU according to the reference image of the second CU to determine a prediction block of the second CU.

[0296] S8033: Add the residual of the second CU parsed from the EL bit stream and the prediction block of the second CU to obtain a reconstructed block of the second CU; wherein the reconstructed blocks of multiple CUs constitute a second reconstructed image.

[0297] For example, S8031 to S8033 may refer to the description of S6021 to S6023 above, which will not be repeated here.

[0298] S804: Determine a reconstructed image of the original image according to the first reconstructed image and the second reconstructed image.

[0299] Exemplarily, when the second image only includes the image of the second area in the original image and does not include the image of the first area in the original image, the image of the first area in the first reconstructed image and the image of the second area in the second reconstructed image can be determined based on the position indication information parsed from the bitstream; then, the image of the first area in the first reconstructed image and the image of the second area in the second reconstructed image are fused to obtain an updated second reconstructed image, that is, a reconstructed image of the original image. For example, the pixel value of the pixel point in the second area in the first reconstructed image can be replaced with the pixel value of the pixel point at the corresponding position in the second area in the second reconstructed image. For another example, the pixel value of the pixel point in the first area in the second reconstructed image can be replaced with the pixel value of the pixel point at the corresponding position in the first area in the first reconstructed image.

[0300] For example, when the second image is the original image, the second reconstructed image obtained after loop filtering is the reconstructed image of the original image. At this time, there is no need to fuse the first reconstructed image and the second reconstructed image to obtain the reconstructed image of the original image. The second reconstructed image can be directly used as the reconstructed image of the original image.

[0301] FIG9A is a schematic diagram illustrating an exemplary compression process. In FIG9A , the second image is the original image. In this case, the second reconstructed image can be used as a reconstructed image of the original image. The encoding and decoding process of FIG9A can be referred to the descriptions in FIG5A and FIG6A (or the descriptions in FIG7A and FIG8 ), and will not be repeated here.

[0302] Figure 9B is a schematic diagram illustrating an exemplary compression process. In Figure 9B , the second image includes the image of the second region of the original image and does not include the image of the first region of the original image. In this case, the second reconstructed image can be used as the reconstructed image of the original image. The encoding and decoding process of Figure 9B can be referred to the descriptions of Figures 5A and 6A and will not be repeated here.

[0303] Figure 9C is a schematic diagram illustrating an exemplary compression process. In Figure 9C , the second image includes the image of the second region of the original image but does not include the image of the first region of the original image. In this case, a reconstructed image of the original image can be generated by fusing the first reconstructed image and the second reconstructed image. The encoding and decoding process of Figure 9C can be referred to the descriptions of Figures 7A and 8 and will not be repeated here.

[0304] Figure 10 is a schematic diagram of an exemplary encoding device. The encoding device in Figure 10 can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0305] An image acquisition module 1001 is configured to acquire a first image and a second image, wherein the first image includes an image of a first region in an original image and does not include an image of a second region in the original image, and the second image includes an image of the second region in the original image. The sizes of the first image and the second image are the same as the sizes of the original image, and the image of the second region in the original image includes user information.

[0306] A base layer encoding module 1002 is configured to perform base layer encoding on the first image to obtain a base layer bit stream;

[0307] The enhancement layer encoding module 1003 is configured to perform enhancement layer encoding on the second image to obtain an enhancement layer bit stream; wherein the base layer bit stream and the enhancement layer bit stream are interwoven into one bit stream.

[0308] In one possible scenario, the image acquisition module 1001, the base layer encoding module 1002 and the enhancement layer encoding module 1003 can all be regarded as modules in an encoder.

[0309] Figure 11 is a schematic diagram of an exemplary decoding device. The decoding device in Figure 11 can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0310] A code stream receiving module 1101 is configured to receive a bit stream, where the bit stream includes a base layer bit stream and an enhancement layer bit stream;

[0311] a base layer decoding module 1102 configured to perform base layer decoding on a base layer bitstream to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of the first image, the first image includes an image of the first region in the original image and does not include an image of the second region in the original image;

[0312] The enhancement layer decoding module 1103 is used to decode the enhancement layer of the enhancement layer bit stream to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, and the second image includes an image of the second area in the original image; wherein the first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

[0313] In one possible scenario, the code stream receiving module 1101, the base layer decoding module 1102, and the enhancement layer decoding module 1103 can all be considered as modules in a decoder.

[0314] In an example, FIG12 shows a schematic block diagram of a device 1200 according to an embodiment of the present application. The device 1200 may include: a processor 1201 and a transceiver / transceiver pin 1202 , and optionally, a memory 1203 .

[0315] The various components of the device 1200 are coupled together via a bus 1204, wherein the bus 1204 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 1204 in the figure.

[0316] Optionally, the memory 1203 may be used to store instructions in the aforementioned method embodiment. The processor 1201 may be used to execute the instructions in the memory 1203 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0317] The apparatus 1200 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0318] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0319] The present application also provides a chip including one or more interface circuits and one or more processors. The one or more processors receive or send data via the one or more interface circuits. When the one or more processors execute computer instructions, the steps of the above embodiment are performed. The interface circuit is a transceiver / transceiver pin 1202.

[0320] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0321] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0322] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0323] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0324] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0325] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0326] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0327] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0328] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0329] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0330] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0331] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0332] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0333] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A layered coding method, It is characterized in that The method comprises: Acquire a first image and a second image, wherein the first image includes an image of a first area in an original image and does not include an image of a second area in the original image, and the second image includes an image of the second area in the original image, a size of the first image and a size of the second image are the same as a size of the original image, and the image of the second area in the original image includes user information; Performing base layer encoding on the first image to obtain a base layer bit stream; performing encoding of an enhancement layer on the second image to obtain an enhancement layer bit stream; The base layer bit stream and the enhancement layer bit stream are interleaved into one bit stream.

2. The method according to claim 1, It is characterized in that The second image also includes an image of the first area in the original image; or, The second image does not include an image of the first area in the original image.

3. The method according to claim 1 or 2, It is characterized in that The encoding of the enhancement layer of the second image to obtain an enhancement layer bit stream includes: Performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the second image to obtain a predicted image of the second image; A residual between the second image and a predicted image of the second image is encoded to obtain the enhancement layer bitstream.

4. The method according to claim 3, It is characterized in that The performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the second image to obtain a predicted image of the second image includes: Dividing the second image to obtain a plurality of coding units CU; For a first CU among the multiple CUs, determining whether the first CU belongs to the second region; When the first CU belongs to the second region, performing inter-frame prediction or intra-frame prediction on the first CU to determine a prediction block of the first CU; When the first CU does not belong to the second region, performing inter-layer prediction on the first CU to determine a prediction block of the first CU; The predicted image of the second image includes prediction blocks of multiple CUs obtained by dividing the second image.

5. The method according to claim 4, It is characterized in that The performing inter-layer prediction on the first CU to determine a prediction block of the first CU includes: Using a reconstructed block in the first reconstructed image that is located at the same position as the first CU as a prediction block of the first CU; The first reconstructed image is a reconstructed image of the first image.

6. The method according to claim 4 or 5, It is characterized in that When the first CU does not belong to the second region, a residual between the prediction block of the first CU and the first CU is 0.

7. The method according to claim 3, It is characterized in that The performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the second image to obtain a predicted image of the second image includes: Dividing the second image to obtain multiple CUs; Performing inter-frame prediction, intra-frame prediction or inter-layer prediction on a second CU among the multiple CUs to determine a prediction block of the second CU; The predicted image of the second image includes prediction blocks of multiple CUs obtained by dividing the second image.

8. The method according to any one of claims 4 to 7, It is characterized in that Any CU among the multiple CUs is located inside or outside the second area.

9. The method according to any one of claims 1 to 8, It is characterized in that The method further comprises: generating a first reconstructed image; The first reconstructed image is a reconstructed image of the first image, and the first reconstructed image is a reference image of the first image of other frames. the first reconstructed image is an image after loop filtering, and the pixel values ​​of the pixels in the second area of ​​the first reconstructed image are preset values.

10. The method according to any one of claims 1 to 9, It is characterized in that The method further comprises: generating a second reconstructed image; Among them, the second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the second image of other frames, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

11. The method according to claim 10, It is characterized in that The method further comprises: When the second image does not include the image of the first area in the original image, fusing the first reconstructed image and the second reconstructed image to obtain an updated second reconstructed image; The first reconstructed image is a reconstructed image of the first image.

12. The method according to any one of claims 1 to 11, It is characterized in that The method further comprises: Acquire location indication information, where the location indication information is used to indicate the location of the second area; The encoded position indication information is added to the base layer bit stream and / or the enhancement layer bit stream.

13. The method according to any one of claims 1 to 12, It is characterized in that The first area in the original image is an area in the original image excluding the second area; The second image is generated by replacing pixel values ​​of pixel points in the first area of ​​the original image; The first image is generated by replacing pixel values ​​of pixel points in a second area in the original image; The replacement method includes at least one of the following: solid color filling, mosaic or Gaussian blur.

14. The method according to any one of claims 1 to 13, It is characterized in that The base layer bitstream and the enhancement layer bitstream include reference frame identification and prediction mode.

15. A layered decoding method, It is characterized in that The method comprises: receiving a bitstream, the bitstream comprising a base layer bitstream and an enhancement layer bitstream; Perform base layer decoding on the base layer bit stream to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of a first image, the first image includes an image of a first area in an original image and does not include an image of a second area in the original image; Decoding the enhancement layer bit stream to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of a second image, and the second image includes an image of a second area in the original image; The first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

16. The method according to claim 15, It is characterized in that The second image also includes an image of the first area in the original image; or, The second image does not include an image of the first area in the original image.

17. The method according to claim 15 or 16, It is characterized in that The bitstream further includes a reference frame identifier and a prediction mode, and the decoding of the enhancement layer bitstream to obtain a second reconstructed image includes: Determining a reference image for the second image according to the reference frame identifier and the prediction mode; Performing prediction according to a reference image of the second image to obtain a predicted image of the second image; The predicted image of the second image is added to the residual of the second image parsed from the enhancement layer bitstream to obtain the second reconstructed image.

18. The method according to claim 17, It is characterized in that The second image includes a plurality of coding units CU, the reference image of the second image includes reference images of the plurality of CUs, and performing prediction according to the reference image of the second image to obtain a predicted image of the second image includes: For a first CU among the multiple CUs, predict the first CU according to a reference image of the first CU to determine a prediction block of the first CU; The predicted image of the second image includes prediction blocks of multiple CUs included in the second image.

19. The method according to any one of claims 15 to 18, It is characterized in that The first reconstructed image is an image after loop filtering, and the pixel values ​​of the pixels in the second area of ​​the first reconstructed image are preset values.

20. The method according to any one of claims 15 to 19, It is characterized in that The image of the second area in the second reconstructed image is an image that has been loop filtered, and the image of the first area in the second reconstructed image is an image that has not been loop filtered.

21. The method according to any one of claims 15 to 20, It is characterized in that The method further comprises: When the second image does not include the image of the first area in the original image, the first reconstructed image and the second reconstructed image are fused to obtain an updated second reconstructed image.

22. The method according to claim 19, It is characterized in that The bit stream further includes encoded position indication information, where the position indication information is used to indicate the position of the second area; and the method further includes: A second area in the first reconstructed image is determined according to the position indication information.

23. The method according to claim 20 or 21, It is characterized in that The bitstream further includes a reference frame identifier and / or encoded position indication information, where the position indication information is used to indicate the position of the second area, and the method further includes: A second region in the second reconstructed image is determined according to the reference frame identifier and / or the position indication information.

24. The method according to claim 21, It is characterized in that The bit stream further includes encoded position indication information, where the position indication information is used to indicate the position of the second area; and the method further includes: determining, according to the position indication information, a first region in the first reconstructed image and a second region in the second reconstructed image; The fusing the first reconstructed image and the second reconstructed image to obtain an updated second reconstructed image includes: The image of the first area in the first reconstructed image and the image of the second area in the second reconstructed image are fused to obtain an updated second reconstructed image.

25. The method according to any one of claims 15 to 24, It is characterized in that The method further comprises: The second reconstructed image is used as the reconstructed image of the original image.

26. An electronic device, It is characterized in that include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the encoding method according to any one of claims 1 to 14.

27. An electronic device, It is characterized in that include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the decoding method according to any one of claims 15 to 25.

28. A chip, It is characterized in that It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps described in any one of claims 1 to claim 25 are performed.

29. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 25.

30. A computer program product, It is characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 25 are performed.

Images