Encoding and decoding method and electronic equipment

Abstract

The embodiment of the invention provides a coding and decoding method and electronic equipment. The coding method comprises the steps that firstly, a first image and a second image are obtained, the first image comprises an image of a first area in an original image and does not comprise an image of a second area in the original image, the second image comprises an image of the second area in the original image, and the size of the first image and the size of the second image are the same as the size of the original image; the image of the second area in the original image contains user information; then, carrying out basic layer coding on the first image to obtain a basic layer bit stream; afterwards, performing enhancement layer coding on the second image to obtain an enhancement layer bit stream; wherein the basic layer bit stream and the enhancement layer bit stream are interwoven into a path of bit stream. Therefore, the coding performance can be improved and the operation overhead can be reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of coding and decoding, and in particular to a coding and decoding method and an electronic device. Background Art

[0002] With the rapid development of the Internet and multimedia technology, as well as the continuous decline in the hardware cost and implementation cost of video surveillance, video surveillance technology has been widely used; for example, video surveillance technology is used to implement remote video surveillance of target areas to assist in public security management, accident warning, etc.

[0003] However, the collected video images may carry user information (such as faces, license plate numbers, etc.). If the video images are leaked, the user information will also be leaked, which may cause losses to the users. Therefore, video privacy protection technology comes into being.

[0004] In the prior art, the encoding end inputs the privacy area image and the non-privacy area image of the video image into two encoders for encoding, and obtains a bit stream of the privacy area image and a bit stream of the non-privacy area image; then, the bit stream of the non-privacy area image and the encrypted bit stream of the privacy area image are transmitted; in this way, video privacy protection can be achieved. However, the privacy area image and the non-privacy area image are encoded independently, which reduces the encoding performance to a certain extent; in addition, the system layer needs to synchronize the bit stream of the privacy area image with the bit stream of the non-privacy area image to ensure that the decoding end can obtain a complete image, which will bring computing overhead. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides an electronic device for a coding and decoding method, which can improve coding performance and reduce computational overhead.

[0006] In a first aspect, an embodiment of the present application provides a coding method, the method comprising: first, obtaining a first image and a second image, the first image comprising an image of a first area in the original image and excluding an image of a second area in the original image, the second image comprising an image of the second area in the original image, the size of the first image and the size of the second image being the same as the size of the original image, and the image of the second area in the original image comprising user information; then, encoding the first image at a base layer to obtain a base layer bit stream; thereafter, encoding the second image at an enhancement layer to obtain an enhancement layer bit stream; wherein the base layer bit stream and the enhancement layer bit stream are interwoven into one bit stream.

[0007] Since the reference queue for encoding the enhancement layer includes the reconstructed image of the base layer (that is, the reconstructed image of the first image (hereinafter referred to as the first reconstructed image)), that is to say, the reference queue of the second image includes the first reconstructed image; compared with the prior art in which the reference queue of the second image (that is, the privacy image) only includes the reconstructed block of the second image of the current frame and the second reconstructed images of other frames (that is, the reconstructed images of the second images of other frames), the present application encodes more reference images for the second image, so that the present application can obtain more accurate prediction blocks, and thus can improve the encoding performance of the enhancement layer (that is, improve the encoding quality or reduce the bit rate overhead), thereby improving the overall encoding performance. In addition, in the present application, the enhancement layer bit stream and the base layer bit stream are interwoven into one bit stream, which facilitates the frame synchronization of the enhancement layer bit stream and the base layer bit stream, and can reduce computational overhead.

[0008] Exemplarily, the original image may refer to an image that is originally expected to be transmitted to the remote end. The original image may be generated in a variety of ways, for example, the original image may be an image captured by an image acquisition device, an image received from other terminal devices, an image generated by AI (Artificial Intelligence), an image processed by an image processing tool, etc., and this application does not limit this.

[0009] Exemplarily, the original image may include a first area and a second area, wherein the second area image of the original image may include user information, and the user information may refer to information that directly or indirectly describes the identity of the user. For example, user information includes, but is not limited to: user name, date of birth, ID number, address, telephone number, face, license plate number, movement posture, clothing, etc., and this application does not limit this. In some scenarios, user information may also be referred to as private information, which may refer to secrets that are not willing to be disclosed or known to others (people outside a certain range), and this secret is irrelevant to other people and social interests.

[0010] It should be noted that the present application uses one encoder to encode the base layer and the enhancement layer, wherein one encoder outputs one bit stream; wherein the BL bit stream and the EL bit stream can be interleaved into one bit stream, which can also be called the bit stream of the original image.

[0011] Exemplarily, for video data including n (n is a positive integer) frames of original images, the steps in the first aspect above can be performed for each frame of original image to obtain a bit stream of n frames of original images (hereinafter referred to as n-frame bit stream); wherein the first image and the second image generated based on the i-th (i is an integer between 1 and n) frame of original image can be referred to as the i-th frame first image and the i-th frame second image. In this way, a bit stream output by the encoder can include n-frame bit streams, and each frame of the bit stream includes an EL bit stream and a BL bit stream.

[0012] It should be noted that the enhancement layer bit stream is output after the base layer bit stream, that is, after the encoder outputs the enhancement layer bit stream, the enhancement layer bit stream is connected to the base layer bit stream. At this time, the enhancement layer bit stream and the base layer bit stream are interlaced to form one bit stream (it can be understood that the base layer bit stream and the enhancement layer bit stream are interlaced and arranged in the bit stream output by the encoder); it is not that the encoder performs additional operations (such as interlacing operations) to make the enhancement layer bit stream and the base layer bit stream interlaced to form one bit stream.

[0013] Based on the above, the present application can be regarded as a layered encoding of the original image, wherein, in the layered encoding process of the original image, the encoding of the base layer corresponds to the encoded image of the base layer as the first image, and the encoding of the enhancement layer corresponds to the encoded image of the enhancement layer as the second image.

[0014] For example, after the encoder obtains the bit stream of the original image, the enhancement layer bit stream in the bit stream of the original image can be encrypted to protect the user information in the original image during the transmission of the original image to the remote end, thereby ensuring the security of the user information in the original image. The encryption operation can be performed by the system layer of the encoder.

[0015] According to the first aspect, the second image also includes the image of the first area in the original image; or the second image does not include the image of the first area in the original image. In other words, the second image only needs to include the image of the second area in the original image.

[0016] Exemplarily, the second image also includes an image of the first area in the original image, that is, the second image is the original image.

[0017] According to the first aspect, or any implementation method of the first aspect above, the enhancement layer of the second image is encoded to obtain an enhancement layer bit stream, including: performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the second image to obtain a predicted image of the second image; encoding the residual between the second image and the predicted image of the second image to obtain an enhancement layer bit stream.

[0018] Exemplarily, the reference image for inter-frame prediction of the second image may be a reconstructed image of the second image of another frame. The reference image for intra-frame prediction of the second image may be a reconstructed block of the second image of the current frame. The reference image for inter-layer prediction of the second image may be a reconstructed image of the first image of the current frame.

[0019] According to the first aspect, or any implementation method of the first aspect above, inter-frame prediction, intra-frame prediction or inter-layer prediction is performed on the second image to obtain a predicted image of the second image, including: dividing the second image to obtain multiple coding units (CodingUnit, CU); for the first CU among the multiple CUs, judging whether the first CU belongs to the second area; when the first CU belongs to the second area, performing inter-frame prediction or intra-frame prediction on the first CU to determine the prediction block of the first CU; when the first CU does not belong to the second area, performing inter-layer prediction on the first CU to determine the prediction block of the first CU; wherein the predicted image of the second image includes the prediction blocks of the multiple CUs obtained by dividing the second image.

[0020] Among them, for the first CU belonging to the second area, the similarity with the reconstructed block in the second area in the reconstructed image of the second image of the other frame is higher, so a more accurate prediction block can be obtained, thereby improving the encoding performance. When the second image is the original image, for the first CU that does not belong to the second area, the similarity with the reconstructed block in the first area in the reconstructed image of the first image of the current frame is higher, so a more accurate prediction block can also be obtained, thereby improving the encoding performance.

[0021] According to the first aspect, or any implementation of the first aspect above, inter-layer prediction is performed on the first CU to determine the prediction block of the first CU, including: using the reconstructed block in the first reconstructed image that is in the same position as the first CU as the prediction block of the first CU; wherein the first reconstructed image is a reconstructed image of the first image. In this way, the MV of the first CU is (0,0), and the number of bits required to encode the MV of the first CU is small, which can reduce the bit rate overhead of the encoding of the enhancement layer, thereby reducing the overall bit rate overhead. In addition, the amount of calculation of the motion search on the encoding side can also be reduced, thereby reducing the encoding complexity.

[0022] According to the first aspect, or any implementation of the first aspect above, when the first CU does not belong to the second region, the residual between the prediction block of the first CU and the first CU is 0. The residual between the first CU and the prediction block of the first CU can be forcibly set to 0. In this way, the number of bits required to encode the residual between the first CU and the prediction block of the first CU is small, which can reduce the coding rate overhead of the enhancement layer, thereby reducing the overall coding rate overhead.

[0023] According to the first aspect, or any implementation of the first aspect above, inter-frame prediction, intra-frame prediction or inter-layer prediction is performed on the second image to obtain a predicted image of the second image, including: dividing the second image to obtain multiple CUs; ​​for the second CU in the multiple CUs, inter-frame prediction, intra-frame prediction or inter-layer prediction is performed respectively to determine the prediction block of the second CU; wherein the predicted image of the second image includes the prediction blocks of the multiple CUs obtained by dividing the second image. In this way, for each second CU, a more accurate prediction block can be determined, which can improve the encoding quality of the second CU.

[0024] According to the first aspect, or any implementation of the first aspect above, any one of the multiple CUs is located inside or outside the second region. That is, the boundary line of the second region does not split any CU; this can be used as a constraint condition for dividing the second image into multiple CUs, so that it is easy to determine whether the first CU belongs to the second region.

[0025] According to the first aspect, or any implementation of the first aspect above, the method also includes: generating a first reconstructed image; wherein the first reconstructed image is a reconstructed image of the first image, and the first reconstructed image is a reference image of the first image of other frames and / or a reference image of the second image; the first reconstructed image is an image after loop filtering, and the pixel value of the pixel point in the second area in the first reconstructed image is a preset value.

[0026] According to the first aspect, or any implementation of the first aspect above, the method further includes: generating a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the second image of other frames, the image of the second area in the second reconstructed image is an image that has been loop filtered, and the image of the first area in the second reconstructed image is an image that has not been loop filtered. In this way, it can be ensured that the image of the first area in the second reconstructed image is the same as the image of the first area in the first reconstructed image; in this case, after the decoding end decodes and obtains the second reconstructed image, the second reconstructed image can be directly used as the reconstructed image of the original image, without the need to fuse the first reconstructed image and the second reconstructed image, thereby reducing computational overhead.

[0027] According to the first aspect, or any implementation of the first aspect above, when the second image does not include the image of the first area in the original image, the first reconstructed image and the second reconstructed image are fused to obtain an updated second reconstructed image. At this time, the updated second reconstructed image is the reconstructed image of the original image; the updated second reconstructed image can be subsequently stored in the decoded image cache as a reference image for subsequent encoding of the second image of other frames.

[0028] According to the first aspect, or any implementation of the first aspect above, the method further includes: obtaining position indication information, the position indication information is used to indicate the position of the second area; adding the encoded position indication information to the base layer bit stream and / or the enhancement layer bit stream. In this way, the position indication information can be transmitted to the decoding end, so that the decoding end can constrain the loop filtering of the first area and the second area according to the position indication information parsed from the base layer bit stream and / or the enhancement layer bit stream, and fuse the first reconstructed image and the second reconstructed image.

[0029] In one possible manner, when the second area is a rectangle, the position indication information may include the coordinates of the upper left corner of the second area and the size of the second area. When the second area is a circle, the position indication information may include the coordinates of the center of the second area and the radius of the second area. It should be understood that when the second area is other shapes, the position indication information may include other information, and the present application does not limit this.

[0030] In one possible manner, the position indication information may be an image block identification map. Specifically, the original image may be divided into m (m is a positive integer) image blocks of a fixed size (which may be set as required, such as 2*2); then, a preset identifier is used for each image block to identify whether it is in the second area. For example, for an image block, when the image block is in the second area, the value of the preset identifier of the image block is set to the first identifier value; when the image block is not in the second area, the value of the preset identifier of the image block is set to the second identifier value; in this way, an image block identification map may be obtained; wherein the image block identification map may include m pixels, one pixel corresponds to an image block in the original image, and the pixel value of each pixel is the value of the preset identifier. Wherein, the first identification value is different from the second identification value, and may be set specifically as required, and the present application does not limit this. For example, the first identification value is "1" and the second identification value is "0"; at this time, among the m pixels contained in the image block identification map, the pixel values ​​of m1 pixels are "1" and the pixel values ​​of m2 pixels are "0". Wherein, m1+m2=m, m1 and m2 are integers greater than or equal to 0.

[0031] It should be understood that the image block identification map may also be referred to as an image block identification matrix; the image block identification matrix may include m elements, one element corresponds to an image block in the original image, and the value of each element is a preset identification value.

[0032] In a possible manner, the above-mentioned image block may be a CU.

[0033] In a possible manner, the above-mentioned image block includes only one pixel. In this case, the position indication information may be a pixel identification map.

[0034] It should be noted that the location indication information may also be expressed in other forms, and this application does not impose any limitation on this.

[0035] According to the first aspect, or any implementation of the first aspect, the base layer bit stream is located before the enhancement layer bit stream.

[0036] According to the first aspect, or any implementation method of the first aspect above, the first area in the original image is the area in the original image excluding the second area; the second image is generated by replacing the pixel values ​​of the pixel points in the first area in the original image; the first image is generated by replacing the pixel values ​​of the pixel points in the second area in the original image; wherein the replacement method includes at least one of the following: solid color filling, mosaic or Gaussian blur.

[0037] It should be understood that the replacement method may also be other methods, and this application does not limit this.

[0038] First, user information detection can be performed on the original image to determine the area where the user information is located. For ease of explanation, the area in the original image containing user information (that is, the area where the user information is located) is referred to as the second area, and the area in the original image that does not contain user information is referred to as the first area; it should be understood that the first area is the area in the original image other than the second area. The method for detecting user information on the original image can be, for example, an AI-based intelligent detection method, and this application does not limit this.

[0039] Next, a second image is generated. In one possible manner, the original image may be used as the second image; that is, the second image includes an image of the first area in the original image and an image of the second area in the original image. In one possible manner, the pixel values ​​of the pixels in the second area in the original image may be kept unchanged, and the pixel values ​​of the pixels in the first area in the original image may be replaced. For example, the replacement may be performed in at least one of the following ways: solid color filling, mosaic, or Gaussian blur, etc., which is not limited in this application.

[0040] For example, the pixel values ​​of the pixels in the first area of ​​the original image may be kept unchanged, and the pixel values ​​of the pixels in the second area of ​​the original image may be replaced, for example, the replacement may be performed in at least one of the following ways: solid color filling, mosaic or Gaussian blur, etc., which is not limited in this application.

[0041] It should be noted that the above steps of detecting user information, generating the first image and generating the second image can be performed by an encoder or by other modules, and this application does not impose any restrictions on this.

[0042] According to the first aspect, or any implementation of the first aspect above, the base layer bitstream and the enhancement layer bitstream include a reference frame identifier and a prediction mode. In this way, the reference frame identifier and the prediction mode are transmitted to the decoding end, so that the decoding end can determine the reference image and then perform prediction based on the reference image.

[0043] Exemplarily, the prediction mode may include but is not limited to an intra-frame prediction mode, an inter-frame prediction mode, an inter-layer prediction mode, and the like.

[0044] In a second aspect, an embodiment of the present application provides a decoding method, the method comprising: first, receiving a bit stream, the bit stream comprising a base layer bit stream and an enhancement layer bit stream; then, decoding the base layer of the base layer bit stream to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of the first image, the first image includes an image of a first area in the original image and does not include an image of a second area in the original image; thereafter, decoding the enhancement layer bit stream to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, the second image includes an image of the second area in the original image; wherein the first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

[0045] Exemplarily, decoding of the base layer and decoding of the enhancement layer may be performed by one decoder.

[0046] Exemplarily, after receiving the bitstream, the decoding end can, on the one hand, input the base layer bitstream in the bitstream into the decoder; on the other hand, decrypt the encrypted enhancement layer bitstream in the bitstream and input it into the decoder; the decoder decodes the base layer and the enhancement layer. The decryption operation can be performed by the system layer of the decoding end.

[0047] According to the second aspect, the second image further includes the image of the first area in the original image; or the second image does not include the image of the first area in the original image.

[0048] According to the second aspect, or any implementation of the second aspect above, the bitstream also includes a reference frame identifier and a prediction mode, and the enhancement layer bitstream is decoded to obtain a second reconstructed image, including: determining a reference image of the second image according to the reference frame identifier and the prediction mode; predicting according to the reference image of the second image to obtain a predicted image of the second image; adding the predicted image of the second image to the residual of the second image parsed from the enhancement layer bitstream to obtain a second reconstructed image.

[0049] According to the second aspect, or any implementation method of the second aspect above, the second image includes multiple coding units CU, the reference image of the second image includes reference images of multiple CUs, and prediction is performed based on the reference image of the second image to obtain a predicted image of the second image, including: for a first CU among the multiple CUs, predicting the first CU based on the reference image of the first CU to determine a prediction block of the first CU; wherein the predicted image of the second image includes prediction blocks of multiple CUs included in the second image.

[0050] According to the second aspect, or any implementation of the second aspect, the first reconstructed image is an image after loop filtering, and the pixel values ​​of the pixels in the second area of ​​the first reconstructed image are preset values.

[0051] According to the second aspect, or any implementation of the second aspect, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

[0052] According to the second aspect, or any implementation of the second aspect above, the method further includes: when the second image does not include the image of the first area in the original image, fusing the first reconstructed image and the second reconstructed image to obtain an updated second reconstructed image.

[0053] According to the second aspect, or any implementation of the second aspect above, the bitstream also includes encoded position indication information, and the position indication information is used to indicate the position of the second area; the method also includes: determining the second area in the first reconstructed image based on the position indication information.

[0054] According to the second aspect, or any implementation of the second aspect above, the bitstream also includes a reference frame identifier and / or encoded position indication information, the position indication information is used to indicate the position of the second area, and the method also includes: determining the second area in the second reconstructed image based on the reference frame identifier and / or the position indication information.

[0055] Exemplarily, when the reference image identified by the reference frame identifier of the first CU is the first reconstructed image, it can be determined that the first CU does not belong to the second region; when the reference image identified by the reference frame identifier of the first CU is not the first reconstructed image, it can be determined that the first CU belongs to the second region. Then, the second region in the first reconstructed image can be determined according to the position of the first CU belonging to the second region.

[0056] According to the second aspect, or any implementation of the second aspect above, the bitstream also includes encoded position indication information, and the position indication information is used to indicate the position of the second area; the method also includes: determining the first area in the first reconstructed image and the second area in the second reconstructed image according to the position indication information; fusing the first reconstructed image and the second reconstructed image to obtain an updated second reconstructed image, including: fusing the image of the first area in the first reconstructed image and the image of the second area in the second reconstructed image to obtain an updated second reconstructed image.

[0057] For example, the pixel values ​​of the pixels in the second region in the first reconstructed image may be replaced by the pixel values ​​of the pixels at the corresponding positions in the second region in the second reconstructed image. For another example, the pixel values ​​of the pixels in the first region in the second reconstructed image may be replaced by the pixel values ​​of the pixels at the corresponding positions in the first region in the first reconstructed image.

[0058] According to the second aspect, or any implementation of the second aspect, the method further includes: using the second reconstructed image as the reconstructed image of the original image.

[0059] Exemplarily, when the second image includes the image of the second area in the original image but does not include the image of the first area in the original image, the updated second reconstructed image may be used as the reconstructed image of the original image.

[0060] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0061] In a third aspect, an embodiment of the present application provides an encoding device, the device comprising:

[0062] an image acquisition module, configured to acquire a first image and a second image, wherein the first image includes an image of a first area in an original image and does not include an image of a second area in the original image, and the second image includes an image of the second area in the original image, the size of the first image and the size of the second image are the same as the size of the original image, and the image of the second area in the original image includes user information;

[0063] A base layer encoding module, used for encoding the first image at a base layer to obtain a base layer bit stream;

[0064] The enhancement layer encoding module is used to encode the enhancement layer of the second image to obtain an enhancement layer bit stream; wherein the base layer bit stream and the enhancement layer bit stream are interwoven into one bit stream.

[0065] It should be understood that the encoding device of the third aspect can be used to execute the encoding method in the first aspect or any possible implementation of the first aspect.

[0066] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0067] In a fourth aspect, an embodiment of the present application provides a decoding device, the device comprising:

[0068] A code stream receiving module, used for receiving a bit stream, wherein the bit stream includes a base layer bit stream and an enhancement layer bit stream;

[0069] A base layer decoding module, configured to decode a base layer bit stream to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of a first image, the first image includes an image of a first region in an original image and does not include an image of a second region in the original image;

[0070] An enhancement layer decoding module is used to decode the enhancement layer of the enhancement layer bit stream to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of a second image, and the second image includes an image of a second area in the original image; wherein the first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

[0071] It should be understood that the decoding device of the fourth aspect can be used to execute the decoding method in the second aspect or any possible implementation of the second aspect.

[0072] The fourth aspect and any implementation of the fourth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0073] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0074] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0075] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, and when the program instructions are executed by the processor, the electronic device executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0076] The sixth aspect and any implementation of the sixth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0077] In the seventh aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the steps in the first aspect or any possible implementation of the first aspect are executed.

[0078] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0079] In an eighth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps in the second aspect or any possible implementation method of the second aspect are executed.

[0080] The eighth aspect and any implementation of the eighth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0081] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0082] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0083] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0084] The tenth aspect and any implementation of the tenth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0085] In the eleventh aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0086] The eleventh aspect and any implementation of the eleventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0087] In the twelfth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0088] The twelfth aspect and any implementation of the twelfth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1A is a schematic diagram of an application scenario shown as an example;

[0090] Figure 1B is a schematic diagram of a compression framework shown as an example;

[0091] Figure 2A is a schematic diagram of a compression framework shown as an example;

[0092] Figure 2B is a schematic diagram of an original image, a first image, and a second image shown exemplarily;

[0093] Figure 3A is a schematic diagram of an exemplary encoding process;

[0094] Figure 3B is a schematic diagram of a bit stream structure shown as an example;

[0095] Figure 4A is a schematic diagram of an exemplary decoding process;

[0096] Figure 4B is a schematic diagram of a first reconstructed image and a second reconstructed image shown exemplarily;

[0097] Figure 5A is a schematic diagram of an exemplary encoding process;

[0098] Figure 5B is a schematic diagram of an exemplary encoding process;

[0099] Figure 5C is a schematic diagram of prediction reference shown as an example;

[0100] Fig. 6A is a schematic diagram of an exemplary decoding process;

[0101] Figure 6B is a schematic diagram of an exemplary decoding process;

[0102] Fig. 7A is a schematic diagram of an exemplary encoding process;

[0103] Figure 7B is a schematic diagram of prediction reference shown as an example;

[0104] Figure 8is a schematic diagram of an exemplary decoding process;

[0105] Fig.9A is a schematic diagram of a compression process shown as an example;

[0106] Fig. 9B is a schematic diagram of a compression process shown as an example;

[0107] Fig. 9C is a schematic diagram of a compression process shown as an example;

[0108] Fig.10 A schematic diagram of an encoding device shown as an example;

[0109] Fig.11 A schematic diagram of a decoding device is shown as an example;

[0110] Fig.12 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0111] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0112] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0113] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0114] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0115] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0116] Figure 1A Schematic diagram of an exemplary application scenario. Figure 1A The application scenario shown is a video surveillance scenario; it should be understood that the present application can also be applied to any scenario that requires video (or image) privacy protection, and the present application does not limit this.

[0117] Reference Figure 1A , for example, image acquisition devices can be deployed at intersections, shopping mall entrances and exits, school gates and other locations, and the image acquisition devices can collect video data, and then transmit the collected video data to at least one terminal device through the network. For example, the video data collected by the image acquisition device deployed at the intersection can be transmitted to the smart screen, the video data collected by the image acquisition device deployed at the shopping mall entrance and exit can be transmitted to the smart screen and tablet computer, the video data collected by the image acquisition device deployed at the school gate can be transmitted to the smart screen and personal computer, and so on. After the terminal device receives the video data sent by the image acquisition device, it can store and display the video data (or the video data processed as required) to facilitate the user to detect the monitored area.

[0118] It should be noted that in this application, the actions of the image acquisition device to collect video data, transmit video data, and the terminal device to store, process and display video data are all carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0119] Exemplarily, the image acquisition device can compress (or encode) the acquired video data to obtain a bit stream (Bit stream; where the bit stream can also be called a code stream or bit stream), and then transmit the bit stream to the terminal device to reduce the amount of transmitted data and reduce bandwidth requirements.

[0120] Figure 1B A schematic diagram of a compression framework is shown as an example.

[0121] Reference Figure 1B , exemplarily, the image acquisition device may include a camera, an encoding module (or encoder) and a sending module. , exemplarily, the encoding module may be a software module or a hardware module, which is not limited in the present application embodiment. It should be understood that, Figure 1B This is only an example of an image acquisition device. In other embodiments of the present application, the image acquisition device has a Figure 1BMore modules are shown, and the embodiments of the present application are not limited to this.

[0122] Reference Figure 1B , exemplarily, the terminal device may include a display module, a decoding module (or decoder) and a receiving module. Exemplarily, the decoding module may be a software module or a hardware module, which is not limited in the present application embodiment. It should be understood that, Figure 1B This is only an example of a terminal device. The terminal devices in other embodiments of the present application have Figure 1B More modules are shown, and the embodiments of the present application are not limited to this.

[0123] Continue to refer to Figure 1B , exemplarily, the process of an image acquisition device sending the raw video data collected by a camera to a terminal device for display is as follows: the camera can output the collected raw video data to the encoding module; then, the encoding module can encode the raw video data to obtain a bit stream, and output the encoded bit stream to the sending module; then, the sending module can send the bit stream to the terminal device. Subsequently, the receiving module of the terminal device can receive the bit stream; then, the bit stream is output to the decoding module; then, the decoding module can decode the bit stream to obtain reconstructed video data and output the reconstructed video data to the display module, and the display module displays the reconstructed video data.

[0124] For example, the following Figure 1B The image acquisition device is called the encoding end, and can Figure 1B The terminal device is called a decoding end; the encoding process of the encoding end and the decoding process of the decoding end can refer to the description of the subsequent embodiments.

[0125] Figure 2A FIG. 1 is a schematic diagram of a compression framework shown as an example. Figure 2A The dashed lines in the figure are optional operations.

[0126] Reference Figure 2A , exemplarily, before encoding at the encoding end, the original image may be preprocessed to obtain a first image and a second image.

[0127] Exemplarily, the original image may refer to an image that is originally expected to be transmitted to the remote end. The original image may be generated in a variety of ways, for example, the original image may be an image captured by an image acquisition device, an image received from other terminal devices, an image generated by AI (Artificial Intelligence), an image processed by an image processing tool, etc., and this application does not limit this.

[0128] Exemplarily, the original image may include user information; wherein user information may refer to information that directly or indirectly describes the identity of the user. For example, user information includes, but is not limited to: user name, date of birth, ID number, address, telephone number, face, license plate number, sports posture, clothing, etc., which is not limited in this application. In some scenarios, user information may also be referred to as private information, which may refer to secrets that are not willing to be disclosed or known to others (people outside a certain range), and this secret is irrelevant to other people and social interests.

[0129] Specifically, the process of preprocessing the original image to obtain the first image and the second image may be as follows:

[0130] First, user information detection can be performed on the original image to determine the area where the user information is located. For ease of explanation, the area containing user information in the original image (that is, the area where the user information is located) is called the second area, and the area not containing user information in the original image is called the first area; it should be understood that the first area in the original image is the area in the original image other than the second area. The method for detecting user information on the original image can be, for example, an intelligent detection method based on AI, and this application does not limit this.

[0131] For example, the original image can be Figure 2B As shown in (1), the second region can be Figure 2B As shown in the dotted box in (1), the first area is Figure 2B (1) except the second area in the original image.

[0132] Next, generate a second image. In one possible way, the original image can be used as the second image; that is, the second image includes an image of the first area in the original image and an image of the second area in the original image. In one possible way, the pixel values ​​of the pixels in the second area in the original image can be kept unchanged, and the pixel values ​​of the pixels in the first area in the original image can be replaced. For example, the replacement can be performed in at least one of the following ways: solid color filling, mosaic, or Gaussian blur, etc., which is not limited in this application. For example, if the replacement method is solid color filling, the second image can be as follows: Figure 2B (2) as shown; Figure 2B In (2), the pixel values ​​of the pixels in the second area of ​​the original image remain unchanged, and the pixel values ​​of the pixels in the first area of ​​the original image are filled with white (it should be understood that other colors can also be used for filling, and this application does not limit this).

[0133] And, generate a first image. Exemplarily, the pixel values ​​of the pixels in the first area of ​​the original image can be kept unchanged, and the pixel values ​​of the pixels in the second area of ​​the original image can be replaced. For example, the replacement can be performed in at least one of the following ways: solid color filling, mosaic or Gaussian blur, etc., which is not limited in this application. For example, if the replacement method is solid color filling, the first image can be as follows Figure 2B (3) as shown; Figure 2B In (3), the pixel values ​​of the pixels in the first area of ​​the original image remain unchanged, and the pixel values ​​of the pixels in the second area of ​​the original image are filled with white (it should be understood that other colors can also be used for filling, and this application does not limit this).

[0134] That is, the first image includes the image of the first area in the original image and does not include the image of the second area in the original image, the second image includes the image of the second area in the original image, and the size of the first image and the size of the second image are the same as the size of the original image.

[0135] It should be noted that the preprocessing operation can be performed by the encoder or by other modules, and this application does not limit this. This application takes the example of preprocessing being performed by other modules for explanation.

[0136] Afterwards, the first image and the second image can be input into the same encoder for encoding, as shown below: Figure 3A Description in the examples.

[0137] Figure 3A FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 3A The steps in the embodiment are performed by an encoder.

[0138] S301, acquiring a first image and a second image.

[0139] S302, encode the base layer of the first image to obtain a base layer bit stream.

[0140] S303, encode the enhancement layer of the second image to obtain an enhancement layer bit stream.

[0141] Exemplarily, first, the first image can be input as a base layer (BaseLayer, BL) image to the encoder, and the encoder encodes the first image at the base layer to obtain a base layer bit stream (hereinafter referred to as BL bit stream). Then, the second image can be input as an enhancement layer (EnhanceLayer, EL) image to the encoder, and the encoder encodes the second image at the enhancement layer to obtain an enhancement layer bit stream (hereinafter referred to as EL bit stream). That is, the encoder first executes S302 and then executes S303.

[0142] It should be noted that the first image and the second image are encoded by the same encoder, wherein one encoder outputs one bit stream.

[0143] In this way, after the encoder completes encoding of the first image and the second image, the BL bit stream and the EL bit stream can be interleaved into one bit stream (this bit stream can also be called the bit stream of the original image). The one bit stream can include the BL bit stream and the EL bit stream, and the BL bit stream is located before the EL bit stream.

[0144] Exemplarily, for video data including n (n is a positive integer) frames of original images, the above steps S301 to S303 can be performed for each frame of original image to obtain a bit stream of n frames of original images (hereinafter referred to as n-frame bit stream); wherein the first image and the second image generated based on the i-th (i is an integer between 1 and n) frame of original image can be referred to as the i-th frame first image and the i-th frame second image. In this way, a bit stream output by the encoder can include n-frame bit streams, each frame of the bit stream includes an EL bit stream and a BL bit stream, and the BL bit stream is located before the EL bit stream; Figure 3B shown.

[0145] Figure 3B FIG. 1 is a schematic diagram of a bit stream structure shown as an example. Figure 3B In the embodiment, a bit stream output by the encoder includes n frame bit streams: the first frame bit stream (i.e., the bit stream of the first frame original image) includes the BL_1 bit stream and the EL_1 bit stream, and the BL_1 bit stream is before the EL_1 bit stream; the second frame bit stream (i.e., the bit stream of the second frame original image) includes the BL_2 bit stream and the EL_2 bit stream, and the BL_2 bit stream is before the EL_2 bit stream; ...; the n frame bit stream (i.e., the bit stream of the n frame original image) includes the BL_n bit stream and the EL_n bit stream, and the BL_n bit stream is before the EL_n bit stream.

[0146] Exemplarily, the header of the BL bitstream output by the encoder includes a frame identifier (such as a picture order count (POC)) and a layer identifier (LayerID); and the header of the EL bitstream includes a frame identifier and a layer identifier. The frame identifier is used to identify a frame bitstream, and the layer identifier is used to identify the BL bitstream and the EL bitstream in each frame bitstream. It should be understood that the value of the frame identifier in each frame bitstream is different, and the value of the layer identifier of the BL bitstream and the EL bitstream in each frame bitstream is different.

[0147] Exemplarily, for the first image and the second image generated based on the same frame of original image, the values ​​of the frame identifiers in the BL bitstream and the EL bitstream are the same; therefore, the header of the EL bitstream may not include the frame identifier, and this application does not impose any limitation on this.

[0148] It should be noted that the EL bit stream is output after the BL bit stream, that is, after the encoder outputs the EL bit stream, the EL bit stream is connected to the BL bit stream. At this time, the EL bit stream and the BL bit stream are interleaved to form a bit stream (it can be understood that the BL bit stream and the EL bit stream are interleaved in the bit stream output by the encoder); it is not that the encoder performs additional operations (such as interleaving operations) to make the EL bit stream and the BL bit stream interleave to form a bit stream.

[0149] Based on the above, the present application can be regarded as a layered encoding of the original image, wherein, in the layered encoding process of the original image, the encoding of the base layer corresponds to the encoded image of the base layer as the first image, and the encoding of the enhancement layer corresponds to the encoded image of the enhancement layer as the second image.

[0150] Since the reference queue for encoding the enhancement layer includes the reconstructed image of the base layer (that is, the reconstructed image of the first image (hereinafter referred to as the first reconstructed image)), that is to say, the reference queue of the second image includes the first reconstructed image; compared with the prior art in which the reference queue of the second image (that is, the privacy image) only includes the reconstructed block of the second image of the current frame and the second reconstructed images of other frames (that is, the reconstructed images of the second images of other frames), the present application encodes more reference images for the second image, so that the present application can obtain more accurate prediction blocks, and thus can improve the encoding performance of the enhancement layer (improve encoding quality or reduce bit rate overhead), thereby improving the overall encoding performance. In addition, in the present application, the enhancement layer bit stream and the base layer bit stream are interwoven into one bit stream, which facilitates the frame synchronization of the enhancement layer bit stream and the base layer bit stream, and can reduce computational overhead.

[0151] Exemplarily, after the encoding end obtains the bit stream of the original image, the EL bit stream in the bit stream of the original image can be encrypted to protect the user information in the original image during the process of transmitting the original image to the remote end, thereby ensuring the security of the user information in the original image. The encryption operation can be performed by the encoding end system layer.

[0152] Exemplarily, after receiving the bit stream, the decoding end may input the BL bit stream in the bit stream into the decoder; on the other hand, the encrypted EL bit stream in the bit stream may be decrypted and input into the decoder; wherein the decryption operation may be performed by the system layer of the decoding end. Next, referring to Figure 2A , the decoder decodes the BL bit stream and the EL bit stream to obtain a reconstructed image of the first image (hereinafter referred to as the first reconstructed image) and a reconstructed image of the second image (hereinafter referred to as the second reconstructed image). After that, the post-processing module performs post-processing based on the first reconstructed image and the second reconstructed image to obtain a reconstructed image of the original image. The decoding process of the decoder can be as follows Figure 4A As shown in the embodiment.

[0153] Figure 4A FIG. 4 is a schematic diagram showing an exemplary decoding process.

[0154] S401, receiving a bit stream, where the bit stream includes a base layer bit stream and an enhancement layer bit stream.

[0155] Exemplarily, after receiving the bit stream, the decoder can read the frame identifier from the bit stream and determine a frame bit stream according to the frame identifier. Then, the layer identifier can be read from the frame bit stream to determine the BL bit stream and the EL bit stream.

[0156] S402, decoding the base layer bit stream to obtain a first reconstructed image.

[0157] Next, the decoder can decode the base layer bitstream to obtain the first reconstructed image. Figure 2B (3) As shown in FIG. Figure 4B (1) as shown.

[0158] S403, decoding the enhancement layer bit stream to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, and the second image includes an image of the second region in the original image.

[0159] Afterwards, the decoder can decode the enhancement layer bitstream to obtain a second reconstructed image. For example, when the second image is the original image (such as Figure 2B (1)), the second reconstructed image can be Figure 4B (2) As shown in FIG. 2 . When the second image includes the image of the second region in the original image and does not include the image of the first region in the original image (such as Figure 2B (2)) When the second reconstructed image is Figure 4B (3) as shown.

[0160] It should be noted that when the encoder performs lossless encoding on the first image and the second image, Figure 4B The first reconstructed image in (1) is Figure 2B The first image in (3) is the same, Figure 4B The second reconstructed image in (2) is Figure 2B The original image in (1) is the same as Figure 4B The second reconstructed image in (3) is Figure 2B (2) is the same as the second image. When the encoder performs lossy encoding on the first image and the second image, Figure 4B The first reconstructed image in (1) is Figure 2B There is a difference in the first image in (3), Figure 4B The second reconstructed image in (2) is Figure 2B There are differences in the original image in (1). Figure 4B The second reconstructed image in (3) is Figure 2B There is a difference in the second image in (2).

[0161] It should be understood that Figure 4B It is only a schematic diagram of the first reconstructed image and the second reconstructed image, and does not mean that the encoding and decoding method of the present application is used for Figure 2B The real first reconstructed image and the second reconstructed image are obtained after encoding and decoding the first image and the second image.

[0162] Refer again Figure 2A , exemplarily, when the second image includes an image of the second area in the original image and does not include an image of the first area in the original image, the first area in the first reconstructed image and the second area in the second reconstructed image can be determined; then, the image of the first area in the first reconstructed image and the image of the second area in the second reconstructed image are fused to obtain a reconstructed image of the original image. For example, the pixel value of a pixel point in the second area in the first reconstructed image can be replaced with the pixel value of a pixel point at a corresponding position in the second area in the second reconstructed image. For another example, the pixel value of a pixel point in the first area in the second reconstructed image can be replaced with the pixel value of a pixel point at a corresponding position in the first area in the first reconstructed image.

[0163] Exemplarily, when the second image is an original image, the second reconstructed image may be used as a reconstructed image of the original image.

[0164] The encoding process of the base layer and the encoding process of the enhancement layer are described below.

[0165] Figure 5A FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 5A In the process of enhancement layer encoding, inter-frame prediction and / or intra-frame prediction are performed on the coding unit (CodingUnit, CU) belonging to the second area in the second image, and inter-layer prediction is performed on the CU that does not belong to the second area (that is, belongs to the first area) in the second image.

[0166] S501, acquiring a first image and a second image; wherein the first image includes an image of a first area in the original image and does not include an image of a second area in the original image, the second image includes an image of the second area in the original image, a size of the first image and a size of the second image are the same as a size of the original image, and the image of the second area in the original image contains user information.

[0167] S502: Acquire location indication information, where the location indication information can be used to indicate the location of the second area.

[0168] In one possible manner, when the second area is a rectangle, the position indication information may include the coordinates of the upper left corner of the second area and the size of the second area. When the second area is a circle, the position indication information may include the coordinates of the center of the second area and the radius of the second area. It should be understood that when the second area is other shapes, the position indication information may include other information, and the present application does not limit this.

[0169] In one possible way, the position indication information may be an image block identification map. Specifically, the original image may be divided into m (m is a positive integer) image blocks of a fixed size (which may be set as required, such as 2*2); then, for each image block, a preset identification is used to identify whether it is in the second area; for example, for an image block, when the image block is in the second area, the value of the preset identification of the image block is set to the first identification value; when the image block is not in the second area, the value of the preset identification of the image block is set to the second identification value; in this way, an image block identification map may be obtained; wherein the image block identification map may include m pixels, one pixel corresponds to one image block, and the pixel value of each pixel is the value of the preset identification. Wherein, the first identification value is different from the second identification value, and may be set specifically as required, and the present application does not limit this. For example, the first identification value is "1" and the second identification value is "0"; at this time, among the m pixels contained in the image block identification map, the pixel values ​​of m1 pixels are "1" and the pixel values ​​of m2 pixels are "0". Wherein, m1+m2=m, m1 and m2 are integers greater than or equal to 0.

[0170] It should be understood that the image block identification map may also be referred to as an image block identification matrix; the image block identification matrix may include m elements, one element corresponds to one image block, and the value of each element is a preset identification value.

[0171] In a possible manner, the above-mentioned image block may be a CU.

[0172] In a possible manner, the above-mentioned image block includes only one pixel. In this case, the position indication information may be a pixel identification map.

[0173] It should be noted that the location indication information may also be expressed in other forms, and this application does not impose any limitation on this.

[0174] S503, performing inter-frame coding or intra-frame coding on the first image to obtain a BL bit stream.

[0175] Figure 5B The figure is a schematic diagram of an exemplary encoding process.

[0176] Reference Figure 5B (1), illustratively, S503 may include the following steps:

[0177] S5031, divide the first image into multiple CUs.

[0178] The following is described by taking one CU (hereinafter referred to as a third CU) among the multiple CUs obtained by dividing the first image as an example, and reference may be made to S5032 to S5033:

[0179] S5032: Perform inter-frame prediction or intra-frame prediction on a third CU among the multiple CUs obtained by dividing the first image, and determine a prediction block of the third CU.

[0180] S5033, encoding a residual between a third CU and a prediction block of the third CU to obtain a bit stream of the third CU; wherein the bit streams of the plurality of CUs obtained by dividing the first image constitute a BL bit stream.

[0181] Exemplarily, it is possible to determine whether to perform inter-frame prediction or intra-frame prediction on the third CU based on pre-configured coding parameters. When it is determined that intra-frame prediction is performed on the third CU, a search can be performed from the reconstructed block of the first image of the current frame to determine the prediction block of the third CU; then, the residual between the third CU and the prediction block of the third CU is determined. Afterwards, the residual is transformed, quantized, and entropy encoded to obtain the bit stream of the third CU. Exemplarily, the residual is transformed to obtain a transform coefficient, which maps the spatial domain residual to the frequency domain coefficient to obtain the transform coefficient. The transform coefficient can then be quantized to obtain the residual coefficient; and then the size of the transform coefficient can be further reduced. Then, the residual coefficient is entropy encoded to obtain the bit stream of the third CU.

[0182] Exemplarily, when it is determined that the third CU is to be inter-predicted, motion search and motion estimation can be performed from the first reconstructed image of other frames (i.e., the reconstructed image of the first image of other frames) to determine the motion vector (MV) of the third CU and the prediction block of the third CU; then, the residual between the third CU and the prediction block of the third CU is determined. After that, the residual is transformed, quantized, and entropy encoded, and the MV of the third CU is entropy encoded to obtain the bit stream of the third CU.

[0183] Exemplarily, the third CU may be inter-predicted and intra-predicted respectively to determine two candidate prediction blocks of the third CU: candidate prediction block 1 and candidate prediction block 2, wherein candidate prediction block 1 is determined by inter-prediction and candidate prediction block 2 is determined by intra-prediction. Then, the prediction block of the third CU may be determined based on the two candidate prediction blocks of the third CU. In one possible manner, the best candidate prediction block is selected from candidate prediction block 1 and candidate prediction block 2 as the prediction block of the third CU. For example, based on candidate prediction block 1 and candidate prediction block 2, the rate-distortion loss of the first image is predicted respectively (it may also be other types of losses, which are not limited in this application), and the candidate prediction block with the smallest rate-distortion loss is selected as the best candidate prediction block. Afterwards, the residual between the third CU and the prediction block of the third CU is determined. Then, the residual is transformed, quantized, and entropy encoded, and the MV of the third CU is entropy encoded to obtain the bit stream of the third CU.

[0184] It should be noted that the operations performed in S5033 for encoding the residual between the third CU and the prediction block of the third CU include but are not limited to: transformation, quantization, entropy coding, etc. For example, the residual between the third CU and the prediction block of the third CU may be transformed and quantized to obtain the residual coefficient of the third CU; then, the residual coefficient of the third CU may be entropy coded to obtain the bit stream of the third CU.

[0185] Exemplarily, a reconstruction operation may be performed during the encoding of the first image to generate a first reconstructed image, wherein the first reconstructed image may be used as a reference image of the first image of other frames and / or a reference image of the second image. The first reconstructed image may also be referred to as a reconstructed image of the base layer.

[0186] Reference Figure 5B (1), exemplarily, the residual coefficients of the third CU may be inversely quantized and inversely transformed to obtain the residual of the third CU (wherein the residual obtained by inversely quantizing and inversely transforming is different from the residual before transforming and quantizing, and the present application does not distinguish between the two in terms of name); then, the residual of the third CU is added to the prediction block of the third CU to obtain the reconstructed block of the third CU. Thereafter, the reconstructed block of the third CU is loop filtered to obtain the reconstructed block of the third CU after loop filtering. In this way, the reconstructed blocks of all CUs obtained by dividing the first image after loop filtering can constitute the first reconstructed image. It should be noted that the reconstructed blocks of all CUs obtained by dividing the first image are loop filtered, that is, the image of the first area and the image of the second image in the first reconstructed image are loop filtered. Optionally, after obtaining the first reconstructed image, the pixel value of the pixel point in the second area of ​​the first reconstructed image can be set to a preset value, such as the median value of the dynamic range (128).

[0187] Specifically, after obtaining the first reconstructed image, the pixel points in the second area of ​​the first reconstructed image may be determined according to the position indication information obtained above, and then the pixel values ​​of the pixel points in the second area of ​​the first reconstructed image may be set to preset values.

[0188] S504, perform inter-frame coding, intra-frame coding or inter-layer coding on the second image to obtain an EL bit stream.

[0189] Reference Figure 5B (2), illustratively, S504 may include the following steps:

[0190] S5041, dividing the second image into multiple coding units CU.

[0191] It should be noted that the constraint condition for dividing the second image into multiple CUs in S5041 is that any CU cannot be divided by the boundary line of the second region; that is, any CU among the multiple CUs obtained by dividing the second image is either located in the second region or outside the second region.

[0192] The following is described by taking one CU (hereinafter referred to as the first CU) among the multiple CUs obtained by dividing the second image as an example, and reference may be made to S5042 to S5045:

[0193] S5042: For a first CU among the multiple CUs obtained by dividing the second image, determine whether the first CU belongs to the second region.

[0194] Exemplarily, for the first CU, it can be determined whether the first CU belongs to the second region. Specifically, the encoder can determine whether the first CU belongs to the second region according to the position information of the first CU and the position indication information obtained above.

[0195] When the first CU belongs to the second region, S5043 may be executed; when the first CU does not belong to the second region, S5044 may be executed.

[0196] S5043 , perform inter-frame prediction or intra-frame prediction on the first CU to determine a prediction block of the first CU.

[0197] Exemplarily, S5043 may refer to the description of the process of determining the prediction block of the third CU in the above S5032, which will not be repeated here.

[0198] It should be noted that in S5043, the reference image for inter-frame prediction of the first CU is the second reconstructed image of other frames (that is, the reconstructed image of the second image of other frames); the reference block for intra-frame prediction of the first CU is the reconstructed block of the second image of the current frame.

[0199] S5044: Perform inter-layer prediction on the first CU to determine a prediction block of the first CU.

[0200] Exemplarily, the reconstructed image of the base layer, that is, the first reconstructed image, can be used as a reference image for inter-layer prediction to determine the prediction block of the first CU. Specifically, for the first CU, the reconstructed block at the same position as the first CU in the first reconstructed image can be used as the prediction block of the first CU. In this way, the MV of the first CU is (0,0), and the number of bits required to encode the MV of the first CU is relatively small, which can reduce the coding rate overhead of the enhancement layer, thereby reducing the overall coding rate overhead. In addition, the amount of calculation of the motion search on the encoding side can also be reduced, thereby reducing the coding complexity.

[0201] Figure 5C A prediction reference diagram is shown as an example.

[0202] Reference Figure 5C For example, assume that time k (k is a positive integer) is the current time, corresponding to the first image of the current frame and the second image of the current frame; time k-1 is the previous time, corresponding to the first reconstructed image of the previous frame and the second reconstructed image of the previous frame.

[0203] Reference Figure 5C For example, inter-frame prediction can be performed for the third CU in the first area of ​​the first image of the current frame, as shown in R01 and R02; wherein R01 indicates that the inter-frame prediction refers to the reconstructed block in the first area of ​​the first reconstructed image of the previous frame, and R02 indicates that the inter-frame prediction refers to the reconstructed block in the second area of ​​the first reconstructed image of the previous frame.

[0204] Reference Figure 5C For example, intra-frame prediction can be performed for the third CU in the first area of ​​the first image of the current frame, as shown in R03 and R04; wherein R03 indicates that the intra-frame prediction refers to the reconstructed block in the first area of ​​the first image of the current frame, and R04 indicates that the intra-frame prediction refers to the reconstructed block in the second area of ​​the first image of the current frame.

[0205] Reference Figure 5C For example, inter-frame prediction can be performed for the third CU in the second area in the first image of the current frame, as shown in R05 and R06; wherein R05 indicates that the inter-frame prediction refers to the reconstructed block in the first area in the first reconstructed image of the previous frame, and R06 indicates that the inter-frame prediction refers to the reconstructed block in the second area in the first reconstructed image of the previous frame.

[0206] Reference Figure 5CFor example, intra-frame prediction can be performed for the third CU in the second area in the first image of the current frame, as shown in R07 and R08; wherein R07 indicates that the intra-frame prediction refers to the reconstructed block in the first area in the first image of the current frame, and R08 indicates that the intra-frame prediction refers to the reconstructed block in the second area in the first image of the current frame.

[0207] Reference Figure 5C For example, inter-layer prediction can be performed for the first CU in the first area of ​​the second image of the current frame, as shown in R09; wherein R09 indicates that the inter-layer prediction can refer to the reconstructed block of the first area in the first reconstructed image of the current frame (i.e., the reconstructed image of the base layer).

[0208] Reference Figure 5C , exemplarily, inter-frame prediction can be performed for the first CU in the second area in the second image of the current frame, as shown in R10 and R11; wherein R10 represents the inter-frame prediction referring to the reconstructed block in the first area in the second reconstructed image of the previous frame (the reconstructed image of the enhancement layer of the previous frame), and R11 represents the inter-frame prediction referring to the reconstructed block in the second area in the second reconstructed image of the previous frame.

[0209] Reference Figure 5C For example, intra-frame prediction can be performed for the first CU in the second area in the second image of the current frame, as shown in R12 and R13; wherein R12 indicates that the intra-frame prediction refers to the reconstructed block in the first area in the second image of the current frame, and R13 indicates that the intra-frame prediction refers to the reconstructed block in the second area in the second image of the current frame.

[0210] S5045, encoding a residual between the first CU and the prediction block of the first CU to obtain a bit stream of the first CU; wherein the bit streams of the multiple CUs obtained by dividing the second image constitute an EL bit stream.

[0211] Exemplarily, when the first CU does not belong to the second region (i.e., the first CU belongs to the first region), the residual between the first CU and the prediction block of the first CU can be forcibly set to 0. In this way, the number of bits required to encode the residual between the first CU and the prediction block of the first CU is small, which can reduce the encoding bit rate overhead of the enhancement layer, thereby reducing the overall bit rate overhead.

[0212] It should be noted that the operations performed in S5045 for encoding the residual between the first CU and the prediction block of the first CU include but are not limited to: transformation, quantization, entropy coding, etc. For example, the residual between the first CU and the prediction block of the first CU may be transformed and quantized to obtain the residual coefficient of the first CU; then, the residual coefficient of the first CU may be entropy coded to obtain the bit stream of the first CU.

[0213] Exemplarily, a reconstruction operation may be performed during the encoding of the second image to obtain a second reconstructed image as a reference image for subsequent encoding of other second images of the frame. The second reconstructed image may also be referred to as a reconstructed image of the enhancement layer.

[0214] Reference Figure 5B (2) Exemplarily, when the first CU belongs to the second region, the residual coefficients of the first CU may be inversely quantized and inversely transformed to obtain the residual of the first CU; then, the residual of the first CU is added to the prediction block of the first CU to obtain the reconstructed block of the first CU. After that, the reconstructed block of the first CU may be loop-filtered to obtain the reconstructed block of the first CU after loop filtering. When the first CU does not belong to the second region, the prediction block of the first CU may be used as the reconstructed block of the first CU, and there is no need to loop-filter the reconstructed block of the first CU. In other words, the reconstructed blocks of the CUs located in the second region among all the CUs obtained by dividing the second image are loop-filtered, and the CUs located in the first region are not loop-filtered, that is, the image of the first region in the second reconstructed image of the current frame is the same as the image of the first region in the first reconstructed image of the current frame.

[0215] Specifically, whether the first CU belongs to the second area can be determined according to the position indication information obtained above. For details, please refer to the description of S5042 above, which will not be repeated here.

[0216] S505, adding the encoded position indication information to the BL bit stream and / or the BL bit stream.

[0217] It should be noted that S505 is an optional step.

[0218] Exemplarily, the encoded position indication information may be added to the front end or the end of the BL bit stream, and / or the encoded position indication information may be added to the end of the EL bit stream.

[0219] Exemplarily, after the encoded position indication information is added to the BL bitstream and / or the BL bitstream, the decoding end can constrain the loop filtering of the first area and the second area according to the position indication information parsed from the BL bitstream and / or the BL bitstream, and fuse the first reconstructed image and the second reconstructed image.

[0220] When the encoded position indication information is not added to the BL bitstream and / or the BL bitstream, the decoding end may derive the position information of the second region in other ways, which will be described in detail later.

[0221] It should be noted that the encoding end can also add encoding control parameters (such as prediction mode (which may include but is not limited to inter-frame prediction mode, intra-frame prediction mode and inter-layer prediction mode), division information (such as CU division method), transformation parameters and quantization parameters, etc.) to the bit stream of the original image, so that the decoding end can use the corresponding control parameters for decoding control and decode the reconstructed image of the original image.

[0222] Fig. 6A FIG. 4 is a schematic diagram showing an exemplary decoding process. Fig. 6A The decoding process and Figure 5A The encoding process corresponds to .

[0223] S601, receiving a bit stream, where the bit stream includes a base layer bit stream, an enhancement layer bit stream, and position indication information.

[0224] Exemplarily, after receiving the bit stream, the decoder may parse the bit stream, determine each frame bit stream according to the frame identifier and layer identifier in the bit stream, and determine the BL bit stream and EL bit stream in each frame bit stream.

[0225] For example, when Figure 5A In the embodiment, when the encoding end executes S505, that is, adds the encoded position indication information to the BL bit stream and / or the BL bit stream, the decoding end may also parse the position indication information from the received bit stream.

[0226] Exemplarily, the decoding end may also parse out control parameters (such as prediction mode (inter-frame prediction or intra-frame prediction), partition information (such as CU partitioning method), transform parameters and quantization parameters, etc.) from the received bit stream.

[0227] The following describes decoding of the BL bit stream and the EL bit stream in a frame bit stream.

[0228] S602: Decode the base layer bit stream to obtain a first reconstructed image.

[0229] Figure 6B FIG. 4 is a schematic diagram of an exemplary decoding process.

[0230] Exemplarily, the decoder may determine the division mode of the first image according to the division information parsed from the bitstream, and further determine the multiple CUs included in the first image; the decoding process of one CU (which may be referred to as a third CU) among the multiple CUs included in the first image may refer to S6021 to S6023:

[0231] S6021: Determine a reference image of a third CU according to a reference frame identifier and a prediction mode parsed from the BL bitstream.

[0232] S6022: Predict the third CU according to the reference image of the third CU to determine a prediction block of the third CU.

[0233] Exemplarily, when the decoder determines that the preset mode of the third CU is the inter-frame prediction mode according to the prediction mode parsed from the BL bitstream, the reference image of the third CU can be determined from the first reconstructed image of other frames according to the reference frame identifier parsed from the BL bitstream; at this time, the decoder can also parse the MV of the third CU from the BL bitstream. Then, according to the MV of the third CU, the prediction block of the third CU is determined from the reference image of the third CU.

[0234] Exemplarily, when the decoder determines that the prediction mode of the third CU is the intra-frame prediction mode based on the prediction mode parsed from the bit stream, the prediction block of the third CU can be determined from the reconstructed pixels adjacent to the third CU space based on the angle information parsed from the BL bit stream (or other information for determining the reference block or pre-agreed intra-frame prediction reference rules).

[0235] S6023, adding the residual of the third CU parsed from the BL bit stream and the prediction block of the third CU to obtain a reconstructed block of the third CU; wherein the reconstructed blocks of multiple CUs constitute a first reconstructed image.

[0236] Exemplarily, the decoder may also parse the residual of the third CU from the BL bitstream; then, the residual of the third CU and the prediction block of the third CU may be added to obtain a reconstructed block of the third CU; thereafter, loop filtering may be performed on the reconstructed block of the third CU, and specific reference may be made to the above Figure 5A The loop filtering process of the reconstructed block of the third CU in the embodiment is not described in detail here.

[0237] It should be noted that in Fig. 6A In an embodiment, the second area in the first reconstructed image can be determined based on the position indication information parsed from the bit stream by the decoder; and then the pixel values ​​of the pixel points in the second area in the first reconstructed image are set to a preset value, such as the dynamic range median (128).

[0238] S603, decoding the enhancement layer bit stream to obtain a second reconstructed image.

[0239] Exemplarily, the decoder may determine the division mode of the second image according to the division information parsed from the bitstream, and further determine the multiple CUs included in the second image; the decoding process for one CU (which may be referred to as the first CU) among the multiple CUs included in the second image may refer to S6031 to S6035:

[0240] S6031, determine a reference image of the first CU according to the reference frame identifier and prediction mode parsed from the EL bit stream.

[0241] S6032: Predict the first CU according to the reference image of the first CU to determine a prediction block of the first CU.

[0242] S6033: Add the residual of the first CU parsed from the EL bit stream and the prediction block of the first CU to obtain a reconstructed block of the first CU; wherein the reconstructed blocks of multiple CUs constitute a second reconstructed image.

[0243] It should be noted that in S6033, when the residual of the first CU parsed from the EL bitstream is 0 and the MV of the first CU is (0,0), the reconstructed block at the same position as the first CU in the first reconstructed image can be used as the prediction block of the first CU.

[0244] Exemplarily, after obtaining the reconstructed block of the first CU, loop filtering may be performed on the reconstructed block of the first CU. For details, refer to the above Figure 5A The description of loop filtering of the reconstructed block of the first CU in the embodiment will not be repeated here.

[0245] It should be noted that in Fig. 6A In an embodiment of the invention, when the decoder parses the position indication information from the bitstream, it can determine whether to perform loop filtering on the reconstructed block of the first CU according to the position indication information parsed from the bitstream. Specifically, it can be determined whether the first CU belongs to the second region according to the position indication information. When the first CU belongs to the second region, the reconstructed block of the first CU can be loop filtered; when the first CU does not belong to the second region, the reconstructed block of the first CU may not be loop filtered.

[0246] When the decoder does not parse the position indication information from the bitstream, after obtaining the reconstructed block of the first CU, if the reference image identified by the reference frame identifier of the first CU is not the reconstructed image of the base layer, it means that the first CU belongs to the second region, and the reconstructed block of the first CU can be loop filtered. If the reference image identified by the reference frame identifier of the first CU is the reconstructed image of the base layer, it means that the first CU does not belong to the second region, and at this time, there is no need to loop filter the reconstructed block of the first CU.

[0247] S604: Use the second reconstructed image as a reconstructed image of the original image.

[0248] Exemplarily, since the image of the first area in the second reconstructed image is the same as the image of the first area in the first reconstructed image, and the image of the second area in the second reconstructed image is a reconstructed image of the image of the second area in the second image; therefore, the second reconstructed image can be directly used as the reconstructed image of the original image.

[0249] Fig. 7AFIG. 1 is a schematic diagram of an exemplary encoding process. Fig. 7A In an embodiment, during the enhancement layer encoding process, at least one of inter-frame prediction, intra-frame prediction and inter-layer prediction can be performed on the CU in the second image, whether it belongs to the second area or the CU belongs to the first area.

[0250] S701, acquiring a first image and a second image; wherein the first image includes an image of a first area in the original image and does not include an image of a second area in the original image, the second image includes an image of the second area in the original image, a size of the first image and a size of the second image are the same as a size of the original image, and the image of the second area in the original image contains user information.

[0251] S702: Acquire location indication information, where the location indication information can be used to indicate the location of the second area.

[0252] S703, performing inter-frame prediction or intra-frame prediction on the first image to obtain a BL bit stream.

[0253] Exemplarily, S703 may include the following steps:

[0254] S7031, divide the first image into multiple CUs.

[0255] S7032, performing inter-frame prediction or intra-frame prediction on a third CU among the multiple CUs obtained by dividing the first image, and determining a prediction block of the third CU.

[0256] S7033, encode the residual between the third CU and the prediction block of the third CU to obtain a bit stream of the third CU; wherein the bit streams of the multiple CUs obtained by dividing the first image constitute a BL bit stream.

[0257] Illustratively, S7031 to S7033 may refer to the description of S5031 to S5033 above, which will not be repeated here.

[0258] It needs to be explained that Figure 5A In different embodiments, Fig. 7A In the embodiment, after loop filtering is performed on all reconstructed blocks of the third CU to obtain the first reconstructed image, there is no need to set the pixel values ​​of the pixels in the second area in the first reconstructed image to the preset values.

[0259] S704, perform inter-frame coding, intra-frame coding, or inter-layer coding on the second image to obtain an EL bit stream.

[0260] Exemplarily, S704 may include the following steps S7041 to S7043:

[0261] S7041, divide the second image into multiple CUs.

[0262] It should be noted that, unlike S5041, in S7041, no constraint condition may be set when dividing the second image into multiple CUs.

[0263] The following is described by taking one CU (hereinafter referred to as the second CU) among the multiple CUs obtained by dividing the second image as an example, and reference may be made to S7042 to S7043:

[0264] S7042, perform inter-frame prediction, intra-frame prediction or inter-layer prediction on the second CU in the multiple CUs, and determine a prediction block of the second CU.

[0265] For example, the process of inter-frame prediction, intra-frame prediction and inter-layer prediction for the second CU can refer to Figure 5A The description in the embodiments is not repeated here.

[0266] Exemplarily, motion search and motion estimation can be performed from the first reconstructed image of the current frame to determine the motion vector (MV) of the second CU and the prediction block of the second CU. That is, in S7042, no restriction is imposed on the position of the reconstructed block referenced by the inter-layer prediction.

[0267] Relative to Figure 5A In terms of embodiments, Fig. 7A In the encoding process of the enhancement layer, the reference queue of the second image has more reference images, so that a more accurate prediction block can be obtained, thereby improving the encoding quality of the enhancement layer, thereby improving the reconstruction quality of the reconstructed image of the original image.

[0268] S7043: Encode a residual between the second CU and the prediction block of the second CU.

[0269] Exemplarily, during the process of encoding the second image, a reconstruction operation is also performed to obtain a second reconstructed image; for details, please refer to the above Figure 5A It should be noted that Figure 5A The difference between the loop filtering of the reconstructed block of the first CU in the embodiment is that Fig. 7A In the embodiment of the present invention, loop filtering may be performed on all reconstructed blocks of the second CU. That is, the reconstructed blocks of all CUs in the second reconstructed image are reconstructed blocks that have been loop filtered.

[0270] It should be noted that when the second image only includes the image of the second area in the original image and does not include the image of the first area in the original image, the image of the first area in the second reconstructed image after loop filtering is not a reconstructed image of the image of the first area in the original image; at this time, in order to serve as a reference for the subsequent encoding of the second image of other frames, the first reconstructed image and the second reconstructed image can be fused to obtain an updated second reconstructed image. At this time, the updated second reconstructed image is the reconstructed image of the original image.

[0271] Figure 7B A prediction reference diagram is shown as an example.

[0272] Reference Figure 7B For example, assume that time k (k is a positive integer) is the current time, corresponding to the first image of the current frame and the second image of the current frame; time k-1 is the previous time, corresponding to the first reconstructed image of the previous frame and the second reconstructed image of the previous frame.

[0273] For example, during the prediction process, Figure 7B The reconstructed block referenced by the third CU of the first image in Figure 5C The reconstructed block referenced by the third CU of the first image is similar and will not be repeated here.

[0274] Reference Figure 7B , exemplarily, inter-layer prediction can be performed for the second CU in the second area in the second image of the current frame, as shown in R09 and R14; wherein R09 indicates that the inter-layer prediction can refer to the reconstructed block of the first area in the first reconstructed image of the current frame (i.e., the reconstructed image of the basic layer), and R14 indicates that the inter-layer prediction can refer to the reconstructed block of the second area in the first reconstructed image of the current frame (i.e., the reconstructed image of the basic layer).

[0275] Reference Figure 7B , exemplarily, inter-frame prediction can be performed for the second CU in the second area in the second image of the current frame, as shown in R10 and R11; wherein R10 represents the inter-frame prediction referring to the reconstructed block in the first area in the second reconstructed image of the previous frame (the reconstructed image of the enhancement layer of the previous frame), and R11 represents the inter-frame prediction referring to the reconstructed block in the second area in the second reconstructed image of the previous frame.

[0276] Reference Figure 7B , exemplarily, intra-frame prediction can be performed for the second CU in the second area in the second image of the current frame, as shown in R12 and R13; wherein R12 indicates that the intra-frame prediction refers to the reconstructed block in the first area in the second image of the current frame, and R13 indicates that the intra-frame prediction refers to the reconstructed block in the second area in the second image of the current frame.

[0277] Similarly, for the second CU in the first area of ​​the second image of the current frame, inter-frame prediction, intra-frame prediction and inter-layer prediction can be performed, as shown in R15 and R16, R17 and R18, R19 and R20, and will not be repeated here.

[0278] S705, adding the encoded position indication information to the BL bit stream and / or the BL bit stream.

[0279] Exemplarily, when the second image is an original image, S705 is an optional step.

[0280] Exemplarily, when the second image includes the image of the second area in the original image and does not include the image of the first area in the original image, S705 is a necessary step.

[0281] It should be noted that the encoding end may also encode the encoding control parameters into the bit stream of the original image, so that the decoding end can use the corresponding control parameters to perform decoding control and decode a reconstructed image of the original image.

[0282] Figure 8 FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 8 The decoding process and Fig. 7A The encoding process corresponds to .

[0283] S801, receiving a bit stream, where the bit stream includes a base layer bit stream, an enhancement layer bit stream, and position indication information.

[0284] S802, decoding the base layer bit stream to obtain a first reconstructed image.

[0285] The decoding process for one CU (which may be referred to as a third CU) among the multiple CUs included in the first reconstructed image may refer to S8021 to S8023:

[0286] S8021, determine a reference image of the third CU according to the reference frame identifier and prediction mode parsed from the BL bitstream.

[0287] S8022: Predict the third CU according to the reference image of the third CU to determine a prediction block of the third CU.

[0288] S8023, adding the residual of the third CU parsed from the BL bit stream and the prediction block of the third CU to obtain a reconstructed block of the third CU; wherein the reconstructed blocks of multiple CUs constitute a first reconstructed image.

[0289] Illustratively, S8021 to S8023 may refer to the description of S6021 to S6023 above, which will not be repeated here.

[0290] S803, decoding the enhancement layer bit stream to obtain a second reconstructed image.

[0291] Exemplarily, S803 may refer to the following description of S8031 to S8033.

[0292] S8031, determine a reference image of the second CU according to the reference frame identifier and prediction mode parsed from the EL bit stream.

[0293] S8032: Predict the second CU according to the reference image of the second CU to determine a prediction block of the second CU.

[0294] S8033, adding the residual of the second CU parsed from the EL bit stream and the prediction block of the second CU to obtain a reconstructed block of the second CU; wherein the reconstructed blocks of multiple CUs constitute a second reconstructed image.

[0295] Illustratively, S8031 to S8033 may refer to the description of S6021 to S6023 above, which will not be repeated here.

[0296] S804: Determine a reconstructed image of the original image according to the first reconstructed image and the second reconstructed image.

[0297] Exemplarily, when the second image only includes the image of the second area in the original image and does not include the image of the first area in the original image, the image of the first area in the first reconstructed image and the image of the second area in the second reconstructed image can be determined based on the position indication information parsed from the bitstream; then, the image of the first area in the first reconstructed image and the image of the second area in the second reconstructed image are fused to obtain an updated second reconstructed image, that is, a reconstructed image of the original image. For example, the pixel value of a pixel point in the second area in the first reconstructed image can be replaced with the pixel value of a pixel point at a corresponding position in the second area in the second reconstructed image. For another example, the pixel value of a pixel point in the first area in the second reconstructed image can be replaced with the pixel value of a pixel point at a corresponding position in the first area in the first reconstructed image.

[0298] Exemplarily, when the second image is the original image, the second reconstructed image obtained after loop filtering is the reconstructed image of the original image. At this time, there is no need to fuse the first reconstructed image and the second reconstructed image to obtain the reconstructed image of the original image. The second reconstructed image can be directly used as the reconstructed image of the original image.

[0299] Fig.9A FIG. 1 is a schematic diagram of a compression process. Fig.9A In the example, the second image is the original image. In this case, the second reconstructed image can be used as the reconstructed image of the original image. Fig.9A The encoding and decoding process can be referred to Figure 5A and Fig. 6A Description in (or refer to Fig. 7Aand Figure 8 ), which will not be repeated here.

[0300] Fig. 9B FIG. 1 is a schematic diagram of a compression process. Fig. 9B In the example, the second image includes the image of the second region in the original image and does not include the image of the first region in the original image. In this case, the second reconstructed image can be used as the reconstructed image of the original image; Fig. 9B The encoding and decoding process can be referred to Figure 5A and Fig. 6A The description in will not be repeated here.

[0301] Fig. 9C FIG. 1 is a schematic diagram of a compression process. Fig. 9C In the example, the second image includes an image of the second region in the original image but does not include an image of the first region in the original image. In this case, a reconstructed image of the original image can be generated by fusing the first reconstructed image and the second reconstructed image. Fig. 9C The encoding and decoding process can be referred to Fig. 7A and Figure 8 The description in will not be repeated here.

[0302] Fig.10 FIG. 1 is a schematic diagram of an exemplary encoding device. Fig.10 The encoding device in can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0303] An image acquisition module 1001 is used to acquire a first image and a second image, wherein the first image includes an image of a first area in an original image and does not include an image of a second area in the original image, and the second image includes an image of the second area in the original image. The size of the first image and the size of the second image are the same as the size of the original image, and the image of the second area in the original image includes user information;

[0304] A base layer encoding module 1002, configured to perform base layer encoding on the first image to obtain a base layer bit stream;

[0305] The enhancement layer encoding module 1003 is used to encode the enhancement layer of the second image to obtain an enhancement layer bit stream; wherein the base layer bit stream and the enhancement layer bit stream are interwoven into one bit stream.

[0306] In one possible scenario, the image acquisition module 1001, the base layer encoding module 1002 and the enhancement layer encoding module 1003 can all be regarded as modules in an encoder.

[0307] Fig.11 FIG. 4 is a schematic diagram of an exemplary decoding device. Fig.11The decoding device in can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0308] The code stream receiving module 1101 is used to receive a bit stream, where the bit stream includes a base layer bit stream and an enhancement layer bit stream;

[0309] A base layer decoding module 1102 is configured to decode a base layer bit stream to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of a first image, the first image includes an image of a first region in an original image and does not include an image of a second region in the original image;

[0310] The enhancement layer decoding module 1103 is used to decode the enhancement layer of the enhancement layer bit stream to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of the second image, and the second image includes an image of the second area in the original image; wherein the first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

[0311] In a possible scenario, the code stream receiving module 1101, the base layer decoding module 1102 and the enhancement layer decoding module 1103 may all be regarded as modules in a decoder.

[0312] In one example, Fig.12 A schematic block diagram of a device 1200 according to an embodiment of the present application is shown. The device 1200 may include: a processor 1201 and a transceiver / transceiver pin 1202 , and optionally, a memory 1203 .

[0313] The various components of the device 1200 are coupled together via a bus 1204, wherein the bus 1204 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, various buses are referred to as bus 1204 in the figure.

[0314] Optionally, the memory 1203 may be used to store instructions in the aforementioned method embodiment. The processor 1201 may be used to execute instructions in the memory 1203, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0315] The apparatus 1200 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0316] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0317] The embodiment of the present application further provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the above embodiment are executed. The interface circuit is a transceiver / transceiver pin 1202.

[0318] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0319] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0320] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0321] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0322] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0323] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0324] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0325] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0326] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0327] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0328] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0329] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0330] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0331] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A layered coding method, It is characterized in that The method comprises: Acquire a first image and a second image, wherein the first image includes an image of a first area in an original image and does not include an image of a second area in the original image, and the second image includes an image of the second area in the original image, a size of the first image and a size of the second image are the same as a size of the original image, and the image of the second area in the original image includes user information; Performing base layer encoding on the first image to obtain a base layer bit stream; performing encoding of an enhancement layer on the second image to obtain an enhancement layer bit stream; The base layer bit stream and the enhancement layer bit stream are interleaved into one bit stream.

2. The method according to claim 1, It is characterized in that The second image also includes an image of the first area in the original image; or, The second image does not include an image of the first area in the original image.

3. The method according to claim 1 or 2, It is characterized in that The encoding of the enhancement layer of the second image to obtain an enhancement layer bit stream includes: Performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the second image to obtain a predicted image of the second image; A residual between the second image and a predicted image of the second image is encoded to obtain the enhancement layer bitstream.

4. The method according to claim 3, It is characterized in that The performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the second image to obtain a predicted image of the second image includes: Dividing the second image to obtain a plurality of coding units CU; For a first CU among the multiple CUs, determining whether the first CU belongs to the second region; When the first CU belongs to the second region, performing inter-frame prediction or intra-frame prediction on the first CU to determine a prediction block of the first CU; When the first CU does not belong to the second region, performing inter-layer prediction on the first CU to determine a prediction block of the first CU; The predicted image of the second image includes prediction blocks of multiple CUs obtained by dividing the second image.

5. The method according to claim 4, It is characterized in that The performing inter-layer prediction on the first CU to determine a prediction block of the first CU includes: Using a reconstructed block in the first reconstructed image that is located at the same position as the first CU as a prediction block of the first CU; The first reconstructed image is a reconstructed image of the first image.

6. The method according to claim 4 or 5, It is characterized in that When the first CU does not belong to the second region, a residual between the prediction block of the first CU and the first CU is 0.

7. The method according to claim 3, It is characterized in that The performing inter-frame prediction, intra-frame prediction or inter-layer prediction on the second image to obtain a predicted image of the second image includes: Dividing the second image to obtain multiple CUs; Performing inter-frame prediction, intra-frame prediction or inter-layer prediction on a second CU among the multiple CUs to determine a prediction block of the second CU; The predicted image of the second image includes prediction blocks of multiple CUs obtained by dividing the second image.

8. The method according to any one of claims 4 to 7, It is characterized in that Any CU among the multiple CUs is located inside or outside the second area.

9. The method according to any one of claims 1 to 8, It is characterized in that The method further comprises: generating a first reconstructed image; Among them, the first reconstructed image is a reconstructed image of the first image, the first reconstructed image is a reference image of the first images of other frames and / or a reference image of the second image; the first reconstructed image is an image after loop filtering, and the pixel values ​​of the pixels in the second area of ​​the first reconstructed image are preset values.

10. The method according to any one of claims 1 to 9, It is characterized in that The method further comprises: generating a second reconstructed image; Among them, the second reconstructed image is a reconstructed image of the second image, the second reconstructed image is a reference image of the second image of other frames, the image of the second area in the second reconstructed image is an image that has undergone loop filtering, and the image of the first area in the second reconstructed image is an image that has not undergone loop filtering.

11. The method according to claim 10, It is characterized in that The method further comprises: When the second image does not include the image of the first area in the original image, fusing the first reconstructed image and the second reconstructed image to obtain an updated second reconstructed image; The first reconstructed image is a reconstructed image of the first image.

12. The method according to any one of claims 1 to 11, It is characterized in that The method further comprises: Acquire location indication information, where the location indication information is used to indicate the location of the second area; The encoded position indication information is added to the base layer bit stream and / or the enhancement layer bit stream.

13. The method according to any one of claims 1 to 12, It is characterized in that The first area in the original image is an area in the original image excluding the second area; The second image is generated by replacing pixel values ​​of pixel points in the first area of ​​the original image; The first image is generated by replacing pixel values ​​of pixel points in a second area in the original image; The replacement method includes at least one of the following: solid color filling, mosaic or Gaussian blur.

14. The method according to any one of claims 1 to 13, It is characterized in that The base layer bitstream and the enhancement layer bitstream include reference frame identification and prediction mode.

15. A layered decoding method, It is characterized in that The method comprises: receiving a bitstream, the bitstream comprising a base layer bitstream and an enhancement layer bitstream; Perform base layer decoding on the base layer bit stream to obtain a first reconstructed image, wherein the first reconstructed image is a reconstructed image of a first image, the first image includes an image of a first area in an original image and does not include an image of a second area in the original image; Decoding the enhancement layer bit stream to obtain a second reconstructed image; wherein the second reconstructed image is a reconstructed image of a second image, and the second image includes an image of a second area in the original image; The first reconstructed image and the second reconstructed image are used to determine the reconstructed image of the original image.

16. The method according to claim 15, It is characterized in that The second image also includes an image of the first area in the original image; or, The second image does not include an image of the first area in the original image.

17. The method according to claim 15 or 16, It is characterized in that The bitstream further includes a reference frame identifier and a prediction mode, and the decoding of the enhancement layer on the enhancement layer bitstream to obtain a second reconstructed image includes: Determining a reference image for the second image according to the reference frame identifier and the prediction mode; Performing prediction according to a reference image of the second image to obtain a predicted image of the second image; The predicted image of the second image is added to the residual of the second image parsed from the enhancement layer bitstream to obtain the second reconstructed image.

18. The method according to claim 17, It is characterized in that The second image includes a plurality of coding units CU, the reference image of the second image includes reference images of the plurality of CUs, and performing prediction according to the reference image of the second image to obtain a predicted image of the second image includes: For a first CU among the multiple CUs, predict the first CU according to a reference image of the first CU to determine a prediction block of the first CU; The predicted image of the second image includes prediction blocks of multiple CUs included in the second image.

19. The method according to any one of claims 15 to 18, It is characterized in that The first reconstructed image is an image after loop filtering, and the pixel values ​​of the pixels in the second area of ​​the first reconstructed image are preset values.

20. The method according to any one of claims 15 to 19, It is characterized in that The image of the second area in the second reconstructed image is an image that has been loop filtered, and the image of the first area in the second reconstructed image is an image that has not been loop filtered.

21. The method according to any one of claims 15 to 20, It is characterized in that The method further comprises: When the second image does not include the image of the first area in the original image, the first reconstructed image and the second reconstructed image are fused to obtain an updated second reconstructed image.

22. The method according to claim 19, It is characterized in that The bit stream further includes encoded position indication information, where the position indication information is used to indicate the position of the second area; and the method further includes: A second area in the first reconstructed image is determined according to the position indication information.

23. The method according to claim 20 or 21, It is characterized in that The bitstream further includes a reference frame identifier and / or encoded position indication information, where the position indication information is used to indicate the position of the second area, and the method further includes: A second region in the second reconstructed image is determined according to the reference frame identifier and / or the position indication information.

24. The method according to claim 21, It is characterized in that The bit stream further includes encoded position indication information, where the position indication information is used to indicate the position of the second area; and the method further includes: determining, according to the position indication information, a first region in the first reconstructed image and a second region in the second reconstructed image; The fusing the first reconstructed image and the second reconstructed image to obtain an updated second reconstructed image includes: The image of the first area in the first reconstructed image and the image of the second area in the second reconstructed image are fused to obtain an updated second reconstructed image.

25. The method according to any one of claims 15 to 24, It is characterized in that The method further comprises: The second reconstructed image is used as the reconstructed image of the original image.

26. An electronic device, It is characterized in that include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the encoding method according to any one of claims 1 to 14.

27. An electronic device, It is characterized in that include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the decoding method according to any one of claims 15 to 25.

28. A chip, It is characterized in that It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps described in any one of claims 1 to claim 25 are performed.

29. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 25.

30. A computer program product, It is characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 25 are performed.

Images