Encoding method, apparatus, and electronic device

Abstract

The application provides an encoding method, device and electronic equipment. The method comprises the following steps: acquiring a video stream; dividing an image included in the video stream into a first block and a second block, the first block containing a foreground in the image, and the second block containing only a background in the image; encoding the first block in a first image to obtain a substream of the first block in the first image, the first image being a to-be-encoded image determined from the video stream according to a first encoding frame rate; encoding the second block in a second image to obtain a substream of the second block in the second image, the second image being a to-be-encoded image determined from the video stream according to a second encoding frame rate; and the substream of the first block in the first image and the substream of the second block in the second image forming a code stream, the first encoding frame rate being greater than the second encoding frame rate, and the second encoding frame rate being less than a maximum encoding frame rate supported by an encoder. The application can break through the maximum encoding frame rate and the maximum encoding channel number supported by the encoder.

Description

Technical Field

[0001] This application relates to the field of encoding and decoding, and more particularly to an encoding method, apparatus and electronic device. Background Technology

[0002] Real-time video scenarios, such as video conferencing and live streaming, place high demands on video smoothness; therefore, video encoding strategies are particularly crucial. Currently, a dynamic frame rate adjustment (VFR) strategy is commonly used in video encoding. This strategy allows the encoder to flexibly adjust the encoding frame rate based on network conditions: when network conditions are good, the encoder can increase the encoding frame rate; while when network conditions are poor, the encoder can decrease the encoding frame rate to ensure a smooth video experience for the decoder.

[0003] In dynamic frame rate adjustment strategies, the maximum configurable encoding frame rate and maximum number of encoding channels are limited by the encoder's maximum encoding capacity. For example, for an encoder that supports one channel of 4K@30FPS (i.e., the maximum encoding frame rate for 4K video is 30 frames per second), under ideal network conditions, the encoder can only encode one channel at a maximum of 30FPS. This means that although dynamic frame rate adjustment strategies can improve the frame rate, they still cannot exceed the encoder's encoding capacity limits, such as the maximum encoding frame rate and the maximum number of encoding channels supported by the encoder. Summary of the Invention

[0004] In view of this, this application provides an encoding method, apparatus, and electronic device. In this method, the upper limits of the encoder's encoding capabilities, such as the maximum encoding frame rate and the maximum number of encoding paths supported by the encoder, can be overcome.

[0005] In a first aspect, this application provides an encoding method, the method comprising: first, acquiring a video stream comprising multiple frames of images; then, dividing the images comprising the video stream into a first block and a second block, the first block containing the foreground of the image, and the second block containing only the background of the image, wherein the first block and the second block are one or more; subsequently, encoding the first block in the first image to obtain a sub-stream of the first block in the first image, the first image being an image to be encoded determined from the video stream according to a first encoding frame rate; and encoding the second block in the second image to obtain a sub-stream of the second block in the second image, the second image being an image to be encoded determined from the video stream according to a second encoding frame rate; the sub-stream of the first block in the first image and the sub-stream of the second block in the second image constitute a bitstream; the first encoding frame rate is greater than the second encoding frame rate, and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder.

[0006] The phrase "encoding a first block in a first image, wherein the first image is an image to be encoded determined from the video stream according to a first encoding frame rate" can be understood as inputting the first block to the encoder according to the first encoding frame rate; this can be seen as the encoder encoding the first block according to the first encoding frame rate. Similarly, the phrase "encoding a second block in a second image, wherein the second image is an image to be encoded determined from the video stream according to a second encoding frame rate" can be understood as inputting the second block to the encoder according to the second encoding frame rate; this can be seen as the encoder encoding the second block according to the second encoding frame rate. In other words, this application uses different encoding frame rates for encoding the first block and the second block.

[0007] In one possible scenario, the first encoding frame rate is greater than the maximum encoding frame rate supported by the encoder, and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder. Thus, the actual encoding frame rate obtained by conversion is less than or equal to the maximum encoding frame rate supported by the encoder; therefore, the encoding method of this application can exceed the maximum encoding frame rate supported by the encoder.

[0008] For example, in a 4K image, the first block occupies 6 / 16 of the image, and the second block occupies 10 / 16. Assuming the first encoding frame rate is 60 FPS and the second encoding frame rate is 10 FPS, the encoder's actual encoding computing power (or the encoder's actual encoding frame rate) is calculated as: 60 FPS * (6 / 16) + 10 FPS * (10 / 16) = 4K@28.75 FPS. This means that providing encoding hardware (or software) at 4K@30 FPS can overcome the limitation and achieve an encoding effect of 4K@60 FPS (this 4K@60 FPS encoding effect can be achieved for the first block containing the foreground of the image).

[0009] For example, in a 4K image, the first block occupies 3 / 8 of the image, and the second block occupies 5 / 8. Assuming the first encoding frame rate is 60 FPS and the second encoding frame rate is 10 FPS, the encoder's actual encoding computing power (or the encoder's actual encoding frame rate) is calculated as: 60 FPS * (3 / 8) + 10 FPS * (5 / 8) = 4K@28.75 FPS. This means that providing encoding hardware (or software) at 4K@30 FPS can overcome the limitation and achieve an encoding effect of 4K@60 FPS.

[0010] In one possible scenario, the first encoding frame rate is less than or equal to the maximum encoding frame rate supported by the encoder, and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder. Thus, the actual encoding frame rate of the encoder obtained by conversion is less than the maximum encoding frame rate supported by the encoder; consequently, the encoder can add at least one additional data channel to the encoding process. Therefore, the encoding method of this application can exceed the maximum number of encoding channels supported by the encoder.

[0011] For example, in a 4K image, the first block occupies 6 / 16 of the frame, and the second block occupies 10 / 16. Assuming the first encoding frame rate is 30 FPS and the second encoding frame rate is 6 FPS, the actual encoding computational power (or the encoder's actual encoding frame rate) for encoding one channel of data is: 30 FPS * (6 / 16) + 6 FPS * (10 / 16) = 15 FPS. Thus, for a 4K@30 FPS encoder, it can encode two channels of data. In other words, providing one 4K@30 FPS encoding hardware (or software) can overcome the limitation and achieve the encoding effect of two 4K@30 FPS channels.

[0012] For example, the two “encoding” steps mentioned above can be performed by the encoder; the two steps of “acquiring” and “dividing” can be performed by other modules.

[0013] For example, the first encoding frame rate can refer to the frame rate at which the first block is input to the encoder; for example, the first encoding frame rate is 60 FPS. From another perspective, the first encoding frame rate can be seen as the instantaneous frame rate at which the encoder encodes the first block (it should be noted that the encoder's encoding frame rate parameter is not adjusted during this process; for example, the encoder's encoding frame rate parameter remains at 30 FPS). For example, the first encoding frame rate can be preset by the system or the user, or it can be determined based on the current network conditions; this application does not impose any restrictions on this.

[0014] For example, the second encoding frame rate can refer to the frame rate at which the second block is input to the encoder, and the second encoding frame rate is less than the first encoding frame rate; for example, the second encoding frame rate is 30 FPS, 20 FPS, 10 FPS, etc. From another perspective, the second encoding frame rate can be seen as the instantaneous frame rate at which the encoder encodes the second block (it should be noted that the encoder's encoding frame rate parameter is not adjusted during this process). For example, the second encoding frame rate can be preset by the system or the user, or it can be determined based on the current network status; this application does not impose any restrictions on this.

[0015] For example, the first encoding frame rate may be an integer multiple of the second encoding frame rate.

[0016] For example, the first block is a block containing the foreground in the image. It can be understood that there are two cases for the first block: one is that the first block only contains the foreground in the image, and the other is that the first block contains both the foreground and the background in the image.

[0017] For example, when the original acquisition frame rate (the frame rate at which the video stream is acquired by an image acquisition device or recording tool, etc.) and the first encoding frame rate are equal, each frame in the video stream can be divided into a first block and a second block. When the first encoding frame rate is not equal to the original acquisition frame rate, the video stream can be downsampled according to the first encoding frame rate to obtain the target video stream; then, each frame in the target video stream is divided into a first block and a second block.

[0018] For example, "both the first block and the second block are one or more" can include the following cases: Case 1, the first block is one and the second block is multiple; Case 2, the first block is multiple and the second block is one; Case 3, the first block is multiple and the second block is multiple; Case 4, the first block is one and the second block is one.

[0019] For example, the substream of the first block in the first image and the substream of the second block in the second image can be merged to obtain a bitstream. It should be understood that "merging" can mean that the encoder actually performs the operation of "merging" or "concatenation", or it can be understood as the encoder outputting a substream after the previous substream after obtaining a substream, without performing the operation of "concatenation" or "merging".

[0020] For example, a bitstream can also be called a bitstream, coded bitstream, etc.

[0021] According to the first aspect, both the first block and the second block are tiles; or, both the first block and the second block are slices.

[0022] For example, a slice can contain a series of macroblocks (the basic coding units in the H.264 video coding standard) or coding tree units (CTUs) (the basic coding units in the H.265 video coding standard). A slice can be divided according to time. A slice can be a horizontal strip (a slice consists of one or more rows of pixels in an image) or an irregular shape.

[0023] For example, a tile can contain a series of CTUs, a tile can be divided according to space, and a tile can be a rectangular area of ​​an image.

[0024] When the encoder is an H.265 encoder, the image can be divided into multiple tiles, meaning both the first and second blocks are tiles; alternatively, the image can be divided into multiple slices, meaning both the first and second blocks are slices. Dividing the image into multiple tiles is more flexible than dividing it into multiple slices.

[0025] When the encoder is an H.264 encoder, the image can be divided into multiple slices, that is, the first block and the second block are both slices.

[0026] According to the first aspect, or any implementation of the first aspect above, the method further includes: encoding the second block in the third image using a skip mode to obtain a sub-stream of the second block in the third image, wherein the third image is an image in the first image other than the one identified as the second image; the sub-stream of the first block in the first image, the sub-stream of the second block in the second image, and the sub-stream of the second block in the third image constitute the bitstream. In this way, the encoded frame rate of the second block can be padded to be the same as the encoded frame rate of the first block.

[0027] It should be noted that "encoding the second block in the third image using skip mode (a type of inter-frame prediction)" essentially means not encoding the residual block and motion vector corresponding to the second block, but instead using the bitstream description information of the second block to generate the substream of the second block. For the encoder, encoding the second block in the third image using skip mode requires very little computational power, which can be ignored; that is, by padding the encoding frame rate of the second block to be the same as that of the first block in this way, it consumes virtually no computational power from the encoder.

[0028] For example, the bitstream description information of the second block may include the location information of the second block and the prediction mode identifier of the second block. The prediction mode identifier of the second block's bitstream has a first identifier value, which indicates that the inter-frame prediction mode of the block is a skip mode. The location information of the second block may include the coordinates of the top-left corner pixel of the second block and the size information of the block.

[0029] Furthermore, it facilitates the decoding side to know the reconstruction method of the second block in the third image. That is, when the decoding side can determine from the substream of the second block that the inter-frame prediction method used by the second block is skip mode, it can copy the reconstructed image of the second block in the previous frame that corresponds to the position of the second block in the current frame (the position of the second block corresponding to this position in the reconstructed image of the previous frame is the same as or close to the position of the second block in the current frame) as the reconstructed image of the second block in the current frame.

[0030] It should be noted that, since users tend to focus on the foreground of an image while watching a video and ignore the background, although the encoding method of this application makes the background of the reconstructed images in adjacent frames the same, this is negligible to the user, or in other words, easily overlooked by the user.

[0031] For the decoding side, the dynamic frame rate adjustment strategy in the prior art will cause the video frame rate to change abruptly when the network conditions suddenly improve or deteriorate, which will lead to a visual discontinuity. However, the frame rate of the reconstructed video stream obtained by the decoding side of this application remains unchanged, and there will be no visual discontinuity.

[0032] According to the first aspect, or any implementation of the first aspect above, the first block includes a first type of block and a second type of block, the motion state of the first type of block satisfies a preset condition, and the motion state of the second type of block does not satisfy the preset condition; encoding the first block in the first image to obtain a sub-stream of the first block in the first image includes: encoding the first type of block in the first image to obtain a sub-stream of the first type of block in the first image; the method further includes: encoding the second type of block in a fourth image to obtain a sub-stream of the second type of block in the fourth image, the fourth image being an image to be encoded determined from the video stream according to a third encoding frame rate, the third encoding frame rate being less than the first encoding frame rate; the sub-stream of the first type of block in the first image, the sub-stream of the second block in the second image, and the sub-stream of the second type of block in the fourth image constitute the bitstream.

[0033] This allows for a further increase in the encoding frame rate of the first type of block. For example, in a 4K image, the proportion of the first type of block is 4 / 16, the proportion of the second type of block is 2 / 16, and the proportion of the third type of block is 10 / 16. Assuming the first encoding frame rate is 80 FPS, the second encoding frame rate is 10 FPS, and the third encoding frame rate is 20 FPS; the actual encoding computing power of the encoder (or the actual encoding frame rate of the encoder) is calculated as: 80 FPS * (4 / 16) + 20 FPS * (2 / 16) + 10 FPS * (10 / 16) = 4K@28.75 FPS. That is, providing encoding hardware (or software) at 4K@30 FPS can overcome the limitation and achieve an encoding effect of 4K@80 FPS (for the first type of block, an encoding effect of 4K@80 FPS can be achieved).

[0034] For example, the motion state can be represented by motion speed, and the preset condition can be that the motion speed is greater than a first threshold; that is, the first type of block is the block in the first block whose motion speed is greater than the first threshold, and the second type of block is the block in the first block whose motion speed is less than the first threshold. It should be understood that the motion state can also be represented by other parameters, or the motion speed can be represented by a combination of other parameters, and this application does not limit this.

[0035] For example, the first block can be further divided into more categories. For instance, multiple first blocks can be divided into multiple categories based on the motion state of each first block. For example, the motion state can be represented by motion speed (e.g., motion detection can be performed on the foreground in the image to determine the motion speed of the foreground). Blocks with motion speeds greater than a first threshold among multiple first blocks can be identified as first-class blocks; blocks with motion speeds greater than a second threshold but less than the first threshold among multiple first blocks can be identified as second-class blocks; blocks with motion speeds greater than a third threshold but less than the second threshold among multiple first blocks can be identified as third-class blocks, and so on. Wherein, the first threshold is greater than the second threshold, the second threshold is greater than the third threshold, and so on.

[0036] Furthermore, the encoding frame rate corresponding to each type of block in the first block can be determined; wherein, the encoding frame rate corresponding to the first type of block can be the first encoding frame rate, the encoding frame rate corresponding to the second type of block can be the third encoding frame rate, the encoding frame rate corresponding to the third type of block can be the fourth encoding frame rate, the encoding frame rate corresponding to the fourth type of block can be the fifth encoding frame rate, and so on. Then, each type of block in the first block can be input into the encoder according to its corresponding encoding frame rate. The encoding frame rate corresponding to the first type of block is greater than the encoding frame rate corresponding to other types of blocks (this application does not restrict the relationship between the encoding frame rate corresponding to other types of blocks in the first block and the second encoding frame rate; optionally, the encoding frame rate corresponding to other types of blocks in the first block is greater than or equal to the second encoding frame rate).

[0037] According to the first aspect, or any implementation of the first aspect above, the method further includes: encoding the second type of blocks in the fifth image using a skip mode to obtain a sub-stream of the second type of blocks in the fifth image, wherein the fifth image is an image in the first image other than the fourth image; the sub-stream of the first type of blocks in the first image, the sub-stream of the second type of blocks in the second image, the sub-stream of the second type of blocks in the fourth image, and the sub-stream of the second type of blocks in the fifth image constitute the bitstream. In this way, the encoded frame rate of the second type of blocks can be padded to be the same as the encoded frame rate of the first type of blocks.

[0038] It should be noted that "encoding the second type of block in the fifth image using skip mode (a type of inter-frame prediction)" essentially means not encoding the residual block and motion vector corresponding to the second type of block, but instead using the bitstream description information of the second type of block to generate the substream of that second type of block. For the encoder, encoding the second type of block in the fifth image using skip mode requires very little computational power, which can be ignored; that is, by padding the encoding frame rate of the second type of block to be the same as that of the first type of block in this way, it basically does not consume any computational power of the encoder.

[0039] For example, the bitstream description information of the second type of block may include the location information of the second type of block and the prediction mode identifier of the second type of block. The prediction mode identifier of the second type of block's bitstream has a first identifier value, which indicates that the inter-frame prediction mode of the block is a skip mode. The location information of the second type of block may include the coordinates of the top-left corner pixel of the second type of block and the size information of the block.

[0040] In addition, it makes it easier for the decoding side to know the reconstruction method of the second type of block in the fifth image. That is, when the decoding side can determine that the inter-frame prediction method used by the second type of block is skip mode through the sub-stream of the second type of block, it can copy the reconstructed image of the second type of block in the previous frame that corresponds to the position of the second type of block in the current frame, and use it as the reconstructed image of the second type of block in the current frame.

[0041] It should be noted that, since users tend to focus their attention on the foreground with large motion displacement while ignoring the foreground with small motion displacement during video viewing, although the encoding method of this application will make the second type of blocks of the reconstructed images of adjacent frames the same, it can be ignored by users, or in other words, it is easy for users to ignore.

[0042] According to the first aspect, or any implementation of the first aspect above, encoding a first block in a first image to obtain a sub-stream of the first block in the first image includes: predicting the first block in the first image to determine a predicted value of the first block in the first image; determining a residual value of the first block in the first image based on the original value of the first block in the first image and the predicted value of the first block in the first image; and entropy encoding the residual value of the first block in the first image to obtain a sub-stream of the first block in the first image.

[0043] According to the first aspect, or any implementation of the first aspect above, encoding the second block in the second image to obtain a sub-stream of the second block in the second image includes: predicting the second block in the second image to determine the predicted value of the second block in the second image; determining the residual value of the second block in the second image based on the original value of the second block in the second image and the predicted value of the second block in the second image; and entropy encoding the residual value of the second block in the second image to obtain a sub-stream of the second block in the second image.

[0044] According to the first aspect, or any implementation thereof, the first block in the first image includes boundary coding units and other coding units. The method further includes: quantizing the residual values ​​of the other coding units of the first block in the first image according to a first quantization parameter, and quantizing the residual values ​​of the boundary coding units of the first block in the first image according to a second quantization parameter, wherein the second quantization parameter is smaller than the first quantization parameter; entropy coding is performed on the residual values ​​of the first block in the first image to obtain a sub-stream of the first block in the first image, including: entropy coding is performed on the quantized residual values ​​of the first block in the first image to obtain a sub-stream of the first block in the first image. Since a smaller quantization parameter results in a smaller error between the quantization result and the actual result, and a higher reconstruction quality, this application can improve the quality of the first block boundary.

[0045] According to the first aspect, or any implementation thereof, the second block in the second image includes boundary coding units and other coding units. The method further includes: quantizing the residual values ​​of the other coding units of the second block in the second image according to a third quantization parameter, and quantizing the residual values ​​of the boundary coding units of the second block in the second image according to a fourth quantization parameter, wherein the fourth quantization parameter is smaller than the third quantization parameter; entropy coding is performed on the residual values ​​of the second block in the second image to obtain a sub-stream of the second block in the second image, comprising: entropy coding is performed on the quantized residual values ​​of the second block in the second image to obtain a sub-stream of the second block in the second image. Since a smaller quantization parameter results in a smaller error between the quantization result and the actual result, and a higher reconstruction quality, this application can improve the quality of the second block boundary.

[0046] Correspondingly, on the decoding side, the decoder can smooth the boundary coding units of two adjacent first blocks, smooth the boundary coding units of two adjacent second blocks, and smooth the boundary coding units of the first block and the second block, so as to make the boundary transition between two adjacent blocks smoother and more fluid, thereby improving the visual smoothness of the reconstructed image.

[0047] Secondly, this application provides an encoding apparatus, which includes a preprocessing module and an encoder, wherein:

[0048] The preprocessing module is used to acquire a video stream, which includes multiple frames of images; divide the images in the video stream into a first block and a second block, wherein the first block contains the foreground of the image and the second block contains only the background of the image, and there are one or more first blocks and second blocks; input the first block of the first image and the second block of the second image to the encoder, wherein the first image is an image to be encoded determined from the video stream according to a first encoding frame rate, and the second image is an image to be encoded determined from the video stream according to a second encoding frame rate; wherein the first encoding frame rate is greater than the second encoding frame rate, and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder;

[0049] The encoder is used to encode a first block in the first image to obtain a sub-stream of the first block in the first image; and to encode a second block in the second image to obtain a sub-stream of the second block in the second image; the sub-stream of the first block in the first image and the sub-stream of the second block in the second image constitute a bitstream.

[0050] According to the second aspect, both the first block and the second block are tiles; or, both the first block and the second block are slices.

[0051] According to the second aspect, or any implementation of the second aspect above, the encoder is further configured to encode the second block in the third image using a skip mode to obtain a sub-stream of the second block in the third image, wherein the third image is an image in the first image other than the one identified as the second image; the sub-stream of the first block in the first image, the sub-stream of the second block in the second image, and the sub-stream of the second block in the third image constitute the bitstream.

[0052] According to the second aspect, or any implementation of the second aspect above, the first block includes a first type of block and a second type of block, the motion state of the first type of block satisfies a preset condition, and the motion state of the second type of block does not satisfy the preset condition; the encoder is used to encode the first type of block in the first image to obtain a sub-stream of the first type of block in the first image; and to encode the second type of block in the fourth image to obtain a sub-stream of the second type of block in the fourth image, the fourth image being an image to be encoded determined from the video stream according to a third encoding frame rate, the third encoding frame rate being less than the first encoding frame rate; the sub-stream of the first type of block in the first image, the sub-stream of the second block in the second image, and the sub-stream of the second type of block in the fourth image constitute the bitstream.

[0053] According to the second aspect, or any implementation of the second aspect above, the encoder is further configured to encode the second type of blocks in the fifth image using a skip mode to obtain a sub-stream of the second type of blocks in the fifth image, wherein the fifth image is an image in the first image other than the fourth image; the sub-stream of the first type of blocks in the first image, the sub-stream of the second blocks in the second image, the sub-stream of the second type of blocks in the fourth image, and the sub-stream of the second type of blocks in the fifth image constitute the bitstream.

[0054] Thirdly, this application provides an encoding device, which includes:

[0055] The acquisition module is used to acquire a video stream, which includes multiple frames of images.

[0056] The segmentation module is used to divide the images included in the video stream into a first block and a second block. The first block contains the foreground of the image, and the second block contains only the background of the image. There are one or more first blocks and the second block.

[0057] An encoding module is configured to encode a first block in a first image to obtain a sub-stream of the first block in the first image, wherein the first image is an image to be encoded determined from the video stream according to a first encoding frame rate; and to encode a second block in a second image to obtain a sub-stream of the second block in the second image, wherein the second image is an image to be encoded determined from the video stream according to a second encoding frame rate; the sub-stream of the first block in the first image and the sub-stream of the second block in the second image constitute a bitstream; wherein the first encoding frame rate is greater than the second encoding frame rate, and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder.

[0058] It should be understood that the encoding device of the third aspect can be used to execute the encoding method of the first aspect and any implementation thereof, which will not be elaborated here.

[0059] The technical effects of the third aspect and any implementation thereof can be found in the first aspect and any implementation thereof, and will not be repeated here.

[0060] Fourthly, this application provides a bitstream generated by an encoding method according to the first aspect and any implementation thereof.

[0061] Fifthly, this application provides an electronic device, including: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, cause the electronic device to perform the method of the first aspect or any possible implementation thereof.

[0062] In a sixth aspect, this application provides a chip including one or more interface circuits and one or more processors; the one or more processors receive or transmit data through the one or more interface circuits, and when the one or more processors execute computer instructions, cause the electronic device to perform the method in the first aspect or any possible implementation of the first aspect.

[0063] In a seventh aspect, this application provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the method of the first aspect or any possible implementation thereof.

[0064] Eighthly, this application provides a computer-readable storage medium storing a bitstream generated according to the method in the first aspect or any possible implementation thereof.

[0065] Ninthly, this application provides a computer program product, characterized in that the computer program product includes computer instructions that, when executed by a computer or processor, cause the steps of the method in the first aspect or any possible implementation of the first aspect to be performed.

[0066] In a tenth aspect, this application also provides a method for storing a bitstream, the method comprising: receiving a bitstream generated according to the method in the first aspect or any possible implementation thereof; and storing the bitstream.

[0067] Eleventhly, this application also provides a method for transmitting a bitstream, the method comprising: storing a bitstream generated according to the method of the first aspect or any possible implementation thereof; and transmitting the bitstream. In this way, the bitstream can be transmitted to a decoding side or a distribution server.

[0068] In a twelfth aspect, this application also provides a method for transmitting a bitstream, the method comprising: receiving a bitstream generated according to the first aspect or any possible implementation thereof; and sending the bitstream. Thus, the bitstream can be sent to a decoding side.

[0069] In a thirteenth aspect, this application also provides a system for storing a bitstream, the system comprising: a receiving module and a storage module; the receiving module is configured to receive a bitstream generated according to the method in the first aspect or any possible implementation thereof; the storage module is configured to store the bitstream.

[0070] In a fourteenth aspect, this application also provides a system for transmitting a bitstream, the system comprising: a transceiver module and a storage module; the storage module is used to store the bitstream generated according to the method in the first aspect or any possible implementation thereof; the transceiver module is used to transmit the bitstream. Thus, the bitstream can be transmitted to a decoding side or a distribution server.

[0071] In a fifteenth aspect, this application also provides a system for transmitting a bitstream, the system comprising: a transceiver module configured to receive a bitstream generated according to a method in accordance with the first aspect or any possible implementation thereof; the transceiver module is further configured to transmit the bitstream. Thus, the bitstream can be transmitted to a decoding side.

[0072] In this embodiment, the electronic device, computer-readable storage medium, computer program product, chip or codec, system, etc. are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above. Attached Figure Description

[0073] Figure 1A This is a schematic diagram illustrating one application scenario of an embodiment of this application;

[0074] Figure 1B This is a schematic diagram illustrating another application scenario of an embodiment of this application;

[0075] Figure 2A This is a schematic diagram of a data transmission process according to an embodiment of this application;

[0076] Figure 2B This is a schematic diagram of another data transmission process according to an embodiment of this application;

[0077] Figure 3A This is a schematic diagram of the structure of an encoding device 300 according to an embodiment of this application;

[0078] Figure 3B This is a schematic diagram illustrating the division of an image into blocks according to an embodiment of this application;

[0079] Figure 3C This is a schematic diagram of the structure of an encoder according to an embodiment of this application;

[0080] Figure 3D This is a schematic diagram of the structure of a decoder according to an embodiment of this application;

[0081] Figure 4A This is a schematic diagram of an encoding process 400 according to an embodiment of this application;

[0082] Figure 4B This is a schematic diagram illustrating another embodiment of the present application where an image is divided into blocks;

[0083] Figure 4C This is a schematic diagram illustrating another embodiment of image division into blocks according to this application;

[0084] Figure 4D This is a schematic diagram illustrating how to determine an image to be encoded from a video stream according to an embodiment of this application;

[0085] Figure 4E This is a schematic diagram of a bitstream structure according to an embodiment of this application;

[0086] Figure 5A This is a schematic diagram of another encoding process 500 according to an embodiment of this application;

[0087] Figure 5B This is a schematic diagram of another bitstream structure according to an embodiment of this application;

[0088] Figure 6A This is a schematic diagram of another encoding process 600 according to an embodiment of this application;

[0089] Figure 6B This is a schematic diagram of another stream structure according to an embodiment of this application;

[0090] Figure 7 This is a schematic diagram of another encoding process 700 according to an embodiment of this application;

[0091] Figure 8 This is a schematic diagram of the frame of another encoder according to an embodiment of this application;

[0092] Figure 9 This is a schematic diagram of another encoding device 900 according to an embodiment of this application;

[0093] Figure 10 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0094] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0095] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0096] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0097] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0098] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0099] In the embodiments of this application, the modules / components shown in the framework diagram (or structural diagram or system diagram) are merely examples of this application. The actual framework (or structure or system) may include more or fewer modules / components than those shown in the diagram, or may have different component configurations. Furthermore, the various components / modules shown in the diagrams may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0100] Figure 1A This is a schematic diagram illustrating one application scenario of an embodiment of this application. Figure 1A The image shown depicts a video conferencing scenario.

[0101] Figure 1A Users located in different locations can conduct video conferences using terminal devices. Figure 1A In this scenario, a user located at location E can create a video conference using a terminal device. Once a user at location E joins the video conference, that user can be called the host, and the terminal device used by the host to join the video conference can be called participant terminal 1. Subsequently, participant A at location A can join the video conference through participant terminal 2, participant B at location B can join through participant terminal 3, participant C at location C can join through participant terminal 4, participant D at location D can join through participant terminal 5, and so on.

[0102] During a video conference, the camera of participant terminal 1 can capture an image of the scene where the host is located (and / or the screen displayed on participant terminal 1 recorded by its screen recording tool) to obtain video stream 1. Subsequently, participant terminal 1 can encode video stream 1 to obtain bitstream 1. Afterwards, participant terminal 1 can send bitstream 1 to multiple participant terminals, including participant terminal 2, participant terminal 3, participant terminal 4, and participant terminal 5, via a cloud server. Each of these participant terminals can decode and reconstruct video stream 1 based on bitstream 1 and then play the reconstructed video stream 1 locally. The camera of participant terminal 2 can capture the image of user A (and / or the screen displayed on participant terminal 2 recorded by its screen recording tool) to obtain video stream 2. Subsequently, participant terminal 2 can encode video stream 2 to obtain bitstream 2. Then, participant terminal 2 can send bitstream 2 to multiple participant terminals, including participant terminal 1, participant terminal 3, participant terminal 4, and participant terminal 5, via a cloud server. Each participant terminal, after decoding the reconstructed video stream 2 based on bitstream 2, can play the reconstructed video stream 2 locally. Similarly, the same applies to other participant terminals, which will not be elaborated further here. In this way, multi-terminal real-time video conferencing can be achieved.

[0103] Figure 1B This is a schematic diagram illustrating another application scenario of this application embodiment. Figure 1B The image shown is from a live stream.

[0104] exist Figure 1B In this scenario, the streamer can activate the live streaming application on their device and broadcast live through the application. Simultaneously, user A can use phone A to enter the streamer's live stream room to watch; user B can use phone B to enter the streamer's live stream room to watch; user C can use tablet C to enter the streamer's live stream room to watch; ...

[0105] During a live stream, the camera of the live streaming device captures images of the scene where the streamer is located (such as images containing the streamer and products) to obtain a video stream. The live streaming device then encodes this video stream to obtain a bitstream. This bitstream is then transmitted via a cloud server to multiple viewing devices, such as mobile phone A, mobile phone B, and tablet C. Each of these devices can decode the bitstream to reconstruct the video stream locally. This allows multiple devices to watch the same live stream simultaneously.

[0106] It should be understood that, Figure 1A and Figure 1B This is only a partial example of the real-time video scenarios described in this application. This application can also be applied to other real-time video scenarios such as distance education, monitoring and security, and this application does not limit these applications.

[0107] For example, the data transmission process in a real-time video scene can be referred to as follows: Figure 2A and Figure 2B .

[0108] Figure 2A This is a schematic diagram of a data transmission process according to an embodiment of this application. Figure 2A Therefore Figure 1A The data transmission process is illustrated using the example of a Chinese participant terminal, which includes a display screen and a video conferencing box.

[0109] Reference Figure 2A The image acquisition module 111 on the display screen 11 of any participating terminal can capture video to obtain a video stream (main video stream) 101 (and / or the screen recording tool of any participating terminal can record the screen 11 to obtain a video stream (secondary video stream) 101). Then, the display screen 11 can send the video stream 101 to the video conferencing box 12. Afterwards, the video conferencing box 12 can encode the video stream 101 to obtain a bitstream 102; subsequently, the video conferencing box 12 can upload the bitstream 102 to the cloud server 13. After receiving the bitstream 102, the cloud server 13 can store the bitstream 102 and distribute it to the video conferencing boxes 14 of other participating terminals. The video conferencing box 14 can decode the bitstream 102 to obtain the reconstructed video stream 103 (which may include reconstructing the main video stream and / or reconstructing the auxiliary video stream); then, the video conferencing box 14 can send the reconstructed video stream 103 to the corresponding other participating terminals, and the other participating terminals can play the reconstructed video stream 103 through the display screen 15.

[0110] Figure 2B This is a schematic diagram of another data transmission process according to an embodiment of this application.

[0111] by Figure 1A Taking mobile phones as an example, the terminals used by Chinese participants at the conference Figure 2B The data transmission process in the process can be as follows: Figure 2BDuring the meeting, any participating user (including the host) can use their mobile phone 16 to capture video to obtain a video stream (the main video stream of the meeting) (and / or the screen recording tool on mobile phone 16 can record the screen displayed on mobile phone 16 to obtain a video stream (the secondary video stream of the meeting)). This video stream can then be encoded to obtain a bitstream 104. Mobile phone 16 can then send the bitstream 104 to cloud server 13. After receiving the bitstream 104, cloud server 13 can store the bitstream 104 and distribute it to other participating users' mobile phones 17. Mobile phone 17 can decode the bitstream 104 and play the reconstructed video stream (the reconstructed meeting video stream).

[0112] by Figure 1B Taking mobile phones as an example, in China, live streaming devices... Figure 2B The data transmission process in the process can be described as follows: Figure 2B In the process, the broadcaster's mobile phone 16 can capture video to obtain a video stream (live video stream) and encode the video stream to obtain a bitstream 104. Then, the broadcaster's mobile phone 16 can send the bitstream 104 to the cloud server 13. After receiving the bitstream 104, the cloud server 13 can store the bitstream 104 and distribute it to the mobile phones 17 of users watching the live stream. The mobile phone 17 can decode the bitstream 104 and play the reconstructed video stream (reconstructed live video stream).

[0113] It should be noted that this application does not limit whether the device used to generate the video stream and the device used to encode the video stream are the same device, nor does it limit whether the device used to decode the video stream and the device used to play the reconstructed video stream are the same device.

[0114] It should be understood that, Figure 1A The participating terminal can also be other terminal devices such as tablets, laptops, smart TVs, etc. Figure 1B The live streaming device can also be other terminal devices; and Figure 1A and Figure 1B The cloud server mentioned can also be other types of servers, such as physical (dedicated) servers, server clusters, etc. Furthermore, the cloud server in this application can have different names depending on the application scenario, for example... Figure 1A The cloud server in the context can also be called a conference server. Figure 1B The cloud server in this context can also be called a live streaming server, etc.

[0115] Figure 3A This is a schematic diagram of the structure of an encoding device 300 according to an embodiment of this application.

[0116] Reference Figure 3AThe encoding device 300 may include a preprocessing module 301 and an encoder 302. It should be understood that both the preprocessing module 301 and the encoder 302 may be implemented in hardware or software, and this application does not impose any restrictions on this.

[0117] For example, the preprocessing module 301 can preprocess the images (here, an image can refer to a complete image) included in the input video stream, such as dividing the image into multiple blocks according to preset rules, and setting or controlling the encoding parameters (such as quantization control parameters, quantization parameters, prediction methods, etc.) corresponding to each block.

[0118] Figure 3B This is a schematic diagram illustrating the division of an image into blocks according to an embodiment of this application.

[0119] Figure 3B (1) shows the division of the image into two slices, where each slice can be called a block; where, Figure 3B (1) The bold dashed line in the middle is the boundary between two slices. A slice can contain a series of macroblocks (the basic coding unit in the H.264 video coding standard) or coding tree units (CTUs) (the basic coding unit in the H.265 video coding standard). Slices can be divided according to time. A slice can be a horizontal strip (a slice consists of one or more rows of pixels in an image) or an irregular shape.

[0120] Figure 3B (2) shows that the image is divided into 6 tiles, and each tile can be called a block; among them, Figure 3B (2) The bold dashed lines in the image are the boundaries between each tile and other tiles. A tile can contain a series of CTUs, tiles can be divided according to space, and a tile can be a rectangular area of ​​an image.

[0121] Figure 3B (1) A small square in the middle represents a CTU or macroblock. Figure 3B (2) A small square in the diagram represents a CTU. This should be understood. Figure 3B (1) This is only one example of how to divide a slice in this application. A complete image can be divided into more slices in other ways. Figure 3B (2) This is only one example of how tiles are divided in this application. A complete image may be divided into more or fewer tiles in other ways.

[0122] For example, encoder 302 can be used to encode each block to output a bitstream (wherein the bitstream may also be referred to as an encoded bitstream, bitstream, etc.).

[0123] In other words, Figure 3A The encoding device can be used to encode each complete image in a video stream to output a bitstream.

[0124] Figure 3C This is a schematic diagram of the structure of an encoder according to an embodiment of this application. Figure 3C What is shown is Figure 3A An example of the structure of encoder 302.

[0125] Reference Figure 3C For example, encoder 302 may include: block module, intra-frame prediction module, inter-frame prediction module, transform module, quantization module, entropy coding module, inverse quantization module, inverse transform module, loop filtering module, and memory.

[0126] Continue to refer to Figure 3C The encoding process of the encoder can be as follows: the encoder's block module acquires the image to be encoded (where, Figure 3C The image input to the encoder can be a complete image or a block within a complete image (such as a tile or slice). The image is then divided into blocks to be encoded (e.g., coding tree units, CUs). For each coding unit, a decision module can... Figure 3C (Not shown in the image) Determine whether to perform inter-frame prediction or intra-frame prediction for this coding unit.

[0127] When it is determined that intra-frame prediction is required for a coding unit, the intra-frame prediction module performs intra-frame prediction on the coding unit to determine the prediction block corresponding to the coding unit (that is, a block composed of the predicted values ​​of each pixel in the coding unit). After obtaining the prediction block corresponding to the coding unit, the residual value between the coding unit and the prediction block corresponding to the coding unit can be determined to obtain the residual block corresponding to the coding unit (that is, a block composed of the residual values ​​between the original values ​​of the pixels at the corresponding positions of the coding unit and the prediction block corresponding to the coding unit). Then, the residual block corresponding to the coding unit is input to the transform module, which transforms the residual block corresponding to the coding unit to obtain the transform result and outputs the transform result to the quantization module. Subsequently, the quantization module can quantize the transformation result, obtain the quantization result, and output the quantization result to the entropy coding module. Then, the entropy coding module can entropy code the quantization result to obtain the encoded data of the residual block corresponding to the coding unit, and write the residual data corresponding to the coding unit into the bitstream segment of the coding unit (the bitstream segments of all coding units in a block form the substream of the block, the sub-blocks of all blocks in a complete image form the bitstream of the image, and the bitstreams of all images contained in a video stream form the bitstream of the video stream (that is, the bitstream mentioned later)).

[0128] For example, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs it to the inverse transform module. Next, the inverse transform module performs an inverse transform on the inverse quantization result to obtain the decoded residual block corresponding to the coding unit. Then, the decoded residual block corresponding to the coding unit can be superimposed with the prediction block corresponding to the coding unit to obtain the reconstructed block corresponding to the coding unit. Subsequently, the reconstructed block corresponding to the coding unit is input to the loop filtering module, which performs loop filtering on the reconstructed block corresponding to the coding unit to obtain the filtered reconstructed block corresponding to the coding unit and outputs it to the memory. (Whereinafter obtaining the filtered reconstructed block corresponding to the coding unit, the loop filtering module can stitch the filtered reconstructed block corresponding to the coding unit to the corresponding position of a complete image to be reconstructed; after obtaining a complete reconstructed image frame, the complete reconstructed image can be output to the memory for storage.)

[0129] When it is determined that inter-frame prediction is required for a coding unit, the inter-frame prediction module can perform inter-frame prediction on the coding unit to determine the prediction block corresponding to the coding unit. For example, the inter-frame prediction module can select a reference frame of the image to be encoded from the reference frame sequence in the memory; for the coding unit in the image to be encoded, it can search for a matching prediction block from the reference frame of the image to be encoded to obtain the prediction block corresponding to the coding unit. After obtaining the prediction block corresponding to the coding unit, the residual block between the coding unit and the prediction block corresponding to the coding unit can be determined; then, the residual block corresponding to the coding unit is input to the transform module, which transforms the residual block corresponding to the coding unit, obtains the transform result, and outputs the transform result to the quantization module. Subsequently, the quantization module can quantize the transform result, obtain the quantization result, and output the quantization result to the entropy coding module; then, the entropy coding module can entropy code the quantization result to obtain the encoded data of the residual block corresponding to the coding unit, and write the residual data corresponding to the coding unit into the bitstream segment of the coding unit. In addition, the inter-frame prediction module can perform inter-frame prediction on the coding unit, determine the motion vector corresponding to the coding unit, encode the motion vector corresponding to the coding unit, and write the encoded data of the motion vector corresponding to the coding unit into the bitstream segment of the coding unit.

[0130] For example, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs it to the inverse transform module. Next, the inverse transform module performs an inverse transform on the inverse quantization result to obtain the decoded residual block corresponding to the coding unit. Then, the decoded residual block corresponding to the coding unit can be superimposed with the prediction block corresponding to the coding unit (wherein, the prediction block corresponding to the coding unit can be determined from the prediction blocks output by the inter-frame prediction module based on the motion vector corresponding to the coding unit) to obtain the reconstructed block corresponding to the coding unit. Subsequently, the reconstructed block corresponding to the coding unit is input to the loop filtering module, which performs loop filtering on the reconstructed block corresponding to the coding unit to obtain the filtered reconstructed block corresponding to the coding unit and outputs it to the memory. (Wherein, after obtaining the filtered reconstructed block corresponding to the coding unit, the loop filtering module can stitch the filtered reconstructed block corresponding to the coding unit to the corresponding position of a complete image to be reconstructed; after obtaining a complete reconstructed image frame, the complete reconstructed image can be output to the memory for storage).

[0131] Figure 3D This is a schematic diagram of the structure of a decoder according to an embodiment of this application. Figure 3D The structure of the decoder and Figure 3C The structure corresponds to that of the encoder.

[0132] Reference Figure 3D The decoder may include: an entropy decoding module, an inverse quantization module, an inverse transform module, an intra-frame prediction module, an inter-frame prediction module, a loop filtering module, and a memory.

[0133] For example, after receiving the bitstream, the decoder can parse the bitstream to obtain the parsing result. The parsing process can be as follows: The encoded data of the residual block corresponding to the encoding unit is extracted from the bitstream. Then, the encoded data of the residual block corresponding to the encoding unit is input to the entropy decoding module, which performs entropy decoding on the encoded data of the residual block corresponding to the encoding unit to obtain the entropy-decoded data of the residual block corresponding to the encoding unit. Next, the entropy-decoded data of the residual block corresponding to the encoding unit can be input to the dequantization module, which performs dequantization to obtain dequantized data, which is then input to the inverse transform module. Subsequently, the inverse transform module performs an inverse transform on the dequantized data to obtain the decoded residual block corresponding to the encoding unit.

[0134] It should be noted that bitstream description information can also be extracted from the bitstream. In this case, there is no need to perform entropy decoding, inverse quantization, and inverse transform operations on the bitstream description information. This bitstream description information can be used in the subsequent decoding process.

[0135] Refer again Figure 3D When it is determined, based on the bitstream description information parsed from the bitstream, that intra-frame prediction is required for a specific coding unit, the intra-frame prediction module can perform intra-frame prediction on that coding unit to obtain the prediction block corresponding to that coding unit (typically, the prediction block corresponding to that coding unit can be determined based on the bitstream description information). Then, the decoded residual block corresponding to that coding unit can be superimposed with the prediction block corresponding to that coding unit to obtain the reconstructed block corresponding to that coding unit. Next, loop filtering can be performed on the reconstructed block corresponding to that coding unit to obtain the filtered reconstructed block corresponding to that coding unit, and the filtered reconstructed block corresponding to that coding unit is output to the memory. (The loop filtering module, after obtaining the filtered reconstructed block corresponding to that coding unit, can stitch the filtered reconstructed block corresponding to that coding unit to the corresponding position in a complete image to be reconstructed; after obtaining a reconstructed image frame, the reconstructed image can be output to the memory for storage).

[0136] Refer again Figure 3DFor example, when it is determined, based on the bitstream description information parsed from the bitstream, that inter-frame prediction is required for a coding unit (at this time, the reconstructed value of the motion vector corresponding to the coding unit can also be parsed from the bitstream), the inter-frame prediction module can perform inter-frame prediction on the coding unit to obtain the prediction block corresponding to the coding unit. For example, the inter-frame prediction module can determine a reference frame of a complete image to be reconstructed from the reference frame sequence in the memory based on the bitstream description information parsed from the bitstream; then, the inter-frame prediction module can determine the prediction block corresponding to the coding unit based on the reconstructed value of the motion vector and the reference frame.

[0137] After obtaining the prediction block corresponding to the coding unit, the decoded residual block corresponding to the coding unit can be superimposed with the prediction block corresponding to the coding unit to obtain the reconstruction block corresponding to the coding unit, which is then output to the loop filtering module. The loop filtering module then performs loop filtering on the reconstruction block corresponding to the coding unit to obtain the filtered reconstruction block corresponding to the coding unit, and outputs the filtered reconstruction block corresponding to the coding unit to the memory. (Note: After obtaining the filtered reconstruction block corresponding to the coding unit, the loop filtering module can stitch the filtered reconstruction block corresponding to the coding unit to the corresponding position in a complete image to be reconstructed; after obtaining a reconstructed image frame, the reconstructed image can be output to the memory.)

[0138] The following combination Figure 3A and Figure 3C The encoding method of this application is explained.

[0139] Figure 4A This is a schematic diagram of an encoding process 400 according to an embodiment of this application. In the encoding process 400, steps S401 to S402 are performed by... Figure 3A Preprocessing module 301 executes the process, and steps S403 to S404 are performed by... Figure 3A The encoder 302 is executed.

[0140] S401, acquire video stream, the video stream includes multiple frames of images.

[0141] For example, the video stream in S401 can refer to a real-time video stream, such as a conference video stream, a live video stream, etc.

[0142] For example, in a video conferencing scenario, an image acquisition device can capture video of the environment in which the participants are currently located to obtain the main video stream of the meeting. Then, the image acquisition device can output this video stream to the preprocessing module 301 of participating terminals such as mobile phones, tablets, and video conferencing boxes.

[0143] For example, in a live streaming scenario, an image acquisition device can capture video of the user's current environment to obtain a live video stream. The image acquisition device can then output this live video stream to the preprocessing module 301 of a live streaming device such as a mobile phone or tablet.

[0144] For example, a video stream may include a sequence of images, which may include multiple frames of images. In S401, the images included in the video stream refer to complete images.

[0145] For ease of explanation later, the frame rate of the video stream captured by the image acquisition device can be referred to as the raw acquisition frame rate. For example, the raw acquisition frame rate can be 120 FPS (Frames per Second).

[0146] S402, the video stream is divided into a first block and a second block, the first block contains the foreground of the image, and the second block contains only the background of the image, and there are one or more first blocks and second blocks.

[0147] For example, when the original acquisition frame rate and the first encoding frame rate are equal, the preprocessing module 301 can divide each frame of the video stream acquired in S401 into a first block and a second block. For each frame of the video stream, the preprocessing module 301 can identify the foreground and background of the frame image; subsequently, the preprocessing module 301 can divide the frame image into two types of blocks according to the foreground and background of the frame image: a first block and a second block; wherein, the first block is a block containing the foreground of the image (it can be understood that there are two cases for the first block, namely, one case is that the first block only contains the foreground of the image, and the other case is that the first block contains both the foreground and the background of the image), and the second block is a block containing only the background of the image.

[0148] For example, the first block may be one or more, and the second block may be one or more.

[0149] For example, the first encoding frame rate can refer to the frame rate at which the preprocessing module 301 inputs the first block to the encoder 302; for example, the first encoding frame rate is 60 FPS. From another perspective, the first encoding frame rate can be seen as the instantaneous frame rate at which the encoder encodes the first block (it should be noted that the encoder's encoding frame rate parameter is not adjusted during this process; for example, the encoder's encoding frame rate parameter remains at 30 FPS). For example, the first encoding frame rate can be preset by the system or the user, or it can be determined based on the current network status; this application does not impose any restrictions on this.

[0150] Figure 4B This is a schematic diagram illustrating another embodiment of the present application where an image is divided into blocks. Figure 4B This illustrates a diagram of dividing an image into multiple tiles, i.e. Figure 4B Both the first and second blocks are tiles.

[0151] Figure 4B In (1), the preprocessing module 301 divides image 1 in the video stream into 16 tiles; of which 6 tiles form the first block and the other 10 tiles form the second block. The 6 tiles forming the first block can be processed as follows: Figure 4B (1) The gray rectangular blocks represent Tile 6, Tile 7, Tile 10, Tile 11, Tile 14, and Tile 15; each of these six tiles contains both the foreground and background of image 1. The ten tiles forming the second block can be arranged as follows: Figure 4B (1) shows the white rectangular blocks, namely Tile 1 to Tile 5, Tile 8, Tile 9, Tile 12, Tile 13 and Tile 16; each of these 10 tiles contains only the background of image 1.

[0152] Figure 4B In (2), the preprocessing module 301 divides image 2 in the video stream into 16 tiles; of which 6 tiles form the first block and the other 10 tiles form the second block. The 6 tiles forming the first block can be processed as follows: Figure 4B (2) The gray rectangular blocks represent Tile 5, Tile 6, Tile 9, Tile 10, Tile 13, and Tile 14; each of these six tiles contains both the foreground and background of image 2. The ten tiles forming the second block can be arranged as follows: Figure 4B (2) shows the white rectangular blocks, namely Tile 1 to Tile 4, Tile 7, Tile 8, Tile 11, Tile 12, Tile 15 and Tile 16; each of these 10 tiles contains only the background of image 2.

[0153] Figure 4C This is a schematic diagram illustrating another embodiment of the present application of dividing an image into blocks. Figure 4C This illustrates a diagram of dividing an image into multiple slices, i.e. Figure 4C Both the first and second blocks are Slices.

[0154] Figure 4C In the video stream, preprocessing module 301 divides image 3 into 8 slices; 3 slices form the first block, and the other 5 slices form the second block. The 3 slices forming the first block can be processed as follows: Figure 4CThe medium gray rectangular blocks represent Slice 4, Slice 5, and Slice 6; each of these three slices contains both the foreground and background of image 3. The five slices forming the second block can be arranged as follows... Figure 4C The white rectangular blocks in the middle are Slice 1 to Slice 3, Slice 7 and Slice 8; each of these 5 slices contains only the background of image 3.

[0155] When encoder 302 is an H.265 encoder, the image can be divided into multiple tiles, meaning both the first and second blocks are tiles; alternatively, the image can be divided into multiple slices, meaning both the first and second blocks are slices. Dividing the image into multiple tiles is more flexible than dividing it into multiple slices.

[0156] When encoder 302 is an H.264 encoder, the image can be divided into multiple slices, that is, the first block and the second block are both slices.

[0157] Furthermore, when an image in a video stream contains multiple foreground objects or has a large foreground area, dividing the image into multiple slices results in a smaller first block size compared to dividing the image into multiple tiles.

[0158] It should be understood that the first and second blocks can be divided in other ways, and this application does not impose any restrictions on this.

[0159] For example, when the first encoding frame rate is not equal to the original acquisition frame rate, the preprocessing module 301 can downsample the video stream according to the first encoding frame rate to obtain the target video stream; then, for each frame image in the target video stream, the foreground and background of the frame image are identified; subsequently, the preprocessing module 301 can divide the frame image into a first block and a second block according to the foreground and background of the frame image.

[0160] For example, the preprocessing module 301 can perform human detection on the image to determine the foreground and background in the image.

[0161] S403, the first block in the first image is encoded to obtain a sub-stream of the first block in the first image, wherein the first image is the image to be encoded determined from the video stream according to the first encoding frame rate.

[0162] For example, the preprocessing module 301 can input the first block to the encoder 302 according to the first encoding frame rate, and the encoder 302 can encode the first block.

[0163] For example, the preprocessing module 301 can determine the image to be encoded (hereinafter referred to as the first image) from the video stream according to the first encoding frame rate. When the first encoding frame rate is equal to the original acquisition frame rate, each frame in the video stream is the first image; when the first encoding frame rate is not equal to the original acquisition frame rate, some images in the video stream are the first images; or, in other words, the preprocessing module 301 can determine each frame in the target video stream as the first image. The first image consists of multiple frames.

[0164] Figure 4D This is a schematic diagram illustrating how to determine an image to be encoded from a video stream according to an embodiment of this application.

[0165] Assuming the original capture frame rate of the video stream is 120 FPS and the first encoding frame rate is 60 FPS, the process of determining the first image from the video stream is illustrated using the first 24 frames of a 1-second video stream as an example.

[0166] Figure 4D The first frame is represented by F0, the second frame by F1, and so on, with the 24th frame represented by F23. The image represented by the gray rectangle in the video stream is determined as the first image, which includes F0, F2, F4, F6, F8, F10, F12, F14, F16, F18, F20, F22, and so on.

[0167] Subsequently, the preprocessing module 301 can sequentially input the first blocks in each frame of the first image to the encoder 302 according to the frame number or timestamp of each frame. When a frame of the first image includes multiple first blocks, the multiple first blocks of the frame can be sequentially input to the encoder 302 according to the index order of these multiple first blocks; or they can be sequentially input to the encoder 302 according to the position coordinates of these multiple first blocks, in a left-to-right and top-to-bottom order. The encoder 302 can sequentially encode the input first blocks to obtain the substream of the first block in each frame of the first image.

[0168] For example, encoder 302 can encode each first block according to the encoding process described above; it will not be repeated here. Specifically, encoder 302 performs intra-frame prediction for the first block in a first image that is an I-frame, and inter-frame prediction for the first block in a first image that is a P-frame or B-frame.

[0169] S404, Encode the second block in the second image to obtain a sub-stream of the second block in the second image. The second image is the image to be encoded determined from the video stream according to the second encoding frame rate, and the first encoding frame rate is greater than the second encoding frame rate.

[0170] For example, the preprocessing module 301 can input the second block to the encoder 302 according to the second encoding frame rate, and the encoder 302 can encode the second block.

[0171] For example, the second encoding frame rate can refer to the frame rate at which the preprocessing module 301 inputs the second block to the encoder 302, and the second encoding frame rate is less than the first encoding frame rate; for example, the second encoding frame rate is 30 FPS, 20 FPS, 10 FPS, etc. From another perspective, the second encoding frame rate can be regarded as the instantaneous frame rate of the encoder encoding the second block (it should be noted that the encoding frame rate parameter of the encoder is not adjusted during this process). For example, the second encoding frame rate can be preset by the system or the user, or it can be determined according to the current network status; this application does not limit this.

[0172] For example, the second encoding frame rate may be less than the maximum encoding frame rate supported by the encoder; this application does not limit the size relationship between the first encoding frame rate and the maximum encoding frame rate supported by the encoder.

[0173] For example, the first encoding frame rate may be an integer multiple of the second encoding frame rate.

[0174] For example, when the first encoding frame rate is the same as the original acquisition frame rate, the preprocessing module 301 can determine the image to be encoded (hereinafter referred to as the second image) from the video stream or target video stream acquired in S401 according to the second encoding frame rate; wherein, the second image consists of multiple frames. When the first encoding frame rate is different from the original acquisition frame rate, the preprocessing module 301 can determine the second image from the target video stream according to the second encoding frame rate.

[0175] Assuming the second encoding frame rate is 10 FPS, continue referring to the above. Figure 4D The image represented by the rectangular blocks with dashed borders in the first image is determined as the second image. The second image includes: F0, F10, F20, ...

[0176] Subsequently, the preprocessing module 301 can sequentially input the second blocks in each frame of the second image to the encoder 302 according to the frame number or timestamp of each frame of the second image. When a frame of the second image includes multiple second blocks, the multiple second blocks of the frame of the second image can be sequentially input to the encoder 302 according to the index order of these multiple second blocks; or the multiple second blocks of the frame of the second image can be sequentially input to the encoder 302 according to the position coordinates of these multiple second blocks, in a left-to-right and top-to-bottom order. The encoder 302 can sequentially encode the input second blocks to obtain the sub-stream of the second blocks in each frame of the second image.

[0177] For example, encoder 302 can encode each second block according to the encoding process described above; it will not be repeated here. Specifically, intra-frame prediction is performed for the second block in the second image that is an I-frame, and inter-frame prediction is performed for the second block in the second image that is a P-frame or B-frame. It should be noted that the encoder performs the encoding operations in S403 and S404 in a similar manner.

[0178] It should be understood that the process of determining the second image described above is essentially determining the second image from multiple frames of the first image; that is, there exists an image in the video stream or target video stream that is identified as both the first image and the second image; for example, Figure 4D F0, F10, and F20 are identified as the first image, and also as the second image. For ease of subsequent description, the images in the first image other than those identified as the second image can be referred to as the third image; for example... Figure 4D In the first image, rectangles with solid borders (including rectangles with solid borders and no squares filled, and rectangles with solid borders and squares filled) can be called the third image. The third images include: F2, F4, F6, F8, F12, F14, F16, F18, F22, ...

[0179] Since the encoder 302 encodes the video stream frame by frame during the encoding process, for the second image, the preprocessing module 301 can input the first and second blocks of the second image into the encoder 302, which then executes steps S403 and S404 to obtain sub-streams of the first and second blocks of the second image (at this time, the first image in S403 refers to the image determined to be the second image, i.e., the first block of the second image is encoded to obtain the sub-stream of the first block of the second image). For the third image, the preprocessing module 301 only needs to input the first block of the third image into the encoder 302, which then executes step S403 (at this time, the first image in S403 refers to the third image other than the one determined to be the second image) to obtain the sub-stream of the first block of the third image.

[0180] Refer again Figure 4D For the three second images F0, F10, and F20, the preprocessing module 301 can input the first block and the second block of each second image to the encoder 302; for the nine third images F2, F4, F6, F8, F12, F14, F16, F18, and F22, the preprocessing module 301 can input the first block of each third image to the encoder 302.

[0181] Specifically, the sub-streams of the first block in the second image and the first block in the third image can be combined to form the sub-stream of the first block; the sub-streams of the second block in the second image can be combined to form the sub-stream of the second block. Then, the encoder 302 can concatenate the sub-streams of the first block and the sub-streams of the second block to obtain the bitstream.

[0182] The following uses two frames from a video stream as an example to illustrate the structure of the sub-stream of the first block, the sub-stream of the second block, and the bitstream.

[0183] For example, suppose Figure 4B (1) is the third image, and Figure 4B (1) The encoding order of the first block in the image is Tile6→Tile7→Tile10→Tile11→Tile14→Tile15; and assuming Figure 4B (2) is the second image, and Figure 4B (2) The encoding order of the first and second blocks in the image is Tile 1→Tile 2→Tile 3→Tile 4→Tile 5→……→Tile 15→Tile 16; and Figure 4B The image in (1) is Figure 4D F8 in the middle, Figure 4B The image in (2) is Figure 4D Press F10 in the middle to complete the process. Figure 4B (1) image and Figure 4B After encoding the image in (2), the resulting substream of the first block can be as follows: Figure 4E As shown in (1), the sub-stream of the second block can be as follows Figure 4E As shown in (2).

[0184] Figure 4E (1) The sub-stream of the first block includes Figure 4B (1) The sub-flows of the 6 tiles (first block) in the image and Figure 4B (2) The image contains 6 sub-streams of tiles (first blocks); each sub-stream of a tile (first block) is a gray rectangle. Each sub-stream of a first block (i.e., each tile) may include the encoded data of the first block (encoded data can also be called payload, and the encoded data may include, for example, the encoded data of the residual block corresponding to the encoded unit included in the first block, the encoded data of the motion vector corresponding to the encoded unit included in the first block, etc.) and the bitstream description information of the first block (such as the index or position coordinates of the first block, the prediction method of the first block, etc.).

[0185] Figure 4E (2) The sub-stream of the second block includes Figure 4B(2) The image contains 10 sub-streams of tiles (second blocks); each sub-stream of a tile (second block) is a white rectangle. Each sub-stream of a second block may include the encoded data of the second block and the bitstream description information of the second block, etc.

[0186] After that, Figure 4E (1) Sub-stream of the first block and Figure 4E (2) The substream concatenation of the second block in the code stream results in the following bitstream: Figure 4E As shown in (3).

[0187] In summary, when the first encoding frame rate is greater than the maximum encoding frame rate supported by the encoder, and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder, the actual encoding frame rate of the encoder obtained by conversion is less than or equal to the maximum encoding frame rate supported by the encoder; it can be seen that the encoding method of this application can break through the maximum encoding frame rate supported by the encoder.

[0188] For example, in a 4K image, the first block occupies 6 / 16 of the image, and the second block occupies 10 / 16. Assuming the first encoding frame rate is 60 FPS and the second encoding frame rate is 10 FPS, the encoder's actual encoding computing power (or the encoder's actual encoding frame rate) is calculated as: 60 FPS * (6 / 16) + 10 FPS * (10 / 16) = 4K@28.75 FPS. This means that providing encoding hardware (or software) at 4K@30 FPS can overcome the limitation and achieve an encoding effect of 4K@60 FPS (this 4K@60 FPS encoding effect can be achieved for the first block containing the foreground of the image).

[0189] For example, in a 4K image, the first block occupies 3 / 8 of the image, and the second block occupies 5 / 8. Assuming the first encoding frame rate is 60 FPS and the second encoding frame rate is 10 FPS, the encoder's actual encoding computing power (or the encoder's actual encoding frame rate) is calculated as: 60 FPS * (3 / 8) + 10 FPS * (5 / 8) = 4K@28.75 FPS. This means that providing encoding hardware (or software) at 4K@30 FPS can overcome the limitation and achieve an encoding effect of 4K@60 FPS.

[0190] When the first encoding frame rate is less than or equal to the maximum encoding frame rate supported by the encoder, and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder, the actual encoding frame rate of the encoder obtained by conversion is less than the maximum encoding frame rate supported by the encoder; thus, the encoder can add encoding at least one additional data channel; it can be seen that the encoding method of this application can break through the maximum number of encoding channels supported by the encoder.

[0191] For example, in a 4K image, the first block occupies 6 / 16 of the frame, and the second block occupies 10 / 16. Assuming the first encoding frame rate is 30 FPS and the second encoding frame rate is 6 FPS, the actual encoding computational power (or the encoder's actual encoding frame rate) for encoding one channel of data is: 30 FPS * (6 / 16) + 6 FPS * (10 / 16) = 15 FPS. Thus, for a 4K@30 FPS encoder, it can encode two channels of data. In other words, providing one 4K@30 FPS encoding hardware (or software) can overcome the limitation and achieve the encoding effect of two 4K@30 FPS channels.

[0192] Based on the above description, it can be seen that the first block in the third image is encoded, but the second block in the third image is not encoded. Thus, the decoding side can decode the reconstructed image of the first block in the third image from the bitstream, but cannot obtain the reconstructed image of the second block in the third image. Therefore, the encoding side needs to tell the decoding side how to reconstruct the reconstructed image of the second block in the third image.

[0193] For example, the encoding side can use a skip mode to encode the second block in the third image, that is, write the bitstream description information of the second block in the third image into the bitstream; in this way, the decoding side can parse the bitstream description information of the second block from the bitstream and reconstruct the reconstructed image of the second block according to the bitstream description information of the second block.

[0194] The following describes the process of the encoder using the skip mode to encode the second block in the third image.

[0195] Figure 5A This is a schematic diagram of another encoding process 500 according to an embodiment of this application. In the encoding process 500, S501 to S502 are performed by... Figure 3A Preprocessing module 301 executes the process, and steps S503 to S506 are performed by... Figure 3A The encoder is executed.

[0196] S501, acquire video stream, the video stream includes multiple frames of images.

[0197] S502, the video stream is divided into a first block and a second block. The first block contains the foreground of the image, and the second block contains only the background of the image. There are one or more first blocks and second blocks.

[0198] S503, the first block in the first image is encoded to obtain a sub-stream of the first block in the first image, wherein the first image is the image to be encoded determined from the video stream according to the first encoding frame rate.

[0199] S504, the second block in the second image is encoded to obtain a sub-stream of the second block in the second image. The second image is the image to be encoded determined from the video stream according to the second encoding frame rate. The first encoding frame rate is greater than the second encoding frame rate.

[0200] For example, S501 to S504 can be referred to the description of S401 to S404 above, and will not be repeated here.

[0201] S505, the second block in the third image is encoded using a skip mode to obtain a sub-stream of the second block in the third image. The third image is the image in the first image other than the one that is determined to be the second image.

[0202] In this way, the encoding frame rate of the second block can be padded to be the same as that of the first block.

[0203] For example, "encoding the second block in the third image using skip mode (a type of inter-frame prediction)" essentially means not encoding the residual block and motion vector corresponding to the second block, but instead generating a sub-stream of the second block using the bitstream description information of the second block. The bitstream description information of the second block may include the location information of the second block and the prediction mode identifier of the second block. The prediction mode identifier included in the bitstream of the second block has a first identifier value, which indicates that the inter-frame prediction mode of the block is skip mode. The location information of the second block may include the coordinates of the top-left pixel of the second block and the size information of the block. It should be understood that when the second block is divided into multiple coding units, the bitstream description information of each coding unit in the second block can be used to generate the bitstream segments of each coding unit; the bitstream segments of multiple coding units in the second block are then concatenated to obtain the sub-stream of the second block.

[0204] During reconstruction, the encoder can copy the filtered reconstruction block of the second block in the second block of the previous reconstructed image that corresponds to the position of the second block in the current frame (the position of the second block in the previous reconstructed image is the same as or close to the position of the second block in the current frame), and use it as the filtered reconstruction block corresponding to that second block in the current frame. Specifically, for each coding unit in the second block of the current frame, the encoder can copy the filtered reconstruction block of the coding unit in the second block of the previous reconstructed image that corresponds to the position of that coding unit in the second block of the current frame, and use it as the filtered reconstruction block corresponding to that coding unit in the second block of the current frame.

[0205] It should be noted that the preprocessing module 301 can set the prediction method of the encoder 302 for the second block in the third image to skip mode; however, it is not necessary to input the second block in the third image into the encoder 302.

[0206] For encoder 302, the computational power required to encode the second block in the third image using the skip mode is very small and can be ignored; that is, by using this method, the encoding frame rate of the second block is made up to be the same as that of the first block, which basically does not consume the computational power of encoder 302.

[0207] It should be noted that, since users tend to focus on the foreground of an image while watching a video and ignore the background, although the encoding method of this application makes the background of the reconstructed images in adjacent frames the same, this is negligible to the user, or in other words, easily overlooked by the user.

[0208] For the decoding side, the dynamic frame rate adjustment strategy in the prior art will cause the video frame rate to change abruptly when the network conditions suddenly improve or deteriorate, which will lead to a visual discontinuity. However, the frame rate of the reconstructed video stream obtained by the decoding side of this application remains unchanged, and there will be no visual discontinuity.

[0209] It should be noted that S503 can be executed for each first image; at the same time, it can be determined whether the first image is identified as a second image or a third image; when the first image is identified as a second image, S504 can be executed; when the first image is identified as a third image, S505 can be executed. That is to say, for a single first image, S503 can be executed, as well as either S504 or S505.

[0210] S506, the sub-stream of the first block in the first image, the sub-stream of the second block in the second image, and the sub-stream of the second block in the third image are spliced ​​together to obtain the bitstream.

[0211] For example, the sub-streams of the first block in the first image (including the sub-streams of the first block in the second image and the sub-streams of the first block in the third image) can form the sub-stream of the first block, and the sub-streams of the second block in the second image and the sub-streams of the second block in the third image can form the sub-stream of the second block.

[0212] The following uses two frames from a video stream as an example to illustrate the structure of the sub-stream of the first block, the sub-stream of the second block, and the bitstream.

[0213] For example, suppose Figure 4B (1) is the third image, and Figure 4B(1) The encoding order of the first and second blocks in the image is Tile 1→Tile 2→Tile 3→Tile 4→Tile 5→……→Tile 15→Tile 16; and assuming Figure 4B (2) is the second image, and Figure 4B (2) The encoding order of the first and second blocks in the image is Tile 1→Tile 2→Tile 3→Tile 4→Tile 5→……→Tile 15→Tile 16; and Figure 4B The image in (1) is Figure 4D F8 in the middle, Figure 4B The image in (2) is Figure 4D Press F10 in the middle to complete the process. Figure 4B (1) image and Figure 4B After encoding the image in (2), the resulting substream of the first block can be as follows: Figure 5B As shown in (1), the sub-stream of the second block can be as follows Figure 5B As shown in (2).

[0214] Figure 5B (1) The sub-stream of the first block includes Figure 4B (1) The sub-flows of the 6 tiles (first block) in the image and Figure 4B (2) The image contains 6 sub-streams of tiles (first blocks); each sub-stream of a tile (first block) is a gray rectangle. Each sub-stream of a first block (i.e., each tile) may include the encoded data of the first block and the bitstream description information of the first block, etc.

[0215] Figure 5B (2) The sub-stream of the second block includes Figure 4B (1) The sub-flows of 10 tiles (second block) in the image and Figure 4B (2) The image contains 10 sub-streams of tiles (second blocks); each tile (second block) is a white rectangle. Figure 4B (1) The substream of each second block (i.e. each tile) in the image may include the bitstream description information of the second block, but does not include the encoded data of the second block. Figure 4B (2) The substream of each second block (i.e. each tile) in the image may include the encoded data of the second block and the bitstream description information of the second block.

[0216] After that, Figure 5B (1) Sub-stream of the first block and Figure 5B (2) The substream concatenation of the second block in the code stream results in the following bitstream: Figure 5B As shown in (3).

[0217] After the encoding device encodes according to the above encoding process 500, the decoding side can parse the bitstream description information of the second block from the substream of the second block. When the decoding side determines that the second block adopts the skip mode based on the bitstream description information of the second block, it can copy the filtered reconstruction block of the second block in the previous frame's reconstructed image that corresponds to the position of the second block in the current frame, and use it as the filtered reconstruction block corresponding to that second block in the current frame. Specifically, for each coding unit of the second block in the current frame, the filtered reconstruction block of the coding unit in the second block of the previous frame's reconstructed image that corresponds to the position of that coding unit in the second block of the current frame can be copied and used as the filtered reconstruction block corresponding to that coding unit in the second block of the current frame.

[0218] For example, the preprocessing module 301 can also classify multiple first blocks into multiple categories based on the motion state of each first block. For instance, the motion state can be represented by motion speed (e.g., the preprocessing module 301 can perform motion detection on the foreground in the image to determine the motion speed of the foreground). Blocks with motion speeds greater than a first threshold among the multiple first blocks can be identified as first-class blocks; blocks with motion speeds greater than a second threshold but less than the first threshold among the multiple first blocks can be identified as second-class blocks; blocks with motion speeds greater than a third threshold but less than the second threshold among the multiple first blocks can be identified as third-class blocks, and so on. Wherein, the first threshold is greater than the second threshold, the second threshold is greater than the third threshold, and so on.

[0219] Furthermore, the prediction processing module 301 can also determine the encoding frame rate corresponding to each type of block in the first block; wherein, the encoding frame rate corresponding to the first type of block can be the first encoding frame rate, the encoding frame rate corresponding to the second type of block can be the third encoding frame rate, the encoding frame rate corresponding to the third type of block can be the fourth encoding frame rate, the encoding frame rate corresponding to the fourth type of block can be the fifth encoding frame rate, and so on. Afterwards, the preprocessing module 301 can input each type of block in the first block into the encoder 302 according to its corresponding encoding frame rate. The encoding frame rate corresponding to the first type of block is greater than the encoding frame rates corresponding to other types of blocks (this application does not restrict the relationship between the encoding frame rates corresponding to other types of blocks in the first block and the second encoding frame rate; optionally, the encoding frame rates corresponding to other types of blocks in the first block are greater than or equal to the second encoding frame rate).

[0220] The following explanation uses the division of the first block into a first-class block and a second-class block as an example.

[0221] Figure 6A This is a schematic diagram of another encoding process 600 according to an embodiment of this application. In the encoding process 600, S601 to S602 are executed by the preprocessing module 301, and S603 to S606 are executed by the encoder 302.

[0222] S601, acquire video stream, the video stream includes multiple frames of images.

[0223] For example, S601 can be described with reference to the above description of S401, and will not be repeated here.

[0224] S602, the image in the video stream is divided into a first block and a second block. The first block contains the foreground of the image, and the second block contains only the background of the image. The first block includes two types of blocks: the first type of block and the second type of block.

[0225] For example, the process of dividing the first block and the second block in S602 can be referred to the description of S402 above, and will not be repeated here.

[0226] For example, in S602, the motion state of the first type of block satisfies a preset condition, while the motion state of the second type of block does not satisfy the preset condition. The motion state can be represented by motion speed, and the preset condition can be that the motion speed is greater than a first threshold. That is, in S602, the first type of block is the block in the first block whose motion speed is greater than the first threshold, and the second type of block is the block in the first block whose motion speed is less than the first threshold.

[0227] It should be understood that the motion state can also be represented by other parameters, or the motion rate can be represented by a combination of other parameters, and this application does not limit this.

[0228] S603, the first type of block in the first image is encoded to obtain a sub-stream of the first type of block in the first image, wherein the first image is the image to be encoded determined from the video stream according to the first encoding frame rate.

[0229] For example, S603 can be described with reference to the above description of S403, and will not be repeated here.

[0230] For example, the first encoding frame rate in S603 can refer to the frame rate at which the preprocessing module 301 inputs the first type of block to the encoder 302. From another perspective, the first encoding frame rate can be regarded as the instantaneous frame rate at which the encoder encodes the first type of block (it should be noted that the encoding frame rate of the encoder is not adjusted during this process).

[0231] S604, Encode the second block in the second image to obtain a sub-stream of the second block in the second image.

[0232] For example, S604 can be described with reference to the above description of S404, and will not be repeated here.

[0233] S605, the second type of block in the fourth image is encoded to obtain a sub-stream of the second type of block in the fourth image. The fourth image is the image to be encoded determined from the video stream according to the third encoding frame rate, which is less than the first encoding frame rate.

[0234] For example, the preprocessing module 301 can input the second type of block to the encoder 302 according to the third encoding frame rate, and the encoder 302 can encode the second block.

[0235] For example, the third encoding frame rate can refer to the frame rate at which the preprocessing module 301 inputs the second type of block to the encoder 302, and the third encoding frame rate is less than the first encoding frame rate; for example, the third encoding frame rate is 30 FPS, 20 FPS, 10 FPS, etc. From another perspective, the third encoding frame rate can be regarded as the instantaneous frame rate of the encoder encoding the second type of block (it should be noted that the actual encoding frame rate of the encoder is not adjusted in this process). For example, the third encoding frame rate can be preset by the system or the user, or it can be determined according to the current network status; this application does not limit this.

[0236] For example, the third encoding frame rate can be greater than, less than or equal to the maximum encoding frame rate supported by the encoder.

[0237] For example, the first encoding frame rate may be an integer multiple of the third encoding frame rate.

[0238] For example, this application does not limit the relationship between the third encoding frame rate and the second encoding frame rate.

[0239] For example, when the first encoding frame rate is the same as the original acquisition frame rate, the preprocessing module 301 can determine the image to be encoded (hereinafter referred to as the fourth image) from the video stream or target video stream acquired in S601 according to the third encoding frame rate; wherein, the fourth image consists of multiple frames. When the first encoding frame rate is different from the original acquisition frame rate, the preprocessing module 301 can determine the fourth image from the target video stream according to the third encoding frame rate.

[0240] Assuming the third encoding frame rate is 30 FPS, continue referring to the above. Figure 4D The image represented by the square-filled rectangular blocks in the first image (including rectangular blocks with solid borders and square-filled blocks and rectangular blocks with dashed borders and square-filled blocks) is determined as the fourth image. The fourth image includes: F0, F4, F8, F12, F16, F20, ...

[0241] Subsequently, the preprocessing module 301 can sequentially input the second-type blocks in each frame of the fourth image to the encoder 302 according to the frame number or timestamp of each frame. When a frame of the fourth image includes multiple second-type blocks, the multiple second-type blocks of the fourth image can be sequentially input to the encoder 302 according to the index order of these multiple second-type blocks; or they can be sequentially input to the encoder 302 according to the position coordinates of these multiple second-type blocks, in a left-to-right and top-to-bottom order. The encoder 302 can sequentially encode the input second-type blocks to obtain a substream of the second-type blocks in each frame of the fourth image.

[0242] For example, encoder 302 can encode each second type block according to the encoding process described above; it will not be repeated here. Specifically, intra-frame prediction is performed for the second type blocks in the fourth image that is an I-frame, and inter-frame prediction is performed for the second type blocks in the fourth image that is a P-frame or B-frame.

[0243] It should be understood that the process of determining the fourth image described above is essentially determining the fourth image from multiple frames of the first image; that is, there exists an image in the video stream or target video stream that is identified as both the first image and the fourth image; for example, Figure 4D F0, F4, F8, F12, F16, and F20 are identified as the first image and also as the fourth image. Based on the description of encoding process 400, it is known that there exists a certain image in the video stream or target video stream that is identified as the first image and also as the second image; therefore, it can be deduced that there exists a certain image in the video stream or target video stream that is identified as the first image, also as the fourth image, and also as the second image; for example, Figure 4D F0 and F20 were identified as the first image, the fourth image, and the second image.

[0244] For ease of explanation later, images in the video stream that are simultaneously identified as the first image, the second image, and the fourth image are referred to as Class I images (e.g., ...). Figure 4D (F0 and F20 in the text); Images in the video stream that are identified as the first and second images, but not as the fourth image, are called second-class images (e.g., F0 and F20 in the text). Figure 4D (F10 in the video stream); Images in the video stream that are identified as the first image but not as the second or fourth image are called third-class images (e.g., F10 in the video stream). Figure 4D (F2, F6, F10, F14, F18, and F22 in the text); Images in the video stream that are identified as the first and fourth images, but not as the second image, are called fourth-class images (e.g., F2, F6, F10, F14, F18, and F22 in the text); Figure 4D (F4, F8, F12, and F16 in the series).

[0245] In other words, for the first type of image, the preprocessing module 301 needs to input the first type block, the second type block and the second block in the first type of image into the encoder 302, and the encoder 302 executes S603 to S605 (during the execution of S603, the first image refers to the first type of image in the first image; during the execution of S605, the second image refers to the first type of image in the second image; during the execution of S604, the fourth image refers to the first type of image in the fourth image).

[0246] For the second type of image, the preprocessing module 301 needs to input the first type block and the second type block in the second type of image into the encoder 302, and the encoder 302 executes S603 and S604 (during the execution of S603, the first image refers to the second type of image in the first image; during the execution of S604, the second image refers to the second type of image in the second image).

[0247] For the third type of image, the preprocessing module 301 needs to input the first type of block in the third type of image into the encoder 302, and the encoder 302 executes S603 (during the execution of S603, the first image refers to the third type of image in the first image).

[0248] For the fourth type of image, the preprocessing module 301 needs to input the first type block and the second type block in the fourth type of image into the encoder 302, and the encoder 302 executes S603 and S605 (during the execution of S603, the first image refers to the fourth type of image in the first image; during the execution of S605, the fourth image refers to the fourth type of image in the fourth image).

[0249] For example, refer to again Figure 4D The preprocessing module 301 can input the first type block, the second type block, and the second block of the two frames of images F0 and F20 into the encoder 302. The preprocessing module 301 can input the first type block and the second block of the one frame of image F10 into the encoder 302. The preprocessing module 301 can input the first type block of the six frames of images F2, F6, F10, F14, F18, and F22 into the encoder 302. The preprocessing module 301 can input the first type block and the second type block of the four frames of images F4, F8, F12, and F16 into the encoder 302.

[0250] S606, stitch together the sub-streams of the first type of block in the first image, the sub-streams of the second block in the second image, and the sub-streams of the second type of block in the fourth image to obtain a bitstream.

[0251] For example, the sub-streams of the first type of block in the first image and the sub-streams of the second type of block in the fourth image can be combined to form the sub-stream of the first block; the sub-streams of the second block in the second image can be combined to form the sub-stream of the second block. Then, the encoder 302 can concatenate the sub-streams of the first block and the sub-streams of the second block to obtain a bitstream.

[0252] For example, suppose Figure 4B Image (1) is a first-class image. Figure 4B (1) In the image, the first type of the six first blocks includes Tile 6, Tile 7, Tile 10, and Tile 11, and the second type of blocks includes Tile 14 and Tile 15; and the Figure 4B The encoding order of each block in image (1) is Tile 1→Tile 2→Tile 3→……→Tile 15→Tile 16. And assuming image 4B(2) is a second-class image, Figure 4B (2) In the image, the first type of blocks in the six first blocks include Tile 5, Tile 6, Tile 9 and Tile 10, and Figure 4B (2) The encoding order of the first type of block and the second type of block in the image is Tile 1→Tile 2→Tile 3→Tile 4→Tile 5→Tile 6→Tile 7→Tile 8→Tile 9→Tile 10→Tile 11→Tile 12→Tile 15→Tile 16; and Figure 4B (1) is Figure 4D F0 in Figure 4B (2) Figure 4D Press F10 in the settings to complete the process. Figure 4B (1) image and Figure 4B After encoding the image in (2), the resulting substream of the first block can be as follows: Figure 6B As shown in (1), the sub-stream of the second block can be as follows Figure 6B As shown in (2).

[0253] Figure 6B (1) The sub-stream of the first block includes Figure 4B (1) The image contains sub-streams of 4 tiles (first-class blocks) and 2 tiles (first-class blocks), and Figure 4B (2) The image contains four sub-streams of tiles (first-class blocks); each first-class block sub-stream is a dark gray rectangle, and each second-class block sub-stream is a light gray rectangle. Each first-class block sub-stream may include the encoded data and bitstream description information of the first-class block; and each second-class block sub-stream may include the encoded data and bitstream description information of the second-class block.

[0254] Figure 6B (2) The sub-stream of the second block includes Figure 4B (1) The sub-flows of 10 tiles (second block) in the image and Figure 4B (2) The image contains 10 sub-streams of Tile (second block); each Tile (second block) sub-stream is a white rectangle, and each sub-stream of the second block (i.e. each Tile) may include the encoded data of the second block and the bitstream description information of the second block, etc.

[0255] After that, Figure 6B (1) Sub-stream of the first block and Figure 6B (2) The code stream is obtained by splicing the sub-streams of the second block.

[0256] This allows for a further increase in the encoding frame rate of the first type of block. For example, in a 4K image, the proportion of the first type of block is 4 / 16, the proportion of the second type of block is 2 / 16, and the proportion of the third type of block is 10 / 16. Assuming the first encoding frame rate is 80 FPS, the second encoding frame rate is 10 FPS, and the third encoding frame rate is 20 FPS; the actual encoding computing power of the encoder (or the actual encoding frame rate of the encoder) is calculated as: 80 FPS * (4 / 16) + 20 FPS * (2 / 16) + 10 FPS * (10 / 16) = 4K@28.75 FPS. That is, providing encoding hardware (or software) at 4K@30 FPS can overcome the limitation and achieve an encoding effect of 4K@80 FPS (for the first type of block, an encoding effect of 4K@80 FPS can be achieved).

[0257] Similarly, the same method described above, which allows the decoding side to reconstruct the second block in the partially reconstructed image, can be used to enable the decoding side to obtain the reconstructed image of the second type of block in the partially reconstructed image; the following encoding process 700 can be referred to.

[0258] Figure 7 This is a schematic diagram of another encoding process 700 according to an embodiment of this application.

[0259] S701, acquire video stream, the video stream includes multiple frames of images.

[0260] S702, the video stream is divided into a first block and a second block. The first block contains the foreground of the image, and the second block contains only the background of the image. There are one or more first blocks and second blocks.

[0261] S703, the first type of block in the first image is encoded to obtain a sub-stream of the first type of block in the first image, wherein the first image is the image to be encoded determined from the video stream according to the first encoding frame rate.

[0262] For example, S701 to S703 can be referred to the description of S601 to S603 above, and will not be repeated here.

[0263] S704, the second block in the second image is encoded to obtain a sub-stream of the second block in the second image. The second image is the image to be encoded determined from the video stream according to the second encoding frame rate, and the first encoding frame rate is greater than the second encoding frame rate.

[0264] S705, the second block in the third image is encoded using a skip mode to obtain a sub-stream of the second block in the third image. The third image is the image in the first image other than the one that is determined to be the second image.

[0265] For example, S704 to S705 can be referred to the description of S504 to S505 above, and will not be repeated here.

[0266] S706, the second type of block in the fourth image is encoded to obtain a sub-stream of the second type of block in the fourth image. The fourth image is the image to be encoded determined from the video stream according to the third encoding frame rate, which is less than the first encoding frame rate.

[0267] For example, S706 can be described with reference to the above description of S605, and will not be repeated here.

[0268] S707, the skip mode is used to encode the second type of block in the fifth image to obtain the sub-stream of the second type of block in the fifth image. The fifth image is the first image in the video stream located between two adjacent fifth images.

[0269] It should be understood that the process of determining the fourth image described above is essentially determining the fourth image from multiple frames of the first image; that is, there exists an image in the video stream or target video stream that is determined to be both the first image and the fourth image. For ease of subsequent description, the image in the first image other than the one determined to be the fourth image can be referred to as the fifth image; such as... Figure 4D The rectangles in the first image that are not filled with squares (including rectangles with solid borders and no squares filled, and rectangles with dashed borders and no squares filled) can be called the fifth image. The fifth images include: F2, F6, F10, F14, F18, F22, ...

[0270] For example, S707 can be described with reference to the above description of S505, and will not be repeated here.

[0271] Refer again Figure 4DFor example, for the fifth image F2, since F2 is also determined to be the first image and the third image, the preprocessing module 301 can input the first type block, the second type block and the second block of F2 into the encoder 302; the encoder 302 encodes the residual block and motion vector corresponding to the first type block in F2 to generate the bitstream of the first type block in F2 (including the bitstream description information of the first type block); and the encoder 302 does not encode the residual block and motion vector corresponding to the second type block, but uses the bitstream description information of the second type block to generate the bitstream of the second type block in F2; and the encoder 302 does not encode the residual block and motion vector corresponding to the second block, but uses the bitstream description information of the second block to generate the bitstream of the second block in F2.

[0272] Refer again Figure 4D For example, for the fifth image F10, since F2 is also determined to be the first image and the second image, the preprocessing module 301 can input the first type block, the second type block and the second block of F10 into the encoder 302; the encoder 302 encodes the residual block and motion vector corresponding to the first type block in F2 to generate the bitstream of the first type block in F2 (including the bitstream description information of the first type block); and the encoder 302 does not encode the residual block and motion vector corresponding to the second type block, but uses the bitstream description information of the second type block to generate the bitstream of the second type block in F2; and the encoder 302 encodes the residual block and motion vector corresponding to the second block to generate the bitstream of the second block in F2 (including the bitstream description information of the second block).

[0273] In this way, the encoding frame rate of the second type of block can be padded to be the same as that of the first type of block.

[0274] It should be noted that the preprocessing module 301 does not need to input the second type of block from the fifth image into the encoder 302. Furthermore, the encoder 302 performs the encoding operations of S705 and S707 in a similar manner.

[0275] For encoder 302, the computational power required to encode the second type of block in the fifth image using the skip mode is very small and can be ignored; that is, by using this method, the encoding frame rate of the second type of block is made up to be the same as that of the first type of block, which basically does not consume the computational power of the encoder.

[0276] It should be noted that, since users tend to focus their attention on the foreground with large motion displacement while ignoring the foreground with small motion displacement during video viewing, although the encoding method of this application will make the second type of blocks of the reconstructed images of adjacent frames the same, it can be ignored by users, or is easily overlooked by users.

[0277] It should be noted that for each first image, S703 can be executed; simultaneously, it can be determined whether the first image is identified as a second image or a third image; when the first image is identified as a second image, S704 can be executed; when the first image is identified as a third image, S705 can be executed. That is, for a single first image, S704, and either S704 or S705, can be executed. Furthermore, it can be determined whether the first image is identified as a fourth image or a fifth image; when the first image is identified as a fourth image, S706 can be executed; when the first image is identified as a fifth image, S707 can be executed. That is, for a single first image, S704, and either S706 or S707, can be executed.

[0278] S708, the sub-streams of the first block in the first image, the second block in the second image, the third block in the fourth image, and the third block in the fifth image are spliced ​​together to obtain a bitstream.

[0279] For example, S708 can be described with reference to the above descriptions of S506 and S606, and will not be repeated here.

[0280] For example, this application can optimize the boundary quality of blocks; for instance, by adjusting the quantizer parameter (QP) of the block editing to enhance the boundary quality of each block.

[0281] For example, the first block may include boundary coding units and other coding units (other coding units may refer to coding units other than boundary coding units included in the first block); for the first block, the encoder 302 of this application may quantize the residual values ​​of other coding units of the first block in the first image according to a first quantization parameter, and quantize the residual values ​​of boundary coding units of the first block in the first image according to a second quantization parameter, wherein the second quantization parameter is smaller than the first quantization parameter (the smaller the quantization parameter, the smaller the error between the quantization result and the actual result, and the higher the reconstruction quality); then, entropy coding is performed on the quantized residual values ​​of the first block in the first image to obtain the sub-stream of the first block in the first image.

[0282] Combination Figure 3CFor example, the transformation result output by the transformation module may include the transformation result of the residual block corresponding to the boundary coding unit and the transformation result of the residual block corresponding to other coding units. Then, the quantization module may quantize the transformation result of the residual block corresponding to the boundary coding unit according to a first quantization parameter to obtain the quantized result of the residual block corresponding to the boundary coding unit (also referred to as the quantized residual block corresponding to the boundary coding unit). The quantization module may also quantize the transformation result of the residual block corresponding to other coding units according to a second quantization parameter to obtain the quantized result of the residual block corresponding to other coding units (also referred to as the quantized residual block corresponding to other coding units). Subsequently, the entropy coding module may perform entropy coding on the quantized residual blocks corresponding to other coding units and the quantized residual blocks corresponding to the boundary coding unit to obtain the substream of the first block.

[0283] For example, the preprocessing module 301 can set the quantization parameters of the encoder 302 (e.g., the preprocessing module 301 can control the parameter control module in the encoder 302 to set the quantization parameters), that is, the preprocessing module 301 can set the quantization parameters of the encoder 302 for the boundary coding units and other coding units of the first block respectively. Of course, the preprocessing module 301 can also set the adjustment strategy of the quantization parameters of the encoder 302 for the boundary coding units and other coding units of the first block, such as dynamically adjusting the quantization parameters of the boundary coding units and other coding units of the first block using QP_Map.

[0284] It should be understood that, for the second block, encoder 302 can enhance the quality of the boundary coding units of the second block in the manner described above, which will not be repeated here.

[0285] Correspondingly, on the decoding side, the decoder can smooth the boundary coding units of two adjacent first blocks, smooth the boundary coding units of two adjacent second blocks, and smooth the boundary coding units of the first block and the second block, so as to make the boundary transition between two adjacent blocks smoother and more fluid, thereby improving the visual smoothness of the reconstructed image.

[0286] The encoder 302 described in the above embodiments is implemented by proprietary encoding hardware. When the encoder 302 is implemented by software (such as a CPU), the encoding process can be as follows:

[0287] Figure 8 This is a schematic diagram of the frame of another encoder according to an embodiment of this application.

[0288] Reference Figure 8The encoder 302, implemented by a CPU, may include an encoding module and a concatenation module. The preprocessing module 301 may send an encoding request to the encoding module; this encoding request may include blocks (including a first block and / or a second block) and block control information (or setting information) (optional). The block control information may include boundary information of the block (including the first block and / or the second block), boundary quantization parameters of the block (including the first block and / or the second block) (or boundary quantization parameter adjustment strategies), etc. Subsequently, the encoding module can encode the blocks in the encoding request according to the block control information to obtain sub-streams of the blocks and send these sub-streams to the concatenation module, which then concatenates the sub-streams of each block to obtain a bitstream.

[0289] Figure 9 This is a schematic diagram of another encoding device 900 according to an embodiment of this application. This encoding device 900 can be used to execute the methods of the foregoing embodiments; therefore, the beneficial effects it can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0290] Acquisition module 901 is used to acquire a video stream, which includes multiple frames of images;

[0291] The segmentation module 902 is used to divide the images included in the video stream into a first block and a second block. The first block contains the foreground of the image, and the second block contains only the background of the image. There are one or more first blocks and second blocks.

[0292] The encoding module 903 is used to encode a first block in a first image to obtain a sub-stream of the first block in the first image, wherein the first image is an image to be encoded determined from the video stream according to a first encoding frame rate; and to encode a second block in a second image to obtain a sub-stream of the second block in the second image, wherein the second image is an image to be encoded determined from the video stream according to a second encoding frame rate; the sub-stream of the first block in the first image and the sub-stream of the second block in the second image constitute a bitstream, wherein the first encoding frame rate is greater than the second encoding frame rate and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder.

[0293] It should be noted that the preprocessing module 301 in the encoding device 300 may include the acquisition module 901 and the block segmentation module 902 in the encoding device 900. When the encoding module 903 in the encoding device 900 is implemented in hardware, the encoding module 903 in the encoding device 900 is the encoder in the encoding device 300; when the encoding module 903 in the encoding device 900 is implemented in software, the encoding module 903 may include... Figure 8 The encoding unit and the splicing unit.

[0294] For example, both the first block and the second block are tiles; or, both the first block and the second block are slices.

[0295] For example, the encoding module 903 is further configured to encode the second block in the third image using a skip mode to obtain a sub-stream of the second block in the third image, wherein the third image is an image in the first image other than the one determined to be the second image; the sub-stream of the first block in the first image, the sub-stream of the second block in the second image, and the sub-stream of the second block in the third image constitute a bitstream.

[0296] For example, the first block includes a first type of block and a second type of block. The motion state of the first type of block meets a preset condition, while the motion state of the second type of block does not meet the preset condition. The encoding module 903 is used to encode the first type of block in the first image to obtain a sub-stream of the first type of block in the first image; and to encode the second type of block in the fourth image to obtain a sub-stream of the second type of block in the fourth image. The fourth image is an image to be encoded determined from the video stream according to a third encoding frame rate, which is less than the first encoding frame rate. The sub-stream of the first type of block in the first image, the sub-stream of the second type of block in the second image, and the sub-stream of the second type of block in the fourth image constitute a bitstream.

[0297] For example, the encoding module 903 is further configured to encode the second type of blocks in the fifth image using a skip mode to obtain a sub-stream of the second type of blocks in the fifth image, wherein the fifth image is an image in the first image other than the one determined to be the fourth image; the sub-stream of the first type of blocks in the first image, the sub-stream of the second type of blocks in the second image, the sub-stream of the second type of blocks in the fourth image and the sub-stream of the second type of blocks in the fifth image constitute a bitstream.

[0298] For example, the encoding module 903 is used to predict a first block in a first image to determine the predicted value of the first block in the first image; determine the residual value of the first block in the first image based on the original value of the first block in the first image and the predicted value of the first block in the first image; and entropy encode the residual value of the first block in the first image to obtain a sub-stream of the first block in the first image.

[0299] For example, the encoding module 903 is used to predict a second block in a second image to determine a predicted value of the second block in the second image; determine a residual value of the second block in the second image based on the original value of the second block in the second image and the predicted value of the second block in the second image; and entropy encode the residual value of the second block in the second image to obtain a sub-stream of the second block in the second image.

[0300] For example, the encoding module 903 is further configured to quantize the residual values ​​of other encoding units of the first block in the first image according to the first quantization parameter, and to quantize the residual values ​​of the boundary encoding units of the first block in the first image according to the second quantization parameter, wherein the second quantization parameter is less than the first quantization parameter; and to entropy encode the quantized residual values ​​of the first block in the first image to obtain a sub-stream of the first block in the first image.

[0301] In one example, Figure 10 The schematic block diagram illustrating an embodiment of the present application shows an apparatus 1000. The apparatus 1000 may include a processor 1001 and a transceiver 1002, and optionally, a memory 1003.

[0302] The various components of device 1000 are coupled together via bus 1004, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus 1004 in the figure.

[0303] Optionally, the memory 1003 can be used to store instructions from the foregoing method embodiments. The processor 1001 can be used to execute the instructions in the memory 1003, control the transceiver 1002 to receive signals, and control the transceiver 1002 to transmit signals.

[0304] The device 1000 may be an electronic device or a chip of an electronic device as described in the above method embodiments. The electronic device may be a terminal device or a server.

[0305] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0306] This application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the above-described related method are executed to achieve the method in the above embodiments. The interface circuit is a transceiver 1002.

[0307] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the methods described in the above embodiments. Exemplarily, the computer-readable storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0308] This application also provides a computer program product comprising computer instructions that, when executed by a computer or processor, cause the computer to perform the aforementioned steps to implement the methods described in the above embodiments. Exemplarily, the computer program product may be stored in random access memory (RAM), flash memory, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, read-only optical discs (CD-ROMs), or any other form of storage medium known in the art.

[0309] This application also provides a method for storing a bitstream, the method comprising: receiving a bitstream generated according to the method of the above embodiments; and storing the bitstream.

[0310] This application also provides a method for transmitting a bitstream, the method comprising: storing the bitstream generated according to the method of the above embodiments; and transmitting the bitstream. In this way, the bitstream can be transmitted to a decoding side or a distribution server.

[0311] This application also provides a method for transmitting a bitstream, the method comprising: receiving a bitstream generated according to the method of the above embodiments; and sending the bitstream. In this way, the bitstream can be sent to the decoding side.

[0312] This application also provides a system for storing a bitstream, the system comprising: a receiving module and a storage module; the receiving module is used to receive a bitstream generated according to the method of the above embodiments; the storage module is used to store the bitstream.

[0313] This application also provides a system for transmitting a bitstream, the system comprising: a transceiver module and a storage module; the storage module is used to store the bitstream generated according to the method of the above embodiments; the transceiver module is used to transmit the bitstream. Thus, the bitstream can be transmitted to a decoding side or a distribution server.

[0314] This application also provides a system for transmitting a bitstream, the system comprising: a transceiver module for receiving a bitstream generated according to the method of the above embodiments; the transceiver module is further configured to transmit the bitstream. Thus, the bitstream can be transmitted to the decoding side.

[0315] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0316] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0317] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0318] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0319] Any content in the various embodiments of this application, as well as any content in the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0320] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An encoding method, characterized in that, The method includes: Acquire a video stream, the video stream comprising multiple frames of images; The video stream includes images divided into a first block and a second block. The first block contains the foreground of the image, and the second block contains only the background of the image. Both the first block and the second block are one or more. Encode a first block in a first image to obtain a sub-stream of the first block in the first image, wherein the first image is an image to be encoded determined from the video stream according to a first encoding frame rate; The second block in the second image is encoded to obtain a sub-stream of the second block in the second image, wherein the second image is an image to be encoded determined from the video stream according to the second encoding frame rate; The sub-stream of the first block in the first image and the sub-stream of the second block in the second image constitute a bitstream. The first encoding frame rate is greater than the second encoding frame rate, and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder.

2. The method according to claim 1, characterized in that, Both the first block and the second block are tiles; or, both the first block and the second block are slices.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The second block in the third image is encoded using a skip mode to obtain a sub-stream of the second block in the third image, wherein the third image is the first image other than the one identified as the second image; The bitstream is composed of the sub-stream of the first block in the first image, the sub-stream of the second block in the second image, and the sub-stream of the second block in the third image.

4. The method according to claim 1 or 2, characterized in that, The first block includes a first type of block and a second type of block. The motion state of the first type of block meets the preset conditions, while the motion state of the second type of block does not meet the preset conditions. Encoding the first block in the first image to obtain a sub-stream of the first block in the first image includes: Encode the first type of block in the first image to obtain a sub-stream of the first type of block in the first image; The method further includes: The second type of block in the fourth image is encoded to obtain a sub-stream of the second type of block in the fourth image, wherein the fourth image is an image to be encoded determined from the video stream according to a third encoding frame rate, the third encoding frame rate being less than the first encoding frame rate; The bitstream consists of a substream of the first type of block in the first image, a substream of the second type of block in the second image, and a substream of the second type of block in the fourth image.

5. The method according to claim 4, characterized in that, The method further includes: The second type of block in the fifth image is encoded using a skip mode to obtain a sub-stream of the second type of block in the fifth image, wherein the fifth image is the first image other than the one identified as the fourth image; The bitstream consists of a substream of the first type of block in the first image, a substream of the second type of block in the second image, a substream of the second type of block in the fourth image, and a substream of the second type of block in the fifth image.

6. The method according to any one of claims 1 to 3, characterized in that, Encoding the first block in the first image to obtain a sub-stream of the first block in the first image includes: A prediction is made for the first block in the first image to determine the predicted value of the first block in the first image; The residual value of the first block in the first image is determined based on the original value of the first block in the first image and the predicted value of the first block in the first image. Entropy encoding is performed on the residual value of the first block in the first image to obtain a sub-stream of the first block in the first image.

7. The method according to any one of claims 1 to 6, characterized in that, Encoding the second block in the second image to obtain a sub-stream of the second block in the second image includes: A prediction is made for the second block in the second image to determine the predicted value of the second block in the second image; Based on the original value of the second block in the second image and the predicted value of the second block in the second image, determine the residual value of the second block in the second image; Entropy encoding is performed on the residual value of the second block in the second image to obtain the sub-stream of the second block in the second image.

8. The method according to claim 6, characterized in that, The first block in the first image includes boundary coding units and other coding units, and the method further includes: The residual values ​​of other coding units in the first block of the first image are quantized according to the first quantization parameter, and the residual values ​​of the boundary coding units in the first block of the first image are quantized according to the second quantization parameter, wherein the second quantization parameter is smaller than the first quantization parameter. The step of entropy encoding the residual value of the first block in the first image to obtain a sub-stream of the first block in the first image includes: Entropy encoding is performed on the quantized residual value of the first block in the first image to obtain a sub-stream of the first block in the first image.

9. An encoding device, characterized in that, The encoding device includes a preprocessing module and an encoder, wherein: The preprocessing module is used to acquire a video stream, the video stream comprising multiple frames of images; divide the images in the video stream into a first block and a second block, the first block containing the foreground of the image, and the second block containing only the background of the image, wherein there are one or more first blocks and second blocks; input the first block of the first image and the second block of the second image to the encoder, wherein the first image is an image to be encoded determined from the video stream according to a first encoding frame rate, and the second image is an image to be encoded determined from the video stream according to a second encoding frame rate; the first encoding frame rate is greater than the second encoding frame rate, and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder; The encoder is used to encode a first block in the first image to obtain a sub-stream of the first block in the first image; and to encode a second block in the second image to obtain a sub-stream of the second block in the second image; the sub-stream of the first block in the first image and the sub-stream of the second block in the second image constitute a bitstream.

10. The apparatus according to claim 9, characterized in that, Both the first block and the second block are tiles; or, both the first block and the second block are slices.

11. The apparatus according to claim 9 or 10, characterized in that, The encoder is further configured to encode the second block in the third image using a skip mode to obtain a sub-stream of the second block in the third image, wherein the third image is an image in the first image other than the one determined to be the second image; The bitstream is composed of the sub-stream of the first block in the first image, the sub-stream of the second block in the second image, and the sub-stream of the second block in the third image.

12. The apparatus according to claim 9 or 10, characterized in that, The first block includes a first type of block and a second type of block. The motion state of the first type of block meets the preset conditions, while the motion state of the second type of block does not meet the preset conditions. The encoder is used to encode a first type of block in the first image to obtain a sub-stream of the first type of block in the first image; The second type of block in the fourth image is encoded to obtain a sub-stream of the second type of block in the fourth image. The fourth image is an image to be encoded determined from the video stream according to a third encoding frame rate, which is less than the first encoding frame rate. The sub-stream of the first type of block in the first image, the sub-stream of the second block in the second image, and the sub-stream of the second type of block in the fourth image constitute the bitstream.

13. The apparatus according to claim 12, characterized in that, The encoder is further configured to encode the second type of blocks in the fifth image using a skip mode to obtain a sub-stream of the second type of blocks in the fifth image, wherein the fifth image is an image in the first image other than the fourth image; the sub-stream of the first type of blocks in the first image, the sub-stream of the second type of blocks in the second image, the sub-stream of the second type of blocks in the fourth image, and the sub-stream of the second type of blocks in the fifth image constitute the bitstream.

14. An encoding device, characterized in that, The device includes: The acquisition module is used to acquire a video stream, which includes multiple frames of images; The segmentation module is used to divide the images included in the video stream into a first block and a second block. The first block contains the foreground of the image, and the second block contains only the background of the image. Both the first block and the second block are one or more. An encoding module is used to encode a first block in a first image to obtain a sub-stream of the first block in the first image, wherein the first image is an image to be encoded determined from the video stream according to a first encoding frame rate; and to encode a second block in a second image to obtain a sub-stream of the second block in the second image, wherein the second image is an image to be encoded determined from the video stream according to a second encoding frame rate; the sub-stream of the first block in the first image and the sub-stream of the second block in the second image constitute a bitstream, wherein the first encoding frame rate is greater than the second encoding frame rate, and the second encoding frame rate is less than the maximum encoding frame rate supported by the encoder.

15. A bitstream, characterized in that, The bitstream is generated by the method according to any one of claims 1 to 8.

16. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is coupled to the processor; The memory stores program instructions that, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1 to 8.

17. A chip, characterized in that, It includes one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 1 to 8 are performed.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on a computer or processor, causes the computer or processor to perform the method as described in any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a computer or processor, cause the steps of the method as described in any one of claims 1 to 8 to be performed.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a bitstream generated by the method according to any one of claims 1 to 8.

21. A method for storing a bitstream, characterized in that, The method includes: The bitstream is received, wherein the bitstream is generated by the method according to any one of claims 1 to 8; And to store the bitstream.

22. A method for transmitting a code stream, characterized in that, The method includes: The bitstream is stored, and the bitstream is generated by the method according to any one of claims 1 to 8; And the transmission of the bitstream.

23. A method for transmitting a code stream, characterized in that, The method includes: The bitstream is received, wherein the bitstream is generated by the method according to any one of claims 1 to 8; And send the bitstream.

24. A system for storing bitstreams, characterized in that, The system includes: a receiving module and a storage module; The receiving module is used to receive a bitstream, which is generated by the method according to any one of claims 1 to 8; the storage module is used to store the bitstream.

25. A system for transmitting a code stream, characterized in that, The system includes: a transceiver module and a storage module; The storage module is used to store the bitstream, which is generated by the method according to any one of claims 1 to 8; the transceiver module is used to transmit the bitstream.

26. A system for transmitting a code stream, characterized in that, The system includes: a transceiver module, The transceiver module is used to receive a bitstream, which is generated by the method according to any one of claims 1 to 8; the transceiver module is also used to send the bitstream.

Images