Video encoding method, video decoding method, and related apparatus

Abstract

The application discloses a video coding method, a video decoding method and related devices. The video coding method comprises: coding a video to obtain coded data; adding a preset syntax in the coded data to obtain a video bitstream, different values of the preset syntax being used to indicate a display mode of the knowledge image, the display mode comprising display in a single stream and non-display in a single stream. The application can reduce coding delay.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding technology, and in particular to a video encoding method, a video decoding method, and related apparatus. Background Technology

[0002] Video image data is relatively large, so it is usually necessary to compress the video pixel data (RGB, YUV, etc.). The compressed data is called a video stream, which is transmitted to the user's end via wired or wireless network for decoding and viewing. The entire video encoding process includes prediction, transformation, quantization, and encoding.

[0003] In video encoding and decoding, to improve compression ratio and reduce the number of codewords to be transmitted, the encoder does not directly encode and transmit pixel values. Instead, it uses intra-frame or inter-frame prediction modes to predict the pixel values ​​of the current block using reconstructed pixels from the coded blocks of the current frame or a reference frame. The pixel value predicted using a certain prediction mode is called the predicted pixel value, and the difference between the predicted pixel value and the original pixel value is called the residual. The encoder only needs to encode a certain prediction mode and the residual generated when using that prediction mode, and the decoder can decode the corresponding pixel value based on this bitstream information. This greatly reduces the number of codewords required for encoding.

[0004] However, during the long-term research and development process, the inventors of this application discovered that current video coding methods still have some shortcomings; for example, there is a relatively long coding delay. Summary of the Invention

[0005] This application provides a video encoding method, a video decoding method, and related apparatus, which can reduce encoding latency.

[0006] To achieve the above objectives, this application provides a video coding method, which includes:

[0007] The original image content of the knowledge image is encoded only once to obtain the encoded data of the knowledge image, which is used for display.

[0008] To achieve the above objectives, this application provides a video coding method, which includes:

[0009] The video is encoded to obtain encoded data;

[0010] A preset syntax is added to the encoded data to obtain a video stream. Different values ​​of the preset syntax are used to indicate the display mode of the knowledge image, which includes displaying in a single stream and not displaying in a single stream.

[0011] In one embodiment, the knowledge image used for display can be used as a random access frame.

[0012] In one embodiment, the method further includes:

[0013] The encoded data of the knowledge image is encapsulated into at least one Network Abstraction Layer (NAL) unit packet, and the NAL unit type carried by the NAL unit packet of the knowledge image for display indicates that the knowledge image can be used as a random access frame.

[0014] In one embodiment, the network abstraction layer unit packet used to display the knowledge image carries a network abstraction layer unit type of display knowledge image, instantaneous decoding refresh frame, or RAPI type.

[0015] In one embodiment, the step of encapsulating the encoded data of the knowledge image into at least one network abstraction layer unit packet includes:

[0016] The playback sequence number of the knowledge image is encoded into the encoded data of the knowledge image.

[0017] In one embodiment, encoding the video to obtain encoded data includes:

[0018] Determine whether to use fragmented transmission for knowledge images;

[0019] If fragmented transmission is used, the knowledge image is divided into at least two image blocks, and then the at least two image blocks of the knowledge image are respectively encoded into different coded data;

[0020] If fragmented transmission is not used, the entire knowledge image frame is directly encoded into a single coded data.

[0021] The knowledge images used for display do not employ fragmented interleaving transmission.

[0022] To achieve the above objectives, this application also provides a video decoding method, the method comprising:

[0023] Decode the video stream to obtain the decoded image of the knowledge image;

[0024] The output displays the decoded image of the knowledge image.

[0025] In one embodiment, the output display of the decoded image of the knowledge image includes:

[0026] In one embodiment, the output display of the decoded image of the knowledge image includes:

[0027] The decoded image of the knowledge image to be displayed is simultaneously placed in both the regular image management section of the decoded image buffer and the knowledge image management section of the decoded image buffer; the output display of the decoded image of the knowledge image in the regular image management section is controlled by an output flag in the regular image management section; or,

[0028] The decoded image of the knowledge image is placed in the knowledge image management section of the decoded image buffer, and the knowledge image is directly output and displayed after decoding and reconstruction; or,

[0029] The decoded image of the knowledge image is placed in the knowledge image management section of the decoded image buffer. Based on the value of the preset syntax, a preset mark is applied to the decoded image of the knowledge image. The preset mark is used to indicate whether the decoded image of the knowledge image is output and displayed.

[0030] In one embodiment, the output display of the decoded image of the knowledge image includes:

[0031] If the knowledge image management section of the decoded image cache contains historical knowledge images, remove the historical knowledge images and move in the decoded images of the knowledge images.

[0032] In one embodiment, the output displays a decoded image of the knowledge image, prior to which:

[0033] Obtain the preset syntax from the video stream;

[0034] If the value of the preset syntax indicates that the knowledge image is to be displayed, then the step of outputting the decoded image of the knowledge image is executed;

[0035] If the value of the preset syntax indicates that the knowledge image is not used for display, the decoded image of the knowledge image is decoded to obtain the decoded image of the same content frame; the decoded image of the same content frame is output and displayed, and the knowledge image and its corresponding same content frame correspond to the same original image content.

[0036] To achieve the above objectives, this application also provides an encoder that includes a processor; the processor is configured to execute instructions to implement the steps of the above method.

[0037] To achieve the above objectives, this application also provides a decoder, which includes a processor; the processor is used to execute instructions to implement the steps of the above method.

[0038] To achieve the above objectives, this application also provides a computer-readable storage medium for storing instruction / program data that can be executed to implement the above methods.

[0039] The video encoding method of this application encodes knowledge images to obtain encoded data; a preset syntax is added to the encoded data to obtain a video bitstream. Different values ​​of the preset syntax are used to indicate the display mode of the knowledge image, so that the knowledge image can be displayed. This eliminates the need to encode the original image content corresponding to the knowledge image multiple times, thereby reducing encoding time. Attached Figure Description

[0040] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0041] Figure 1 This is a schematic diagram illustrating the encoding order of multiple images in an image sequence in related technologies;

[0042] Figure 2 This is a schematic diagram of one embodiment of the video encoding method of this application;

[0043] Figure 3 This is a schematic diagram of an embodiment of the video encoding method of this application;

[0044] Figure 4 This is a schematic diagram of another embodiment of the video encoding method of this application;

[0045] Figure 5 This is a schematic diagram of one embodiment of the video decoding method of this application;

[0046] Figure 6 This is a schematic diagram of another embodiment of the video decoding method of this application;

[0047] Figure 7 This is a schematic diagram of one embodiment of the encoder of this application;

[0048] Figure 8 This is a schematic diagram of the structure of one embodiment of the decoder of this application;

[0049] Figure 9 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. In addition, unless otherwise specified (e.g., "or additionally" or "or in alternatives"), the term "or" as used herein refers to a non-exclusive "or" (i.e., "and / or"). Furthermore, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0051] Some video codec standards (such as the SVAC3 standard) introduce the concept of a library picture. In this application, a library picture is represented by an L-frame. In current standards, a library picture is a long-term reference frame that uses intra-frame coding. The library picture serves only as a reference frame and is not used for display. Library pictures are identified by their Library Picture Index (IDX). Since library pictures are not currently used for display, a playback sequence number (i.e., POC or DOI) is not used to identify them.

[0052] like Figure 1 As shown, these standards also introduce the concept of RL (reference library) frames, which are P-frames or B-frames that reference only the knowledge image. Generally, the original image of an RL frame is the same as the original image of the L-frame it references. Therefore, the original image corresponding to the L-frame is encoded twice. The first encoding is used as the knowledge image (e.g., L0), which is not used for display. The second encoding is used as a P-frame or B-frame that references only that knowledge image (e.g., RL0), which is used for display. Consequently, there is an encoding delay before the regular image is displayed due to the repeated encoding of certain frames.

[0053] Based on this, this application proposes a video encoding method. This method encodes a knowledge image to obtain encoded data. A preset syntax is added to the encoded data to obtain a video stream. Different values ​​of the preset syntax are used to indicate whether the knowledge image is used for display, and / or whether to force the use of a skip mode to encode frames with the same content. The frames with the same content and the knowledge image correspond to the same original image content. Thus, the knowledge image can be selectively displayed, avoiding multiple encodings of the original image content corresponding to the knowledge image, thereby reducing encoding time. Alternatively, the skip mode can be selectively forced to encode frames with the same content of the knowledge image, reducing the time spent on the second encoding of the original image content corresponding to the knowledge image. Therefore, the method of this application can reduce encoding latency.

[0054] The aforementioned frames with identical content can be understood as frames where the knowledge image is encoded first, and then the corresponding original image content is encoded again. Generally, in schemes where the original image content corresponding to the knowledge image is encoded twice, RL frames are usually frames with identical content. However, in schemes where the original image content corresponding to the knowledge image is encoded only once, frames with identical content to the knowledge image generally do not appear, and thus RL frames are generally not frames with identical content.

[0055] Specifically, such as Figure 2 As shown, the video encoding method proposed in this application may include the following steps. It should be noted that the step numbers are for simplification only and are not intended to limit the order of execution of the steps.

[0056] S101: Encode the video to obtain encoded data.

[0057] S102: Add preset syntax to the encoded data to obtain the video bitstream.

[0058] Optionally, the knowledge image can be encoded to obtain encoded data. Then, a preset syntax can be added to the encoded data. Different values ​​of the preset syntax are used to indicate the display mode of the knowledge image. In this way, the knowledge image can be displayed selectively, so that the original image corresponding to the knowledge image does not need to be encoded multiple times, which can reduce the encoding time.

[0059] The video includes at least one image sequence, and each image sequence includes multiple image frames. Optionally, an image sequence may include at least one knowledge image.

[0060] The display mode can include displaying in a single stream and / or not displaying in a single stream. Specifically, if all knowledge images in a sequence are not used for output display, and both regular images and knowledge images in the sequence are transmitted in the same bitstream, then the display mode of that sequence can be understood as not displaying in a single stream. Regular images are I-frames, P-frames, and B-frames. Conversely, if all knowledge images in a sequence are used for output display, and both regular images and knowledge images in the sequence are transmitted in the same bitstream, then the display mode of that sequence can be understood as displaying in a single stream.

[0061] The display mode can also include partial display in a single stream. That is, in a sequence, part of the knowledge images are used for output display, and the remaining part of the knowledge images are not used for output display, and both the regular images and knowledge images in the sequence are transmitted in the same bitstream. Then the display mode of the knowledge images in the sequence can be understood as partial display in a single stream.

[0062] The display mode can also include display in two streams. Specifically, if regular images and knowledge images in a sequence are transmitted in different bitstreams, and the knowledge images in the sequence are used for output display, then the display mode of the knowledge images in the sequence can be understood as displaying in two streams.

[0063] The display mode can also include not displaying in both streams. Specifically, if regular images and knowledge images are transmitted in different bitstreams in a sequence, and the knowledge images in that sequence are not used for output display, then the display mode of the knowledge images in that sequence can be understood as not displaying in both streams.

[0064] The display mode of a sequence of knowledge images can be at least one of the following: displaying in a single stream, not displaying in a single stream, partially displaying in a single stream, displaying in a dual stream, and not displaying in a dual stream. Of course, the display mode of a sequence of knowledge images is not limited to this and can also be other display modes.

[0065] In one feasible approach, encoding video may include: encoding at least one parameter of the image sequence in the video (e.g., video encoding configuration and compatibility level, video image size and aspect ratio, video frame rate and bit rate, inter-frame prediction and intra-frame prediction settings, quantization parameters, entropy coding mode, reference frame settings, etc.) to obtain encoded data; then, a preset syntax may be added to the encoded data so that the preset syntax and the original parameters of the image sequence in the encoded data form a sequence parameter set (SPS) of the image sequence. That is, in this feasible approach, a preset syntax may be added to the sequence parameter set (SPS) of the image sequence to uniformly control the display mode of all known image frames in the sequence through the preset syntax.

[0066] Furthermore, since the function of controlling whether a knowledge image is output and displayed is a function within the larger tool of the knowledge image, the encoding and transmission of the syntax elements controlling whether to output and display the knowledge image only need to be performed when the larger tool of the knowledge image is open. Specifically, the preset syntax can be transmitted only in the sequence parameter set of the knowledge image. Of course, in other embodiments, the preset syntax can be transmitted in the sequence parameter set regardless of whether it is the sequence parameter set of the knowledge image or not; that is, the preset syntax can be transmitted in both the sequence parameter set of the knowledge image and the sequence parameter set of the regular image.

[0067] In another possible approach, encoding the video may include: encoding at least one parameter of the image sequence in the video (e.g., video encoding configuration and compatibility level, video image size and aspect ratio, video frame rate and bit rate, inter-frame prediction and intra-frame prediction settings, quantization parameters, entropy coding mode, reference frame settings, etc.) to obtain encoded data of the image sequence parameters; then, a preset syntax may be added to the encoded data of the image sequence parameters so that the preset syntax and the original parameters of the image sequence in the encoded data form a sequence parameter set of the image sequence. That is, in this possible approach, a preset syntax may be added to the sequence parameter set (SPS) of the image sequence to indicate the activation status of the "knowledge image display scheme" in the image sequence through the preset syntax in the sequence parameter set.

[0068] Encoding the video may further include: encoding the knowledge images in the aforementioned image sequence to obtain encoded data of the knowledge images; then, adding preset syntax to the encoded data of the knowledge images so that the preset syntax in the encoded data of the knowledge images is used to indicate the display mode of the knowledge images. The encoded data of the knowledge images may include image parameter sets and image content data. Optionally, the encoded data of the image sequence parameters and the encoded data of at least one knowledge image can be combined into a single video stream packet, that is, the sequence parameter set and the encoded data of at least one knowledge image can be placed in the same stream packet and transmitted together to the decoding end.

[0069] In another feasible approach, encoding video may include: encoding a knowledge image to obtain encoded data of the knowledge image; then, adding preset syntax to the encoded data of the knowledge image so that the preset syntax in the encoded data of the knowledge image is used to indicate the display mode of the knowledge image. The preset syntax can be added to the image parameter set of the knowledge image.

[0070] For the original image content of a knowledge image, it can include at least the following two different processing methods.

[0071] The first approach involves not displaying the decoded image of the knowledge image at the decoding end. In this case, the encoding end encodes the original image content corresponding to the knowledge image twice: once for encoding the knowledge image not intended for display, and once for encoding the same content frame (i.e., the RL image) of the knowledge image to be displayed. When encoding the same content frame for display, conventional encoding methods can be used to re-encode the original image content corresponding to the knowledge image. For example, the reconstructed image of the knowledge image can be used as a reference image to perform inter-frame encoding on the original image content corresponding to the knowledge image, resulting in encoded data for the same content frame. However, this second encoding of the original image content corresponding to the knowledge image also incurs significant time overhead. Furthermore, since the decoded image of the knowledge image and the decoded image of the same content frame may differ in this scheme, both the decoded image of the knowledge image and the decoded image of the same content frame need to be stored in a cache at the decoding end, resulting in significant cache overhead. In other words, this processing method has substantial hardware overhead.

[0072] The second approach is to control the decoding end to display the decoded image of the knowledge image, that is, to set the knowledge image for display. In this way, the original image content corresponding to the knowledge image does not need to be encoded again for display. That is, after encoding the knowledge image, the next frame of the knowledge image can be encoded directly, thus reducing encoding time. Furthermore, for the original image content of the knowledge image, the decoding end will only decode one image, thus avoiding the problem of "needing to simultaneously store the decoded image of the knowledge image and the decoded image of its corresponding content frame in the cache." This approach also saves caching overhead. For example, the knowledge image can be encoded to obtain encoded data; then the encoded data is sent to the decoding end; the decoding end decodes the encoded data to obtain the decoded image of the knowledge image; and the decoding end displays the decoded image of the knowledge image.

[0073] To facilitate video playback by the decoding end based on the video bitstream, when a knowledge image is set for display, its playback sequence number can be encoded into the knowledge image's encoded data. This allows the decoding end to determine the playback order of the knowledge images, thus facilitating video playback. In other words, knowledge images and regular images can be numbered together with their playback sequence numbers (i.e., DOI / POC).

[0074] Of course, in other embodiments, the encoding end may also encode the original image content corresponding to the knowledge image multiple times, wherein the first encoding is the knowledge image, and the knowledge image can be used for output display.

[0075] Based on the above, "whether the knowledge image is used for display" refers to whether the knowledge image is output for display at the decoding end; that is, "display" means that the decoded output is used for display. Specifically, if the knowledge image is used for display, the decoding end will output the decoded image of the knowledge image for display after decoding it. If the knowledge image is not used for display, the decoding end will not output the decoded image of the knowledge image after decoding it; if it cannot be output, it cannot be used for display.

[0076] Alternatively, any of the above processing methods can be executed by default.

[0077] Alternatively, one of the two processing methods above can be selected as the processing method for the original image content corresponding to the knowledge image.

[0078] In some embodiments, there are two processing methods for the processing of the original image content corresponding to a knowledge image. The encoding end can decide to use one of the two processing methods to process the original image content of the knowledge image. After the decision is made, a preset syntax can be added to the encoding data of the knowledge image and / or the sequence parameter set of the image sequence to which the knowledge image belongs. This allows the decoding end to know which of the two processing methods the encoding end uses to process the original image content of the knowledge image.

[0079] Of course, in other embodiments, a preset syntax can also be used to indicate which of the two processing methods described above and other processing methods is used to process the original image content of the knowledge image.

[0080] For example, as shown in Table 1, a 2-bit syntactic element identifier `library_picture_mode_index` can be added to the Sequence Parameter Set (SPS). This identifier controls the switching of the display mode for knowledge images, providing unified control over all knowledge image frames in the sequence. The variable `LibraryPictureEnableFlag` indicates whether the knowledge image tool is enabled for this sequence. After enabling the knowledge image tool, each SPS needs to transmit the syntactic element `library_picture_mode_index`, as shown in Table 1. `library_picture_mode_index = "0"` indicates that all knowledge images in the sequence are not displayed; `library_picture_mode_index = "1"` indicates that all knowledge images in the sequence are displayed; `library_picture_mode_index = "2"` indicates the display mode in dual-stream mode, i.e., knowledge images have a separate bitstream, and regular images have a separate bitstream, which are output separately; `library_picture_mode_index = "3"` is reserved for possible subsequent display modes.

[0081] Table 1. Another schematic diagram of sequence parameter sets.

[0082]

[0083] For example, as shown in Table 2, a 2-bit syntactic element identifier `library_picture_mode_index` can be added to the Sequence Parameter Set (SPS). This identifier is used to control the switching of the display mode of the knowledge image, thus providing unified control over all knowledge image frames in the sequence. In this embodiment, the `library_picture_mode_index` syntactic element is transmitted only in the SPS of the knowledge image. As shown in Table 2, the variable `LibraryStreamFlag` represents whether it is the SPS of a knowledge image; `library_picture_mode_index` is transmitted only when it is the SPS of a knowledge image. Specifically, `library_picture_mode_index = "0"` indicates that all knowledge images in the sequence are not displayed; `library_picture_mode_index = "1"` indicates that all knowledge images in the sequence are displayed; `library_picture_mode_index = "2"` indicates the display mode in dual-stream mode, i.e., knowledge images have a separate stream, and regular images have a separate stream, which are output separately; `library_picture_mode_index = "3"` is reserved for possible subsequent display modes.

[0084] Table 2. A schematic diagram of the sequence parameter set for knowledge images.

[0085]

[0086] In addition, the decision to use patch transmission (i.e., fragmented transmission) to process knowledge images can be made based on the business scenario.

[0087] The business scenario can be determined based on the bit overhead of the knowledge image and / or the image bitrate requirements.

[0088] For example, if the bitrate requirement for knowledge images is low, the knowledge image can be processed without using patch-based transmission, i.e. Figure 1 As shown, the knowledge image is transmitted directly as a whole frame, that is, the entire knowledge image frame is directly encoded into a bitstream; if the bitrate requirement of the knowledge image is high, such as... Figure 3 As shown, knowledge images can be processed using a patch-based transmission method. This involves dividing the knowledge image into at least two image blocks, and then encoding these blocks into different bitstreams. In this way, the decoding end can only decode the knowledge image by obtaining the bitstream containing the encoded data of all image blocks, reducing the impact of large bitstream data on the transmission channel. More preferably, before transmitting the bitstream of the "image encoded from the reference knowledge image," the bitstream packet containing the encoded data of all image blocks of the reference knowledge image is transmitted, so that the decoding end obtains the decoded image of the knowledge image before decoding the "image encoded from the reference knowledge image."

[0089] The bitstream requirements for knowledge images can be related to video resolution requirements, image clarity requirements, and / or the update frequency of knowledge images.

[0090] Optionally, if the update frequency of the knowledge image frame is low (e.g., below the frequency threshold) and a higher bitrate is required to encode the knowledge image, i.e., the bitstream requirement of the knowledge image is relatively high, then the knowledge image can be processed by using a patch transmission method. If the update frequency of the knowledge image frame is high (e.g., above the frequency threshold) and the generated knowledge image does not need to have high image quality, i.e., the bitstream requirement of the knowledge image is relatively low, then the knowledge image can be transmitted as a whole frame, and the impact on the transmission channel is not significant.

[0091] Alternatively, if the video resolution requirement is high (e.g., above the resolution threshold), a higher bitrate is needed to encode the knowledge image, meaning the bitrate requirement for the knowledge image is relatively high. In this case, the knowledge image can be processed using a patch transmission method. If the video resolution requirement is low (e.g., below the resolution threshold), and the generated knowledge image does not need to have high image quality, meaning the bitrate requirement for the knowledge image is relatively low, then transmitting the knowledge image as a whole frame is sufficient, and the impact on the transmission channel is not significant.

[0092] For example, if the bit overhead of the knowledge image is small (e.g., less than the bit threshold), the knowledge image can be processed without using the patch transmission method, that is, the knowledge image can be transmitted as a whole frame, that is, the whole frame of knowledge image is directly encoded into a bitstream. If the bit overhead of the knowledge image is large (e.g., greater than the bit threshold), the knowledge image can be processed using the patch transmission method, that is, the knowledge image is divided into at least two image blocks, and then at least two image blocks of the knowledge image are encoded into different bitstreams, which can reduce the impact of the transmission channel caused by the large bitstream data.

[0093] Optionally, regardless of whether the original image content of the knowledge image is transmitted in patches or in whole frames, the processing method can be either of the two methods described above, and there is no limitation. In a specific embodiment, the processing method for the original image content of the knowledge image transmitted in patches can be the first processing method described above. The processing method for the original image content of the knowledge image transmitted in whole frames can be either the second processing method or the first processing method described above.

[0094] Furthermore, the choice between the two processing methods can be determined based on whether multiple fragments of a knowledge image are transmitted continuously during patch transmission. Specifically, if a knowledge image is transmitted continuously in patches, it may or may not be used for display; if a knowledge image is transmitted interleaved in patches, it is not used for display, i.e., the original image content of the knowledge image is processed using the first processing method described above. Wherein, if at least one frame of regular image is inserted between multiple fragments of a knowledge image, then the knowledge image uses interleaved patch transmission. If at least one frame of regular image is not inserted between multiple fragments of a knowledge image, then the knowledge image uses continuous patch transmission. Thus, when a knowledge image is transmitted in one patch, or multiple patches but not interleaved (e.g., ...), the processing method can be applied to the knowledge image. Figure 3 When L0 is shown, it can be used as a display frame, reducing the latency of a knowledge image frame and eliminating the need for additional buffering; when the knowledge image is transmitted in multiple interleaved patches (e.g., Figure 3 When L1 is shown, it is not used as a display frame to avoid impacting the transmission channel.

[0095] For example, in some actual business scenarios, the frequency of updating knowledge image frames can be higher, and the generated knowledge images do not need to have high image quality. In this case, transmitting the knowledge image as a whole frame is sufficient, and the impact on the transmission channel is not significant. The knowledge image can be displayed directly, and then the image content of the next frame can be encoded. This reduces the latency of displaying one frame of image and eliminates the need for additional buffering. Specifically, such as... Figure 4 As shown, L0 and L1 represent knowledge image frames. The knowledge image with POC=0 is the first frame of the encoded video sequence, and the preset syntax value of the knowledge image with POC=0 is set to the value corresponding to the "knowledge image display scheme". The RL frame with POC=1 is the image content of the second frame of the video sequence. In some other business scenarios, the update frequency of knowledge image frames is low, and a higher bitrate is required to encode the knowledge image. In this case, to avoid impacting the transmission channel, it is necessary to transmit the knowledge image in patches. In this case, the first processing method can be used to process the original image content corresponding to the knowledge image.

[0096] This application also provides a video encoding method according to yet another embodiment. In this embodiment, the video encoding method includes the following steps: encoding the original image content of the knowledge image only once to obtain the encoded data of the knowledge image.

[0097] In this embodiment, when the original image content of the knowledge image is encoded only once, the decoding end can decode the knowledge image and use it for display.

[0098] In one feasible approach, the decision to encode a knowledge image or all knowledge images in a sequence only once can be determined based on the business scenario. Optionally, the business scenario can be determined based on the bit overhead and / or image bitrate requirements of the knowledge image. For example, if the bit overhead of the knowledge image is greater than a bit threshold, the knowledge image is encoded twice to obtain encoded data of the knowledge image and encoded data of the same content frame. In this scenario, the knowledge image data decoded by the decoder is not used for display, but the same content frame of the knowledge image decoded by the decoder is used for display. If the bit overhead of the knowledge image is less than the bit threshold, the knowledge image is encoded only once to obtain encoded data of the knowledge image. In this scenario, the knowledge image decoded by the decoder is used for display. Alternatively, in other feasible approaches, the original image content of all knowledge images can be encoded only once by default.

[0099] In scenarios requiring random access, such as live streaming and video-on-demand, the decoding end can search forward or backward from the live or video-on-demand time to find a random access frame. This frame then serves as the starting point for decoding and playback. Thus, the random access frame is a display frame that needs to be played. Related technologies use regular I-frames and / or RL frames as random access frames. However, since knowledge images are not used for display in these technologies, they are not used as random access frames. Therefore, when the nearest random access frame to the live or video-on-demand time is an RL frame, the knowledge image referenced by that RL frame is decoded first, and then the decoded knowledge image is used as a reference to decode the RL frame. This requires decoding at least two frames before playback can begin from that RL frame.

[0100] Optionally, if the original image content of the knowledge image is encoded only once, since the knowledge image can be used for display, it can be used as a random access frame. In this way, the decoding end can decode the knowledge image and play it. Thus, by using the knowledge image used for display as a random access frame, video playback can be started from the random access frame after decoding one frame, which can improve the problem of stuttering during random access frame playback of the video stream.

[0101] In one technique, the decoding end determines whether a frame is a random access frame by searching the NAL (Network Abstraction Layer) unit type of each frame. This can be achieved at the system layer of the decoding end by using the NAL unit type of each frame. In this case, to enable the decoding end to use the knowledge image for display as a random access frame, the encoding end can set the NAL unit type of the knowledge image before transmitting the video stream to indicate that the knowledge image can be used as a random access frame.

[0102] For example, as shown in Table 3, a new nal unit type can be added, such as adding nal unit type "17" in Table 3: display knowledge image, and setting the nal unit type of the knowledge image used for display to the newly added type "display knowledge image". In this way, when the system layer at the decoding end finds that the nal unit type of a knowledge image is 17, it can know that the knowledge image is a knowledge image used for display, and thus it can be used as a random access frame.

[0103] For example, the nal unit type of the knowledge image used for display can also be set to the nal unit type of the IDR (Instantaneous Decoding Refresh) frame. This reuses the nal unit type of the current IDR frame, since the IDR itself represents a random access frame. Encoding the knowledge image for display with an IDR nal type indicates that the knowledge image for display can reuse the IDR operations in random access related operations, and the decoding end can then recognize that the knowledge image for display is a random access frame. It's understandable that although the knowledge image for display is encoded with an IDR nal type, its internal structure is entirely related to the knowledge image itself; that is, it is still processed using knowledge image-related decoding and caching methods.

[0104] Table 3. NAL Unit Type Diagram

[0105]

[0106] For example, the nal unit type of the knowledge image used for display can also be set to RAPI type, that is, reusing the current RAPI type. Since a RAPI image itself represents a random access frame, encoding and transmitting a knowledge image for display as a RAPI type indicates that the knowledge image used for display can reuse RAPI operations in random access related operations, and the decoding end can then know that the knowledge image used for display is a random access frame. It is understandable that although the knowledge image used for display is encoded and transmitted as a RAPI type, its internal structure is entirely based on knowledge image content; that is, it is still processed using knowledge image-related decoding and caching methods.

[0107] In the case where the knowledge image used for display is used as a random access frame, the RL frame that references the knowledge image used for display may not be used as a random access frame. In this way, the random access frame that is closest to the random access time such as the live broadcast time or the on-demand time is searched is used as the knowledge image used for display, thereby improving the problem of stuttering during the playback of random access frames in the video stream.

[0108] Of course, in some cases, regardless of whether the knowledge image is used for display, the RL frame referencing the knowledge image can be used as a random access frame. However, it is preferable that if the random access frame closest to the live or on-demand time is an RL frame, it can be further confirmed whether the knowledge image referenced by the RL frame is a random access frame. If the knowledge image referenced by the RL frame is not a random access frame, then the RL frame is directly used as the random access point, i.e., playback starts from the RL frame. If the knowledge image referenced by the RL frame is a random access frame, then the knowledge image referenced by the RL frame can be used as the random access point, i.e., playback starts from the knowledge image referenced by the RL frame.

[0109] Of course, in other embodiments, regardless of whether the knowledge image is displayed, it may not be used as a random access frame; that is, it is stipulated that the knowledge image will not be used as a random access frame. In such random access, the decoding end still treats the RL frame as a random access frame. For example, the NAL unit type of both the knowledge image used for display and the knowledge image not used for display is set to NAL unit type "12"—knowledge image frame. Thus, the decoding end can know that the image frame is a knowledge image frame and is not used as a random access frame based on NAL unit type "12"—knowledge image frame.

[0110] Before transmitting the video stream, the encoded video data can be encapsulated into NAL packets. Each NAL packet carries a NAL unit type, indicating the type of data contained within it. This allows the system layer to process different types of data differently. Thus, for NAL packets containing knowledge images for display, the NAL unit type can be specifically configured according to the various implementation methods described above, enabling the decoding end to determine the random access point based on the NAL unit type carried by the NAL packet.

[0111] Please see Figure 5 , Figure 5 This is a flowchart illustrating one embodiment of the video decoding method of this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow that approach. Figure 5 The illustrated process sequence is limited. In this embodiment, the video decoding method includes the following steps:

[0112] S201: Obtain preset syntax from video stream.

[0113] The different values ​​of the preset syntax are used to indicate the display mode of the knowledge image.

[0114] S202: Based on preset syntax, output and display knowledge images or frames containing the same content as knowledge images.

[0115] The step of outputting and displaying a knowledge image or a frame containing the same content as the knowledge image based on the preset syntax includes: the value of the preset syntax instructing the knowledge image to be displayed, decoding the knowledge image and outputting and displaying it.

[0116] Optionally, the step of outputting and displaying a knowledge image or a frame with the same content as the knowledge image based on the preset syntax includes: decoding the knowledge image when the value of the preset syntax indicates that the knowledge image is not used for display; and decoding and outputting a decoded image of the frame with the same content based on the decoded image of the knowledge image.

[0117] Furthermore, the specific details of the video decoding method correspond to the specific details of the video encoding method in the above embodiments, and can be referred to the relevant content of the video encoding method in the above embodiments, which will not be repeated here.

[0118] This can be achieved by decoding and outputting a knowledge image or a frame containing the same content as a knowledge image, based on a preset syntax. In one implementation, if the preset syntax indicates that the knowledge image is to be displayed, the knowledge image is decoded; the decoded image is then output to display the knowledge image. In another implementation, if the preset syntax indicates that the knowledge image is not to be displayed, the knowledge image is decoded; based on the decoded image of the knowledge image, a decoded image containing the same content is obtained; the decoded image containing the same content is then output to display the decoded image.

[0119] Please see Figure 6 , Figure 6 This is a flowchart illustrating another embodiment of the video decoding method of this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow that approach. Figure 6 The illustrated process sequence is limited. In this embodiment, the video decoding method includes the following steps:

[0120] S301: Decode the video stream to obtain the decoded image of the knowledge image;

[0121] S302: Output the decoded image of the knowledge image.

[0122] In related technologies, if an image needs to be output and displayed, it needs to be placed in the regular image management section of the DPB (decoded picture buffer), and a label indicating whether the image has been output and displayed is used. However, currently, knowledge images are only placed in the knowledge image management section of the DPB, without corresponding labels to control the output. Based on this, this application proposes several implementation methods to enable the output and display of knowledge images.

[0123] Firstly, the knowledge image intended for display can be placed in both the regular image management section and the knowledge image management section of the DPB. The regular image management section manages the output of the knowledge image, while the knowledge image management section allows the knowledge image to be referenced. This approach is relatively simple and direct. Specifically, the decoding end places the knowledge image intended for display in both the regular image management section and the knowledge image management section of the DPB. When decoding an image frame that uses the knowledge image intended for display as a reference frame, the decoding end references the knowledge image within the knowledge image management section of the DPB. The knowledge image placed in the regular image management section of the DPB can also be marked as "referenced" and "output". Since the knowledge image used for display is also placed in the knowledge image management section of the DPB, the knowledge image placed in the regular image management section of the DPB can be unreferenced and thus marked as "unreferenced". After the knowledge images placed in the regular image management section of the DPB are output and displayed in the playback order, the output knowledge image in the regular image management section of the DPB can be marked as "output". Thus, the knowledge image has been marked as "unreferenced" and "output", and can be removed from the regular image management section of the DPB.

[0124] Secondly, the knowledge image used for display can be directly output after decoding and reconstruction. In this implementation, the knowledge image only needs to be placed in the knowledge image management section of the DPB.

[0125] The decoding end can include a DPB and a temporary buffer. After the knowledge image is decoded and reconstructed, the knowledge image can be placed in the temporary buffer, and then the knowledge image in the temporary buffer can be output and displayed. That is, the knowledge image can be directly output and displayed through the temporary buffer. In this way, there is no need to control the output and display of the knowledge image through the management of the DPB. Instead, it is only necessary to control the reference of the knowledge image through the DPB. Thus, the knowledge image only needs to be placed in the knowledge image management part of the DPB for reference. In this way, the DPB does not need to store an extra frame of knowledge image content, which reduces some of the DPB usage overhead compared to the previous implementation method. That is, it does not increase the usage overhead of the DPB.

[0126] In other embodiments, knowledge images can also be directly output and displayed by means of a knowledge image cache specifically used to store knowledge images.

[0127] To facilitate the output and display of knowledge images stored in temporary caches, a bypass output path can be established for the temporary caches, etc., so that the knowledge images to be displayed in the temporary caches can be output and displayed through this output path. This bypass output path can be a newly built output circuit.

[0128] Third, the knowledge image is still only placed in the knowledge image management section of the DPB, and a new output-option flag is added to the knowledge image management section. If the preset syntax representation in the above implementation is not displayed, the new flag is "not output"; if the preset syntax representation in the above implementation is displayed, the new flag is "outputtable". Knowledge images marked as "outputtable" are then output and displayed, without increasing the DPB's usage overhead. That is, the "outputtable" flag is set for knowledge images based on the value of the preset syntax in the above implementation. Specifically, if the value of the preset syntax indicates that the knowledge image frame is not used for display, then the knowledge image frame is marked as "not output"; if the value of the preset syntax indicates that the knowledge image frame is used for display, then the knowledge image frame is marked as "outputtable".

[0129] For example, after a knowledge image frame is decoded and placed into the knowledge image management section of the DPB, if the corresponding library_picture_mode_index is "0", it means that the knowledge image frame is not used for display, and the knowledge image frame can be marked as "not output"; if the library_picture_mode_index is "1", it means that the knowledge image frame is used for display, and the knowledge image frame can be marked as "output".

[0130] In all three implementation methods described above, after decoding a complete knowledge image frame, the newly decoded knowledge image frame must be placed into the knowledge image management section of the DPB so that the knowledge image frame can be referenced by the knowledge image management section of the DPB.

[0131] Optionally, if the knowledge image management part of the DPB still contains previous knowledge images (i.e., historical knowledge images) after decoding a complete knowledge image frame, the previous knowledge images can be moved out and then the currently decoded knowledge image can be moved in; if the knowledge image management part of the DPB is empty at this time, the currently decoded knowledge image can be moved in directly.

[0132] Of course, in some embodiments, if the knowledge image management part of the DPB still has historical knowledge images, the historical knowledge images do not need to be removed, that is, the currently decoded knowledge image can be directly moved in.

[0133] In addition, regular images (i.e., images used for display other than knowledge images) can still be output and displayed using the DPB management method.

[0134] Please see Figure 7 , Figure 7This is a schematic diagram of one embodiment of the encoder of this application. The encoder 10 includes a processor 12, which executes instructions to implement the video encoding method described above. For details of the implementation process, please refer to the description of the above embodiment, which will not be repeated here.

[0135] Processor 12 can also be referred to as a CPU (Central Processing Unit). Processor 12 may be an integrated circuit chip with signal processing capabilities. Processor 12 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor, or processor 12 can be any conventional processor.

[0136] The encoder 10 may further include a memory 11 for storing instructions and data required for the processor 12 to run.

[0137] The processor 12 is used to execute instructions to implement the methods provided by any embodiment and any non-conflicting combination of the video encoding methods of this application described above.

[0138] Please see Figure 8 , Figure 8 This is a schematic diagram of one embodiment of the decoder of this application. The decoder 20 includes a processor 22, which executes instructions to implement the video decoding method described above. For detailed implementation processes, please refer to the description of the above embodiment; further details will not be repeated here.

[0139] Processor 22 can also be referred to as CPU (Central Processing Unit). Processor 22 may be an integrated circuit chip with signal processing capabilities. Processor 22 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor, or processor 22 can be any conventional processor.

[0140] The decoder 20 may further include a memory 21 for storing instructions and data required for the processor 22 to run.

[0141] The processor 22 is used to execute instructions to implement the methods provided by any embodiment and any non-conflicting combination of the video decoding methods of this application described above.

[0142] Please see Figure 9 , Figure 9This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of this application. The computer-readable storage medium 30 in this embodiment stores instruction / program data 31. When executed, this instruction / program data 31 implements the methods provided by any embodiment of the video encoding method and video decoding method of this application, as well as any non-conflicting combination thereof. The instruction / program data 31 can be formed into a program file and stored in the storage medium 30 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium 30 includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0145] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0146] The above are merely embodiments of this application and do not limit the scope of this patent application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A video encoding method, characterized in that, The method includes: The original image content of the knowledge image is encoded only once to obtain the encoded data of the knowledge image, wherein the knowledge image is used for display, and the knowledge image used for display does not use fragmented interleaving transmission.

2. A video encoding method, characterized in that, The method includes: The video is encoded to obtain encoded data; A preset syntax is added to the encoded data to obtain a video bitstream. Different values ​​of the preset syntax are used to indicate the display mode of the knowledge image. The display mode includes displaying in a single stream or not displaying in a single stream. The knowledge image used for display is transmitted in segments or in whole frames. The knowledge image transmitted in segments is not used for display.

3. The video encoding method according to claim 1 or 2, characterized in that, The knowledge image used for display can be used as a random access frame.

4. The video encoding method according to claim 3, characterized in that, The method further includes: The encoded data of the knowledge image is encapsulated into at least one Network Abstraction Layer (NAL) unit packet, and the NAL unit type carried by the NAL unit packet of the knowledge image for display indicates that the knowledge image can be used as a random access frame.

5. The video encoding method according to claim 4, characterized in that, The network abstraction layer unit (BAL) packet used for displaying the knowledge image carries a network abstraction layer unit type of display knowledge image, instantaneous decoding refresh frame, or RAPI type.

6. The video encoding method according to claim 4, characterized in that, The step of encapsulating the encoded data of the knowledge image into at least one network abstraction layer unit packet includes: The playback sequence number of the knowledge image is encoded into the encoded data of the knowledge image.

7. The video encoding method according to claim 2, characterized in that, The process of encoding the video to obtain encoded data includes: Determine whether to use fragmented transmission for knowledge images; If fragmented transmission is used, the knowledge image is divided into at least two image blocks, and then the at least two image blocks of the knowledge image are respectively encoded into different coded data; If fragmented transmission is not used, the entire frame of knowledge image can be directly encoded into a single coded data; The fragmented transmission includes fragmented interleaved transmission and fragmented continuous transmission.

8. A video decoding method, characterized in that, The method includes: Decode the video stream to obtain the decoded image of the knowledge image; If the knowledge image is used for display, a decoded image of the knowledge image is output to display it. The knowledge image used for display is transmitted in fragmented continuous transmission or in whole frame transmission. Knowledge images transmitted in fragmented interleaved transmission are not used for display.

9. The video decoding method according to claim 8, characterized in that, The output displays the decoded image of the knowledge image, which previously included: Obtain the preset syntax from the video stream; If the value of the preset syntax indicates that the knowledge image is to be displayed, then the step of outputting the decoded image of the knowledge image is executed; If the value of the preset syntax indicates that the knowledge image is not used for display, the decoded image of the knowledge image is decoded to obtain the decoded image of the same content frame; the decoded image of the same content frame is output and displayed, and the knowledge image and its corresponding same content frame correspond to the same original image content.

10. The video decoding method according to claim 9, characterized in that, The output displays the decoded image of the knowledge image, including: The decoded image of the knowledge image to be displayed is simultaneously placed in both the regular image management section of the decoded image buffer and the knowledge image management section of the decoded image buffer; the output display of the decoded image of the knowledge image in the regular image management section is controlled by an output flag in the regular image management section; or, The decoded image of the knowledge image is placed in the knowledge image management section of the decoded image buffer, and the knowledge image is directly output and displayed after decoding and reconstruction; or, The decoded image of the knowledge image is placed in the knowledge image management section of the decoded image buffer. Based on the value of the preset syntax, a preset mark is applied to the decoded image of the knowledge image. The preset mark is used to indicate whether the decoded image of the knowledge image is output and displayed.

11. The video decoding method according to claim 8, characterized in that, The output displays the decoded image of the knowledge image, including: If the knowledge image management section of the decoded image cache contains historical knowledge images, remove the historical knowledge images and move in the decoded image of the knowledge images.

12. An encoder, characterized in that, The encoder includes a processor; the processor is configured to execute instructions to implement the steps of the method as described in any one of claims 1-7.

13. A decoder, characterized in that, The decoder includes a processor; the processor is configured to execute instructions to implement the steps of the method as described in any one of claims 8-11.

14. A computer-readable storage medium storing instruction / program data thereon, characterized in that, When the instruction / program data is executed, it implements the steps of the method described in any one of claims 1-11.

Images