Video coding method, video decoding method and related devices

Abstract

The invention discloses a video coding method, a video decoding method and related devices. The video coding method comprises the following steps: coding a video to obtain coded data; a preset syntax is added to the coded data to obtain a video code stream, different values of the preset syntax are used for indicating a display mode of the knowledge image, and the display mode comprises display in a single stream and non-display in the single stream. According to the invention, coding delay can be reduced.

Description

Technical Field

[0001] The present application relates to the field of video encoding and decoding technology, and in particular to a video encoding method, a video decoding method and related devices. Background Art

[0002] The amount of video image data is relatively large, and it is usually necessary to compress the video pixel data (RGB, YUV, etc.). The compressed data is called a video stream, which is transmitted to the user end through a wired or wireless network and then decoded for viewing. The entire video encoding process includes prediction, transformation, quantization, encoding and other processes.

[0003] In video encoding and decoding, in order to improve the compression rate and reduce the codewords that need to be transmitted, the encoder will not directly encode and transmit the pixel values, but will use the intra-frame or inter-frame prediction mode to predict the pixel values ​​of the current block using the reconstructed pixels of the encoded block of the current frame or reference frame. The pixel value predicted by a certain prediction mode is called the predicted pixel value, and the difference between the predicted pixel value and the original pixel value is called the residual. The encoder only needs to encode a certain prediction mode and the residual generated when the prediction mode is used, and the decoder can decode the corresponding pixel value based on the code stream information. This greatly reduces the codewords required for encoding.

[0004] However, during the long-term research and development process, the inventors of the present application discovered that the current video encoding method still has some shortcomings; for example, there is a relatively long encoding delay. Summary of the invention

[0005] The present application provides a video encoding method, a video decoding method and related devices, which can reduce encoding delay.

[0006] To achieve the above object, the present application provides a video encoding method, the method comprising:

[0007] The original image content of the knowledge image is encoded only once to obtain the encoded data of the knowledge image, wherein the knowledge image is used for display.

[0008] To achieve the above object, the present application provides a video encoding method, the method comprising:

[0009] Encode the video to obtain encoded data;

[0010] A preset syntax is added to the encoded data to obtain a video code stream, and different values ​​of the preset syntax are used to indicate a display mode of the knowledge image, and the display mode includes displaying in a single stream and not displaying in a single stream.

[0011] In one embodiment, the knowledge image for display can be used as a random access frame.

[0012] In one embodiment, the method further comprises:

[0013] The coded data of the knowledge image is encapsulated into at least one network abstraction layer unit packet, and the network abstraction layer unit type carried by the network abstraction layer unit packet of the knowledge image for display indicates that the knowledge image can be used as a random access frame.

[0014] In one embodiment, the network abstraction layer unit type carried by the network abstraction layer unit packet of the knowledge image for display is a display knowledge image type, an instantaneous decoding refresh frame type or a RAPI type.

[0015] In one embodiment, encapsulating the encoded data of the knowledge image into at least one network abstraction layer unit packet comprises:

[0016] The playback sequence number of the knowledge image is encoded into the encoding data of the knowledge image.

[0017] In one embodiment, encoding the video to obtain encoded data includes:

[0018] Determine whether to adopt fragmented transmission for the knowledge image;

[0019] If slice transmission is adopted, the knowledge image is cut into at least two image blocks, and then the at least two image blocks of the knowledge image are respectively encoded into different encoding data;

[0020] If the fragmented transmission is not adopted, the whole frame of knowledge image is directly encoded into one coded data.

[0021] Among them, the knowledge image used for display does not adopt slice interleaving transmission.

[0022] To achieve the above object, the present application also provides a video decoding method, the method comprising:

[0023] Decode the video code stream to obtain a decoded image of the knowledge image;

[0024] The output is a decoded image showing the knowledge image.

[0025] In one embodiment, the outputting displays a decoded image of the knowledge image, comprising:

[0026] In one embodiment, the outputting displays a decoded image of the knowledge image, comprising:

[0027] Put the decoded image of the knowledge image for display into the conventional image management part of the decoded image buffer and the knowledge image management part of the decoded image buffer at the same time; use the output flag in the conventional image management part to control the output display of the decoded image of the knowledge image in the conventional image management part; or,

[0028] The decoded image of the knowledge image is placed in the knowledge image management part of the decoded image buffer area, and the knowledge image is directly output and displayed after decoding and reconstruction is completed; or,

[0029] The decoded image of the knowledge image is placed in the knowledge image management part of the decoded picture cache area, and the decoded image of the knowledge image is marked with a preset mark based on the value of the preset syntax, and the preset mark is used to indicate whether the decoded image of the knowledge image is output and displayed.

[0030] In one embodiment, the outputting displays a decoded image of the knowledge image, comprising:

[0031] If a historical knowledge image is stored in the knowledge image management part of the decoded picture buffer area, the historical knowledge image is removed and the decoded image of the knowledge image is moved in.

[0032] In one embodiment, the outputting displays a decoded image of the knowledge image, and the outputting previously includes:

[0033] Get the preset syntax from the video stream;

[0034] If the value of the preset syntax indicates that the knowledge image is for display, the step of outputting a decoded image that displays the knowledge image is performed;

[0035] If the value of the preset syntax indicates that the knowledge image is not for display, a decoded image of the same content frame is obtained by decoding based on the decoded image of the knowledge image; the decoded image of the same content frame is output and displayed, and the knowledge image and its corresponding same content frame correspond to the same original image content.

[0036] To achieve the above objectives, the present application also provides an encoder, which includes a processor; the processor is used to execute instructions to implement the steps of the above method.

[0037] To achieve the above objectives, the present application also provides a decoder, which includes a processor; the processor is used to execute instructions to implement the steps of the above method.

[0038] To achieve the above objectives, the present application also provides a computer-readable storage medium, which is used to store instructions / program data, and the instructions / program data can be executed to implement the above method.

[0039] The video encoding method of the present application encodes a knowledge image to obtain encoded data; a preset syntax is added to the encoded data to obtain a video code stream, and different values ​​of the preset syntax are used to indicate a display mode of the knowledge image, so that the knowledge image can be displayed. In this way, there is no need to encode the original image content corresponding to the knowledge image multiple times, which can reduce the encoding time. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 It is a schematic diagram of the encoding order of multiple images in an image sequence in the related art;

[0042] Figure 2 It is a schematic diagram of an implementation method of the video encoding method of the present application;

[0043] Figure 3 is a schematic diagram of an embodiment of a video encoding method of the present application;

[0044] Figure 4 is a schematic diagram of another embodiment of the video encoding method of the present application;

[0045] Figure 5 It is a schematic diagram of an implementation method of a video decoding method of the present application;

[0046] Figure 6 is a schematic diagram of another implementation of the video decoding method of the present application;

[0047] Figure 7 It is a structural schematic diagram of an implementation mode of an encoder of the present application;

[0048] Figure 8 It is a structural diagram of an implementation mode of a decoder of the present application;

[0049] Fig. 9 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of the present application. In addition, unless otherwise specified (for example, "or in addition" or "or in an alternative"), the term "or" as used herein refers to a non-exclusive "or" (that is, "and / or"). Moreover, the various embodiments described herein are not necessarily mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments.

[0051] The concept of library picture is introduced in some video coding standards (such as SVAC3 standard). In this application, L frame is used to represent library picture. In the current standard, library picture is a long-term reference frame, which adopts intra-frame coding. Library picture is only used as reference frame and is not used for display. Library picture is identified by its library picture index IDX. Since library picture is not used for display at present, the playback sequence number (i.e. POC or DOI) is not used to identify library picture.

[0052] like Figure 1 As shown, the concept of RL (reference library) frame is also introduced in these standards. RL frame refers to P frame or B frame that only refers to knowledge image. Generally speaking, the original image of RL frame is the same as the original image of L frame that RL frame refers to. Therefore, the original image corresponding to L frame is repeatedly encoded twice, wherein the first encoding is a knowledge image (such as L0) which is not used for display, and the second encoding is a P frame or B frame (such as RL0) that only refers to the knowledge image and is used for display. Therefore, before the regular image is displayed, there will be a coding delay caused by the repeated encoding of some frames.

[0053] Based on this, the present application proposes a video encoding method, which encodes a knowledge image to obtain encoded data; adds a preset syntax to the encoded data to obtain a video code stream, and different values ​​of the preset syntax are used to indicate whether the knowledge image is used for display and / or whether to force the use of a skip mode to encode the same content frame. The same content frame and the knowledge image correspond to the same original image content, so that the knowledge image can be selectively displayed, so that the original image content corresponding to the knowledge image does not need to be encoded multiple times, which can reduce the encoding time; or, the same content frame of the knowledge image can be selectively forced to use a skip mode to encode, which can reduce the second encoding time of the original image content corresponding to the knowledge image, and thus the method of the present application can reduce the encoding delay.

[0054] The above-mentioned same content frames can be understood as frames in which the original image content corresponding to the knowledge image is encoded again after the knowledge image is encoded. Generally speaking, in the scheme in which the original image content corresponding to the knowledge image is encoded twice, the RL frames are generally the same content frames. In the scheme in which the original image content corresponding to the knowledge image is encoded only once, the same content frames of the knowledge image generally do not appear, so the RL frames are generally not the same content frames.

[0055] Specifically, Figure 2 As shown, a video encoding method proposed in the present application may include the following steps. It should be noted that the following step numbers are only used to simplify the description and are not intended to limit the execution order of the steps.

[0056] S101: Encode the video to obtain encoded data.

[0057] S102: Adding a preset syntax to the coded data to obtain a video code stream.

[0058] Optionally, the knowledge image can be encoded to obtain encoded data, and then a preset syntax can be added to the encoded data, wherein different values ​​of the preset syntax are used to indicate the display mode of the knowledge image. In this way, the knowledge image can be displayed selectively, thereby eliminating the need to encode the original image corresponding to the knowledge image multiple times, thereby reducing encoding time.

[0059] The video includes at least one image sequence, and each image sequence includes a plurality of image frames. Optionally, an image sequence may include at least one knowledge image.

[0060] Among them, the display mode may include displaying in a single stream and / or not displaying in a single stream. Among them, if all knowledge images in a sequence are not used for output display, and the regular images and knowledge images in the sequence are transmitted in the same code stream, then the display mode of the sequence can be understood as not displaying in a single stream. Among them, regular images are I frames, P frames and B frames. Among them, if all knowledge images in a sequence are used for output display, and the regular images and knowledge images in the sequence are transmitted in the same code stream, then the display mode of the sequence can be understood as displaying in a single stream.

[0061] The display mode may also include partial display in a single stream. That is, part of the knowledge images in a sequence are used for output display, and the remaining part of the knowledge images are not used for output display, and both the regular images and the knowledge images in the sequence are transmitted in the same code stream, then the display mode of the knowledge images in the sequence can be understood as partial display in a single stream.

[0062] The display mode may also include displaying in dual streams. Specifically, if the conventional images and the knowledge images in a sequence are transmitted in different code streams, and the knowledge images in the sequence are used for output display, then the display mode of the knowledge images in the sequence can be understood as displaying in dual streams.

[0063] The display mode may also include not displaying in dual streams. Specifically, if the conventional images and knowledge images in a sequence are transmitted in different code streams, and the knowledge images in the sequence are not used for output display, the display mode of the knowledge images in the sequence can be understood as not displaying in dual streams.

[0064] The display mode of a sequence of knowledge images can be at least one of the above-mentioned display in a single stream, no display in a single stream, partial display in a single stream, display in dual streams, and no display in dual streams. Of course, the display mode of a sequence of knowledge images is not limited to this, and can also be other display modes.

[0065] In one feasible manner, encoding a video may include: encoding at least one parameter of an image sequence in the video (such as the configuration and compatibility level of video encoding, the size and aspect ratio of video images, the frame rate and bit rate of the video, inter-frame prediction and intra-frame prediction settings, quantization parameters, entropy coding modes, reference frame settings, etc.) to obtain encoded data; then a preset syntax may be added to the encoded data so that the preset syntax and the original parameters of the image sequence in the encoded data constitute a sequence parameter set of the image sequence, that is, in this feasible manner, a preset syntax may be added to a sequence parameter set (SPS) of an image sequence so that the display mode of all known image frames in the sequence can be uniformly controlled through the preset syntax.

[0066] Furthermore, since the function of controlling whether the knowledge image is output and displayed is a function under the knowledge image tool, the encoding and transmission of the syntax elements that control whether the output and display are displayed will be performed only when the knowledge image tool is opened. Specifically, the preset syntax can be transmitted only in the sequence parameter set of the knowledge image. Of course, in other embodiments, regardless of whether a sequence parameter set is a sequence parameter set of the knowledge image, the preset syntax can be transmitted in the sequence parameter set, that is, the preset syntax can be transmitted in both the sequence parameter set of the knowledge image and the sequence parameter set of the conventional image.

[0067] In another feasible manner, encoding a video may include: encoding at least one parameter of an image sequence in the video (e.g., video encoding configuration and compatibility level, video image size and aspect ratio, video frame rate and bit rate, inter-frame prediction and intra-frame prediction settings, quantization parameters, entropy coding mode, reference frame settings, etc.) to obtain encoded data of the image sequence parameters; then a preset syntax may be added to the encoded data of the image sequence parameters so that the preset syntax and the original parameters of the image sequence in the encoded data constitute a sequence parameter set of the image sequence, that is, in this feasible manner, a preset syntax may be added to a sequence parameter set (SPS) of an image sequence to indicate, through the preset syntax in the sequence parameter set, the activation status of a "display scheme for knowledge images" in the image sequence.

[0068] Encoding the video may also include: encoding the knowledge image in the above-mentioned image sequence to obtain the encoded data of the knowledge image; then a preset syntax may be added to the encoded data of the knowledge image, so that the preset syntax in the encoded data of the knowledge image is used to indicate the display mode of the knowledge image. The encoded data of the knowledge image may include an image parameter set and image content data. Optionally, the encoded data of the image sequence parameters and the encoded data of at least one knowledge image may form a video bitstream package, that is, the sequence parameter set and the encoded data of at least one knowledge image may be placed in the same bitstream package and transmitted together to the decoding end.

[0069] In another achievable manner, encoding a video may include: encoding a knowledge image to obtain encoding data of the knowledge image; and then adding a preset syntax to the encoding data of the knowledge image, so that the preset syntax in the encoding data of the knowledge image is used to indicate a display mode of the knowledge image. The preset syntax may be added to an image parameter set of the knowledge image.

[0070] For the original image content of a knowledge image, it may include at least the following two different processing methods.

[0071] The first processing method is that the decoding end does not display the decoded image of the knowledge image. In this way, the encoding end will encode the original image content corresponding to the knowledge image twice, one of which is used to encode as a knowledge image not for display, and the other is used to encode the same content frame (i.e., RL image) of the knowledge image for display. When encoding the same content frame for display, the original image content corresponding to the knowledge image can be directly encoded again using the conventional encoding method of the prior art. For example, the original image content corresponding to the knowledge image is inter-frame encoded using the reconstructed image of the knowledge image as a reference image to obtain the encoded data of the same content frame. In this way, the time overhead of the second encoding of the original image content corresponding to the knowledge image is also large. In addition, since in this scheme, there may be differences between the decoded image of the knowledge image and the decoded image of the same content frame, the decoded image of the knowledge image and the decoded image of the same content frame are obtained by decoding at the decoding end. Generally, it is necessary to retain the decoded image of the knowledge image and the decoded image of the same content frame in the cache at the same time, which leads to a large cache overhead, that is, this processing method has a large hardware overhead.

[0072] The second processing method is to control the decoding end to display the decoded image of the knowledge image, that is, to set the knowledge image to be displayed, so that the original image content corresponding to the knowledge image does not need to be encoded again for display, that is, after encoding the knowledge image, the next frame image of the knowledge image can be directly encoded, thereby reducing the encoding time; in addition, corresponding to the original image content of the knowledge image, the decoding end will only decode to obtain one image, so there is no problem of "needing to retain the decoded image of the knowledge image and the decoded image of the same content frame in the cache at the same time", that is, using this solution can also save cache overhead. Exemplarily, the knowledge image can be encoded to obtain the encoded data of the knowledge image; then the encoded data of the knowledge image is sent to the decoding end; the decoding end decodes the encoded data of the knowledge image to obtain the decoded image of the knowledge image; the decoding end displays the decoded image of the knowledge image.

[0073] In order to facilitate the decoding end to play the video based on the video code stream, when the knowledge image is set to display, the play sequence number of the knowledge image can be encoded into the coded data of the knowledge image, so that the decoding end can determine the play sequence of the knowledge image, thereby facilitating the decoding end to play the video. That is, the knowledge image and the regular image can be numbered together with the play sequence number (i.e., DOI / POC).

[0074] Of course, in other embodiments, the encoding end may also encode the original image content corresponding to the knowledge image multiple times, wherein the first encoding is a knowledge image, and the knowledge image can be used for output display.

[0075] Based on the above content, it can be known that "whether the knowledge image is used for display" refers to whether the knowledge image is output and displayed at the decoding end, that is, "display" means decoding output and being used for display. Specifically, if the knowledge image is used for display, the decoding end will output and display the decoded image of the knowledge image after decoding it. If the knowledge image is not used for display, the decoding end will not output the decoded image of the knowledge image after decoding it, and if it cannot be output, it cannot be used for display.

[0076] Optionally, any of the above processing modes may be executed by default.

[0077] Alternatively, one of the above two processing methods may be selected as the processing method for the original image content corresponding to the knowledge image.

[0078] That is, in some embodiments, there are two processing methods for selecting the processing method of the original image content corresponding to a knowledge image, that is, the encoding end can decide to use the above two processing methods to process the original image content of the knowledge image. After the decision, the preset syntax can be added to the encoding data of the knowledge image and / or the sequence parameter set of the image sequence to which the knowledge image belongs, so that the decoding end can know which of the above two processing methods the encoding end uses to process the original image content of the knowledge image through the encoding data of the knowledge image and / or the sequence parameter set of the image sequence to which the knowledge image belongs.

[0079] Of course, in other embodiments, a preset syntax may also be used to indicate which of the above two processing methods and other processing methods is used to process the original image content of the knowledge image.

[0080] Exemplarily, as shown in Table 1, a 2-bit syntax element identifier library_picture_mode_index can be added to the sequence parameter set (SPS), and the identifier is used to control the switching of the display mode of the knowledge image. At this time, all knowledge image frames in the sequence are uniformly controlled. The variable LibraryPictureEnableFlag represents whether the knowledge image tool is enabled for the sequence. After the knowledge image tool is enabled, each SPS needs to transmit the syntax element library_picture_mode_index, as shown in Table 1. library_picture_mode_index = "0" means that all knowledge images in the sequence are not displayed; library_picture_mode_index = "1" means that all knowledge images in the sequence are displayed; library_picture_mode_index = "2" is used to indicate the display mode in dual-stream mode, that is, the knowledge image has a separate code stream and the conventional image has a separate code stream, which are output separately; library_picture_mode_index = "3" is used to reserve for subsequent display modes that may exist.

[0081] Table 1 Another schematic diagram of sequence parameter set

[0082]

[0083] For another example, as shown in Table 2, a 2-bit syntax element identifier library_picture_mode_index can be added to the sequence parameter set (SPS), and the identifier is used to control the switching of the display mode of the knowledge image. At this time, all knowledge image frames in the sequence are uniformly controlled. In this embodiment, the syntax element library_picture_mode_index is only transmitted in the SPS of the knowledge image. As shown in Table 2, the variable LibraryStreamFlag represents whether it is the SPS of the knowledge image. If it is the SPS of the knowledge image, library_picture_mode_index is transmitted. Among them, library_picture_mode_index = "0" means that all knowledge images in the sequence are not displayed; library_picture_mode_index = "1" means that all knowledge images in the sequence are displayed; library_picture_mode_index = "2" is used to indicate the display mode in the dual-stream mode, that is, the knowledge image has a separate code stream and the conventional image has a separate code stream, which are output separately; library_picture_mode_index = "3" is used to reserve for subsequent display modes that may exist.

[0084] Table 2 A schematic diagram of the sequence parameter set of knowledge images

[0085]

[0086] In addition, it is also possible to determine whether to use patch transmission (i.e., fragment transmission) to process the knowledge image based on the business scenario.

[0087] The business scenario may be determined based on the bit overhead of the knowledge image and / or the image bit rate requirement, etc.

[0088] For example, if the bit rate requirement of the knowledge image is low, the knowledge image can be processed without using the patch transmission method, that is, Figure 1 As shown in , the knowledge image is directly transmitted as a whole frame, that is, the whole frame of the knowledge image is directly encoded into a bit stream; if the bit rate requirement of the knowledge image is high, such as Figure 3 As shown, the knowledge image can be processed by a patch transmission method, that is, the knowledge image is cut into at least two image blocks, and then the at least two image blocks of the knowledge image are respectively encoded into different code streams, so that the decoding end can only decode the knowledge image after obtaining the code stream containing the encoded data of all image blocks of the knowledge image, which can reduce the impact of too large code stream data on the transmission channel. It is more preferred that before transmitting the code stream of the "image encoded with reference to the knowledge image", the code stream package containing the encoded data of all image blocks of the referenced knowledge image will be transmitted, so that the decoding end can decode and obtain the decoded image of the knowledge image before decoding the "image encoded with reference to the knowledge image".

[0089] The code stream requirement of the knowledge image may be related to the video resolution requirement, the image clarity requirement and / or the update frequency of the knowledge image.

[0090] Optionally, if the update frequency of the knowledge image frame is low (for example, lower than the frequency threshold), and a higher bit rate is required to encode the knowledge image, that is, the code stream requirement of the knowledge image is relatively high, then the knowledge image can be processed by patch transmission; if the update frequency of the knowledge image frame is high (for example, higher than the frequency threshold), and the generated knowledge image does not need to have a high image quality, that is, the code stream requirement of the knowledge image is relatively low, then the knowledge image can be transmitted as a whole frame, and the impact on the transmission channel is not great.

[0091] Alternatively, if the video resolution requirement is high (for example, higher than the resolution threshold), a higher bit rate is needed to encode the knowledge image, that is, the bit rate requirement of the knowledge image is relatively high. At this time, the knowledge image can be processed by patch transmission; if the video resolution requirement is low (for example, lower than the resolution threshold), and the generated knowledge image does not need to have a high image quality, that is, the bit rate requirement of the knowledge image is relatively low, then the knowledge image can be transmitted in the entire frame, and the impact on the transmission channel is not great.

[0092] For another example, if the bit overhead of the knowledge image is small (for example, less than the bit threshold), the knowledge image can be processed without using the patch transmission method, that is, the knowledge image can be directly transmitted as a whole frame, that is, the whole frame of the knowledge image can be directly encoded into a code stream; if the bit overhead of the knowledge image is large (for example, greater than the bit threshold), the knowledge image can be processed by using the patch transmission method, that is, the knowledge image is divided into at least two image blocks, and then the at least two image blocks of the knowledge image are respectively encoded into different code streams, which can reduce the impact of too much code stream data on the transmission channel.

[0093] Optionally, whether the original image content of the knowledge image is transmitted in patches or in a whole frame, the processing method may be any of the two processing methods described above, and no limitation is made here. In a specific embodiment, the processing method of the original image content of the knowledge image transmitted in patches may be the first processing method described above. The processing method of the original image content of the knowledge image transmitted in a whole frame may be the second processing method or the first processing method described above.

[0094] Furthermore, it is also possible to determine which of the two processing methods to use based on whether multiple slices of a knowledge image are transmitted continuously during patch transmission. Specifically, if a knowledge image is transmitted continuously in patches, the knowledge image may be used for display or not; and if a knowledge image is transmitted interleaved in patches, the knowledge image is not used for display, that is, the original image content of the knowledge image is processed by the first processing method mentioned above. Among them, if at least one frame of a regular image is inserted and transmitted between multiple slices of a knowledge image, the knowledge image uses the slice interleaved transmission method. If at least one frame of a regular image is not inserted and transmitted between multiple slices of a knowledge image, the knowledge image uses the slice continuous transmission method. In this way, when a knowledge image is transmitted as one patch, or multiple patches are transmitted but not interleaved (for example, Figure 3 When the knowledge image is transmitted in multiple patches (e.g. L0), it can be used as a display frame, while reducing the delay of a frame of knowledge image, and no additional buffering is required; when the knowledge image is transmitted in multiple patches (e.g. Figure 3 L1) as shown, it is not used as a display frame, thus avoiding impact on the transmission channel.

[0095] For example, in some actual business scenarios, the frequency of updating the knowledge image frame can be higher, and the generated knowledge image does not need to have a high image quality. In this case, the knowledge image can be transmitted as a whole frame, and the impact on the transmission channel is not great. In this case, the knowledge image can be displayed directly, and then the image content of the next frame can be encoded. This can reduce the delay of displaying a frame of image, and no additional cache is required. Specifically, Figure 4 As shown, L0 and L1 represent knowledge image frames, the knowledge image with POC=0 is the first frame of the encoded video sequence, and the preset syntax value of the knowledge image with POC=0 is set to the value corresponding to the "knowledge image display scheme", and the RL frame with POC=1 is the second frame image content of the video sequence. In other business scenarios, the update frequency of the knowledge image frame is low, and a higher bit rate is required to encode the knowledge image. In this case, in order to avoid the impact on the transmission channel, the knowledge image needs to be transmitted in patches. At this time, the first processing method can be used to process the original image content corresponding to the knowledge image.

[0096] The present application also provides a video encoding method in another embodiment. In this embodiment, the video encoding method includes the following steps: encoding the original image content of the knowledge image only once to obtain the encoded data of the knowledge image.

[0097] In this implementation, when the original image content of the knowledge image is encoded only once, the decoding end can decode the knowledge image and use it for display.

[0098] In one achievable manner, it can be determined whether to encode a knowledge image or all knowledge images in a sequence only once based on the business scenario. Optionally, the business scenario can be determined based on the bit overhead of the knowledge image and / or the image bit rate requirements. For example, when the bit overhead of the knowledge image is greater than the bit threshold, the knowledge image is encoded twice to obtain the encoded data of the knowledge image and the encoded data of the same content frame, respectively. In this scenario, the knowledge image data decoded by the decoding end is not used for display, and the same content frame of the knowledge image decoded by the decoding end is used for display; when the bit overhead of the knowledge image is less than the bit threshold, the knowledge image is encoded only once to obtain the encoded data of the knowledge image. In this scenario, the knowledge image decoded by the decoding end is used for display. Alternatively, in other achievable manners, it can be assumed that the original image content of all knowledge images is encoded only once.

[0099] In situations where random access is required, such as live broadcast and on-demand broadcast, the decoding end can search forward or backward from the live broadcast time or on-demand broadcast time to find a random access frame, and then start decoding and playing with the random access frame as the random access point. In this way, the random access frame is a displayed frame that needs to be played. The related technology uses conventional I frames and / or RL frames as random access frames. And because the knowledge image is not used for display in the related technology, the related technology does not use the knowledge image as a random access frame. In this way, when the random access frame closest to the random access moment such as the live broadcast moment or on-demand moment is the RL frame, the knowledge image referenced by the RL frame will be decoded first, and then the RL frame will be decoded with the decoded knowledge image as a reference. In this way, at least two frames of images need to be decoded before playback can start from the RL frame.

[0100] Optionally, in the case where the original image content of the knowledge image is encoded only once, since the knowledge image can be used for display, the knowledge image can be used as a random access frame, so that the decoding end can play the knowledge image after decoding it. By using the knowledge image for display as a random access frame, it is possible to decode a frame of the image and then play the video from the random access frame, which can improve the problem of stuttering in the playback of random access frames of the video stream.

[0101] In one technology, the decoding end determines whether each frame is a random access frame by searching the NAL unit (Network Abstraction Layer unit) type of each frame. Among them, the system layer of the decoding end can determine whether each frame is a random access frame by the NAL unit type of each frame. In this case, in order to achieve the purpose of the decoding end using the knowledge image for display as a random access frame, the encoding end can set the NAL unit type of the knowledge image for display before transmitting the video code stream to indicate that the knowledge image can be used as a random access frame.

[0102] For example, as shown in Table 3, a new NAL unit type can be added, such as the NAL unit type "17" in Table 3: display knowledge image, and the NAL unit type of the knowledge image used for display is set to the newly added type "display knowledge image", so that when the system layer at the decoding end finds that the NAL unit type of a knowledge image is 17, it can be known that the knowledge image is a knowledge image for display, and thus it can be used as a random access frame.

[0103] For another example, the NAL unit type of the knowledge image for display can also be set to the NAL unit type of IDR, that is, the NAL unit type of the current IDR (Instantaneous Decoding Refresh) frame can be reused, because IDR itself represents a random access frame, so the knowledge image for display is encoded and transmitted with a NAL type of IDR, indicating that the knowledge image for display can reuse the operation of IDR in random access related operations, and the decoding end can know that the knowledge image for display is a random access frame. It can be understood that although the knowledge image for display is encoded and transmitted with a NAL type of IDR, all of its contents are related to the knowledge image, that is, the knowledge image for display is still processed by knowledge image related decoding and caching and other processing methods.

[0104] Table 3 NAL unit type table

[0105]

[0106] For another example, the nal unit type of the knowledge image for display can also be set to the RAPI type, that is, the current RAPI type is reused, because the RAPI image itself represents a random access frame, so the knowledge image for display is encoded and transmitted with a RAPI type, indicating that the knowledge image for display can reuse the RAPI operation in random access related operations, and the decoding end can know that the knowledge image for display is a random access frame. It can be understood that although the knowledge image for display is encoded and transmitted with a RAPI type, all of its contents are related to the knowledge image, that is, the knowledge image for display is still processed by knowledge image related decoding and caching and other processing methods.

[0107] Among them, when the knowledge image for display is used as a random access frame, the RL frame that refers to the knowledge image for display may not be used as a random access frame, so that the random access frame searched closest to the random access moment such as the live broadcast moment or the on-demand moment is the knowledge image for display, so as to improve the problem of stuttering in the playback of random access frames of the video stream.

[0108] Of course, in some cases, regardless of whether the knowledge image is used for display, the RL frame that references the knowledge image can be used as a random access frame. However, it is more preferred that, when the random access frame that is closest to the random access moment such as the live broadcast moment or the on-demand moment is a RL frame, it can be further confirmed whether the knowledge image referenced by the RL frame is a random access frame; if the knowledge image referenced by the RL frame is not a random access frame, the RL frame is directly used as a random access point, that is, playback starts from the RL frame; and if the knowledge image referenced by the RL frame is a random access frame, the knowledge image referenced by the RL frame can be used as a random access point, that is, playback starts from the knowledge image referenced by the RL frame.

[0109] Of course, in other embodiments, regardless of whether the knowledge image is displayed or not, the knowledge image may not be used as a random access frame, that is, it is specified that the knowledge image is not used as a random access frame. When performing random access in this way, the decoding end still uses the RL frame as a random access frame. Exemplarily, the nal unit type of the knowledge image used for display and the knowledge image not used for display are both set to nal unit type "12" - knowledge image frame, so that the decoding end can know that the image frame is a knowledge image frame and is not used as a random access frame based on the nal unit type "12" - knowledge image frame.

[0110] Among them, before transmitting the video code stream, the video coding data can be encapsulated into NAL packets, each of which will carry a NAL unit type to indicate what type of data is in the current NAL packet, so that the system layer can perform different processing according to different types of data. In this way, for the NAL packet whose data is the data of the knowledge image for display, the type of NAL unit it carries can be set specifically according to the above-mentioned various implementation methods, so that the decoding end can confirm the random access point based on the NAL unit type carried by the NAL packet.

[0111] See also Figure 5 , Figure 5 FIG. 1 is a flow chart of an embodiment of a video decoding method of the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily use Figure 5 The process sequence shown is limited. In this embodiment, the video decoding method includes the following steps:

[0112] S201: Obtaining a preset syntax from a video bitstream.

[0113] Among them, different values ​​of the preset syntax are used to indicate the display mode of the knowledge image.

[0114] S202: Based on a preset syntax, output and display the knowledge image or the same content frame of the knowledge image.

[0115] Among them, outputting and displaying the knowledge image or the same content frame of the knowledge image based on the preset syntax includes: when the value of the preset syntax indicates that the knowledge image is used for display, decoding the knowledge image and outputting and displaying it.

[0116] Optionally, outputting and displaying the knowledge image or the same content frame of the knowledge image based on the preset syntax includes: decoding the knowledge image when the value of the preset syntax indicates that the knowledge image is not for display; and decoding and outputting and displaying the decoded image of the same content frame based on the decoded image of the knowledge image.

[0117] Furthermore, the specific details of the video decoding method correspond to the specific contents of the video encoding method in the above-mentioned embodiment. The relevant contents of the video encoding method in the above-mentioned embodiment may be referred to and will not be elaborated here.

[0118] The knowledge image or the same content frame of the knowledge image can be decoded and output based on the preset syntax. In one implementation, when the value of the preset syntax indicates that the knowledge image is used for display, the knowledge image is decoded; and a decoded image showing the knowledge image is output. In another implementation, when the value of the preset syntax indicates that the knowledge image is not used for display, the knowledge image is decoded; based on the decoded image of the knowledge image, a decoded image of the same content frame is obtained by decoding; and a decoded image showing the same content frame is output.

[0119] See also Figure 6 , Figure 6 FIG. 1 is a flow chart of another embodiment of the video decoding method of the present application. It should be noted that if there is substantially the same result, this embodiment does not use Figure 6 The process sequence shown is limited. In this embodiment, the video decoding method includes the following steps:

[0120] S301: Decode the video code stream to obtain a decoded image of the knowledge image;

[0121] S302: Output a decoded image showing the knowledge image.

[0122] In the related art, if an image needs to be output and displayed, it needs to be placed in the regular image management part of the DPB (decoded picture buffer), and the label "has been output" is used to mark whether the current image has been output and displayed. However, currently, knowledge images are only placed in the knowledge image management part of the DPB, and there is no corresponding label to control the output. Based on this, the present application proposes a variety of implementation methods to enable knowledge images to be output and displayed.

[0123] First, the knowledge image for display can be placed in both the conventional image management part of the DPB and the knowledge image management part of the DPB. The conventional image management part of the DPB is used to manage the output of the knowledge image, and the knowledge image management part of the DPB can be used to allow the knowledge image to be referenced. This is relatively simple and direct to implement. Specifically, the decoding end will place the knowledge image for display in both the conventional image management part of the DPB and the knowledge image management part of the DPB; when the decoding end decodes an image frame that uses the knowledge image for display as a reference frame, it will refer to the knowledge image in the knowledge image management part of the DPB; and the knowledge image for display placed in the conventional image management part of the DPB can also be marked by whether it is "referenced" and whether it is "output". Since the knowledge images for display are also placed in the knowledge image management part of the DPB at the same time, the knowledge images placed in the regular image management part of the DPB may not be referenced, so that they can be marked as "not referenced". In this way, after the knowledge images placed in the regular image management part of the DPB are output and displayed in the playback order, the output and displayed knowledge images in the regular image management part of the DPB can be marked as "output". In this way, the knowledge image has been marked as "not referenced" and "output", so that the knowledge image can be removed from the regular image management part of the DPB.

[0124] Secondly, the knowledge image for display can be directly output after decoding and reconstruction. In this implementation, the knowledge image only needs to be put into the knowledge image management part of the DPB.

[0125] Among them, the decoding end may include DPB and temporary cache, etc. After the knowledge image decoding and reconstruction is completed, the knowledge image can be first placed in the temporary cache, etc., and then the knowledge image in the temporary cache can be output and displayed, that is, the knowledge image can be directly output and displayed by directly outputting the temporary cache. In this way, there is no need to control the output display of the knowledge image through the management method of DPB, so that only the reference of the knowledge image needs to be controlled through DPB, so that the knowledge image only needs to be placed in the knowledge image management part of DPB, so that the knowledge image can be referenced through the knowledge image management part of DPB, so that DPB does not need to store an extra frame of knowledge image content, which reduces some DPB usage overhead compared to the previous implementation method, that is, the DPB usage overhead is not increased.

[0126] In other embodiments, the knowledge image may be directly output and displayed by directly outputting the knowledge image cache which is specially used to store the knowledge image.

[0127] In order to facilitate the output and display of the knowledge image in the temporary cache, a bypass output path can be built for the temporary cache, so that the knowledge image to be output and displayed in the temporary cache can be output and displayed through the output channel. The bypass output path can be a newly built output circuit.

[0128] Thirdly, the knowledge image is still only put into the knowledge image management part of DPB, and a mark of whether it can be exported is added to the knowledge image management part. If the preset syntax of the above implementation mode indicates that it is not displayed, the newly added mark is "not exportable", and if the preset syntax of the above implementation mode indicates display, the newly added mark is "exportable". The knowledge image marked as "exportable" is output and displayed, and the usage overhead of DPB will not be increased at this time. That is, the knowledge image can be marked as "exportable" according to the value of the preset syntax in the above implementation mode. Specifically, if the value of the preset syntax shows that the frame of knowledge image is not used for display, the frame of knowledge image is marked as "not exportable"; if the value of the preset syntax shows that the frame of knowledge image is used for display, the frame of knowledge image is marked as "exportable".

[0129] For example, after a frame of knowledge image is decoded and put into the knowledge image management part of DPB, if the library_picture_mode_index corresponding to the knowledge image is "0", it means that the frame of knowledge image is not used for display, then this frame of knowledge image can be marked as "not outputtable"; library_picture_mode_index is "1", which means that the frame of knowledge image is used for display, then this frame of knowledge image is marked as "outputtable".

[0130] In the above three implementation methods, after decoding a complete frame of knowledge image, the newly decoded frame of knowledge image needs to be put into the knowledge image management part of DPB so that the knowledge image frame can be referenced by the knowledge image management part of DPB.

[0131] Optionally, if after decoding a complete frame of knowledge image, there are still previous knowledge images (i.e., historical knowledge images) in the knowledge image management part of DPB, the previous knowledge image can be moved out and the currently decoded knowledge image can be moved in; if the knowledge image management part of DPB is empty at this time, the currently decoded knowledge image can be directly moved in.

[0132] Of course, in some embodiments, if there are still historical knowledge images in the knowledge image management part of the DPB, the historical knowledge images may not be removed, that is, the currently decoded knowledge images may be directly moved in.

[0133] In addition, conventional images (i.e., images used for display other than knowledge images) can still be output and displayed using the management method of DPB.

[0134] See also Figure 7 , Figure 71 is a schematic diagram of the structure of an embodiment of the encoder of the present application. The encoder 10 includes a processor 12, and the processor 12 is used to execute instructions to implement the above video encoding method. The specific implementation process can be found in the description of the above embodiment, which will not be repeated here.

[0135] The processor 12 may also be referred to as a CPU (Central Processing Unit). The processor 12 may be an integrated circuit chip having signal processing capabilities. The processor 12 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 12 may also be any conventional processor, etc.

[0136] The encoder 10 may further include a memory 11 for storing instructions and data required for the processor 12 to run.

[0137] The processor 12 is used to execute instructions to implement the method provided by any embodiment of the video encoding method of the present application and any non-conflicting combination.

[0138] See also Figure 8 , Figure 8 2 is a schematic diagram of a decoder of the present application. The decoder 20 includes a processor 22, which is used to execute instructions to implement the above video decoding method. The specific implementation process is described in the above embodiment, which will not be repeated here.

[0139] The processor 22 may also be referred to as a CPU (Central Processing Unit). The processor 22 may be an integrated circuit chip having signal processing capabilities. The processor 22 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 22 may also be any conventional processor, etc.

[0140] The decoder 20 may further include a memory 21 for storing instructions and data required for the processor 22 to run.

[0141] The processor 22 is used to execute instructions to implement the method provided by any embodiment of the video decoding method of the present application and any non-conflicting combination.

[0142] See also Fig. 9 , Fig. 9It is a schematic diagram of the structure of a computer-readable storage medium in the implementation mode of the present application. The computer-readable storage medium 30 of the embodiment of the present application stores instructions / program data 31, and when the instructions / program data 31 are executed, the method provided by any embodiment of the video encoding method and video decoding method of the present application and any non-conflicting combination is implemented. Among them, the instructions / program data 31 can form a program file and be stored in the above-mentioned storage medium 30 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) executes all or part of the steps of each implementation method of the present application. The aforementioned storage medium 30 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0145] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0146] The above are only implementation methods of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video encoding method, characterized in that: The method comprises: The original image content of the knowledge image is encoded only once to obtain the encoded data of the knowledge image, wherein the knowledge image is used for display.

2. A video encoding method, characterized in that: The method comprises: Encode the video to obtain encoded data; A preset syntax is added to the encoded data to obtain a video code stream, and different values ​​of the preset syntax are used to indicate a display mode of the knowledge image, and the display mode includes displaying in a single stream or not displaying in a single stream.

3. The video encoding method according to claim 1 or 2, characterized in that: The knowledge image for display can be used as a random access frame.

4. The video encoding method according to claim 3, characterized in that: The method further comprises: The coded data of the knowledge image is encapsulated into at least one network abstraction layer unit packet, and the network abstraction layer unit type carried by the network abstraction layer unit packet of the knowledge image for display indicates that the knowledge image can be used as a random access frame.

5. The video encoding method according to claim 4, characterized in that: The network abstraction layer unit type carried by the network abstraction layer unit packet of the knowledge image for display is a display knowledge image type, an instantaneous decoding refresh frame type or a RAPI type.

6. The video encoding method according to claim 4, characterized in that: The step of encapsulating the encoded data of the knowledge image into at least one network abstraction layer unit packet comprises: The playback sequence number of the knowledge image is encoded into the encoding data of the knowledge image.

7. The video encoding method according to claim 2, characterized in that: The step of encoding the video to obtain encoded data includes: Determine whether to adopt fragmented transmission for the knowledge image; If slice transmission is adopted, the knowledge image is cut into at least two image blocks, and then the at least two image blocks of the knowledge image are respectively encoded into different encoding data; If the fragmented transmission is not adopted, the whole frame of knowledge image is directly encoded into one coded data; The slice transmission includes slice interleaving transmission, and the knowledge image used for display does not use the slice interleaving transmission.

8. A video decoding method, characterized in that: The method comprises: Decode the video code stream to obtain a decoded image of the knowledge image; Output is a decoded image showing the knowledge image.

9. The video decoding method according to claim 8, characterized in that: The output displays the decoded image of the knowledge image, which previously includes: Get the preset syntax from the video stream; If the value of the preset syntax indicates that the knowledge image is for display, the step of outputting a decoded image that displays the knowledge image is performed; If the value of the preset syntax indicates that the knowledge image is not for display, a decoded image of the same content frame is obtained by decoding based on the decoded image of the knowledge image; the decoded image of the same content frame is output and displayed, and the knowledge image and its corresponding same content frame correspond to the same original image content.

10. The video decoding method according to claim 9, characterized in that: The output displays a decoded image of the knowledge image, including: Put the decoded image of the knowledge image for display into the conventional image management part of the decoded image buffer and the knowledge image management part of the decoded image buffer at the same time; use the output flag in the conventional image management part to control the output display of the decoded image of the knowledge image in the conventional image management part; or, The decoded image of the knowledge image is placed in the knowledge image management part of the decoded image buffer area, and the knowledge image is directly output and displayed after decoding and reconstruction is completed; or, The decoded image of the knowledge image is placed in the knowledge image management part of the decoded picture cache area, and the decoded image of the knowledge image is marked with a preset mark based on the value of the preset syntax, and the preset mark is used to indicate whether the decoded image of the knowledge image is output and displayed.

11. The video decoding method according to claim 8, characterized in that: The output displays a decoded image of the knowledge image, including: If a historical knowledge image is stored in the knowledge image management part of the decoded picture buffer area, the historical knowledge image is removed and the decoded image of the knowledge image is moved in.

12. An encoder, characterized in that: The encoder comprises a processor; the processor is used to execute instructions to implement the steps of the method according to any one of claims 1-7.

13. A decoder, characterized in that: The decoder comprises a processor; the processor is used to execute instructions to implement the steps of the method according to any one of claims 8-11.

14. A computer-readable storage medium having instructions / program data stored thereon, characterized in that: When the instructions / program data are executed, the steps of the method according to any one of claims 1 to 11 are implemented.

Images