Video encoding methods, video decoding methods, encoders, decoders, and media

Abstract

This application proposes a video encoding method, a video decoding method, a video encoder, a video decoder, and a computer storage medium based on knowledge images. The video encoding method includes: encoding several video frames and their reference knowledge image frames to obtain a first video bitstream, wherein the first video bitstream includes a video encoding layer and a network abstraction layer; obtaining all fragments of each knowledge image frame based on the network abstraction layer; placing all fragments of each knowledge image frame at the same position in the first video bitstream; and generating a second video bitstream based on several video frames and the knowledge image frames after position shifts, and storing it. By collecting knowledge image fragment streams using the above video encoding method, fast local playback can be achieved, reducing parsing costs and thus reducing latency.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding technology, and in particular to a video encoding method, a video decoding method, a video encoder, a video decoder, and a computer storage medium based on knowledge images. Background Technology

[0002] In video encoding and decoding, to improve compression ratio and reduce the number of codewords to be transmitted, the encoder does not directly encode and transmit pixel values. Instead, it uses intra-frame or inter-frame prediction modes to predict the pixel values ​​of the current block using reconstructed pixels from the coded blocks of the current frame or a reference frame. The pixel value predicted using a certain prediction mode is called the predicted pixel value, and the difference between the predicted pixel value and the original pixel value is called the residual. The encoder only needs to encode a certain prediction mode and the residual generated when using that prediction mode, and the decoder can decode the corresponding pixel value based on this bitstream information. This greatly reduces the number of codewords required for encoding.

[0003] The existing SVAC3 video codec standard introduces the concept of a library picture, with L-frames in subsequent proposals representing library pictures. A library picture is a long-term reference frame encoded using I-frames. It serves only as a reference frame and is not used for display. Library pictures are identified by their Library Picture Index (IDX), rather than the Point of Content (POC) or DOI of other frames in the bitstream.

[0004] However, the existing knowledge image stream organization method, which encodes and stores the data directly, presents the following problems for local playback: In scenarios such as playback and fast forward, multiple parsing operations are required in the bitstream to collect all the knowledge image slices before decoding can begin. The parsing cost is related to the number of knowledge image slices, which in turn affects playback latency. Summary of the Invention

[0005] To address the aforementioned technical problems, this application proposes a video encoding method, a video decoding method, a video encoder, a video decoder, and a computer storage medium based on knowledge images.

[0006] To address the aforementioned technical problems, this application proposes a video coding method based on knowledge images, the video coding method comprising:

[0007] Encode several video frames and their reference knowledge image frames to obtain a first video stream, wherein the first video stream includes a video coding layer and a network abstraction layer;

[0008] Based on the network abstraction layer, all fragments of each knowledge image frame are obtained;

[0009] Place all segments of each knowledge image frame in the same position in the first video stream;

[0010] A second video stream is generated and stored based on several video frames and knowledge image frames after the position is moved.

[0011] The step of placing all segments of each knowledge image frame at the same position in the first video bitstream includes:

[0012] All segments of each knowledge image frame are placed at the same position in the first video stream before the video frame referencing the knowledge image frame.

[0013] The step of placing all segments of each knowledge image frame at the same position in the first video bitstream includes:

[0014] Place all segments of each knowledge image frame in the original position of the first segment in the first video stream.

[0015] The step of placing all segments of each knowledge image frame at the same position in the first video bitstream includes:

[0016] Place all segments of each knowledge image frame in the position of the original last segment in the first video stream.

[0017] The video encoding method further includes:

[0018] In response to a remote streaming playback request, the second video stream is converted into the third video stream, wherein the third video stream may be the same as or different from the complete first video stream;

[0019] The third video stream is transmitted to the user terminal via the network.

[0020] The third video stream consists of the target video frame corresponding to the remote streaming playback request in the first video stream and the knowledge image frame it references.

[0021] The step of obtaining all slices of each knowledge image frame based on the network abstraction layer includes:

[0022] Obtain the slice network abstraction layer for each knowledge image frame slice;

[0023] Based on the segmentation network abstraction layer, obtain the target knowledge image frame segment that belongs to the knowledge image end segment;

[0024] Based on the target knowledge image frame segmentation, determine all segments of each knowledge image frame.

[0025] The step of obtaining all slices of each knowledge image frame based on the network abstraction layer includes:

[0026] Obtain the slice network abstraction layer for each knowledge image frame slice;

[0027] Based on the segmentation network abstraction layer, target knowledge image frame segments belonging to knowledge image coding delimitation segments are obtained;

[0028] According to the target knowledge image frame segmentation, determine all segments of each knowledge image frame;

[0029] The knowledge image encoding delimiter is either the first knowledge image segment or the last knowledge image segment.

[0030] The step of obtaining all slices of each knowledge image frame based on the network abstraction layer includes:

[0031] Obtain the set of image parameters shared by all segments of each knowledge image frame;

[0032] Syntax for obtaining the number of slices of the image frame marked in the image parameter set;

[0033] All fragments of each knowledge image frame are obtained according to the number of fragments in the syntax.

[0034] The step of obtaining all fragments of each knowledge image frame according to the number of fragments in the syntax includes:

[0035] The number of fragments for each knowledge image frame is calculated based on the sum of the number of fragments in the syntax and the preset number.

[0036] All fragments of each knowledge image frame are determined according to the number of fragments in each knowledge image frame.

[0037] The step of obtaining all slices of each knowledge image frame based on the network abstraction layer includes:

[0038] Obtain the set of image parameters shared by all segments of each knowledge image frame;

[0039] Syntax for obtaining the total length of the bitstream of the labeled knowledge image frames in the image parameter set;

[0040] All fragments of each knowledge image frame are obtained according to the total length of the bitstream in the syntax.

[0041] To address the aforementioned technical problems, this application proposes a video decoding method based on knowledge images, the video decoding method comprising:

[0042] Acquire a video stream, wherein the video stream includes a video encoding layer and a network abstraction layer;

[0043] Based on the network abstraction layer, all fragments of each knowledge image frame are obtained, wherein all fragments of each knowledge image frame are stored in the same location;

[0044] Decode the knowledge image frames in the video stream according to all segments of each knowledge image frame;

[0045] Decode other video frames in the video stream according to the knowledge image frames.

[0046] To address the aforementioned technical problems, this application also proposes a video encoder, which includes a memory and a processor coupled to the memory;

[0047] The memory is used to store program data, and the processor is used to execute the program data to implement the video encoding method described above.

[0048] To address the aforementioned technical problems, this application also proposes a video decoder, which includes a memory and a processor coupled to the memory;

[0049] The memory is used to store program data, and the processor is used to execute the program data to implement the video decoding method described above.

[0050] To address the aforementioned technical problems, this application also proposes a computer storage medium for storing program data, which, when executed by a computer, is used to implement the aforementioned video encoding method and / or video decoding method.

[0051] Compared with existing technologies, the beneficial effects of this application are as follows: A video encoder encodes several video frames and their reference knowledge image frames to obtain a first video stream, wherein the first video stream includes a video coding layer and a network abstraction layer; based on the network abstraction layer, all fragments of each knowledge image frame are obtained; all fragments of each knowledge image frame are placed at the same position in the first video stream; a second video stream is generated and stored according to several video frames and the knowledge image frames after position shifting. Through the above video encoding method, the knowledge image fragment stream is collected, enabling fast local playback and reducing parsing costs, thereby reducing latency. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] in:

[0054] Figure 1 This is a schematic diagram of an embodiment of the frame reference relationship in a bitstream containing I-frames and P-frames provided in this application;

[0055] Figure 2 This is a schematic diagram of an embodiment of the frame reference relationship in a bitstream containing I-frames, P-frames and B-frames provided in this application;

[0056] Figure 3 This is a schematic diagram of another embodiment of the frame reference relationship in a bitstream containing I-frames and P-frames provided in this application;

[0057] Figure 4 This is a schematic diagram of an embodiment of the frame reference relationship and the position of the knowledge image in the bitstream under the IPPP configuration in the prior art;

[0058] Figure 5 This is a schematic diagram of another embodiment of the frame reference relationship and the position of the knowledge image in the bitstream under the IPPP configuration in the prior art;

[0059] Figure 6 This is a schematic diagram of the video encoding / decoding and storage related business scenarios provided in this application;

[0060] Figure 7 This is a flowchart illustrating an embodiment of the video encoding method provided in this application;

[0061] Figure 8 This is a flowchart illustrating a specific business scenario of the video encoding method in this application;

[0062] Figure 9 This is a schematic diagram of an embodiment of the frame reference relationship and the position of the knowledge image in the bitstream provided in this application;

[0063] Figure 10 This is a schematic diagram of the LDPB update process during the encoding or decoding process provided in this application;

[0064] Figure 11 This is a schematic diagram of an embodiment of the knowledge image reconstruction process at storage provided in this application;

[0065] Figure 12 This is a schematic diagram of another embodiment of the knowledge image reconstruction process at the storage location provided in this application;

[0066] Figure 13 This is a schematic diagram of an embodiment of the knowledge image reconstruction process for remote streaming playback provided in this application;

[0067] Figure 14 This is a flowchart illustrating an embodiment of the video decoding method provided in this application;

[0068] Figure 15 This is a schematic diagram of the structure of an embodiment of the video encoder provided in this application;

[0069] Figure 16 This is a schematic diagram of the structure of an embodiment of the video decoder provided in this application;

[0070] Figure 17 This is a schematic diagram of the structure of an embodiment of the computer storage medium provided in this application. Detailed Implementation

[0071] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0072] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0073] Video image data is relatively large, so it is usually necessary to compress the video pixel data (RGB, YUV, etc.). The compressed data is called a video stream, which is transmitted to the user's end via wired or wireless network for decoding and viewing. The entire video encoding process includes prediction, transformation, quantization, and encoding.

[0074] In video encoding, the most commonly used color encoding methods include YUV and RGB. The color encoding method used in this invention is YUV. Y represents luminance, which is the grayscale value of the image; U and V (i.e., Cb and Cr) represent chrominance, which are used to describe the color and saturation of the image. Each Y luminance block corresponds to one Cb and one Cr chrominance block, and each chrominance block also corresponds to only one luminance block.

[0075] A video stream consists of consecutive frames. Each frame is decoded and played in sequence to form a video picture. Common frame types in existing video encoding and decoding standards include I-frames, P-frames, and B-frames.

[0076] I-frames are intra-coded frames, meaning they are independent frames with all their own encoding and decoding information, and can be encoded and decoded independently without referencing other frames. I-frames require complete encoding of all content within the frame, generally resulting in a larger bitstream and lower compression ratio.

[0077] P-frames are inter-frame predictive coded frames, which require reference frames in the display order as reference images for encoding and decoding.

[0078] B-frames are bidirectional inter-frame predictive coded frames, requiring reference frames from the past and future in the display order for encoding and decoding.

[0079] Figure 1 This is a schematic diagram of the frame reference relationship in a bitstream containing I-frames and P-frames. Figure 2 This is a schematic diagram of the frame reference relationship in a bitstream containing I-frames, P-frames, and B-frames. Figure 1 and Figure 2 In this context, POC (pic_order_cnt) represents the playback order of video frames, and DOI (decode order index) represents the encoding and decoding order of video frames. From... Figure 2 As can be seen, when B-frames are present in the bitstream, the encoding / decoding order and playback order of the frames can be different.

[0080] It should be noted that the frame reference relationships in a video sequence can be combined in various ways. Figure 1 and Figure 2 This is just to illustrate a common reference relationship.

[0081] In video playback, random access is often required. For example, when frames are dropped during live streaming, the decoder cannot establish reference relationships between frames and cannot decode normally. In this case, the decoder will search for the next randomly accessed frame and start decoding again. Or, in video-on-demand, when playing video from a certain moment, the decoder generally starts decoding from the previous randomly accessed frame corresponding to that moment.

[0082] To meet the requirements of random access, the encoder periodically inserts a random access frame into the bitstream. In existing technologies, random access frames are all I-frames. However, I-frames generally have a large bitstream size, which significantly impacts bandwidth during transmission, so they are typically not inserted frequently during encoding. But the larger the periodic interval for inserting I-frames, the longer the waiting time during random access. The encoder needs to make a trade-off and choose an appropriate I-frame period. One solution to this problem in existing technologies is to insert P-frames or B-frames that only reference the previous random access frame between random access frames.

[0083] For ease of understanding, in the following description, frames with POC values ​​of 0, 1, 2... will be referred to as frames 0, 1, 2... . Figure 3 In this code, frame 0 is an I-frame, and frames 1, 2, 3, and 4 are P-frames, with frame 3 directly referencing the I-frame. If frame 1 or 2 is lost, as long as the I-frame is still in the decoder, it will not affect the normal decoding of frames 3 and 4.

[0084] When playing video on demand, if you want to randomly play the 3rd frame, you only need to read and decode frames 0 and 3. And... Figure 1 In the reference model, three frames (0, 1, 2) need to be read and decoded before the third frame can be decoded to obtain the image.

[0085] It is evident that inserting P-frames or B-frames that only reference the previous random access frame can reduce the reading and decoding overhead required for random access.

[0086] Specifically, an IDR frame is a type of random access frame. An IDR frame is a special type of I-frame that, in addition to enabling random access, will clear all buffered frames in the encoding / decoding buffer if encountered during encoding / decoding. Therefore, frames encoded / decoded after an IDR frame will not reference frames encoded / decoded before the IDR frame.

[0087] The existing SVAC3 video codec standard introduces the concept of a knowledge picture (library picture), with L-frames in subsequent proposals representing knowledge pictures. Simultaneously, the concept of RL (reference library) frames is introduced; RL frames refer to P-frames or B-frames that reference only the knowledge picture. A knowledge picture is a long-term reference frame encoded using I-frames. It serves only as a reference frame and is not used for display. Knowledge pictures are identified by their knowledge picture index (IDX), rather than the POC or DOI of other frames in the bitstream.

[0088] Regarding the structure of knowledge images in the bitstream: In existing technologies, knowledge images are encoded using I-frames. However, because the QP (Queries Per Frame) of encoded knowledge images is generally small, encoding is slow, and the bitrate is generally high, interleaving an entire frame of knowledge image bitstream into the bitstream can cause significant bitrate spikes and decoding jitter. Therefore, the patch mechanism in the existing SVAC3 standard is utilized. The knowledge image is divided into multiple patches, which are then interleaved with multiple display images. Only one patch is encoded at a time and added to the bitstream, ultimately resulting in an encoded output bitstream that interleaves the knowledge base patch bitstream with the display image bitstream. (The following text is incomplete and requires further context.) Figure 4 and Figure 5 Examples of frame reference relationships and knowledge images in the bitstream under IPPP configuration in existing technologies.

[0089] Regarding the management of knowledge image cache frames: In the existing technology, when an image starts to reference a new knowledge image or starts to encode / decode a new knowledge image, the previous knowledge image is replaced. That is to say, all frames in the current sequence can refer to at most one knowledge image, and that knowledge image must be the most recently encoded / decoded knowledge image.

[0090] Regarding the indexing of knowledge images: In the existing technology, knowledge images support an index of 0-511, that is, only 512 frames of knowledge images are supported to be referenced by the entire sequence of frames.

[0091] Regarding the configuration of knowledge image frames (L frames) and RL frames: In the prior art, knowledge images are encoded as P frames or B frames by treating all original I frames as RL frames according to the I frame interval.

[0092] The Reference Picture Lists (RPL) record the reference relationships for each frame and are used to update the reference picture buffer and clean up subsequent frames that are no longer referenced. In SVAC3, the syntax of the RPL is as follows:

[0093] RPLs include RPL0 and RPL1, which record the preceding and following reference frames, respectively. The following syntax is for a specific RPL0 / RPL1. As you can see, the RPL records the number of reference frames (num_of_ref_pic) and marks whether each reference frame is a knowledge image (library_index_flag). For knowledge image reference frames, it records their index (referenced_library_picture_index); for other non-knowledge image reference frames, it records (abs_delta_doi), from which the index can be further calculated. The following table gives the syntactic definition of RPLs and their meanings:

[0094]

[0095]

[0096] For each frame of an image, there is a corresponding Reference Frame Layout (RPL). The RPL contains all reference frames used by the current frame and subsequent frames. The reference frames used by the current frame are the first num_ref_default_active_minus1 reference frames among these frames. num_ref_default_active_minus1, like the RPL, is also expressed in the syntax. That is, the RPL of a frame includes:

[0097] (1) The number of reference frames used in the current frame and the frames after the current frame.

[0098] (2) Mark all reference frames as knowledge image frames.

[0099] (3) Reference index of all reference frames.

[0100] At the same time, each frame can determine, based on the syntax num_ref_default_active_minus1, which are the first few frames among all RPL reference frames that the current frame needs to reference.

[0101] The SVAC3 NAL Unit includes a 3-byte (24-bit) prefix of 0x000001, and the following NAL header-related syntax: 1 bit of forbidden_zero_bit, 1 bit of nal_ref_idc, 4 bits of nal_unit_type, 1 bit of encryption_idc, and 1 bit of authentication_idc. The NAL header-related syntax is 1 byte (8 bits), so the NAL unit is 2 bytes in total. The NAL header-related syntax is shown below:

[0102]

[0103] The 4-bit nal_unit_type can have 16 possible values, and the uses of these 16 values ​​are as follows:

[0104]

[0105]

[0106] This application proposes a method for reconstructing knowledge image streams by addressing changes in the organization of the knowledge image stream during transmission and storage. Specifically, it includes the specific format of the reconstructed knowledge image stream and the reconstructing method during storage. A concrete practical application scenario of this knowledge image stream reconstructing method is as follows: Figure 6As shown, the image is captured and encoded at the acquisition end, and the encoded bitstream is transmitted over the network and stored at the storage end. There are two service modes at the storage end: local playback and remote streaming playback.

[0107] Please refer to the details. Figure 7 and Figure 8 , Figure 7 This is a flowchart illustrating an embodiment of the video encoding method provided in this application. Figure 8 This is a flowchart illustrating a specific business scenario of the video encoding method described in this application.

[0108] like Figure 1 As shown, the specific steps are as follows:

[0109] Step S11: Encode several video frames and their reference knowledge image frames to obtain a first video stream, wherein the first video stream includes a video coding layer and a network abstraction layer.

[0110] In this embodiment, the video encoder encodes several video frames and their referenced knowledge image frames to obtain a first video bitstream. The encoding process for the current video frame is as follows:

[0111] When the video encoder encodes the current video frame, it extracts the RPL information of the current video frame, namely the first reference image queue configuration set. The first reference image queue configuration set includes the reference frame index of the current video frame, which includes reference knowledge image frame indexes and / or reference non-knowledge image frame indexes.

[0112] The video encoder obtains the reference frame index of the current video frame based on the first reference image queue configuration set.

[0113] In this embodiment, the RPL information records the knowledge image frame and the non-knowledge image frame referenced by the current video frame. Please refer to [link / reference] for details. Figure 9 , Figure 9 The RPL information of frame 1 (P) records the index of frame 0 (RL) and the index of frame 0 (L). The knowledge image of frame 1 is generated from the original image of frame 3 in the original sequence.

[0114] In existing technologies, the buffer can only hold one knowledge image frame at a time, and the video frame to be encoded can only reference the most recent knowledge image frame, i.e. Figure 6 In the prior art, the 9th frame P can only refer to the 8th frame RL and the 1st frame L, because only the 1st frame L is cached in the buffer, and the 0th frame L has been deleted.

[0115] This application uses LDPB to manage multiple L-frames, allowing multiple L-frames to be cached simultaneously in the buffer. Figure 9As shown, the reference frame index of frame 9 (P frame) includes frame 8 (RL frame) and frame 0 (L frame).

[0116] The video encoder retrieves reference knowledge image frames from the buffer based on the reference frame index, and encodes the current video frame according to the reference knowledge image frames to obtain the encoded bitstream of the current video frame.

[0117] In this embodiment, the video encoder obtains a reference knowledge image frame from the buffer using a reference frame index. The reference knowledge image frame is then used as a reference frame to encode the current video frame, thereby obtaining the video bitstream of the current video frame.

[0118] In addition, if the reference frame index also includes the index of non-knowledge image frames, that is, the index of other P frames, other B frames or other RL frames, the video encoder can also obtain other reference non-knowledge image frames, and use the reference non-knowledge image frames pointed to by the index and the reference knowledge image frames together to encode the current video frame, thereby obtaining the video bitstream of the current video frame.

[0119] The L-frame management mechanism mentioned above is called LDPB. It can manage multiple L-frames, meaning that multiple L-frames can exist during sequence encoding / decoding and be selected for use by subsequent encoded / decoded frames. The implementation process of this management mechanism is as follows:

[0120] First, a syntax needs to be added to specify the length of the LDPB and the maximum number of L frames that can exist in the buffer at the same time.

[0121] Secondly, for each L-frame in LDPB, its status is marked, including "referenced" and "not referenced", which respectively indicate whether the frame is referenced by the current encoding / decoding frame.

[0122] Please refer to the details. Figure 10 , Figure 10 This is a schematic diagram illustrating the LDPB update process during the encoding or decoding process provided in this application. Details are as follows:

[0123] 1. Before encoding or decoding, the referenced status of all L frames in the LDPB needs to be marked according to the RPL information. If the frame in the current LDPB is not included in the RPL of the current frame, it needs to be marked as "not referenced"; otherwise, the status remains "referenced".

[0124] 2. Frames marked as "not referenced" in the LDPB indicate that subsequent frames will not reference this frame, meaning you can exit the LDPB. Clear the "not referenced" frames from the LDPB, and keep the "referenced" frames.

[0125] 3. If the current frame is an L-frame, after encoding or decoding this L-frame, the generated reconstructed frame is added to the LDPB and marked as "referenced".

[0126] The LDPB management mechanism is described below through a specific implementation method:

[0127] Taking the bitstream structure in SVAC3 as an example. In existing monitoring scenarios, when a PTZ camera is used to capture images from multiple angles at a fixed point, for example, a device cyclically captures two fixed scenes, the knowledge image generated the first time for each scene can be kept in the buffer, reducing the need for repeated transmission of the same scene's knowledge image multiple times. Please continue reading. Figure 9 , Figure 9 This is a configuration where subsequent frames reference the knowledge image from previous frames. For example, in frame 0, the camera captures the first location and generates knowledge image 0; in frame 3, it captures the second location and generates knowledge image 1; and in frame 8, it switches back to the first location. In this case, the knowledge image in frame 0 can be directly referenced, eliminating the need to retransmit a new knowledge image.

[0128] After encoding, the first video stream includes a video coding layer and a network abstraction layer. The video coding layer is responsible for effectively representing the content of the video data, while the network abstraction layer is responsible for formatting the data and providing header information to ensure the data is suitable for transmission on various channels and storage media. The video coding method of this application quickly indicates the end of the image collection process at the network abstraction layer by adding syntax or end-of-episode markers, thereby rapidly realizing the knowledge image reconstruction process.

[0129] Step S12: Obtain all fragments of each knowledge image frame based on the network abstraction layer.

[0130] In the embodiments of this application, the video encoder employs a method for quickly indicating the end of the slice collection process, including but not limited to:

[0131] (1) Add a new NAL type for slices, and mark the current slice as the last knowledge image slice.

[0132] (2) Add a new NAL type for CRR slices, marking the current slice as the first or last knowledge image slice.

[0133] (3) Add syntax to the image header to mark the total number of knowledge image pieces.

[0134] (4) Add syntax to the image header to mark the total length of the knowledge image stream.

[0135] Among them, if the method of adding NAL type is adopted, namely (1) and (2) above, the specific organization form of NAL includes but is not limited to:

[0136] (1) Increase the total number of bytes in the NAL unit to N (N>1), and add a new number to the NAL type in the NAL unit to represent the above type. If the total number of syntactic bits in the NAL unit after the increase does not reach the total number of bytes N, then set the remaining bits as reserved bits, which can be used for purposes including but not limited to the following: time domain scalability, spatial domain scalability, quality scalability, encryption authentication, etc.

[0137] (2) Use the reserved position number in the existing NAL type and set it to the above type.

[0138] (3) Replace a certain type in the existing NAL type, use its number, and set it to the above type.

[0139] The four termination methods described above will be introduced below with specific examples:

[0140] (1) Add a new NAL type for slices, and mark the current slice as the last knowledge image slice.

[0141] The following is a specific example based on the SVAC3 standard.

[0142] First, the total number of bytes in the NAL unit is expanded to 2 bytes, and the nal_unit_type in the NAL is expanded to 5 bits, that is, the value range of nal_unit_type is 0 to 31.

[0143]

[0144] The specific values ​​for nal_unit_type are as follows:

[0145] 0 reserve 1 non-IDR image coded patch_data_rbsp() 2 IDR image coded patch_data_rbsp() 3 SVC enhancement layer coded patch_data_rbsp() for non-IDR images 4 SVC enhancement layer coded patch_data_rbsp() for IDR image 5 Monitoring extended data unit surveillance_extension_rbsp() 6 Supplemental enhancement information sei_rbsp() 7 Sequence parameter set seq_parameter_set_rbsp() 8 Image parameter set pic_parameter_set_rbsp() 9 Security parameter set sec_parameter_set_rbsp() 10 authentication data authentication_data_rbsp() 11 end of stream_of_stream_rbsp() 12 Knowledge Image Frame patch_data_rbsp() 13 Reserved for SVAC audio 14 Knowledge Image DRAP Access Point, RL Frame patch_data_rbsp() 15 SVC enhancement layer image parameter set pic_parameter_set_rbsp() 16 Knowledge image end patch end_patch_data_rbsp() 17～31 reserve

[0146] Specifically, the syntax definition of knowledge image end_patch_data_rbsp() is added to nal_unit_type of the end method (1), thereby marking the current slice as the last knowledge image slice.

[0147] (2) Add a new CRR end NAL type to mark the current slice as the first or last knowledge image slice.

[0148] The following is a specific example based on the SVAC3 standard.

[0149] First, the total number of bytes in the NAL unit is expanded to 2 bytes, and the nal_unit_type in the NAL is expanded to 5 bits, that is, the value range of nal_unit_type is 0 to 31.

[0150]

[0151]

[0152] The specific values ​​for nal_unit_type are as follows:

[0153]

[0154] Specifically, the syntax definition of knowledge image encoding delimiter patch_data_delimiter_rbsp() is added to nal_unit_type in the termination method (2), thereby marking the current patch as the first knowledge image patch or the last knowledge image patch.

[0155] For example, for the first and last knowledge image coded pieces, its nal_unit_type is patch_data_delimiter_rbsp(), and for the other knowledge image coded pieces, its nal_unit_type is patch_data_rbsp().

[0156] (3) Add a num syntax to the image header to mark the total number of knowledge image pieces.

[0157] The following is a specific example based on the SVAC3 standard:

[0158]

[0159] Among them, library_patch_num_minus1 represents the number of knowledge image frame slices minus one, with a value range of 0 to 255.

[0160] In other implementations, `library_patch_num_minusn` can represent the number of knowledge image frame fragments minus `n`, where `n` is a non-zero integer. By setting `minus`, the amount of data stored in the syntax can be effectively reduced. For example, if the number of knowledge image frame fragments is 10, the value corresponding to `library_patch_num_minus1` would be stored as 9. During decoding, the video decoder only needs to add 1 to the stored value to obtain the number of knowledge image frame fragments.

[0161] (4) Add a size syntax to the image header to mark the total length of the knowledge image stream.

[0162] The following is a specific example based on the SVAC3 standard:

[0163]

[0164] Where library_picture_size represents the total length of the knowledge image stream, with a value ranging from 0 to 2. 32 -1.

[0165] Step S13: Place all segments of each knowledge image frame in the same position in the first video stream.

[0166] In this embodiment, the video encoder adds a knowledge image reassembly process to the storage end, adjusts the bitstream order, and collects the knowledge image slices together for storage.

[0167] The knowledge image reassembly method of this application addresses the changes in the organization of the knowledge image stream throughout the entire business scenario as follows: After image acquisition and encoding, the knowledge image stream is organized in the existing manner to ensure stable bitrate during transmission; a knowledge image reassembly process is added before the storage process to collect the knowledge image fragments together; for remote users' streaming playback needs, the stream needs to be transmitted to the user end via the network, and the stored stream is reorganized and transmitted to maintain stable bitrate.

[0168] Among them, knowledge image reconstruction methods include, but are not limited to:

[0169] (1) The collected knowledge image stream is placed in the original position of the first segment in the bit stream, which facilitates fast decoding and playback during on-demand playback.

[0170] (2) Place the collected knowledge image stream at any position before the first frame in the stream that needs to reference it.

[0171] (3) Place the collected knowledge image stream in the original last segment position.

[0172] Specifically, a schematic diagram of an example of the knowledge image reconstructing process at storage is shown below. Figure 11 This demonstrates the changing process of knowledge image bitstream organization across various business scenarios.

[0173] like Figure 11 As shown, after image acquisition and encoding, the knowledge image stream is organized in the existing manner to ensure stable bit rate during transmission.

[0174] A knowledge image reassembly process is added before the storage process to collect knowledge image fragments together. The collected knowledge image stream is then placed in the original position of the first fragment in the bitstream, facilitating fast decoding and playback during on-demand viewing.

[0175] For remote users' streaming playback needs, the bitstream needs to be transmitted to the user's end via the network. When transmitting to the knowledge image stream, the first segment is transmitted first, and the remaining segments are transmitted in the same way in subsequent display frames.

[0176] Placing the shard in the original first shard position can avoid the following problems:

[0177] (1) In the case of streaming, if the knowledge image piece is not placed in the original first fragment position, the position where the knowledge image piece should start to be transmitted needs to be calculated in advance, which increases the delay.

[0178] (2) In the case of streaming, if the knowledge image piece that needs to be transmitted at the same time as the current display frame is after this frame, it is necessary to search for the knowledge image stream backward, which increases the delay.

[0179] (3) After the display frame is transmitted quickly, several remaining knowledge image pieces are not transmitted, the bit rate suddenly increases, and the delay increases.

[0180] Specifically, another example of the knowledge image reconstructing process at storage is illustrated as follows: Figure 12 This demonstrates the changing process of knowledge image bitstream organization across various business scenarios.

[0181] like Figure 12 As shown, after image acquisition and encoding, the knowledge image stream is organized in the existing manner to ensure stable bit rate during transmission.

[0182] A knowledge image reassembly process is added before the storage process to collect knowledge image fragments together. The collected knowledge image stream is then placed at the position of the last fragment before the original first frame that needs to be referenced, which facilitates fast decoding and playback during on-demand viewing.

[0183] For remote users' streaming playback needs, the bitstream needs to be transmitted to the user's end via the network. The position where each knowledge image needs to be transmitted earliest is calculated, for example, after POC4 in this case. When this frame is transmitted, the knowledge image stream is searched backward and a slice is transmitted. Then, the display frame and the knowledge image slice are transmitted again. The knowledge image slice stream is interspersed in subsequent display frames in the original way.

[0184] Step S14: Generate a second video stream and store it according to several video frames and knowledge image frames after the position is moved.

[0185] Furthermore, such as Figure 6 and Figure 8 As shown, in the scenario of remote user streaming playback, for the remote user's streaming playback needs, the video encoder reassembles the knowledge image bitstream into a bitstream for network transmission, that is, the second video bitstream is reassembled into the first video bitstream and decoded for remote viewing by the user through network transmission.

[0186] In addition, the video encoder can reorganize the bitstream format according to specific playback requirements; please refer to [link / reference needed]. Figure 13 , Figure 13This is a schematic diagram of an embodiment of the knowledge image reconstruction process for remote streaming playback provided in this application.

[0187] like Figure 13 As shown, in this embodiment, the playback request is randomly accessed from the RL frame of POC8. In this case, only the RL frame and its reference L frame need to be transmitted. The reference L frame is searched backward from the RL frame. After the L frame is transmitted as a whole or in segments, the RL frame of POC8 and the subsequent display frames are transmitted. By parsing the specific remote streaming playback request, the video encoder can encode only the portion of the video frames that the user needs to play and their reference frames, instead of encoding the entire video every time, which can effectively improve encoding and transmission efficiency.

[0188] In this embodiment, a video encoder encodes several video frames and their reference knowledge image frames to obtain a first video stream, wherein the first video stream includes a video coding layer and a network abstraction layer; based on the network abstraction layer, all fragments of each knowledge image frame are obtained; all fragments of each knowledge image frame are placed at the same position in the first video stream; and a second video stream is generated and stored according to the several video frames and the knowledge image frames after the position is moved. By using the above video encoding method to collect knowledge image fragment streams, local fast playback can be achieved, reducing parsing costs and thus reducing latency.

[0189] This application proposes adding a knowledge image reassembly method to storage applications. By collecting knowledge image fragment streams and grouping them together, local fast playback can be achieved, reducing parsing costs and thus latency. This application also proposes to rapidly reassemble knowledge images by adding knowledge image fragment types or adding syntactic indicators to the image header to indicate the end of the fragment collection process. This allows for the rapid collection of all knowledge image fragments with minimal parsing cost, thereby reducing latency.

[0190] Please continue reading. Figure 14 , Figure 14 This is a flowchart illustrating an embodiment of the video decoding method provided in this application.

[0191] like Figure 14 As shown, the specific steps are as follows:

[0192] Step S21: Obtain the video stream, which includes a video encoding layer and a network abstraction layer.

[0193] Step S22: Obtain all fragments of each knowledge image frame based on the network abstraction layer, wherein all fragments of each knowledge image frame are stored in the same location.

[0194] Step S23: Decode the knowledge image frames in the video stream according to all segments of each knowledge image frame.

[0195] Step S24: Decode other video frames in the video stream according to the knowledge image frames.

[0196] The video decoding method of this application is applied to a local playback scenario where the video decoder is used. By decoding all segments of the knowledge image frame stored in the same location, fast local playback is achieved, reducing parsing costs and thus reducing latency.

[0197] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0198] To implement the above video encoding method, this application also proposes a video encoder, which can be found in the following details. Figure 15 , Figure 15 This is a schematic diagram of the structure of an embodiment of the video encoder provided in this application.

[0199] The video encoder 400 in this embodiment includes a processor 41, a memory 42, an input / output device 43, and a bus 44.

[0200] The processor 41, memory 42, and input / output device 43 are respectively connected to the bus 44. The memory 42 stores program data, and the processor 41 is used to execute the program data to implement the video encoding method described in the above embodiments.

[0201] In this embodiment, processor 41 can also be referred to as a CPU (Central Processing Unit). Processor 41 may be an integrated circuit chip with signal processing capabilities. Processor 41 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 41 can be any conventional processor.

[0202] To implement the above video decoding method, this application also proposes a video decoder, which can be found in the following details. Figure 16 , Figure 16 This is a schematic diagram of the structure of an embodiment of the video decoder provided in this application.

[0203] The video decoder 500 in this embodiment includes a processor 51, a memory 52, an input / output device 53, and a bus 54.

[0204] The processor 51, memory 52, and input / output device 53 are respectively connected to the bus 54. The memory 52 stores program data, and the processor 51 is used to execute the program data to implement the video decoding method described in the above embodiments.

[0205] In this embodiment, processor 51 can also be referred to as a CPU (Central Processing Unit). Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 51 can be any conventional processor.

[0206] This application also provides a computer storage medium; please refer to the following: Figure 17 , Figure 17 This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the video encoding method and / or video decoding method of the above embodiment.

[0207] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0208] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A video coding method based on knowledge images, characterized in that, The video encoding method includes: Encode several video frames and their reference knowledge image frames to obtain a first video stream, wherein the first video stream includes a video coding layer and a network abstraction layer; Based on the network abstraction layer, all fragments of each knowledge image frame are obtained; Place all segments of each knowledge image frame in the same position in the first video stream; A second video stream is generated and stored based on several video frames and knowledge image frames after the position is moved. The process of obtaining all fragments of each knowledge image frame based on the network abstraction layer includes: Obtain the segmentation network abstraction layer of each knowledge image frame segment; obtain the target knowledge image frame segment belonging to the end segment of the knowledge image based on the segmentation network abstraction layer; determine all segments of each knowledge image frame according to the target knowledge image frame segment; Alternatively, obtain the segmentation network abstraction layer of each knowledge image frame segment; obtain the target knowledge image frame segment belonging to the knowledge image coding delimitation segment based on the segmentation network abstraction layer; determine all segments of each knowledge image frame according to the target knowledge image frame segment; wherein, the knowledge image coding delimitation segment is the first knowledge image segment or the last knowledge image segment; Alternatively, obtain the image parameter set shared by all fragments of each knowledge image frame; obtain the syntax that marks the number of fragments of the knowledge image frame in the image parameter set; obtain all fragments of each knowledge image frame according to the number of fragments in the syntax; Alternatively, obtain the image parameter set shared by all fragments of each knowledge image frame; obtain the syntax that marks the total length of the bitstream of the knowledge image frame in the image parameter set; and obtain all fragments of each knowledge image frame according to the total length of the bitstream in the syntax.

2. The video encoding method according to claim 1, characterized in that, Placing all segments of each knowledge image frame at the same position in the first video stream includes: All segments of each knowledge image frame are placed at the same position in the first video stream before the video frame referencing the knowledge image frame.

3. The video encoding method according to claim 1 or 2, characterized in that, Placing all segments of each knowledge image frame at the same position in the first video stream includes: Place all segments of each knowledge image frame in the original position of the first segment in the first video stream.

4. The video encoding method according to claim 1 or 2, characterized in that, Placing all segments of each knowledge image frame at the same position in the first video stream includes: Place all segments of each knowledge image frame in the position of the original last segment in the first video stream.

5. The video encoding method according to claim 1, characterized in that, The video encoding method further includes: In response to a remote streaming playback request, the second video stream is converted into a third video stream, wherein the third video stream may be the same as or different from the complete first video stream; The third video stream is transmitted to the user terminal via the network.

6. The video encoding method according to claim 5, characterized in that, The third video stream consists of the target video frame corresponding to the remote streaming playback request in the first video stream and the knowledge image frame it references.

7. The video encoding method according to claim 1, characterized in that, The step of obtaining all fragments of each knowledge image frame according to the number of fragments in the syntax includes: The number of fragments for each knowledge image frame is calculated based on the sum of the number of fragments in the syntax and the preset number. All fragments of each knowledge image frame are determined according to the number of fragments in each knowledge image frame.

8. A video decoding method based on knowledge images, characterized in that, The video decoding method includes: Acquire a video stream, wherein the video stream includes a video encoding layer and a network abstraction layer; Based on the network abstraction layer, all fragments of each knowledge image frame are obtained, wherein all fragments of each knowledge image frame are stored in the same location; Decode the knowledge image frames in the video stream according to all segments of each knowledge image frame; Decode other video frames in the video stream according to the knowledge image frames; The process of obtaining all fragments of each knowledge image frame based on the network abstraction layer includes: Obtain the segmentation network abstraction layer of each knowledge image frame segment; obtain the target knowledge image frame segment belonging to the end segment of the knowledge image based on the segmentation network abstraction layer; determine all segments of each knowledge image frame according to the target knowledge image frame segment; Alternatively, obtain the segmentation network abstraction layer of each knowledge image frame segment; obtain the target knowledge image frame segment belonging to the knowledge image coding delimitation segment based on the segmentation network abstraction layer; determine all segments of each knowledge image frame according to the target knowledge image frame segment; wherein, the knowledge image coding delimitation segment is the first knowledge image segment or the last knowledge image segment; Alternatively, obtain the image parameter set shared by all fragments of each knowledge image frame; obtain the syntax that marks the number of fragments of the knowledge image frame in the image parameter set; obtain all fragments of each knowledge image frame according to the number of fragments in the syntax; Alternatively, obtain the image parameter set shared by all fragments of each knowledge image frame; obtain the syntax that marks the total length of the bitstream of the knowledge image frame in the image parameter set; and obtain all fragments of each knowledge image frame according to the total length of the bitstream in the syntax.

9. A video encoder, characterized in that, The video encoder includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the video encoding method as described in any one of claims 1 to 7.

10. A video decoder, characterized in that, The video decoder includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the video decoding method as described in claim 8.

11. A computer storage medium, characterized in that, The computer storage medium is used to store program data, which, when executed by the computer, is used to implement the video encoding method as described in any one of claims 1 to 7, and / or the video decoding method as described in claim 8.

Images