Method for generating and processing transport stream and program stream and apparatus therefor

Abstract

The application discloses a kind of generation and processing method and its device of transmission stream and program stream.Processing method and device can analyze the dependent relationship identification information in transmission stream or program stream, to obtain the dependent relationship between main bit stream and knowledge bit stream, and according to the dependent relationship, to obtain main bit stream and the knowledge bit stream referred to, and with the knowledge image in knowledge bit stream as reference image, the image in main bit stream is decoded.Generation method and device can put main bit stream and the knowledge bit stream dependent as basic stream into transmission stream packet package or program stream packet package, and put dependent relationship identification information in the transmission stream packet package or program stream packet package to describe the dependent relationship between the basic stream to which the main bit stream belongs and the basic stream to which the knowledge bit stream belongs.The application can conveniently and compatibly realize the dependent relationship description between main bit stream and knowledge bit stream, and also does not waste data amount.

Description

Technical Field

[0001] This invention belongs to the field of image or video compression, storage and transmission technology, and specifically relates to methods for generating and processing transport streams and program streams. Background Technology

[0002] 1. Traditional video coding schemes

[0003] In existing video sequence processing, to enable random access functionality in the encoded video sequence, the video sequence is divided into multiple segments with random access capabilities (referred to as random access segments), such as... Figure 1 As shown, a video sequence includes at least one random access segment. Each random access segment corresponds to a display period and includes one random access image and multiple non-random access images. Each image has its own display time to describe when the image is displayed or played. Images in a random access segment can be intra-frame coded, or coded using inter-frame prediction with reference to other images in the random access segment. The referenced images can be images to be displayed or synthetic images that cannot be displayed, etc. However, in the prior art, an image displayed after a random access image (excluding leading pictures) can only reference other images in the random access segment to which that image belongs, and cannot reference images in random access segments before or after the random access segment to which that image belongs, such as... Figure 1 As shown. Specifically, the dependency between the current image and the candidate reference image can be described in the following ways:

[0004] In existing video coding schemes (such as H.264 / AVC or H.265 / HEVC), the dependency between the current image and candidate reference images is described by the reference image configuration set of the video compression layer. This configuration set describes the numbering difference between the reference image and the current image. The reason only the numbering difference is described in the reference image configuration set is that in existing video coding schemes, the candidate reference image and the current image belong to the same independent, decodeable random access segment, and they can only use the same numbering rule, such as chronological numbering. Therefore, the candidate reference image can be accurately located based on the numbering difference between the current image and the candidate reference image. If the reference image and the current image use different numbering rules, since existing video coding schemes do not provide a method to describe different numbering rules in the bitstream, the same numbering difference will point to different candidate reference images, causing the codec to be unable to use the correct candidate reference image.

[0005] In Scalable Video Coding (SVC) and Multiview Video Coding (MVC), such as Figure 2 As shown, based on existing inter-frame prediction (using only candidate reference images within the same layer / viewpoint), SVC / MVC expands the range of candidate reference images for the current image using inter-layer / inter-viewpoint prediction. These expanded candidate reference images have the same ID (e.g., the same timestamp) as the current image but do not belong to the same layer of independently decorable segments. SVC / MVC uses layer identifiers at the video compression layer to describe the dependencies between bitstreams of different layers / viewpoints, and jointly uses the same image ID to describe the dependencies between images between layers / viewpoints.

[0006] In AVS2's background frame technology, such as Figure 3 As shown, the dependency between encoded images and scene images is described by the identifier of the reference image type in the video compression layer. Specifically, AVS2 uses identifiers to describe specific scene image types (i.e., G images and GB images) and uses a specific reference buffer (i.e., scene image buffer) to manage G / GB images. Simultaneously, it uses identifiers to describe whether the current image references a G / GB image and uses a specific reference image queue construction method (i.e., by default, G / GB images are placed at the last reference image position in the reference image queue). Ultimately, this allows the current image, numbered according to the rules, to reference candidate reference images that do not follow the rules (i.e., GB images), or candidate reference images that use the same rule numbering as the current image but whose number difference exceeds the constraint range (i.e., G images). However, this technique limits the scene image buffer to only one candidate reference image at any given time, and this candidate reference image still belongs to the same independent decodeable segment as the current image.

[0007] 2. Knowledge-based video coding scheme

[0008] The aforementioned mechanism in the existing technology limits the number of available reference images for the image to be encoded, and cannot effectively improve the efficiency of image encoding and image decoding.

[0009] To mine and utilize the information of mutual reference between images during encoding across multiple randomly accessed segments, when encoding (or decoding) an image, the encoder (or decoder) can select an image from a database that has a similar texture content to the currently encoded (or decoded) image as a reference image. This reference image is called a knowledge base image, and the database storing the set of these reference images is called a knowledge base. This method of encoding and decoding at least one image in a video by referencing at least one knowledge base image is called library-based video coding, also known as long-span-related information-based video coding (or simply long-span coding). Encoding a video sequence using library-based video coding produces a knowledge stream containing the encoded bitstream of the knowledge base image and a main stream containing the bitstreams obtained by encoding each frame of the video sequence with reference to the knowledge base image. These two bitstreams are similar to the base layer bitstream and enhancement layer bitstream produced by scalable video coding (SVC), respectively; that is, the main stream depends on the knowledge stream, and the knowledge stream can be interspersed with the main stream to form a concatenated bitstream. However, the dependency relationships between the hierarchical bitstreams in the dual-stream organization method of knowledge-based video coding and the hierarchical bitstream organization method of SVC are different. The difference is that the dual-stream layers of SVC depend on each other according to a certain aligned time period, while the video layer depends on the knowledge layer in the dual-stream of knowledge-based video coding according to a non-aligned time period.

[0010] In knowledge-based image encoding and decoding techniques, knowledge images are acquired and used to provide additional candidate reference images for image encoding and decoding. Figure 4 This paper demonstrates the dependencies between sequential images and knowledge images in encoding and decoding techniques utilizing knowledge images. Knowledge images enable sequential images to leverage a wide range of relevant information, improving encoding and decoding efficiency. However, existing techniques cannot effectively support the description of dependencies between sequential images and knowledge images, nor can they efficiently manage knowledge images.

[0011] 3. Existing methods for generating and processing transport streams and program streams.

[0012] The method for generating and processing transport streams and program streams combines one or more audio, video bitstreams or other basic data streams into a single or multiplexed stream for storage and transmission. The syntax information in the data stream is used for synchronous decoding and display of audio and video information, while ensuring that the data buffer in the decoder does not overflow or underflow.

[0013] A transport stream is a type of stream defined primarily for program transmission and storage in environments where significant errors (often manifesting as bit errors or packet loss) may occur. It is a single stream consisting of one or more programs, which may contain one or more different clock bases. Multiple PES (Packetized Elementary Stream) packets composed of multiple elementary streams within the same program share a single clock base. Transport stream packets are 184 bytes long. The header of a transport stream packet specifies the predetermined time for each byte received from the channel to enter the program stream decoder, serving as a reference for clock correction and buffer management. Its data structure is as follows: Figure 5 As shown.

[0014] A program stream is a stream defined for the transmission and storage of a program in environments with low error rates where the system encoding process is a primary consideration. It is a single stream composed of one or more PES packets sharing a common time base, containing only one program. The length of the program stream packets is variable. The header of the program stream packet specifies the predetermined time for each byte received from the channel to enter the program stream decoder, serving as a reference for clock correction and buffer management. Its data structure is as follows: Figure 6 As shown.

[0015] Each transport stream and program stream contains information that identifies the characteristics of the elementary streams that make up a program and the relationships between them. This information may include the language used in the audio channels and the relationships between layers in multi-layer video coding. For example, transport stream packets may carry Program-Specific Information (PSI) tables specifying the data content included in the transport stream packets. PSI tables carried in the transport stream include the following four: a) Program Association Table (PAT); b) Program Map Table (PMT); c) Conditional Access Table (CAT); and d) Network Information Table (NIT). Program stream packets may carry Program Stream Map (PSM) and Program Stream Directory (PSD). These tables, containing descriptive information, can record the attributes of each transport stream / program stream, or each program, or each elementary stream, which can be achieved by using different program element descriptors. For example, hierarchy_descriptor can describe the attributes of basic flows that have hierarchical relationships in a program and identify the dependencies between basic flows.

[0016] A primary stream refers to any encoded video bitstream, encoded audio bitstream, or other encoded bitstream within a PES packet. A primary stream is transmitted with one and only one sequence of PES packets, each with a specific stream_id.

[0017] The PES packet header begins with a 32-bit start code, which also identifies the stream or stream type to which the packet data belongs. Its data structure is as follows: Figure 7 As shown. The PES packet header may contain decoding and presentation timestamps (DTS and PTS). The PES packet header also contains other optional fields. The PES packet data field contains a variable number of consecutive bytes from a basic stream.

[0018] For a program that contains only a single video elementary stream, the transport stream's data structure is as follows: Figure 8 As shown. The first line is a series of TS packets arranged in a data streaming manner. The content in the boxes in the second line is part of the data contained in the header or payload of the corresponding TS packet. In this example, the first TS packet contains PAT, the second TS packet contains PMT, and subsequent TS packets contain PES packets.

[0019] For a video containing only a single bit stream, the program stream data structure is as follows: Figure 9 As shown in the diagram. The first row is a series of PS packets arranged in a data streaming manner. The content of the boxes in the second row is part of the data contained in the header or payload of the corresponding PS packet. In this example, the first PS packet contains PSM, and subsequent PS packets contain PES packets. Summary of the Invention

[0020] While existing methods for generating and processing transport and program streams can support the processing of layered coding (e.g., temporal layered coding of SVC and MVC, or HEVC and VVC), the dependencies between the multi-layered video streams generated by different layered coding methods are not consistent. Therefore, special designs are needed for each type of layered coding to accurately describe the dependencies between the multi-layered video streams in the transport and program streams, facilitating rapid data processing and correct decoding at the receiving end. This invention addresses the aforementioned problem by innovatively designing a solution for the dependency characteristics of the master stream and knowledge stream using large-span coding. Furthermore, since the dependency description between the master stream and knowledge stream using large-span coding is not achieved through timestamp synchronization but through dependency identification (because a knowledge image in the knowledge stream can be shared as a reference by images in multiple random access segments of the master stream, thus making synchronization impossible using a single timestamp), and existing transport and program streams can only use timestamps for synchronization, this invention also addresses this issue by innovatively designing a convenient and compatible method for describing the dependency between the master stream and knowledge stream.

[0021] Therefore, the first objective of this invention is to provide a method for processing transport streams / program streams, comprising:

[0022] Parse the dependency identification information of the target program in the transport stream / program stream to obtain the dependency relationship between the main bit stream and the knowledge bit stream in the target program;

[0023] Acquire the main stream and knowledge stream in the target program, wherein at least one knowledge image in the knowledge stream is referenced by at least two non-temporally adjacent images in the main stream;

[0024] The knowledge image in the knowledge bitstream is used as a reference image to decode the image in the main bitstream.

[0025] Preferably, the dependency identification information is recorded in the program mapping table of the transport stream or the program stream mapping of the program stream. The program mapping table is located in the transport stream packet, and the program stream mapping is located in the program stream packet, recording the numbering information and attribute information of the basic streams contained in the program.

[0026] Preferably, the dependency identification information is recorded in the program element descriptor in the program mapping table of the transport stream or the program stream mapping of the program stream. The descriptor is used to identify whether the corresponding basic stream is the primary stream of the reference knowledge bit stream. If so, the descriptor also identifies the index number of all knowledge bit streams referenced by the primary stream.

[0027] Preferably, the decoding time is recorded in the basic stream packet to which the knowledge image in the knowledge bitstream and the main bitstream image in the main bitstream belong. The decoding time recorded in the basic stream packet to which the knowledge image in the knowledge bitstream belongs should be earlier than the decoding time recorded in the basic stream packet to which the main bitstream image in the main bitstream belongs. The main bitstream image is the first image in the main bitstream to be decoded with reference to the knowledge image.

[0028] A second objective of this invention is to provide a method for generating a transport stream / program stream, comprising:

[0029] Acquire the knowledge bitstream and main bitstream of the program, wherein at least one knowledge image in the knowledge bitstream is referenced by at least two non-temporally adjacent images in the main bitstream;

[0030] The knowledge bitstream and the main bitstream are packetized as basic streams to obtain basic stream packets, which are then placed into transport stream packets or program stream packets.

[0031] Dependency identification information is included in the transport stream packet or program stream packet. This information describes the dependency relationship between the base stream to which the primary bit stream belongs and the base stream to which the knowledge bit stream belongs.

[0032] Output the generated transport stream or program stream.

[0033] Preferably, the process of placing dependency identification information into the transport stream packet or program stream packet includes the following operations: placing the dependency identification information into the program mapping table of the transport stream or the program stream mapping of the program stream, wherein the program mapping table is located in the transport stream packet and the program stream mapping is located in the program stream packet, both of which record the numbering information and attribute information of the basic streams contained in the program.

[0034] Preferably, the process of placing dependency identification information into the transport stream packet or program stream packet includes the following operations: placing the dependency identification information into the program element descriptor in the program mapping table of the transport stream or the program stream mapping of the program stream. The descriptor is used to identify whether the corresponding basic stream is the primary bit stream of the reference knowledge bit stream. If so, the descriptor also identifies the index number of all knowledge bit streams referenced by the primary bit stream.

[0035] Preferably, the step of packetizing the knowledge bitstream and the main bitstream as basic streams to obtain basic stream packets further includes the following operation: setting the decoding time recorded in the basic stream packet according to the time when the knowledge image contained in the basic stream packet is used, the decoding time should be earlier than the decoding time recorded in the basic stream packet to which the main bitstream image in the main bitstream belongs, wherein the main bitstream image is the first image in the main bitstream to be decoded with reference to the contained knowledge image.

[0036] A third objective of this invention is to provide a processing apparatus for a transport stream / program stream, comprising:

[0037] Parsing module: Parses the dependency identification information of the target program in the transport stream / program stream to obtain the dependency relationship between the main bit stream and the knowledge bit stream in the target program;

[0038] Acquisition module: Acquires the knowledge bitstream and main bitstream in the target program, wherein at least one knowledge image in the knowledge bitstream is referenced by at least two non-temporally adjacent images in the main bitstream;

[0039] Decoding module: Uses the knowledge image in the knowledge bitstream as a reference image to decode the image in the main bitstream.

[0040] A fourth objective of this invention is to provide an apparatus for generating a transport stream / program stream, comprising:

[0041] Acquisition module: Acquires the knowledge bitstream and main bitstream of the target program, wherein at least one knowledge image in the knowledge bitstream is referenced by at least two non-temporally adjacent images in the main bitstream;

[0042] Packetization module: The knowledge bitstream and the main bitstream are used as basic streams to be packetized to obtain basic stream packets, and then put into transport stream packets or program stream packets.

[0043] Insertion module: Insert dependency identification information into the transport stream packet or program stream packet. This information describes the dependency relationship between the base stream to which the main bit stream belongs and the base stream to which the knowledge bit stream belongs.

[0044] Output module: Outputs the generated transport stream or program stream.

[0045] The beneficial effects of this invention are as follows: First, by adding dependency identification information between basic streams, the invention enables the storage and transmission of video programs containing primary and knowledge streams via transport or program streams. This allows the better compression effect obtained by the primary and knowledge streams to be effectively used for data storage and transmission, reducing bandwidth consumption in practical applications. Second, the dependency identification information designed in this invention is fully compatible with existing standard technologies and will not incur additional overhead for updating existing equipment. Furthermore, the use of flexible program element descriptors enables flexible use of dependencies, avoiding the potential waste of syntax elements in existing methods when processing primary and knowledge streams. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A schematic diagram of image dependencies in a video sequence segmented into random access fragments using existing technology 1;

[0048] Figure 2 A schematic diagram of image dependencies in a video sequence segmented into random access fragments using existing technology 2;

[0049] Figure 3 A schematic diagram of image dependencies in a video sequence segmented into random access fragments using existing technology 3;

[0050] Figure 4 A schematic diagram of image dependencies in a video sequence segmented into random access fragments using existing technology four;

[0051] Figure 5 A schematic diagram of a data structure for transport stream packets in the prior art;

[0052] Figure 6 A schematic diagram of a data structure for program stream packets in the prior art;

[0053] Figure 7 A schematic diagram of a data structure for PES packets in the prior art;

[0054] Figure 8 A schematic diagram of a data structure for transport streams in the prior art;

[0055] Figure 9 A schematic diagram of a data structure for a program stream in the prior art;

[0056] Figure 10 A schematic diagram of a method for processing a transport stream / program stream disclosed in this invention;

[0057] Figure 11 A schematic diagram of a method for generating a transport stream / program stream disclosed in this invention;

[0058] Figure 12 A schematic diagram of a data structure for a transport stream in one embodiment of the present invention;

[0059] Figure 13 A schematic diagram of a data structure for a program stream in one embodiment of the present invention;

[0060] Figure 14 A schematic diagram of a transmission stream / program stream processing device disclosed in this invention;

[0061] Figure 15 A schematic diagram of a transport stream / program stream generation device disclosed in this invention. Detailed Implementation

[0062] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Note that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way. Terminology definitions:

[0063] Before describing the embodiments, let's define the necessary terms:

[0064] Knowledge image: also called knowledge base image. A knowledge image is an image that is outside the set of images to be displayed in the random access segment to which the current image belongs and the next nearest random access segment. A knowledge image is a reference image used to provide a reference for the image to be encoded or decoded.

[0065] RL (Reference to Library) image: An image encoded or decoded solely by referencing a knowledge image. In one implementation, the RL image immediately follows the sequence header data; random access is achieved by decoding the external knowledge image first and then the RL image. In another implementation, the RL image immediately follows the sequence header data and the knowledge image data; random access is achieved by decoding the knowledge image first and then the RL image. In yet another implementation, the RL image follows the sequence header data, but contains auxiliary enhancement information, extended information, or user information; random access is achieved by decoding the knowledge image first and then the RL image.

[0066] Knowledge bitstream: also called knowledge codestream, refers to a bitstream that contains only knowledge images. At least one knowledge image in the knowledge bitstream is referenced by at least two non-temporally adjacent images in the main bitstream.

[0067] Main bitstream: Also called main bitstream, it refers to the bitstream to which the reference knowledge image belongs, but this bitstream does not contain the knowledge image data. In one implementation, the knowledge stream and the main bitstream are interleaved and concatenated into a single bitstream.

[0068] Example

[0069] In this embodiment, a method for processing transport streams / program streams is provided, such as... Figure 10 As shown, this method identifies the presence of knowledge bitstreams and the primary bitstream of reference knowledge bitstreams in the program by parsing the knowledge bitstream dependency identification information contained in the transport stream / program stream. If the primary bitstream of the reference knowledge bitstream exists, the method locates the knowledge bitstream that the primary bitstream depends on for decoding processing. Specifically, the processing operations are as follows:

[0070] 1. Acquire transport streams or program streams;

[0071] 2. Parse the dependency identification information in the transport stream or program stream to obtain the dependency relationship between the main bit stream and the knowledge bit stream;

[0072] 3. Based on the aforementioned dependencies, process the transport stream or program stream to obtain the master bit stream and its referenced knowledge bit stream;

[0073] 4. Use the knowledge image in the knowledge bitstream as a reference image to decode the image in the main bitstream.

[0074] In a corresponding embodiment, a method for generating a transport stream / program stream is provided at the sending end, such as... Figure 11 As shown, this method is used to generate the stream processed by the above-described processing method. Specifically, the generation operation is as follows:

[0075] 1. Obtain the main bitstream and its dependent knowledge bitstreams;

[0076] 2. The primary bit stream and the knowledge bit stream are respectively divided into basic stream packets to obtain basic stream packets, and then placed into transport stream packets or program stream packets;

[0077] 3. Include dependency identification information in the transport stream packet or program stream packet. This information describes the dependency relationship between the base stream to which the primary bit stream belongs and the base stream to which the knowledge bit stream belongs.

[0078] 4. Output the generated transport stream or program stream.

[0079] The methods disclosed in the above embodiments provide key operations for the generation and processing of transport streams and program streams. These operations can be further extended in different implementations and application scenarios.

[0080] In one implementation, the operation of "processing the transport stream or program stream according to the dependency relationship to obtain the primary bit stream and the knowledge bit stream it references" further includes: parsing the packet header information of the packets in the transport stream or program stream, and obtaining the program information and the basic stream information contained in the transport stream or program stream based on the packet header information.

[0081] Specifically, for transport streams, the Program Association Table (PAT) and Program Mapping Table (PMT) carried in the transport stream packets are parsed from the packet header information, as well as the Conditional Acceptance Table (CAT) and Network Information Table (NIT) that may exist in the transport stream packets. From these tables, the program number information, the basic stream information contained in the program, and the attribute information of the basic stream are parsed.

[0082] Specifically, for a program stream, the Program Stream Mapping Segment (PSM) and Program Stream Address Segment (PSD) carried in the program stream packet are parsed from the packet header information. From these tables, the program number information, the basic stream information contained in the program, and the attribute information of the basic stream are parsed.

[0083] Dependency identification information is obtained from the PAT or PMT of the transport stream or the PSM of the program stream to identify whether the basic stream in the program is a knowledge bit stream or a master bit stream, and to obtain the dependency relationship between the master bit stream and the knowledge bit stream.

[0084] In another implementation, the operation of "processing the transport stream or program stream according to the dependency relationship to obtain the primary bit stream and its referenced knowledge bit stream" further includes: obtaining the PES packet to which the target program belongs from the program stream packet according to the parsed information, and extracting the basic stream data in the target program from the PES packet. Then, based on the dependency relationship identification information obtained from the parsed information, identifying and obtaining the basic stream containing the primary bit stream and the basic stream containing the knowledge bit stream.

[0085] In another implementation, the operation of "decoding the image in the main bitstream using the knowledge image in the knowledge bitstream as the reference image" further includes: when the image to be decoded in the main bitstream is an RL image, decoding is performed using only the knowledge image in the knowledge bitstream as the reference image; when the image to be decoded in the main bitstream is a P / B image, decoding is performed using a set of reference images consisting of its original short-term reference image and (if the image references a knowledge image) the knowledge image.

[0086] In one implementation, the method for generating the transport stream and program stream further includes: the primary stream and the knowledge stream are each treated as different basic streams and packetized into PES packets. The order of these PES packets can be as follows: first, a PES packet containing the knowledge stream is generated, then a PES packet containing the primary stream is generated; or first, a PES packet containing the primary stream is generated, then a PES packet containing the knowledge stream is generated; or the two can be generated alternately. In yet another implementation, the method for generating the transport stream and program stream further includes: the primary stream and the knowledge stream are combined into a single basic stream and packetized into PES packets, in which case the data from the knowledge stream and the primary stream cannot coexist in the same PES packet.

[0087] In another implementation, the operation of "inserting dependency identification information into the transport stream packet or program stream packet" further includes:

[0088] For transport streams, the dependency identification information is placed in the Program Association Table (PAT) or Program Map Table (PMT) carried in the transport stream packets, as well as in the Conditional Acceptance Table (CAT) and Network Information Table (NIT), which may also exist.

[0089] For a program stream, the dependency identification information is placed in the program stream mapping segment PSM or program stream address segment PSD carried by the program stream packet.

[0090] In the above embodiments and their implementations, there are various specific methods for generating and processing dependency identification information, which are described in detail below.

[0091] In one implementation, to ensure compatibility with existing standard methods, an attempt is made to leverage existing syntactic elements and add new semantics to distinguish between the subject stream and the knowledge stream and to describe the dependencies between them.

[0092] Specifically, in the adaptation field of a transport stream, there is a syntax element `elementary_stream_priority_indicator` that describes the importance of the incoming streams contained in the transport stream packet to which this adaptation field belongs. This syntax element can be reused, and new semantics can be added to distinguish between primary streams and knowledge streams and to describe the dependencies between them. For example, the `elementary_stream_priority_indicator` syntax element in the adaptation field adds the following italicized semantics (the other semantics are the original semantics of this field / syntax element):

[0093] "This field is 1 bit and indicates the priority of the basic stream data carried in the payload data among transport stream packets with the same PEID. A value of '1' indicates that the payload data has a higher priority than other transport stream packets. For video, this field can only be '1' if the payload data contains one or more bytes from the intra-coded image. A value of '0' indicates that the payload data has the same priority as other transport stream packets whose value is not '1'. For AVS3 encoded video, when there are knowledge bitstreams and main bitstreams, a value of '0' indicates that the payload data belongs to the main bitstream, and a value of '1' indicates that the payload data belongs to the knowledge bitstream. When decoding the main bitstream data, the knowledge bitstream data must be decoded first."

[0094] The advantage of this implementation is that it does not require adding new syntax elements. However, since the adaptation field only exists in the transport stream packets and mainly contains some unimportant information, it is an optional field. Therefore, this approach may introduce other redundant syntax elements and cannot solve the problem of describing dependencies in the program stream.

[0095] In another implementation, considering that the program map (PMT) of the transport stream and the program stream map (PSM) of the program stream record the basic streams contained in each program, additional identifiers can be added to the PMT and PSM of the transport stream to describe the dependencies between the primary bit stream and the knowledge bit stream. As shown in the table below, the italicized syntax elements are the newly added syntax elements used to identify the dependencies between the primary bit stream and the knowledge bit stream.

[0096] Table 1 Syntax Elements of Transport Stream Program Map Table

[0097]

[0098] The existing syntax elements and their definitions in the program mapping table can be accurately described in the existing transport stream methods, and will not be repeated here. The semantics of the newly added syntax element identifiers are as follows:

[0099] The knowledge bitstream identifier is_library_stream

[0100] This field is 1 bit. It defines whether the primary stream corresponding to this identifier is a knowledge bitstream in the program. A value of '1' indicates that the primary stream corresponding to this identifier is a knowledge bitstream, which is depended upon by the primary bitstream in the same program; a value of '0' indicates that the primary stream corresponding to this identifier is a primary bitstream, which depends on the knowledge bitstream in the same program, and its dependent knowledge bitstream should be decoded before decoding the primary bitstream.

[0101] The purpose of adding the conditional statement "if(stream_type==0xD4)" to the program mapping table is to reduce the data waste of the syntax element is_library_stream when processing video bitstreams with non-large span encoding. This is because it is only necessary to use the syntax element is_library_stream when the video stream being processed uses the AVS3 encoding method (stream_type is defined as 0xD) and generates a main bitstream and a knowledge bitstream.

[0102] The advantage of this implementation is that the dependency relationship between the main bit stream and the knowledge bit stream can be quickly obtained when parsing PMT or PSM. However, this identifier will waste data when only the main bit stream is generated using the AVS3 encoding method, and it is also not compatible with the original standard method.

[0103] In another implementation, for the generation and processing of program streams, a new syntax element is added to the program stream mapping of the program stream, as shown in the table below.

[0104] Table 2 Syntax of Program Stream Mapping

[0105]

[0106]

[0107] The existing syntax elements and their definitions in the program mapping table can be accurately described in the existing transport stream methods, and will not be repeated here. The semantics of the newly added syntax element identifiers are as follows:

[0108] The knowledge bitstream identifier is_library_stream

[0109] This field is 1 bit. It defines whether the primary stream corresponding to this identifier is a knowledge bitstream in the program. A value of '1' indicates that the primary stream corresponding to this identifier is a knowledge bitstream, which is depended upon by the primary bitstream in the same program; a value of '0' indicates that the primary stream corresponding to this identifier is a primary bitstream, which depends on the knowledge bitstream in the same program, and its dependent knowledge bitstream should be decoded before decoding the primary bitstream.

[0110] This implementation method is comparable in advantages and disadvantages to the previous one.

[0111] In another implementation, considering that program element descriptors in the transport stream and program stream can describe the attributes of the basic stream, additional descriptors can be added to describe the dependencies between the primary stream and the knowledge stream. This embodiment designs an AVS3 long-span coded video stream descriptor to represent program elements containing video including the primary stream and knowledge stream, encoded using an AVS3 coding method based on long-span related information. These program elements are multiplexed across multiple streams.

[0112] Table 3 AVS3 Large Span Encoded Video Stream Descriptors

[0113]

[0114] The semantics of the syntax elements are as follows, where the defined number of bits and values ​​are for illustrative purposes only and can be any available number of bits and values:

[0115] The descriptor_tag field

[0116] This field is 8 bits long and is used to identify each descriptor.

[0117] descriptor_length field

[0118] This field is 8 bits long. It specifies the number of bytes in the descriptor that immediately follows this field.

[0119] The knowledge bitstream identifier is_library_stream

[0120] This field is 1 bit. It defines whether the primary stream corresponding to this descriptor is a knowledge bitstream in the program. A value of '1' indicates that the primary stream corresponding to this descriptor is a knowledge bitstream and is depended upon by the primary bitstream in the same program; a value of '0' indicates that the primary stream corresponding to this descriptor is a primary bitstream, which depends on the knowledge bitstream in the same program, and its dependent knowledge bitstream should be decoded before decoding the primary bitstream.

[0121] The above method uses program element descriptors, which are compatible with existing standards, and the descriptors can be freely selected by the generator, thus avoiding data waste. This is an implementation method that is both compatible and saves data. However, the above implementation only uses a 1-bit identifier to determine whether the basic stream in the program is a knowledge bitstream or a main bitstream, assuming that the main bitstream must refer to the knowledge bitstream. This cannot cope with various application scenarios.

[0122] Therefore, in another implementation, the syntactic elements in the program element descriptor are expanded. In addition to identifying the main bitstream and knowledge bitstream, the index number of the mainstream referenced knowledge bitstream is further identified, supporting scenarios where one main bitstream can reference multiple knowledge bitstreams. The syntax and semantics of this expanded descriptor are shown in the table below.

[0123] Table 4 AVS3 Large Span Encoded Video Stream Descriptors

[0124]

[0125] The semantics of the syntax elements are as follows, where the defined number of bits and values ​​are for illustrative purposes only and can be any available number of bits and values:

[0126] The descriptor_tag field

[0127] This field is 8 bits long and is used to identify each descriptor.

[0128] descriptor_length field

[0129] This field is 8 bits long. It specifies the number of bytes in the descriptor that immediately follows this field.

[0130] The knowledge bitstream identifier is_library_stream

[0131] This field is 1 bit. It defines whether the primary stream corresponding to this descriptor is a knowledge bitstream in the program. A value of '1' indicates that the primary stream corresponding to this descriptor is a knowledge bitstream; a value of '0' indicates that the primary stream corresponding to this descriptor is a primary bitstream, which depends on the knowledge bitstream in the same program, and its dependent knowledge bitstream should be decoded before decoding the primary bitstream.

[0132] The field num_ref_library_stream is used to count the number of knowledge streams referenced.

[0133] This field is 8 bits long. It defines the number of underlying bitstreams containing knowledge bitstreams that the primary bitstream references when the underlying bitstream corresponding to this descriptor is a primary bitstream. The value of num_ref_library_stream can be 0, in which case the primary bitstream does not reference knowledge bitstreams.

[0134] The referenced knowledge stream index number library_stream_id

[0135] This field is 8 bits long. It defines the index number of the basic bitstream containing the knowledge bitstream that this descriptor references when the underlying bitstream is a primary bitstream.

[0136] In one embodiment, all the above embodiments and their implementations should also satisfy the constraint that when the knowledge bitstream and the main bitstream are packetized into PES packets, the decoding time DTS recorded in the PES packet to which the knowledge image in the referenced knowledge bitstream belongs should be earlier than the decoding time recorded in the basic stream packet to which the main bitstream image in the main bitstream to which the referenced knowledge bitstream belongs, wherein the main bitstream image is the first image in the main bitstream to be decoded with reference to the knowledge image.

[0137] In one embodiment, the above constraints can be satisfied by other methods. One implementation method is to expand the decoding time DTS of the PES packet to which the knowledge image belongs in the knowledge bitstream into multiple DTSs. These DTSs correspond one-to-one with the decoding times of the PES packets to which all the main bitstream images referencing the knowledge image belong in the main bitstream. That is, the PES packet containing the knowledge image is decoded for reference at multiple times. In another implementation method, the PES packet to which the knowledge image belongs does not use DTS to establish a dependency relationship with the main bitstream image. Instead, new image-level dependency identification information can be added to the PES packet to which the main bitstream belongs. This identification information can index the PES packets to which all the knowledge images referenced by the main bitstream image in the PES packet belong. Compared with the previous implementation method, this is an explicit dependency expression method.

[0138] In yet another embodiment, when using a main bitstream and its dependent knowledge bitstreams present in the program, the data structure of the transport stream generated using the aforementioned method should be as follows: Figure 12 As shown. A video program in this TS stream contains two basic streams: the knowledge bitstream (whose TS packet PID = 0x11) and the main bitstream (whose TS packet PID = 0x12). In the PMT, avs3crr_descriptor() is added to each of the two basic streams of the program. The descriptor corresponding to the knowledge bitstream has an is_library_stream flag of 1, while the descriptor corresponding to the main bitstream has an is_library_stream flag of 0.

[0139] Similarly, in another embodiment, when using the above method to generate a program stream, its data structure should be as follows: Figure 13 As shown. A video program in this PS stream contains two basic streams: the knowledge bitstream (whose PS packet PID = 0x11) and the main bitstream (whose PS packet PID = 0x12). In the PSM, avs3crr_descriptor() is added to each of the program's two basic streams. The descriptor corresponding to the knowledge bitstream has an is_library_stream value of 1, while the descriptor corresponding to the main bitstream has an is_library_stream value of 0.

[0140] In another embodiment, when processing the received transport stream or program stream, the receiving end needs to perform a series of buffering and demultiplexing operations before it can obtain the basic stream data from the transport stream or program stream for decoding. This embodiment describes the operations that the receiving end should implement to achieve correct buffering and decoding of the basic stream data, avoiding overflow or underflow of the basic stream data in the data buffer.

[0141] The above provides methods for generating and processing transport streams and program streams. Correspondingly, the operations of the above embodiments and implementations can be implemented as corresponding devices and modules.

[0142] In one embodiment, a processing apparatus for a transport stream / program stream is provided, such as... Figure 14 As shown, the device includes a processor and storage, as well as a set of modules capable of performing the following operations:

[0143] 1. Acquisition Module: Used to acquire transport streams or program streams;

[0144] 2. Parsing module: Parses the dependency identification information in the transport stream or program stream to obtain the dependency relationship between the main bit stream and the knowledge bit stream;

[0145] 3. Processing module: Based on the aforementioned dependencies, processes the transport stream or program stream to obtain the master bit stream and its referenced knowledge bit stream;

[0146] 4. Decoding Module: This module decodes the image in the main bitstream using the knowledge image in the knowledge bitstream as a reference image. The programs for this module are stored in memory and executed by the processor.

[0147] In another embodiment corresponding to this one, a transmission stream / program stream generation device is provided at the transmitting end, such as... Figure 15 As shown, the device includes a processor and storage, as well as a set of modules capable of performing the following operations:

[0148] 1. Acquisition Module: Acquires the main bitstream and its dependent knowledge bitstreams;

[0149] 2. Packetization module: Packets the main bit stream and the knowledge bit stream as basic streams and places them into transport stream packets or program stream packets respectively;

[0150] 3. Insertion Module: Insert dependency identification information into the transport stream packet or program stream packet. This information describes the dependency relationship between the base stream to which the primary bit stream belongs and the base stream to which the knowledge bit stream belongs.

[0151] 4. Output module: Outputs the generated transport stream or program stream.

[0152] The programs for these modules are stored in memory and executed by the processor.

[0153] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the above preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention should not depart from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for processing a transport stream / program stream, comprising: The dependency identification information of the target program in the transport stream / program stream is parsed to obtain the dependency relationship between the main bit stream and the knowledge bit stream in the target program. The dependency identification information is recorded in the program mapping table of the transport stream or the program stream mapping of the program stream. The program mapping table is located in the transport stream packet, and the program stream mapping is located in the program stream packet. The numbering information and attribute information of the basic streams contained in the program are recorded. Acquire the main stream and knowledge stream in the target program, wherein at least one knowledge image in the knowledge stream is referenced by at least two non-temporally adjacent images in the main stream; The knowledge image in the knowledge bitstream is used as a reference image to decode the image in the main bitstream.

2. The method according to claim 1, characterized in that: The dependency identification information is recorded in the program element descriptor in the program mapping table of the transport stream or the program stream mapping of the program stream. The descriptor is used to identify whether the corresponding basic stream is the primary stream of the reference knowledge bit stream. If so, the descriptor also identifies the index number of all knowledge bit streams referenced by the primary stream.

3. The method of claim 1, wherein, The decoding time is recorded in the basic stream packet to which the knowledge image in the knowledge bitstream and the main bitstream image in the main bitstream belong. The decoding time recorded in the basic stream packet to which the knowledge image in the knowledge bitstream belongs should be earlier than the decoding time recorded in the basic stream packet to which the main bitstream image in the main bitstream belongs. The main bitstream image is the first image in the main bitstream to be decoded with reference to the knowledge image.

4. A method of generating a transport stream / program stream, characterized by include: Acquire the knowledge bitstream and main bitstream of the program, wherein at least one knowledge image in the knowledge bitstream is referenced by at least two non-temporally adjacent images in the main bitstream; The knowledge bitstream and the main bitstream are packetized as basic streams to obtain basic stream packets, which are then placed into transport stream packets or program stream packets. Dependency identification information is included in the transport stream packet or program stream packet. This information describes the dependency relationship between the base stream to which the primary bit stream belongs and the base stream to which the knowledge bit stream belongs, including the following operations: The dependency identification information is placed in the program mapping table of the transport stream or the program stream mapping of the program stream. The program mapping table is located in the transport stream packet, and the program stream mapping is located in the program stream packet. Both record the numbering information and attribute information of the basic streams contained in the program. Output the generated transport stream or program stream.

5. The method of claim 4, wherein, The process of inserting dependency identification information into the transport stream packet or program stream packet includes the following operations: The dependency identification information is placed in the program element descriptor in the program mapping table of the transport stream or the program stream mapping of the program stream. The descriptor is used to identify whether the corresponding basic stream is the primary stream of the reference knowledge bit stream. If so, the descriptor also identifies the index number of all knowledge bit streams referenced by the primary stream.

6. The method of claim 4, wherein, The process of dividing the knowledge bitstream and the main bitstream into basic stream packets further includes the following operations: Based on the time when the knowledge image contained in the basic stream packet is used, the decoding time recorded in the basic stream packet is set. This decoding time should be earlier than the decoding time recorded in the basic stream packet to which the main stream image in the main stream belongs, wherein the main stream image is the first image in the main stream to be decoded with reference to the contained knowledge image.

7. A processing apparatus for a transport stream / program stream, comprising: Parsing module: Parses the dependency identification information of the target program in the transport stream / program stream to obtain the dependency relationship between the main bit stream and the knowledge bit stream in the target program. The dependency identification information is recorded in the program mapping table of the transport stream or the program stream mapping of the program stream. The program mapping table is located in the transport stream packet, and the program stream mapping is located in the program stream packet. It records the numbering information and attribute information of the basic streams contained in the program. Acquisition module: Acquires the knowledge bitstream and main bitstream in the target program, wherein at least one knowledge image in the knowledge bitstream is referenced by at least two non-temporally adjacent images in the main bitstream; Decoding module: Uses the knowledge image in the knowledge bitstream as a reference image to decode the image in the main bitstream.

8. An apparatus for generating a transport stream / program stream, comprising: Acquisition module: Acquires the knowledge bitstream and main bitstream of the target program, wherein at least one knowledge image in the knowledge bitstream is referenced by at least two non-temporally adjacent images in the main bitstream; Packetization module: The knowledge bitstream and the main bitstream are used as basic streams to be packetized to obtain basic stream packets, and then put into transport stream packets or program stream packets. Insertion Module: Insert dependency identification information into the transport stream packet or program stream packet. This information describes the dependency relationship between the base stream to which the primary bit stream belongs and the base stream to which the knowledge bit stream belongs, including the following operations: The dependency identification information is placed in the program mapping table of the transport stream or the program stream mapping of the program stream. The program mapping table is located in the transport stream packet, and the program stream mapping is located in the program stream packet. Both record the numbering information and attribute information of the basic streams contained in the program. Output module: Outputs the generated transport stream or program stream.

Images