Method, apparatus, computer readable medium and device for processing video stream

Abstract

Embodiments of the present application provide a video stream processing method, device, computer readable medium and equipment. The video stream processing method comprises: parsing a video stream to be encapsulated to obtain a video coding parameter set and video frame data of the video stream; if it is determined according to the video coding parameter set that the video stream contains a multi-view video stream, generating a program map table corresponding to the multi-view video stream, adding a registration descriptor to the program map table, and using a value of a format identifier in the registration descriptor to indicate that the video stream is a multi-view video stream; and performing encapsulation processing on the program map table, the video coding parameter set and the video frame data to generate a transport stream data packet. The technical solution of the embodiments of the present application realizes the expansion of the encapsulation mode of the multi-view video stream, increases the application scenarios of the multi-view video stream, and meets the increasing demand of users for multi-view video content.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to a method, apparatus, computer-readable medium, and device for processing video streams. Background Technology

[0002] With the continuous development of digital technology, immersive media is gradually entering people's daily lives. Immersive media refers to the use of technologies such as virtual reality (VR) to place users in a virtual environment, providing an immersive experience. Among them, multi-view video is an important component of immersive media, bringing users the most intuitive interactive experience.

[0003] Before a video playback device plays video content, a video production device can acquire the video content and encode it to obtain a corresponding video stream. Then, the video production device can encapsulate the video stream and transmit the resulting encapsulated file to the video playback device, which can then play the corresponding video content based on the encapsulated file. However, the encapsulation methods for multi-view video streams proposed in related technologies are limited, thus restricting the application of multi-view video streams and failing to meet the transmission and storage requirements of multi-view video. Summary of the Invention

[0004] The embodiments of this application provide a video stream processing method, apparatus, computer-readable medium, and device, which expand the encapsulation method of multi-view video streams, increase the application scenarios of multi-view video streams, and meet the growing demand of users for multi-view video content.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part by practice of this application.

[0006] According to one aspect of the embodiments of this application, a video stream processing method is provided, comprising: parsing a video stream to be encapsulated to obtain a video encoding parameter set and video frame data of the video stream; if it is determined from the video encoding parameter set that the video stream contains a multi-view video stream, generating a program mapping table corresponding to the multi-view video stream, wherein a registration descriptor is added to the program mapping table, and the value of a format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream; and encapsulating the program mapping table, the video encoding parameter set, and the video frame data to generate a transport stream data packet.

[0007] According to one aspect of the embodiments of this application, a video stream processing method is provided, comprising: acquiring a transport stream data packet, wherein the transport stream data packet is obtained by encapsulating a program map table, a video encoding parameter set of the video stream, and video frame data, wherein a registration descriptor is added to the program map table, and the value of a format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream; decapsulating the transport stream data packet to obtain a video data encapsulation unit, as well as the program map table and the video encoding parameter set; and reassembling the video data encapsulation unit according to the program map table and the video encoding parameter set to obtain a multi-view video stream.

[0008] According to one aspect of the embodiments of this application, a video stream processing apparatus is provided, comprising: a parsing unit configured to parse a video stream to be encapsulated to obtain a video encoding parameter set and video frame data of the video stream; a generation unit configured to generate a program map table corresponding to the multi-view video stream if it is determined from the video encoding parameter set that the video stream contains a multi-view video stream, wherein a registration descriptor is added to the program map table, and the value of a format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream; and a processing unit configured to encapsulate the program map table, the video encoding parameter set, and the video frame data to generate a transport stream data packet.

[0009] In some embodiments of this application, based on the foregoing scheme, the generation unit is further configured to: if it is determined from the video coding parameter set that the video stream contains a multi-viewpoint video stream, generate three-dimensional reference display information from the video coding parameter set, add the three-dimensional reference display information to the program mapping table, wherein the three-dimensional reference display information contains identification information of multiple viewpoints in the multi-viewpoint video stream.

[0010] In some embodiments of this application, based on the foregoing scheme, the program mapping table contains basic stream information, the registration descriptor is the first descriptor in the basic stream information, and the three-dimensional reference display information is added to the second descriptor in the basic stream information.

[0011] In some embodiments of this application, based on the foregoing scheme, the label value of the second descriptor is a set value, and the length value of the second descriptor is the length of the three-dimensional reference display information.

[0012] In some embodiments of this application, based on the foregoing scheme, the processing unit is configured to encapsulate the video encoding parameter set and the video frame data as a whole, and encapsulate the program mapping table separately to generate encapsulated transport stream data packets.

[0013] In some embodiments of this application, based on the foregoing scheme, the processing unit is configured to encapsulate the video encoding parameter set and the keyframe data in the video stream as a whole.

[0014] In some embodiments of this application, based on the foregoing scheme, the processing unit is configured to: sequentially write the video frame data into the transport stream data packet according to the arrangement order of the video frames contained in the video stream; wherein each video frame contains multiple video data encapsulation units corresponding to different viewpoints.

[0015] In some embodiments of this application, based on the foregoing scheme, the first video data encapsulation unit included in each video frame is an Access Unit Delimiter (AUD) video data encapsulation unit, which is used to indicate the type of each video frame.

[0016] In some embodiments of this application, based on the foregoing scheme, each video data encapsulation unit includes a start code, which is used to indicate the starting position of the video data encapsulation unit.

[0017] In some embodiments of this application, based on the foregoing scheme, the generation unit is further configured to: add basic stream information of the multi-view video stream to the program mapping table, wherein the basic stream information includes video stream type and total descriptor length.

[0018] According to one aspect of the embodiments of this application, a video stream processing apparatus is provided, comprising: an acquisition unit configured to acquire transport stream data packets, the transport stream data packets being obtained by encapsulating a program map table, a video encoding parameter set of the video stream, and video frame data, wherein a registration descriptor is added to the program map table, and the value of a format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream; a decapsulation unit configured to decapsulate the transport stream data packets to obtain a video data encapsulation unit, the program map table, and the video encoding parameter set; and a processing unit configured to reassemble the video data encapsulation unit according to the program map table and the video encoding parameter set to obtain a multi-view video stream.

[0019] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video stream processing method as described in the above embodiments.

[0020] According to one aspect of the embodiments of this application, a computer device is provided, including: one or more processors; and a storage device for storing one or more computer programs, which, when executed by the one or more processors, cause the computer device to implement the video stream processing method as described in the above embodiments.

[0021] According to one aspect of the embodiments of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads from the computer-readable storage medium and executes the computer program, causing the computer device to perform the video stream processing methods provided in the various alternative embodiments described above.

[0022] In some embodiments of this application, after obtaining the video stream to be encapsulated, the video stream can be parsed to obtain the video encoding parameter set and video frame data. If the video stream is determined to contain a multi-view video stream based on the video encoding parameter set, a program mapping table corresponding to the multi-view video stream is generated. A registration descriptor is added to this program mapping table, and the value of the format identifier in the registration descriptor indicates that the video stream is a multi-view video stream. Then, the program mapping table, video encoding parameter set, and video frame data can be encapsulated to generate a transport stream data packet. It is evident that the technical solution of this application can ensure that the multi-view video stream can be correctly identified based on the parsed video encoding parameter set, and further, the encapsulation processing of the multi-view video stream can be achieved through a program mapping table containing registration descriptors. This expands the encapsulation methods for multi-view video streams, increases the application scenarios of multi-view video streams, and meets the growing demand of users for multi-view video content.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0024] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown;

[0025] Figure 2 A schematic diagram showing the placement of video encoding and decoding devices in a streaming system;

[0026] Figure 3 This illustration shows a structural diagram of a packaged file provided in an embodiment of this application;

[0027] Figure 4 A flowchart illustrating a video stream processing method according to an embodiment of this application is shown;

[0028] Figure 5 A flowchart illustrating a video stream processing method according to an embodiment of this application is shown;

[0029] Figure 6 A flowchart of a video encapsulation processing method according to an embodiment of this application is shown;

[0030] Figure 7 This illustration shows a schematic diagram of the application of the technical solution of this application in a video live streaming scenario;

[0031] Figure 8 This illustration shows a schematic diagram of the application of the technical solution of this application in a video-on-demand scenario;

[0032] Figure 9 A block diagram of a video stream processing apparatus according to an embodiment of this application is shown;

[0033] Figure 10 A block diagram of a video stream processing apparatus according to an embodiment of this application is shown;

[0034] Figure 11 A schematic diagram of the structure of a computer system suitable for implementing the computer device of the present application is shown. Detailed Implementation

[0035] Exemplary embodiments will now be described in a more comprehensive manner with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to these examples; rather, these embodiments are provided so that this application will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0036] Furthermore, the features, structures, or characteristics described in this application can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to provide a full understanding of the embodiments of this application. However, those skilled in the art will recognize that when implementing the technical solutions of this application, not all the detailed features in the embodiments may be used, one or more specific details may be omitted, or other methods, elements, devices, steps, etc., may be employed.

[0037] In this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0040] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0041] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0042] like Figure 1 As shown, an exemplary system architecture 100 applying the technical solutions of this application includes multiple terminal devices that can communicate with each other via, for example, a network 150. For example, system architecture 100 may include a first terminal device 110 and a second terminal device 120 interconnected via network 150. Figure 1 In the embodiments, the first terminal device 110 and the second terminal device 120 can perform unidirectional data transmission or bidirectional data transmission.

[0043] For example, the first terminal device 110 can encode video data (e.g., a video image stream captured by the terminal device 110) and then transmit it to the second terminal device 120 via the network 150. The encoded video data is transmitted in the form of one or more encoded video streams. The second terminal device 120 can receive the encoded video data from the network 150, decode the encoded video data to recover the video data, and display the video image based on the recovered video data.

[0044] In one embodiment of this application, system architecture 100 may include a third terminal device 130 and a fourth terminal device 140 that perform bidirectional transmission of encoded video data, such as during a video conference. For bidirectional data transmission, each of the third terminal device 130 and the fourth terminal device 140 may encode video data (e.g., a video image stream captured by the terminal device) for transmission over network 150 to the other terminal device. Each of the third terminal device 130 and the fourth terminal device 140 may also receive encoded video data transmitted by the other terminal device, decode the encoded video data to recover the video data, and display the video images on an accessible display device based on the recovered video data.

[0045] exist Figure 1 In the embodiments shown, the first terminal device 110, the second terminal device 120, the third terminal device 130 and the fourth terminal device 140 may be servers or terminals, but the principles disclosed in this application are not limited to these.

[0046] Servers can be standalone physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals can be smartphones, tablets, laptops, desktop computers, smart speakers, smart voice interaction devices, smartwatches, smart home appliances, in-vehicle terminals, aircraft, etc., but are not limited to these.

[0047] Figure 1 The network 150 shown represents any number of networks, including, for example, wired and / or wireless communication networks, that transmit encoded video data between the first terminal device 110, the second terminal device 120, the third terminal device 130, and the fourth terminal device 140. The communication network 150 may exchange data in circuit-switched and / or packet-switched channels. This network may include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of network 150 may be irrelevant to the operation of the disclosure herein.

[0048] In one embodiment of this application, Figure 2The illustration shows the placement of video encoding and decoding devices in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital television (TV), and storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0049] The streaming system may include an acquisition subsystem 213, which may include a video source 201 such as a digital camera, which creates an uncompressed video image stream 202. In an embodiment, the video image stream 202 includes samples captured by a digital camera. The video image stream 202 is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data 204 (or encoded video bitstream 204). The video image stream 202 may be processed by an electronic device 220, which includes a video encoding device 203 coupled to the video source 201. The video encoding device 203 may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video data 204 (or encoded video bitstream 204) is depicted as a thin line to emphasize the lower data volume of the encoded video data 204 (or encoded video bitstream 204), which may be stored on a streaming server 205 for future use. One or more streaming client subsystems, such as Figure 2 Client subsystems 206 and 208 can access streaming server 205 to retrieve copies 207 and 209 of encoded video data 204. Client subsystem 206 may include, for example, a video decoding device 210 in electronic device 230. Video decoding device 210 decodes the incoming copy 207 of the encoded video data and produces an output video picture stream 211 that can be displayed on display 212 (e.g., a screen) or another presentation device. In some streaming systems, the encoded video data 204, video data 207, and video data 209 (e.g., video stream) may be encoded according to certain video encoding / compression standards.

[0050] It should be noted that electronic devices 220 and 230 may include other components not shown in the figures. For example, electronic device 220 may include a video decoding device, and electronic device 230 may also include a video encoding device.

[0051] In one embodiment of this application, taking High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) from international video coding standards, as well as the Chinese national video coding standard AVS, as examples, after an input video frame image, the video frame image is divided into several non-overlapping processing units according to a block size. Each processing unit performs a similar compression operation. This processing unit is called a Coding Tree Unit (CTU) or Largest Coding Unit (LCU). The CTU can be further subdivided into more refined units to obtain one or more basic Coding Units (CUs). The CU is the most basic element in a coding process.

[0052] In another embodiment, this processing unit can also be called a tile, which is a rectangular area of ​​a multimedia data frame that can be independently decoded and encoded. In the Alliance for Open Media Video 1 (AV1) standard, the tile can be further subdivided into one or more superblocks (SBs). The SB is the starting point for block partitioning and can be further divided into multiple subblocks. The superblocks are then further subdivided into one or more blocks. Each block is the most basic element in a coding process. Optionally, an SB can contain several blocks (Bs).

[0053] The above method of dividing video frame images can be called a block partition structure. The following introduces some concepts in the encoding process:

[0054] Predictive coding includes intra-frame prediction and inter-frame prediction. The original video signal is predicted from a selected reconstructed video signal to obtain a residual video signal. The encoder needs to decide which predictive coding mode to choose for the current coding unit (or coding block) and inform the decoder. Intra-frame prediction refers to predicting a signal from a region within the same image that has already been encoded and reconstructed; inter-frame prediction refers to predicting a signal from another encoded image (a reference image) that is different from the current image.

[0055] Transform and Quantization: After the residual video signal undergoes transformation operations such as Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT), the signal is transformed into the transform domain, and these are called transform coefficients. The transform coefficients are then subjected to lossy quantization, losing some information to make the quantized signal more suitable for compression. In some video coding standards, there may be more than one transform method to choose from. Therefore, the encoder needs to select one of the transform methods for the current coding unit (or coding block) and inform the decoder. The fineness of quantization is usually determined by the quantization parameter (QP). A larger QP value means that coefficients with a wider range of values ​​will be quantized into the same output, which usually leads to greater distortion and a lower bit rate. Conversely, a smaller QP value means that coefficients with a smaller range of values ​​will be quantized into the same output, which usually leads to less distortion and a higher bit rate.

[0056] Entropy coding, or statistical coding, involves statistically compressing the quantized transform-domain signal based on the frequency of each value, ultimately outputting a binary (0 or 1) compressed bitstream. Simultaneously, other information generated during encoding, such as the selected coding mode and motion vector data, also requires entropy coding to reduce the bit rate. Statistical coding is a lossless coding method that effectively reduces the bit rate required to represent the same signal. Common statistical coding methods include Variable Length Coding (VLC) and Content-Adaptive Binary Arithmetic Coding (CABAC).

[0057] Context-Based Binary Arithmetic Coding (CABAC) primarily involves three steps: binarization, context modeling, and binary arithmetic coding. After binarizing the input syntax elements, the binary data can be encoded using either a regular coding mode or a bypass coding mode. The bypass coding mode eliminates the need to assign a specific probability model to each binary bit; the input binary bit bin value is directly encoded using a simple bypass encoder, thus accelerating the overall encoding and decoding speed. Generally, different syntax elements are not completely independent, and even identical syntax elements possess a certain degree of memory. Therefore, according to conditional entropy theory, using other encoded syntax elements for conditional coding can further improve coding performance compared to independent coding or memoryless coding. This encoded symbol information used as conditions is called the context. In the regular coding mode, the binary bits of the syntax elements sequentially enter the context modeler. The encoder assigns an appropriate probability model to each input binary bit based on the values ​​of previously encoded syntax elements or binary bits; this process is called context modeling. The context model corresponding to a grammatical element can be located using the context index increment (ctxIdxInc) and the context index start (ctxIdxStart). After the bin value and the assigned probability model are fed into the binary arithmetic encoder for encoding, the context model needs to be updated based on the bin value, which is the adaptive process in encoding.

[0058] Loop Filtering: The transformed and quantized signal undergoes inverse quantization, inverse transform, and prediction compensation to obtain a reconstructed image. Due to the effects of quantization, the reconstructed image differs from the original image in some aspects, resulting in distortion. Therefore, filtering operations can be performed on the reconstructed image, such as deblocking filters (DB), sample adaptive offset (SAO), or adaptive loop filters (ALF), to effectively reduce the distortion caused by quantization. Since these filtered reconstructed images will be used as a reference for subsequent coded images to predict future image signals, the above filtering operations are also called loop filtering, i.e., filtering operations within the coding loop.

[0059] It should be noted that after the video encoding device encodes the video data to obtain a video stream, the video encapsulation device can encapsulate the video stream. This encapsulates the video stream into a file container according to a preset file format to form a video encapsulation file for storage and transmission. Correspondingly, after receiving the video encapsulation file, the receiver can decapsulate it in the reverse manner of the encapsulation process and decode it in the reverse manner of the encoding process to obtain the video content. Optionally, the video encoding device and the video encapsulation device can be the same device or different devices; the device for decapsulating the video encapsulation file and the device for decoding can be the same device or different devices.

[0060] In this embodiment of the application, the encapsulation of a video stream using the Moving Picture Experts Group 2 Transport Stream (MPEG2-TS) standard is used as an example. MPEG2-TS is a standard format for transmitting video, audio and other data, and is particularly suitable for data transmission over imperfectly reliable media such as broadcasting or the Internet.

[0061] like Figure 3 As shown, the MPEG2-TS structure can be roughly divided into three layers from bottom to top. The bottom layer is the Elementary Stream (ES), which is the raw bitstream output by the encoder, such as the H.264 / H.265 stream output by the video encoder and the Advanced Audio Coding (AAC) stream output by the audio encoder. The middle layer is the Packetized Elementary Stream (PES) data packets. The ES stream is first encapsulated into PES, that is, a video or audio frame is encapsulated into a PES packet. The core of PES encapsulation is to add a PES header (PESHeader) to the beginning of the video or audio frame, thus obtaining the PES packet. The top layer is the Transport Stream (TS) data packets, that is, the PES packets are sliced, and a TS header (TS Header) is added to the beginning of each slice. The final TS packet size is fixed at 188 bytes. If it is less than 188 bytes, padding bits are added to the end of the TS packet. This is done by inserting an adaptation field. The MPEG2-TS stream is composed of individual TS data packets, which are implemented using Fields.

[0062] In addition to the PES and TS encapsulations mentioned above, the MPEG2-TS standard also defines various Program Specific Information (PSI) or Service Information (SI) to ensure that the decoding side can correctly identify and parse the MPEG2-TS stream. PSI stores metadata information needed for parsing and recognizing the MPEG2-TS stream, including the Program Association Table (PAT) and the Program Map Table (PMT). SI functions similarly to PSI, generally storing application-level information; SI is not mandatory.

[0063] The PAT mentioned above is mainly used to provide the Program ID (PID) information of the PMT. The PMT is mainly used to provide media stream information in the program stream. Optionally, the structure of the PMT can be as shown in Table 1 below:

[0064]

[0065] Table 1

[0066] Referring to Table 1, the PMT contains the following information in sequence:

[0067] table_id (an 8-bit unsigned integer): Indicates the type of table. For PMT, it is usually a specific value. In Table 1, uimsbf represents an unsigned integer.

[0068] section_syntax_indicator (1-bit binary string): Fixed at 1, indicating that the standard section syntax is followed. bslbf in Table 1 represents a binary string;

[0069] '0' (1-bit binary string): Reserved bit, usually set to 0;

[0070] reserved (2 bits of binary string): Reserved bits, usually set to 11;

[0071] section_length (12-bit unsigned integer): Represents the length from the next byte to the end of the section (excluding the CRC32 checksum), with a maximum value of 4093 bytes;

[0072] program_number (16-bit unsigned integer): Represents the program number to which the current PMT belongs, corresponding to the program_number in the PAT;

[0073] reserved (2 bits of binary string): Reserved bits, usually set to 11;

[0074] version_number (5-bit unsigned integer): PMT version number, updated when PMT content changes;

[0075] current_next_indicator (1-bit binary string): Indicates whether the PMT currently in use is the next one to be used.

[0076] section_number (an 8-bit unsigned integer): The number of the current section, which is usually 0x00 in a PMT;

[0077] last_section_number (an 8-bit unsigned integer): The number of the last section, which is usually 0x00 in a PMT;

[0078] reserved (3 bits of binary string): Reserved bits, usually set to 111;

[0079] PCR_PID (13-bit unsigned integer): The PID of the Program Clock Reference (PCR), used for the timestamp to synchronize the decoder;

[0080] reserved (4-bit binary string): Reserved bits, usually set to 1111;

[0081] program_info_length (12-bit unsigned integer): Indicates the total length of the descriptors that follow it. If it is 0, there are no descriptors.

[0082] The first for loop containing descriptor() contains one or more descriptor descriptors, providing additional information about the entire program; the exact number is determined by program_info_length.

[0083] stream_type (an 8-bit unsigned integer): Specifies the type of the basic stream, for example, 0x1B for H.264, 0x24 for H.265, and 0x0F for AAC;

[0084] reserved (3 bits of binary string): Reserved bits, usually set to 111;

[0085] elementary_PID (13-bit unsigned integer): The PID value that identifies the ES stream. The PID field also exists in the TS Header, so that the decoder knows which ES stream each TS packet belongs to.

[0086] reserved (4-bit binary string): Reserved bits, usually set to 1111;

[0087] ES_info_length (12-bit unsigned integer): Represents the total length of the descriptors that immediately follow. This part is the Elementary Stream Information (ES_info), which contains the descriptors associated with the basic stream.

[0088] The second for loop contains descriptor(): This part contains one or more descriptors that provide additional information about the underlying stream, the exact number of which is determined by ES_info_length;

[0089] CRC_32 (32 bits): Used to verify whether an error occurred during the transmission of the PMT.

[0090] In the PMT structure shown in Table 1, the second for loop containing `descriptor()` stores the stream information for all ES streams. The decoder can obtain which streams are included in the current TS stream by parsing the ES info part of the PMT. Furthermore, `ES_info_length` and the descriptor in the second for loop containing `descriptor()` are used to represent the stream's metadata. Metadata is crucial for the decoder; missing metadata may lead to decoding failure. The descriptor structure begins with an 8-bit tag and an 8-bit length, followed by the specific content of the descriptor. The structure of the descriptor content is determined by the specific descriptor.

[0091] In some embodiments of this application, when using MPEG2-TS to transmit HEVC (H.265) streams, the general processing procedure includes the following aspects:

[0092] 1. In PMT, stream_type is set to 0x24, and ES_info_length can be empty, meaning there is no descriptor;

[0093] 2. The parameter sets of the HEVC stream, namely the Video Parameter Set (VPS), Sequence Parameter Set (SPS), and Picture Parameter Set (PPS), need to be placed together with the video frame data. For example, the parameter set must be placed together with the keyframe. This method is specified in the Annex-B specification of the HEVC standard (which is a bitstream format specification in the video coding standard) and is usually referred to as the Annex-B method.

[0094] 3. Since HEVC streams need to be stored in Annex-B format in MPEG2-TS, each Network Abstract Layer Unit (NALU) of the HEVC stream needs to have a start code (such as 0x00 0x00 0x000x01 or 0x00 0x00 0x01).

[0095] 4. Typically, each HEVC frame consists of one or more NALUs. The first NALU in each frame must be an Access Unit Delimiter (AUD) NALU. The purpose of this AUD NALU is to distinguish between different frames. The decoder can easily differentiate between different image frames using the AUD NALU. The AUD NALU structure is shown in Table 2 below:

[0096]

[0097] Table 2

[0098] Referring to Table 2, the AUD NALU contains pic_type, which is a 3-bit unsigned integer (u(3)) used to indicate the type of image in the currently accessed unit. This field helps the decoder understand the type of frame to be processed, thereby selecting an appropriate decoding strategy. Optionally, in H.264 / AVC, the value of pic_type and its meaning are as follows:

[0099] 0: Indicates an intra-coded frame (I-frame), which is a frame entirely encoded by internal information and does not rely on other frames for prediction;

[0100] 1: Indicates a predicted-coded frame (P-frame), which is obtained by forward prediction from a previous reference frame;

[0101] 2: Represents a bidirectional predicted-coded frame (B-frame), which is obtained by bidirectional prediction from the preceding and following reference frames.

[0102] 3 to 7: Retain unused items.

[0103] Referring to Table 2, the `rbsp_trailing_bits()` function call within the AUD NALU is used to add necessary trailing bits to ensure the correct end of the Raw Byte Sequence Payload (RBSP) data. In the RBSP, each NALU's data may require padding with additional bits to ensure byte alignment or meet specific encoding requirements. `rbsp_trailing_bits()` is responsible for adding these padding bits and typically includes a termination bit and several trailing bits. Optionally, the termination bit can be a fixed 1, while the trailing bits can be several 0 bits until the desired byte boundary is reached.

[0104] In some optional embodiments, the structure of the HEVC NALU header can be as shown in Table 3:

[0105]

[0106] Table 3

[0107] As shown in Table 3, the first bit of the first byte of the NALU header is a forbidden_zero_bit, which defaults to 0, indicating that the NALU is valid. Otherwise, it indicates that the NALU is invalid, so that the receiver can correct errors or discard the NALU.

[0108] The second to seventh bits of the first byte of the NALU header can be used to indicate the type of NALU (nal_unit_type), including 32 categories for indicating encoded data (VCL NALU) and 32 categories for indicating other related data (non-VCL NALU). Among the other related data (non-VCL NALU), there are categories for indicating VPS, SPS, PPS and SEI respectively.

[0109] The last bit of the first byte and the first five bits of the second byte of the NALU header are the NALU layer ID (nuh_layer_id), which defaults to 0 and can be used to identify the viewpoint.

[0110] The last three bits of the second byte of the NALU header are the NALU temporal level label + 1 (nuh_temporal_id_plus1), which defaults to 1. Subtracting 1 from the value can be used to indicate the NALU temporal level label.

[0111] It should be noted that the right column in Table 3 represents the character type, such as f(1) representing the forbidden bit, which occupies the first bit of the NALU header; u(6) representing an unsigned 6-bit integer; u(3) representing an unsigned 3-bit integer.

[0112] Multiview HEVC streams are essentially multi-layer HEVC streams, typically with two viewpoints (i.e., perspectives): a left viewpoint and a right viewpoint. One viewpoint is the primary viewpoint, and the other is the secondary viewpoint; generally, the left viewpoint is the primary viewpoint, and the right viewpoint is the secondary viewpoint, but this can be determined based on the user's selection. The primary viewpoint uses layer 0 of the HEVC stream, called the Base Layer, specifically defined by a nuh_layer_id of 0 in the nal_unit_header. The secondary viewpoint's nuh_layer_id needs to be greater than 0, typically 1, but can also be set to other values ​​greater than 0; this is called the Secondary Stereo Layer.

[0113] In a multiview HEVC stream, different layers are encoded separately, so each layer has its own parameter set. The VPS is shared by different layers and there is only one copy, while the SPS and PPS are independent copies for each layer. The parameter sets of different layers are distinguished and correspond to each other through the nuh_layer_id in the nal_unit_header.

[0114] The parameter set information of a normal HEVC stream is stored using HEVCDecoderConfigurationRecord (the specific syntax structure is shown in Table 4). However, since a multiview HEVC stream has two layers of bitstreams, and different layers require different parameter set information, using only HEVCDecoderConfigurationRecord is insufficient to store complete parameter set information. Therefore, LHEVCDecoderConfigurationRecord (the specific syntax structure is shown in Table 5) can be defined to store the parameter set information of other layers.

[0115]

[0116] Table 4

[0117] Referring to Table 4, the syntax structure of HEVCDecoderConfigurationRecord is shown. Specifically, ConfigurationVersion = 1 indicates that the configuration version is 1. The parameter general_profile_space indicates whether the current NALU belongs to a general profile space. For example, a value of 1 indicates that the NALU conforms to the standard HEVC profile requirements, while a value of 0 indicates that the NALU belongs to a specific non-standard profile space, which can be used for specific applications or extensions. The parameter general_tier_flag indicates the NALU's tier information, such as Baseline Tier, Main Tier, and High Tier. The parameter general_profile_idc indicates the identifier of the general profile to which the NALU belongs. The parameter general_profile_compatibility_flags indicates the HEVC profiles that the current NALU is compatible with, allowing the decoder to determine whether the NALU is compatible with a specific HEVC profile and tier.

[0118] The parameter `general_constraint_indicator_flags` indicates whether the current NAL unit meets a series of specific coding constraints. These constraints are designed to ensure consistent video quality across different decoders and playback devices by satisfying certain characteristics or requirements of video coding. Constraints can include intra-frame prediction constraints, quantization parameter constraints, color space constraints, chroma sampling format constraints, and cyclic filtering constraints. The parameter `general_level_idc` indicates the HEVC coding level used by the current NALU. The HEVC standard defines several different levels, each corresponding to specific coding parameters and performance requirements, such as bit rate, frame rate, and resolution. The parameter `min_spatial_segmentation_idc` indicates the identifier of the minimum spatial segmentation level supported by the encoder, associated with SPS. HEVC supports dividing frames into different regions and encoding each region independently, applying different coding parameters to different regions to improve coding efficiency and video quality.

[0119] The parameter `parallelismType` indicates a parameter or identifier associated with HEVC parallel processing. This parameter is related to SPS, and parallel coding techniques can include slicing, threading, etc. The parameter `chromaFormat` indicates the sampling format of the chroma components in the video sequence, such as 4:2:0, 4:2:2, and 4:4:4. Chroma components typically represent the color information of an image, and the chroma sampling format determines how the chroma components are sampled and their resolution relative to the luma components. The parameter `bitDepthLumaMinus8` is a parameter in SPS that indicates the bit depth of the luma component (Luma) minus 8. Bit depth refers to the number of bits used per pixel value, which determines the color gamut and precision of the image. In HEVC, the bit depth of the luma component can be any value from 8 to 16 bits; bits are saved by subtracting 8 and storing the difference. The parameter `bitDepthchromaMinus8` is a parameter in SPS that indicates the bit depth of the chroma component minus 8. Bits are saved by subtracting 8 from its value and storing the difference. The bit depth of the luminance component determines its sampling precision, while the bit depth of the chroma component determines its color precision.

[0120] The `reserved` field is reserved for future use. The `reserved` field is optional, has a variable length, and its content should not be decoded or used during decoding. This field is reserved for future standard versions or applications to support possible extensions or other features. All bits in this field are set to 1, such as `reserved='11111'b` and `reserved='111111'b` in Table 4, which are standard placeholders. The parameter `avgFrameRate` is the average frame rate, indicating the average display rate of frames in the video sequence, configured in SPS and VPS. The parameter `constantFrameRate` is a parameter in SPS indicating whether the video sequence has a constant frame rate; if so, the value is 1, otherwise 0. The parameter `numTemporalLayers` indicates the number of temporal layers in the video sequence, used for hierarchical coding. Hierarchical coding refers to encoding the video sequence into multiple different quality layers or temporal layers to support different network conditions and device capabilities. The temporal layer allows the encoder to encode video at different temporal resolutions, thus providing better temporal scalability.

[0121] The `temporalIdNested` parameter indicates whether nested temporal hierarchy is enabled. Temporal hierarchy is a hierarchical architecture used to describe dependencies between video frames. The `lengthsizeMinusone` parameter indicates the size of the NALU's length field minus one. This parameter is set in SPS or VPS and applies to all NALU units. The `numOfArrays` parameter represents the array length. The `array_completeness` parameter indicates whether the current array contains a complete set of parameters. A value of 0 indicates that the current array is incomplete, containing an undefined field or an unfilled byte; a value of 1 indicates that the current array is complete. The `NAL_unit_type` parameter indicates the type of NALU, such as SPS, PPS, etc. The `numNalus` parameter indicates the number of NALUs. The `nalUnitLength` parameter indicates the length of the current NALU inside the for loop. The `bit(8*nalUnitLength)nalUnit` parameter indicates the actual data of the current NALU, with a length of 8 times `nalUnitLength` bits.

[0122] The syntax structure of LHEVCDecoderConfigurationRecord is shown in Table 5:

[0123]

[0124] Table 5

[0125] Referring to Table 5, the syntax structure of LHEVCDecoderConfigurationRecord is very similar to that of HEVCDecoderConfigurationRecord, except that some fields are reduced. ConfigurationVersion=1 indicates configuration version 1. Similarly, the parameter min_spatial_segmentation_idc indicates the identifier of the minimum spatial segmentation level supported by the encoder; the parameter parallelismType indicates the parameter or identifier for parallel processing with HEVC; the parameter numTemporalLayers indicates the number of temporal layers in the video sequence, used for hierarchical encoding; the parameter temporalIdNested indicates whether nested temporal layers are enabled; the parameter lengthsizeMinusone indicates the size of the NALU length field minus one; the parameter numOfArrays indicates the length of the array; the parameter array_completeness indicates whether the current array contains the complete parameter set; the parameter NAL_unit_type indicates the type of NALU; the parameter numNalus indicates the number of NALUs; and the parameter nalUnitLength indicates the length of the current NALU inside the for loop. The parameter bit(8*nalUnitLength) indicates the actual data of the current NALU, with a length of 8 times nalUnitLength bits.

[0126] In addition to the parameter set information VPS, SPS, and PPS, the multiview HEVC stream also requires a supplemental enhancement information (SEI) NALU, hereinafter referred to as 3d SEI, called three-dimensional reference display information (SEI). The specific syntax structure is shown in Table 6.

[0127]

[0128] Table 6

[0129] Referring to Table 6, the parameter `prec_ref_display_width` indicates the pre-decoding display width of the reference image, used to inform the decoder of the original display size of the reference image so that it can be correctly scaled or processed during decoding. The parameter `ref_viewing_distance_flag` indicates whether there is viewing distance information for the reference view so that it can be properly processed and displayed during decoding. The parameter `prec ref viewing dist` indicates the pre-decoding viewing distance. The parameter `num_ref_displays_minus1` indicates the number of reference displays minus 1, that is, the number of display devices or screens used to present the reference image minus 1. The above 3D SEI takes two viewpoints as an example, including the left viewpoint and the right viewpoint, represented by the parameters `left viewid[i]` and `right viewid[i]`. For example, in the NALU header, the left viewpoint is the main viewpoint with a layerID of 0, and the right viewpoint has a layerID of 1. Then the value of `leftview id[i]` is 0, and the value of `right view id[i]` is 1. In this loop, other viewpoints, such as multiple secondary viewpoints, can also be included.

[0130] The parameter `exponent_ref_display_width[i]` indicates the exponent portion of the reference display width, and the parameter `mantissa_ref_display_width[i]` indicates the mantissa portion of the reference display width. The parameter `exponent_ref_viewing_distance[i]` indicates the exponent portion of the reference viewing distance. The parameter `mantissa_ref_viewing_distance[i]` indicates the mantissa portion of the reference viewing distance. The parameter `additional_shift_present_flag[i]` indicates whether additional offset information exists. `num_sample_shift_plus512[i]`: indicates an additional offset, typically used to adjust the size or position of an image. The parameter `three_dimensional_reference_displays_extension_flag` indicates whether three-dimensional reference display extension information exists. If this flag is true, more parameters may be included to describe information about the three-dimensional reference display.

[0131] It should be noted that the aforementioned 3D SEI stores parameters required for playing multi-view videos, such as viewpoint ID and rendering precision. The parameters left view id[i] and right view id[i] can be derived from the VPS, while the values ​​of the remaining parameters can be configured by the user. Therefore, the player can render and play the video according to the user's desired rendering based on the information stored in this 3D SEI.

[0132] Considering the limitations of the encapsulation methods for multiview HEVC streams proposed in related technologies, which restricts the application of multiview HEVC streams, the technical solution of this application proposes a new video stream processing scheme, which expands the encapsulation methods for multiview HEVC streams, increases the application scenarios of multiview HEVC streams, and meets the growing demand of users for multiview video content.

[0133] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0134] Figure 4 A flowchart illustrating a video stream processing method according to an embodiment of this application is shown. This video stream processing method can be executed by a device with computing processing capabilities, such as a terminal device or a server. (Refer to...) Figure 4 As shown, the video stream processing method includes at least S410 to S430, which are detailed below:

[0135] In S410, the video stream to be encapsulated is parsed to obtain the video encoding parameter set and video frame data of the video stream.

[0136] In some optional embodiments, the video stream can be a video bitstream obtained by encoding video content, also known as a video encoded stream. This video stream can be a HEVC stream, AVC stream, Multiview HEVC stream, etc. For the transmission and storage of the video content, the video encapsulation device can encapsulate the video stream to obtain a video encapsulation file. That is, the video stream is the video stream to be encapsulated. Optionally, the video stream to be encapsulated can be received by the video encapsulation device from other devices, or it can be obtained by encoding the video content itself; this application does not limit this.

[0137] In some alternative embodiments, the video encoding parameter set of the video stream is mainly used to describe the encoding parameters of the video stream, such as VPS, SPS, PPS, etc.

[0138] In some optional embodiments, the video frame data is used to represent specific video content and may be encoded data obtained by encoding video frames.

[0139] In S420, if it is determined from the video coding parameter set that the video stream contains a multi-view video stream, a program mapping table corresponding to the multi-view video stream is generated. A registration descriptor is added to the program mapping table, and the value of the format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream.

[0140] In some optional embodiments, a multi-view video stream contains video content from multiple viewpoints. This video content can be a sequence of videos captured simultaneously from different angles by a camera array. For example, a multi-view video stream can be a video stream with two viewpoints. These two viewpoints can be a sequence of videos captured simultaneously from two angles by a camera array, and can be viewed by the user's left and right eyes respectively. Based on the parallax of the images seen by the user's left and right eyes (the difference in line of sight generated by binoculars), the user can have a three-dimensional stereoscopic visual experience.

[0141] In some alternative embodiments, the multiview video stream may be a multiview HEVC stream. When determining whether a video stream is a multiview video stream, NAL units can be extracted from the video stream, and the nal_unit_type field in the NAL unit header can be checked. If a specific NAL unit type exists, it may indicate that this is a multiview HEVC video stream.

[0142] In some optional embodiments, since there is a special parameter set VPS in a multiview HEVC stream, the presence of a VPS NAL unit can be checked. If a VPS NAL unit exists, the VPS can be parsed to determine whether the video stream is a multiview video stream. For example, the VPS contains information describing the relationships between all views, such as the field vps_max_layers_minus1 representing the maximum number of layers minus 1. If this value is greater than 0, it indicates that this is a multi-layer or multi-view video stream; another example is the field vps_num_views_minus1 representing the number of views minus 1. If this value is greater than 0, it confirms that this is a multiview video stream.

[0143] In some alternative embodiments, since the SEI message in an HEVC stream can contain additional information about the video stream, a specific SEI message may be included for a multiview HEVC stream to provide information about the multi-view content. Therefore, the three_dimensional_reference_displays_info SEI message can be searched and parsed. This message contains crucial information such as left and right view IDs and rendering precision, which is essential for correctly decoding and displaying 3D content. If this message is found, it can be determined that the video stream is a multiview video stream.

[0144] In some alternative embodiments, for multiview video streams, when encapsulated with MPEG2-TS, a program map table (PMT) for the multiview video stream can be generated, and a registration descriptor can be added to the PMT to indicate that the video stream is a multiview video stream by the value of the format identifier in the registration descriptor.

[0145] In one specific embodiment, the syntax structure of the registration descriptor can be as shown in Table 7:

[0146]

[0147] Table 7

[0148] Referring to Table 7, the Registration Descriptor contains the following information:

[0149] The descriptor_tag (an 8-bit unsigned integer) is used to uniquely identify the type of the descriptor. Each type of descriptor has a unique descriptor_tag value. Therefore, the decoder can identify the current descriptor as a Registration Descriptor through this tag.

[0150] The descriptor_length (an 8-bit unsigned integer) represents the length of the remaining part of the descriptor (excluding the descriptor_tag and descriptor_length itself), in bytes. It is mainly used to help the decoder determine the end position of the descriptor content and ensure that subsequent data is read correctly.

[0151] The `format_identifier` (a 32-bit unsigned integer) is used to uniquely identify the specific format or protocol followed by the video stream. In the embodiments of this application, the `format_identifier` can be used to identify whether the video stream is a multiview video stream. For example, for a multiview HEVC stream, the `format_identifier` can be set to "MVHV" to identify it as a multiview HEVC stream, which allows the decoder to perform corresponding processing according to the requirements of the specific format.

[0152] The `for` loop structure indicates that the `additional_identification_info` field may appear multiple times, the exact number determined by `N`. This design allows the descriptor to include multiple additional identification information items to adapt to different scenario requirements. The `additional_identification_info` (an 8-bit binary string) provides additional identification information. When `format_identifier` is insufficient to fully describe the video stream, the `additional_identification_info` field can provide more detailed information. For example, in some cases, it may contain version numbers, extension flags, etc.

[0153] Based on the structure of the registration descriptors shown in Table 7, the value of the format identifier format_identifier can be used to identify whether a video stream is a multi-view video stream.

[0154] In some optional embodiments, if it is determined from the video coding parameter set that the video stream contains a multiview video stream, then 3D reference display information can also be generated based on the video coding parameter set. This 3D reference display information contains identification information of multiple viewpoints in the multiview video stream, and then the 3D reference display information is added to the Program Map (PMT). Optionally, for a multiview HEVC stream, the 3D reference display information can be the 3dSEI described in the foregoing embodiments, and its specific syntax structure can be referred to Table 6 above.

[0155] In some optional embodiments, the Program Map (PMT) contains Basic Stream Information (ES_info). The registration descriptor set in the above embodiments can be the first descriptor in the Basic Stream Information, while the 3D reference display information can be added to the second descriptor in the Basic Stream Information. In this embodiment, by setting the first descriptor in the PMT's ES_info to the registration descriptor in the above embodiments, the current video stream can be marked as a multi-view video stream through this registration descriptor. This allows the decoder to recognize that this is a special stream type and take appropriate processing measures accordingly. Meanwhile, since the 3D reference display information contains key information required for playing multi-view video, such as left and right view IDs and rendering precision, this information is crucial for correctly decoding and displaying 3D content. Therefore, by including the 3D reference display information as part of the descriptor in the PMT's ES_info, it can be ensured that these important parameters are transmitted along with the video stream, enabling the decoder to obtain and utilize this information for correct rendering. Furthermore, by placing the 3D reference display information in the second descriptor of ES_info, the decoder can decode the 3D reference display information as early as possible to render the multi-view video.

[0156] In some optional embodiments, the tag value of the second descriptor can be a set value, and the length of the second descriptor is the length of the 3D reference display information. It should be noted that since the standard defines various descriptors, the tag value in the range of 0x40 to 0xFF is reserved for user-defined or custom purposes. The standard also uses values ​​from 0x40 to 0x7F. To prevent conflicts, the tag value of the second descriptor can be set to a larger value (e.g., 0xA0). This avoids conflict with existing standards while providing sufficient flexibility to support new application requirements.

[0157] In some optional embodiments, basic stream information of the multi-view video stream can also be added to the program map table, such as the video stream type (stream_type) and total descriptor length (ES_info_length) shown in Table 1.

[0158] In S430, the program mapping table, video encoding parameter set, and video frame data are encapsulated to generate transport stream data packets.

[0159] In some optional embodiments, the video coding parameter set and video frame data can be encapsulated as a whole, while the program map table (PMT) can be encapsulated separately to generate the encapsulated transport stream data packet. In this embodiment, by encapsulating the video coding parameter set together with the video frame data, the decoder can immediately obtain the necessary parameter set information upon receiving a keyframe without additional lookup. This reduces waiting time and complexity, improves decoding speed, and is also compatible with existing standards. Furthermore, by encapsulating the PMT separately, the correlation between different primary streams (such as video content from different viewpoints) becomes clearer, facilitating accurate identification and processing of each stream by the receiving end. This is particularly important for supporting multi-view video streams.

[0160] In some optional embodiments, encapsulating the video encoding parameter set and video frame data as a whole can specifically involve encapsulating the video encoding parameter set and keyframe data in the video stream as a whole.

[0161] In some optional embodiments, when generating transport stream data packets through encapsulation processing, video frame data can be written into the transport stream data packets sequentially according to the arrangement order of the video frames contained in the video stream; wherein each video frame contains multiple video data encapsulation units corresponding to multiple viewpoints. In this embodiment, since multiple viewpoints in a multi-view video stream are rendered simultaneously, by ensuring that each video frame contains multiple video data encapsulation units corresponding to each viewpoint, it can be guaranteed that the multi-view video stream can be rendered correctly. Optionally, the video data encapsulation unit can be a NALU; if the multi-view video stream contains two viewpoints, then each video frame needs to contain two NALUs corresponding to each viewpoint.

[0162] In some optional embodiments, the first video data encapsulation unit included in each video frame is an Access Entity Delimiter (AUD) video data encapsulation unit, which is used to indicate the type of each video frame. Optionally, the AUD video data encapsulation unit can be an AUD NALU, and the specific structure can refer to the technical solutions of the foregoing embodiments.

[0163] In some optional embodiments, each video data encapsulation unit includes a start code, which indicates the starting position of the video data encapsulation unit. Optionally, the start code may be 0x000x00 0x00 0x01 or 0x00 0x000x01, etc.

[0164] Figure 4 This description focuses on the technical solutions of the embodiments of this application from the perspective of video stream encapsulation. The following is a combination of... Figure 5 The technical solutions of the embodiments of this application will be described from the perspective of decapsulating video encapsulation files.

[0165] Figure 5 A flowchart illustrating a video stream processing method according to an embodiment of this application is shown. This video stream processing method can be executed by a device with computing processing capabilities, such as a terminal device or a server. (Refer to...) Figure 5 As shown, the video stream processing method includes at least S510 to S530, which are detailed below:

[0166] In S510, a transport stream data packet is obtained. This transport stream data packet is obtained by encapsulating the program map table, the video encoding parameter set of the video stream, and the video frame data. A registration descriptor is added to the program map table. The value of the format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream.

[0167] Optionally, the relevant descriptions in the transport stream data packets, program mapping tables, and registration descriptors can refer to the technical solutions of the foregoing embodiments, and will not be repeated here.

[0168] In S520, the transport stream data packets are decapsulated to obtain video data encapsulation units, as well as program mapping tables and video coding parameter sets.

[0169] In some optional embodiments, since the transport stream (TS) packets are of a fixed length of 188 bytes and begin with a 4-byte header containing a PID (Process ID) to identify which elementary stream or control information the packet belongs to, the PMT can be parsed when the PID of a TS packet corresponds to a program's PMT. The PMT contains information about all elementary streams within the program, including the PID and stream type. For multiview HEVC streams, special attention must be paid to the registration descriptor and the second descriptor carrying 3D reference display information. Therefore, if a registration descriptor with a set format_identifier (such as "MVHV" in the example above) is found, it indicates that this is a multiview HEVC stream.

[0170] Meanwhile, since video encoding parameter sets are usually transmitted along with video frame data (such as keyframes) and carried in NALUs, the decoder can parse these parameter sets in order to correctly decode subsequent video frames.

[0171] In some optional embodiments, for each TS packet of the video elementary stream, the PES packet is first reassembled based on the packet header information, and then the NALU is further parsed. The boundaries of the NALU can be located using the start code (0x000001 or 0x00000001). Different types of NALUs are distinguished based on the nal_unit_type field in the NALU header, such as AUD NALU, VPS, SPS, PPS, and specific video frame data.

[0172] In S530, the video data encapsulation unit is reorganized according to the program mapping table and the video coding parameter set to obtain a multi-view video stream.

[0173] In some optional embodiments, for multi-view video streams, each frame contains multiple NALUs corresponding to different viewpoints (e.g., left and right viewpoint NALUs). These multiple viewpoint NALUs are treated as a complete frame for processing. The previously extracted 3D reference display information is used to correctly configure the 3D display settings, ensuring that different viewpoints can be displayed synchronously and accurately. After completing the above steps, the decoder will obtain the correct video frame data, ready for display or further processing (such as transcoding, analysis, etc.).

[0174] In summary, the technical solution of this application extends the encapsulation method of multiview video streams, increases the application scenarios of multiview video streams, and meets the growing demand of users for multiview video content. The following description uses MPEG2-TS encapsulation of a multiview HEVC stream as an example to illustrate the technical solution of this application:

[0175] In some optional embodiments, when encapsulating a multiview HEVC stream using MPEG2-TS, the stream_type in the PMT uses the same value as a normal HEVC stream, i.e., 0x24. To indicate that the current stream is a multiview HEVC stream, a registration descriptor can be written according to the structure in Table 7 above, with descriptor_tag set to 0x5, descriptor_length set to 4, and format_identifier defined as "MVHV," an abbreviation for multiview HEVC.

[0176] In some optional implementations, the 3D SEI required by the multiview HEVC stream can be placed in the second descriptor of ES_info, where the descriptor's tag is defined as 0xA0 to avoid conflicts with other custom values. The length of the descriptor is the size of the 3D SEI, and the content of the descriptor is the content of the 3D SEI.

[0177] In some optional embodiments, for the sake of simplicity and compatibility with ordinary HEVC streams, the parameter set transmission and storage of multiview HEVC streams are kept consistent with those of ordinary HEVC streams. That is, the parameter set of multiview HEVC streams can be placed together with keyframes. In this way, if the parsing side does not support multiview HEVC streams, it can only parse and decode the parameter set and frame data of the base layer, which will not cause errors in incompatible decoders. Meanwhile, decoders that are compatible with multiview HEVC streams can successfully parse all the required layer parameter set information.

[0178] In some optional implementations, consistent with the requirements of a regular HEVC stream, the frame data of a multiview HEVC stream can use Annex-B format, meaning each NALU has a start code, and each NALU in each frame must be an AUD NALU. Compared to a regular HEVC stream, a multiview HEVC frame has two viewpoints, each corresponding to a separate NALU: one for the Base Layer and one for the Secondary Stereo Layer. Because the left and right viewpoints need to be rendered simultaneously, their timing information must be identical. Therefore, these two NALUs must be combined and considered as a complete frame.

[0179] Extending MPEG2-TS using the methods described above allows for efficient storage and transmission of Multiview HEVC. The specific encapsulation process is as follows: Figure 6 As shown:

[0180] Reference Figure 6 As shown, a video encapsulation processing method according to an embodiment of this application includes the following steps:

[0181] S601, acquire video stream.

[0182] In some optional embodiments, the encoder can encode and output an HEVC stream based on user-defined encoder parameters and video frame (e.g., YUV, RGB) data. In this step, the user can choose to encode a normal HEVC stream or a multiview HEVC stream.

[0183] S602, parses the parameter set of the video stream.

[0184] In some optional embodiments, parsing the parameter set of the video stream is mainly used in S603 to determine whether the current HEVC stream is a normal HEVC stream or a multiview HEVC stream. This step can also obtain parameter set information (such as VPS, SPS, PPS) needed for subsequent steps, including parameter set information for the Base Layer and Secondary Stereo Layer.

[0185] S603, determine whether the current HEVC stream is a multiview HEVC stream. If yes, execute S604; if it is a normal HEVC stream, execute S607.

[0186] S604 generates 3D SEI.

[0187] In some optional embodiments, a 3D SEI can be generated based on the VPS and user-configured parameters, where left_view_id and right_view_id have been obtained in the parameter parsing stage of S102, and the remaining parameters are configured by the user.

[0188] S605, write the registration descriptor in the PMT.

[0189] In some optional embodiments, the registration descriptor written in the PMT can be a registration descriptor defined for the multiview HEVC stream, and the specific technical solution can be referred to in the foregoing embodiments.

[0190] S606, writes the 3D SEI descriptor in the PMT.

[0191] In some alternative embodiments, the 3D SEI can be written into a second descriptor in the PMT.

[0192] S607, write other information in PMT.

[0193] In some optional embodiments, basic stream information of the multi-view video stream can also be added to the PMT, such as the video stream type (stream_type) and total descriptor length (ES_info_length) shown in Table 1.

[0194] S608 writes AVC / HEVC frames and generates TS packets.

[0195] In some optional embodiments, video frame data can be written to the TS packets of the MPEG2-TS stream in the order of the HEVC stream. When the stream is multiview HEVC, each video frame contains a left-view NALU and a right-view NALU until the end.

[0196] according to Figure 6 As shown in the encapsulation process, if the video stream to be encapsulated is a regular HEVC stream, then steps S601, S602, S607, and S608 can be used to encapsulate the video stream. If the video stream to be encapsulated is a Multiview HEVC stream, then in addition to executing steps S601, S602, S607, and S608, steps S604, S605, and S606 are also required to complete the encapsulation process for the Multiview HEVC stream.

[0197] The following describes some application scenarios to which the technical solutions of the embodiments of this application are applicable:

[0198] (1) Video live streaming scenario

[0199] Reference Figure 7 The diagram illustrates the application of the technical solution of this application in a live video streaming scenario. Figure 7 As shown, in a live video streaming scenario, the video encapsulation device can be an electronic device used by the broadcaster. This device provides functions for video production, encoding, and encapsulation of multi-viewpoint videos, and can run a live video streaming application. The video decapsulation device is an electronic device used by viewers of the broadcast. It provides functions for decapsulating video files, rendering and playing the multi-viewpoint videos. This device can include multiple displays, each used to display video content corresponding to multiple viewpoints.

[0200] Specifically, the video encapsulation device can acquire the video stream to be encapsulated, then parse the video stream to obtain the video encoding parameter set and video frame data. If the video encoding parameter set determines that the video stream contains a multi-view video stream, a program map table corresponding to the multi-view video stream is generated. A registration descriptor is added to this program map table, and the value of the format identifier in the registration descriptor indicates that the video stream is a multi-view video stream. Then, the program map table, video encoding parameter set, and video frame data can be encapsulated to generate a transport stream (TS) data packet. Optionally, the video encapsulation device can encode and encapsulate the audio content corresponding to the video content of the video stream, and the resulting TS data packet may include an audio encapsulation packet.

[0201] Furthermore, after obtaining the TS data packet, the video encapsulation device can push the TS data packet to the cloud server using a streaming tool. Upon receiving the TS data packet, the cloud server processes it according to user requirements, such as transcoding, noise reduction, enhancement, and analysis. Alternatively, the video encapsulation device can also push the TS data packet to a content delivery network (CDN) using a streaming tool. The cloud server can then retrieve the TS data packet from the CDN and process it according to user requirements. Finally, the viewer's video decapsulation device can retrieve the TS data packet through a live video application, perform decapsulation and decoding processing, and render the decoded video content based on display parameters to display (play) the video content from the TS data packet.

[0202] Optionally, the video decoding device can also decapsulate the audio packet to obtain the audio stream, and then decode the audio stream to obtain the audio content corresponding to the video content. This allows the audio content to be played synchronously while the video content is played. This approach further expands the application scenarios of multi-view video streams, meeting the transmission and storage needs of multi-view video. Thus, the entire video cloud chain, from encoding, streaming, cloud processing, and distribution, can support multi-view video, satisfying the needs for viewing, transmitting, storing, and processing multi-view video.

[0203] (2) Video on demand scenario

[0204] Video on Demand (VOD) refers to playing video content according to the request of viewers; it can also be understood as transmitting content that a user wants to watch (click or select) to the requesting user. (See also...) Figure 8 The diagram illustrates the application of the technical solution of this application in a video-on-demand scenario. Figure 8 As shown, in a video-on-demand scenario, the video encapsulation device can be an electronic device of a platform providing video content (such as immersive media content) or a user. This video encapsulation device can provide functions for video production, video encoding, and video encapsulation of multi-viewpoint videos. The video decapsulation device is an electronic device used by viewers using the video-on-demand function. The video decapsulation device can provide functions for decapsulating video files, thereby rendering and playing multi-viewpoint videos. This video decapsulation device can include multiple displays, each used to display video content corresponding to multiple viewpoints.

[0205] Specifically, the video encapsulation device can acquire the video stream to be encapsulated, then parse the video stream to obtain the video encoding parameter set and video frame data. If the video encoding parameter set determines that the video stream contains a multi-view video stream, a program map table corresponding to the multi-view video stream is generated. A registration descriptor is added to this program map table, and the value of the format identifier in the registration descriptor indicates that the video stream is a multi-view video stream. Then, the program map table, video encoding parameter set, and video frame data can be encapsulated to generate a TS data packet. Optionally, the video encapsulation device can encode and encapsulate the audio content corresponding to the video content of the video stream, and the resulting TS data packet may include an audio encapsulation packet.

[0206] Furthermore, after obtaining the TS data packet, the video encapsulation device can store the TS data packet in cloud storage, such as a cloud server or other devices that process TS data packets. The TS data packet can then be retrieved from the cloud storage and processed. Subsequently, the video decapsulation device can retrieve the corresponding TS data packet from the cloud storage based on the user-selected video content, perform decapsulation and decoding processing on the TS data packet, and render the decoded video content based on display parameters to display (play) the video content of the TS data packet. Figure 8 Taking a user wearing VR glasses to watch the video content corresponding to the TS data packet as an example, it can include video content corresponding to two viewpoints respectively. The VR glasses can also play the corresponding audio content in the TS data packet. In this way, the transmission and storage needs of multi-view video can be met.

[0207] It should be noted that the above embodiments are illustrated using multiview HEVC as an example. In other embodiments of this application, other types of multiview video streams may also be used. Furthermore, the technical solutions of the embodiments of this application are not limited to the application scenarios listed above, but can be applied to all application scenarios involving multiview video stream encapsulation.

[0208] The following describes an apparatus embodiment of this application, which can be used to execute the video stream processing method described above in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the video stream processing method described above in this application.

[0209] Figure 9 A block diagram of a video stream processing apparatus according to an embodiment of the present application is shown. The video stream processing apparatus can be applied to a device with computing processing capabilities, such as a terminal device or a server.

[0210] Reference Figure 9As shown, a video stream processing apparatus 900 according to an embodiment of this application includes: a parsing unit 902, a generation unit 904, and a processing unit 906.

[0211] The system includes: a parsing unit 902 configured to parse the video stream to be encapsulated, obtaining the video encoding parameter set and video frame data of the video stream; a generation unit 904 configured to generate a program mapping table corresponding to the multi-view video stream if it is determined from the video encoding parameter set that the video stream contains a multi-view video stream, wherein a registration descriptor is added to the program mapping table, and the value of the format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream; and a processing unit 906 configured to encapsulate the program mapping table, the video encoding parameter set, and the video frame data to generate a transport stream data packet.

[0212] In some embodiments of this application, based on the foregoing scheme, the generation unit 904 is further configured to: if it is determined from the video coding parameter set that the video stream contains a multi-viewpoint video stream, generate three-dimensional reference display information from the video coding parameter set, add the three-dimensional reference display information to the program mapping table, wherein the three-dimensional reference display information contains identification information of multiple viewpoints in the multi-viewpoint video stream.

[0213] In some embodiments of this application, based on the foregoing scheme, the program mapping table contains basic stream information, the registration descriptor is the first descriptor in the basic stream information, and the three-dimensional reference display information is added to the second descriptor in the basic stream information.

[0214] In some embodiments of this application, based on the foregoing scheme, the label value of the second descriptor is a set value, and the length value of the second descriptor is the length of the three-dimensional reference display information.

[0215] In some embodiments of this application, based on the foregoing scheme, the processing unit is configured to encapsulate the video encoding parameter set and the video frame data as a whole, and encapsulate the program mapping table separately to generate encapsulated transport stream data packets.

[0216] In some embodiments of this application, based on the foregoing scheme, the processing unit 906 is configured to encapsulate the video encoding parameter set and the keyframe data in the video stream as a whole.

[0217] In some embodiments of this application, based on the foregoing scheme, the processing unit 906 is configured to: sequentially write the video frame data into the transport stream data packet according to the arrangement order of the video frames contained in the video stream; wherein each video frame contains multiple video data encapsulation units corresponding to different viewpoints.

[0218] In some embodiments of this application, based on the foregoing scheme, the first video data encapsulation unit included in each video frame is an Access Unit Delimiter (AUD) video data encapsulation unit, which is used to indicate the type of each video frame.

[0219] In some embodiments of this application, based on the foregoing scheme, each video data encapsulation unit includes a start code, which is used to indicate the starting position of the video data encapsulation unit.

[0220] In some embodiments of this application, based on the foregoing scheme, the generation unit 904 is further configured to: add basic stream information of the multi-view video stream to the program mapping table, wherein the basic stream information includes video stream type and total descriptor length.

[0221] Figure 10 A block diagram of a video stream processing apparatus according to an embodiment of the present application is shown. The video stream processing apparatus can be applied to a device with computing processing capabilities, such as a terminal device or a server.

[0222] Reference Figure 10 As shown, a video stream processing apparatus 1000 according to an embodiment of this application includes: an acquisition unit 1002, a decapsulation unit 1004, and a processing unit 1006.

[0223] The acquisition unit 1002 is configured to acquire transport stream data packets, which are obtained by encapsulating a program map table, a video encoding parameter set for the video stream, and video frame data. A registration descriptor is added to the program map table, and the value of the format identifier in the registration descriptor indicates that the video stream is a multi-view video stream. The decapsulation unit 1004 is configured to decapsulate the transport stream data packets to obtain a video data encapsulation unit, the program map table, and the video encoding parameter set. The processing unit 1006 is configured to reassemble the video data encapsulation unit according to the program map table and the video encoding parameter set to obtain a multi-view video stream.

[0224] Figure 11 A schematic diagram of the structure of a computer system suitable for implementing the computer device of the present application is shown.

[0225] It should be noted that, Figure 11The computer system 1100 of the computer device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0226] like Figure 11 As shown, the computer system 1100 may include a Central Processing Unit (CPU) 1101, which can perform various appropriate actions and processes based on a program stored in Read-Only Memory (ROM) 1102 or a program loaded from storage portion 1108 into Random Access Memory (RAM) 1103, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1103. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. An input / output (I / O) interface 1105 is also connected to bus 1104.

[0227] The following components can be connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1110 as needed so that computer programs read from them can be installed into storage section 1108 as needed.

[0228] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by central processing unit (CPU) 1101, it performs various functions defined in the system of this application.

[0229] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a computer program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0230] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0231] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0232] In another aspect, this application also provides a computer-readable medium, which may be included in the computer device described in the above embodiments; or it may exist independently and not assembled into the computer device. The computer-readable medium carries one or more computer programs, which, when executed by the computer device, cause the computer device to perform the methods described in the above embodiments.

[0233] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0234] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, and includes several instructions to cause a computer device to execute the method according to the embodiments of this application.

[0235] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0236] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for processing a video stream, characterized in that, include: The video stream to be encapsulated is parsed to obtain the video encoding parameter set and video frame data of the video stream; If it is determined from the video encoding parameter set that the video stream contains a multi-view video stream, then a program mapping table corresponding to the multi-view video stream is generated. A registration descriptor is added to the program mapping table, and the value of the format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream. The program mapping table, the video encoding parameter set, and the video frame data are encapsulated to generate transport stream data packets.

2. The video stream processing method according to claim 1, characterized in that, The video stream processing method further includes: If it is determined from the video coding parameter set that the video stream contains a multi-view video stream, then three-dimensional reference display information is generated from the video coding parameter set, and the three-dimensional reference display information contains identification information of multiple viewpoints in the multi-view video stream; Add the three-dimensional reference display information to the program map table.

3. The video stream processing method according to claim 2, characterized in that, The program mapping table contains basic stream information, the registration descriptor is the first descriptor in the basic stream information, and the three-dimensional reference display information is added to the second descriptor in the basic stream information.

4. The video stream processing method according to claim 3, characterized in that, The label value of the second descriptor is a set value, and the length value of the second descriptor is the length of the three-dimensional reference display information.

5. The video stream processing method according to claim 1, characterized in that, The program mapping table, the video encoding parameter set, and the video frame data are encapsulated to generate transport stream data packets, including: The video encoding parameter set and the video frame data are encapsulated as a whole, and the program mapping table is encapsulated separately to generate the encapsulated transport stream data packet.

6. The video stream processing method according to claim 5, characterized in that, The video encoding parameter set and the video frame data are encapsulated as a whole, including: The video encoding parameter set and the keyframe data in the video stream are encapsulated as a whole.

7. The video stream processing method according to claim 1, characterized in that, The program mapping table, the video encoding parameter set, and the video frame data are encapsulated to generate transport stream data packets, including: The video frame data is written into the transport stream data packet in the order of the video frames contained in the video stream. Each video frame contains multiple video data encapsulation units corresponding to different viewpoints.

8. The video stream processing method according to claim 7, characterized in that, The first video data encapsulation unit contained in each video frame is an Access Entity Delimiter (AUD) video data encapsulation unit, which is used to indicate the type of each video frame.

9. The video stream processing method according to claim 7, characterized in that, Each of the video data encapsulation units contains a start code, which is used to indicate the starting position of the video data encapsulation unit.

10. The method for processing a video stream according to any one of claims 1 to 9, characterized in that, The video stream processing method further includes: Add the basic stream information of the multi-view video stream to the program mapping table. The basic stream information includes the video stream type and the total length of the descriptor.

11. A method for processing a video stream, characterized in that, include: Acquire transport stream data packets, which are obtained by encapsulating a program map table, a video encoding parameter set of a video stream, and video frame data. A registration descriptor is added to the program map table, and the value of the format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream. The transport stream data packets are decapsulated to obtain a video data encapsulation unit, the program mapping table, and the video encoding parameter set; The video data encapsulation unit is reassembled according to the program mapping table and the video encoding parameter set to obtain a multi-view video stream.

12. A video stream processing apparatus, characterized in that, include: The parsing unit is configured to parse the video stream to be encapsulated to obtain the video encoding parameter set and video frame data of the video stream. The generation unit is configured to generate a program mapping table corresponding to the multi-view video stream if it is determined from the video encoding parameter set that the video stream contains a multi-view video stream. The program mapping table contains a registration descriptor, and the value of the format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream. The processing unit is configured to encapsulate the program mapping table, the video encoding parameter set, and the video frame data to generate transport stream data packets.

13. A video stream processing apparatus, characterized in that, include: The acquisition unit is configured to acquire transport stream data packets, which are obtained by encapsulating a program map table, a video encoding parameter set of a video stream, and video frame data. A registration descriptor is added to the program map table, and the value of the format identifier in the registration descriptor is used to indicate that the video stream is a multi-view video stream. The decapsulation unit is configured to decapsulate the transport stream data packets to obtain a video data encapsulation unit, the program mapping table, and the video encoding parameter set. The processing unit is configured to reassemble the video data encapsulation unit according to the program mapping table and the video encoding parameter set to obtain a multi-view video stream.

14. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the video stream processing method according to any one of claims 1 to 10, or the video stream processing method according to claim 11.

15. A computer device, characterized in that, include: One or more processors; A memory for storing one or more computer programs that, when executed by one or more processors, cause the computer device to implement the video stream processing method of any one of claims 1 to 10, or the video stream processing method of claim 11.

16. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, wherein a processor of a computer device reads from the computer-readable storage medium and executes the computer program, causing the computer device to perform the video stream processing method of any one of claims 1 to 10, or to perform the video stream processing method of claim 11.

Images