System and method for transcoding video streams

Abstract

This transcoding system (ST) comprises: - a video stream segmentation module (USEG) configured to segment at least one video stream (FVi) into a plurality of segments (SEGij); - an input storage module (USTKIN) configured to store said segments (SEGij), each associated with metadata (Mij); - a transcoding resource allocation module (UALLOC) configured to dynamically allocate computing resources (VM, CONT) to transcode each of said segments (SEGij) into a transcoded segment (SEGTij), said transcoded segment (SEGTij) being associated with the metadata (Mij) associated with the segment (SEGij) from which it was transcoded, at least a part of said segments being transcoded in parallel; - a video stream reconstruction module (UPACK) configured to constitute at least one transcoded video stream (FVT) from said transcoded segments (SEGTij) and their metadata (Mij). Figure for the abridged version: Fig. 1

Description

Title of the invention: System and method for transcoding video streams. Technical field

[0001] The invention relates to the general field of video stream transcoding. More specifically, the invention relates to the context in which the stream is transcoded as it is received by the transcoding system, in other words, in real time or in "live" mode. Background of the invention

[0002] Online video streaming via streaming platforms requires more and more resources.

[0003] Nowadays, users demand high-definition videos, which requires efficient compression and resources to maintain this quality without visible degradation. Codecs (such as H.264, H.265, or AVI) allow for reduced file sizes while maintaining acceptable quality, but they require significant CPU and GPU resources for video encoding and decoding.

[0004] Furthermore, the platforms use adaptive streaming techniques (such as HLS or MPEG-DASH) that automatically adjust the video quality according to the bandwidth available to the user. This technology requires sophisticated algorithms, the storage of multiple versions of a video, and servers capable of switching between versions.

[0005] The platforms must also adapt to a wide variety of devices (smartphones, tablets, smart TVs, computers) and formats. This requires costly transcoding operations to ensure compatibility with these different terminals, screen sizes, and resolutions, resulting in the management of multiple video streams and formats.

[0006] Platforms must minimize latency to provide a smooth user experience, particularly in the context of live broadcasts.

[0007] The Applicant has found that current transcoding systems are not satisfactory. Object and summary of the invention

[0008] Thus, the present disclosure relates to a transcoding system comprising:

[0009] - a video stream segmentation module configured to segment at least one video stream in a plurality of segments;

[0010] - an input storage module configured to store said segments, each associated with metadata;

[0011] - a transcoding resource allocation module configured to allocate dynamically compute resources to transcode each of said segments into a transcoded segment, a said transcoded segment being associated with the metadata associated with the segment from which it was transcoded, at least a part of said segments being transcoded in parallel;

[0012] - a video stream reconstruction module configured to constitute at least one transcoded video stream from said transcoded segments and their metadata.

[0013] Accordingly, this disclosure relates to a transcoding process comprising the following steps:

[0014] - segmentation of at least one video stream into a plurality of segments;

[0015] - for storing said segments, each associated with metadata;

[0016] - dynamic allocation of computing resources to transcode each of said segments into a transcoded segment, at least a part of said segments being transcoded in parallel;

[0017] - association of each of said transcoded segments with the metadata associated with the segment from which it was transcoded;

[0018] -of the constitution of at least one transcoded video stream from said transcoded segments and their metadata.

[0019] Thus, and in general, the present disclosure proposes a solution to parallelize the transcoding of audio / video streams.

[0020] The mechanism put in place makes it possible to segment the incoming high-quality streams and to process all or part of these segments on different transcoding instances.

[0021] The transcoding system and method proposed by this disclosure can be applied to any type of stream, for example, for virtual reality applications. Video streams can, for example, be: - live 2D image streams (in English "live"); - VR180 or VR360 live streams; - monoscopic and stereoscopic flows; - streams in "top / bottom" or "side-by-side" format. In one embodiment, segment transcoding is performed by containers running in virtual machines (examples of resources as defined in the disclosure). For example, a container can be dynamically allocated for each segment to be transcoded and removed once the transcoding is complete. All these transcoders can run in parallel. A transcoder (or transcoding instance) has its own computing power.

[0022] Transcoding performed within a container may or may not rely on the resources of a graphics card.

[0023] Depending on the desired quality levels, this parallelization makes it possible to transcode very high-quality content. It also makes it possible to improve and / or modulate the latency induced by transcoding, that is to say, the delay between the input stream and the output stream of the transcoding system.

[0024] For example, in one embodiment, the computing resources include at least one virtual machine, the number of which is dynamically determined according to one or more of the following criteria:

[0025] - resources of said at least one virtual machine;

[0026] - a type of said transcoding; and

[0027] - a service quality criterion.

[0028] In one embodiment, the computing resources include at least one container allocated dynamically according to the number of segments to be transcoded.

[0029] In one embodiment, each segment comprises at least one, preferably only one, image group starting with an I-frame type frame.

[0030] In one embodiment, the metadata associated with a segment includes:

[0031] - an identifier of the stream from which the segment is extracted; - a rank of the segment in said flow; and - a timestamp representing the moment of reception of said segment by the input storage module.

[0032] In one embodiment, said timestamps are all determined using the same clock.

[0033] This advantageously allows for the synchronization of segments extracted from different streams. The transcoded segments, associated with the same metadata as the segments from which they were extracted, are thus automatically synchronized.

[0034] Thus, in one embodiment, at least one transcoded video stream may include segments transcoded from segments extracted from different streams, according to an arrangement defined by a scenario.

[0035] These scenarios allow, for example, combining streams from several cameras that acquire the same scene from different angles.

[0036] The reconstitution (or packaging) module may optionally determine the protocol for broadcasting the reconstituted stream (for example DASH, HLS over HTTP, broadcast over UDP, QUIC) and reconstitute the stream according to this protocol from the transcoded segments.

[0037] In a particular embodiment, the different stages of the transcoding process are determined by computer program instructions.

[0038] Consequently, the invention also relates to a computer program on an information medium, this program being capable of being implemented in a system of transcoding or more generally in a computer, this program includes instructions adapted to the implementation of the steps of a transcoding process as mentioned above.

[0039] This program may use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0040] This disclosure also relates to a computer-readable information or recording medium containing instructions for a computer program as mentioned above.

[0041] The information or recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a hard disk drive.

[0042] On the other hand, the information or recording medium can be a transmissible medium such as an electrical or optical signal, which can be transmitted via an electrical or optical cable, by radio, or by other means. The program according to the invention can, in particular, be downloaded onto an Internet-type network.

[0043] Alternatively, the information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the drawings

[0044] Other features and advantages of the present invention will become apparent from the description below, with reference to the accompanying drawings which illustrate an example of an embodiment without being limiting in any way. In the figures:

[0045] - Fig. 1 represents, schematically, a transcoding system conforming to a particular mode of this disclosure;

[0046] - Figure 2 represents, in flowchart form, the main steps of a transcoding process conforming to a particular mode of this disclosure;

[0047] - Figure 3 schematically represents the physical architecture of a computer capable of implementing a transcoding process conforming to a particular mode of this disclosure. Detailed description of the invention

[0048] Fig. 1 represents an ST transcoding system conforming to a particular embodiment of the present disclosure.

[0049] The ST transcoding system includes an FV video stream reception URX module and a video stream segmentation USEG module configured to segment at least one FV video stream into a plurality of SEG segments.

[0050] In a particular embodiment, the video stream comprises Groups of Pictures (GOPs), each GOP consisting of a sequence of images composed of different types of frames, typically known to those skilled in the art as I-frames, P-frames, and B-frames. In such an encoding scheme, each L-frame contains all the information necessary to be decoded on its own and serves as a reference point for the other frames in its GOP.

[0051] In the example embodiment described here, the ST transcoding system includes a USTK1N input storage module configured to store SEG segments after associating them with M metadata.

[0052] Figure 1 illustrates a situation in which the USTK^ input storage module comprises: - SEGn, ..., SEGy segments of a first FV1 video stream associated with MH metadata; and - SEG2i, ..., SEG2j segments of a second FV2 video stream associated with M2i metadata

[0053] Not all streams necessarily have the same number of segments and not all segments necessarily have the same number of image groups.

[0054] In the embodiment described here, the My metadata associated with a SEGy segment includes: - an identifier i of the stream from which this segment is extracted; - a rank j of the segment in the flow i; and - a timestamp representing the moment of reception of this segment by the USTK1N input storage module.

[0055] In the embodiment described here, the USTK1N input storage module includes a CLK clock and the metadata M associated with the different segments are all timestamped using the same CLK clock.

[0056] In the embodiment described here, the ST transcoding system includes a UALLOC transcoding resource allocation module.

[0057] In one embodiment, the UALLOC module dynamically determines a number of NVM virtual machines based on one or more criteria among: - the resources of these virtual machines (number of cores, memory, ...); - of a transcoding TCod type, defined by: - the input encoding, i.e., the one that receives the incoming stream, which impacts the complexity of the decoding (for example H.264), and the output encoding, which impacts the complexity of the encoding (for example HEVC). - the input resolution (e.g., 720p) and output resolution (e.g., 1080p), which determine whether resizing operations are necessary The input bit rate (e.g., 3 Mbps) and output bit rate (e.g., 2.5 Mbps) - optionally the desired container type, for example fMP4, - the cost of using virtual machines, - energy constraints such as the consumption of virtual machines required for transcoding, as an alternative to GPU transcoding; - and a CQOs quality of service criterion, for example to adjust the latency, the transcoding power (defined by the product of the number of individual transcoders and the power of these transcoders) must be increased to decrease the latency between the input and output of the transcoding system.

[0058] In one embodiment, the UALLOC module dynamically allocates the CONT containers inside the VM virtual machines according to the load, i.e. according to the number of segments to be transcoded.

[0059] For example, as soon as a SEGy segment is recorded in the USTK^ input storage module, the UALLOC allocation module dynamically allocates a container in a virtual machine in order to transcode that segment.

[0060] In [Fig. 1], the allocation of a CNT3 container of a virtual machine VM1 for transcoding the SEG2i segment has been illustrated as an example, the transcoded segment SEGT2i being associated with the metadata of the SEG2i segment.

[0061] In a very advantageous way, the different segments are transcoded independently and in parallel.

[0062] In the embodiment described here, the transcoded SEGTÿ segments are stored, in association with their metadata My, in an output storage module USTKout.

[0063] The transcoded SEGTÿ segments are not necessarily received by the output storage module USTKOuT in the same order as the original SEGy segments by the input storage module USTK1N.

[0064] In the embodiment described here, the ST transcoding system includes a UPACK module for reconstructing video streams from the transcoded SEGTÿ segments and their metadata My.

[0065] In the embodiment described here, the UPACK video stream reconstruction module uses an SCN scenario to determine how to construct an FVT transcoded video stream from the transcoded segments.

[0066] In the example of [Fig. 1], it is assumed that the video streams FV1 and FV2 are video streams representing the same scene acquired simultaneously by two cameras and that the transcoded video stream FVT comprises: - a first SEGTn segment corresponding to the transcoding of the first SEGn segment of the first FV1 stream; - a second SEGT22 segment corresponding to the transcoding of the second SEG22 segment of the second FV2 stream; and - a third segment SEGTn corresponding to the transcoding of the third segment SEGn of the first stream FV1.

[0067] The scheduling of transcoded segments resulting from the encoding of segments from different video streams is facilitated by the fact that the metadata of the different segments include a timestamp made with the same clock.

[0068] In the particular embodiment described here, each SEG segment is made up of a group of GOP images.

[0069] Alternatively, each SEG segment comprises several GOP image groups.

[0070] For example, assuming: - a first flow whose GPOs have a duration of 2 seconds; and - a second one whose GPOs have a duration of 6 seconds; and It may be useful to create segments such as: - the segments obtained from the GOPs of the first stream comprise three GOPs; and - The segments obtained from the GOPs of the second stream contain only one GOP; so that all segments have the same duration. This makes it possible to create a transcoded stream that mixes segments from both streams without loss of coherence.

[0071] Of course, an FVT transcoded video stream can contain SEGTÿ transcoded segments resulting from the encoding of SEGÿ segments of the same stream.

[0072] Figure 2 represents, in flowchart form, the main steps of a PT transcoding process conforming to a particular mode of this disclosure.

[0073] In the embodiment described here, the PT process includes a step E10 of receiving at least one video stream FVi.

[0074] In the embodiment described herein, the PT method includes a step E20 of segmenting the received video stream(s) FV into a plurality of SEG segments. In a particular embodiment described herein, each SEG segment consists of a group of GOP images, alternatively each SEGÿ segment is made up of one or more groups of GOP images.

[0075] In the embodiment described here, the PT process includes a step E30 of segment storage, each segment SEGÿ being associated (step E40) with metadata My.

[0076] In one embodiment described herein, the metadata Mÿ associated with a SEGÿ segment comprises: - an identifier i of the stream from which this segment is extracted; - a rank j of the segment in the flow i; and - a timestamp representative of the moment of receipt of this segment.

[0077] In the embodiment described here, the PT process includes a step E50 of dynamic allocation of computing resources to transcode (step E60) each of said segments into a transcoded segment, at least part of the segments being transcoded in parallel.

[0078] These computing resources include, for example, containers that run in virtual machines.

[0079] In the embodiment described here, the PT process includes a step E70 of associating each of the transcoded SEGTÿ segments with the metadata Mij associated with the SEGÿ segment from which it was transcoded.

[0080] In the embodiment described here, the PT process includes a step E80 of at least one FVT transcoded video stream from SEGTÿ transcoded segments and their Mÿ metadata. The ordering of the transcoded segments in the FVT transcoded video stream is, for example, defined by an SCN scenario.

[0081] Figure 3 schematically represents the hardware architecture of an ORD computer capable of implementing a transcoding process in accordance with a particular mode of this disclosure.

[0082] In this respect, the computer ORD may include a processor PROC, random access memory, read-only memory MEM, non-volatile memory, and communication means COM. The memory MEM constitutes an information storage medium according to the invention, readable by the computer and by the processor PROC, on which a computer program PROG according to the invention is stored. The computer program PROG includes instructions for performing steps of a transcoding process according to this disclosure, when the program PROG is executed by the processor PROC.

[0083] The PROG computer program defines functional and software modules of the ST transcoding system described previously with reference to [Fig. 1], and in particular:

[0084] - a video stream segmentation module into a plurality of segments;

[0085] - an input storage module for these segments, each associated with metadata;

[0086] - a dynamic allocation module for computing resources for transcoding of these segments into transcoded segments, a transcoded segment being associated with the metadata associated with the segment from which it was transcoded, at least a part of said segments being transcoded in parallel;

[0087] - a video stream reconstruction module configured to constitute at least one transcoded video stream from transcoded segments and their metadata. Example of implementation

[0088] In an example of a real-time streaming implementation, the ST transcoding system in accordance with disclosure can be used to transcode video streams broadcast according to the RTMP (Real-Time Messaging Protocol).

[0089] For example, one can consider source streams acquired by Insta360 Titan cameras from Insta360, or QooCam 8K Enterprise from KanDao Technology.

[0090] For further information on these cameras, those skilled in the art may refer to the following websites:

[0091] https: / / www.insta360.com / fr / product / insta360-titan / and

[0092] The source streams can be configured, in particular: i / the encoded (H264 or HEVX), ii / bitrate, iii / exposure; and iv / the number of images per second (in English frames per second).

[0093] In this implementation example, the video streams are segmented into GOP image groups comprising 60 frames, each group comprising one I-frame and 59 P-frames, for a duration of 2 seconds.

[0094] Each segmented packet is associated with metadata comprising a flow identifier and a rank of the segment in the flow.

[0095] In this particular embodiment, the allocation module allocates:

[0096] - 2 containers in which the stream segmentation modules are executed (1 (by flow); - 1 container in which the USTK1N input storage module is executed; - 1 container in which the USTK0Ut output storage module is executed; - 1 transcoding type container for each of the segments to be transcoded; - 1 container in which an instance of the UPACK reconstruction module is executed to reconstruct the first transcoded stream; and - 1 container in which an instance of the UPACK reconstruction module is executed to reconstruct the second transcoded stream.

[0097] The containers are run in virtual machines, created as needed when the physical resources required to run the containers are insufficient.

[0098] In the embodiment described here, each virtual machine has 64 virtual processors (processing units) and 64 gigabytes of RAM.

[0099] In a particular embodiment, the transcoding system can store a plurality of video segments and the reconstructed stream can be reconstructed much later, in the context of delayed broadcasting, replay, in a summarized version, highlighting only, ...

[0100] In one embodiment, the first frame of each stream includes additional metadata pts=O. This metadata makes it possible to detect a dropout in the live stream, for example due to frequent reconnections of a camera and to avoid integrating streams of too short durations into the reconstructed stream.

Claims

Demands

1. A transcoding system (ST) comprising: - a video stream segmentation module (USEG) configured to segment at least one video stream (FV;) into a plurality of segments (SEGy); - an input storage module (USTK^) configured to store said segments (SEGy), each associated with metadata (My); - a transcoding resource allocation module (UALLOC) configured to dynamically allocate computing resources (VM, CONT) to transcode each of said segments (SEGy) into a transcoded segment (SEGTy), said transcoded segment (SEGTy) being associated with the metadata (Mij) associated with the segment (SEGy) from which it was transcoded, at least a portion of said segments being transcoded in parallel; - a video stream reconstruction module (UPACK) configured to constitute at least one transcoded video stream (FVT) from said transcoded segments (SEGTy) and their metadata (My).

2. Transcoding system (TS) according to claim 1 wherein each segment comprises at least one, preferably only one, image group beginning with an I-frame type frame.

3. Transcoding system (TS) according to claim 1 or 2 wherein the metadata (My) associated with said segment (SEGy) comprise: - an identifier (j) of the stream from which said segment is extracted; - a rank (i) of said segment in said stream; and - a timestamp representative of the time of receipt of said segment by said input storage unit (USTK^).

4. Transcoding system (TS) according to any one of claims 1 to 3 wherein said computing resources (VM, CONT) comprise at least one virtual machine (VM) the number of which (NVm) is dynamically determined according to one or more of the following criteria: - resources of said at least one virtual machine; - a type (TCod) of said transcoding; and - a quality of service criterion (CQOs).

5. Transcoding system (TS) according to any one of claims 1 to 4 in which said computing resources (VM, CONT) comprise at least one container allocated dynamically according to the number of segments (SEGy) to be transcoded.

6. Transcoding system (ST) according to any one of claims 3 to 5 wherein said timestamps are all determined using the same clock (CLK).

7. Transcoding system (TS) according to claim 6 in which said at least one transcoded video stream (TVS) comprises transcoded segments (SEGTÿ) from segments (SEGÿ) extracted from different streams (TVS), according to an arrangement defined by a scenario (SCN).

8. Transcoding method (PT) comprising steps of: - segmenting (E20) at least one video stream (FVi) into a plurality of segments (SEGÿ); - storing (E30) said segments (SEGÿ), each associated (E40) with metadata (My); - dynamically allocating (E50) computing resources (VM, CONT) to transcode each of said segments (SEGÿ) into a transcoded segment (SEGTÿ), at least a part of said segments being transcoded in parallel; - associating (E70) each of said transcoded segments (SEGTÿ) with the metadata (Mij) associated with the segment (SEGÿ) from which it was transcoded; - constructing (E80) at least one transcoded video stream (FVT) from said transcoded segments (SEGTÿ) and their metadata (My).

9. Computer program (PROG) comprising instructions for carrying out the steps of the transcoding process according to claim 8 when said program is executed by a computer (ORD).

10. Computer-readable recording medium (MEM) on which is recorded a computer program (PROG) comprising instructions for carrying out the steps of the transcoding process according to claim 8.

Images