Method, apparatus and storage medium for processing media stream based on network-based media processing function

Abstract

A method and apparatus for processing a media stream by a network based media processing (NBMP) function, the media stream including segments with a common header, performed by at least one processor. The method includes obtaining an input stream including one or more segments; identifying a start code in the segments, wherein the start code occurs at the beginning of a segment and the start code is a unique code indicating the start of a segment; identifying a sequence number in the segments, wherein the sequence number indicates a position of each segment in a plurality of segments; splitting the input stream into sub-streams based on the start code and the sequence number and signaling a location of a respective common header in each sub-stream; merging the sub-streams based on the start code and the sequence number and signaling a location of the common header; and generating an output stream.

Description

[0001] Cross Reference to Related Applications

[0002] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 219,234, filed July 7, 2021, and U.S. Patent Application No. 17 / 702,096, filed March 23, 2022, the disclosures of which are incorporated by reference in their entireties. TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate to extensions to the segmenter and merger function reference templates for network-based media processing (NBMP). The extensions provide additional features for segment-based segmenter and merger functions, such as support for common headers, and use of metadata for start codes and sequence numbers. BACKGROUND

[0004] Networks and cloud platforms can be used to run various applications. The NBMP standard defines a set of tools for independent processing of media segments with equal duration. The NBMP AMD1 defines segmenter and merger function templates. Segment-based segmenter and merger functions are used to divide media data into parallel sub-streams, where each stream can be processed independently, and then the processed streams are merged together to create the processing result of the original stream.

[0005] The NBMP segmenter and merger functions can perform parallel processing of multiple media streams with clean boundaries. However, these NBMP functions cannot address common headers in multiple media streams or sub-streams thereof. Therefore, the NBMP segmenter and merger functions cannot process media streams with headers at their beginning. Further, the NBMP standard design only supports timing metadata for finding segment boundaries and does not work with streams having start codes.

[0006] Aspects of the present disclosure address these issues and describe segment-based segmentation and merging functions for media parallel processing that support metadata for common headers and sequence numbers. SUMMARY

[0007] One or more example embodiments of the present disclosure provide a method and apparatus for segment-based segmentation and merging functions for media parallel processing on a cloud platform.

[0008] According to an embodiment, a method of processing a media stream by a network based media processing (NBMP) function, the media stream comprising segments with a common header, is provided, the method being performed by at least one processor. The method can comprise: obtaining an input stream comprising a plurality of segments, wherein the input stream comprises a first common header located in the plurality of segments; identifying a start code in the plurality of segments, wherein the start code is located at the beginning of a segment and the start code is a unique code indicating the start of the segment; identifying a sequence number of each segment in the plurality of segments, wherein the sequence number indicates a position of each segment in the plurality of segments; segmenting the input stream into a plurality of sub-streams using a NBMP segmenter function and signaling a location of a respective common header of each sub-stream based on the start code and the sequence number; merging the plurality of sub-streams using a NBMP merger function and signaling a location of a second common header based on the start code and the sequence number; and generating an output stream based on the merged plurality of sub-streams, wherein the output stream comprises the second common header.

[0009] According to an embodiment, an apparatus for processing a media stream by a network based media processing (NBMP) function, the media stream comprising segments with a common header, is provided. The apparatus can comprise at least one memory storing computer program code (instructions) and at least one processor configured to read the program code and operate as instructed by the program code. The program code can comprise: obtaining code configured to cause the at least one processor to obtain an input stream comprising a plurality of segments, wherein the input stream comprises a first common header located in the plurality of segments; first identifying code configured to cause the at least one processor to identify a start code in the plurality of segments, wherein the start code is located at the beginning of a segment and the start code is a unique code indicating the start of the segment; second identifying code configured to cause the at least one processor to identify a sequence number of each segment in the plurality of segments, wherein the sequence number indicates a position of each segment in the plurality of segments; segmenting code configured to cause the at least one processor to segment the input stream into a plurality of sub-streams using a NBMP segmenter function and signaling a location of a respective common header of each sub-stream based on the start code and the sequence number; merging code configured to cause the at least one processor to merge the plurality of sub-streams using a NBMP merger function and signaling a location of a second common header based on the start code and the sequence number; and generating code configured to cause the at least one processor to generate an output stream based on the merged plurality of sub-streams, wherein the output stream comprises the second common header.

[0010] According to one or more embodiments, a non-transitory computer-readable medium storing computer code can be provided. The computer code can be configured to, when executed by at least one processor, cause the at least one processor to: obtain an input stream comprising a plurality of segments, wherein the input stream comprises a first common header located in the plurality of segments; identify a start code in the segments, wherein the start code is located at a beginning of one segment and the start code is a unique code indicating a start of the segment; identify a sequence number for each of the plurality of segments, wherein the sequence number indicates a position of each segment in the plurality of segments; based on the start code and the sequence number, segment the input stream into a plurality of sub-streams using an NBMP splitter function and signal a location of a respective common header for each sub-stream; based on the start code and the sequence number, merge the plurality of sub-streams using an NBMP merger function and signal a location of a second common header; and generate an output stream based on the merged plurality of sub-streams, wherein the output stream comprises the second common header.

[0011] Additional aspects will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description, or can be learned by practice of the presented embodiments of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other aspects, features, and advantages of embodiments of the disclosure will become more apparent from the following description in conjunction with the following accompanying drawings, in which:

[0013] Figure 1 is a schematic illustration of a communication system in accordance with one or more embodiments.

[0014] Figure 2 is a simplified example illustration of a streaming media environment in accordance with one or more embodiments.

[0015] Figure 3 is a block diagram of an NBMP system in accordance with one or more embodiments.

[0016] Figure 4 is an example diagram of an NBMP splitter and merger process in accordance with one or more embodiments.

[0017] Figure 5 is a flow diagram of an example process for processing an input media stream containing a common header using temporal metadata.

[0018] Figure 6 is a block diagram of an example process for processing a media stream comprising segments with a common header in accordance with one or more embodiments.

[0019] Figure 7 is a block diagram of an example of computer code for processing a media stream comprising segments with a common header in accordance with one or more embodiments. DETAILED DESCRIPTION

[0020] The present disclosure relates to a method and apparatus for parallel processing of splitting and merging functions of segments in a media stream.

[0021] Embodiments of the present disclosure are described in detail with reference to the attached drawings. However, the examples of the embodiments can be implemented in various forms, and the present disclosure should not be construed as being limited to the examples described herein. Rather, the examples of the embodiments are provided so that the technical solutions of the present disclosure are more fully comprehensive and complete, and the ideas of the examples of the embodiments are fully conveyed to those skilled in the art. The drawings are only exemplary illustrations of the present disclosure, and are not necessarily drawn to scale. The same reference numbers in the drawings represent the same or similar components, and thus repeated descriptions of these components are omitted.

[0022] The proposed features discussed below can be used individually or in any combination in any order. Some of the block diagrams shown in the drawings are functional entities, and do not necessarily correspond to physically or logically independent entities. Further, the embodiments can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits), or in the form of software, or in different network and / or processor devices and / or microcontroller devices. In one example, one or more processors execute computer program code stored in one or more non-volatile computer readable media.

[0023] Figure 1 is a schematic diagram of an environment 100 in which the methods, apparatuses, and systems described herein can be implemented, in accordance with an embodiment. As shown in Figure 1 The environment 100 can include a user device 110, a platform 120, and a network 130, as shown. The devices of the environment 100 can be interconnected by wired connections, wireless connections, or a combination of wired and wireless connections.

[0024] The user device 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information related to the platform 120. For example, the user device 110 can include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch), or a similar device. In some implementations, the user device 110 can receive information from and / or send information to the platform 120.

[0025] The platform 120 includes one or more devices as described elsewhere herein. In some embodiments, the platform 120 may include a cloud server or a group of cloud servers. In some embodiments, the platform 120 may be designed to be modular so that software components can be swapped in or out based on specific needs. In this way, the platform 120 can be easily and / or quickly reconfigured for different uses.

[0026] In some embodiments, as shown, the platform 120 can be hosted in a cloud computing environment 122. It is worth noting that although the embodiments described herein describe the platform 120 as being hosted in a cloud computing environment 122, in some embodiments, the platform 120 is not cloud-based (i.e., can be implemented outside of a cloud computing environment) or can be partially cloud-based.

[0027] Cloud computing environment 122 includes an environment that hosts platform 120. Cloud computing environment 122 can provide computing, software, data access, storage, and other services without requiring end users (e.g., user devices 110) to be aware of the physical location and configuration of the systems and / or devices hosting platform 120. As shown, cloud computing environment 122 can include a set of computing resources 124 (collectively, "computing resources 124" and individually, "computing resource 124").

[0028] Computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some embodiments, computing resources 124 may host platform 120. Cloud resources may include computing instances executed on computing resources 124, storage devices provided on computing resources 124, data transmission devices provided by computing resources 124, and the like. In some embodiments, computing resources 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0029] Further Figure 1 As shown, the computing resources 124 include a set of cloud resources, such as one or more application programs ("APP") 124-1, one or more virtual machines ("VM") 124-2, virtualized storage ("VS") 124-3, one or more hypervisors ("HYP") 124-4, etc.

[0030] The applications 124-1 include one or more software applications that can be provided to and / or accessed by the user devices 110 and / or the platform 120. The applications 124-1 need not install and execute software applications on the user devices 110. For example, the applications 124-1 can include software related to the platform 120 and / or any other software that can be provided through the cloud computing environment 122. In some embodiments, one application 124-1 can send / receive information to or from one or more other applications 124-1 through a virtual machine 124-2.

[0031] The virtual machines 124-2 include a software implementation of a machine (e.g., a computer) that executes programs, similar to a physical machine. The virtual machines 124-2 can be system virtual machines or process virtual machines, depending on the degree with which the virtual machines 124-2 use and correspond to any real machines. A system virtual machine can provide a complete system platform that supports execution of a complete operating system ("OS"). A process virtual machine can execute a single program and can support a single process. In some embodiments, the virtual machines 124-2 can execute on behalf of users (e.g., the user devices 110) and can manage the infrastructure of the cloud computing environment 122, such as data management, synchronization, or long-duration data transfers.

[0032] The virtualized storage 124-3 includes one or more storage systems and / or one or more devices that use virtualization techniques in the storage systems or devices of the computing resources 124. In some embodiments, the types of virtualization, in the context of a storage system, can include block virtualization and file virtualization. Block virtualization can refer to abstraction (or separation) of logical storage from physical storage so that the storage system can be accessed without considering the physical storage or heterogeneous structure. The separation can allow an administrator of the storage system to flexibly manage storage for end users. File virtualization can eliminate dependencies between data accessed at a file level and locations of physical storage files. This can optimize storage usage, server consolidation, and / or performance of non-disruptive file migrations.

[0033] The hypervisor 124-4 can provide a hardware virtualization technique that allows multiple operating systems (e.g., "guest operating systems") to execute concurrently on a host computer, such as the computing resources 124. The hypervisor 124-4 can provide a virtual operating platform to the guest operating systems and can manage execution of the guest operating systems. Multiple instances of various operating systems can share virtualized hardware resources.

[0034] The network 130 includes one or more wired and / or wireless networks. For example, the network 130 may include a cellular network (e.g., a fifth generation (5G) network, a Long-Term Evolution (LTE) network, a third generation (3G) network, a Code Division Multiple Access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-optic-based network, etc., and / or a combination of these or other types of networks.

[0035] Figure 1 The number and arrangement of devices and networks shown are provided as examples. Figure 1 There may be more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks than those shown. Figure 1 Two or more of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one set of devices (eg, one or more devices) of environment 100 may perform one or more functions described as being performed by another set of devices of environment 100.

[0036] Figure 2 yes Figure 1 1. The device 200 may correspond to the user device 110 and / or the platform 120. Figure 2 As shown, device 200 may include a bus 210 , a processor 220 , a memory 230 , a storage component 240 , an input component 250 , an output component 260 , and a communication interface 270 .

[0037] Bus 210 includes a component that permits communication among the components of device 200. Processor 220 is implemented in hardware, firmware, or a combination of hardware and software. Processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some embodiments, processor 220 includes one or more processors capable of being programmed to perform a function. Memory 230 includes a random access memory (RAM), a read only memory (ROM), and / or another type of dynamic or static storage device (e.g., a flash memory, a magnetic storage device, and / or an optical storage device) that stores information and / or instructions for use by processor 220.

[0038] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 can include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, and / or a solid state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium, along with a corresponding drive.

[0039] Input component 250 includes a component that permits device 200 to receive information, such as via user input, for example, a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and / or a microphone. Additionally, or alternatively, input component 250 can include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output component 260 includes a component that provides output information from device 200, such as a display, a speaker, and / or one or more light-emitting diodes (LEDs).

[0040] Communication interface 270 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter) that enables device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 270 can permit device 200 to receive information from another device and / or provide information to another device. For example, communication interface 270 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, and / or the like.

[0041] Device 200 can perform one or more processes described herein. Device 200 can perform these processes in response to processor 220 executing software instructions stored by non-volatile computer-readable media (e.g., memory 230 and / or storage component 240). Computer-readable media is defined herein as non-volatile memory devices. Memory devices include storage space within a single physical storage device or storage space distributed across multiple physical storage devices.

[0042] The software instructions may be read into the memory 230 and / or storage component 240 from another computer-readable medium or from another device via the communication interface 270. When executed, the software instructions stored in the memory 230 and / or storage component 240 may cause the processor 220 to perform one or more of the processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more of the processes described herein. Accordingly, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0043] Figure 2 The number and arrangement of components shown are provided as examples. Figure 2 The device 200 may include more components, fewer components, different components, or components arranged differently than those shown. Additionally or alternatively, one or more components of the device 200 may perform one or more functions described as being performed by another group of components of the device 200.

[0044] In an embodiment of the present disclosure, a network-based media processing (NBMP) system is provided. Figure 3 The NBMP system architecture according to an embodiment of the present application is shown, which can be implemented for cloud processing. NBMP system 300 includes an NBMP source 310, an NBMP workflow manager 320, a function repository 330, one or more media processing entities (MPEs) 340, a media source 350, and a media receiver 360. NBMP source 310, NBMP workflow manager 320, function repository 330, MPE 340, media source 350, and media receiver 360 may include or be implemented by at least one or more processors and a memory storing code, the code being configured to cause at least one or more processors to perform the functions of NBMP source 310, NBMP workflow manager 320, function repository 330, MPE 340, media source 360, and media receiver 360, respectively.

[0045] The NBMP source 310 can communicate workflow descriptions with the NBMP workflow manager 320 via the NBMP workflow API 311. The NBMP source 310 can also communicate function descriptions with the function repository 330 via the function discovery API 313. For example, the NBMP source 310 can send one or more workflow description documents (WDDs) to the NBMP workflow manager 320 and can read function descriptions of functions stored in the function repository 330, which are media processing functions such as media decoding, feature point extraction, camera parameter extraction, projection method, seam information extraction, blending, post-processing, and encoding functions stored in the memory of the function repository 330. The NBMP workflow manager 320 can communicate with the function repository 330 via the function discovery API 312, which can be the same or different API as the function discovery API 313, and can communicate with one or more of the MPEs 340 via API 314 (e.g., MPE API).

[0046] The media processing entities 340 can include one or more tasks 341. The NBMP workflow manager 320 can also communicate with the tasks 341 via API 315 (e.g., NBMP task API). The NBMP workflow manager 320 can use the API 315 to establish, configure, manage, and monitor one or more tasks 341 of a workflow that can be performed by one or more MPEs 340. To configure, manage, and monitor the tasks 341 of a workflow, the NBMP workflow manager 320 can send messages such as requests to one or more of the MPEs 340 and / or the tasks 341, where each message can have a number of descriptors, each of which has a number of parameters. Additionally, the communications between the NBMP source 310, the NBMP workflow manager 320, the function repository 330, and the MPEs 340 can be considered control flow.

[0047] Each task 341 may include a media processing function 343 and a configuration 342 for the media processing function 343. Each task 341 in the corresponding media processing entity 340 may also communicate with each other to facilitate data flow between tasks. In an embodiment, the NBMP workflow manager 320 may select a task based on the description of the task in the WDD and search the function repository 330 via the function discovery API 312 to find an appropriate function to be run as the task 341 of the current workflow. One or more MPEs 340 may be configured to receive media content from a media source 350, process the media content according to a workflow including the task 341 created by the NBMP workflow manager 320, and output the processed media content to a media receiver 360. In an embodiment, one or more MPEs 340 may be provided in parallel for multiple media streams 316 and 317 between the media source 350 and the media receiver 360.

[0048] Media source 350 may include a memory for storing media and may be integrated with or separate from NBMP source 310. In an embodiment, NBMP workflow manager 320 may notify NBMP source 310 when a workflow is ready, and media source 350 may transmit media content to one or more MPEs 340 based on the notification that the workflow is ready, and one or more MPEs 340 may transmit the media content to media sink 360. The communication between media source 350, MPE 340, and media sink 360 may be considered a data stream.

[0049] Figure 4 An example of NBMP segmentation and merging of segments of a media stream for parallel processing is shown. Figure 4 As shown, the NBMP splitter and merger process 400 includes a splitter 420 and a merger 430. Figure 4 In

[15] , task T is converted into n instances of task T, where n instances run in parallel. The input media stream 410 to task T can be continuous. Splitter 420 converts the media stream into N media substreams. Each substream is processed by an instance of T. Merger 430 then interweaves the substreams to generate output 450. Output 450 is equivalent to the output stream of task T (i.e., media stream 440).

[0050] like Figure 4 As shown, the 1:N splitter 420 and the N:1 merger 430 operate on segment boundaries. Each segment has associated with it start, duration, and length metadata. Since the segments are independent, the substreams are independent of each other in terms of being processed by tasks T. In an embodiment, tasks T0, ..., T N-1is an instance of task T and can process segments simultaneously or individually. Since segments and substreams are independent, each instance of task T (i.e., T0, ..., T N-1 ) can run at its own speed.

[0051] According to an embodiment, two concepts may be added to the functions of the splitter 420 and the merger 430 to split a stream into sub-streams having a common header and merge the sub-streams, and to detect segment boundaries using a start code and a sequence number.

[0052] [Processing public headers]

[0053] In some embodiments, the media stream input to the segmenter 420 may have a common header in the stream preceding the segments, such as a VPS / SPS header in a video elementary stream or a segment initialization / CMAF main header in DASH / CMAF.

[0054] In some embodiments, the N output substreams from splitter 420 can be a uniform stream. Therefore, a common header can be copied to all N output substreams. Although merger 430 can receive a header in each of its N inputs, output 450 must have only one header, and duplication of input headers must be avoided. Therefore, the NBMP splitter and merger functions need to properly support processing common headers.

[0055] According to an embodiment, information about the common header may be provided as input metadata to the segmenter 420 and the merger 430. Any stream input to the segmenter 420 (e.g., the input media stream 410 and / or the media stream 440) may have at most one common header. Further, all streams input to the merger 430 (e.g., in the streams generated by tasks T0, ..., T N-1 The N streams output after processing the segments) must have at most one identical common header. The location of the common header can be provided by time metadata and / or start code and sequence number. When the location of the common header is provided by time metadata, the start, duration, and length constraints of the common header are defined. For example, if the duration constraint of a media segment is 0, then all other media segments of the media stream must have a non-zero duration. When the location of the common header is provided by start code and sequence number, the common header of the media segment will have the same start code as other media segments. However, if the sequence number of the media segment is 0, then all other media segments of the media stream must have a non-zero sequence number.

[0056] Reference below Figure 5 An example of processing a common header when the location of the common header is provided by time metadata is described. The processing may be performed by the NBMP workflow manager 320.

[0057] Figure 5is a flow diagram of an example process 500 for processing an input media stream containing a common header by using temporal metadata.

[0058] like Figure 5 As shown, at S501, the segmenter 420 finds the common header segment in the media stream. Then, the process 500 continues with the media segment metadata and as shown in FIG. Figure 4 The media stream is divided as described in the embodiment of the present invention. That is, the media stream is divided into N sub-streams (S502). At S503, the splitter 420 copies the common header to each of the N sub-streams. At S504, the N sub-streams are then processed by tasks T0, ..., T N-1 The process 500 then proceeds to the merger 430.

[0059] At S505, the merger 430 finds all common header segments in the N substreams input to the merger 430, and these common header segments are copied to the N substreams at S503. At S506, the merger 430 copies only one common header from one of the N substream segments and discards the other common headers from the other substreams. The merger 430 then merges the N substreams together (S507), as shown in FIG. Figure 4 At S508 , the merger 430 marks the common header in the merged output metadata from S507 .

[0060] The common header can also be processed when its location is provided by the start code and sequence number. NBMPAMD1 uses the following information in the metadata input to its splitter and merger functions (ie, SDLT as described below) to perform this processing.

[0061] S: Start of segment in T scale.

[0062] D: Duration of the segment in T scale.

[0063] L: The length of the fragment in bytes.

[0064] T: Time scale (number of ticks in seconds).

[0065] Embodiments of the present disclosure provide an alternative approach. That is, in the input stream, each segment begins with a unique code that is not repeated anywhere else in the segment or in any other segment. This unique code can be present only at the beginning of each segment. Thus, the start code can be a marker indicating the start of a new segment.

[0066] If the input segments have the same start mode, it is not necessary to have the start time and duration of each segment. However, with respect to merger 430, the segments must be merged in the correct order. Thus, sequence number metadata can be added to the input of merger 430. In embodiments, sequence number metadata showing the order of each segment can also be added to the input of splitter 420 to maintain uniformity. The media segment sequence number is a non-zero number that monotonically increases over time.

[0067] According to embodiments, the following extension can be provided to the splitter function of NBMP workflow manager 320. Specifically, the extension can be provided to splitter 420.

[0068] urn:mpeg:mpeg i:nbmp:2020:split

[0069] In embodiments, assuming that the input media stream is continuous without any gap, splitter 420 (including the above extension) can follow a set of functional requirements. This set of functional requirements for splitter 420 is described below.

[0070] Splitter 420 can include one input buffer and N output buffers, where N is a configuration parameter for the number of splits. The input and output buffers can be, for example, first-in-first-out (FIFO) buffers.

[0071] Splitter 420 operates on one input segment at a time and generates N output segments. The input segment has a set of consecutive samples and a maximum duration D in a time scale T, where both D and T are configuration parameters. The input segment can include timing metadata or sequence metadata. The timing metadata includes a time scale t = T, a start time s in the time scale t, a duration d in the time scale t, and a length l (bytes) as metadata and functional constraints. The sequence metadata includes a unique start code that is the same as all other input segments, and a continuously increasing sequence number. The input segment has no overlapping samples with other input segments.

[0072] The set of input segments to splitter 420 covers the entire duration of the input. That is, there are no samples of the input that are missed in the total duration. Splitter 420 operates on the input segments in increasing order. That is, an input segment covering an earlier duration will be processed before any input segment covering a later duration.

[0073] The media stream output at each output buffer at any time can include zero or more output segments. An output segment can include consecutive sample durations of maximum duration D in a time scale t. An output segment can also include timing metadata or sequence metadata. The timing metadata includes a time scale t = T, a start time s in the time scale t, a duration d in the time scale t, and a length l (bytes) as metadata and functional constraints. The sequence metadata includes a start code that is the same and unique to all other N output segments, and a sequence number that is incrementally consecutive. The collection of all output segments from the N output buffers of the splitter 420 together cover the entire duration of the input media stream (i.e., no input samples are missed in the collection of output segments).

[0074] The splitter 420 operates on the input segments and divides them into N output segments. Each of the N output buffers can receive one of the N output segments. The input segments can be divided into N output segments having the order s0, s1,..., s N-1 where the segment s i covers the duration of the media just between the segments s i-1 and s i+1 . The output buffers can be ordered as O0, O1,..., O N-1 . Thus, the output segment s i may be placed in the buffer O i , and so on.

[0075] In an embodiment, when there is a common header, the splitter 420 can repeat the common header in each output. For example, if the input media stream 410 (and similarly the media stream 440) has a common header, the splitter 420 will repeat the common header in each substream. Further, the first segment can be a common header segment, and the duration of the first segment can be zero.

[0076] Table 1 shows an example of a splitter function description template according to an embodiment. As shown in Table 1, the splitter function description template includes descriptors, parameters, parameter types, and their respective descriptions.

[0077] Table 1 - Splitter function description template

[0078]

[0079]

[0080] Table 2 shows example splitter configuration parameters according to embodiments. As shown in Table 2, the splitter configuration parameters can include units, types, and valid ranges. Definitions of the splitter configuration parameters are also shown in Table 2. The splitter configuration parameters of Table 1 can be provided in the splitter 420.

[0081] Table 2 - Splitter Configuration Parameters

[0082]

[0083]

[0084]

[0085] Table 3 shows splitter variable parameters (e.g., management interface FIFO (MIFIFO) buffer parameters) according to embodiments, including units, types, and valid ranges of the parameters.

[0086] Table 3 - Splitter Variable Parameters

[0087]

[0088] According to embodiments, the following extensions can be provided to the merger function of the NBMP workflow manager 320. Specifically, the extensions can be provided to the merger 430.

[0089] urn:mpeg:mpeg i:nbmp:2020:merge

[0090] In embodiments, the merger 430 (including the above extensions) can follow a set of functional requirements assuming that the input media streams are continuous without any gaps. This set of functional requirements for the merger 430 is described below.

[0091] The merger 430 includes N input buffers and one output buffer, where N is a configuration parameter for the number of splits. The output buffer can be, for example, a first-in-first-out (FIFO) buffer.

[0092] The merger 430 operates one input segment for each input buffer at a time, for a total of N segments, and generates one output segment. Each of the input segments can have a set of consecutive samples and a maximum duration D in a time scale T, where D and T are both configuration parameters. Each of the input segments can also include timing metadata or sequence metadata. The timing metadata includes a time scale t = T, a start time s in the time scale t, a duration d in the time scale t, and a length l (bytes). The sequence metadata includes a same and unique start code, where all input segments and sequence numbers are ordered in increasing order. The input segments have no overlapping samples with other input segments.

[0093] The set of input segments input to the N buffers of merger 430 covers the entire duration of the input. That is, no samples of the input are omitted in the total duration. Merger 430 operates on the input segments in increasing order. That is, a segment covering an earlier duration is processed before any segment covering a later duration.

[0094] The media stream at the output buffer at any time can include zero or more output segments. Each output segment can include a maximum duration of N*D consecutive sample durations in the time scale t. The output segment can also include timing metadata or sequence metadata. The timing metadata includes the time scale t=T, the start time s in the time scale t, the duration d in the time scale t, and the length l (bytes). The sequence metadata includes the same and unique start code, where all N output segments and sequence numbers are sorted in increasing order. The collection of output segments of all output buffers in the media stream together cover the entire duration of the input media stream (i.e., no samples of the input are omitted in the collection of output segments).

[0095] Merger 430 operates on each input segment from each input buffer and merges the input segments into one output segment. One segment from each input buffer, s0, s1,..., s N-1 , respectively, from input buffers I0, I1,..., IN, respectively, can be processed. N-1 Segment s i may include samples that are exactly in the time interval between the time interval of s i-1 and s i+1 . Segments s0, s1,..., s N-1 may be concatenated in increasing order into one output segment. That is, s0may be concatenated with s1, then with s2, and so on until s N-1 . Segments s0, s1,..., s N-1 may be concatenated such that the output segment includes all samples of segments s0, s1,..., s N-1 , without duplicate samples, and has the earliest presentation of EPT, or their sequence numbers are sorted in increasing order.

[0096] In an embodiment, when there is a common header at each input of the merger 430, the merger 430 can duplicate the common header once at its output. For example, if N processed substreams each have a common header, the merger 430 can have only one instance of the common header in the output 450. In one or more embodiments, the first segment of the output 450 can be the common header, and the duration of the first segment can be zero. As such, the merger 430 can keep only one common header and discard the common headers of other input segments present in the substreams produced by the splitter 420.

[0097] Table 4 shows an example of a merger function description template according to an embodiment. As shown in Table 4, the merger function description template includes descriptors, parameters, parameter types, and their respective descriptions.

[0098] Table 4 - Merger function description template

[0099]

[0100]

[0101] Table 5 shows merger configuration parameters according to an embodiment. As shown in Table 5, the merger configuration parameters can include units, types, and valid ranges. Definitions of the merger configuration parameters are also shown in Table 5. The merger configuration parameters of Table 5 can be provided in the merger 430.

[0102] Table 5 - Merger configuration parameters

[0103]

[0104]

[0105] Table 6 shows merger variable parameters (e.g., management interface FIFO (MIFIFO) buffer parameters) according to an embodiment, including units, types, and valid ranges of the parameters.

[0106] Table 6 - Merger variable parameters

[0107]

[0108] Figure 6 is a flowchart of an example method 600 for identifying segment boundaries in segments having a unique start code, according to one or more embodiments.

[0109] In some embodiments, Figure 6 One or more of the process blocks of the method 600 can be performed by the platform 120. In some embodiments, Figure 6One or more of the process blocks can be performed by another device or set of devices (such as user device 110) separate from or including platform 120.

[0110] As shown, at operation 610, method 600 can include obtaining an input stream containing a plurality of segments, where the input stream includes a first common header located in the plurality of segments. Figure 6

[0111] At operation 620, method 600 can include identifying a start code located in the plurality of segments. The start code can be located at the beginning of one segment. The start code is a unique code that indicates the start of the segment.

[0112] At operation 630, method 600 can include identifying a sequence number for each of the plurality of segments. The sequence number indicates the position of each segment in the plurality of segments. That is, the sequence number can be used to determine the order in which the plurality of sub-streams of the input stream are processed.

[0113] At operation 640, method 600 can include using splitter 420 or an NBMP splitter function to split the input stream into a plurality of sub-streams based on the start code and the sequence number and signal the location of the respective common header for each sub-stream.

[0114] At operation 650, method 600 can include using merger 430 or an NBMP merger function to merge the plurality of sub-streams based on the start code and the sequence number and signal the location of the second common header.

[0115] At operation 660, method 600 can include generating an output stream having a common header that is produced once in a first segment of the output stream.

[0116] Although Figure 6 Example blocks of the method are shown, but in some embodiments, the method can include more blocks, fewer blocks, different blocks, or blocks arranged differently than those depicted in Figure 6 Additionally or alternatively, two or more of the blocks of the method can be performed in parallel. The method can be performed in NBMP system 300, but is not limited thereto.

[0117] Figure 7 is a block diagram of an example of computer code for identifying segment boundaries in segments having a unique start code in accordance with one or more embodiments.

[0118] According to embodiments of the disclosure, at least one processor with a memory storing computer code can be provided. The computer code can be configured to perform any number of aspects of the disclosure when executed by the at least one processor.

[0119] ​For example, referring to Figure 7 , the computer code 700 can be implemented in the NBMP system 300, but is not limited thereto.

[0120] As shown in Figure 7 , the computer code 700 can include obtaining code 710, first identifying code 720, second identifying code 730, splitting code 740, merging code 750, and generating code 760.

[0121] The obtaining code 710 can be configured to cause the at least one processor to obtain an input stream including a plurality of segments, wherein the input stream includes a first common header located in the plurality of segments.

[0122] The first identifying code 720 can be configured to cause the at least one processor to identify a start code in the plurality of segments. The start code can be located at the beginning of one segment, and the start code is a unique code indicating the start of the segment.

[0123] The second identifying code 730 can be configured to cause the at least one processor to identify a sequence number in each of the plurality of segments. The sequence number indicates the position of each segment in the plurality of segments, and can be used to determine the processing order of the segments.

[0124] The splitting code 740 can be configured to cause the at least one processor to split the input stream into a plurality of sub-streams based on the start code and the sequence number using the splitter 420 or the NBMP splitter function and signal the location of the respective common header of each sub-stream.

[0125] The merging code 750 can be configured to cause the at least one processor to merge the plurality of sub-streams based on the start code and the sequence number using the merger 430 or the NBMP merger function and signal the location of the second common header.

[0126] The generating code 760 can be configured to cause the at least one processor to generate an output stream based on the merged plurality of sub-streams, wherein the output stream includes the second common header.

[0127] Although Figure 7 example blocks of the computer code 700 of an apparatus or device according to an embodiment are shown, in some embodiments the apparatus can include more, less, or different blocks than those depicted in Figure 7 . Additionally or alternatively, two or more of the blocks of the apparatus can be combined.

[0128] The techniques described above for signaling common headers when processing input media streams can be used alone or in combination in any order. Further, each of the methods (or embodiments) can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors can execute programs stored in a non-transitory computer-readable medium.

[0129] The present disclosure provides illustrations and descriptions but is not intended to be exhaustive or to limit implementations to the precise form disclosed. Modifications and variations are possible in light of the disclosure, or can be acquired from practice of the implementations.

[0130] As used herein, the term component is intended to be broadly interpreted to include hardware, firmware, or a combination of hardware and software.

[0131] It will be apparent that systems and / or methods, described herein, can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and / or methods were described herein without reference to specific software code — it being understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0132] Even though combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure to only these permutations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below can stand on its own as a separate implementation, the disclosure of possible implementations includes each and every combination of the dependent claims with each other dependent claim.

[0133] No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and can be used interchangeably with “one or more.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, or the like), and can be used interchangeably with “one or more.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise.

Claims

1. A method of processing a media stream by a network based media processing (NBMP) function, performed by at least one processor, the method comprising: The method comprises: obtaining an input stream comprising a plurality of segments, wherein the input stream comprises a first common header located before the plurality of segments; identifying a start code for each segment of the plurality of segments, wherein the start code is located at the beginning of each segment and the start code is a unique code indicating the start of the each segment; identifying a sequence number for each segment of the plurality of segments, wherein the sequence number indicates the position of the each segment in the plurality of segments; segmenting the input stream into a plurality of sub-streams using a network-based media processing (NBMP) segmenter function based on the start code and the sequence number, copying the first common header to each of the plurality of sub-streams, and signaling the location of the respective common header in each sub-stream; merging the plurality of sub-streams using a network-based media processing (NBMP) merger function based on the start code and the sequence number, copying only one common header from one of the plurality of sub-streams to the location of a second common header in an output stream, discarding the common headers of the other sub-streams, and signaling the location of the second common header; and generating the output stream based on the merged plurality of sub-streams, wherein the output stream comprises the second common header.

2. The method of claim 1, wherein, The respective common header of each sub-stream is the same.

3. The method of claim 1, wherein the location of the respective common header of each sub-stream is unique.

4. The method of claim 1, wherein, Each segment included in the input stream corresponds to one of the plurality of sub-streams.

5. The method of claim 1, wherein, The plurality of segments included in the input stream covers the entire duration of the input stream.

6. The method of claim 1, wherein, The plurality of sub-streams are processed in an increasing order based on the sequence number.

7. The method of claim 1, wherein, The segment of the second common header is the first segment of the output stream and has a duration of zero.

8. An apparatus for processing a media stream by a network based media processing (NBMP) function, the apparatus characterized in that, The apparatus comprises: at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising: obtaining code configured to cause the at least one processor to obtain an input stream comprising a plurality of segments, wherein the input stream comprises a first common header located before the plurality of segments; first identifying code configured to cause the at least one processor to identify a start code for each segment of the plurality of segments, wherein the start code is located at the beginning of each segment and the start code is a unique code indicating the start of the segment; second identifying code configured to cause the at least one processor to identify a sequence number for each segment of the plurality of segments, wherein the sequence number indicates the position of the each segment in the plurality of segments; segmenting the input stream into a plurality of sub-streams using a network-based media processing (NBMP) segmenter function based on the start code and the sequence number, copying the first common header to each of the plurality of sub-streams, and signaling the location of the respective common header in each sub-stream; merging the plurality of sub-streams using a network-based media processing (NBMP) merger function based on the start code and the sequence number, copying only one common header from one of the plurality of sub-streams to the location of a second common header in an output stream, discarding the common headers of the other sub-streams, and signaling the location of the second common header; and generating the output stream based on the merged plurality of sub-streams, wherein the output stream comprises the second common header. splitting code configured to cause the at least one processor to split, based on the start code and the sequence number, the input stream into a plurality of sub-streams using a network based media processing (NBMP) splitter function, copy the first common header to each of the plurality of sub-streams, and signal a location of each sub-stream's respective common header; merging code configured to cause the at least one processor to merge, based on the start code and the sequence number, the plurality of sub-streams using a network based media processing (NBMP) merger function, copy only one common header from one of the plurality of sub-streams to a location of a second common header in an output stream, discard common headers of other sub-streams, and signal the location of the second common header; and generating code configured to cause the at least one processor to generate the output stream based on the merged plurality of sub-streams, wherein the output stream includes the second common header.

9. The apparatus of claim 8, wherein, the each sub-stream's respective common header is identical.

10. The apparatus of claim 8, wherein, the location of the each sub-stream's respective common header is unique.

11. The apparatus of claim 8, wherein, each segment included in the input stream corresponds to one of the plurality of sub-streams.

12. The apparatus of claim 8, wherein, the plurality of segments included in the input stream cover an entire duration of the input stream.

13. The apparatus of claim 8, wherein, the plurality of sub-streams are processed in an increasing order based on the sequence number.

14. The apparatus of claim 8, wherein, the segment of the second common header is a first segment of the output stream and has a duration of zero.

15. A non-transitory computer-readable storage medium storing instructions, the method comprising: the instructions include one or more instructions that, when executed by at least one processor of an apparatus that processes a media stream by a network based media processing (NBMP) function, cause the at least one processor to implement the method of any of claims 1 to 7.

Images