Method, apparatus and computer program for wire formatting segmented media metadata for parallel processing on a cloud platform

Abstract

The method, executed by at least one processor, includes segmenting a media stream into a plurality of media segments in a multi-dimensional space. The method includes determining respective metadata associated with each of the plurality of media segments. The method includes encapsulating the plurality of metadata into a predefined wire format, each encapsulated metadata including a position or sequence associated with each of the plurality of media segments. The method includes processing the plurality of media segments in parallel based on the encapsulated metadata. The method further includes merging the plurality of media segments into a media stream after parallel processing.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 298,922, filed January 12, 2022, and U.S. Patent Application No. 17 / 991,530, filed November 21, 2022, the disclosures of which are incorporated herein by reference in their entirety.

[0002] [Technical field] The present disclosure relates generally to wire formats, and more particularly to methods and apparatus for wire formatting segmented media metadata for parallel processing on cloud platforms. [Background technology]

[0003] The Network-Based Media Processing (NBMP) framework defines interfaces, including both data formats and Application Programming Interfaces (APIs), between entities connected through a digital network for media processing. The NBMP standard defines a set of tools for independent processing of media segments. The framework enables dynamic creation of media processing pipelines as well as access to processed media data and metadata in real-time or a deferred manner. Network and cloud platforms are used to run various applications. While metadata parameters are defined, the NBMP standard does not define an interoperable wire format for segment metadata. Summary of the Invention

[0004] The following presents a simplified summary of one or more embodiments of the present disclosure in order to provide a basic understanding of such embodiments. This summary is not an extensive overview of all contemplated embodiments, and is not intended to identify key or critical elements of all embodiments or to delineate the scope of any or all embodiments. Its sole purpose is to present some concepts of one or more embodiments of the present disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0005] According to an exemplary embodiment, a method executed by at least one processor includes segmenting a media stream into a plurality of media segments in a multi-dimensional space. The method includes determining respective metadata associated with each of the plurality of media segments. The method includes encapsulating the plurality of metadata into a predetermined wire format, each encapsulated metadata including a position or sequence associated with each of the plurality of media segments. The method includes processing the plurality of media segments in parallel based on the encapsulated metadata. The method further includes merging the plurality of media segments into a media stream after parallel processing.

[0006] According to an exemplary embodiment, an apparatus includes at least one memory configured to store computer program code and at least one processor configured to read the computer program code and operate as directed by the computer program code. The computer program code includes segmentation code configured to cause the at least one processor to segment a media stream into a plurality of media segments in a multi-dimensional space. The computer program code includes determination code configured to cause the at least one processor to determine respective metadata associated with each of the plurality of media segments. The computer program code includes encapsulation code configured to cause the at least one processor to encapsulate the plurality of metadata into a predetermined wire format, each encapsulated metadata including a position or sequence associated with each of the plurality of media segments. The computer program code includes parallel processing code configured to cause the at least one processor to process the plurality of media segments in parallel based on the encapsulated metadata. The computer program code further includes merging code configured to cause the at least one processor to merge the plurality of media segments into a media stream after parallel processing.

[0007] According to an exemplary embodiment, a non-transitory computer-readable medium storing instructions, when executed by a processor, causes the processor to perform a method including segmenting a media stream into a plurality of media segments in a multi-dimensional space. The method includes determining respective metadata associated with each of the plurality of media segments. The method includes encapsulating the plurality of metadata into a predetermined wire format, each encapsulated metadata including a position or sequence associated with each of the plurality of media segments. The method includes processing the plurality of media segments in parallel based on the encapsulated metadata. The method further includes merging the plurality of media segments into a media stream after parallel processing.

[0008] SUMMARY Disclosed herein are methods, apparatus, and non-transitory computer-readable storage media for wire formatting of segmented media metadata for parallel processing on cloud platforms.

[0009] Additional embodiments will be set forth in the description that follows, and in part will be obvious from the description, and / or may be learned by practice of presented embodiments of the present disclosure.

[0010] These and other aspects, features, and aspects of embodiments of the present disclosure will become apparent from the following description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram of an example architecture for network-based media processing (NBMP), in accordance with various embodiments of the present disclosure. [Figure 2] 1 is a diagram of a template of a splitter and merger in an NBMP architecture in accordance with various embodiments of the present disclosure. [Figure 3]1 is an example of segment position metadata as a JavaScript Object Notation (JSON) object according to various embodiments of the present disclosure. [Figure 4] 1 is an example of segment sequence metadata as a JSON object according to various embodiments of the present disclosure. [Figure 5] 1 is a flowchart of an example process for segmenting and processing a media stream using a wire format for media stream metadata, according to various embodiments of the present disclosure. [Figure 6] 1 illustrates an example computer system according to various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following detailed description of the example embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0013] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit implementations to the precise form disclosed. Modifications and variations are possible in light of the foregoing disclosure or may be acquired from practice of the examples. Moreover, one or more features or components of one embodiment may be incorporated into or combined with other embodiments (or one or more features of other embodiments). Moreover, in the flowcharts and descriptions of operations provided below, it will be understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least in part), and the order of one or more operations may be swapped.

[0014] It will be apparent that the systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, even if the operation and behavior of a system and / or method is described herein without reference to specific software code, it will be understood that software and hardware can be designed to implement the system and / or method based on the description herein.

[0015] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim within its scope.

[0016] No element, act, or instruction used herein should be construed as critical or required unless explicitly stated otherwise. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar terms are used. Also, as used herein, terms such as "has," "have," "having," "include," and "including" are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on" unless expressly stated otherwise. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include A only, B only, or both A and B.

[0017] References throughout this specification to "one embodiment," "an embodiment," or similar terms mean that a particular feature, structure, or characteristic described in connection with the illustrated embodiment is included in at least one embodiment of the solution. Thus, appearances of the phrases "in one embodiment," "in an embodiment," and similar phrases throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0018] Furthermore, the described features, advantages, and characteristics of the present disclosure may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize, in light of the description herein, that the present disclosure may be practiced without one or more particular features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in particular embodiments that may not be present in all embodiments of the present disclosure.

[0019] Embodiments of the present disclosure are directed to a wire format for segmented media metadata used in NBMP processing of media segments. Figure 1 illustrates an embodiment of an NBMP reference architecture (100). The NBMP reference architecture (100) may include a media source 102 that may provide a media flow (e.g., a media stream) to a media processing entity (MPE) (104). The MPE (104) may also provide the media flow to a media sink (106).

[0020] The reference architecture 100 may further include an NBMP source 108 that provides an NBMP workflow API and a workflow description to an NBMP workflow manager 110. The NBMP workflow manager 110 may receive NBMP tasks from the MPE 104 and provide MPE APIs to the MPE 104. The reference architecture 100 may further include a function repository 112 that interacts with the NBMP source 108 and the NBMP workflow manager 110.

[0021] Figure 2 shows an example of a splitter and merger template defined in the NBMP standard. The splitter and merger functions can be used for parallel processing of segments. A media stream may be continuous. However, the splitter function may convert the media stream into N media substreams. Each substream may be processed by an instance of T, and the substreams are then interleaved together to generate an output (e.g., equivalent to task T) stream.

[0022] A 1:N splitter function and an N:1 merger function may operate on the segment functions. Each segment may have (i) a start, (ii) a duration, and (iii) length metadata, or (i) a start code and (ii) a sequence number associated with the segment. Since the segments are independent, the substreams are therefore independent from each other in that they are processed by task T. In some embodiments, tasks T0, ..., T N-1 does not need to process the segments simultaneously. Because the segments and substreams are independent, each instance of a task can run at a rate independent of the others (e.g., each instance of a task can run at its own rate). The previous NBMP standard only addressed one-dimensional segmentation of media data.

[0023] According to some embodiments, each segment may use one of the following metadata: 1. Location Metadata: a. Scaling vector T =[t0,t1, ,t M-1 ], S and D Scale factor, b. Unit t i Each index s i A start vector representing the start point of a media segment in M-dimensional space. S =[s0,s1,...,s M-1 ], c. Unit t i Each index d i A length vector representing the hyperspace that a media segment covers in the M-dimensional space. D =[d0,d1, ,d M-1 ], and d. The size of segment L in bytes. 2. Or, sequence metadata: a. Each index n i A sequence vector, which represents the sequence of media segments in the M-dimensional space, n =[n0,n1, ,n M-1 ], b. A unique code at which all segments start, a start code C that is not repeated in the middle of any segment, and c. The size of segment L in bytes.

[0024] The NBMP standard does not define a wire format for the above metadata.

[0025] According to some embodiments, the wire format for the metadata of one or more segments may include a byte stream format for location metadata and / or sequence metadata. The byte stream format may include the associated Multipurpose Internet Mail Extension (MIME). According to some embodiments, the wire format for the metadata of one or more segments may include a JavaScript Object Notation (JSON) format for location metadata and / or sequence metadata. The JSON format may include the associated MIME.

[0026] According to some embodiments, the parameter C , S , D , and T are defined as follows. C may be an M-dimensional vector [c0, c1, ···, c M-1 , where the element c i has an index i, and index i + 1 is nested in index i, that is, one increment of index i of the vector is considered a larger increment than any increment of indices i + 1, i + 2, ···, M - 1 (0 ≤ i < M).

[0027] A multi-dimensional segment having dimension M is a point S = [s0, s1, ···, s M-1 and a length D = [d0, d1, ···, dM-1 ], where s i and d i is a non-negative integer. If a non-integer value is required, use the vector T =[t0,t1, ,t M-1 ] is the scale factor t of dimension i i and the actual starting point and length in dimension i are s i / t i and d i / t i At this time, t i is a positive integer.

[0028] Table 1 shows an example of the byte format for position metadata for a single segment. [Table 1]

[0029] The table can be repeated for additional segments by one or more runs of the same table for multiple segments: Table 1, Table 1, ..., Table 1.

[0030] Table 2 shows an example of the byte format for sequence metadata for a single segment. In some embodiments, because the start code C is common to all segments (e.g., the same start code in all segments), the start code C is not carried as part of the sequence metadata and has its own input to the function. [Table 2]

[0031] The table can be repeated for additional segments by one or more runs of the same Table 2 for multiple segments: Table 2, Table 2, ..., Table 2.

[0032] Figure 3 shows an example JSON object (300) of position metadata for one or more segments. Figure 4 shows an example JSON object (400) of sequence metadata for one or more segments. In some embodiments, the start code C may be common to all segments, so the start code C may not be carried as part of the sequence metadata and has its own input to the function.

[0033] Table 3 shows example MIME types for each wire format (e.g., byte format and JSON object). [Table 3]

[0034] Table 4 shows an example of extended step descriptor parameters for advertising supported formats by a function in its Function Description Document (FDD). The underlined lines are new parameters. [Table 4] JPEG0007736332000005.jpg226170JPEG0007736332000006.jpg227170JPEG0007736332000007.jpg91170

[0035] FIG. 5 depicts a flowchart of an embodiment of a process (500) for segmenting and processing a media stream using a wire format for media stream metadata. The process (500) may begin at operation (502), where a media stream is segmented into a plurality of media segments. The process continues at operation (504), where respective metadata associated with each of the plurality of media segments is determined. The process continues at operation (506), where the metadata for each media segment is encapsulated into a predetermined wire format, such as a byte format or a JSON format. The metadata may be positional metadata or sequence metadata. The metadata may include a position or sequence associated with each of the plurality of media segments. The process continues at operation (508), where the plurality of media segments are processed in parallel based on the encapsulated metadata. The process continues at operation (510), where, after parallel processing, the plurality of media segments are merged into a media stream.

[0036] The techniques of the embodiments of the present disclosure described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 6 illustrates a computer system (600) suitable for implementing embodiments of the disclosed subject matter.

[0037] Computer software may be coded using any suitable machine code or computer language that may undergo mechanisms such as assembly, compilation, linking, etc. to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc. directly or through interpretation, microcode execution, etc.

[0038] The instructions may be executed by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0039] 6 for computer system (600) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components depicted in the exemplary embodiment of computer system (600).

[0040] The computer system (600) may include certain human interface input devices. Such human interface input devices may respond to input by one or more users through, for example, tactile input (e.g., keystrokes, swipes, dataglove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), and olfactory input (not shown). Human interface devices may also be used to capture certain media not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0041] The input human interface devices may include one or more (only one of each is shown) of a keyboard (601), a mouse (602), a trackpad (603), a touchscreen (610), a dataglove, a joystick (605), a microphone (606), a scanner (607), and a camera (608).

[0042] The computer system (600) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (610), datagloves, or joystick (605), although haptic feedback devices that do not function as input devices may also be present). For example, such devices may be audio output devices (e.g., speakers (609), headphones (not shown)), visual output devices (such as CRT screens, LCD screens, plasma screens, OLED screens, etc., each of which may or may not have touchscreen input capabilities, each of which may or may not have haptic feedback capabilities, some of which may be capable of outputting two-dimensional visual output or three or more dimensional output by means of, for example, stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown) (610)), and printers (not shown).

[0043] The computer system (600) may also include human-accessible storage devices and their associated media, such as CD / DVD ROM / RW (620) with media (621) such as CDs / DVDs, thumb drives (622), removable hard drives or solid state drives (623), legacy magnetic media such as tape and floppy disks (not shown), specialized ROM / ASIC / PLD-based devices such as security dongles (not shown), etc.

[0044] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0045] The computer system 600 may include interfaces to one or more communication networks. Networks may be, for example, wireless, wired, or optical. Networks may also be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, and the like; TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; and vehicular and industrial networks including CAN Bus. Certain networks generally require an external network interface adapter that connects to a particular general-purpose data port or peripheral bus 649 (e.g., a USB port on the computer system 600); others are typically integrated into the core of the computer system (e.g., an Ethernet interface on a PC computer system or a cellular network interface on a smartphone computer system) by attachment to a system bus, as described below. Using any of these networks, the computer system 600 may communicate with other entities. Such communications may be one-way, receive-only (such as a television broadcast), one-way transmit-only (from the CANBus to a particular CANBus device), or bidirectional (e.g., to other computer systems using a local or wide area digital network). Such communications may include communications to a cloud computing environment (655). Specific protocols and protocol stacks may be used in each of the networks and network interfaces as described above.

[0046] The above-mentioned Newman interface device, human-accessible storage device, and network interface (654) may be attached to the core (640) of the computer system (600).

[0047] The core (640) may include one or more central processing units (CPUs) (641), graphics processing units (GPUs) (642), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (643), task-specific hardware accelerators (644), etc. These devices may be connected through a system bus (648), along with read-only memory (ROM) (645), random access memory (646), internal mass storage such as a non-user-accessible internal hard drive, SSD, etc. (647). In some computer systems, the system bus (648) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (648) or through a peripheral bus (649). Peripheral bus architectures include PCI, USB, etc. A graphics adapter (650) may also be included with the core (640).

[0048] The CPU (641), GPU (642), FPGA (643), and accelerator (644) can execute specific instructions that, in combination, may constitute the above-mentioned computer code. The computer code may be stored in ROM (645) or RAM (646). Temporary data may also be stored in RAM (646), while persistent data may be stored, for example, in internal mass storage (647). Rapid storage and retrieval from any of the memory devices is made possible through the use of cache memory, which may be closely associated with one or more of the CPU (641), GPU (642), mass storage (647), ROM (645), RAM (646), etc.

[0049] The computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0050] By way of example and not limitation, a computer system having the architecture (600), and in particular the core (640), can provide functionality as a result of processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage, as introduced above, or may be storage specific to the core (640) that is non-transitory in nature, such as the core's internal mass storage (647) or ROM (645). Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (640). The computer-readable media may include one or more memory devices or chips, depending on particular needs. The software may cause the core (640), and in particular the processor (including CPU, GPU, FPGA, etc.) therein, to perform particular processes or particular portions of particular processes described herein, including defining data structures stored in RAM (646) and modifying such data structures according to software-defined processes. Additionally, or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 644), which may operate in place of or together with software to perform particular processes or portions of particular processes described herein. References to software may include logic, and vice versa, as appropriate. References to computer-readable media may encompass circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0051] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0052] It is understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed herein represents example approaches. Based on design preferences, it is understood that the specific order or hierarchy of blocks within a process / flowchart may be rearranged. Also, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0053] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail of integration. Furthermore, one or more of the above components described above may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or medium) having computer-readable program instructions for causing a processor to perform operations.

[0054] A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards, or mechanically encoded devices such as ridge structures in grooves in which instructions are stored, and any suitable combination thereof. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted through wires.

[0055] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmission cables, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within each computing / processing device.

[0056] The computer-readable program code / instructions for carrying out operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., over the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects or operations.

[0057] These computer-readable program instructions can be supplied to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to create a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, result in means for implementing the functions / acts specified in the flowchart and / or block diagram blocks. These computer-readable program instructions can be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium having the instructions stored thereon has an article of manufacture including instructions that implement aspects of the functions / acts specified in the flowchart and / or block diagram blocks.

[0058] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions executed on the computer, other programmable apparatus, or other device implement aspects of the functions / operations specified in the flowchart and / or block diagram blocks.

[0059] The flowcharts and block diagrams in the figures represent the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks than those shown in the figures. In some alternative implementations, the functions shown in the blocks may be performed in a different order than that shown in the figures. For example, two blocks shown in succession may actually be executed concurrently or nearly concurrently, or the blocks may be executed in the reverse order, depending on the functionality involved. Each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or a combination of dedicated hardware and computer instructions.

[0060] It will be apparent that the systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement these systems and / or methods is not a limitation of the implementation. Thus, although the operation and behavior of the systems and / or methods have been described herein without reference to specific software code, it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0061] The foregoing disclosure also encompasses the embodiments listed below.

[0062] (1) A method executed by at least one processor, comprising: segmenting the media stream into a plurality of media segments in a multi-dimensional space; determining respective metadata associated with each of the plurality of media segments; encapsulating a plurality of pieces of metadata into a predetermined wire format, each piece of encapsulated metadata including a position or sequence associated with each of the plurality of media segments; processing the plurality of media segments in parallel based on the encapsulated metadata; merging the plurality of media segments into the media stream after said parallel processing; A method comprising:

[0063] (2) The method of feature (1), the metadata for each media segment from the plurality of media segments includes location metadata. method.

[0064] (3) The method of feature (3), the predetermined wire format comprises a byte stream format; The byte stream format encapsulates one or more of a multidimensional scale vector, a position vector, a length vector, and size information. method.

[0065] (4) The method of feature (2), the predetermined wire format is a JSON array; wherein the array elements of the plurality of media segments encapsulate one or more of a multi-dimensional scale vector, a position vector, a length vector, and size information; method.

[0066] (5) The method of feature (1), the metadata for each media segment from the plurality of media segments includes sequence metadata; method.

[0067] (6) The method of feature (5), the predetermined wire format is a byte stream format; The byte stream format encapsulates one or more of a multi-dimensional sequence vector and size information. method.

[0068] (7) The method of feature (5), the predetermined wire format is a JSON array; wherein an array element of the plurality of media segments encapsulates one or more of a multi-dimensional sequence vector and size information; method.

[0069] (8) The method of feature (1), The predetermined wire format is Multipurpose Internet Mail Extensions (M I ME), method.

[0070] (9) at least one memory configured to store computer program code; and at least one processor configured to read the computer program code and operate as directed by the computer program code; The computer program code segmentation code configured to cause the at least one processor to segment the media stream into a plurality of media segments in a multi-dimensional space; determining code configured to cause the at least one processor to determine respective metadata associated with each of the plurality of media segments; encapsulation code configured to encapsulate a plurality of pieces of metadata into a predetermined wire format, each piece of encapsulated metadata including a position or sequence associated with each of the plurality of media segments, on the at least one processor; parallel processing code configured to cause the at least one processor to process the plurality of media segments in parallel based on the encapsulated metadata; merging code configured to cause the at least one processor to merge the plurality of media segments into the media stream after the parallel processing; 1. An apparatus comprising:

[0071] (10) The device of feature (9), the metadata for each media segment from the plurality of media segments includes location metadata. Device.

[0072] (11) The device of feature (10), the predetermined wire format comprises a byte stream format; The byte stream format encapsulates one or more of a multidimensional scale vector, a position vector, a length vector, and size information. Device.

[0073] (12) The device of feature (10), the predetermined wire format is a JSON array; wherein the array elements of the plurality of media segments encapsulate one or more of a multi-dimensional scale vector, a position vector, a length vector, and size information; Device.

[0074] (13) The device of feature (9), the metadata for each media segment from the plurality of media segments includes sequence metadata; Device.

[0075] (14) The device of feature (13), the predetermined wire format is a byte stream format; The byte stream format encapsulates one or more of a multi-dimensional sequence vector and size information. Device.

[0076] (15) The device of feature (13), the predetermined wire format is a JSON array; wherein an array element of the plurality of media segments encapsulates one or more of a multi-dimensional sequence vector and size information; Device.

[0077] (16) The device of feature (9), The predetermined wire format is Multipurpose Internet Mail Extensions (M I ME), Device.

[0078] (17) A non-transitory computer-readable medium storing instructions, The instructions, when executed by a processor, cause the processor to: segmenting the media stream into a plurality of media segments in a multi-dimensional space; determining respective metadata associated with each of the plurality of media segments; encapsulating a plurality of pieces of metadata into a predetermined wire format, each piece of encapsulated metadata including a position or sequence associated with each of the plurality of media segments; processing the plurality of media segments in parallel based on the encapsulated metadata; merging the plurality of media segments into the media stream after said parallel processing; A non-transitory computer-readable medium for carrying out a method comprising:

[0079] (18) The non-transitory computer-readable medium of feature (17), the metadata for each media segment from the plurality of media segments includes location metadata. Non-transitory computer-readable medium.

[0080] (19) The non-transitory computer-readable medium of feature (18), the predetermined wire format comprises a byte stream format; The byte stream format encapsulates one or more of a multidimensional scale vector, a position vector, a length vector, and size information. Non-transitory computer-readable medium.

[0081] (20) The non-transitory computer-readable medium of feature (18), the predetermined wire format is a JSON array; wherein the array elements of the plurality of media segments encapsulate one or more of a multi-dimensional scale vector, a position vector, a length vector, and size information; Non-transitory computer-readable medium.

Claims

1. 1. A method executed by at least one processor, comprising: segmenting the media stream into a plurality of media segments in a multi-dimensional space; determining respective metadata associated with each of the plurality of media segments; encapsulating a plurality of pieces of metadata into a predetermined wire format, each encapsulated piece of metadata including a position or a sequence associated with each of the plurality of media segments, the predetermined wire format being specified by a syntax element segment-metadata-supported-formats in a Function Description Document (FDD), the syntax element segment-metadata-supported-formats specifying supported wire formats for the positions and the sequences, respectively; processing the plurality of media segments in parallel based on the encapsulated metadata; merging the plurality of media segments into the media stream after said parallel processing; A method having the following.

2. the metadata for each media segment from the plurality of media segments includes location metadata. The method of claim 1.

3. the predetermined wire format comprises a byte stream format; The byte stream format encapsulates one or more of a multidimensional scale vector, a position vector, a length vector, and size information. The method of claim 2.

4. the predetermined wire format is a JSON sequence; wherein the array elements of the plurality of media segments encapsulate one or more of a multi-dimensional scale vector, a position vector, a length vector, and size information; The method of claim 2.

5. the metadata for each media segment from the plurality of media segments includes sequence metadata; The method of claim 1.

6. the predetermined wire format is a byte stream format; The byte stream format encapsulates one or more of a multi-dimensional sequence vector and size information. The method of claim 5.

7. the predetermined wire format is a JSON sequence; wherein an array element of the plurality of media segments encapsulates one or more of a multi-dimensional sequence vector and size information; The method of claim 5.

8. the predetermined wire format includes Multipurpose Internet Mail Extensions (MIME); The method of claim 1.

9. at least one memory configured to store computer program code; at least one processor configured to read the computer program code and to act as directed by the computer program code; wherein the computer program code, when executed by the at least one processor, causes the at least one processor to perform the method of any one of claims 1 to 8. Device.

10. A computer program comprising instructions, The instructions, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8. Computer program.

Images