Method and apparatus for wire formats for segmented media metadata for parallel processing on a cloud platform

Abstract

A method performed by at least one processor comprises the step of segmenting a media stream into a plurality of media segments in a multidimensional space. The method comprises the step of determining metadata associated with each of the plurality of media segments. The method comprises the step of encapsulating the plurality of metadata into a predetermined wire format—each encapsulated metadata includes a location or sequence associated with each of the plurality of media segments. The method comprises the step of processing the plurality of media segments in parallel based on the encapsulated metadata. After parallel processing, the method further comprises the step of merging the plurality of media segments into a media stream.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application is based on and claims priority to U.S. Patent Application No. 63 / 298,922 filed January 12, 2022 and U.S. Patent Application No. 17 / 991,530 filed November 21, 2022, the entire contents of which are incorporated herein by reference.

[0003] Technology field

[0004] The present disclosure generally relates to wire formats, and more specifically to methods and apparatus for wire formats for segmented media metadata for parallel processing in a cloud platform. Background Technology

[0005] The Network-Based Media Processing (NBMP) framework defines interfaces, including both data formats and Application Programming Interfaces (APIs), between entities connected via digital networks for media processing. The NBMP standard defines a set of tools for the independent processing of media segments. The framework enables the dynamic generation of media processing pipelines, as well as access to processed media data and metadata in real-time or deferred manner. Network and cloud platforms are utilized to run various applications. While metadata parameters are defined, the NBMP standard does not define interoperable wire formats for segment metadata.

[0006] The following is a simplified overview of one or more embodiments of the present disclosure to provide a basic understanding of such embodiments. This overview is not an extensive overview of all embodiments considered, nor is it intended to identify the essential or important elements of all embodiments, nor is it intended to describe the scope of any or all embodiments. Its sole purpose is to present some concepts of one or more embodiments of the present disclosure in a simplified form as an introduction to the more detailed description to be presented later.

[0007] According to an exemplary embodiment, a method performed by at least one processor comprises the step of segmenting a media stream into a plurality of media segments in a multidimensional space. The method comprises the step of determining each of the plurality of media segments and their associated metadata. The method comprises the step of encapsulating the plurality of metadata into a predetermined wire format—each encapsulated metadata including a location or sequence associated with each of the plurality of media segments. The method comprises the step of processing the plurality of media segments in parallel based on the encapsulated metadata. After parallel processing, the method further comprises the step of merging the plurality of media segments into a media stream.

[0008] According to an exemplary embodiment, the device comprises at least one memory configured to store computer program code, and at least one processor configured to read the computer program code and operate as directed by the computer program code, wherein the computer program code comprises segmentation code configured to cause at least one processor to segment a media stream into a plurality of media segments in a multidimensional space. The computer program code comprises determination code configured to cause at least one processor to determine each metadata associated with each of the plurality of media segments. The computer program code comprises encapsulation code configured to cause at least one processor to encapsulate the plurality of metadata into a predetermined wire format—each encapsulated metadata comprising a location or sequence associated with each of the plurality of media segments. The computer program code comprises parallel processing code configured to cause at least one processor to process the plurality of media segments in parallel based on the encapsulated metadata. The computer program code further comprises merging code configured to cause at least one processor to merge the plurality of media segments into a media stream after parallel processing.

[0009] According to an exemplary embodiment, a non-transient computer-readable medium stores instructions, and the instructions, when executed by a processor, cause the processor to execute a method comprising the step of segmenting a media stream into a plurality of media segments in a multidimensional space. The method comprises the step of determining each of the plurality of media segments and its associated metadata. The method comprises the step of encapsulating the plurality of metadata into a predetermined wire format—each encapsulated metadata comprising a location or sequence associated with each of the plurality of media segments. The method comprises the step of parallelizing the plurality of media segments based on the encapsulated metadata. After parallelizing, the method further comprises the step of merging the plurality of media segments into a media stream.

[0010] Methods, apparatuses, and non-transient computer-readable media for wire formats for segmented media metadata for parallel processing in a cloud platform are disclosed by the present disclosure.

[0011] Additional embodiments will be disclosed in the following description, will be partially apparent from the description, and / or may become known through the practice of the embodiments presented in this disclosure. Brief explanation of the drawing

[0012] The above and other aspects, features, and aspects of the embodiments of the present disclosure will become apparent from the following description taken together with the accompanying drawings. FIG. 1 is a diagram of an exemplary network-based media processing (NBMP) architecture according to various embodiments of the present disclosure. FIG. 2 is a drawing of a splitter and merger template in an NBMP architecture according to various embodiments of the present disclosure. FIG. 3 is an example of segment location metadata as a JSON (JavaScript Object Notation) object according to various embodiments of the present disclosure. FIG. 4 is an example of segment sequence metadata as a JSON object according to various embodiments of the present disclosure. FIG. 5 is a flowchart of an exemplary process for segmenting and processing a media stream using a wire format for media stream metadata according to various embodiments of the present disclosure. FIG. 6 illustrates an exemplary computer system according to various embodiments of the present disclosure. Specific details for implementing the invention

[0013] The following detailed description of exemplary embodiments refers to the accompanying drawings. Identical reference numbers in different drawings may identify identical or similar elements.

[0014] The foregoing disclosure is provided for illustrative purposes only and is not intended to be comprehensive or to limit implementations to the exact form disclosed. Modifications and variations may be possible in light of the foregoing disclosure or may be obtained from the practice of implementations. Additionally, one or more features or components of one embodiment may be included in or combined with other embodiments (or one or more features of other embodiments). Furthermore, it will be understood that in the flowcharts and descriptions of operations provided below, one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least partially), and the order of one or more operations may be changed.

[0015] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual special control hardware or software code used to implement these systems and / or methods does not limit the implementations. Accordingly, the operation and behavior of the systems and / or methods have been described herein without reference to specific software code, and it is understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0016] Specific combinations of features are cited in the claims and / or disclosed in the specification, but such combinations are not intended to limit the disclosure of possible implementations. In practice, many of these features may be combined in ways not specifically cited in the claims and / or disclosed in the specification. Each dependent claim listed below may depend directly on only one claim, but the disclosure of possible implementations includes each dependent claim in combination with all other claims in the set of claims.

[0017] Any element, action, or instruction used in this specification shall not be interpreted as important or essential unless explicitly stated so. Additionally, as used in this specification, articles (“a” and “an”) are intended to include one or more items and may be used interchangeably with “one or more.” If only one item is intended, the term “one” or similar language is used. Also, as used in this specification, terms such as “has,” “have,” “having,” “include,” and “including” are intended to be open-ended terms. Additionally, the phrase “based on” is intended to mean “at least partially based on” unless explicitly stated otherwise. Also, expressions such as “at least one of [A] and [B]” or “at least one of [A] or [B]” should be understood to include only A, only B, or both A and B.

[0018] Throughout this specification, references to “one embodiment,” “an embodiment,” or similar language mean that a specific feature, structure, or characteristic described in relation to the indicated embodiment is included in at least one embodiment of the present solution. Accordingly, throughout this specification, phrases such as “in one embodiment,” “in an embodiment,” and similar language may all refer to the same embodiment, but are not necessarily so.

[0019] Additionally, the described features, advantages, and characteristics of the present disclosure may be combined in any suitable manner in one or more embodiments. A person skilled in the art will recognize, in light of the description in this specification, that the present disclosure may be practiced without one or more of the specific features or advantages of a particular embodiment. In other cases, additional features and advantages that may not be present in all embodiments of the present disclosure may be recognized in specific embodiments.

[0020] Embodiments of the present disclosure relate to wire formats for segmented media metadata used in NBMP processing of media segments. FIG. 1 illustrates an embodiment of an NBMP reference architecture (100). The NBMP reference architecture (100) may include a media source (102) capable of providing a media flow (e.g., a media stream) to a media processing entity (MPE) (104). The MPE (104) may also provide the media flow to a media sink (106).

[0021] The reference architecture (100) may further include an NBMP source (108) that provides an NBMP workflow API and a workflow description to an NBMP workflow manager (110). The NBMP workflow manager (110) may receive NBMP tasks from an MPE (104) and provide an MPE API to the MPE (104). The reference architecture (100) may further include a function repository (112) that interacts with the NBMP source (108) and the NBMP workflow manager (110).

[0022] Figure 2 illustrates an example of a splitter and merger function template defined in the NBMP standard. Splitter and merger functions can be used for parallel processing of segments. Media streams can be continuous. However, splitter functions can convert a media stream into N media substreams. Each substream can be processed by an instance of T, and then the substreams are interleaved together to produce an output stream (e.g., an equivalent of task T).

[0023] 1:N splitter and N:1 merger functions can act on segment boundaries. Each segment may have (i) a start, (ii) a duration, and (iii) length metadata, or (i) a start code and (ii) a sequence number associated with the segment. Since such segments are independent, consequently, the substreams are independent of each other in that they are processed by task T. In some embodiments, tasks T0, ...T N-1 There is no need to process segments simultaneously. Because segments and substreams are independent, each instance of the task can be executed at independent rates (for example, each instance of the task can be executed at its own rate). Conventional NBMP standards deal only with the 1-D segmentation of media data.

[0024] According to some embodiments, each segment may use one of the following metadata:

[0025] 1. Location Metadata:

[0026] a. Scaling vector , and Scale factors for,

[0027] b. Starting vector is unit t i Each index s in i Represents the starting point of a media segment in an M-dimensional space having,

[0028] c. Length vector is unit t i Each index d in i It represents the hyperspace of media segments covered in an M-dimensional space having, and

[0029] d. Size of segment L in bytes.

[0030] 2. Or, sequence metadata:

[0031] a. Sequence vector is each index n i Represents a sequence of media segments in an M-dimensional space having,

[0032] b. Start code C, a unique code that all segments start with together, and the code is not repeated in the middle of any segments, and

[0033] c. Size of segment L in bytes.

[0034] The NBMP standard does not define a wire format for the above metadata.

[0035] According to some embodiments, a wire format for metadata of one or more segments may include a byte stream format for location metadata and / or sequence metadata. The byte stream format may include an associated multipurpose internet mail extension (MIME). According to some embodiments, a wire format for metadata of one or more segments may include a JavaScript Object Notation (JSON) format for location metadata and / or sequence metadata. The JSON format may include an associated MIME.

[0036] According to some embodiments, parameters and is defined as follows. is a vector with M dimensions It can be, and element c i It has index i, and index i+1 is nested at index i, which means that a single increment at index i of the vector is considered a larger increment than any increment at indices i+1, i+2, ..., M-1, and 0≤i <M이다.

[0037] A multidimensional segment with dimension M is a point and length It can be defined as a segment representing information about samples within a space starting from, where s i and d i are non-negative integers. If non-integer values ​​are needed, a vector is the scaling factor t for dimension i. i It can be represented as, where the actual starting point and length in dimension i are s, respectively. i / t i and d i / t i and, here t i can be a positive integer.

[0038] Table 1 illustrates an exemplary byte format for location metadata for a single segment.

[0039] Table 1

[0040]

[0041] A table can be repeated for additional segments by connecting one or more identical tables: Table 1, Table 1, ..., Table 1 for multiple segments.

[0042] Table 2 shows an exemplary byte format for sequence metadata of a single segment. In some embodiments, because the start code C is common across all segments (e.g., the same start code across all segments), the start code C is not carried as part of the sequence metadata but has its own input for the function.

[0043] Table 2

[0044]

[0045] The table may be repeated for additional segments by one or more connections of the same Table 2: Table 2, Table 2, ..., Table 2 for multiple segments.

[0046] FIG. 3 illustrates an exemplary JSON object (300) for location metadata for one or more segments. FIG. 4 illustrates an exemplary JSON object (400) for sequence metadata for one or more segments. In some embodiments, since the start code C may be common to all segments, the start code C may not be carried as part of the sequence metadata and has its input to the function.

[0047] Table 3 illustrates exemplary MIME types for each wire format (e.g., byte format and JSON object).

[0048] Table 3

[0049]

[0050] Table 4 shows exemplary extended step descriptor parameters for signaling formats supported by the function in its Function Description Document (FDD). Underlined rows are new parameters.

[0051]

[0052]

[0053]

[0054] FIG. 5 illustrates a flowchart of an embodiment of a process (500) for segmenting and processing a media stream using a wire format for media stream metadata. The process (500) may begin with an operation (502) in which the media stream is segmented into multiple media segments. The process proceeds to an operation (504) in which the metadata associated with each of the multiple media segments is determined. The process proceeds to an operation (506) in which the metadata of each media segment is encapsulated into a predetermined wire format, such as a byte format or a JSON format. The metadata may be positional metadata or sequence metadata. The metadata may include a position or sequence associated with each of the multiple media segments. The process proceeds to an operation (508) in which the multiple media segments are processed in parallel based on the encapsulated metadata. After parallel processing, the process proceeds to an operation (510) in which the multiple media segments are merged into a media stream.

[0055] The technologies of the embodiments of the above-described disclosure may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, FIG. 6 illustrates a computer system (600) suitable for implementing embodiments of the disclosed subject matter.

[0056] Computer software may be coded using any suitable machine code or computer language that may undergo assembly, compilation, linking, or similar mechanisms to generate code containing instructions that can be executed directly or through interpretation, micro-code execution, etc., by computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0057] The commands can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0058] The components of the computer system (600) illustrated in FIG. 6 are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependency or requirement in relation to any one or a combination thereof of the components illustrated in the exemplary embodiments of the computer system (600).

[0059] The computer system (600) may include specific human interface input devices. Such human interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, applause), visual input (e.g., gestures), and olfactory input (not doped). Human interface devices may also be used to capture specific media that are not necessarily directly related to conscious input by a human, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., 2D video, 3D video including stereoscopic video).

[0060] Input human interface devices may include one or more of a keyboard (601), a mouse (602), a trackpad (603), a touch screen (610), a data glove, a joystick (605), a microphone (606), a scanner (607), and a camera (608) (only one of each is shown).

[0061] The computer system (600) may also include specific human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (for example, tactile feedback via a touch-screen (610), data glove, or joystick (605), but there may also be tactile feedback devices that do not function as input devices). For example, such devices may be audio output devices (e.g., speakers (609), headphones (not shown)), visual output devices (e.g., screens (610) including CRT screens, LCD screens, plasma screens, OLED screens, each having or not having touch-screen input capability, each having or not having haptic feedback capability - some of these may output two-dimensional visual output or output of more than three dimensions through means such as stereographic output; virtual reality glasses (not shown), holographic displays and smoke tanks (not shown)), and printers (not shown).

[0062] The computer system (600) may also include human-accessible storage devices and media associated therewith, such as optical media including a CD / DVD ROM / RW (620) having a media (621) such as a CD / DVD, a thumb drive (622), a removable hard drive or solid-state drive (623), legacy magnetic media such as tape and floppy disk (not shown), specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0063] Those skilled in the art should also understand that the term "computer-readable media" as used in connection with the subject matter disclosed herein does not include transmission media, carriers, or other transient signals.

[0064] The computer system (600) may also include an interface to one or more communication networks. The networks may be, for example, wireless, wired, or optical. The networks may also be local, wide-area, metropolitan, automotive and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks, e.g., Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, automotive and industrial networks including CANBus, etc. Certain networks generally require external network interface adapters attached to specific general-purpose data ports or peripheral buses (649) (e.g., USB ports of the computer system (600), etc.). Others are generally integrated into the core of the computer system (600) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). By using any of these networks, the computer system (600) may communicate with other entities. Such communication may be unidirectional, receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., CANbus for specific CANbus devices), or bidirectional to other computer systems using local or wide-area digital networks, for example. Such communication may include communication to a cloud computing environment (655). Specific protocols and protocol stacks may be used in each of the aforementioned networks and network interfaces.

[0065] The aforementioned human interface devices, human-accessible storage devices, and network interfaces (654) can be attached to the core (640) of the computer system (600).

[0066] The core (640) may include one or more central processing units (CPUs) (641), graphics processing units (GPUs) (642), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (643), hardware accelerators (644) for specific tasks, etc. These devices may be connected via a system bus (648), along with internal mass storage (647), such as read-only memory (ROM) (645), random access memory (646), internal non-user accessible hard drives, SSDs, etc. In some computer systems, the system bus (648) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (648) or via a peripheral bus (649). Architectures for peripheral buses include PCI, USB, etc. A graphics adapter (650) may be included in the core (640).

[0067] CPUs (641), GPUs (642), FPGAs (643), and accelerators (644) can be combined to execute specific instructions that can constitute the aforementioned computer code. The computer code may be stored in ROM (645) or RAM (646). Transient data may also be stored in RAM (646), while persistent data may be stored, for example, in internal mass storage (647). High-speed storage and retrieval of any of the memory devices may be made possible through the use of a cache memory that may be closely associated with one or more CPUs (641), GPUs (642), mass storage (647), ROM (645), RAM (646), etc.

[0068] Computer-readable media may have computer code for performing operations implemented by various computers. The media and computer code may be those specifically designed and configured for the purposes of this disclosure, or they may be of a kind well known and available to those skilled in the field of computer software.

[0069] As an example rather than a limitation, a computer system (600) having an architecture, and specifically a core (640), may provide functionality as a result of processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented on one or more types of tangible computer-readable media. Such computer-readable media may be media associated with specific storage of the core (640) that is of a non-transient nature, such as core-internal mass storage (647) or ROM (645), as well as user-accessible mass storage as described above. Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (640). The computer-readable media may include one or more memory devices or chips as needed. Software may enable the core (640) and, specifically, the processors within it (including a CPU, GPU, FPGA, etc.) to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (646) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise implemented in a circuit (e.g., accelerator (644)) that can operate instead of or with the software to execute specific processes or specific parts of specific processes described herein. Reference to software may include logic, where appropriate, and vice versa. Reference to computer-readable media may include, where appropriate, a circuit (integrated circuit (IC), etc.) storing software for execution, a circuit implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0070] The foregoing disclosure is for illustrative purposes only and is not intended to be comprehensive or to limit implementations to the exact form disclosed. Modifications and variations may be possible in light of the foregoing disclosure or may be obtained from the practice of the implementations.

[0071] It is understood that the specific order or hierarchy of blocks in the processes / flow diagrams disclosed herein is an example of exemplary approaches. It is understood that the specific order or hierarchy of blocks in the processes / flow diagrams may be rearranged based on design preferences. Additionally, some blocks may be combined or omitted. The appended method claims present elements of various blocks in a sample order and are not intended to be limited to the specific order or hierarchy presented.

[0072] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail of integration. Additionally, one or more of the aforementioned components may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transient storage medium (or media) having computer-readable program instructions for causing a processor to perform operations.

[0073] A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, but is not limited thereto. A non-comprehensive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), eraseable and programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in which instructions are written, and any suitable combination of the foregoing. A computer-readable storage medium should not be interpreted as being the transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmitting media (e.g., optical pulses passing through fiber optic cables), or electrical signals transmitted through wires, as used herein.

[0074] The computer-readable program instructions described herein may be downloaded to each computing / processing device from an external computer or external storage device or from a computer-readable storage medium via a network, e.g., the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored on a computer-readable storage medium within each computing / processing device.

[0075] Computer-readable program code / instructions for performing operations may be source code or object code written in any combination of one or more programming languages, including assembler instructions, ISA (instruction-set-architecture) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" programming language or similar programming languages. Computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit comprising, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information of computer-readable program instructions to personalize the electronic circuit to perform modes or operations.

[0076] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing unit, and may generate a machine such that instructions executed through the processor of the computer or other programmable data processing unit create means for implementing functions / operations specified in blocks or blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium capable of instructing a computer, a programmable data processing unit, and / or other devices to function in a specific manner, such that the computer-readable storage medium in which the instructions are stored comprises a manufactured article containing instructions that implement modes of functions / operations specified in blocks or blocks of a flowchart and / or block diagram.

[0077] Computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device, so that a series of operational steps are performed on the computer, other programmable device, or other device to create a process implemented by the computer, so that the instructions executed on the computer, other programmable device, or other device implement functions / operations specific to blocks or blocks of a flowchart and / or block diagram.

[0078] The flowcharts and block diagrams in the drawings illustrate the architecture, function, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of instructions containing one or more executable instructions for implementing a specified logical function(s). The method, computer system, and computer-readable media may include additional blocks, fewer blocks, different blocks, or blocks arranged differently from those depicted in the drawings. In some alternative implementations, the functions mentioned in the blocks may occur out of the order mentioned in the drawings. For example, two blocks depicted consecutively may, in fact, be executed simultaneously or substantially simultaneously depending on the related functions, or the blocks may sometimes be executed in reverse order. It should also be noted that each block of the block diagrams and / or flowchart examples, and combinations of blocks within the block diagrams and / or flowchart examples, may be implemented by special-purpose hardware-based systems that perform specified functions or operations or perform combinations of special-purpose hardware and computer instructions.

[0079] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual special control hardware or software code used to implement these systems and / or methods does not limit the implementations. Accordingly, the operation and behavior of the systems and / or methods have been described herein without reference to specific software code, and it is understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0080] The above disclosure also includes the embodiments listed below:

[0081] (1) A method performed by at least one processor, comprising: segmenting a media stream into multiple media segments in a multidimensional space; determining metadata associated with each of the multiple media segments; encapsulating the multiple metadata into a predetermined wire format, wherein each encapsulated metadata includes a location or sequence associated with each of the multiple media segments; processing the multiple media segments in parallel based on the encapsulated metadata; and merging the multiple media segments into a media stream after processing in parallel.

[0082] (2) A method of feature (1), wherein the metadata of each media segment from a plurality of media segments includes location metadata.

[0083] (3) A method of feature (2) wherein a predetermined wire format includes a byte stream format, and one or more of a multidimensional scale vector, a position vector, a length vector, and size information are encapsulated in the byte stream format.

[0084] (4) A method of feature (2) wherein the predetermined wire format is a JSON array, and one or more of the multidimensional scale vector, position vector, length vector, and size information are encapsulated as array elements for multiple media segments.

[0085] (5) A method of feature (1), wherein the metadata of each media segment from a plurality of media segments includes sequence metadata.

[0086] (6) A method of feature (5) wherein a predetermined wire format is a byte stream format, and one or more of a multidimensional sequence vector and size information are encapsulated in the byte stream format.

[0087] (7) A method of feature (5) wherein the predetermined wire format is a JSON array, and one or more of the multidimensional sequence vector and size information are encapsulated as array elements for multiple media segments.

[0088] (8) As a method of feature (1), the predetermined wire format includes a multi-purpose internet mail extension (MIME).

[0089] (9) A device comprising: at least one memory configured to store computer program code; and at least one processor configured to read computer program code and operate as directed by computer program code, wherein the computer program code comprises: a segmentation code configured to cause at least one processor to segment a media stream into a plurality of media segments in a multidimensional space; a determination code configured to cause at least one processor to determine each of the plurality of media segments and their respective metadata; an encapsulation code configured to cause at least one processor to encapsulate the plurality of metadata into a predetermined wire format—each encapsulated metadata includes a location or sequence associated with each of the plurality of media segments—; a parallel processing code configured to cause at least one processor to process the plurality of media segments in parallel based on the encapsulated metadata; and a merging code configured to cause at least one processor to merge the plurality of media segments into a media stream after parallel processing.

[0090] (10) A device of feature (9), wherein the metadata of each media segment from a plurality of media segments includes location metadata.

[0091] (11) A device of feature (10), wherein a predetermined wire format includes a byte stream format, and one or more of a multidimensional scale vector, a position vector, a length vector, and size information are encapsulated in the byte stream format.

[0092] (12) A device of feature (10), wherein the predetermined wire format is a JSON array, and one or more of a multidimensional scale vector, position vector, length vector, and size information are encapsulated as array elements for multiple media segments.

[0093] (13) A device of feature (9), wherein the metadata of each media segment from a plurality of media segments includes sequence metadata.

[0094] (14) A device of feature (13) wherein the predetermined wire format is a byte stream format, and one or more of the multidimensional sequence vector and size information are encapsulated in the byte stream format.

[0095] (15) A device of feature (13), wherein the predetermined wire format is a JSON array, and one or more of the multidimensional sequence vector and size information are encapsulated as array elements for multiple media segments.

[0096] (16) A device of feature (9), wherein the predetermined wire format includes the Multipurpose Internet Mail Extension (MIME).

[0097] (17) A non-transient computer-readable medium for storing instructions, wherein the instructions, when executed by a processor, enable the processor to perform a method comprising: segmenting a media stream into multiple media segments in a multidimensional space; determining each of the multiple media segments with its own metadata; encapsulating the multiple metadata into a predetermined wire format, wherein each encapsulated metadata includes a location or sequence associated with each of the multiple media segments; processing the multiple media segments in parallel based on the encapsulated metadata; and merging the multiple media segments into a media stream after processing in parallel.

[0098] (18) A non-transient computer-readable medium of feature (17), wherein the metadata of each media segment from a plurality of media segments includes location metadata.

[0099] (19) A non-transient computer-readable medium of feature (18), wherein a predetermined wire format includes a byte stream format, and one or more of a multidimensional scale vector, position vector, length vector, and size information are encapsulated in the byte stream format.

[0100] (20) A non-transient computer-readable medium of feature (18), wherein the predetermined wire format is a JSON array, and one or more of a multidimensional scale vector, position vector, length vector, and size information are encapsulated as array elements for multiple media segments.

Claims

Claim 1 A data processing method performed by at least one processor, comprising: segmenting a media stream into a plurality of media segments in a multidimensional space; determining metadata associated with each of the plurality of media segments; encapsulating the plurality of metadata into a predetermined wire format supported by a function in a function description document (FDD) among a plurality of wire formats, wherein each encapsulated metadata includes a position or sequence associated with each of the plurality of media segments; processing the plurality of media segments in parallel based on the encapsulated metadata; and merging the plurality of media segments into the media stream after the parallel processing. Claim 2 A data processing method according to claim 1, wherein the metadata of each media segment from the plurality of media segments includes location metadata. Claim 3 A data processing method according to paragraph 2, wherein the predetermined wire format includes a byte stream format, and one or more of a multidimensional scale vector, a position vector, a length vector, and size information are encapsulated in the byte stream format. Claim 4 A data processing method according to paragraph 2, wherein the predetermined wire format is a JSON array, and one or more of a multidimensional scale vector, a position vector, a length vector, and size information are encapsulated as array elements for the plurality of media segments. Claim 5 A data processing method according to claim 1, wherein the metadata of each media segment from the plurality of media segments includes sequence metadata. Claim 6 A data processing method according to claim 5, wherein the predetermined wire format is a byte stream format, and one or more of a multidimensional sequence vector and size information are encapsulated in the byte stream format. Claim 7 A data processing method according to claim 5, wherein the predetermined wire format is a JSON array, and one or more of a multidimensional sequence vector and size information are encapsulated into array elements for the plurality of media segments. Claim 8 A data processing method according to claim 1, wherein the predetermined wire format includes a Multi-purpose Internet Mail Extension (MIME). Claim 9 A data processing device comprising: at least one memory configured to store computer program code; and at least one processor configured to read said computer program code and operate as directed by said computer program code to perform a data processing method of any one of claims 1 to 8. Claim 10 A non-transient computer-readable medium for storing instructions, wherein the instructions, when executed by a processor, cause the processor to execute a data processing method of any one of claims 1 to 8. Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images