Multidimensional Metadata for Parallel Processing of Segmented Media Data

Abstract

A method executed by at least one processor, comprising: segmenting a multidimensional media stream into a plurality of segments of multidimensional media in a multidimensional space; representing each of the plurality of segments of multidimensional media using a respective sequence vector, each sequence vector including one or more predefined multidimensional metadata, the predefined multidimensional metadata including one of a start vector, a length vector, and a scaling vector, and a start code; and deriving a network based media processing (NBMP) workflow based on the respective sequence vector of each of the plurality of segments.

Description

[Technical field]

[0001] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 298,534, filed January 11, 2022, and U.S. Patent Application No. 17 / 993,583, filed November 23, 2022, the disclosures of which are incorporated by reference in their entireties herein.

[0002] This disclosure provides extensions to one-dimensional NBMP segment metadata to support splitting and merging multi-dimensional media data into segments, each of which can be processed independently. [Background technology]

[0003] Networks and cloud platforms are used to run a variety of applications. The network-based media processing (NBMP) standard defines a set of tools for independently processing one-dimensional media segments. However, when media is multi-dimensional, there may be a need to segment media data into multiple dimensions, which is currently not addressed by the NBMP standard. Summary of the Invention

[0004] According to some embodiments, a method is provided that is executed by at least one processor. The method includes segmenting a multidimensional media stream into a plurality of segments of multidimensional media in a multidimensional space. The method further includes representing each of the plurality of segments of the multidimensional media with a respective sequence vector, each sequence vector including one or more predefined multidimensional metadata, the predefined multidimensional metadata including one of: (1) a start vector, a length vector, and a scaling vector, and (2) a start code. The method further includes deriving a network based media processing (NBMP) workflow based on the respective sequence vector of each of the plurality of segments.

[0005] According to some embodiments, an apparatus includes at least one memory configured to store program code and at least one processor configured to read and operate as instructed by the program code. The program code includes segmentation code configured to cause the at least one processor to segment a multidimensional media stream into a plurality of segments of multidimensional media in a multidimensional space. The program code further includes representation code configured to cause the at least one processor to represent each segment of the plurality of segments of multidimensional media using a respective sequence vector, each sequence vector including one or more predefined multidimensional metadata, the predefined multidimensional metadata including one of: (1) a start vector, a length vector, and a scaling vector, and (2) a start code. The program code further includes derivation code configured to cause the at least one processor to derive a network based media processing (NBMP) workflow based on the respective sequence vector of each segment of the plurality of segments. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform a step of segmenting a multidimensional media stream into a plurality of segments of multidimensional media in a multidimensional space. The instructions further cause the at least one processor to perform a step of representing each segment of the plurality of segments of multidimensional media using a respective sequence vector, each sequence vector including one or more predefined multidimensional metadata, the predefined multidimensional metadata including one of: (1) a start vector, a length vector, and a scaling vector, and (2) a start code.The instructions further cause the at least one processor to derive a network based media processing (NBMP) workflow based on the respective sequence vector of each segment of the plurality of segments. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram of an environment in which the methods, apparatus, and systems described herein may be implemented, according to some embodiments.

[0007] [Diagram 2] A block diagram of example components of one or more devices according to some embodiments.

[0008] [Diagram 3] FIG. 1 is a block diagram of an NBMP system, according to some embodiments.

[0009] [Figure 4] FIG. 1 illustrates a workflow diagram for parallel processing of segments, according to some embodiments.

[0010] [Diagram 5] FIG. 2 is a block diagram of an apparatus for processing media content in the NBMP according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements, and in which:

[0012] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Moreover, one or more features or components of one embodiment may be incorporated in or combined with another embodiment (or one or more features of another embodiment). Additionally, in the flowcharts and operational descriptions provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least in part), and the order of one or more operations may be switched.

[0013] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, although the operation and behavior of the systems and / or methods are described herein without reference to specific software code, it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0014] Although particular combinations of features are set forth in the claims and / or disclosed in the specification, these combinations do not limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically set forth in the claims and / or disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

[0015] No element, act, or instruction used herein should be construed as critical or essential unless expressly stated. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." When only one item is intended, the term "one" or similar language is used. Also, as used herein, terms such as "has," "have," "having," "include," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood as including only A, only B, or both A and B.

[0016] References throughout this specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in connection with the illustrated embodiment is included in at least one embodiment of the solution. Thus, "in one embodiment," "in an embodiment," and similar language throughout this specification may, but need not, all refer to the same embodiment.

[0017] Furthermore, the described features, advantages, and characteristics of the present disclosure may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize in light of the description herein that the present disclosure may be practiced without one or more of the specific features or advantages of a particular embodiment. In other cases, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the present disclosure.

[0018] Figure 1 is a diagram of an environment 100 in which the methods, apparatus, and systems described herein may be implemented, according to an embodiment. As shown in Figure 1, the environment 100 may include a user device 110, a platform 120, and a network 130. The devices of the environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0019] User device 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information related to platform 120. For example, user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or a similar device. In some implementations, user device 110 may receive information from platform 120 and / or transmit information to platform 120.

[0020] Platform 120 includes one or more devices, as described elsewhere herein. In some implementations, platform 120 may include a cloud server or a group of cloud servers. In some implementations, platform 120 may be designed to be modularized, such that software components may be swapped in or out depending on particular needs. As such, platform 120 may be easily and / or quickly reconfigured for different uses.

[0021] In some implementations, as shown, platform 120 may be hosted in a cloud computing environment 122. In particular, although the implementations described herein describe platform 120 as being hosted within cloud computing environment 122, in some implementations platform 120 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0022] Cloud computing environment 122 includes an environment that hosts platform 120. Cloud computing environment 122 may provide services, such as computing, software, data access, storage, etc., that do not require end-user (e.g., user device 100) knowledge of the physical location or configuration of the systems and / or devices that host platform 120. As shown, cloud computing environment 122 may include a group of computing resources 124 (collectively referred to as “computing resources 124” or individually referred to as “computing resource 124”).

[0023] Computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, computing resources 124 may host platform 120. Cloud resources may include computing instances running on computing resources 124, storage devices provided on computing resources 124, data transfer devices provided by computing resources 124, etc. In some implementations, computing resources 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0024] As further shown in FIG. 1, the computational resources 124 include a group of cloud resources, such as one or more applications (“APP”) 124-1, one or more virtual machines (“VM”) 124-2, virtualized storage (“VS”) 124-3, and one or more hypervisors (“HYP”) 124-4.

[0025] Application 124-1 includes one or more software applications that may be provided to or accessed by user device 110 and / or platform 120. Application 124-1 may eliminate the need to install and run a software application on user device 110. For example, application 124-1 may include software associated with platform 120 and / or any other software that may be provided via cloud computing environment 122. In some implementations, one application 124-1 may send information to / receive information from one or more other applications 124-1 via virtual machine 124-2.

[0026] Virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 124-2 may be either a system virtual machine or a processing virtual machine, depending on the extent to which virtual machine 124-2 uses and corresponds to any real machine. A system virtual machine may provide a complete system platform that supports the execution of a complete operating system ("OS"). A processing virtual machine may execute a single program and support a single process. In some implementations, virtual machine 124-2 may execute on behalf of a user (e.g., user device 110) and manage the infrastructure of cloud computing environment 122, such as data management, synchronization, long duration data transfer, etc.

[0027] Virtualized storage 124-3 includes one or more storage systems and / or one or more devices that use virtualization techniques within the storage systems or devices of computing resources 124. In some implementations, within the context of storage systems, types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage so that the storage system can be accessed regardless of the physical storage or heterogeneous structure. This separation may allow storage system administrators flexibility in how they manage storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and where the file is physically stored. This may allow for optimization of performance of storage usage, server consolidation, and / or ongoing file migration.

[0028] Hypervisor 124-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer, such as computer resources 124. Hypervisor 124-4 may present a virtual operating platform to the guest operating systems and may manage the execution of the guest operating systems. Multiple instances of different operating systems may share virtualized hardware resources.

[0029] Network 130 may include one or more wired and / or wireless networks. For example, network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad-hoc network, an intranet, the Internet, an optical fiber-based network, etc., and / or a combination of these or other types of networks.

[0030] The number and arrangement of devices and networks shown in Figure 1 are provided as an example. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different arrangements of devices and / or networks than those shown in Figure 1. Furthermore, two or more devices shown in Figure 1 may be implemented within a single device, or a single device shown in Figure 1 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) of environment 100 may perform one or more functions that are described as being performed by another set of devices of environment 100.

[0031] Figure 2 is a block diagram of example components of one or more devices of Figure 1. Device 200 may correspond to user device 110 and / or platform 120. As shown in Figure 2, device 200 may include a bus 210, a processor 220, a memory 230, a storage component 240, an input component 250, an output component 260, and a communication interface 270.

[0032] The bus 210 includes components that enable communication between the components of the device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 220 includes one or more processors that can be programmed to perform functions. The memory 230 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 220.

[0033] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact disks (CDs), digital versatile disks (DVDs), floppy disks, cartridges, magnetic tapes, and / or other types of non-transitory computer-readable media along with corresponding drives.

[0034] Input components 250 include components that enable device 200 to receive information via user input (e.g., a touchscreen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 260 include components that provide output information from device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0035] Communications interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable device 200 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communications interface 270 may enable device 200 to receive information from another device and / or provide information to another device. For example, communications interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0036] Device 200 may perform one or more operations described herein. Device 200 may perform these operations in response to processor 220 executing software instructions stored by a non-transitory computer-readable medium, such as memory 230 and / or storage component 240. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0037] Software instructions may be read into memory 230 and / or storage component 240 from another computer-readable medium or from another device via communications interface 270. When executed, the software instructions stored in memory 230 and / or storage component 240 may cause processor 220 to perform one or more operations described herein. Additionally or alternatively, hardwired circuitry may be used in place of, or in combination with, software instructions to perform one or more operations described herein. Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0038] The number and arrangement of components shown in Figure 2 is provided as an example. In practice, device 200 may include additional, fewer, different, or differently arranged components than those shown in Figure 2. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions that are described as being performed by another set of components of device 200.

[0039] In one embodiment of the present disclosure, there is provided an NBMP system 300. Referring to Figure 3, the NBMP system 300 includes an NBMP source 310, an NBMP workflow manager 320, a capability repository 330, one or more media processing entities 350, a media source 360, and a media sink 370.

[0040] NBMP source 310 may receive instructions from a third party entity, may communicate with NBMP workflow manager 320 via NBMP workflow, and may communicate with capability repository 330 via capability discovery API 391. For example, NBMP source 310 may send a workflow description document (WDD) to NBMP workflow manager 320, or may read capability descriptions of functions stored in capability repository 330, such as media processing functions stored in the memory of capability repository 330, such as media decoding, feature point extraction, camera parameter extraction, projection method, seam information extraction, blending, post-processing, and encoding functions. NBMP source 310 may include or be implemented by at least one processor and a memory storing code configured to cause at least the processor to perform the functions of NBMP source 310.

[0041] An NBMP source 310 may request the NBMP workflow manager 320 to create a workflow that includes tasks 352 to be performed by one or more media processing entities 350 by sending a workflow description document, which may include multiple descriptors, and each descriptor may have multiple parameters.

[0042] For example, the NBMP source 310 may select a function stored in the function repository 330 and send a workflow description document to the NBMP workflow manager 320, which includes various descriptors for descriptive details such as input and output data, required functionality, and workflow requirements. The workflow description document may include a set of task descriptions of tasks 352 to be performed by one or more media processing entities 350, and a connection map of inputs and outputs. When the NBMP workflow manager 320 receives such information from the NBMP source 310, the NBMP workflow manager 320 may create a workflow by instantiating tasks based on the function names and connecting the tasks according to the connection map.

[0043] Alternatively or additionally, NBMP source 310 may request NBMP workflow manager 320 to create a workflow by using a set of keywords. For example, NBMP source 310 may send NBMP workflow manager 320 a workflow description document that may include a set of keywords that NBMP workflow manager 320 may use to find appropriate functions stored in function repository 330. When NBMP workflow manager 320 receives such information from NBMP source 310, NBMP workflow manager 320 may create a workflow by searching for appropriate functions using keywords that may be specified in process descriptors of the workflow description document and may use other descriptors in the workflow description document to provide tasks and connect them to create the workflow.

[0044] NBMP workflow manager 320 may communicate with capability repository 330 via capability discovery API 393, which may be the same or a different API as capability discovery API 391, and may communicate with one or more of media processing entities 350 via an API 394 (e.g., an NBMP task API). NBMP workflow manager 320 may include or be implemented by at least one processor and a memory storing code configured to cause at least the processor to perform the functions of NBMP workflow manager 320.

[0045] The NBMP workflow manager 320 may use the API 394 to set up, configure, manage, and monitor one or more tasks 352 of a workflow that may be performed by one or more media processing entities 350. In some embodiments, the NBMP workflow manager 320 may use the API 394 to update and destroy tasks 352. To set up, manage, and monitor tasks 352 of a workflow, the NBMP workflow manager 320 may send messages, such as requests, to one or more media processing entities 350, where each message may have multiple descriptors and each of the descriptors may have multiple parameters. The tasks 352 may each include a media processing function 354 and a configuration 353 for the media processing function 354.

[0046] In some embodiments, after receiving a workflow description document from the NBMP source 310 that does not include a list of tasks (e.g., includes a list of keywords instead of a list of tasks), the NBMP workflow manager 320 may select a task based on a description of the task in the workflow description document and search the function repository 330, via function discovery API 393, to find an appropriate function to be performed as a task 352 of the current workflow. For example, the NBMP workflow manager 320 may select a task based on keywords provided in the workflow description document. After an appropriate function is identified using a set of keywords or task descriptions provided by the NBMP source 310, the NBMP workflow manager 320 may use API 394 to configure the selected task in the workflow. For example, the NBMP workflow manager 320 may extract configuration data from the information received from the NBMP source and configure the task 352 based on the configuration data.

[0047] The one or more media processing entities 350 may be configured to receive media content from a media source 360, process the media content according to a workflow including tasks 352 created by the NBMP workflow manager 320, and output the processed media content to a media sink 370. The one or more media processing entities 350 may include or be implemented by at least one processor and a memory storing code configured to cause at least the processor to perform functions of the media processing entity 350.

[0048] Media source 360 ​​may include memory for storing media and may be integral with or separate from NBMP source 310. In some embodiments, NBMP workflow manager 320 may notify NBMP source 310 when a workflow is prepared, and media source 360 ​​may send media content to one or more of media processing entities 350 based on the notification that a workflow is prepared.

[0049] The media sink 370 may include or be implemented by at least one processor and at least one display configured to display media processed by the one or more media processing entities 350.

[0050] As discussed above, messages from the NBMP source 310 to the NBMP workflow manager 320 (e.g., a workflow description document to request the creation of a workflow) and messages from the NBMP workflow manager 320 to one or more media processing entities 350 (e.g., messages to cause a workflow to occur) may include multiple descriptors, each of which may have multiple parameters. In some cases, communications between any of the components of the NBMP system 300 using APIs may include multiple descriptors, each of which may have multiple parameters.

[0051] Embodiments may relate to a method for identifying and signaling non-mandatory inputs, outputs, and tasks in a workflow executed on a cloud platform.

[0052] A required output of a workflow may be an output of that workflow that must produce data for the workflow to be considered to be running properly. A required input of a workflow may be an input that must be processed in order for the workflow to create its required output. A properly running workflow may be a workflow that processes all of its required inputs and produces all of its required outputs. A required task of a workflow may be a task that is necessary to run properly and process data required for the workflow to run properly. For example, a required task may be a task that processes a required input and / or produces a required output. In some embodiments, a required input may be an input that is necessary for a required task to run, and a required output may be an output that is required as a required input of a required task, or an output that is required as an output for the workflow as a whole. A non-required input may be an input that is not required by the workflow to produce a required output of a workflow. For example, a workflow may produce all of the required outputs if all of the required inputs are provided, even if no non-required inputs are provided. A non-required task may be a task that is included in a workflow, but is not a required task. For example, a non-required task may process a non-required input and produce a non-required output.

[0053] The NBMP standard defines splitter and merger function templates. Figure 4 shows a diagram of a workflow 400 corresponding to one example of this process using the NBMP splitter / merge function for parallel processing of segments. In Figure 4, task T 410 may be transformed into n instances of task T that run in parallel.

[0054] In some embodiments, the media stream is contiguous. Splitter 420 may convert the media stream into N media substreams. Each substream may be processed by an instance of T430A-C, and then the substreams are interleaved together in merger 440 to produce output 450 equivalent to the output stream of task T410. The 1:N splitter and N:1 merge functions operate on segment boundaries. Each segment may have start, duration, and length metadata, or a start code and sequence number associated with it. Because the segments are independent, the substreams are independent from each other in that they are processed by task T410. Tasks T0, ..., T N-1 The segments do not need to be processed simultaneously. Because the segments and substreams may be independent, each instance of task T410 may execute at its own rate. However, the current NBMP standard only addresses 1D segmentation of media data.

[0055] The current NBMP standard defines the following format for segment metadata: Each segment meets the following requirements: A. Consecutive sets of samples B. The maximum duration of D on a time scale T (D and T are configuration parameters)

[0056] Each segment may use one of the following metadata: 1. Timing Metadata A. Start time s on time scale t B. Time scale t=T C. Duration d on time scale t D. Length l (bytes) 2. Or sequence metadata A. The same unique start code for all other segments B. Sequence numbers in increasing order

[0057] In both cases, the media is segmented only in one dimension (e.g., typically time). However, media signals tend to be multi-dimensional.

[0058] Embodiments of the present disclosure include multi-dimensional metadata for multi-dimensional segment processing using the following. 1. Start and length 2. Start code and sequence vector

[0059] The following definitions may be used. - C is an M-dimensional vector [c0, c1,..., c M-1 , and the element c i has an index i, and the index i + 1 is nested within the index i, which means that an increment of 1 in the index i of the vector is considered a larger increment than any increment of the indices i + 1, i + 2,..., M - 1 (0 ≤ i < M). - A multi-dimensional segment having dimension M is defined as a segment representing information about samples in a space starting at the point S = [s0, s1,..., s M-1 and length D = [d0, d1,.., d M-1 , where s i and d i are non-negative integers. If non-integer values are required, the vector T = [t0, t1,..., t M-1 represents the scale factor t i for dimension i, and the actual starting point and length in dimension i are s i / t i and d i / t i respectively, where t i is a positive integer. - Scaling vector T = [t0, t1,..., t M-1 , S and D scale factors for - Unit t i at each index s iA start vector representing the starting point of a media segment in an M-dimensional space, having S =[s0, s1,..., s M-1 - Unit t i For each index d i A length vector representing the hyperspace covered by a media segment in an M-dimensional space, having D =[d0, d1,..., d M-1 - The size of segment L in bytes

[0060] Parameter s i And d i May be non-negative integers, and t i Is a positive integer. The segments may be ordered in increasing order as defined by the following. - A segment K having the start vector [s K,0 ,..., s K,M-1 is before a segment L having the start point [s L,0 ,..., s L,M-1 in the following cases. o For at least one i (0 ≤ i < M), s K,i < s L,i . o For any 0 ≤ j < i, s K,j ≤ s L,j . - The increasing order set of segments is a 1D sequence of segments where each segment is located after all other segments that are before it in the sequence.

[0061] Each segment may carry the following metadata. - t , s , d , L

[0062] As follows, the following metadata may be defined for each segment. - Each having index n i ​​A sequence vector representing a sequence of media segments in an M-dimensional space having n =[n0,n1,...,n M-1 . - A start code C, which is a unique code at which all segments start and which is not repeated anywhere within a segment. - The size of segment L in bytes.

[0063] Parameter s i and d i may be non-negative integers, and t i may be a positive integer. The segments may be ordered in increasing order as defined by the following. - A segment K having the sequence vector [n K,0 ,...,n K,M-1 is before a segment L having the starting point [n L,0 ,...,n L,M-1 if o For at least one i (0 ≦ i < M), n K,i < n L,i . o For any 0 ≦ j < i, n K,j ≦ n L,j . - In an increasing order set of segments, each segment is a 1D sequence of segments such that each segment is located after all other segments in the sequence that are before it.

[0064] Each segment may carry the following metadata. - n , C, L

[0065] In the case of M = 1 (e.g., one-dimensional), the above design may be reduced to existing 1D metadata. a. Segment time metadata: t, s, d, L b. Segment start code: n, c, L

[0066] 5 is a flow chart of an example process 500 for signaling multidimensional metadata. In some implementations, one or more of the process blocks of FIG. 5 may be performed by any of the elements described above, such as, for example, NBMP system 300, or any element included in that system, such as NBMP workflow manager 320.

[0067] As shown in FIG. 5, process 500 includes segmenting a multi-dimensional media stream into a sequence of segments (act 510).

[0068] As further shown in FIG. 5, process 500 may include deriving a network-based media processing workflow based on the sequence of the plurality of segments (act 520).

[0069] As further shown in FIG. 5, process 500 may include adding multidimensional metadata to the sequence of segments of the multidimensional media (act 530).

[0070] In an embodiment, operation 530 may further include expressing a starting point for each segment of the plurality of segments using multi-dimensional metadata.

[0071] In an embodiment, the process 500 may further include expressing the position of each segment in the hyperspace using a start vector, a length vector, and a scaling vector associated with the length vector and the start vector.

[0072] In an embodiment, the process 500 may further include ordering the sequences of the multiple segments in increasing order of the sequence vector using rules defined by the multi-dimensional metadata.

[0073] In an embodiment, operation 530 may further include representing each segment of the plurality of segments with the same unique code point, where each segment includes the same unique code point and where each segment starts with the same unique code point.

[0074] In an embodiment, the process 500 may further include ordering the plurality of segments based on the sequence vector representation of the plurality of segments.

[0075] In an embodiment, the process 500 may further include ordering the multiple segments in increasing order of the sequence vector based on a rule defined by the sequence vector representation.

[0076] Figure 5 illustrates example blocks of process 500, and in some implementations process 500 may include additional, fewer, different, or differently arranged blocks than those illustrated in Figure 5. Additionally or alternatively, two or more of the blocks of process 500 may be performed in parallel.

[0077] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0078] As used herein, the term component is intended to be interpreted broadly as hardware, firmware, or a combination of hardware and software.

[0079] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, although the operation and behavior of the systems and / or methods are described herein without reference to specific software code, it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0080] Although combinations of features are set forth in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically set forth in the claims and / or disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in a claim set. No element, act, or instruction used herein should be construed as critical or essential unless expressly recited. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Additionally, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more." When only one item is intended, the term "one" or similar language is used. Also, as used herein, the terms "has," "have," "having," and the like are intended to be open-ended terms. Further, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise.

Claims

1. 1. A method executed by at least one processor, comprising: segmenting the multi-dimensional media stream into a plurality of segments of multi-dimensional media in a multi-dimensional space; representing each segment of the plurality of segments of the multidimensional media with a respective sequence vector, the respective sequence vector including one or more predefined multidimensional metadata, the predefined multidimensional metadata including one of: (1) a start vector, a length vector, and a scaling vector; and (2) a start code; and deriving a network based media processing (NBMP) workflow based on the respective sequence vectors of each segment of the plurality of segments.

2. The method of claim 1 , further comprising ordering the plurality of segments in increasing order of sequence vectors using rules defined by the multi-dimensional metadata.

3. 2. The method of claim 1, further comprising expressing each segment of the plurality of segments with a same unique code point, each segment including the same unique code point, and each segment starting with the same unique code point.

4. The method of claim 3 , further comprising ordering the plurality of segments based on a sequence vector representation of the plurality of segments.

5. The method of claim 4 , further comprising ordering the plurality of segments in increasing order of sequence vectors based on a rule defined by the sequence vector representation.

6. An apparatus comprising: at least one memory configured to store program code; and at least one processor configured to read and execute said program code to perform the method of any one of claims 1 to 5.

7. A program which, when executed by at least one processor, causes the at least one processor to perform the method according to any one of claims 1 to 5.

Images