Method and apparatus for scheduling network-based media processing workflows on a cloud platform

Abstract

A method for scheduling a network based media processing (NBMP) workflow on a cloud platform is provided. The method includes obtaining an input workflow including an input media stream, generating a modified workflow by partitioning the input media stream into one or more tasks, scheduling the one or more tasks on the cloud platform based on a scheduling descriptor, the scheduling descriptor including scheduling type information, and processing the modified workflow based on the scheduling of the one or more tasks.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 218,815, filed July 6, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to the field of media processing, and more specifically to an apparatus and method for scheduling a network-based media processing (NBMP) workflow on a cloud platform, a computer device, and a non-transitory computer-readable storage medium. Background Technology

[0004] Networks and cloud platforms can be used to run a variety of applications. The Network-Based Media Processing (NBMP) standard provides specifications for defining, instantiating, and running workflows on cloud platforms. In some cases, workflows can be run in parts, task-by-task, or in groups of tasks at a time. In such cases, real-time processing of the workflow may not be necessary, and scheduling may be required to account for the limited computing resources allocated to the workflow or to avoid peak traffic times on the cloud. However, the current functionality of the NBMP standard does not provide a method for scheduling workflows or portions of workflows on cloud platforms.

[0005] Therefore, the technical problem to be solved by this application is how to provide a method for scheduling a workflow or a part of a workflow on a cloud platform. Summary of the Invention

[0006] One or more exemplary embodiments of this disclosure provide methods and apparatus for scheduling workflows or portions of workflows on a cloud platform.

[0007] According to one aspect of this disclosure, a method is provided for scheduling a network-based media processing (NBMP) workflow on a cloud platform, the method comprising: acquiring an input workflow, the input workflow including an input media stream; generating a modified workflow by dividing the input media stream into one or more tasks; scheduling one or more tasks on the cloud platform based on a scheduling descriptor, the scheduling descriptor including scheduling type information; and processing the modified workflow based on the scheduling of one or more tasks.

[0008] According to another aspect of the present invention, an apparatus is provided for scheduling a network-based media processing (NBMP) workflow on a cloud platform, the apparatus comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising: acquisition code configured to cause the at least one processor to acquire an input workflow, the input workflow including an input media stream; generation code configured to cause the at least one processor to generate a modified workflow by dividing the input media stream into one or more tasks; scheduling code configured to cause the at least one processor to schedule one or more tasks on the cloud platform based on a scheduling descriptor, the scheduling descriptor including scheduling type information; and processing code configured to cause the at least one processor to process the modified workflow based on the scheduling of one or more tasks.

[0009] According to another aspect of the present invention, an apparatus is provided for scheduling a network-based media processing (NBMP) workflow on a cloud platform. The apparatus includes: an acquisition unit configured to cause at least one processor to acquire an input workflow, the input workflow including an input media stream; a generation unit configured to cause the at least one processor to generate a modified workflow by dividing the input media stream into one or more tasks; a scheduling unit configured to cause the at least one processor to schedule the one or more tasks on the cloud platform based on a scheduling descriptor, the scheduling descriptor including scheduling type information; and a processing unit configured to cause the at least one processor to process the modified workflow based on the scheduling of the one or more tasks.

[0010] According to another aspect of the present invention, a computer device is provided, including a processor and a memory. The memory is used to store program code and transmit the program code to the processor; the processor is used to execute, according to instructions in the program code: acquiring an input workflow, the input workflow including an input media stream; generating a modified workflow by dividing the input media stream into one or more tasks; scheduling one or more tasks on a cloud platform based on a scheduling descriptor, the scheduling descriptor including scheduling type information; and processing the modified workflow based on the scheduling of one or more tasks.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable medium storing instructions including one or more of the following, which, when executed by at least one processor of an apparatus for scheduling a network-based media processing (NBMP) workflow on a cloud platform, cause the at least one processor to: acquire an input workflow, the input workflow including an input media stream; generate a modified workflow by dividing the input media stream into one or more tasks; schedule one or more tasks on the cloud platform based on a scheduling descriptor, the scheduling descriptor including scheduling type information; and process the modified workflow based on the scheduling of one or more tasks.

[0012] The method and apparatus for scheduling a Network-Based Media Processing (NBMP) workflow on a cloud platform according to embodiments of this application first acquire an input workflow, the input workflow including an input media stream; then, a modified workflow is generated by dividing the input media stream into one or more tasks; next, one or more tasks are scheduled on the cloud platform based on a scheduling descriptor, the scheduling descriptor including scheduling type information; and finally, the modified workflow is processed based on the scheduling of one or more tasks. Through this method, the NBMP standard can be extended to support the scheduling of workflows or portions of workflows.

[0013] Additional aspects will be set forth in part in the description which follows, and in part will become apparent from the description, or may be realized by practice of the embodiments proposed in this disclosure. Attached Figure Description

[0014] The above and other aspects, features, and embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0015] Figure 1 It is a schematic diagram of a communication system according to one or more embodiments.

[0016] Figure 2 This is a simplified example diagram of a streaming environment according to one or more implementation methods.

[0017] Figure 3 This is a block diagram of an NBMP system according to one or more implementations.

[0018] Figure 4 This is an example diagram of NBMP splitter and merger processing according to one or more implementations.

[0019] Figure 5This is a block diagram of an example process for scheduling a network-based media processing (NBMP) workflow or a portion thereof on a cloud platform, according to one or more implementations.

[0020] Figure 6 It is a block diagram of an example of computer code for scheduling a network-based media processing (NBMP) workflow or a portion thereof on a cloud platform, according to one or more implementations. Detailed Implementation

[0021] Embodiments of this disclosure relate to methods and apparatus for signaling modifications in a network-based media processing (NBMP) workflow, and more particularly, to apparatus and methods for performing step-based operations in media processing on a cloud platform.

[0022] The embodiments of this disclosure are exemplary in nature and will be fully described with reference to the accompanying drawings. However, examples of implementations can be implemented in a variety of forms and should not be construed as limiting this disclosure to the examples described herein. Rather, examples of implementations are provided to make the technical solutions of this disclosure more comprehensive and complete, and to fully convey the ideas of the examples of implementations to those skilled in the art. The accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of parts are omitted.

[0023] The proposed features discussed below can be used individually or in any order. Some block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. Furthermore, implementations can be: implemented in hardware, which may include processing circuitry systems (e.g., one or more processors or one or more integrated circuits) or microcontroller devices; implemented in software; implemented in different network and / or processor devices; or implemented by a combination of hardware and software components. In one example, one or more processors execute computer program code stored on one or more non-transitory computer-readable media.

[0024] Figure 1 This is a diagram of an environment 100 in which the methods, apparatus, and systems described herein can be implemented according to an embodiment. For example... Figure 1 As shown, environment 100 may include user equipment 110, platform 120, and network 130. The devices in environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0025] User equipment 110 may include one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with platform 120. For example, user equipment 110 may include computing devices (e.g., desktop computers, laptop computers, tablet computers, handheld computers, smart speakers, servers, etc.), mobile phones (e.g., smartphones, cordless phones, etc.), wearable devices (e.g., smart glasses or smartwatches), or similar devices. In some implementations, user equipment 110 may receive information from platform 120 and / or transmit information to platform 120.

[0026] Platform 120 may include one or more devices as described elsewhere herein. In some implementations, platform 120 may include a cloud server or a group of cloud servers. In some implementations, platform 120 may be designed to be modular, allowing software components to be swapped in or out as needed. Therefore, platform 120 can be easily and / or quickly reconfigured for different purposes.

[0027] In some implementations, as shown, platform 120 may be hosted in a cloud computing environment 122. It is worth noting that while the implementations described herein depict platform 120 as hosted in a cloud computing environment 122, in some implementations, platform 120 may not be cloud-based (i.e., it may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0028] The cloud computing environment 122 may include the environment of the hosting platform 120. The cloud computing environment 122 can provide services such as computing, software, data access, and storage, which do not require end users (e.g., user equipment 110) to know the physical location and configuration of the systems and / or devices of the hosting platform 120. As shown, the cloud computing environment 122 may include a set of computing resources 124 (collectively referred to as "computing resources 124" and individually as "computingresource 124").

[0029] Computing resource 124 may include one or more personal computers, workstations, server devices, or other types of computing and / or communication devices. In some implementations, computing resource 124 may host platform 120. Cloud resources may include: computing instances executing in computing resource 124, storage devices provided in computing resource 124, data transmission devices provided by computing resource 124, etc. In some implementations, computing resource 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0030] like Figure 1 As further shown, computing resource 124 may include a set of cloud resources, such as one or more applications (“Application, APP”) 124-1, one or more virtual machines (“Virtual Machine, VM”) 124-2, virtualized storage devices (“Virtualized Storage, VS”) 124-3, one or more hypervisors (“Hypervisor, HYP”) 124-4, etc. However, this disclosure is not limited thereto, and therefore, according to other example embodiments, computing resource 124 may include other types of cloud resources.

[0031] Application 124-1 may include one or more software applications that can be provided to or accessed by user equipment 110 and / or platform 120. Application 124-1 may eliminate the need to install and execute software applications on user equipment 110. For example, application 124-1 may include software associated with platform 120 and / or any other software that can be provided via cloud computing environment 122. In some implementations, an application 124-1 may send information to or receive information from one or more other applications 124-1 via virtual machine 124-2.

[0032] Virtual machine 124-2 may include a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 124-2 may be a system virtual machine or a process virtual machine, depending on the extent to which virtual machine 124-2 uses and corresponds to any real machine. A system virtual machine may be a complete system platform that supports the execution of a complete operating system (“OS”). A process virtual machine may execute a single program and may support a single process. In some implementations, virtual machine 124-2 may execute on behalf of a user (e.g., user device 110) and may manage the infrastructure of the cloud computing environment 122, such as data management, synchronization, or long-duration data transfer.

[0033] Virtualized storage device 124-3 may include one or more storage systems and / or one or more devices that utilize virtualization technology within the storage system or device of computing resource 124. In some implementations, within the context of the storage system, the type of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the extraction (or separation) of logical storage from physical storage, enabling access to the storage system regardless of physical storage or heterogeneous architecture. Separation can allow storage system administrators greater flexibility in managing storage for end users. File virtualization can eliminate the dependency between data accessed at the file level and the location where the file is physically stored. This can enable performance optimization for storage usage, server consolidation, and / or non-disruptive file migration.

[0034] Hypervisor 124-4 can provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer such as computing resource 124. Hypervisor 124-4 can present a virtual operating platform to the guest operating system and manage the execution of the guest operating system. Multiple instances of various operating systems can share virtualized hardware resources.

[0035] Network 130 may include one or more wired and / or wireless networks. For example, network 130 may include cellular networks (e.g., fifth-generation (5G) networks, long-term evolution (LTE) networks, third-generation (3G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMN), local area networks (LAN), wide area networks (WAN), metropolitan area networks (MAN), telephone networks (e.g., public switched telephone networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-optic networks, etc., and / or combinations of these or other types of networks.

[0036] Figure 1 The number and arrangement of devices and networks shown are provided as examples. In practice, with Figure 1Compared to the devices and / or networks shown, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks arranged differently. Furthermore, Figure 1 The two or more devices shown can be implemented within a single device, or Figure 1 The single device shown can be implemented as multiple distributed devices. Additionally or alternatively, a group of devices in environment 100 (e.g., one or more devices) can perform one or more functions described as being performed by another group of devices in environment 100.

[0037] Figure 2 yes Figure 1 A block diagram of example components of one or more devices. Device 200 may correspond to user device 110 and / or platform 120. Figure 2 As shown, device 200 may include bus 210, processor 220, memory 230, storage unit 240, input unit 250, output unit 260 and communication interface 270.

[0038] Bus 210 includes components that allow communication between parts of device 200. Processor 220 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 220 may be a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Accelerated Processing Unit (APU), microprocessor, microcontroller, Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), or another type of processing unit. In some implementations, processor 220 may include one or more processors that can be programmed to perform functions. Memory 230 may include Random Access Memory (RAM), Read Only Memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 220.

[0039] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state drives), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, and / or other types of non-transitory computer-readable media and corresponding drives.

[0040] Input component 250 may include components that allow device 200 to receive information, for example, via user input (e.g., a touchscreen display, keyboard, keypad, mouse, button, switch, and / or microphone). Alternatively or additionally, input component 250 may include sensors for sensing information (e.g., a Global Positioning System (GPS) component, accelerometer, gyroscope, and / or actuator). Output component 260 includes components that provide output information from device 200 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0041] Communication interface 270 may include transceiver-like components (e.g., transceiver and / or separate receiver and transmitter) that enable device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 270 may allow device 200 to receive information from and / or provide information to another device. For example, communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0042] Device 200 can perform one or more of the processes described herein. Device 200 can perform these processes in response to processor 220 executing software instructions stored in non-transitory computer-readable media such as memory 230 and / or storage unit 240. Computer-readable media are defined herein as non-transitory memory devices. Memory devices include memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0043] Software instructions can be read into memory 230 and / or storage unit 240 from another computer-readable medium or from another device via communication interface 270. When executed, the software instructions stored in memory 230 and / or storage unit 240 can cause processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry can be used in place of or in combination with the software instructions to perform one or more processes described herein. Therefore, the implementations described herein are not limited to any particular combination of hardware circuitry and software.

[0044] Figure 2 The number and arrangement of components shown are provided as an example. In practice, with Figure 2 Compared to the components shown, device 200 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a set of components of device 200 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 200.

[0045] In embodiments of this disclosure, a network-based media processing (NBMP) system is provided. Figure 3 An NBMP architecture 300 according to an embodiment described herein is illustrated and can be implemented relative to cloud processing. The NBMP system 300 includes an NBMP source 310, an NBMP workflow manager 320, a function store 330, one or more Media Processing Entities (MPEs) 340, a media source 350, and a media sink 360. The NBMP source 310, NBMP workflow manager 320, function store 330, MPE 340, media source 350, and media sink 360 may include, or be implemented by, at least one or more processors and memory, wherein the memory stores code configured to cause at least one or more processors to respectively execute the functions of the NBMP source 310, NBMP workflow manager 320, function store 330, MPE 340, media source 350, and media sink 360.

[0046] NBMP source 310 can communicate workflow descriptions with NBMP workflow manager 320 via NBMP Workflow API (Application Programming Interface, API) 311. NBMP source 310 can also communicate function descriptions with function store 330 via function discovery API 313. For example, NBMP source 310 can send a Workflow Description Document (WDD) to NBMP workflow manager 320 and can read function descriptions of functions stored in function store 330, which are media processing functions stored in the memory of function store 330, such as media decoding, feature point extraction, camera device parameter extraction, projection methods, seam information extraction, mixing, post-processing, and encoding functions. NBMP workflow manager 320 can communicate with function store 330 via function discovery API 312, which may be the same as or different from function discovery API 313, and can communicate with one or more MPEs 340 via API 314 (e.g., MPE API).

[0047] Media Processing Entity (MPE) 340 may include one or more tasks 341. NBMP Workflow Manager 320 may also communicate with tasks 341 via API 315 (e.g., the NBMP Task API). NBMP Workflow Manager 320 can use API 315 to set up, configure, manage, and control one or more tasks 341 that can execute workflows by one or more MPEs 340. To configure, manage, and control the tasks 341 of the workflow, NBMP Workflow Manager 320 may send messages such as requests to one or more of the MPEs 340 and / or tasks 341, where each message may have several descriptors, each of which has several parameters. Additionally, communication between NBMP source 310, NBMP Workflow Manager 320, function store 330, and MPE 340 can be considered a control flow.

[0048] Task 341 may each include a media processing function 343 and a configuration 342 for the media processing function 343. Each of the tasks 341 in the corresponding media processing entity 340 may also communicate with each other to facilitate data flow between tasks. In an implementation, the NBMP workflow manager 320 may select a task based on the task description in the WDD to search the function store 330 via the function discovery API 312 to find the appropriate function to run as task 341 for the current workflow. One or more MPEs 340 may be configured to receive media content from a media source 350, process the media content according to a workflow including task 341 created by the NBMP workflow manager 320, and output the processed media content to a media receiver 360. In an implementation, one or more MPEs 340 may be provided in parallel for multiple media streams 316 and 317 between the media source 350 and the media receiver 360.

[0049] Media source 350 may include a memory for storing media and may be integrated with or separate from NBMP source 310. In one implementation, NBMP workflow manager 320 may notify NBMP source 310 when a workflow is ready, and media source 350 may transmit media content to one or more MPEs 340 based on the workflow ready notification, and one or more MPEs 340 may transmit media content to media receiver 360. Communication between media source 350, MPEs 340, and media receiver 360 can be considered as a data stream.

[0050] Figure 4 An example of NBMP segmenting and merging media stream 440 of task T for parallel processing, according to an embodiment of this disclosure, is shown. Figure 4 As shown, the NBMP splitter and merger process 400 includes a splitter 420 and a merger 430. In Figure 4 In this process, the task T that inputs the media stream is transformed into n instances of task T (i.e., T0, ..., Tn). N-1 The task T consists of n instances that run in parallel. The input media stream 410 can be continuous. The splitter 420 converts the media stream into N media sub-streams. Each sub-stream is processed by an instance of T. The merger 430 then interweaves the sub-streams to produce an output 450. The output 450 is equivalent to the output stream of task T (i.e., media stream 440).

[0051] like Figure 4As shown, the 1:N splitter 420 and N:1 merger 430 function with respect to segment boundaries. Each segment has metadata associated with it, including start, duration, and length. Because the segments are independent, the substreams are independent of each other in terms of being processed by task T. In the implementation, tasks T0, ..., T... N-1 It is an instance of task T, and segments can be processed simultaneously or individually. Since segments and substreams are independent, each instance of task T (i.e., T0, ..., T1) is a separate instance of task T. N-1 It can run at its own speed.

[0052] According to embodiments, a method for scheduling the execution of a workflow or a portion thereof may be provided. This method may include scheduling time for processing the entire workflow or scheduling time for processing tasks within the workflow. According to embodiments, the workflow may be divided into multiple tasks (i.e., subprocesses), and each of the multiple tasks in the workflow may be scheduled using any one (or combination of) the following schemes: based on sequence, based on duration, based on events, and based on time slots. However, this disclosure is not limited thereto, and therefore, another scheme may be provided for scheduling tasks (i.e., subprocesses) of a workflow.

[0053] According to one implementation, each of a plurality of tasks can be scheduled sequentially. In this case, each task is run or processed completely once, followed by the next task. For example, the method may include scheduling to execute a first task of a plurality of tasks completely, and scheduling to execute a second task of a plurality of tasks after the first task is completed. That is, each of the plurality of tasks is scheduled to be executed one after another after the previous tasks are completed. Although the implementation describes executing the second task after the first task is completed, this disclosure is not limited thereto. Therefore, according to another implementation, the second task (or subsequent task) may begin before the first task is completed. In this case, parallel processing of the tasks can be achieved.

[0054] According to the implementation, each of a plurality of tasks can be scheduled based on duration. In this case, each task in the workflow runs for the duration of input or output. For example, the method may include scheduling to execute a first task of a plurality of tasks for a certain period of time, and scheduling to execute a second task of a plurality of tasks for a certain period of time after executing the first task for the certain period of time. Here, the execution of the first and second tasks may correspond to using an input media stream or generating an output media stream.

[0055] According to one embodiment, executing the first task for a certain period of time may correspond to using a 1-minute input media stream or generating a 1-minute output media stream. Subsequently, executing the second task for a certain period of time may correspond to using a 1-minute input media stream or generating a 1-minute output media stream. Although 1 minute is shown as a duration, this disclosure is not limited thereto, and therefore, according to another embodiment, the duration may be a different amount of time (i.e., 5 minutes of input or output). Furthermore, according to another embodiment, the durations scheduled for executing the first and second tasks may be different. For example, the first task may be scheduled to execute for a first period of time, and the second task may be scheduled to execute for a second period of time. According to this embodiment, parallel processing of the first and second tasks can be achieved.

[0056] According to one implementation, each of a plurality of tasks can be scheduled based on an event. In this case, each task in the workflow runs with a start event and stops with a stop event. For example, the method may include scheduling the execution of the first task of a plurality of tasks based on the occurrence of a first scene of a movie.

[0057] According to one implementation, each of a plurality of tasks can be scheduled based on a time slot. In this case, each task in the workflow begins according to a specific scheduling time. For example, the method may include scheduling the first of a plurality of tasks to be executed between 2:00 AM and 3:00 AM.

[0058] According to the implementation, the schemes described above can be run for a set of tasks (or task groups). However, this disclosure is not limited thereto, and therefore, according to the implementation, some of these schemes can be run for the entire workflow. For example, the entire workflow can be scheduled based on duration, events, or time slots.

[0059] A detailed description of each element in the scheduling scheme is provided below:

[0060] Sequence-based scheduling

[0061] Depending on the implementation, the workflow manager may have to run or execute each of the multiple tasks in the workflow sequentially. In this case, the workflow manager may move in a wave-like manner towards the output, starting the execution of the first task of the workflow with the input media stream, generating the output media stream, buffering the output, and then starting the next task. Alternatively, the workflow manager may start the execution of tasks one after another.

[0062] According to one embodiment, the scheduling of task order can be performed by any of a plurality of components in the NBMP system in one of a variety of ways. According to another embodiment, the NBMP client defines the order of tasks to be executed. According to yet another embodiment, the workflow manager predetermines the order. However, this disclosure is not limited thereto, and therefore, the order of tasks to be executed can be defined by another component.

[0063] In both scenarios described above, the workflow manager needs to reconfigure the workflow by adding a cache / store for each output of the task. The workflow manager either needs information about the maximum size of these caches / stores, or it needs to provide the size of each input so that the workflow manager can estimate the size of the caches / stores.

[0064] According to the implementation, the "Completed" input may be useful because each task that receives a "True" flag at its "Completed" input knows that its task is completed and can generate an event for the workflow manager indicating when it is time for the next task.

[0065] According to the implementation method, the NBMP client and NBMP workflow manager may have the following requirements.

[0066] Depending on the implementation, the NBMP client may need to define or indicate / identify the scheduling scheme mode (i.e., one of sequence, duration, event, and time slot). Furthermore, in the case of duration-based scheduling, the NBMP client may optionally define the order of tasks. Additionally, the NBMP client may need to define or indicate / identify the maximum size of storage for each output or the maximum size of inputs in the workflow.

[0067] Depending on the implementation, the NBMP workflow manager may need to indicate or signal its capability regarding scheduling scheme modes (i.e., sequence, duration, event, and time slot). For example, the NBMP workflow manager may need to indicate whether the NBMP workflow supports scheduling scheme modes (i.e., sequence, duration, event, and time slot). In the case of sequence-based scheduling, the NBMP workflow must indicate whether it can derive the sequence and / or whether it can derive the required buffer size from the input size. Furthermore, the NBMP workflow may need to confirm whether it can run requests to perform tasks based on the sequence scheme mode.

[0068] Duration-based scheduling

[0069] According to one implementation, the workflow manager may need to run or execute each of multiple tasks in the workflow based on duration. Here, the method may include scheduling to execute a first task of the multiple tasks for a first duration, and scheduling to execute a second task of the multiple tasks for a second duration after executing the first task for a certain period. According to one implementation, the first duration and the second duration may be the same. According to another implementation, the first duration and the second duration may be different. According to one implementation, the duration may be a 5-minute input media stream or an output media stream.

[0070] In this scenario, the workflow manager can proceed in a wave-like fashion towards the output, starting the execution of the first task in the workflow that begins with the input media stream, generating the output media stream, buffering the output, and then starting the next task. Alternatively, the workflow manager can start the execution of tasks one after another. Depending on the implementation, the duration of each task scheduling can be based on either the input or output media stream. For example, the first task can be executed until 5 minutes of input media stream has been processed or 5 minutes of output media stream has been generated.

[0071] According to one implementation, the scheduling of task durations can be performed by any of several components in the NBMP system in one of several ways. According to one implementation, the NBMP client defines the duration allocated to the tasks to be executed. According to another implementation, the workflow manager estimates the duration.

[0072] In both scenarios described above, the workflow manager needs to reconfigure the workflow by adding a cache / store for each output of the task. The workflow manager either needs information about the maximum size of the cache / store for a specific duration, or it needs to provide the size of each input so that the workflow manager can estimate the size of the cache / store.

[0073] According to the implementation method, the NBMP client and NBMP workflow manager may have the following requirements.

[0074] Depending on the implementation, the NBMP client may need to define or indicate / identify the scheduling scheme mode (i.e., sequence, duration, event, and time slot). Furthermore, in the case of a duration-based scheduling scheme, the NBMP client may define or indicate / identify the duration of the input (i.e., the input media stream) or the required duration of the output (i.e., the output media stream), the order of tasks (if needed or provided), the maximum storage size for each output in the workflow, or the maximum size of the input.

[0075] Depending on the implementation, the NBMP workflow manager may need to indicate or signal its capabilities regarding scheduling scheme modes (i.e., sequence, duration, event, and time slot). For example, the NBMP workflow manager may need to indicate whether the NBMP workflow supports scheduling scheme modes (i.e., sequence, duration, event, and time slot). In the case of duration-based scheduling, the NBMP workflow must indicate whether it can derive the sequence and / or whether it can derive the required buffer size from the input size. Furthermore, the NBMP workflow must indicate whether it can derive the duration of the input required to generate an output of a given duration. Additionally, the NBMP workflow may need to confirm whether it can run requests to perform tasks based on the sequence scheme mode.

[0076] Event-based scheduling

[0077] According to one implementation, the workflow manager may need to run or execute each of multiple tasks in the workflow based on events. Here, the method may include triggering the start and stop of each task based on events. According to one implementation, an event may be a specific scene in a movie, i.e., the first scene in the movie, or a specific part of music. According to one implementation, the event is published by the workflow manager. Since any task has a "running" state, the workflow manager can start a task by changing its state to "running". To pause a task, the workflow manager can change its state to "idle". Furthermore, since "idle" does not distinguish between a lack of input and being requested to remain "idle" by the workflow manager, according to one implementation, a new state, the "paused" state, is provided, in which the task can be changed from "running" to remaining idle even if input is available.

[0078] According to the implementation, the workflow manager can either schedule events based on its own internal logic or receive events from NBMP clients. Subsequently, also in this scheme, the output buffer / storage for each task should be managed.

[0079] According to one implementation, the scheduling of event-triggered tasks can be performed by any of several components in the NBMP system in one of a variety of ways. According to another implementation, if the event is driven by an NBMP client, the NBMP client limits the maximum size of the intermediate cache / store. According to yet another implementation, if the workflow manager schedules its own events to trigger the start and / or stop of processing tasks, the workflow manager presupposes this size.

[0080] In both scenarios described above, the workflow manager may need to reconfigure the workflow by adding a cache / store for each output of the task. The workflow manager either needs information about the maximum size of the cache / store for a specific duration, or it needs to provide the size of each input so that the workflow manager can estimate the size of the cache / store.

[0081] According to the implementation method, the NBMP client, NBMP workflow manager, and NBMP task may have the following requirements.

[0082] Depending on the implementation, the NBMP client may need to define or indicate / identify the scheduling scheme mode (i.e., sequence, duration, event, and time slot). Furthermore, if the scheme mode is event-based scheduling, the NBMP client may define or indicate / identify whether the event is driven by the NBMP client or by a custom signal from the workflow manager used to initiate its own event scheme. Additionally, if the event is driven by the NBMP client, the NBMP client may define or indicate / identify the maximum size of storage used for each output in the workflow.

[0083] Depending on the implementation, the NBMP workflow manager may need to indicate or signal its capabilities regarding scheduling scheme modes (i.e., sequence, duration, event, and time slot). For example, the NBMP workflow manager may need to indicate whether the NBMP workflow supports scheduling scheme modes (i.e., sequence, duration, event, and time slot). In the case of event-based scheduling, the NBMP workflow must indicate whether it supports workflow-driven events. Furthermore, the NBMP workflow may need to confirm whether it can run requests to perform tasks based on a sequence scheme mode.

[0084] Depending on the implementation, the NBMP workflow manager may need to support a "paused" state and support changing the state between a "running" state and a "paused" state.

[0085] Slot-based scheduling

[0086] According to the implementation, the NBMP client can schedule each of the multiple tasks in a workflow based on time slots, and the workflow manager runs each task according to the given time slot. Here, the time slot could be from 2 AM to 3 AM, executing the first task among multiple tasks.

[0087] According to the implementation, task scheduling can be performed by any of the multiple components in the NBMP system in one of a variety of ways. According to the implementation, since intermediate buffers / storage need to be allocated, the NBMP client can limit the size of the buffer.

[0088] According to the implementation method, the NBMP client and NBMP workflow manager may have the following requirements.

[0089] Depending on the implementation, the NBMP client may need to define or indicate / identify the scheduling scheme pattern (i.e., sequence, duration, event, and time slot). Furthermore, when the scheme pattern is time slot-based scheduling, the NBMP client can define or indicate / identify the scheduling pattern for each task. For example, the NBMP client can use an explicit scheduling pattern where the scheduling has a specific time, or the NBMP client can use an indexed scheduling pattern that provides start, end, increment, and duration schemes (such as for loops).

[0090] Furthermore, when driven by an NBMP client, the NBMP client can limit or indicate / identify the maximum size of storage used for each output in the workflow.

[0091] Depending on the implementation, the NBMP workflow manager may need to indicate or signal its capabilities regarding scheduling scheme modes (i.e., sequence, duration, event, and time slot). For example, the NBMP workflow manager may need to indicate whether the NBMP workflow supports scheduling scheme modes (i.e., sequence, duration, event, and time slot). In the case of time slot-based scheduling, the NBMP workflow must indicate whether it supports a scheduling mode (i.e., explicit scheduling or indexed scheduling). Furthermore, the NBMP workflow may need to confirm whether it can run requests to perform tasks based on a sequence scheme mode.

[0092] Based on the above description, the NBMP client, NBMP workflow manager, and NBMP task have been shown to perform specific operations. However, this disclosure is not limited to these components performing the novel aspects of this disclosure. Therefore, according to other embodiments, other components (or combinations of components) in the NBMP system can perform the operations discussed above in this disclosure.

[0093] According to the example implementation, the scheduling descriptor can be added to the NBMP standard specification in the following manner:

[0094] Table 1—Schedule Descriptors

[0095] Parameter name type cardinality id P 1 describe P 0-1 Scheduling type P 0-1 order string array 0-1 Duration P 0-1 Number of segments P 0-1 io logo P 0-1 Operating mode P 0-1 Time slot O 0-1 situation P 1

[0096] According to one implementation, the scheduling descriptor may indicate the parameter name, type, and cardinality. According to another implementation, the parameter name may include an identifier (id), description, scheduling type, sequence, duration, number of segments, I / O flags, operating mode, time slot, and status. However, this disclosure is not limited to the parameters listed in Table 1, and therefore, according to another implementation, additional parameters may be provided to perform the scheduling of tasks in the workflow.

[0097] Furthermore, according to the example implementation, the time slot object in the scheduling descriptor in Table 1 may include the following parameters shown in Table 2:

[0098] Table 2—Time Slot Objects

[0099] Parameter name type cardinality model P 1 Explicit array of objects 0-1 start P 0-1 Finish P 0-1 Increment P 0-1 Duration P 0-1 Time scale P 0-1

[0100] Furthermore, according to the example implementation, the explicit objects in the time slot objects in Table 2 may include the following parameters shown in Table 3:

[0101] Table 3—Explicit Objects

[0102] Parameter name type cardinality start P 1 Finish P 1

[0103] Furthermore, according to the example implementation, a detailed definition of the scheduling descriptor parameters for the NBMP standard specification can be provided in the following manner:

[0104] Table 4—Schedule Descriptor Parameters

[0105]

[0106]

[0107] According to the implementation, the scheduling descriptor can be used in the following ways. For example, an NBMP client can include a scheduling descriptor in a Workflow Description Document (WDD) update call for any of the following:

[0108] • The capabilities of the workflow manager are obtained by including the scheduler descriptor and "status" = "capability" in the WDD;

[0109] • Schedule workflows by including scheduling descriptors in WDD;

[0110] • Schedule tasks by including a scheduling descriptor in the Task Description Document (TDD);

[0111] • Schedule a group of tasks by including a scheduler descriptor in the task group object; and

[0112] • Consider scheduling any of the above by making a request with "Status" = "Consideration" to see if the workflow can manage scheduling when requesting a scheduling descriptor.

[0113] However, this disclosure is not limited to the use of schedule descriptors as discussed above. Therefore, according to another example implementation, schedule descriptors can be used by the NBMP system in another manner.

[0114] Figure 5 This is a flowchart of an example method 500 for scheduling a network-based media processing (NBMP) workflow on a cloud platform, according to one or more implementations.

[0115] In some implementations, Figure 5 One or more processing blocks can be executed by platform 120. In some implementations, Figure 5 One or more processing blocks may be executed by another device or a group of devices, such as user equipment 110, that is separate from or includes platform 120.

[0116] like Figure 5 As shown, in operation 510, method 500 includes acquiring a workflow. The workflow may include an input media stream.

[0117] In operation 520, method 500 includes generating a modified workflow by dividing the input media stream into one or more tasks.

[0118] In operation 530, method 500 includes scheduling one or more tasks on a cloud platform based on a scheduling descriptor that includes scheduling type information. According to an implementation, the scheduling descriptor includes command information corresponding to the scheduling type information, and wherein the command information includes information for: obtaining the capabilities of the cloud platform; determining whether the scheduling type information in the scheduling request is feasible; and / or providing a response including the result of determining whether the scheduling type information in the scheduling request is feasible.

[0119] According to an implementation, the scheduling type information includes one of sequence information, duration information, event information, or time slot information. According to an implementation, based on the determination that the scheduling type information is sequence information, scheduling is performed to execute each of one or more tasks sequentially. According to an implementation, based on the determination that the scheduling type information is duration information, scheduling is performed to execute a first task among one or more tasks for a first duration. According to an implementation, based on the determination that the scheduling type information is event information, scheduling is performed to execute the first task among one or more tasks based on the occurrence of an event in the input media stream as a trigger to start or stop the execution of the first task. When executing one or more tasks based on event information, a new pause state is defined for each of the one or more tasks, in which each of the one or more tasks maintains its internal state and information without processing any additional data in the corresponding task. According to an implementation, based on the determination that the scheduling type information is time slot information, scheduling is performed to execute the first task among one or more tasks in a specified time slot. The time slot information includes: an explicit scheduling mode, which provides a precise time frame for scheduling; or an indexed scheduling mode, which provides a start, end, increment, and duration scheme for scheduling.

[0120] In operation 540, method 500 includes processing the modified workflow based on the scheduling of one or more tasks.

[0121] although Figure 5 An example block of the method is shown, but in some implementations, the method may include... Figure 5 The blocks described herein are compared to additional blocks, fewer blocks, different blocks, or blocks arranged differently. Alternatively or concurrently, two or more blocks in a method may be executed in parallel.

[0122] Figure 6 This is a block diagram of an example of computer code for scheduling a network-based media processing (NBMP) workflow on a cloud platform, according to one or more implementations.

[0123] According to embodiments of this disclosure, at least one processor may be provided having a memory storing computer code. The computer code may be configured to execute any number of aspects of this disclosure when executed by the at least one processor.

[0124] For example, refer to Figure 6 Computer code 600 can be implemented in NBMP system 300.

[0125] like Figure 6As shown, computer code 600 may include acquisition code 610, generation code 620, scheduling code 630, and processing code 640.

[0126] The acquisition code 610 may include code configured to enable at least one processor to acquire an input workflow that includes an input media stream.

[0127] The generated code 620 may include code configured to enable at least one processor to generate a modified workflow by dividing the input media stream into one or more tasks;

[0128] The scheduling code 630 may include code configured to enable at least one processor to schedule one or more tasks on a cloud platform based on a scheduling descriptor that includes scheduling type information.

[0129] Processing code 640 may include code configured to cause at least one processor to process the modified workflow based on the scheduling of one or more tasks.

[0130] although Figure 6 Example blocks of computer code 600 for an apparatus or device according to an embodiment are shown, but in some implementations, the apparatus may include... Figure 6 The blocks depicted herein are compared to additional blocks, fewer blocks, different blocks, or blocks arranged differently. Alternatively or additionally, two or more blocks in the device can be combined.

[0131] This disclosure also provides an apparatus for scheduling a network-based media processing (NBMP) workflow on a cloud platform. The apparatus includes: an acquisition unit configured to cause at least one processor to acquire an input workflow, the input workflow including an input media stream; a generation unit configured to cause the at least one processor to generate a modified workflow by dividing the input media stream into one or more tasks; a scheduling unit configured to cause the at least one processor to schedule the one or more tasks on the cloud platform based on a scheduling descriptor, the scheduling descriptor including scheduling type information; and a processing unit configured to cause the at least one processor to process the modified workflow based on the scheduling of the one or more tasks.

[0132] In some examples, the scheduling type information includes one of sequence information, duration information, event information, or time slot information.

[0133] In some examples, the scheduling unit is also configured to cause the at least one processor to schedule each of the one or more tasks to be executed sequentially when the scheduling type information is determined to be the sequence information.

[0134] In some examples, the scheduling unit is also configured to cause the at least one processor to schedule the execution of a first task among the one or more tasks for a first duration when the scheduling type information is determined to be the duration information.

[0135] In some examples, the scheduling unit is also configured to cause the at least one processor to schedule the execution of the first task among the one or more tasks based on an event occurring in the input media stream that triggers the start or stop of the first task when the scheduling type information is determined to be the event information.

[0136] In some examples, when performing one or more tasks based on the event information, a new pause state is defined for each of the one or more tasks, in which each of the one or more tasks retains its internal state and information without processing any additional data in the corresponding task.

[0137] In some examples, the scheduling unit is also configured to cause the at least one processor to schedule, upon determining that the scheduling type information is the time slot information, to execute the first of the one or more tasks in a specified time slot.

[0138] In some examples, the time slot information further includes: an explicit scheduling mode, in which a precise time frame is provided for the scheduling; or an indexed scheduling mode, in which a start, end, increment, and duration scheme is provided for the scheduling.

[0139] In some examples, the scheduling descriptor includes command information corresponding to the scheduling type information, wherein the command information includes information for performing the following operations: obtaining the capabilities of the cloud platform; determining whether the scheduling type information in the scheduling request can be implemented; or providing a response including the result of determining whether the scheduling type information in the scheduling request can be implemented.

[0140] This disclosure also provides a computer device including a processor and a memory. The memory is used to store program code and transmit the program code to the processor; the processor is used to execute, according to instructions in the program code: acquiring an input workflow, the input workflow including an input media stream; generating a modified workflow by dividing the input media stream into one or more tasks; scheduling one or more tasks on a cloud platform based on a scheduling descriptor, the scheduling descriptor including scheduling type information; and processing the modified workflow based on the scheduling of one or more tasks.

[0141] The techniques described above for scheduling network-based media processing (NBMP) workflows on a cloud platform can be used individually or in any combination in any order. Furthermore, each of the methods (or implementations) can be implemented using a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors can execute a program stored on a non-transitory computer-readable medium.

[0142] This disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementation to the exact forms disclosed. Modifications and variations can be made based on this disclosure, or from the practice of implementing the methods.

[0143] As used in this article, the term component is intended to be interpreted broadly as hardware, firmware, or a combination of hardware and software.

[0144] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited in its implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the descriptions herein.

[0145] Even if combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways that are not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly refer to only one claim, the disclosure of possible implementations includes every dependent claim combined with every other claim in the claim set.

[0146] Unless explicitly stated otherwise, elements, actions, or instructions used herein should not be construed as critical or necessary. Furthermore, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Additionally, as used herein, the term “group” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with “one or more.” The term “one” or similar language is used where only one item is intended. Moreover, as used herein, the terms “has,” “have,” “having,” etc., are intended to be open-ended terms. Furthermore, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.”

Claims

1. A method for scheduling a network based media processing (NBMP) workflow on a cloud platform, the method comprising: obtaining an input workflow, the input workflow comprising an input media stream; generating a modified workflow by dividing the input media stream into one or more tasks; scheduling the one or more tasks on the cloud platform based on a scheduling descriptor containing scheduling type information, the scheduling descriptor indicating parameter names comprising an identifier, a description, a scheduling type, an order, a duration, a segment number, an io flag, a run mode, a time slot, and a condition, wherein the time slot comprises a mode, an explicit, a start, an end, an increment, a duration, and a time scale, the explicit being an object array, the explicit comprising a start and an end; and processing the modified workflow based on the scheduling of the one or more tasks.

2. The method of claim 1, wherein, the scheduling type information comprises one of order information, duration information, event information, or time slot information.

3. The method of claim 2, further comprising: upon determining that the scheduling type information is the order information, scheduling to execute each of the one or more tasks in order.

4. The method of claim 2, further comprising: upon determining that the scheduling type information is the duration information, scheduling to execute a first task of the one or more tasks for a first duration.

5. The method of claim 2, further comprising: upon determining that the scheduling type information is the event information, scheduling to execute a first task of the one or more tasks based on an event occurring in the input media stream for triggering to start or stop the first task.

6. The method of claim 5, wherein, upon executing the one or more tasks based on the event information, defining a new paused state for each of the one or more tasks in which each of the one or more tasks holds an internal state and information without processing any further data in the respective task.

7. The method of claim 2, further comprising: upon determining that the scheduling type information is the time slot information, scheduling to execute a first task of the one or more tasks in a specified time slot.

8. The method of claim 7, wherein, the time slot information further comprises: an explicit scheduling mode in which an exact time frame is provided for the scheduling; or an indexed scheduling mode in which a start, an end, an increment, a duration scheme is provided for the scheduling.

9. The method according to any one of claims 1 to 8, characterized in that, the scheduling descriptor comprises command information corresponding to the scheduling type information, and wherein the command information comprises information for: obtaining capabilities of the cloud platform; determining whether the scheduling type information in a scheduling request is achievable; or providing a response comprising a result of determining whether the scheduling type information in the scheduling request is achievable.

10. An apparatus for scheduling a network based media processing (NBMP) workflow on a cloud platform, the apparatus comprising: an obtaining unit configured to cause the at least one processor to obtain an input workflow, the input workflow comprising an input media stream; a generating unit configured to cause the at least one processor to generate a modified workflow by dividing the input media stream into one or more tasks; a scheduling unit configured to cause the at least one processor to schedule the one or more tasks on the cloud platform based on a scheduling descriptor containing scheduling type information, the scheduling descriptor indicating parameter names comprising an identifier, a description, a scheduling type, an order, a duration, a segment number, an io flag, a run mode, a time slot, and a status, wherein the time slot comprises a mode, an explicit, a start, an end, an increment, a duration, and a time scale, the explicit being an object array, the explicit comprising a start and an end; and a processing unit configured to cause the at least one processor to process the modified workflow based on the scheduling of the one or more tasks.

11. The apparatus of claim 10, wherein, The scheduling type information comprises one of order information, duration information, event information, or time slot information.

12. The apparatus of claim 11, further comprising: scheduling to perform each of the one or more tasks in order upon determining that the scheduling type information is the order information.

13. The apparatus of claim 11, further comprising: scheduling to perform a first task of the one or more tasks for a first duration upon determining that the scheduling type information is the duration information.

14. The apparatus of claim 11, further comprising: scheduling to perform a first task of the one or more tasks based on an occurrence of an event in the input media stream for triggering to start or stop the first task upon determining that the scheduling type information is the event information.

15. The apparatus of claim 14, wherein, defining a new paused state for each of the one or more tasks upon performing the one or more tasks based on the event information, in which the new paused state keeps each of the one or more tasks with an internal state and information without processing any further data in the corresponding task.

16. The apparatus of claim 11, further comprising: scheduling to perform a first task of the one or more tasks in a specified time slot upon determining that the scheduling type information is the time slot information.

17. The apparatus of claim 16, wherein, The time slot information further comprises: an explicit scheduling mode in which an exact time frame is provided for the scheduling; or an indexed scheduling mode in which a start, an end, an increment, a duration scheme are provided for the scheduling.

18. The apparatus of any one of claims 10-17, wherein, The scheduling descriptor comprises command information corresponding to the scheduling type information, and wherein the command information comprises information for: obtaining a capability of the cloud platform; determining whether the scheduling type information in a scheduling request is achievable; or providing a response comprising a result of determining whether the scheduling type information in the scheduling request is achievable.

19. A non-transitory computer-readable medium storing instructions, the instructions comprising one or more instructions that, when executed by at least one processor of an apparatus for scheduling a network based media processing (NBMP) workflow on a cloud platform, cause the at least one processor to perform the method of any of claims 1-9.

20. A computer device, comprising a processor and a memory, wherein the memory is configured to store program codes and transmit the program codes to the processor; and the processor is configured to execute the method of any of claims 1-9 according to instructions in the program codes. ​ ​

Images