Method and apparatus for performing step-based operations in media processing on a cloud platform

Abstract

Methods and apparatus for performing step-based operations in a network-based media processing (NBMP) workflow. The method includes obtaining a workflow comprising an input media stream, the input media stream having a plurality of segments; identifying boundaries between the plurality of segments in the input media stream according to one of metadata-based segment boundary signaling or code-based segment boundary signaling; and processing the plurality of segments based on the identified boundaries between the plurality of segments.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 218,817, filed July 6, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to the field of media processing, and more specifically to methods and apparatus for performing step-based operations in media processing on a cloud platform, computer equipment, and non-transitory computer-readable storage media. Background Technology

[0004] Networks and cloud platforms can be used to run various applications. The Network Based Media Processing (NBMP) standard provides specifications for defining, instantiating, and running workflows on cloud platforms. The NBMP standard also includes a set of tools for independently processing media segments with the same duration. For example, the NBMP step descriptor provides information about operational support based on function steps. Here, the NBMP step descriptor can define the maximum duration / size of a segment and the unit of operation, as well as other parameters. However, the NBMP standard's step descriptor functionality does not have defined segment boundaries.

[0005] Therefore, the technical problem to be solved by this disclosure is how to properly handle the inputs and outputs in a step descriptor by defining segment boundaries. Summary of the Invention

[0006] One or more exemplary embodiments of this disclosure provide methods and apparatus for providing step descriptor functionality by defining segment boundaries.

[0007] According to one aspect of this disclosure, a method is provided for performing step-based operations in media processing on a cloud platform, the method comprising: acquiring a workflow including an input media stream having multiple segments; identifying boundaries between the multiple segments in the input media stream based on either metadata-based segment boundary signaling or code-based segment boundary signaling; and processing the multiple segments based on the identified boundaries between the multiple segments.

[0008] According to another aspect of this disclosure, an apparatus is provided for performing step-based operations in media processing on a cloud platform, the apparatus comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising: acquisition code configured to cause the at least one processor to acquire a workflow including an input media stream comprising a plurality of segments; identification code configured to cause the at least one processor to identify boundaries between the plurality of segments in the input media stream based on either metadata-based segment boundary signaling or code-based segment boundary signaling; and processing code configured to cause the at least one processor to process the plurality of segments based on the identified boundaries between the plurality of segments.

[0009] According to another aspect of this disclosure, an apparatus is provided for performing step-based operations in media processing on a cloud platform, the apparatus comprising: an acquisition unit configured to cause at least one processor to acquire a workflow including an input media stream comprising a plurality of segments; an identification unit configured to cause the at least one processor to identify boundaries between the plurality of segments in the input media stream based on either metadata-based segment boundary signaling or code-based segment boundary signaling; and a processing unit configured to cause the at least one processor to process the plurality of segments based on the identified boundaries between the plurality of segments.

[0010] According to another aspect of the present invention, a computer device is provided, including a processor and a memory. The memory is used to store program code and transmit the program code to the processor; the processor is used to execute, according to instructions in the program code: acquiring a workflow including an input media stream having multiple segments; identifying boundaries between the multiple segments in the input media stream based on either metadata-based segment boundary signaling or code-based segment boundary signaling; and processing the multiple segments based on the identified boundaries between the multiple segments.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable medium storing instructions including one or more of the following, which, when executed by at least one processor of a means for performing step-based operations in media processing on a cloud platform, cause the at least one processor to: acquire a workflow including an input media stream having a plurality of segments; identify boundaries between the plurality of segments in the input media stream based on either metadata-based segment boundary signaling or code-based segment boundary signaling; and process the plurality of segments based on the identified boundaries between the plurality of segments.

[0012] The method and apparatus for performing step-based operations in media processing on a cloud platform, according to embodiments of this disclosure, firstly acquire a workflow including an input media stream having multiple segments; then, identify boundaries between the multiple segments in the input media stream based on either metadata-based segment boundary signaling or code-based segment boundary signaling; and finally, process the multiple segments based on the identified boundaries between them. In this way, a method is provided to indicate segment boundaries with signals, thereby appropriately processing inputs and outputs in a step descriptor by defining the segment boundaries.

[0013] Additional aspects will be set forth in part in the description which follows, and in part will become apparent from the description, or may be realized by practice of the embodiments proposed in this disclosure. Attached Figure Description

[0014] The above and other aspects, features, and embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0015] Figure 1 It is a schematic diagram of a communication system according to one or more embodiments.

[0016] Figure 2 This is a simplified example diagram of a streaming environment according to one or more implementation methods.

[0017] Figure 3 This is a block diagram of an NBMP system according to one or more implementations.

[0018] Figure 4 This is an example diagram of NBMP splitter and merger processing according to one or more implementations.

[0019] Figure 5 This is a block diagram of an example process for performing step-based operations in media processing on a cloud platform, according to one or more implementations.

[0020] Figure 6 It is a block diagram of an example of computer code for performing step-based operations in media processing on a cloud platform, according to one or more embodiments. Detailed Implementation

[0021] Embodiments of this disclosure relate to methods and apparatus for signaling modifications in a network-based media processing (NBMP) workflow, and more particularly, to apparatus and methods for performing step-based operations in media processing on a cloud platform.

[0022] The embodiments of this disclosure are exemplary in nature and will be fully described with reference to the accompanying drawings. However, examples of implementations can be implemented in a variety of forms and should not be construed as limiting this disclosure to the examples described herein. Rather, examples of implementations are provided to make the technical solutions of this disclosure more comprehensive and complete, and to fully convey the ideas of the examples of implementations to those skilled in the art. The accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of parts are omitted.

[0023] The proposed features discussed below can be used individually or in any order. Some block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. Furthermore, implementations can be: implemented in hardware, which may include processing circuitry systems (e.g., one or more processors or one or more integrated circuits) or microcontroller devices; implemented in software; implemented in different network and / or processor devices; or implemented by a combination of hardware and software components. In one example, one or more processors execute computer program code stored on one or more non-transitory computer-readable media.

[0024] Figure 1 This is a diagram of an environment 100 in which the methods, apparatus, and systems described herein can be implemented according to an embodiment. For example... Figure 1 As shown, environment 100 may include user equipment 110, platform 120, and network 130. The devices in environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0025] User equipment 110 may include one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with platform 120. For example, user equipment 110 may include computing devices (e.g., desktop computers, laptop computers, tablet computers, handheld computers, smart speakers, servers, etc.), mobile phones (e.g., smartphones, cordless phones, etc.), wearable devices (e.g., smart glasses or smartwatches), or similar devices. In some implementations, user equipment 110 may receive information from platform 120 and / or transmit information to platform 120.

[0026] Platform 120 may include one or more devices as described elsewhere herein. In some implementations, platform 120 may include a cloud server or a group of cloud servers. In some implementations, platform 120 may be designed to be modular, allowing software components to be swapped in or out as needed. Therefore, platform 120 can be easily and / or quickly reconfigured for different purposes.

[0027] In some implementations, as shown, platform 120 may be hosted in a cloud computing environment 122. It is worth noting that while the implementations described herein depict platform 120 as hosted in a cloud computing environment 122, in some implementations, platform 120 may not be cloud-based (i.e., it may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0028] The cloud computing environment 122 may include the environment of the hosting platform 120. The cloud computing environment 122 can provide services such as computing, software, data access, and storage, which do not require end users (e.g., user equipment 110) to know the physical location and configuration of the systems and / or devices of the hosting platform 120. As shown, the cloud computing environment 122 may include a set of computing resources 124 (collectively referred to as "computing resources 124" and individually as "computingresource 124").

[0029] Computing resource 124 may include one or more personal computers, workstations, server devices, or other types of computing and / or communication devices. In some implementations, computing resource 124 may host platform 120. Cloud resources may include: computing instances executing in computing resource 124, storage devices provided in computing resource 124, data transmission devices provided by computing resource 124, etc. In some implementations, computing resource 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0030] like Figure 1 As further shown, computing resource 124 may include a set of cloud resources, such as one or more applications (“Application, APP”) 124-1, one or more virtual machines (“Virtual Machine, VM”) 124-2, virtualized storage devices (“Virtualized Storage, VS”) 124-3, one or more hypervisors (“Hypervisor, HYP”) 124-4, etc. However, this disclosure is not limited thereto, and therefore, according to other example embodiments, computing resource 124 may include other types of cloud resources.

[0031] Application 124-1 may include one or more software applications that can be provided to or accessed by user equipment 110 and / or platform 120. Application 124-1 may eliminate the need to install and execute software applications on user equipment 110. For example, application 124-1 may include software associated with platform 120 and / or any other software that can be provided via cloud computing environment 122. In some implementations, an application 124-1 may send information to or receive information from one or more other applications 124-1 via virtual machine 124-2.

[0032] Virtual machine 124-2 may include a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 124-2 may be a system virtual machine or a process virtual machine, depending on the extent to which virtual machine 124-2 uses and corresponds to any real machine. A system virtual machine may be a complete system platform that supports the execution of a complete operating system (“OS”). A process virtual machine may execute a single program and may support a single process. In some implementations, virtual machine 124-2 may execute on behalf of a user (e.g., user device 110) and may manage the infrastructure of the cloud computing environment 122, such as data management, synchronization, or long-duration data transfer.

[0033] Virtualized storage device 124-3 may include one or more storage systems and / or one or more devices that utilize virtualization technology within the storage system or device of computing resource 124. In some implementations, within the context of the storage system, the type of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the extraction (or separation) of logical storage from physical storage, enabling access to the storage system regardless of physical storage or heterogeneous architecture. Separation allows storage system administrators greater flexibility in managing storage for end users. File virtualization eliminates the dependency between data accessed at the file level and the location where the file is physically stored. This enables performance optimization for storage usage, server consolidation, and / or non-disruptive file migration.

[0034] Hypervisor 124-4 can provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer such as computing resource 124. Hypervisor 124-4 can present a virtual operating platform to the guest operating system and manage the execution of the guest operating system. Multiple instances of various operating systems can share virtualized hardware resources.

[0035] Network 130 may include one or more wired and / or wireless networks. For example, network 130 may include cellular networks (e.g., fifth-generation (5G) networks, long-term evolution (LTE) networks, third-generation (3G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMN), local area networks (LAN), wide area networks (WAN), metropolitan area networks (MAN), telephone networks (e.g., public switched telephone networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-optic networks, etc., and / or combinations of these or other types of networks.

[0036] Figure 1 The number and arrangement of devices and networks shown are provided as examples. In practice, with Figure 1 Compared to the devices and / or networks shown, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks arranged differently. Furthermore, Figure 1 The two or more devices shown can be implemented within a single device, or Figure 1 The single device shown can be implemented as multiple distributed devices. Additionally or alternatively, a group of devices in environment 100 (e.g., one or more devices) can perform one or more functions described as being performed by another group of devices in environment 100.

[0037] Figure 2 yes Figure 1 A block diagram of example components of one or more devices. Device 200 may correspond to user device 110 and / or platform 120. Figure 2 As shown, device 200 may include bus 210, processor 220, memory 230, storage unit 240, input unit 250, output unit 260 and communication interface 270.

[0038] Bus 210 includes components that allow communication between parts of device 200. Processor 220 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 220 may be a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Accelerated Processing Unit (APU), microprocessor, microcontroller, Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), or another type of processing unit. In some implementations, processor 220 may include one or more processors that can be programmed to perform functions. Memory 230 may include Random Access Memory (RAM), Read Only Memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 220.

[0039] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state drives), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, and / or other types of non-transitory computer-readable media and corresponding drives.

[0040] Input component 250 may include components that allow device 200 to receive information, for example, via user input (e.g., a touchscreen display, keyboard, keypad, mouse, button, switch, and / or microphone). Alternatively or additionally, input component 250 may include sensors for sensing information (e.g., a Global Positioning System (GPS) component, accelerometer, gyroscope, and / or actuator). Output component 260 includes components that provide output information from device 200 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0041] Communication interface 270 may include transceiver-like components (e.g., transceiver and / or separate receiver and transmitter) that enable device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 270 may allow device 200 to receive information from and / or provide information to another device. For example, communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0042] Device 200 can perform one or more of the processes described herein. Device 200 can perform these processes in response to processor 220 executing software instructions stored in non-transitory computer-readable media such as memory 230 and / or storage unit 240. Computer-readable media are defined herein as non-transitory memory devices. Memory devices include memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0043] Software instructions can be read into memory 230 and / or storage unit 240 from another computer-readable medium or from another device via communication interface 270. When executed, the software instructions stored in memory 230 and / or storage unit 240 can cause processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry can be used in place of or in combination with the software instructions to perform one or more processes described herein. Therefore, the implementations described herein are not limited to any particular combination of hardware circuitry and software.

[0044] Figure 2 The number and arrangement of components shown are provided as an example. In practice, with Figure 2 Compared to the components shown, device 200 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a set of components of device 200 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 200.

[0045] In embodiments of this disclosure, a network-based media processing (NBMP) system is provided. Figure 3An NBMP architecture 300 according to an embodiment described herein is illustrated and can be implemented relative to cloud processing. The NBMP system 300 includes an NBMP source 310, an NBMP workflow manager 320, a function store 330, one or more Media Processing Entities (MPEs) 340, a media source 350, and a media sink 360. The NBMP source 310, NBMP workflow manager 320, function store 330, MPE 340, media source 350, and media sink 360 may include, or be implemented by, at least one or more processors and memory, wherein the memory stores code configured to cause at least one or more processors to respectively execute the functions of the NBMP source 310, NBMP workflow manager 320, function store 330, MPE 340, media source 350, and media sink 360.

[0046] NBMP source 310 can communicate workflow descriptions with NBMP workflow manager 320 via NBMP Workflow API (Application Programming Interface, API) 311. NBMP source 310 can also communicate function descriptions with function store 330 via function discovery API 313. For example, NBMP source 310 can send a Workflow Description Document (WDD) to NBMP workflow manager 320 and can read function descriptions of functions stored in function store 330, which are media processing functions stored in the memory of function store 330, such as media decoding, feature point extraction, camera device parameter extraction, projection methods, seam information extraction, mixing, post-processing, and encoding functions. NBMP workflow manager 320 can communicate with function store 330 via function discovery API 312, which may be the same as or different from function discovery API 313, and can communicate with one or more MPEs 340 via API 314 (e.g., MPE API).

[0047] Media Processing Entity (MPE) 340 may include one or more tasks 341. NBMP Workflow Manager 320 may also communicate with tasks 341 via API 315 (e.g., the NBMP Task API). NBMP Workflow Manager 320 can use API 315 to set up, configure, manage, and control one or more tasks 341 that can execute workflows by one or more MPEs 340. To configure, manage, and control the tasks 341 of the workflow, NBMP Workflow Manager 320 may send messages such as requests to one or more of the MPEs 340 and / or tasks 341, where each message may have several descriptors, each of which has several parameters. Additionally, communication between NBMP source 310, NBMP Workflow Manager 320, function store 330, and MPE 340 can be considered a control flow.

[0048] Task 341 may each include a media processing function 343 and a configuration 342 for the media processing function 343. Each of the tasks 341 in the corresponding media processing entity 340 may also communicate with each other to facilitate data flow between tasks. In an implementation, the NBMP workflow manager 320 may select a task based on the task description in the WDD to search the function store 330 via the function discovery API 312 to find the appropriate function to run as task 341 for the current workflow. One or more MPEs 340 may be configured to receive media content from a media source 350, process the media content according to a workflow including task 341 created by the NBMP workflow manager 320, and output the processed media content to a media receiver 360. In an implementation, one or more MPEs 340 may be provided in parallel for multiple media streams 316 and 317 between the media source 350 and the media receiver 360.

[0049] Media source 350 may include a memory for storing media and may be integrated with or separate from NBMP source 310. In one implementation, NBMP workflow manager 320 may notify NBMP source 310 when a workflow is ready, and media source 350 may transmit media content to one or more MPEs 340 based on the workflow ready notification, and one or more MPEs 340 may transmit media content to media receiver 360. Communication between media source 350, MPEs 340, and media receiver 360 can be considered as a data stream.

[0050] Figure 4 An example of NBMP segmenting and merging media stream 440 of task T for parallel processing, according to an embodiment of this disclosure, is shown. Figure 4As shown, the NBMP splitter and merger process 400 includes a splitter 420 and a merger 430. In Figure 4 In this process, the task T that inputs the media stream is transformed into n instances of task T (i.e., T0, ..., Tn). N-1 The task T consists of n instances that run in parallel. The input media stream 410 can be continuous. The splitter 420 converts the media stream into N media sub-streams. Each sub-stream is processed by an instance of T. The merger 430 then interweaves the sub-streams to produce an output 450. The output 450 is equivalent to the output stream of task T (i.e., media stream 440).

[0051] like Figure 4 As shown, the 1:N splitter 420 and N:1 merger 430 function with respect to segment boundaries. Each segment has metadata associated with it, including start, duration, and length. Because the segments are independent, the substreams are independent of each other in terms of being processed by task T. In the implementation, tasks T0, ..., T... N-1 It is an instance of task T, and segments can be processed simultaneously or individually. Since segments and substreams are independent, each instance of task T (i.e., T0, ..., T1) is a separate instance of task T. N-1 It can run at its own speed.

[0052] According to embodiments of this disclosure, methods for indicating segment boundaries using signals are described. These methods may include metadata methods or start code methods. According to embodiments, segment boundaries need to be indicated using signals in the NBMP step descriptor to better support step-based operations in media processing on cloud platforms. According to embodiments, the NBMP step descriptor can provide information about operational support based on function steps. Here, a function may have information such as a descriptor indicating whether the function can operate in step mode, where independent and / or parallel processing can be performed based on defined segments. The descriptor may define the maximum duration / size of the segment and the number of operations, as well as other parameters.

[0053] [Metadata Method]

[0054] According to embodiments of this disclosure, in a method for indicating segment boundaries using metadata with signals, metadata can be provided to define the segment start, segment duration, and segment size. Such definitions can be provided for any of the segments of task T. For example, metadata definitions can be as follows:

[0055] • S: Segment start at the T scale

[0056] • D: Segment duration at the T scale

[0057] • L: Segment length in bytes

[0058] • T: Time scale (number of ticks in seconds)

[0059] According to the metadata-based signaling approach, any media stream input or output for any function should have a corresponding metadata input or output carrying segment boundary metadata. Therefore, when performing step-based operations in media processing on a cloud platform, segments of the media stream 440 of task T can be appropriately split and merged for parallel processing.

[0060] [Start Code Method]

[0061] According to embodiments of this disclosure, a start code-based method for indicating segment boundaries with signals can be provided as an alternative to the metadata method. According to the signaling method using start codes, a start code can be given for the beginning of each segment. In this case, the media stream input can indicate the start code given for each segment. Furthermore, the media stream output can indicate the start code given for each segment.

[0062] According to the signaling method based on start codes, any media stream input or output for any function should have corresponding metadata input or output carrying a start code that identifies segment boundaries. Therefore, when performing step-based operations in media processing on a cloud platform, start codes can be used to appropriately perform segmentation and merging of media stream 440 for task T for parallel processing.

[0063] According to the implementation, by providing the same start pattern (i.e., start code) for each of the input segments, the process for splitting and merging segments can easily identify the segment boundaries. Therefore, start time information and duration information for each segment (as is the case in metadata-based methods) are not required to indicate segment boundaries with signals.

[0064] Although a start code indicating the start of a segment is provided according to the implementation, this disclosure is not limited to the start code. Therefore, according to other implementations, patterns or codes may be provided at different locations on the input and output streams to indicate segment boundaries. For example, codes such as end codes may be provided at the end of a segment to define the boundaries between segments.

[0065] According to the example implementation, the step descriptor of the NBMP standard specification can be extended in the following ways:

[0066] Table 1—Step Descriptors

[0067] name type cardinality Step-by-step mode P 0-1 Duration of segment P 0-1 Segment metadata P 0-1 Segment start code P 0-1 Operation unit P 0-1 Time overlap P 0-1 Dimension P 0-1 High-dimensional segment divisor P 0-1 High-dimensional segment description P 0-1 High-dimensional segment sorting P 0-1 High-dimensional overlap P 0-1

[0068] For example, segment metadata descriptors and segment start code descriptors can be added to the step descriptor of the NBMP standard specification.

[0069] Furthermore, according to the example implementation, the step descriptor parameters of the NBMP standard specification can be extended in the following ways:

[0070] Table 2—Step Descriptor Parameters

[0071]

[0072]

[0073] For example, parameters for the segment metadata descriptor and the segment start code descriptor can be added to the step descriptor parameters of the NBMP standard specification.

[0074] According to the example implementation, the segment metadata descriptor can be of Boolean type and has the following definition: if "true", the function supports receiving / providing segment metadata for each media input / output to detect segment boundaries. The default value can be "false".

[0075] According to the example implementation, the segment start code descriptor can be a boolean type and has the following definition: if it is "true", the function supports receiving / providing a start code for each media input / output to detect segment boundaries.

[0076] According to an example implementation, a method for identifying segment boundaries may include determining whether either segment metadata or segment start code description is true, and identifying segment boundaries using a metadata method based on the determination that the segment metadata is true, and identifying segment boundaries using a start code method based on the determination that the segment start code is true.

[0077] Figure 5 This is a flowchart of an example method 500 for performing step-based operations in a network-based media processing (NBMP) workflow, according to one or more implementations.

[0078] In some implementations, this can be executed by platform 120. Figure 5 One or more processing blocks. In some implementations, Figure 5 One or more processing blocks may be executed by another device or a group of devices, such as user equipment 110, that is separate from or includes platform 120.

[0079] like Figure 5 As shown, in operation 510, method 500 includes acquiring a workflow. Here, the workflow may include an input media stream, and the input media stream may include multiple segments;

[0080] In operation 520, method 500 includes identifying boundaries between multiple segments in an input media stream based on either metadata-based segment boundary signaling or code-based segment boundary signaling. According to an implementation, code-based segment boundary signaling may include a start code given at the beginning of each segment to indicate the boundaries between the multiple segments.

[0081] In operation 530, method 500 includes processing multiple segments based on the boundaries between the identified multiple segments.

[0082] According to an implementation, method 500 may further include generating a modified workflow by dividing multiple segments into one or more tasks for independent processing of the modified workflow. One or more tasks of the modified workflow are processed in parallel.

[0083] According to an implementation, method 500 may further include: receiving, in code-based segment boundary signaling, a code corresponding to a plurality of segments in an input media stream; and determining that each time the code appears in the input media stream, the code indicates a boundary between adjacent segments.

[0084] According to an implementation, method 500 may further include determining whether the input media stream supports either metadata-based segment boundary signaling or code-based segment boundary signaling.

[0085] According to an implementation, method 500 may further include: providing a first flag corresponding to metadata-based segment boundary signaling; and providing a second flag corresponding to code-based segment boundary signaling.

[0086] According to an implementation, method 500 may further include merging one or more outputs from processing one or more tasks of the modified workflow to generate an output media stream.

[0087] According to an implementation, method 500 may further include applying either metadata-based segment boundary signaling or code-based segment boundary signaling to the output media stream.

[0088] although Figure 5 An example block of the method is shown, but in some implementations, the method may include... Figure 5 The blocks described herein are compared to additional blocks, fewer blocks, different blocks, or blocks arranged differently. Alternatively or concurrently, two or more blocks in a method may be executed in parallel.

[0089] Figure 6 This is a block diagram of an example of computer code for performing step-based operations in a network-based media processing (NBMP) workflow, according to one or more implementations.

[0090] According to embodiments of this disclosure, at least one processor may be provided having a memory storing computer code. The computer code may be configured to execute any number of aspects of this disclosure when executed by the at least one processor.

[0091] For example, refer to Figure 6 Computer code 600 can be implemented in NBMP system 300.

[0092] like Figure 6 As shown, computer code 600 may include acquisition code 610, identification code 620 and processing code 630.

[0093] The acquisition code 610 can be configured to cause at least one processor to acquire the workflow. The workflow may include an input media stream, and the input media stream may include multiple segments.

[0094] The identification code 620 can be configured to enable at least one processor to identify the boundaries between multiple segments in an input media stream based on either metadata-based segment boundary signaling or code-based segment boundary signaling.

[0095] Processing code 630 can be configured to cause at least one processor to process multiple segments based on the boundaries between the identified multiple segments.

[0096] although Figure 6 Example blocks of computer code 600 for an apparatus or device according to an embodiment are shown, but in some implementations, the apparatus may include... Figure 6 The blocks depicted herein are compared to additional blocks, fewer blocks, different blocks, or blocks arranged differently. Alternatively or additionally, two or more blocks in the device can be combined.

[0097] This disclosure also provides an apparatus for performing step-based operations in media processing on a cloud platform, the apparatus comprising: an acquisition unit configured to cause at least one processor to acquire a workflow including an input media stream, the input media stream including a plurality of segments; an identification unit configured to cause the at least one processor to identify boundaries between the plurality of segments in the input media stream based on either metadata-based segment boundary signaling or code-based segment boundary signaling; and a processing unit configured to cause the at least one processor to process the plurality of segments based on the identified boundaries between the plurality of segments.

[0098] In some examples, the device further includes:

[0099] A generation unit is configured to cause the at least one processor to generate a modified workflow by dividing the plurality of segments into one or more tasks for independent processing of the modified workflow.

[0100] In some examples, the code-based segment boundary signaling includes a start code given at the beginning of each segment to indicate the boundary between the plurality of segments.

[0101] In some examples, the device further includes:

[0102] A receiving unit configured to cause the at least one processor to receive, in the code-based segment boundary signaling, codes or patterns corresponding to the plurality of segments in the input media stream; and

[0103] A determining unit is configured to cause the at least one processor to determine that each time the code appears in the input media stream, the code indicates a boundary between adjacent segments.

[0104] In some examples, the device further includes:

[0105] A determining unit is configured to cause the at least one processor to determine whether the input media stream supports either metadata-based segment boundary signaling or code-based segment boundary signaling.

[0106] In some examples, the device further includes:

[0107] A providing unit, configured to cause the at least one processor to provide:

[0108] The first flag corresponding to the metadata-based segment boundary signaling; and

[0109] The second flag corresponding to the code-based segment boundary signaling.

[0110] In some examples, the one or more tasks of the modified workflow are processed in parallel.

[0111] In some examples, the device further includes:

[0112] A merging unit is configured to cause the at least one processor to merge one or more outputs from processing one or more tasks of the modified workflow to generate an output media stream.

[0113] In some examples, the device further includes:

[0114] An application unit is configured to cause the at least one processor to apply either the metadata-based segment boundary signaling or the code-based segment boundary signaling to the output media stream.

[0115] This disclosure also provides a computer device including a processor and a memory. The memory is used to store program code and transmit the program code to the processor; the processor is used to execute, according to instructions in the program code: acquiring a workflow including an input media stream having multiple segments; identifying boundaries between the multiple segments in the input media stream based on either metadata-based segment boundary signaling or code-based segment boundary signaling; and processing the multiple segments based on the identified boundaries between the multiple segments.

[0116] The techniques described above for performing signal-indicating segment boundaries in step-based operations within a network-based media processing (NBMP) workflow can be used individually or in any combination in any order. Furthermore, each of the methods (or implementations) can be implemented using a processing circuitry system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors can execute a program stored on a non-transitory computer-readable medium.

[0117] This disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementation to the exact form disclosed. Modifications and variations can be made based on this disclosure, or from the practice of implementing the methods.

[0118] As used in this article, the term component is intended to be interpreted broadly as hardware, firmware, or a combination of hardware and software.

[0119] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited in its implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the descriptions herein.

[0120] Even if combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways that are not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly refer to only one claim, the disclosure of possible implementations includes every dependent claim combined with every other claim in the claim set.

[0121] Unless explicitly stated otherwise, elements, actions, or instructions used herein should not be construed as critical or necessary. Furthermore, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Additionally, as used herein, the term “group” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with “one or more.” The term “one” or similar language is used where only one item is intended. Moreover, as used herein, the terms “has,” “have,” “having,” etc., are intended to be open-ended terms. Furthermore, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.”

Claims

1. A method for performing step-based operations in media processing on a cloud platform, characterized in that, The method includes: Acquire a workflow that includes an input media stream, the input media stream comprising multiple segments; The boundaries between the plurality of segments in the input media stream are identified based on either segment boundary signaling based on segment metadata or segment boundary signaling based on the segment start code; and The multiple segments are processed based on the boundaries between the identified multiple segments; Add the parameters of the segment metadata descriptor and the segment start code descriptor to the step descriptor parameters. The step descriptor parameters include step mode, segment duration, operation unit, segment metadata descriptor, segment start code descriptor, time overlap, dimension, high-dimensional segment divisor, high-dimensional description, high-dimensional segment order, high-dimensional overlap, and high-dimensional operation unit. If the segment metadata descriptor is true, it indicates that the function supports receiving / providing segment metadata for each media input / output to detect segment boundaries; if the segment metadata descriptor is false, it does not support this function. If the segment start code descriptor is true, the function indicates that it supports receiving / providing a start code for each media input / output to detect segment boundaries; if the segment start code descriptor is false, it is not supported.

2. The method of claim 1, wherein, Also includes: A modified workflow is generated by dividing the multiple segments into one or more tasks, allowing the modified workflow to be processed independently.

3. The method according to claim 1 or 2, characterized in that, The segment boundary signaling based on the segment start code includes a start code given at the beginning of each segment to indicate the boundary between the plurality of segments.

4. The method according to claim 1 or 2, characterized in that, Also includes: In the segment boundary signaling based on the segment start code, a segment start code corresponding to the plurality of segments in the input media stream is received; as well as Each time the segment start code appears in the input media stream, the segment start code indicates the boundary between adjacent segments.

5. The method according to claim 1 or 2, characterized in that, Also includes: Determine whether the input media stream supports either segment boundary signaling based on segment metadata or segment boundary signaling based on the segment start code.

6. The method of claim 1 or 2, wherein, Also includes: Provide a first flag corresponding to the segment boundary signaling based on segment metadata; as well as Provide a second flag corresponding to the segment boundary signaling based on the segment start code.

7. The method of claim 2, wherein, The one or more tasks in the modified workflow are processed in parallel.

8. The method of claim 2, wherein, Also includes: The outputs from one or more tasks that process the modified workflow are combined to generate an output media stream.

9. The method of claim 8, wherein, Also includes: The segment boundary signaling based on segment metadata or the segment boundary signaling based on the segment start code is applied to the output media stream.

10. An apparatus for performing step-based operations in media processing on a cloud platform, characterized in that, The device includes: Memory, which stores instructions; and A processor that communicates with the memory, wherein, when the processor executes the instructions, the processor is configured to cause the device to perform the method according to any one of claims 1 to 9.

11. A non-transitory computer-readable medium storing instructions, the instructions comprising one or more instructions that, when executed by at least one processor of a means for performing step-based operations in media processing on a cloud platform, cause the at least one processor to perform the method of any one of claims 1 to 9.

12. A computer device, comprising a processor and a memory, characterized in that, The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method of any one of claims 1 to 9 according to the instructions in the program code.

Images