Methods and systems for multi-format media processing

Abstract

To provide methods and systems for media processing which are capable of accepting any media input and converting the media input into one or more media outputs.SOLUTION: Multiple video sources are provided, each of which produces a corresponding video stream. A microservice server is provided which comprises microservices adapted to perform at least one of decoding, encoding, scaling, routing and transmission of the corresponding video streams. Multiple computational resources available for use by the microservices are provided. A resource management subsystem is provided for orchestrating the microservices according to project parameters of one or more media projects, allocating and coordinating the multiple the computational resources required for the one or more media projects, and outputting corresponding media processed by the microservices for the one or more media projects.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Application Nos. 63 / 633,095 and 63 / 633,100, filed April 12, 2024, both of which are incorporated herein in their entireties and made a part hereof for all purposes. [Background technology]

[0002] The present invention relates to the field of media processing and routing. More particularly, the present invention relates to an improved method and system for processing and routing media received in any format for output in any format, and which dynamically utilizes resources. The present invention also relates to an object-based user interface that enables automated media processing and routing. Summary of the Invention [Problem to be solved by the invention]

[0003] In the prior art, media processing and routing was typically a complex process involving hardware, wiring, software, and dedicated resources for each function. Such systems faced challenges such as diverse input and output formats, resource allocation, and operational complexity. This complexity led to high costs and management difficulties.

[0004] Figure 1 shows an example of a prior art media processing and routing architecture 10. This system uses multiple hardware decoders 12 to decode media received in a variety of formats. The decoded media typically passes through a hardware-based NxN audio / video router 14 and is then sent to a hardware-based encoder 16 depending on the encoding format required at the destination. Multiple hardware decoders and encoders are required to accommodate the various encoding formats. Furthermore, if the format of the input media (e.g., video resolution) differs from the format of the output media, the signal must be routed through additional processors, such as a video scaler, to convert the input media to the required output format.

[0005] Such complex hardware-based routing systems required configuration of the system (both hardware and software) for each media file or media project processed, and such configuration procedures typically could not be reused for subsequent media files or media projects (e.g., television programs, live broadcasts, commercials, etc.).

[0006] It is therefore desirable to provide a software-based method and system for processing and routing media content that allows for ease of use and repeatable use.

[0007] It would also be desirable to provide a more versatile and efficient system that addresses the challenges and limitations of traditional hardware-based systems. It would also be desirable to provide a simplified user interface for a video processing system that enables automated video processing and routing. It would also be desirable to provide an object-based system that allows for the repeated use of the same video processing and routing parameters when processing the same video source (e.g., the same television program or live broadcast format) without the need to reconfigure physical switches or software settings each time.

[0008] The methods and systems of the present invention provide these and other advantages. [Means for solving the problem]

[0009] The present invention relates to a method and system for processing and routing media that is received in any format and output in any format, making dynamic use of resources.

[0010] An embodiment of a media processing system capable of accepting any media input and transforming the media input into one or more media outputs includes a microservice server including a plurality of video sources (each video source producing a corresponding video stream), microservices configured to at least one of decode, encode, scale, route, and transmit the corresponding video streams, a plurality of computational resources available to the microservices, and a resource management subsystem that coordinates the microservices according to project parameters of one or more media projects, allocates and coordinates the plurality of computational resources needed by the microservices for the one or more media projects, and outputs corresponding media processed by the microservices for the one or more media projects.

[0011] The media project may be an object-based media project. The project object defines project parameters for the media project. The project parameters may include identification of one or more video sources to include in the media project, one or more destinations, encoding formats for one or more media destinations, and corresponding video and audio bitrates.

[0012] The resource management subsystem identifies the microservices required for each media project based on the project parameters and one or more video sources and one or more output destinations identified in the project object. The resource management subsystem can then define one or more process workflows for executing the project through the interconnected microservices.

[0013] Each of the plurality of computing resources may include or utilize one or more of on-premise computing resources and cloud-based computing resources.

[0014] Each of the plurality of computing resources may have an associated resource agent that analyzes, monitors, and reports resource availability of the corresponding computing resource.

[0015] The resource management subsystem can include an orchestration service that determines which microservices are required to complete each media project based on project parameters and determines the interconnections between the microservices; a central resource management system that monitors and manages computing resources based on communications from resource agents; and a task assignment service that initiates one or more process workflows for each media project based on information provided by the orchestration service and the central resource management system.

[0016] The orchestration service can enable dynamic allocation of computing resources by the task allocation service based on real-time changes in computing resource availability.

[0017] Video sources can be added to project objects via a drag and drop user interface.

[0018] The project object can be provided to an object microservice, which provides information about the project object and an identification of the video source for the media project to a system backend. The system backend provides the video source and project object to an orchestration service, which can update the system backend with status information about the usage and availability of computing resources.

[0019] Project objects can be reused in repeatable media projects.

[0020] The system may further comprise a plurality of distribution systems for distributing the media to one or more media destinations, which may comprise one or more of a satellite uplink, a terrestrial broadcast system, an SDI router, and an IP streaming system.

[0021] Media inputs and outputs include SDI®, RTMP, RTSP, TS, SPTS, MPTS, HLS, SRT, NDI, Zyxi, YouTube®, Facebook®, TikTok®, Zoom®, TVU Grid, TVU Anywhere, TVU Partyline, TVU RPS, and more.

[0022] The present invention also includes a method for processing arbitrary media inputs and transforming the media inputs into one or more media outputs. The method may include providing a plurality of video sources (each of the plurality of video sources generating a corresponding video stream); providing a microservice server including microservices adapted to at least one of decode, encode, scale, route, and transmit the corresponding video streams; providing a plurality of computational resources available to the microservices; and providing a resource management subsystem for coordinating the microservices according to project parameters of one or more media projects, allocating and coordinating the plurality of computational resources required by the microservices for the one or more media projects, and outputting the corresponding media processed by the microservices for the one or more media projects. Method embodiments of the present invention may also include various features and functionality of the system embodiments described above. [Brief explanation of the drawings]

[0023] The present invention will now be described in conjunction with the accompanying drawings, in which like reference numbers indicate like elements and in which: [Figure 1] FIG. 1 shows a prior art media processing system. [Figure 2] FIG. 2 depicts a high-level block diagram of a media processing system according to one embodiment of the present invention. [Figure 3] FIG. 3 shows a block diagram of a media processing system according to one embodiment of the present invention. [Figure 4] FIG. 4 shows a process flow diagram of a media processing system according to one embodiment of the present invention. [Figure 5] FIG. 5 shows an example of a user interface of a media processing system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0024] The following detailed description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the present invention. Rather, the detailed description of exemplary embodiments will provide those skilled in the art with an effective description for implementing embodiments of the present invention. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the present invention as set forth in the appended claims.

[0025] The present invention also relates to a user interface or API for media processing systems that is specifically designed to simplify the operations required for video processing and routing. The present invention provides an intuitive user interface that minimizes complexity, allowing operators to focus on content and project management.

[0026] The use of APIs simplifies the implementation of media processing software. The process involves defining a "project object" to which users can add various output formats and destinations. Once created, users simply drag and drop input sources (e.g., live or recorded media files) onto the project object. Using the API, input objects (media files) can be connected to the project object. The platform then automatically procures the necessary resources, decodes, scales, and encodes the media signal to the format required for the output destination defined in the project object, and outputs the encoded signal in the defined format to the corresponding output destination.

[0027] The method and system of the present invention are configured to achieve the following functions. 1. Accept any media input format and transform it for delivery to multiple destinations in a variety of media formats. 2. Dynamically leverage compute resources, whether cloud-based or on-premise, depending on the needs of your media project. 3. The object-based control system allows users to create "project objects" as output objects and specify the desired output format and destination, simplifying operations and reducing the need for dedicated resources for each function, thereby achieving cost savings. 4. Provides an intuitive user interface that simplifies media routing and conversion processes (e.g., decoding and re-encoding). 5. Allow users to drag and drop input sources onto these project objects. 6. Enables automatic resource sourcing and signal routing, including decoding, scaling, and encoding to meet specifications defined by project objects. 7. Reduced operational complexity allows for greater focus on content and project management.

[0028] FIG. 2 illustrates a high-level block diagram of the present invention. As shown in FIG. 2, the system 20 of the present invention accepts multiple input video sources 18 in any type of video format. Based on project requirements, the system 20 determines which input sources to route to one or more specific outputs 21 and decodes and re-encodes the input sources as needed for the designated outputs 21. Thus, video sources 18 provided in any encoding format can be output to one or more destinations 21 in any corresponding encoding format. For example, media inputs and media outputs may include any one of SDI®, RTMP, RTSP, TS, SPTS, MPTS, HLS, SRT, NDI, Zyxi, YouTube®, Facebook®, TikTok®, Zoom®, TVU Grid, TVU Anywhere, TVU Partyline, TVU RPS, etc.

[0029] 3 illustrates a block diagram of an exemplary embodiment of a system 20 according to the present invention. The system 20 includes a plurality of video sources 18, each generating a corresponding video stream. A microservice server 25 is provided that includes microservices 23 configured to at least one of decode, encode, scale, route, and transmit the corresponding video streams. The system also includes a plurality of computational resources 24 available to the microservices 23. A resource management subsystem 26 is provided to coordinate the microservices 23 according to project parameters of one or more media projects, allocate and coordinate the plurality of computational resources 24 required by the microservices 23 for the one or more media projects, and output corresponding media processed by the microservices 23 for the one or more media projects.

[0030] Agents 22 are installed on the computational resources 24, analyze and monitor the resources available on the corresponding computational resources 24, and report to the resource management subsystem 26. In particular, the agents 22 can determine the status and available capabilities of each computational resource 24 for the microservices 23 required for the current media project. The computational resources 24 may utilize or include cloud-based services 28 (e.g., AWS, Azure, Google Cloud), or may be hardware based virtual machines or physical computer devices located on- or off-premises, or both. Each agent 22 consists of a process running on each computational resource 24. The agents 22 (or separate programs running on or communicating with each computational resource 24) are also responsible for downloading applications required for media processing to each computational resource 24. Alternatively, the applications may be provided in Docker format and accessed from a central server.

[0031] The microservices 23 are hardware and / or software adapted to at least one of decoding, encoding, scaling, routing, and transmitting the corresponding video streams, any of which may be located on a cloud-based or physical computer system.

[0032] A media project is an object-based media project (described below in connection with Figure 4). The project object defines the project parameters of a media project. The project parameters consist of the identification of one or more video sources included in the media project, one or more destinations, the encoding format of one or more media destinations, and the corresponding video and audio bitrates.

[0033] The resource management subsystem 26 identifies one or more microservices 23 required for each media project based on the project parameters, one or more video sources 18 and one or more output destinations identified in the project object.

[0034] The resource management subsystem 26, through the interconnected microservices 23, executes project parameters for each media project and defines one or more process workflows for invoking the identified microservices 23 to execute the media project.

[0035] The resource management subsystem 26 may include an orchestration service 30 that determines the microservices 23 required to complete each media project based on project parameters and determines the interconnections between the microservices 23, a central resource management system 31 that monitors and manages the computational resources 24 based on communications from the agents 22, and a task assignment service 32 that initiates one or more process workflows for each media project based on information provided by the orchestration service 30 and the central resource management system 31.

[0036] The central resource management system 31 monitors and manages the use of all computing resources 24 based on communications from the agents 22. The central resource management system 31 can also directly monitor the status of the computing resources 24. The central resource management system 31 provides a comprehensive overview of all available computing resources 24 and their current status, facilitating efficient allocation of computing resources 24 to the required microservices 23. For media projects, the orchestration service 30 plans and connects functional modules (e.g., microservices 23, such as decoders, encoders, video scalers, routers, and transmitters) based on project requests. The orchestration service 30 analyzes received project requests, creates task lists, and determines the interconnections of the various microservices 23 required for the completion of the media project. The orchestration service 30 can also determine the optimal allocation of computing resources 24 required for the completion of the media project, including dynamic resource allocation based on real-time changes in resource availability. The task allocation service 32 initiates the process workflow of the media project and, based on data from the orchestration service 30 and the central resource management system 31, allocates the microservices 23 to appropriate computing resources 24 based on resource availability and requirements communicated from the orchestration service 30.

[0037] The various components of system 20 may be located on-premise and / or in different physical or cloud-based locations and connected by any combination of wired and / or wireless networks now known or to be developed (e.g., the Internet, an intranet, an extranet, an EPN, a VPN, a LAN, a WLAN, etc.).

[0038] FIG. 4 illustrates an exemplary embodiment of a process flow according to the present invention. First, a project object 40 is defined. The project object 40 identifies various project parameters for a media project (e.g., a television program, a live broadcast, a sporting event, an IP video stream, etc.). The project parameters include the destination or output address of the media project, as well as one or more encoding formats and corresponding parameters, such as resolution, video bitrate, and audio bitrate, that correspond to the destination address of the media project. Because multiple destinations may be defined in the project object 40, the encoding format corresponding to each destination may also be defined, if necessary. For example, a program object 40 may define the output configuration for a particular program (e.g., "The Morning Show") and may have, for example, three outputs (one output is a 1080i SDI output, another output is an RTMP 1080i 5Mbps output, and a final output is an HLS 1080p output).

[0039] Identification of one or more video sources 18 required for a media project can be added to a project object 40 via a drag-and-drop user interface 36 associated with the system front-end. An object microservice 42 provides information about the project object 40 and input objects (video sources 18) to a system back-end 44. The project object 40 automatically identifies an appropriate decoder (e.g., one of the microservices 23) for decoding the video source and an appropriate encoder (e.g., another of the microservices 23) associated with the destination defined by the project object 40. The system back-end 44 then sends the selected input objects along with the project object 40 to the orchestration service 30. The project object 40 can be saved and reused for recurring media projects (e.g., daily or weekly television programs, network football games, etc.). As described above, the orchestration service 30 determines the optimal allocation of microservices 23 (e.g., encoders, decoders, transmitters, etc.) to complete the media project and determines the available computational resources 24 required by the microservices 23 based on resource availability information received from the resource management system 31 (provided by the agent 22 as described above in connection with Figure 3). Such decisions are made by the orchestration service 30 based on predetermined business rules for resource allocation. For example, if a project object 40 contains a single output destination pointing to a specific SDI output, that specific output process must reside on the physical hardware (resource 24) where the SDI output port is located. Input signals are routed to the compute resource 24 that contains that output process. The orchestration service 30 can update the system backend 44 with the current resource status and provide instructions to the task assignment service 32 regarding which microservices 23 and compute resources 24 to use to satisfy the parameters of the project object 40.The microservices 23 decode and re-encode the media input for the corresponding destinations, and then output the encoded media to an appropriate distribution system 46 (e.g., via corresponding computational resources 24) to distribute the media to one or more media destinations. Such a system 46 may include one or more of a satellite uplink, an over-the-air broadcast uplink, or an SDI router to distribute the media files to the appropriate destinations. The distribution system 46 may also output IP video streams to various destinations.

[0040] 5 illustrates an exemplary embodiment of a user interface 36 according to the present invention. This user interface is associated with a system front end. The user interface 36 can present video sources 18 in tiled form on a first portion 48 of a computer touchscreen and can present an output section on a second portion 50 of the touchscreen that includes output tiles corresponding to project objects 40. In the present invention, a user can touch and drag any video source 18 from the first portion 48 to a project object 40 on the second portion 50. For example, the illustration shows the selection and finger drag of a video source 18' from the first portion 48 to the second portion 50.

[0041] As described above, project object 40 identifies various parameters of a media project (e.g., a television program, a live broadcast, a sporting event, etc.), which may include a destination or output address for the media project, as well as one or more encoding formats and corresponding parameters, resolution, video bitrate, and audio bitrate that correspond to the media project's destination address.

[0042] The project object 40 automatically identifies the appropriate microservices 23 to decode the selected video source and the appropriate microservices 23 to encode / re-encode the video source for each destination defined by the project object 40. The project object 40 can be saved and reused for recurring media projects (e.g., daily or weekly television programs, network football matches, etc.).

[0043] When you add an input source 18 to a project object 40, the system automatically provisions the necessary microservices 23 and corresponding compute resources 24 to handle decoding, scaling, encoding, and routing the signal to one or more output destinations (e.g., television channel, YouTube® channel, streaming service, social media, etc.) as described above.

[0044] The present invention revolutionizes traditional approaches to media routing and transformation, providing a simplified, user-friendly interface and automated processes that significantly reduce complexity and resource requirements in media production environments.

[0045] It should be appreciated that the present invention provides advantageous methods and systems for processing, routing, and delivering media. While the present invention has been described with reference to various illustrative embodiments, numerous modifications and adaptations are possible without departing from the spirit and scope of the invention as set forth in the following claims.

Claims

1. 1. A media processing system capable of accepting any media input and transforming the media input into one or more media outputs, comprising: multiple video sources, each generating a corresponding video stream; a microservices server including microservices adapted to at least one of decoding, encoding, scaling, routing, and transmitting the corresponding video streams; a plurality of computing resources available to the microservices; a resource management subsystem that coordinates the microservices according to project parameters of one or more media projects, allocates and coordinates a plurality of computing resources required by the microservices for the one or more media projects, and outputs corresponding media processed by the microservices for the one or more media projects; a media processing system including

2. 2. The media processing system of claim 1, the media project is an object-based media project; The Project object defines the project parameters of a media project. the project parameters include identification of one or more video sources included in the media project, one or more destinations, encoding formats for one or more media destinations, and corresponding video and audio bitrates; Media processing system.

3. 3. The media processing system of claim 2, the resource management subsystem identifies microservices required for each of the media projects based on the project parameters and the one or more video sources and the one or more destinations identified in the project object. Media processing system.

4. 4. The media processing system of claim 3, the resource management subsystem defines one or more process workflows via interconnected microservices for executing the project parameters for each of the media projects and for invoking the identified microservices to execute the media projects. Media processing system.

5. 5. The media processing system of claim 4, each of the plurality of computing resources includes or utilizes one or more of an on-premise computing resource and a cloud-based computing resource; Media processing system.

6. 5. The media processing system of claim 4, Associating each of the plurality of computing resources with a resource agent that analyzes, monitors, and reports resource availability of the corresponding computing resource; Media processing system.

7. 7. The media processing system of claim 6, The resource management subsystem: an orchestration service for determining which of the microservices are required to complete each media project based on the project parameters and for determining the interconnections between the microservices; a central resource management system that monitors and manages computing resources based on communications from the resource agents; a task assignment service that initiates one or more process workflows for each media project based on information provided by the orchestration service and the central resource management system; a media processing system,

8. 8. The media processing system of claim 7, the orchestration service enables the task allocation service to dynamically allocate computing resources based on real-time changes in the availability of the computing resources. Media processing system.

9. 8. The media processing system of claim 7, the video source is added to the project object via a drag-and-drop user interface; Media processing system.

10. 8. The media processing system of claim 7, The project object is provided to an object microservice; The object microservice provides information about the project object and an identification of the video source of the media project to the system backend; the system backend provides the video source and the project object to the orchestration service; Media processing system.

11. 11. The media processing system of claim 10, the orchestration service updates the system backend with status information regarding the usage and availability of the computing resources; Media processing system.

12. 3. The media processing system of claim 2, the project object is reusable for repeatable media projects; Media processing system.

13. 2. The media processing system of claim 1, further comprising a plurality of delivery systems for delivering the media to one or more media destinations; Media processing system.

14. 14. The media processing system of claim 13, The distribution system includes one or more of a satellite uplink, a terrestrial broadcast system, an SDI router, and an IP streaming system.

15. 14. The media processing system of claim 13, the media input and the media output include one of SDI (registered trademark), RTMP, RTSP, TS, SPTS, MPTS, HLS, SRT, NDI, Zyxi, YouTube (registered trademark), Facebook (registered trademark), TikTok (registered trademark), Zoom (registered trademark), TVU Grid, TVU Anywhere, TVU Partyline, and TVU RPS; Media processing system.

16. 1. A method for processing any media input and transforming the media input into one or more media outputs, comprising: providing a plurality of video sources, each generating a corresponding video stream; providing a microservices server including microservices adapted to at least one of decoding, encoding, scaling, routing, and transmitting the corresponding video stream; providing a plurality of computing resources for use by the microservices; providing a resource management subsystem for coordinating the microservices according to project parameters of one or more media projects, allocating and coordinating a plurality of computing resources required by the microservices for the one or more media projects, and outputting corresponding media processed by the microservices for the one or more media projects; A method comprising:

Images