Method and system for multi-format media processing

Abstract

Methods and systems are provided for multi-format media processing that are capable of accepting any media input and converting the media input to one or more media outputs. A plurality of video sources are provided, each of which generates a corresponding video stream. A micro-service server is provided that includes micro-services adapted to at least one of decode, encode, scale, route, and transmit the corresponding video streams. A plurality of computing resources are provided for use by the micro-service. A resource management subsystem is provided for orchestrating the micro-service according to item parameters for one or more media items, allocating and coordinating the plurality of computing resources required for the micro-service for the one or more media items, and outputting corresponding media processed by the micro-service for the one or more media items.

Description

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 633,095, filed April 12, 2024, and U.S. Provisional Application No. 63 / 633,100, filed April 12, 2024, both of which are incorporated herein in their entireties and made a part hereof by reference for all purposes. Technical Field

[0002] The present invention relates to the field of media processing and routing. More specifically, the present invention relates to improved methods and systems for processing and routing media received in any format, output in any format, and dynamically utilizing resources. The present invention also relates to an object-based user interface that facilitates automated media processing and routing. Background Art

[0003] Conventional media processing and routing are often complex processes involving hardware, wiring, and software, as well as dedicated resources for each function. Such systems often face challenges in handling a variety of input and output formats, resource allocation, and operational complexity. This complexity also leads to high costs and management challenges.

[0004] Figure 1 An example of a prior art architecture 10 for media processing and routing is shown, which utilizes multiple hardware decoders 12 for decoding media received in various formats. The decoded media then typically passes through a hardware-based NxN audio / video router 14, which sends the media to a designated hardware-based encoder 16 depending on the encoding format required by the destination. The variety of possible encoding formats requires multiple hardware decoders and encoders. Furthermore, when the input media format (e.g., video resolution) differs from the output media format, this will require routing the signal through an additional processor (e.g., a video scaler) to convert the input media to match the required output format.

[0005] For such complex hardware-based routing systems, the system (hardware and software) needs to be configured for each media file or media item being processed. Typically, such configuration routines cannot be saved for subsequent use with the same media file or media item (e.g., a television program, a live event, a commercial, etc.).

[0006] It would be advantageous to provide a software-based method and system for processing and routing of media content that provides repeatability. It would also be advantageous to provide a more versatile and efficient system that addresses the challenges and limitations of prior art hardware-based systems.

[0007] It would also be advantageous to provide a simplified user interface for a video processing system that implements automated video processing and routing. It would be further advantageous to provide an object-based system that allows the same video processing and routing parameters to be reused, rather than requiring resetting physical switches and software settings each time a similar video source (e.g., the same television program or the same live event format) is processed.

[0008] The methods and systems of the present invention provide the foregoing and other advantages. Summary of the Invention

[0009] The present invention relates to methods and systems for processing and routing media received in any format to be output in any format and dynamically utilizing resources.

[0010] An example embodiment of a media processing system capable of accepting any media input and converting the media input into one or more media outputs may include: multiple video sources, each of the multiple video sources generating a corresponding video stream; a microservice server, which includes a microservice suitable for at least one of decoding, encoding, scaling, routing and transmitting the corresponding video stream; multiple computing resources available to the microservices; and a resource management subsystem for orchestrating the microservices according to project parameters of one or more media projects, allocating and coordinating the multiple computing resources required by the microservices for the one or more media projects, and outputting corresponding media processed by the microservices for the one or more media projects.

[0011] The media project may be an object-based media project. The project object defines project parameters of the media project. The project parameters may include identification information of one or more of the video sources to be included in the media project, one or more output destinations, one or more media output destination encoding formats, and corresponding video and audio bit rates.

[0012] Based on the project parameters, the resource management subsystem identifies the microservices required for each of the media projects based on the one or more video sources and the one or more output destinations identified in the project object. The resource management subsystem can then define one or more process workflows through interconnected microservices, the workflows being used to implement the project parameters for each of the media projects and activate the identified microservices to implement the media projects.

[0013] Each of the plurality of computing resources may include or utilize one or more of local computing resources and cloud-based computing resources.

[0014] A resource agent may be associated with each of the plurality of computing resources, the resource agent analyzing, monitoring, and reporting resource availability of the corresponding computing resource.

[0015] The resource management subsystem may include: an orchestration service for determining the microservices required to complete each of the media projects based on the project parameters and for determining the interconnections between the microservices; a central resource management system for monitoring and managing the computing resources based on communications from the resource agents; and a task allocation service for initiating the one or more process workflows for each of the media projects based on information provided by the orchestration service and by the central resource management system.

[0016] The orchestration service may implement dynamic allocation of the computing resources through the task allocation service based on real-time changes in availability of the computing resources.

[0017] The video sources may be added to the project object via a drag-and-drop user interface.

[0018] The project object can be provided to an object microservice. The object microservice provides information about the project object and identification of the video source of the media project to a system backend. The system backend provides the video source and the project object to the orchestration service. The orchestration service can update the system backend with status information regarding the usage and availability of the computing resources.

[0019] For repeatable media items, the item object may be reusable.

[0020] The system may further include a plurality of distribution systems for distributing the media to one or more media output destinations. The distribution systems may include one or more of a satellite uplink, an over-the-air broadcast system, an SDI router, and an IP streaming system.

[0021] The media input and the media output may include one of SDI, RTMP, RTSP, TS, SPTS, MPTS, HLS, SRT, NDI, Zyxi, YouTube, Facebook, TikTok, Zoom, TVU Grid, TVU Anywhere, TVU Partyline, and TVU RPS, among others.

[0022] The present invention also includes a method for processing any media input and converting the media input into one or more media outputs. The method may include: providing multiple video sources, each of the multiple video sources generating a corresponding video stream; providing a microservice server including a microservice adapted to perform at least one of decoding, encoding, scaling, routing, and transmitting the corresponding video stream; providing multiple computing resources available to the microservices; and providing a resource management subsystem for orchestrating the microservices according to project parameters of one or more media projects, allocating and coordinating the multiple computing resources required by the microservices for the one or more media projects, and outputting corresponding media processed by the microservices for the one or more media projects.

[0023] Method embodiments of the present invention may also incorporate the various features and functionality of the system embodiments discussed above. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be described below with reference to the accompanying drawings, wherein like reference numerals represent like elements, and:

[0025] Figure 1 Demonstrate state-of-the-art systems for media processing;

[0026] Figure 2 shows a high-level block diagram of a media processing system according to an example embodiment of the present invention;

[0027] Figure 3 A block diagram showing a media processing system according to an example embodiment of the present invention;

[0028] Figure 4 A process flow diagram showing an example embodiment of a media processing system according to the present invention; and

[0029] Figure 5 An example user interface of a media processing system according to an example embodiment of the invention is shown. DETAILED DESCRIPTION

[0030] The following detailed description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the present invention. Instead, the following detailed description of the exemplary embodiments will provide those skilled in the art with an illustrative description of embodiments for implementing the present invention. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present invention as set forth in the appended claims.

[0031] The present invention relates to a software-based method and system for processing, routing, and distributing media. The system is capable of accepting any media input format and distributing it to one or more destinations in a variety of formats. The system dynamically leverages both cloud and on-premises resources and manages them through an object-based control system. Its architecture includes microservice servers, computing resources, agents, resource management, task services, and orchestration services to optimize operational efficiency and flexibility in media handling.

[0032] The present invention also relates to a user interface or API for a media processing system that is specifically designed to simplify the operations required for video processing and video routing. The present invention provides an intuitive user interface that minimizes complexity, allowing operators to focus on content and project management. Using the API simplifies the implementation of media processing software. The process involves defining a 'project object' to which users can add various output formats and destinations. Once created, users can simply drag and drop input sources (e.g., live or recorded media files) into the project object. The API can be used to connect input objects (media files) to the project object. The platform then automatically obtains the necessary resources, decodes, scales, and encodes the media signal into the specified format required for the destination defined by the project object, and outputs the encoded signal to the corresponding destination in the defined format.

[0033] The method and system of the present invention are suitable for:

[0034] 1. Accept any media input format and convert it for delivery to multiple destinations in various media formats;

[0035] 2. Dynamically utilize computing resources, whether cloud-based or on-premises, based on the needs of media projects;

[0036] 3. Simplify operations through an object-based control system, by enabling a process in which users create 'project objects' as output objects, specify the desired output format and destination, and reduce the need for dedicated resources for each function, thereby saving costs;

[0037] 4. Provide an intuitive user interface that simplifies media routing and conversion processes (e.g., decoding and re-encoding);

[0038] 5. Provide users with the ability to drag and drop input sources into these project objects;

[0039] 6. Implement automatic resource acquisition and signal routing, including decoding, scaling, and encoding, to meet the specifications defined by the project objectives; and

[0040] 7. More focus on content and project management due to reduced operational complexity.

[0041] Figure 2A high-level block diagram of the present invention is shown. Figure 2 As shown in FIG, the system 20 of the present invention accepts multiple input video sources 18 in any type of video format. The system 20 then determines which input source will be routed to one or more specific outputs 21 based on the project's requirements and decodes and re-encodes the input source as needed for the designated outputs 21. Thus, video sources 18 provided in any encoding format can be output to one or more destinations 21 in any corresponding encoding format. For example, media inputs and media outputs can include any of SDI, RTMP, RTSP, TS, SPTS, MPTS, HLS, SRT, NDI, Zyxi, YouTube, Facebook, TikTok, Zoom, TVU Grid, TVU Anywhere, TVU Partyline, TVU RPS, and the like.

[0042] Figure 3 A block diagram of an example embodiment of a system 20 according to the present invention is shown. The system 20 may include multiple video sources 18, each of which generates a corresponding video stream. A microservice server 25 is provided, which includes a microservice 23 adapted for at least one of decoding, encoding, scaling, routing, and transmitting the corresponding video streams. The system also includes multiple computing resources 24 available to the microservices 23. A resource management subsystem 26 is provided for orchestrating the microservices 23 according to project parameters of one or more media projects, allocating and coordinating the multiple computing resources 24 required by the microservices 23 for the one or more media projects, and outputting the corresponding media processed by the microservices 23 for the one or more media projects.

[0043] Agents 22 can be installed on computing resources 24 and analyze, monitor, and report the resources available at their corresponding computing resources 24 to the resource management subsystem 26. Specifically, agents 22 can determine the status and available capacity of their corresponding computing resources 24 for the microservices 23 required for the current media project. Computing resources 24 can utilize or include cloud-based services 28 (e.g., AWS, AZURE, Google Cloud) and / or be virtual machines or hardware based on physical computer devices located locally or off-premises. Agents 22 can each include a process running on each computing resource 24. Agents 22 (or separate programs running on or communicating with each computing resource 24) are also responsible for downloading any necessary applications to their corresponding computing resources 24 for media processing. Alternatively, applications can be Dockerized and accessed from a central server.

[0044] The microservice 23 may be hardware and / or software suitable for at least one of decoding, encoding, scaling, routing, and transmission of the corresponding video stream, any of which may be cloud-based or located on a physical computer system.

[0045] Media items are object-based media items (combined with Figure 4 The project object defines the project parameters of a media project. The project parameters include identification information of one or more video sources to be included in the media project, one or more output destinations, one or more media output destination encoding formats, and corresponding video and audio bit rates.

[0046] Based on the project parameters, the resource management subsystem 26 identifies one or more microservices 23 required for each of the media projects based on the one or more video sources 18 and one or more output destinations identified in the project object.

[0047] The resource management subsystem 26 defines one or more process workflows for implementing project parameters for each of the media projects through the interconnected microservices 23 and initiates the identified microservices 23 to implement the media projects.

[0048] The resource management subsystem 26 may include: an orchestration service 30, which is used to determine the microservices 23 required to complete each of the media projects based on project parameters and to determine the interconnections between the microservices 23; a central resource management system 31, which monitors and manages computing resources 24 based on communications from the agent 22; and a task allocation service 32, which is used to initiate one or more process workflows for each of the media projects based on information provided by the orchestration service 30 and by the central resource management system 31.

[0049] Based on communications from agents 22, a central resource management system 31 oversees and manages the use of all computing resources 24. The central resource management system 31 can also directly monitor the status of computing resources 24. The central resource management system 31 provides a comprehensive overview of all available computing resources 24 and their current status, thereby facilitating the efficient allocation of computing resources 24 to the microservices 23 required for a media project. An orchestration service 30 plans and connects functional modules (e.g., microservices 23 such as decoders, encoders, video scalers, routers, transmitters, etc.) based on project requests. The orchestration service 30 analyzes incoming project requests, creates a task list, and determines the necessary interconnections between the various microservices 23 required to complete the media project. The orchestration service 30 can also determine the optimal allocation of computing resources 24 required to complete the media project, including dynamic resource allocation based on real-time changes in resource availability. A task allocation service 32 initiates the process workflow for the media project and, based on data from the orchestration service 30 and the central resource management system 31, assigns microservices 23 to appropriate computing resources 24 based on the resource availability and requirements communicated by the orchestration service 30.

[0050] The various components of system 20 may be located locally and / or at different physical and cloud-based locations, and may be connected via a combination of wired and / or wireless networks (e.g., the Internet, intranet, extranet, EPN, VPN, LAN, WLAN, etc.) now known or yet to be developed.

[0051] Figure 4 An example embodiment of a process flow according to the present invention is shown. First, a project object 40 is defined. The project object 40 identifies various project parameters of a media project (e.g., a television program, a live broadcast, a sporting event, an IP video stream, etc.). The project parameters may include a destination or output address for the media project, and one or more of an encoding format and corresponding parameters, resolution, video bit rate, and audio bit rate corresponding to the destination address of the media project. Since multiple destinations can be defined in the project object 40, the corresponding encoding format for each of such destinations is also defined as needed. For example, a program object 40 may define the configuration of the output of a particular program (e.g., The Morning Show), for example, with three outputs, one output may be a 1080i SDI output, another output may be an RTMP 1080i 5Mbps, and the last output may be an HLS1080p output.

[0052] Identification information for one or more video sources 18 required for a media project can be added to a project object 40 via a drag-and-drop user interface 36 associated with the system front end. Object microservices 42 provide information about the project object 40 and the input objects (video sources 18) to the system back end 44. The project object 40 then automatically identifies the appropriate decoder (e.g., one of the microservices 23) for decoding the video source and the appropriate encoder (e.g., another of the microservices 23) associated with the destination defined by the project object 40. The system back end 44 then sends the selected input objects along with the project object 40 to the orchestration service 30. The project object 40 can be saved and reused for recurring media projects (e.g., daily or weekly television programs, network football games, etc.).

[0053] As described above, the orchestration service 30 will determine the optimal allocation of microservices 23 (e.g., encoders, decoders, transmitters, etc.) for completing the media project, and based on information received from the resource management system 31 regarding resource availability (such as provided by the agent 22, as described above in conjunction with Figure 3(as discussed above) determines the computing resources 24 required and available to microservices 23. This determination can be made by orchestration service 30 based on predefined business rules for resource allocation. For example, if a project object 40 contains an output destination pointing to a specific SDI output, then the specific output process should reside on the physical hardware (resource 24) where the SDI output port is located. Input signals are then routed to the computing resource 24 containing the output process.

[0054] Orchestration service 30 may update system backend 44 with the current resource status and provide instructions to task allocation service 32 regarding which microservices 23 and computing resources 24 will be used to meet the parameters of project object 40. After decoding the media input and re-encoding it for the corresponding destination, microservice 23 will then output the encoded media (e.g., via the corresponding computing resource 24) to an appropriate distribution system 46 for distribution to one or more media output destinations. Such system 46 may include one or more of a satellite uplink, an over-the-air broadcast uplink, or an SDI router for distributing the media files to the appropriate destination. Distribution system 46 may also output IP video streams to various destinations.

[0055] Figure 5 An example embodiment of a user interface 36 according to the present invention is shown. The user interface is associated with the system front end. The user interface 36 may provide video sources 18 in a tiled format on a first portion 48 of a computer touch screen and provide an output portion containing output tiles corresponding to project objects 40 on a second portion 50 of the touch screen. With the present invention, a user can touch and drag any of the video sources 18 from the first portion 48 to a project object 40 on the second portion 50. For example, Figure 5 Selection of video source 18 ′ and its finger dragging from first portion 48 to second portion 50 is shown.

[0056] As discussed above, the project object 40 identifies various parameters of a media project (e.g., a television program, a live broadcast, a sporting event, etc.). The parameters may include a destination or output address for the media project, as well as one or more of an encoding format and corresponding parameters, resolution, video bitrate, and audio bitrate corresponding to the destination address of the media project. Since multiple destinations can be defined in a project object, the corresponding encoding format for each of such destinations is also defined.

[0057] The project object 40 then automatically identifies the appropriate microservice 23 for decoding the selected video source and the appropriate microservice 23 for encoding / re-encoding the video source for each of the destinations defined by the project object 40. The project objects 40 can be saved and reused for recurring media projects (such as daily or weekly TV shows, network football games, etc.).

[0058] When an input source 18 is added to a project object 40, the system automatically acquires the necessary microservices 23 and corresponding computing resources 24 and handles decoding, scaling, encoding, and signal routing to one or more output destinations (e.g., a TV channel, a YouTube channel, a streaming service, social media, etc.), as discussed above.

[0059] The present invention revolutionizes traditional methods of media routing and conversion, providing a simplified user-friendly interface and automated process that significantly reduces complexity and resource requirements in media production environments.

[0060] It should now be appreciated that the present invention provides advantageous methods and systems for media processing, routing, and distribution.

[0061] While the present invention has been described in conjunction with various illustrated embodiments, many modifications and adaptations can be made thereto without departing from the spirit and scope of the invention as set forth in the claims.

Claims

1. A media processing system capable of accepting any media input and converting the media input into one or more media outputs, comprising: a plurality of video sources, each of the plurality of video sources generating a corresponding video stream; a microservice server comprising a microservice adapted to at least one of decode, encode, scale, route, and transmit the corresponding video stream; a plurality of computing resources available for use by the microservices; and A resource management subsystem is configured to orchestrate the microservices according to project parameters of one or more media projects, allocate and coordinate the computing resources required by the microservices for the one or more media projects, and output corresponding media processed by the microservices for the one or more media projects.

2. The media processing system according to claim 1, wherein: The media item is an object-based media item; An item object defines the item parameters of the media item; and The project parameters include identification information for one or more of the video sources to be included in the media project, one or more output destinations, one or more media output destination encoding formats, and corresponding video and audio bit rates.

3. The media processing system according to claim 2, wherein: Based on the project parameters, the resource management subsystem identifies the microservices required for each of the media projects based on the one or more video sources and the one or more output destinations identified in the project object.

4. A media processing system according to claim 3, wherein the resource management subsystem defines one or more process workflows through interconnected microservices, the workflows being used to implement the project parameters of each of the media projects and to start the identified microservices to implement the media projects.

5. The media processing system of claim 4, wherein each of the plurality of computing resources comprises or utilizes one or more of local computing resources and cloud-based computing resources.

6. The media processing system of claim 4, wherein a resource agent is associated with each of the plurality of computing resources, the resource agent analyzing, monitoring, and reporting resource availability of the corresponding computing resource.

7. The media processing system according to claim 6, wherein the resource management subsystem comprises: an orchestration service for determining, based on the project parameters, which of the microservices is required to complete each of the media projects and for determining interconnections between the microservices; a central resource management system that monitors and manages the computing resources based on communications from the resource agents; and A task allocation service is used to initiate the one or more process workflows for each of the media items based on information provided by the orchestration service and by the central resource management system.

8. The media processing system of claim 7, wherein the orchestration service implements dynamic allocation of the computing resources through the task allocation service based on real-time changes in availability of the computing resources.

9. The media processing system of claim 7, wherein the video source is added to the project object via a drag-and-drop user interface.

10. The media processing system according to claim 7, wherein: The project object is provided to the object microservice; The object microservice provides information about the project object and identification of the video source of the media project to the system backend; The system backend provides the video source and the project object to the orchestration service.

11. The media processing system of claim 10, wherein the orchestration service updates the system backend with status information regarding usage and availability of the computing resources.

12. The media processing system of claim 2, wherein the item object is reusable for a repeatable media item.

13. The media processing system of claim 1, further comprising a plurality of distribution systems for distributing the media to one or more media output destinations.

14. The media processing system of claim 13, wherein the distribution system comprises one or more of a satellite uplink, an over-the-air broadcast system, an SDI router, and an IP streaming system.

15. The media processing system of claim 13, wherein the media input and the media output comprise one of SDI, RTMP, RTSP, TS, SPTS, MPTS, HLS, SRT, NDI, Zyxi, YouTube, Facebook, TikTok, Zoom, TVUGrid, TVU Anywhere, TVU Partyline, and TVU RPS.

16. A method for processing any media input and converting the media input into one or more media outputs, comprising: providing a plurality of video sources, each of the plurality of video sources generating a corresponding video stream; providing a microservice server comprising a microservice adapted to perform at least one of decoding, encoding, scaling, routing, and transmitting the corresponding video stream; Providing a plurality of computing resources that can be used by the microservices; and A resource management subsystem is provided for orchestrating the microservices according to project parameters of one or more media projects, allocating and coordinating the plurality of computing resources required by the microservices for the one or more media projects, and outputting corresponding media processed by the microservices for the one or more media projects.

Images