A heterogeneous hardware multi-board fusion resource scheduling system

Abstract

The present application discloses a heterogeneous hardware multi-board fusion resource scheduling system, including a task allocation unit, a receiving control unit, a multi-board processing unit, and a media terminal device. The task allocation unit is used to distribute conference video images and multimedia data processing tasks to the receiving control unit, and transmit the images to the multi-board processing unit through the network flow to sequentially complete the decoding, scaling, splicing, and encoding of the multimedia data, and finally display it on the terminal or media terminal device. The present application can be compatible with different heterogeneous media, shield the differences between heterogeneous hardware units, break through the isolation of heterogeneous resources, effectively allocate data to hardware units with low working rates for processing, and improve execution efficiency. At the same time, the present application is not subject to the physical limitations of the host interface, is suitable for large-scale operational meetings, and can be infinitely expanded as needed, solving the current limitations caused by single-board cards.

Description

Technical Field

[0001] The present application relates to a board resource scheduling system, which belongs to the field of resource allocation technology, and in particular to a heterogeneous hardware multi-board fusion resource scheduling system. Background Art

[0002] During video image processing, individual image processing tasks are usually assigned to a single media board for processing. Each hardware module completes its own task independently. Limited by the performance of a single board, it cannot complete the task when large tasks are required. Summary of the Invention

[0003] The present application provides a heterogeneous hardware multi-board fusion resource scheduling system that is compatible with different heterogeneous media boards, such as different graphics processors (GPUs), systems on a chip (SOCs), field programmable gate arrays (FPGAs), etc.; it can split a single large-scale media processing task and assign it to different boards, thereby improving media processing capabilities; and it can realize heterogeneous hardware multi-board intelligent resource scheduling. The so-called heterogeneous media boards refer to multiple media boards used for scheduling. The multiple media boards can be produced by different manufacturers or different models of the same manufacturer, and can be used to process code streams.

[0004] The heterogeneous hardware multi-board fusion resource scheduling system includes:

[0005] A task allocation unit is used to split the overall task into different multimedia data tasks, decompose the data code stream of each multimedia data task into multiple groups of encoding and decoding programs, and transmit them to different receiving modules in the receiving control unit as needed;

[0006] Receiving control unit, used to send the received data stream to the adapted board for processing;

[0007] A multi-card processing unit is used to decode the data stream according to the signaling, scale it according to the terminal display requirements, integrate and splice all scaled streams to be displayed on the terminal, re-encode them and send them to the media terminal device;

[0008] Media terminal device, used to send its own video stream to the board and receive and display the images sent by the corresponding boards.

[0009] Optionally, the task allocation unit is configured to send the task information and the data code stream to the receiving processing unit according to a mapping algorithm, so that the data code stream is forwarded to a corresponding board for processing.

[0010] Optionally, the mapping algorithm includes: determining the number of boards required for code stream encoding and decoding according to the conference mode, and determining the style of each board according to the number of boards, and sending the number and style information to the receiving control unit to forward the corresponding code stream to its corresponding board for processing.

[0011] Optionally, the receiving control unit includes a plurality of receiving modules, each of which is used to obtain a plurality of multimedia data tasks to be processed and multimedia data to be processed.

[0012] Optionally, the multi-board processing unit is used to allocate board resources according to a scheduling algorithm.

[0013] Optionally, the scheduling algorithm includes at least one of the following scheduling modes: manual specified allocation, balanced mode allocation, and single board priority mode allocation.

[0014] Preferably, the manual designated allocation means setting a priority according to the importance of the meeting, and for a meeting with a high priority, designating a board to use resources first.

[0015] Preferably, the balanced mode allocation refers to evenly allocating board resources for data tasks to be processed.

[0016] Preferably, the single board priority mode means that when the system is idle or the workload is small (eg, there are boards remaining in the system), single boards are used first.

[0017] Optionally, the system also includes several terminals, each terminal sends an image to a task allocation unit, the task allocation unit summarizes the data code streams of the images of each terminal, determines the multimedia data task based on the total code stream processing task and determines the corresponding board card for processing, sends the data code stream to the determined board card through the receiving control unit for data code stream processing, and sends the processed code stream to the designated media terminal.

[0018] Optionally, the multi-board processing unit includes:

[0019] a decoding subunit, configured to control a plurality of decoding modules to decode the data code stream of the multimedia data task to obtain a plurality of decoded multimedia data, wherein the plurality of decoding modules, the plurality of multimedia data to be processed and the plurality of decoded multimedia data correspond one to one;

[0020] a scaling control subunit, configured to control a plurality of scaling modules to scale the decoded multimedia data to obtain scaled multimedia data sets, each scaled multimedia data set including at least one scaled multimedia data set, and a one-to-one correspondence between the plurality of scaling modules, the plurality of decoded multimedia data sets, and the plurality of scaled multimedia data sets;

[0021] a splicing control subunit, configured to control a plurality of splicing modules to splice the scaled multimedia data included in the multimedia data set to obtain a plurality of spliced ​​multimedia data, wherein the plurality of splicing modules correspond to the plurality of spliced ​​multimedia data in a one-to-one manner;

[0022] an encoding control subunit, configured to control the plurality of encoding modules to encode the spliced ​​multimedia data to obtain a plurality of encoded multimedia data, wherein the plurality of encoding modules, the plurality of spliced ​​multimedia data, and the plurality of encoded multimedia data correspond one to one;

[0023] The sending subunit is used to control a plurality of sending modules to send the encoded multimedia data to a plurality of media terminals for display. The plurality of sending modules, the plurality of encoded multimedia data and the plurality of media terminals correspond one to one.

[0024] This application first uses the scaling control subunit to scale the required data to a specified size, then splices and encodes it according to the needs, and finally compresses all the decoded data streams and flows them into the corresponding splicing module.

[0025] Optionally, the decoding module is used to decode the multimedia data to be processed based on the recognition result to obtain decoded multimedia data. The recognition result is the data to be decoded.

[0026] Optionally, the splicing module is configured to obtain scaled multimedia data sets output by a plurality of the scaling modules, and splice the scaled multimedia data sets to obtain spliced ​​multimedia data.

[0027] Optionally, the media terminal device includes at least one media terminal, and the media terminal device is used to display the spliced ​​multimedia data as needed; and transmit the spliced ​​multimedia data to a designated terminal after encoding.

[0028] The beneficial effects of this application include:

[0029] 1) The heterogeneous hardware multi-board fusion resource scheduling system provided in this application can be used to implement large-scale tasks. The overall task is received through the unified interface of the task allocation unit. Based on the resource characteristics of each heterogeneous hardware unit in the heterogeneous hardware architecture (i.e., the amount of tasks carried by each hardware), the scheduling of the overall task is determined, and the subtasks contained in the task are distributed to the adapted heterogeneous media for execution through a unified hardware and application interface. Since the task is received through a unified interface and its subtasks are distributed through a unified hardware and application interface, it is compatible with different media, shielding the differences between heterogeneous hardware units, breaking through the isolation of heterogeneous resources, and effectively allocating them to hardware units with low working rates for processing, thereby improving execution efficiency.

[0030] 2) This application uses multiple boards for resource scheduling, avoiding the huge interface performance requirements of transmitting GB / s data between boards in traditional multi-card systems. Unlike single-board cards, multi-board cards can decode one channel of video data multiple times and decode it according to different scheduling requirements as needed. In this process, the decoded original video data will not be transmitted between boards. Instead, different code streams will flow into the most suitable board for processing according to instructions, and then all processed data will be encoded into the required format again and sent, ultimately presenting different picture effects on the terminal display. This design avoids the transmission of decoded original video data between boards, greatly improving transmission efficiency and scalability.

[0031] 3) This application avoids the physical limitations of host interfaces on the number of boards used in data processing, improving data processing efficiency. It also expands capacity, enabling large-scale conferences with increased attendees. This application significantly improves performance over existing technologies, enabling unlimited capacity expansion as needed, addressing the current limitations of single-board systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the framework of the current single-board scheduling multimedia data processing method;

[0033] Figure 2 This is a schematic diagram of the overall framework of a heterogeneous hardware multi-board fusion resource scheduling system in one embodiment of the present application;

[0034] Figure 3 This is a flow chart of the execution of a heterogeneous hardware multi-board fusion resource scheduling system in one embodiment of the present application;

[0035] Figure 4 This is a schematic diagram of the image processing flow in a conference mode for this application;

[0036] Figure 5 for Figure 4 Schematic diagram of multi-board architecture based on mapping algorithm in this mode;

[0037] Figure 6 This is a flow chart of a multi-board resource scheduling algorithm in one embodiment of the present application. DETAILED DESCRIPTION

[0038] The present application is described in detail below with reference to embodiments, but the present application is not limited to these embodiments.

[0039] See Figure 1, which shows the current single-board scheduling multimedia data processing process, including: transmitting the conference video image to the decoding module of the decoding unit through the network stream, decoding the multimedia data, passing the decoded multimedia data to the scaling unit for scaling processing, and then controlling the splicing module in the splicing unit to splice the scaled multimedia data included in the multimedia data set, and then using the encoding module in the encoding unit to encode the spliced ​​multimedia data, and finally sending the multimedia data to the media terminal device through the sending module of the sending unit for terminal display.

[0040] However, using a single board for data processing has its drawbacks: a video channel on a single board can only be decoded once, and the picture can only be scaled for display, that is, the picture seen is the same, but the size is different. This cannot meet the display requirements of the picture effects of multi-venue meetings. In addition, since the number of single boards is physically limited by the host interface, the data processing efficiency is also low.

[0041] This application uses multiple boards for data processing and provides a resource scheduling system and method. In one embodiment, the heterogeneous hardware multi-board fusion resource scheduling system includes:

[0042] The task allocation unit is used to split the overall task into different multimedia data tasks, and then the data code stream of each multimedia data task is decomposed into multiple groups of encoding and decoding programs, which are transmitted to different receiving modules in the receiving control unit as needed;

[0043] Receiving control unit, used to send the received data stream to the adapted board for processing;

[0044] A multi-card processing unit is used to decode the data stream according to the signaling, scale it according to the terminal display requirements, integrate and splice all scaled streams to be displayed on the terminal, re-encode them and send them to the media terminal device;

[0045] Media terminal device, used to send its own video stream to the board and receive and display the images sent by the corresponding boards.

[0046] In one embodiment, the multi-board processing unit includes multiple decoding modules, multiple scaling modules, multiple splicing modules, multiple encoding modules and multiple sending modules, which are used to execute the following process: obtaining multiple multimedia data to be processed; controlling multiple decoding modules to decode the multiple multimedia data to be processed to obtain multiple decoded multimedia data, wherein the multiple decoding modules, the multiple multimedia data to be processed and the multiple decoded multimedia data correspond one to one; controlling multiple scaling modules to scale the multiple decoded multimedia data to obtain multiple scaled multimedia data sets, each scaled multimedia data set includes at least one scaled multimedia data, and the multiple scaling modules, the multiple decoded multimedia data, and the multiple encoding modules to decode the multiple multimedia data. The multimedia data corresponds one-to-one to the multiple scaled multimedia data sets; the multiple splicing modules are controlled to splice the scaled multimedia data included in the multiple scaled multimedia data sets to obtain multiple spliced ​​multimedia data, and the multiple splicing modules correspond one-to-one to the multiple spliced ​​multimedia data; the multiple encoding modules are controlled to encode the multiple spliced ​​multimedia data to obtain multiple encoded multimedia data, and the multiple encoding modules, the multiple spliced ​​multimedia data, and the multiple encoded multimedia data correspond one-to-one; the multiple sending modules are controlled to send the multiple encoded multimedia data to the multiple media terminals, and the multiple sending modules, the multiple encoded multimedia data, and the multiple media terminals correspond one-to-one.

[0047] like Figure 2 As shown, when the system receives different multimedia data tasks, the task allocation unit splits the received code stream into multiple sets of codec programs, and inputs the code stream into the corresponding board for data processing according to the instructions, where each board uses Figure 1 The complete architecture of the single-board card is shown. The processed data is then displayed on the terminal according to the signaling. The receiving module in the receiving control unit typically receives data at 100kbps-10mbps, which is decoded into high-capacity 5Gbps data. This data is then scaled and reassembled as needed to form 100kbps-10mbps data, which is then encoded and displayed on the media terminal.

[0048] like Figure 3 The overall process framework diagram shown in the figure involves two algorithms:

[0049] Algorithm 1: Mapping Algorithm

[0050] According to the conference mode, the designated board can specify different encoding modules for one-to-one processing after different code stream input.

[0051] In one embodiment, the mapping algorithm includes: first determining a conference mode, then determining how many boards are required for this mode, and determining the style of each board based on the number of boards. After determination, the transceiver module is notified to forward the corresponding code stream to each corresponding board for execution.

[0052] Algorithm 2: Scheduling algorithm.

[0053] There are three scheduling methods:

[0054] 1. Manually specify the corresponding board to process the corresponding code stream;

[0055] 2. Set the balanced mode to enable task resource allocation;

[0056] 3. Allocate resources based on the method of prioritizing single boards.

[0057] The advantage of the manually specified allocation algorithm is that it sets priorities for important meetings among multiple meetings, so that resources can be used first for meetings; the advantage of the balanced mode is that it evenly distributes board resources, which significantly increases the number of participants in a single meeting; the advantage of the single-board priority mode is that it fully utilizes the utilization rate of the single board and can hold multiple meetings at the same time.

[0058] The conference mode can have M terminals, N styles and other modes, such as Figure 4 As shown, in one mode (taking the scenario of eight terminal venues as an example), the image processing process is as follows: terminal one is identified as the speaker, and the image seen by terminal one is the image display of the branch venue terminals of terminals two, three, four, and five, and the image of terminal one will be sent to media terminal 1; the image seen by terminals two, three, four, and five in the branch venues is the image sent by media terminal 1 corresponding to board one, and the images of terminals two, three, four, and five will be displayed on media terminals 2, 3, 4, and 5; terminals six, seven, and eight also see the image of the speaker (that is, the image of media terminal 1), and according to demand, terminals six, seven, and eight do not need to be presented on the media terminal.

[0059] Figure 5 Figure 4 is a schematic diagram of a multi-board architecture based on a mapping algorithm in mode 4, where board one receives the code streams from terminals one, two, three, four, and five, and sends the processed data to the media terminal as needed; board two receives the code stream from terminal one, and sends the final processed data to terminals six, seven, and eight for terminal display.

[0060] Figure 6 This is a flow chart of a multi-board resource scheduling algorithm in one embodiment. The specific flow is as follows:

[0061] The overall task is split into separate data streams, with images transmitted to different receiving modules as needed. This process is then broken down into multiple horizontal groups. Since data cannot be transferred between boards, received streams are processed by the corresponding boards according to a schedule, with arbitrary allocations between them based on actual needs. The task dispatch unit receives the network data stream and passes it to the receiving module, where it is then distributed to different boards for decoding as needed. Streams cannot be processed across boards; only unprocessed streams are passed to one board. Each board processes its own stream, and the decompressed stream cannot be distributed. The splicing module in the splicing unit reassembles and re-encodes all the scaled streams for the two fixed outputs. Another board then decodes the requested stream, scales it, splices it, and encodes it for transmission. Because different streams have different terminal display requirements, multiple scaling modules process the decoded data. Different display formats require different scaling modules to compress the streams at specific ratios.

[0062] For example, the receiving module 1 processes the code stream containing image 1, scales it as required after decoding, and then enters the splicing module to splice it into data containing image 1, which is then encoded and sent to the required terminal for display.

[0063] Receiving module 2 processes the code stream containing images 2, 3, 4, and 5, scales it as required after decoding, and then enters the splicing module to splice it into data containing images 2, 3, 4, and 5, which is then sent to the required terminal for display after encoding.

[0064] The scheduling subunit within the multi-board processing unit includes three scheduling methods: manually specifying the corresponding board to process the corresponding bitstream; enabling balanced mode for task resource allocation; and allocating resources based on a single-board priority approach. Therefore, within a single board, data with the same requirements can be horizontally grouped, scaled by a designated scaling module, and then sent to different splicing units for splicing. Therefore, in one embodiment, the workflow for receiving modules 3, 4, and 5 is as follows: receiving module 3 receives data containing images 3 and 5, decodes and scales it, and then splices it with the scaled data containing images 2 and 4 from receiving modules 2 and 4, forming data containing images 2, 3, 4, and 5, which is then encoded and sent.

[0065] The receiving modules 6, 7 and 8 only need to receive the data stream containing the image 1 as required.

[0066] The difference between a multi-board and a single-board is that a single-board decodes a single video channel only once, scaling it to different displays as needed. The resulting image remains consistent, just at different sizes. The significance of a multi-board is that a single video channel can be decoded multiple times. Each board needs to decode according to different scheduling requirements, and different streams flow to the most suitable board for processing as instructed. All processed data is then re-encoded into the required format for transmission, ultimately presenting different visual effects on the terminal display.

[0067] This application can avoid the physical limitations of host interfaces on the number of boards used in data processing, improving data processing efficiency. This increased capacity allows for large-scale conference operations and increased attendance, significantly improving performance. This application allows for unlimited capacity expansion as needed, addressing the current limitations of single-board systems.

[0068] The above descriptions are merely a few embodiments of the present application and do not constitute any form of limitation to the present application. Although the present application discloses the preferred embodiments as above, they are not intended to limit the present application. Any technical personnel familiar with the present profession, without departing from the scope of the technical solution of the present application, using the technical content disclosed above to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A heterogeneous hardware multi-board fusion resource scheduling system, characterized in that: The system includes: A task allocation unit is used to split the overall task into different multimedia data tasks, decompose the data code stream of each multimedia data task into multiple groups of encoding and decoding programs, and transmit them to different receiving modules in the receiving control unit as needed; The receiving control unit is used to send the received data code stream to the adapted heterogeneous board for processing; Media terminal device, used to send its own video stream to the board and receive and display the images sent by the corresponding heterogeneous boards; A multi-board processing unit, comprising: a decoding subunit, configured to control a plurality of decoding modules to decode the data code stream of the multimedia data task to obtain a plurality of decoded multimedia data, wherein the plurality of decoding modules, the plurality of multimedia data to be processed and the plurality of decoded multimedia data correspond one to one; a scaling control subunit, configured to control a plurality of scaling modules to scale the decoded multimedia data to obtain scaled multimedia data sets, each scaled multimedia data set including at least one scaled multimedia data set, and a one-to-one correspondence between the plurality of scaling modules, the plurality of decoded multimedia data sets, and the plurality of scaled multimedia data sets; a splicing control subunit, configured to control a plurality of splicing modules to splice the scaled multimedia data included in the multimedia data set to obtain a plurality of spliced ​​multimedia data, wherein the plurality of splicing modules correspond to the plurality of spliced ​​multimedia data in a one-to-one manner; an encoding control subunit, configured to control the plurality of encoding modules to encode the spliced ​​multimedia data to obtain a plurality of encoded multimedia data, wherein the plurality of encoding modules, the plurality of spliced ​​multimedia data, and the plurality of encoded multimedia data correspond one to one; The sending subunit is used to control a plurality of sending modules to send the encoded multimedia data to a plurality of media terminals for display. The plurality of sending modules, the plurality of encoded multimedia data and the plurality of media terminals correspond one to one.

2. The heterogeneous hardware multi-board fusion resource scheduling system according to claim 1 is characterized in that: The task allocation unit is used to send the task information and the data code stream to the receiving processing unit according to the mapping algorithm, so that the data code stream is forwarded to the corresponding board for processing.

3. The heterogeneous hardware multi-board fusion resource scheduling system according to claim 2 is characterized in that: The mapping algorithm includes: determining the number of boards required for code stream encoding and decoding according to the conference mode, and determining the style of each board according to the number of boards, and sending the number and style information to the receiving control unit to forward the corresponding code stream to its corresponding board for processing.

4. The heterogeneous hardware multi-board fusion resource scheduling system according to claim 1, characterized in that: The receiving control unit includes a plurality of receiving modules, each of which is used to obtain a plurality of multimedia data tasks to be processed and multimedia data to be processed.

5. The heterogeneous hardware multi-board fusion resource scheduling system according to claim 1, characterized in that: The multi-board processing unit is used to allocate board resources according to a scheduling algorithm.

6. The heterogeneous hardware multi-board fusion resource scheduling system according to claim 5, characterized in that: The scheduling algorithm includes at least one of the following scheduling modes: manual assignment, balanced mode assignment, and single board priority mode assignment; Manual allocation means setting priorities according to the importance of the meetings. For meetings with high priorities, the designated board will be given priority to use resources. The balanced mode allocation means evenly allocating board resources for data tasks to be processed; The single board priority mode means that when the system is idle or the workload is small, the single board is used first.

7. The heterogeneous hardware multi-board fusion resource scheduling system according to claim 1, characterized in that: The system also includes several terminals, each of which sends an image to a task allocation unit. The task allocation unit summarizes the data code streams of the images of each terminal, determines the multimedia data task and the corresponding board for processing based on the total code stream processing task, sends the data code stream to the determined board through the receiving control unit for data code stream processing, and sends the processed code stream to the designated media terminal.

8. The heterogeneous hardware multi-board fusion resource scheduling system according to claim 1, characterized in that: The media terminal device includes at least one media terminal, and the media terminal is used to display the spliced ​​multimedia data as needed; and transmit the spliced ​​multimedia data to a designated terminal after encoding.

9. The heterogeneous hardware multi-board fusion resource scheduling system according to claim 1, characterized in that: The decoding module is used to decode the multimedia data to be processed based on the recognition result to obtain decoded multimedia data.

10. The heterogeneous hardware multi-board fusion resource scheduling system according to claim 1, characterized in that: The splicing module is used to obtain the scaled multimedia data sets output by the plurality of scaling modules, and splice the scaled multimedia data sets to obtain spliced ​​multimedia data.

Images