Distributed audio and video real-time processing system based on edge calculation

Abstract

The invention discloses a distributed audio and video real-time processing system based on edge computing, relates to the technical field of audio and video data processing, and solves the technical problems that when large-scale and high-concurrency processing requirements are met, data transmission delay is obvious, the load of a central server is too heavy, and user experience is affected. According to the invention, the resource allocation processing module synthesizes edge node configuration and sub-task demand calculation resources, the sub-task demand calculation resources are sorted and allocated from large to small, the resource utilization efficiency is improved, and the secondary allocation processing module processes independent matching signals and analyzes the adjacency of unallocated and allocated sub-tasks, so that the resource utilization efficiency is improved. According to the method, the sub-tasks are distributed according to the priority and the corresponding edge nodes of the distributed sub-tasks in sequence, the task coherence and collaboration are improved, according to a secondary matching analysis signal, the undistributed sub-tasks are classified and sorted according to the priority, the same-level tasks are matched with the edge nodes from large to small according to demand computing resources, different-level tasks are matched alternately, and it is ensured that the high-priority tasks are processed preferentially.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video data processing, and in particular to a distributed audio and video real-time processing system based on edge computing. Background Art

[0002] In the current wave of digitalization, audio and video application scenarios are becoming increasingly diverse. From daily video conferencing and online live broadcasts to smart security monitoring and remote medical consultations in professional fields, strict requirements are placed on real-time audio and video processing.

[0003] The patent application with publication number CN104349205B discloses an audio and video processing method and system, which includes an audio and video conversion module, an audio and video mixing module, an audio and video cross module, an output module and / or a remote transmission module connected in sequence; the audio and video conversion module converts the input video signal into video data of a preset format and preset resolution, converts the input audio signal into audio data, and outputs the video data and audio data to the audio and video mixing module; the audio and video mixing module inserts audio data into the blanking area of ​​the video data line to obtain audio and video data, converts the audio and video data into serialized data and outputs it to the audio and video cross module; the audio and video cross module crosses the serialized data; the output module outputs the cross-linked data, or the remote transmission module converts and encodes the cross-linked data in a format, and outputs the encoded data through a network.

[0004] However, the traditional centralized cloud computing architecture aggregates all audio and video processing tasks to the central server. When faced with large-scale, high-concurrency processing needs, it exposes many drawbacks. Network bandwidth becomes a bottleneck, data transmission delays are significant, the central server is overloaded, and problems such as freezes and packet loss often occur, seriously affecting the user experience. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a distributed audio and video real-time processing system based on edge computing, which solves the problem of significant data transmission delay and excessive load on the central server, which affects the user experience when facing large-scale and high-concurrency processing needs.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a distributed audio and video real-time processing system based on edge computing, comprising:

[0007] The processing information collection module is used to collect audio and video information and edge node information, and transmit the two to the processing information analysis module;

[0008] The processing information analysis module is used to analyze the acquired audio and video information, divide the audio and video information into multiple groups of subtasks, and calculate the required computing resources corresponding to the subtasks. Then, the free resources of different edge nodes are obtained according to the edge node information, and the processing information is generated and transmitted to the resource allocation processing module.

[0009] The resource allocation processing module is used to analyze the acquired processing information, allocate and process the subtasks by comprehensively considering the node configuration of the edge nodes and the required computing resources of the subtasks to obtain allocation information, and generate a judgment signal according to the relationship between the number of unallocated subtasks and the total number of edge nodes, and transmit the judgment signal to the secondary allocation processing module;

[0010] The secondary allocation processing module is used to process the obtained judgment signal. For the single matching signal, the allocation processing is performed according to the priority of the unassigned subtask to generate resource allocation information. For the secondary matching analysis signal, the task classification information is obtained by classifying the unassigned subtask according to the priority of the unassigned subtask, and the resource allocation information is generated by combining the corresponding demand calculation resources, and the resource allocation information is transmitted to the real-time processing information output module at the same time;

[0011] The real-time processing information output module is used to display the acquired resource allocation information to the corresponding operator.

[0012] As a further solution of the present invention, the specific manner in which the processing information analysis module calculates the required computing resources of the subtask is:

[0013] Acquire audio and video information, and divide the audio and video information into subtasks and label them as n, where n=1, 2, ..., m, where m represents the number of subtasks. Then analyze the required computing resources for subtask n, obtain the resolution and frame rate information corresponding to subtask n, and calculate the amount of data that subtask n needs to process per unit time. Substitute the obtained resolution and frame rate information into the formula Qn=Fn×Zn×Rn to calculate the amount of processed data Qn corresponding to subtask n per unit time, where Fn is the resolution of the subtask, Zn is the frame rate of the subtask, and Rn is the byte corresponding to the pixel. Then obtain the number of operations corresponding to subtask n, and estimate the required computing resources corresponding to subtask n based on the amount of data processed per unit time Qn.

[0014] As a further solution of the present invention, the specific manner in which the processing information analysis module generates processing information is:

[0015] Then, all edge nodes are obtained and recorded as i, and i=1, 2, ..., j, where j represents the number of edge nodes, and the idle computing resources corresponding to the edge node i are obtained, the required computing resources of the subtask and the idle computing resources of the edge node are combined to obtain processing information, and then it is transmitted to the resource allocation processing module.

[0016] As a further solution of the present invention, the specific manner in which the resource allocation processing module analyzes the processing information is as follows:

[0017] Obtain the subtask required computing resources and edge node idle computing resources in the processing information, then obtain the configuration requirements corresponding to the subtask, and obtain the node configuration of the edge node. At the same time, filter the edge nodes based on the subtask configuration requirements and record them as filtered edge nodes. Then, use the filtered edge nodes corresponding to the maximum idle computing resources as the allocation object for allocation processing to generate allocation information.

[0018] As a further solution of the present invention, the specific manner in which the resource allocation processing module generates the determination signal is:

[0019] According to the obtained allocation information, the unassigned subtasks are obtained, and the number of all edge nodes is obtained at the same time, and it is determined whether the unassigned subtasks can complete the single matching requirement. If it can be completed, a separate matching signal is generated. Otherwise, if it cannot be completed, a secondary matching analysis signal is generated, and both are transmitted to the secondary allocation processing module at the same time.

[0020] As a further solution of the present invention, the specific manner in which the secondary allocation processing module analyzes the individual matching signals to generate resource allocation information is as follows:

[0021] Analyze the individual matching signals in the judgment signal, obtain all the unassigned subtasks and assigned subtasks, and analyze the adjacency between the unassigned subtasks and the assigned subtasks, obtain the tasks with adjacency, and then obtain the priority of the unassigned subtasks in the acquired adjacent tasks, and allocate them in ascending order according to the edge node number sequence corresponding to the assigned subtasks, and generate resource allocation information.

[0022] As a further solution of the present invention, the specific manner in which the secondary allocation processing module processes the secondary matching analysis signal to generate resource allocation information is:

[0023] Get all the unassigned subtasks, determine the priorities of the unassigned subtasks, and sort them from large to small according to their priorities. Then get the unassigned subtasks with the same priority and record them as tasks of the same level. At the same time, select and assign edge nodes according to the priorities of the tasks of the same level. Get all the tasks of the same level, and get the corresponding required computing resources in the tasks of the same level. Then match the required computing resources with the edge nodes and generate resource allocation information.

[0024] The present invention provides a distributed audio and video real-time processing system based on edge computing. Compared with the prior art, it has the following beneficial effects:

[0025] The present invention uses a resource allocation processing module to comprehensively calculate resources based on edge node configuration and subtask requirements, screens edge nodes based on subtask configuration requirements, and sorts and allocates computing resources from large to small based on subtask requirements, thereby improving resource utilization efficiency. The secondary allocation processing module is flexible in processing different judgment signals. For a single matching signal, the adjacency of unallocated and allocated subtasks is analyzed, and the edge nodes corresponding to the allocated subtasks are allocated in order according to priority, thereby improving task coherence and coordination. For the secondary matching analysis signal, the unallocated subtasks are classified and sorted by priority, and the same-level tasks are matched with edge nodes from large to small based on the required computing resources, and tasks of different levels are interspersed and matched, thereby ensuring that high-priority tasks are processed first and that resource allocation is more reasonable. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a block diagram of the system principle of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] For example, see Figure 1 The present application provides a distributed audio and video real-time processing system based on edge computing, including a processing information acquisition module, a processing information analysis module, a resource allocation processing module, a secondary allocation processing module and a real-time processing information output module, and combines Figure 1 It can be known that the above functional modules are electrically connected in a unidirectional manner.

[0029] The processing information collection module is used to collect audio and video information and edge node information, and transmit the two to the processing information analysis module.

[0030] The processing information analysis module is used to analyze the acquired audio and video information, divide the audio and video information into multiple groups of subtasks, and calculate the required computing resources corresponding to the subtasks. Then, the free resources of different edge nodes are obtained according to the edge node information, and the processing information is generated and transmitted to the resource allocation processing module.

[0031] The audio and video information is obtained, and the audio and video information is divided into subtasks and labeled as n, where n = 1, 2, ..., m, where m represents the number of subtasks, and the division here is based on the decomposition of the audio and video content, specifically including: video frame processing subtask, audio frame processing subtask, key frame extraction subtask and audio and video synchronization subtask, and then the required computing resources of subtask n are analyzed, and the specific analysis method is as follows:

[0032] Obtain the resolution and frame rate information corresponding to subtask n, and calculate the amount of data that subtask n needs to process per unit time. Substitute the obtained resolution and frame rate information into the formula Qn=Fn×Zn×Rn to calculate the amount of processed data Qn corresponding to subtask n per unit time, where Fn is the resolution of the subtask, Zn is the frame rate of the subtask, and Rn is the byte corresponding to the pixel. Then obtain the number of operations corresponding to subtask n, and estimate the required computing resources corresponding to subtask n in combination with the amount of data processed per unit time Qn. The specific calculation formula is required computing resources=number of operations×Qn. Similarly, calculate the required computing resources corresponding to all subtasks.

[0033] Then, all edge nodes are obtained and recorded as i, and i=1, 2, ..., j, where j represents the number of edge nodes, and the idle computing resources corresponding to the edge node i are obtained, the required computing resources of the subtask and the idle computing resources of the edge node are combined to obtain processing information, and then it is transmitted to the resource allocation processing module.

[0034] The resource allocation processing module analyzes the acquired processing information, allocates and processes the subtasks by comprehensively considering the node configuration of the edge nodes and the required computing resources of the subtasks to obtain allocation information, and generates a judgment signal based on the relationship between the unallocated subtasks and the total number of edge nodes, and transmits the judgment signal to the secondary allocation processing module.

[0035] Obtain the subtask demand computing resources and edge node idle computing resources in the processing information, then obtain the configuration requirements corresponding to the subtask, and obtain the node configuration of the edge node, and at the same time, filter the edge nodes based on the subtask configuration requirements, and specifically filter the configuration based on a single subtask, and in the analysis process, sort and analyze the subtask demand computing resources from large to small, and record them as filtered edge nodes, and then use the filtered edge node corresponding to the largest idle computing resource as the allocation object for allocation processing to generate allocation information;

[0036] For example, there are three subtasks, namely video encoding (denoted as subtask A), audio noise reduction (denoted as subtask B), and image recognition (denoted as subtask C), and their required computing resources are 80 units, 50 units, and 30 units respectively. The edge nodes are Node1, Node2, and Node3, and their idle computing resources are 90 units, 60 units, and 40 units respectively.

[0037] When screening, first consider subtask A. Since its required computing resources are the largest, 80 units, Node1's idle computing resources of 90 units meet the requirement, while Node2 and Node3 do not. Then look at subtask B, which requires 50 units of computing resources, which can be met by Node1 and Node2. For subtask C, Node1, Node2, and Node3 can all meet its required computing resources of 30 units.

[0038] After the screening is completed, the screening edge node corresponding to the maximum idle computing resource is used as the allocation object for allocation processing. Node1 has the largest idle computing resource, so subtask A is allocated to Node1 for processing.

[0039] According to the obtained allocation information, the unassigned subtasks are obtained, and the number of all edge nodes is obtained at the same time, and it is determined whether the unassigned subtasks can complete the single matching requirement, and the single matching requirement here is expressed as whether the number of edge nodes can match all unassigned subtasks one by one. If it can be completed, a separate matching signal is generated, otherwise if it cannot be completed, a secondary matching analysis signal is generated, and both are transmitted to the secondary allocation processing module at the same time. For example, if the number of unassigned subtasks is 6 groups, and the corresponding total number of edge nodes is 5, then the single matching requirement is not met in this case. On the contrary, if the number of unassigned subtasks is 5 groups, and the total number of edge nodes is 6, then the single matching requirement is met in this case.

[0040] Embodiment 2, as Embodiment 2 of the present invention, this embodiment is implemented on the basis of Embodiment 1, and the difference from Embodiment 1 is as follows:

[0041] The secondary allocation processing module is used to process the acquired judgment signal. For the single matching signal, the allocation processing is performed according to the priority of the unassigned subtask to generate resource allocation information. For the secondary matching analysis signal, the task classification information is obtained by classifying the unassigned subtask according to the priority of the subtask, and the resource allocation information is generated in combination with the corresponding demand calculation resources, and then transmitted to the real-time processing information output module.

[0042] Analyze the individual matching signals in the judgment signal, obtain all the unassigned subtasks and the assigned subtasks, and analyze the adjacency between the unassigned subtasks and the assigned subtasks, and the adjacency analysis here is specifically expressed as judging whether there is a sequence relationship between the corresponding tasks, such as the order of precedence, the order of top and bottom, and obtain the tasks with adjacency, and the tasks here include the unassigned subtasks and the assigned subtasks, and then obtain the priority of the unassigned subtask in the acquired adjacent tasks, and the priority here is represented by the corresponding configuration requirement parameter, and match the unassigned subtasks according to the priority. The specific matching method is: allocate according to the edge node number order corresponding to the assigned subtasks from small to large, and generate resource allocation information;

[0043] For example, suppose there is a series of audio and video processing tasks, in which the assigned subtask A is responsible for the initial encoding of the video, and the assigned subtask B is responsible for the format conversion of the encoded video. At this time, the unassigned subtask C is to optimize the quality of the converted video. From the task logic point of view, subtask C and subtask B have a sequential relationship. In this case, subtask C is a task that is adjacent to the assigned subtask;

[0044] Continuing with the above example, assume that the assigned subtask A is assigned to the edge node Node2, and the assigned subtask B is assigned to the edge node Node3. The unassigned subtask C is judged to be adjacent to subtask B, and has a higher priority among the unassigned subtasks with adjacency. Since the edge node numbering sequence corresponding to the assigned subtasks is Node2, Node3 (from small to large), according to the rules, subtask C is given priority to be assigned to Node2. If Node2 has insufficient resources, it is assigned to Node3. If there are other unassigned subtasks D, which are adjacent to the assigned subtask A and have a lower priority, they will also be assigned to Node2 first in the above order. If Node2 cannot carry it, Node3 will be considered.

[0045] Process the secondary matching analysis signal in the judgment signal to obtain all unassigned subtasks, and determine the priority of the unassigned subtasks. The priority determination here is represented by the corresponding configuration requirement parameters, and the unassigned subtasks are sorted from large to small according to their priorities. Then, the unassigned subtasks with the same priority are obtained and recorded as tasks of the same level. At the same time, the edge nodes are selected and allocated according to the priorities corresponding to the tasks of the same level. The specific allocation method is:

[0046] Get all the tasks of the same level, and get the corresponding required computing resources in the tasks of the same level at the same time, then match them with the edge nodes according to the required computing resources, and the matching order here is to allocate them from large to small according to the required computing resources. Specifically, first allocate the tasks of the same level with the highest priority, and when allocating, sort and allocate the unallocated subtasks in the tasks of the same level according to the required computing resources, and generate resource allocation information;

[0047] Then, all tasks of different levels are obtained, and the priorities corresponding to tasks of different levels are obtained. At the same time, tasks of different levels are interspersed and matched according to priorities, and resource allocation information is generated, which is then transmitted to the real-time processing information output module.

[0048] Assume that there are the following unassigned subtasks and their priorities:

[0049] Unassigned subtask A (video super-resolution processing): priority 8

[0050] Unassigned subtask B (image filtering): priority 6

[0051] Unassigned subtask C (adding audio effects): priority 4;

[0052] Mark unassigned subtasks with the same priority as peer tasks. For example, if there are unassigned subtasks D and E, and their priorities are both 5, then they are peer tasks. For peer tasks, their corresponding required computing resources will be obtained at the same time. Assume that the required computing resources of peer tasks D and E are as follows:

[0053] The same level task D (maybe some kind of video effect enhancement): requires 100 computing resources;

[0054] The same level task E (may be another video effect enhancement): the required computing resources are 80 units;

[0055] The same-level tasks are allocated and processed in the order of required computing resources from large to small. First, according to the idle computing resources of the edge nodes, the same-level tasks with the largest required computing resources are allocated first. When allocating, start from the edge node with the most abundant computing resources to ensure that the task can run on the node that meets its resource requirements. Suppose there are edge nodes Node1, Node2 and Node3, and their idle computing resources are 150 units, 120 units and 90 units respectively. For the same-level task D, since its required computing resources are 100 units, it will be allocated to Node1 first; and for the same-level task E, since Node1 has 50 units left after allocating D, which is not enough to meet the needs of E, E will be allocated to Node2. At the same time, detailed resource allocation information is generated;

[0056] Next, all tasks of different levels (i.e., unassigned subtasks with different priorities) are obtained and their corresponding priorities are re-obtained. For these tasks of different levels, they will be interspersed and matched according to their priorities. That is, after completing the assignment of tasks of the same level, high-priority tasks of different levels are assigned to appropriate edge nodes according to the priority order, while considering the idle computing resources of edge nodes.

[0057] Taking the above unassigned subtasks A, B, and C as an example, according to the priority order, first consider allocating the unassigned subtask A (priority 8) to the edge node. Assuming that Node1 still has 50 units of idle computing resources after allocating the same-level task D, and the unassigned subtask A requires 80 units of computing resources, the idle computing resources of Node2 and Node3 will be checked at this time. If Node2 has enough resources, A will be allocated to Node2. Then consider the allocation of unassigned subtasks B and C in turn, and also allocate them according to the idle computing resources of the edge nodes and the computing resources they each need.

[0058] A real-time processing information output module is used to display the acquired resource allocation information to the corresponding operator.

[0059] Embodiment 3, as the embodiment 3 of the present invention, focuses on combining the implementation processes of embodiment 1 and embodiment 2 for implementation.

[0060] Some of the data in the above formulas are numerically calculated by taking their dimensions. At the same time, the contents not described in detail in this specification belong to the existing technologies well known to those skilled in the art.

[0061] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A distributed audio and video real-time processing system based on edge computing, characterized in that: include: The processing information collection module is used to collect audio and video information and edge node information, and transmit the two to the processing information analysis module; The processing information analysis module is used to analyze the acquired audio and video information, divide the audio and video information into multiple groups of subtasks, and calculate the required computing resources corresponding to the subtasks. Then, the free resources of different edge nodes are obtained according to the edge node information, and the processing information is generated and transmitted to the resource allocation processing module. The resource allocation processing module is used to analyze the acquired processing information, allocate and process the subtasks by comprehensively considering the node configuration of the edge nodes and the required computing resources of the subtasks to obtain allocation information, and generate a judgment signal according to the relationship between the number of unallocated subtasks and the total number of edge nodes, and transmit the judgment signal to the secondary allocation processing module; The secondary allocation processing module is used to process the obtained judgment signal. For the single matching signal, the allocation processing is performed according to the priority of the unassigned subtask to generate resource allocation information. For the secondary matching analysis signal, the task classification information is obtained by classifying the unassigned subtask according to the priority of the unassigned subtask, and the resource allocation information is generated by combining the corresponding demand calculation resources, and the resource allocation information is transmitted to the real-time processing information output module at the same time; The real-time processing information output module is used to display the acquired resource allocation information to the corresponding operator.

2. According to the distributed audio and video real-time processing system based on edge computing according to claim 1, it is characterized in that: The specific method of processing the required computing resources of the information analysis module calculation subtask is: Acquire audio and video information, and divide the audio and video information into subtasks and label them as n, where n=1, 2, ..., m, where m represents the number of subtasks. Then analyze the required computing resources for subtask n, obtain the resolution and frame rate information corresponding to subtask n, and calculate the amount of data that subtask n needs to process per unit time. Substitute the obtained resolution and frame rate information into the formula Qn=Fn×Zn×Rn to calculate the amount of processed data Qn corresponding to subtask n per unit time, where Fn is the resolution of the subtask, Zn is the frame rate of the subtask, and Rn is the byte corresponding to the pixel. Then obtain the number of operations corresponding to subtask n, and estimate the required computing resources corresponding to subtask n based on the amount of data processed per unit time Qn.

3. According to the distributed audio and video real-time processing system based on edge computing according to claim 1, it is characterized in that: The specific method for the processing information analysis module to generate processing information is as follows: Then, all edge nodes are obtained and recorded as i, and i=1, 2, ..., j, where j represents the number of edge nodes, and the idle computing resources corresponding to the edge node i are obtained, the required computing resources of the subtask and the idle computing resources of the edge node are combined to obtain processing information, and then it is transmitted to the resource allocation processing module.

4. According to the distributed audio and video real-time processing system based on edge computing according to claim 1, it is characterized in that: The specific method in which the resource allocation processing module analyzes the processing information is as follows: Obtain the subtask required computing resources and edge node idle computing resources in the processing information, then obtain the configuration requirements corresponding to the subtask, and obtain the node configuration of the edge node. At the same time, filter the edge nodes based on the subtask configuration requirements and record them as filtered edge nodes. Then, use the filtered edge nodes corresponding to the maximum idle computing resources as the allocation object for allocation processing to generate allocation information.

5. A distributed audio and video real-time processing system based on edge computing according to claim 1, characterized in that: The specific method in which the resource allocation processing module generates the judgment signal is: According to the obtained allocation information, the unassigned subtasks are obtained, and the number of all edge nodes is obtained at the same time, and it is determined whether the unassigned subtasks can complete the single matching requirement. If it can be completed, a separate matching signal is generated. Otherwise, if it cannot be completed, a secondary matching analysis signal is generated, and both are transmitted to the secondary allocation processing module at the same time.

6. A distributed audio and video real-time processing system based on edge computing according to claim 1, characterized in that: The specific method in which the secondary allocation processing module analyzes the individual matching signals to generate resource allocation information is as follows: Analyze the individual matching signals in the judgment signal, obtain all the unassigned subtasks and assigned subtasks, and analyze the adjacency between the unassigned subtasks and the assigned subtasks, obtain the tasks with adjacency, and then obtain the priority of the unassigned subtasks in the acquired adjacent tasks, and allocate them in ascending order according to the edge node number sequence corresponding to the assigned subtasks, and generate resource allocation information.

7. A distributed audio and video real-time processing system based on edge computing according to claim 1, characterized in that: The specific method in which the secondary allocation processing module processes the secondary matching analysis signal to generate resource allocation information is: Get all the unassigned subtasks, determine the priorities of the unassigned subtasks, and sort them from large to small according to their priorities. Then get the unassigned subtasks with the same priority and record them as tasks of the same level. At the same time, select and assign edge nodes according to the priorities of the tasks of the same level. Get all the tasks of the same level, and get the corresponding required computing resources in the tasks of the same level. Then match the required computing resources with the edge nodes and generate resource allocation information.

Images