Video analysis task unloading optimization method and system in Internet of Things gateway

Abstract

The invention discloses a video analysis task unloading optimization method and system in an Internet of Things gateway. The method comprises the following steps: estimating the residual computing power of an FPGA (Field Programmable Gate Array); the method comprises the following steps: dividing a video analysis task through a pre-constructed CPU-FPGA analysis model; and unloading the video analysis task from the CPU to the FPGA by taking the minimum total delay of the video analysis task as a target under the condition that the residual computing power of the FPGA is given through a pre-constructed MVAD algorithm. Network data packets with transport-parse are preferentially processed, leaving the remaining computational power to accelerate video analysis based on the required execution time. In order to estimate the residual computing power, two schemes are developed for non-preemptive and general situations. Finally, the unloading optimization method can significantly improve the video analysis and network processing throughput under the conditions of resource shortage and poor network, so that the Internet of Things gateway has excellent performance improvement and lower overhead, thereby improving the application efficiency and solving the reconstruction time and resource consumption.

Description

Technical Field

[0001] The present invention relates to a method and system for optimizing video analysis task offloading in an Internet of Things gateway, and belongs to the technical field of the Internet of Things. Background Art

[0002] Traditional IoT gateways serve as a bridge connecting IoT devices to the cloud, transmitting data, processing various communication protocols, and managing traffic at both ends like routers. However, current IoT gateways also need to encrypt sensitive data, compress video or image data, and filter unnecessary data to reduce bandwidth consumption and cloud pressure. In order to respond in a timely manner and reduce the burden on the cloud, existing IoT gateways usually process data locally, among which video analysis is the most commonly used. By analyzing the video, the IoT gateway can extract characters or objects in the image and feed the results back to the cloud or edge to perform tasks such as OCR (Optical Character Recognition), vehicle counting, and target tracking. For image data from IoT devices, IoT gateways usually use deep neural networks (DNN) for local image processing. Considering the dynamic nature of the remaining computing power, it is difficult to minimize the overall delay in performing video analysis tasks. Summary of the invention

[0003] The present invention provides a method and system for optimizing video analysis task offloading in an Internet of Things gateway, which solves the problems disclosed in the background technology. The analysis delay is optimized by scheduling the execution position of the DNN layer; a CPU-FPGA analysis model is established, and an analysis task offloading algorithm is designed to make full use of the limited remaining computing power.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: a video analysis task offloading optimization method in an Internet of Things gateway, comprising:

[0005] When the FPGA is shared in the time dimension, the amount of time given by the video analysis is used to measure the remaining computing power of the FPGA;

[0006] The video analysis task is divided into two parts, one part is processed by the CPU and the other part is processed by the FPGA, through the pre-built CPU-FPGA analysis model.

[0007] Through the pre-built MVAD algorithm, the video analysis task is offloaded from the CPU to the FPGA with the goal of minimizing the total delay of the video analysis task given the remaining computing power of the FPGA.

[0008] Further, the formula for measuring the remaining computing power of the FPGA using the amount of time given by video analysis includes:

[0009] The remaining computing power in each cycle when there is no network resource preemption ; For delay tolerance, set the cycle length to ,make is the number of network packets generated in the previous cycle, Processing network packet throughput for FPGA.

[0010] Further, the formula for measuring the remaining computing power of the FPGA using the amount of time given by video analysis includes:

[0011] The remaining computing power per cycle when network resources are allowed to be preempted ; For the length of a cycle, set , is the delay tolerance; Indicates that network information is used to estimate network traffic in the next period of time The number of time periods; express The i-th period before, where ;make and Respectively, in the cycle The actual and predicted number of network packets generated in Indicates that in the cycle The number of network packets generated in the AR model is expressed as ,in is white noise, A smoothing value that reflects the periodic characteristics of network packets; Processing network packet throughput for FPGA.

[0012] Furthermore, the construction method of the CPU-FPGA analysis model is:

[0013] The CPU-FPGA analysis model uses DNN, which is structured as a DAG communication graph. Each node in the graph represents a single layer of DNN. represents the DAG of DNN, where is a set of vertices representing DNN layers, where and Represent the input layer and output layer respectively; is a set of links, links ( , )∈ express Must be in Previously processed, Feed its output to ;

[0014] make and The CPU and FPGA are The processing time of If and its successors are processed in different devices, The time required to transfer its output data between the CPU or memory and the FPGA through the PCI-E bus;

[0015] definition , , , respectively represent the computation delay, transmission delay, and FPGA computation delay at each layer’s CPU;

[0016] The second division is: ,in and ; Processed by the CPU, Processed by FPGA; ; The final result of the vertex will be transferred between CPU / memory and FPGA through PCI-E bus; There will be processing lag at the CPU, This will cause transmission delays. Will cause processing delays on the FPGA;

[0017] make Indicates the total delay of processing the video analysis task. After the video analysis task is divided, each frame is processed by the CPU, transmitted to the FPGA by the CPU or memory through the PCI-E bus, and processed by the FPGA. The total delay is the sum of the delays of the three processing stages.

[0018] Delays during CPU processing:

[0019]

[0020] Delays in FPGA processing stage:

[0021]

[0022] Delays during the transmission phase:

[0023]

[0024] ;

[0025] The delay of the FPGA processing stage cannot exceed the allocated computing power; Indicates the remaining computing power allocated, with the following constraints:

[0026] .

[0027] Furthermore, the construction method of the MVAD algorithm is:

[0028] The video analysis task partitioning problem is transformed into a weighted ST graph cut problem; a set of graph cuts is generated, each graph cut corresponds to a partition, and the optimal partition is selected from this set. Comply with the prescribed thresholds;

[0029] Splitting the nodes of a graph into two different sets is called a cut; The st-cut is a cut such that the source s and the sink t are in different sets; the size of a cut is specified as the sum of the prices of each connection in the cut set; Each node in carries 3 delay values , , ; Construct an auxiliary graph , so that each link captures only a single delay value; according to , merge two nodes S and T, specify the virtual entry and exit nodes respectively; establish S and each vertex The relationship between , the purpose is to capture FPGA calculation delay; by giving each vertex and T is assigned a value β to interconnect, and obtain the CPU calculation delay; for the link ( , ) , to capture the transmission delay , , .

[0030] There are 3 outgoing links, The communication delay can only be calculated once at most; in Figure Introduce auxiliary vertices, for any vertex with out-degree greater than 1 , add an auxiliary vertex ,Link( , );from arrive The successor connection is now removed from arrive The connection of the successor of The out-degree is greater than 1, and an auxiliary vertex is added and links ( , );Link( , )and( , ) are replaced by links ( , )and( , ).

[0031] Without considering S and T, if a vertex v has one successor, then the link starting from v corresponds to its communication delay; if v has multiple successor nodes, then all links starting from v are not considered;

[0032] Each st-cut with α = 1, β = 1, γ = 1 corresponds to a DNN partition; if If the link is cut, then will be processed by the FPGA, ; If the cut is from to T, then Will process on the edge, ; If the cut is from arrive The link The information will be exchanged between the CPU and FPGA. ;

[0033] Search Minimum st cut under several different values ​​of α, β and γ;

[0034] The minimum cut result is determined by the ratio of the three parameters. First, fix one of the three, γ = 1, and only modify the other two. There is a two-dimensional search space for α and β. First, a coarse-grained initial search is performed on the two-dimensional plane to find the top result. Then, a finer-grained search is applied to the optimal solution. Repeat the above steps until the enhanced performance is less than a limit Or at most h rounds.

[0035] Accordingly, the present invention also provides a video analysis task offloading optimization system in an Internet of Things gateway, comprising:

[0036] The module for measuring the remaining computing capacity of the FPGA is used to measure the remaining computing capacity of the FPGA using the time quantity given by the video analysis when the FPGA is shared in the time dimension;

[0037] The CPU-FPGA analysis model is used to divide the video analysis task into two parts, one part is processed by the CPU and the other part is processed by the FPGA.

[0038] The MVAD algorithm module is used to offload the video analysis task from the CPU to the FPGA with the goal of minimizing the total delay of the video analysis task given the remaining computing power of the FPGA.

[0039] Accordingly, the present invention also provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform any of the methods described above.

[0040] Accordingly, the present invention also provides a computing device, comprising:

[0041] One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods described above.

[0042] The beneficial effects achieved by the present invention are:

[0043] The present invention optimizes the offloading mode of video analysis tasks, so that most CPU resources are allocated to network functions, and hardware acceleration is more effective than software-based processing. This is crucial for allocating FPGA resources to accelerate video analysis, while reducing the demand for hardware resources.

[0044] The present invention can reduce the delay of network processing by optimizing the unloading mode of video analysis tasks, which means that the time from video data transmission to analysis and processing can be shortened, thereby improving the response speed and real-time performance of the entire system.

[0045] The performance of video analysis and network processing of the present invention is that by giving priority to network processing, the IoT gateway can process network data packets faster and improve network processing throughput. The non-preemptive estimation scheme (NPES) assumes that network traffic should not cause any preemption of video analysis, which not only improves fairness between tasks, adapts to different application scenarios, reduces competition for hardware resources, improves system stability and reliability, but also helps to improve resource utilization.

[0046] The present invention realizes resource occupancy and delay when processing video analysis tasks by optimizing the scheduling and transmission mode of tasks. At the same time, the optimization algorithm reduces the overhead of FPGA switches, improves the utilization rate of FPGA resources, hardware scalability and efficiency of video analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of a method for optimizing video analysis task offloading in an Internet of Things gateway of the present invention;

[0048] Figure 2 It is a schematic diagram of the process flow of estimating the remaining computing capacity of FPGA in the present invention;

[0049] Figure 3 Different representation diagrams of the YOLOV4 network model used by DNN in the present invention;

[0050] Figure 4 Schematic diagram of the conversion of the YOLOV4 network model into the ST problem. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0052] Embodiment 1, as Figure 1 As shown, this embodiment introduces a video analysis task offloading optimization method in an Internet of Things gateway, including:

[0053] 1. Remaining computing capacity estimation strategy

[0054] When the FPGA is shared in the time dimension, the amount of time given by video analysis can be used to measure the remaining computing power. is the length of a cycle, each A sub-period for network functions and a sub-cycle for video analysis composition.

[0055] The traffic load from the application largely determines the duration of the Network Function Virtualization (NFV) sub-cycle. Accurately estimating the application traffic load depends on whether the system is designed for a single purpose or for general applications. For example, a single application such as a traffic monitoring camera that transmits traffic video to a remote control center and can recognize the license plates of cars appearing in the video is an example of a single-purpose system. On the other hand, a general engine is developed to facilitate video analysis of upper-layer applications, such as mobile phones with functions such as voice recognition, fingerprint authentication, and face recognition. The framework in the present invention can be applied to both at the same time.

[0056] If the system is designed for a single application, then a comprehensive traffic load model can be developed that takes into account the application characteristics. Therefore, we first examine an extreme case where the application traffic estimation is not subject to any preemption, which is called the non-preemptive estimation scheme (NPES).

[0057] like Figure 2 As shown, the remaining computing capacity estimation method of FPGA includes:

[0058] A. Non-preemptive estimation scheme

[0059] The non-preemptive estimation scheme (NPES) can be used to estimate the remaining computing power, with the goal of ensuring that the cycle is non-preemptive and maximizing the remaining computing power of each cycle. .

[0060] The remaining computing power of a cycle depends on: 1) the duration of the cycle; 2) the number of packets processed in the cycle; 3) the network packet processing throughput of the FPGA. In order to accurately calculate the remaining computing power of each cycle, it is necessary to know the number of packets processed in the cycle. Therefore, for the cycle NPES chooses to buffer all network packets and in the next cycle Process it at the beginning.

[0061] The first goal of NPES is to ensure that preemption does not occur. As mentioned earlier, each packet has a latency tolerance determined by the application. To prevent preemption, packets must be processed before tolerable delays are exceeded.

[0062] Solution: If the FPGA's network packet processing throughput Than the maximum rate of network packet generation More significant and periodic No more than , then NPES can run without preemption.

[0063] No preemption means that all packets can be received within the delay tolerance Therefore, when and , all packets can be processed before verifying the tolerance delay. Indicates data packet The delay required to wait for processing.

[0064] It consists of two parts: 1) The data packet is generated during its cycle The buffered time is recorded as ; 2) Wait for NIC processing in cycle middle The time required for the previously generated unprocessed data packets is denoted as Obviously, .make express The number of unprocessed packets generated before, Since the maximum generation rate of network packets is ,therefore , combined with and , we have .therefore, .like , then .

[0065] Network packets are processed faster on the FPGA than they are generated, so longer cycles result in more spare computing power. Therefore, we set the cycle length to ,Right now .make is the number of network packets generated in the previous cycle, represents the throughput of network data packets processed on FPGA, from which we can calculate .

[0066] B. Trajectory-driven estimation scheme

[0067] For the general case where preemption is allowed, the present invention proposes a TDES to maximize the remaining computing power in each cycle. The main idea is to predict the number of packets generated in this time period. If the exact number of packets can be predicted, it is recorded as , then the remaining computing power can be calculated as ,in Network packet processing throughput for FPGAs.

[0068] Compared with relying on a constant injection rate and probability function, using network traffic trajectories to estimate application traffic can produce more accurate predictions. To this end, an autoregressive (AR) predictor is used to predict the traffic load of the next period based on the traffic load of the previous periods. Let Indicates that network information is used to estimate network traffic in the next period of time The number of time periods. express The i-th period before, where .make and Respectively, in the cycle The actual and predicted number of network packets generated in Indicates that in the cycle The number of network packets generated within. The AR model can be expressed as ,in is white noise, A smoothed value that reflects the periodic characteristics of network packets. The following formula is used to calculate the actual number of packets: and periodic network packets The mean square error (MSE) between: .

[0069] A smaller MSE can better reflect the periodic characteristics. Therefore, by reasonably choosing and , which can minimize the MSE.

[0070] The length of will affect the remaining computing capacity, because the delay tolerance of each data packet is , which means that it can be Because the exact generation time of network data packets is difficult to predict, the cycle length is set to ,Right now , the remaining computing power can be calculated .

[0071] 2. Video Analysis Task Offloading Optimization Algorithm

[0072] The remaining computing power allocated to the video analysis task at any given moment may not be enough to perform the complete video analysis task. In order to utilize the limited remaining computing power, this paper proposes a CPU-FPGA analysis model, in which some layers of the DNN are executed on the CPU, while the remaining layers are executed on the FPGA. With the goal of minimizing the total delay of executing the video analysis task, the analysis task is offloaded from the CPU to the FPGA. The analysis task offloading problem is modeled as a difficult polynomial problem and a heuristic algorithm is designed to solve the problem.

[0073] A. CPU - FPGA Analysis Model

[0074] 1) DNN as a graph: A DNN is structured as a directed acyclic graph (DAG) communication graph. Each node in the graph represents a single layer of the neural network, which is inseparable and must be processed at the edge or in the cloud. The connections within the graph show the communication and dependencies between layers.

[0075] make represents the DAG of DNN, where is a representation of DNN (where, and Represents the vertex sets of the input layer and the output layer respectively. is a set of links, links ( , )∈ express Must be in Previously processed, Feed its output to . Figure 3 The directed acyclic graph of the initial V4 network is given.

[0076] The processing time of each layer depends on where it is processed (i.e., CPU or FPGA). and The CPU and FPGA are The processing time of If and its successors in different devices (e.g., Processed on the CPU, and at least one of its successors is processed on the FPGA), The time required to transfer its output data between the CPU or memory and the FPGA through the PCI-E bus.

[0077] definition , , They represent the three main delays of each layer: computation delay at the CPU, transmission delay, and computation delay at the FPGA.

[0078] 2) DNN partitioning: DNN is divided into two parts, one managed by the CPU and the other managed by the FPGA. Algebraically, The second division is: ,in and . Processed by the CPU, Processed by FPGA. . The final result of the vertex will be transferred between CPU / memory and FPGA through PCI-E bus. There will be processing lag at the CPU, This will cause transmission delays. This will cause processing delays on the FPGA.

[0079] 3) Delay components and constraints: Let Indicates the total delay of processing the video analysis task. After the division is completed, each frame is processed by three stages, namely, CPU processing, CPU or memory sent to FPGA through PCI-E bus, and FPGA processing. The total delay is the sum of the delays of the three stages.

[0080] The three phases of postponement are described below.

[0081] In the CPU processing stage:

[0082]

[0083] In the FPGA calculation stage:

[0084]

[0085] During the transmission phase:

[0086]

[0087] Therefore, there is .

[0088] It is worth noting that the system’s CPU is always ready to perform video analysis tasks, while the FPGA can only perform video analysis tasks within the limited remaining computing power. Therefore, the delay of the FPGA calculation stage cannot exceed the allocated computing power. Represents the remaining computing power allocated. There are the following constraints:

[0089]

[0090] B. Problem Raising and Analysis

[0091] The goal of the Analytic Task Offload (ATO) problem of the present invention is to reduce the total delay in executing video analysis tasks given a given computing power. Schematic diagram of the structure of DNN, the processing pause of each layer on the CPU and FPGA and the transmission delay of each layer. It is worth noting that the hardware implementation of the DNN layer (i.e., FPGA) is significantly faster than a general-purpose processor, so reducing the total delay is equivalent to reducing the total delay of the CPU processing stage and the transmission stage. Therefore, the total delay of the CPU processing stage and the transmission stage is taken as the target. The decision variable is the partition vertex VP. The constraint is that the delay of the FPGA calculation stage does not exceed the allocated computing power. In summary, the ATO optimization problem is defined as follows:

[0092] Question 1: Given , , , and , under the premise of satisfying the constraints, determine and ,make Minimum.

[0093] Question 2: (Intro D) Given , , , , and , is there a partition ( , ) makes and ?

[0094] Given an example of the above problem, we have items, and the weight of the items is The cost of an item is expressed as Indicates that the weight limit is Indicates that cost constraints are The task is to determine whether there is a subset X of objects such that and Based on this, an example of the problem definition IntroD is as follows: , .make , (because , so define . Introduction , make , . Assume that problem Intro D has a solution ,in , This means and , which can be expressed as , which proves is a solution to the original knapsack problem.

[0095] Assume X solves the knapsack problem, so and The latter can be expressed as This verifies Resolved IntroD issue.

[0096] C. ATO algorithm

[0097] Since the ATO problem is difficult to solve, it is impossible to find an ideal solution worldwide in polynomial time. To this end, this patent invented the Minimize Video Analytic Delay (MVAD) algorithm, which provides a regional optimal solution. The principle of developing MVAD is as follows: First, the partitioning problem is transformed into a weighted st graph cut problem. Then, a set of graph cuts is generated, each graph cut corresponds to a partition, and then the optimal partition is selected from this set, which is in The prescribed thresholds are followed.

[0098] First, we need to solve how to transform the partitioning problem into an st graph cut problem. Splitting the nodes of a graph into two different sets is a cut. The st-cut is a cut such that the source s and the sink t are in different sets. The size of a cut is specified as the sum of the prices of each link in the cut set. Each node in carries 3 delay values , , The delay added to the total delay depends on where the node is processed. For this purpose, an auxiliary graph can be constructed , so that each link only captures a single delay value. The structure of Figure 4 As shown: 1) According to , merge two nodes S and T, specifying virtual entry and exit nodes respectively; 2) establish S and each vertex The relationship between , referred to as "red edge", is intended to capture FPGA calculation delay; 3) by giving each vertex and T is assigned a value β (called "blue links") to interconnect in order to derive CPU computational latency; 4) to link ( , ) , to capture the transmission delay. Here , , .

[0099] However, these are not sufficient, since a vertex may have multiple successors and its communication delay is calculated multiple times. For example, Figure 3 (b) There are 3 outgoing links, but The communication delay can only be calculated once at most. Introduce auxiliary vertices, that is, for any vertex with out-degree greater than 1 , add an auxiliary vertex ,Link( , ).from arrive The successor connection is now removed from arrive is replaced by the connection of the successor. Figure 4 As shown, the vertex The out-degree is greater than 1, so an auxiliary vertex is added and links ( , ).Link( , )and( , ) are replaced by links ( , )and( , ).

[0100] Now, without considering S and T, if a vertex v has a successor, then the link starting from v corresponds to its communication delay (such as Figure 4 If v has multiple successors, then Figure 4 All links starting from v to The delay has been taken into account.

[0101] so, Each st-cut with α = 1, β = 1, γ = 1 corresponds to a DNN partition. More specifically, if (Red links such as Figure 4 If the link is cut, will be processed by the FPGA, i.e. If the cut is from to T (blue link like Figure 4 As shown in the figure, then Will process on the edge, i.e. If the cut is from arrive Links (black links such as Figure 4 shown), then The information will be exchanged between the CPU and FPGA, that is, The total cost of cutting the red link is the FPGA processing time The total cost of a blue link disconnection is equal to the CPU - computation time The total cost of cutting black links is equal to the transmission time.

[0102] The second step is to generate a set of partitions and select the best partition. This is done by finding Minimum st cut for several different values ​​of α, β, and γ. Noting that the minimum partitioning problem can be solved in polynomial time, we use Goldberg's algorithm, which has a time complexity of . In this way, a partition is generated with a specific α, β, and γ tuple. Afterwards, this partition is evaluated for suitability for ATO. If it outperforms all existing solutions, then it is considered a new solution for ATO. The previous process can be repeated for a wide spectrum of α, β, and γ tuples until no solution is found.

[0103] Obviously, The result of the minimum cut is determined by the ratio of the three parameters, not their exact values. Therefore, one of the three can be fixed, i.e., γ = 1, and only the other two can be modified. After that, there is a two-dimensional search space for α and β. First, a coarse-grained initial search is performed on the two-dimensional plane to approximately find the top-level result. Then, in the second stage, a finer-grained search is applied near the optimal solution for further improvement. The above steps are repeated until the enhanced performance is less than a limit Or at most h rounds.

[0104]

[0105] The overall algorithm MVAD ( ) is shown in Algorithm 1. The design function minimum cut ( ) (line 14) calculates The function search() (lines 10 to 17) aims to search the given space The best cut in , , and the granularity δ (lines 12 to 13), i.e., each change and The step length is For each α and β, search() calls mincut() to compute the minimum cut. Lines 15 to 16 ensure that the cut set meets the given FPGA computing power limit and the sum of CPU computing and transmission delays does not increase.

[0106] The algorithm starts setting the search granularity is 1 (line 1), and the search space is large enough (lines 2-3). Subsequently, it calls search() (line 7) with the granularity Traverse the given space S and return the best α and β found so far. Then MVAD() narrows the search space S (line 7) to the neighborhood of the best α and β of the current iteration and Adjust to a finer granularity (row 8). This area S and fineness is used to search ( ). The end point of the loop is that the enhanced performance is less than a limit ε (line 4), or the number of calculation rounds exceeds h. Finally, it returns the cut with the best performance (line 9). Obviously, for the final neighborhood of α and β, we can get an ideal localization result. MVAD ( ) is a polynomial time algorithm with a computational complexity of .

[0107] Embodiment 2, this embodiment introduces the specific working process of the video analysis task offloading optimization method in the Internet of Things gateway;

[0108] In order to maximize the computing power of the transport analysis processing unit (TAPU) and reduce the impact on network processing, the technology of the present invention implements a video analysis task offloading optimization algorithm in an IoT gateway, develops algorithms for remaining computing power estimation and efficient task offloading algorithms for video analysis, and implements a prototype of a gateway transport analysis unit system and integrates it into an IoT portal.

[0109] According to the modules implemented above, the specific process of the video analysis task offloading optimization method is as follows:

[0110] Network traffic dataset: This example uses a real smart grid dataset published in reality. It is power data from three different smart homes, recording power consumption and generation of wind turbines and solar panels at a resolution of 1 second, and measuring circuit-level power usage at intervals of several seconds. In addition, the dataset contains environmental information and some real data from residents. All types of data are transmitted to the cloud periodically. For example, the real-time voltage value is sensed once a second and uploaded to the cloud every 20 seconds.

[0111] Video Analysis Settings: The camera is configured to capture 15 frames per second (FPS) with a resolution of 720p. To detect animations in the video, the system uses the YOLOv2 object detection model.

[0112] Evaluation criteria: Since network traffic is regular, the gateway-based video analysis task offloading optimization algorithm adopts a non-preemptive estimation scheme by default.

[0113] Benchmarks: The two approaches of this patent are compared with the following two approaches; 1) O-VA-FPGA offloads only the video analytics to the Intel Max10 FPGA and implements all the network functions in software. 2) D-FPGA offloads the video analytics and network functions to two dedicated Intel Max10 FPGAs without sharing.

[0114] Metrics: First, the throughput and latency of the video analytics software for video analytics and network processing were evaluated. Then, the FPGA utilization was analyzed, which is the ratio of the actual FPGA resources used to the total FPGA resources. This ratio can indicate the degree to which the FPGA resources are effectively used. It should be noted that low FPGA utilization leads to over-provisioning of hardware resources and increases system cost. For example, when a low-cost, low-computation FPGA has a high utilization rate, it can meet the performance requirements. There is no need to use a high-priced, high-computation chip. When the utilization rate is low, it will cause over-provisioning of resources and increase system cost.

[0115] Therefore, due to the increased workload of video analysis and the fixed estimated remaining computing power of the FPGA, some tasks can only be completed on the CPU, which increases the workload of the CPU in the DNN partition. Similarly, when the network load increases, the utilization of the FPGA also increases to nearly 100%. Since the priority algorithm processes the network function first, the FPGA allocates more resources to the network function image, thereby reducing the resources obtained by the video analysis image, thereby improving the efficiency and responsiveness of video analysis.

[0116] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0117] Embodiment 3 is based on the same inventive concept as Embodiment 1. This embodiment introduces a video analysis task offloading optimization system in an Internet of Things gateway, including:

[0118] The module for measuring the remaining computing capacity of the FPGA is used to measure the remaining computing capacity of the FPGA using the time quantity given by the video analysis when the FPGA is shared in the time dimension;

[0119] The CPU-FPGA analysis model is used to divide the video analysis task into two parts, one part is processed by the CPU and the other part is processed by the FPGA.

[0120] The MVAD algorithm module is used to offload the video analysis task from the CPU to the FPGA with the goal of minimizing the total delay of the video analysis task given the remaining computing power of the FPGA.

[0121] Example 4 is based on the same inventive concept as Example 1. This example introduces a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform a video analysis task offloading optimization method in an Internet of Things gateway.

[0122] Example 5 is based on the same inventive concept as Example 1. This example introduces a computing device, comprising one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing a video analysis task offloading optimization method in an Internet of Things gateway.

[0123] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0127] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A video analysis task offloading optimization method in an Internet of Things gateway, characterized in that: include: When the FPGA is shared in the time dimension, the amount of time given by the video analysis is used to measure the remaining computing power of the FPGA; The video analysis task is divided into two parts, one part is processed by the CPU and the other part is processed by the FPGA, through the pre-built CPU-FPGA analysis model. Through the pre-built MVAD algorithm, the video analysis task is offloaded from the CPU to the FPGA with the goal of minimizing the total delay of the video analysis task given the remaining computing power of the FPGA.

2. The video analysis task offloading optimization method in the Internet of Things gateway according to claim 1 is characterized in that: The formula for measuring the remaining computational power of the FPGA using the amount of time given by the video analysis includes: The remaining computing power in each cycle when there is no network resource preemption ; For delay tolerance, set the cycle length to ,make is the number of network packets generated in the previous cycle, Processing network packet throughput for FPGA.

3. The video analysis task offloading optimization method in the Internet of Things gateway according to claim 1 is characterized in that: The formula for measuring the remaining computational power of the FPGA using the amount of time given by the video analysis includes: The remaining computing power per cycle when network resources are allowed to be preempted ; For the length of a cycle, set , is the delay tolerance; Indicates that network information is used to estimate network traffic in the next period of time The number of time periods; express The i-th period before, where ;make and Respectively, in the cycle The actual and predicted number of network packets generated in Indicates that in the cycle The number of network packets generated in the AR model is expressed as ,in is white noise, A smoothing value that reflects the periodic characteristics of network packets; Processing network packet throughput for FPGA.

4. The video analysis task offloading optimization method in the Internet of Things gateway according to claim 1 is characterized in that: The CPU-FPGA analysis model is constructed as follows: The CPU-FPGA analysis model uses DNN, which is structured as a DAG communication graph. Each node in the graph represents a single layer of DNN. represents the DAG of DNN, where is a set of vertices representing DNN layers, where and Represent the input layer and output layer respectively; is a set of links, links ( , )∈ express Must be in Previously processed, Feed its output to ; make and The CPU and FPGA are The processing time of If and its successors are processed in different devices, The time required to transfer its output data between the CPU or memory and the FPGA through the PCI-E bus; definition , , , respectively represent the computation delay, transmission delay, and FPGA computation delay at each layer’s CPU; The second division is: ,in and ; Processed by the CPU, Processed by FPGA; ; The final result of the vertex will be transferred between CPU / memory and FPGA through PCI-E bus; There will be processing lag at the CPU, This will cause transmission delays. Will cause processing delays on the FPGA; make Indicates the total delay of processing the video analysis task. After the video analysis task is divided, each frame is processed by the CPU, transmitted to the FPGA by the CPU or memory through the PCI-E bus, and processed by the FPGA. The total delay is the sum of the delays of the three processing stages. Delays during CPU processing: ； Delays in FPGA processing stage: ； Delays during the transmission phase: ； ； The delay of the FPGA processing stage cannot exceed the allocated computing power; Represents the remaining computing power allocated, with the following constraints: 。 5. The video analysis task offloading optimization method in the Internet of Things gateway according to claim 1 is characterized in that: The construction method of the MVAD algorithm is: The video analysis task partitioning problem is transformed into a weighted ST graph cut problem; a set of graph cuts is generated, each graph cut corresponds to a partition, and the optimal partition is selected from this set. Comply with the prescribed thresholds; Splitting the nodes of a graph into two different sets is called a cut; The st-cut is a cut such that the source s and the sink t are in different sets; the size of a cut is specified as the sum of the prices of each connection in the cut set; Each node in carries 3 delay values , , ; Construct an auxiliary graph , so that each link captures only a single delay value; according to , merge two nodes S and T, specify virtual entry and exit nodes respectively; establish S and each vertex The relationship between , the purpose is to capture FPGA calculation delay; by giving each vertex and T is assigned a value β to interconnect, and obtain the CPU calculation delay; to give the link ( , ) , to capture the transmission delay , , ; There are 3 outgoing links, The communication delay can only be calculated once at most; in Figure Introduce auxiliary vertices, for any vertex with out-degree greater than 1 , add an auxiliary vertex ,Link( , );from arrive The successor connection is now removed from arrive The successor of the vertex The out-degree is greater than 1, and an auxiliary vertex is added and links ( , );Link( , )and( , ) are replaced by links ( , )and( , ); Without considering S and T, if a vertex v has a successor, then the link starting from v corresponds to its communication delay; If v has multiple successor nodes, all links starting from v are not considered; Each st-cut with α = 1, β = 1, γ = 1 corresponds to a DNN partition; if If the link is cut, then will be processed by the FPGA, ; If the cut is from to T, then Will process on the edge, ; If the cut is from arrive The link The information will be exchanged between the CPU and FPGA. ; Search Minimum st cut under several different values ​​of α, β and γ; The minimum cut result is determined by the ratio of the three parameters. First, fix one of the three, γ = 1, and only modify the other two. There is a two-dimensional search space for α and β. First, a coarse-grained initial search is performed on the two-dimensional plane to find the top result. Then, a finer-grained search is applied to the optimal solution. Repeat the above steps until the enhanced performance is less than a limit Or at most h rounds.

6. A video analysis task offloading optimization system in an Internet of Things gateway, characterized in that: include: The module for measuring the remaining computing capacity of the FPGA is used to measure the remaining computing capacity of the FPGA using the time quantity given by the video analysis when the FPGA is shared in the time dimension; The CPU-FPGA analysis model is used to divide the video analysis task into two parts, one part is processed by the CPU and the other part is processed by the FPGA. The MVAD algorithm module is used to offload the video analysis task from the CPU to the FPGA with the goal of minimizing the total delay of the video analysis task given the remaining computing power of the FPGA.

7. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions which, when executed by a computing device, cause the computing device to perform any one of the methods according to claims 1 to 5 .

8. A computing device, characterized in that include: One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods according to claims 1 to 5.

Images