Accelerator based on multi-image FPGA, accelerator implementation method, terminal device and computer readable storage medium

Abstract

This invention relates to an accelerator based on a multi-mirror FPGA, an accelerator implementation method, a terminal device, and a computer-readable storage medium. The accelerator includes a multi-mirror FPGA, comprising multiple mirrors pre-stored in the mirror flash memory of the FPGA. Hardware resources of the multi-mirror FPGA are used in a time-domain multiplexing manner via function calls using a hardware function abstraction method. An offline manager, connected to the multi-mirror FPGA, configures the multiple mirrors in the multi-mirror FPGA and determines the functional load of the multi-mirror FPGA. An online manager, also connected to the multi-mirror FPGA, is configured to control the switching of FPGA mirrors to allocate FPGA resources. The offline manager and the online manager are configured to respectively call preset hardware functions to implement the corresponding hardware function abstraction method of the multi-mirror FPGA. This invention enables upper layers to use the multi-mirror FPGA via function calls, facilitating the achievement of acceleration goals and improving the flexibility and ease of use of the solution.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) gateway technology, and in particular to an accelerator based on a multi-mirror FPGA, an accelerator implementation method, a device, and a computer storage medium. Background Technology

[0002] Today, IoT gateways are becoming a key element in bringing traditional and next-generation devices into the Internet of Things (IoT). IoT gateways are crucial for connectivity because they act as a bridge between devices and the cloud, integrating network protocols, helping manage storage and edge data analytics, and facilitating the secure flow of data between edge devices and the cloud. With the exponential growth in the number of connected devices, the booming development of data, and the ever-increasing demands for data security, IoT gateways today need to support a growing range of functionalities, including handling network protocols and the secure transmission of critical data, and supporting video analytics. The current generation of IoT gateways faces pressure to upgrade their hardware.

[0003] The current first-generation IoT gateways, namely those based on the traditional CPU (Central Processing Unit) architecture, and specifically the first-generation gateway acceleration solutions based on CPU architecture, face pressure for hardware upgrades. This solution aims to improve communication protocol compatibility and device management functions, including by using a scalable software router designed to process data packets in parallel with multiple CPU cores, increasing network throughput. However, in this solution, the accelerator function is limited; under the CPU architecture, it can only accelerate network functions and cannot, or cannot significantly, accelerate video analytics functions.

[0004] Second-generation IoT gateways add GPUs (Graphics Processing Units) to traditional CPU architectures, resulting in a CPU-GPU-based second-generation gateway acceleration solution. While this approach can meet the computational requirements of regular network functions and video analytics, it still has several drawbacks for IoT gateways with advanced network functions (such as data encryption and decryption, data compression, etc.). For example, existing CPU-GPU-based second-generation gateway acceleration solutions, building upon first-generation solutions, utilize video analytics neural network models on GPUs to further accelerate network packet processing. However, in this approach, the gateway accelerator is treated as a single unit, and different functions cannot share accelerator resources, leading to low accelerator utilization. Furthermore, executing advanced software network functions consumes most of the CPU resources, leaving little or no CPU resources for video analytics data preprocessing, resulting in low GPU utilization.

[0005] Existing technologies also include gateway acceleration solutions based on FPGA (Field Programmable Gate Array). FPGA, as flexible and customizable hardware, has been widely used as an accelerator. For example, automated tools are used to generate custom FPGA-based accelerators for various functions, including but not limited to: using FPGA resources to handle a portion of the CPU workload, significantly reducing CPU resource utilization; building key-value stores on FPGAs, effectively improving throughput; offloading transaction execution of video workloads to FPGAs to increase video processing speed; and implementing relevant protocols on FPGAs to reduce CPU workload, etc.

[0006] However, existing FPGA-based gateway acceleration solutions still have the following drawbacks: 1. Each function requires a custom FPGA as a dedicated hardware accelerator, and accelerator resources cannot be shared between different functions, which not only makes the overall cost of the gateway device expensive but also makes the device bulky; 2. Existing multi-image FPGAs cannot automatically switch between different images according to needs or settings, lacking flexibility.

[0007] Furthermore, current FPGAs, such as the Intel MAX10 series FPGAs, only offer a manual switching mode. Figure 4 As shown, only manual switches SW1 and SW2 are provided. One switch is used to select the mirror via the CONF IG SEL pin, and the other switch is used to trigger reconfiguration via the RU nCONF IG pin. To complete the mirror switching operation, SW1 and SW2 need to be manually triggered. This is inconvenient for switching the selection of the FPGA. Summary of the Invention

[0008] The main objective of this invention is to provide an accelerator based on a multi-mirror FPGA, an accelerator implementation method, a terminal device, and a computer storage medium, aiming to solve the drawbacks of existing FPGA-based gateway acceleration solutions or second-generation gateway acceleration solutions based on CPU-GPU architecture, and to improve the performance of the gateway accelerator.

[0009] To achieve the above objectives, this invention proposes an accelerator based on a multi-mirror FPGA, the accelerator comprising:

[0010] A multi-mirror FPGA, comprising multiple mirrors pre-stored in the mirror flash memory of an FPGA, utilizes the hardware resources of the multi-mirror FPGA in the form of function calls through time-domain multiplexing using a hardware function abstraction method.

[0011] An offline manager, connected to the multi-mirror FPGA, configures multiple mirrors within the multi-mirror FPGA and determines the functional load of the multi-mirror FPGA; and

[0012] An online manager, connected to the multi-mirror FPGA, is configured to control the switching of FPGA mirrors for FPGA resource allocation.

[0013] The offline manager and the online manager are configured to call preset hardware functions to implement the corresponding hardware function abstraction method of the multi-image FPGA.

[0014] On one hand, hardware function abstraction methods include function address mapping between function names and corresponding implementation memory addresses, so as to load the required image into the configuration flash memory through function calls.

[0015] On one hand, the offline manager includes a mirror library, a network function mirror decision-maker, and a video analysis mirror decision-maker. The mirror library is configured with and stores images of multiple mirrored FPGAs. The network function mirror decision-maker selects a first image from the mirror library to be loaded onto the FPGA according to network method requirements. The video analysis mirror decision-maker selects a second image from the mirror library for video analysis according to video analysis requirements.

[0016] On the one hand, the number of the first image and the second image is greater than or equal to one.

[0017] In one aspect, the online manager includes a mirror scheduler, a network method process manager, and a video analytics process manager. The mirror scheduler is used to control the switching of FPGA mirrors to allocate FPGA resources between the network method process manager and the video analytics process manager.

[0018] On one hand, the mirror scheduler prioritizes allocating FPGA resources for network functions, while utilizing the remaining FPGA resources for video analytics.

[0019] On one hand, when existing network packets are verifying the delay tolerance, a preemption signal is sent to the mirror scheduler. When the mirror scheduler receives the preemption signal, the network method process manager switches the FPGA mirror to the network function mirror.

[0020] This invention also proposes an accelerator implementation method based on a multi-mirror FPGA as described above, the accelerator implementation method comprising:

[0021] The offline manager configures the mirroring of the multi-mirror FPGA and determines the functional load of the multi-mirror FPGA.

[0022] Switching between FPGA images is controlled via the online manager; and

[0023] The offline manager and the online manager respectively call preset hardware functions to implement the corresponding hardware function abstraction method of the multi-mirror FPGA.

[0024] Among them, the hardware resources of multi-mirror FPGAs are used in the form of function calls through time-domain multiplexing by utilizing hardware function abstraction methods.

[0025] On the one hand, the functional load of the multi-mirror FPGA is determined by the network function mirror decision-maker and the video analysis mirror decision-maker.

[0026] On one hand, a mirror scheduler is provided to control the switching of FPGA mirrors in order to allocate FPGA resources between the network method process manager and the video analytics process manager.

[0027] The present invention also proposes a terminal device comprising: an accelerator, a memory, and a processor based on a multi-mirror FPGA as described above, wherein the accelerator is configured to execute the accelerator implementation method as described above.

[0028] The present invention also proposes a computer-readable storage medium storing an accelerator implementation program, which, when executed by a processor, implements the steps of the accelerator implementation method as described above.

[0029] Compared to existing gateway acceleration solutions, this invention addresses the problems of low accelerator utilization, high cost, and bulky size caused by the limited functionality and inability to share resources with other functions. It improves the accelerator's economy, intelligence, and miniaturization. Furthermore, this invention solves the problem of existing acceleration solutions requiring dedicated custom FPGAs for each function through hardware function abstraction. This allows upper layers to use multiple mirrored FPGAs via function calls, facilitating acceleration goals and improving the solution's flexibility and ease of use. In addition, the accelerator of this invention is smaller and more cost-effective.

[0030] This invention provides a hardware accelerator for computationally intensive video analytics and increasingly advanced network functions, thereby meeting the performance requirements brought about by these functions, realizing the performance requirements and development trends of video analytics and network functions in IoT gateways, and solving the drawbacks of existing technologies. Attached Figure Description

[0031] Figure 1 This is a system architecture diagram of an accelerator based on a multi-mirror FPGA according to a preferred embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the device structure of the terminal device hardware operating environment according to a preferred embodiment of the present invention;

[0033] Figure 3 This is a flowchart illustrating an accelerator implementation method according to a preferred embodiment of the present invention.

[0034] Figure 4 This is a diagram illustrating the existing mechanism for manually triggering mirror switching.

[0035] Figure 5 This is a schematic diagram of a multi-image switching mechanism for an accelerator implementation method according to a preferred embodiment of the present invention.

[0036] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0037] The detailed description below is intended to describe various configurations of the subject matter, and not merely to indicate configurations that the subject matter can adopt. The accompanying drawings are incorporated herein and form part of the detailed specification. The detailed specification includes specific details to provide a thorough understanding of the subject matter. However, it will be apparent to those skilled in the art that the subject matter is not limited to the specific details described herein and can be implemented using one or more embodiments. In one or more instances, structures and components are shown in block diagram form to avoid obscuring the concepts of the subject matter. One or more embodiments of this disclosure are illustrated by one or more figures and / or described in conjunction with one or more figures.

[0038] Terminology Explanation

[0039] Field-Programmable Gate Arrays (FPGAs): Logic circuits described in hardware description languages ​​(Verilog or VHDL) can be quickly programmed onto FPGAs for execution using logic synthesis, placement, and routing tools. These programmable logic components can be used to implement basic digital logic gates (such as AND gates, OR gates, XOR gates, and NOT gates) or more complex combinational logic functions.

[0040] Dual / Dual-image: The image file is the final file of the FPGA design, also known as a bitfile or bitstream file, used to configure the FPGA's configuration bitstream. Dual-image refers to storing two image files simultaneously on a single FPGA, running two sets of user-designed combinational logic gates respectively; multi-image refers to storing two or more image files simultaneously on a single FPGA, running multiple sets of user-designed combinational logic gates respectively. Multi-image FPGAs can pre-store multiple images in the FPGA's image flash memory, enabling fast switching between images. A typical example of this type of dual / dual-image FPGA is the Max10 series FPGA manufactured by Intel.

[0041] Central Processing Unit (CPU): The CPU is one of the main components of an electronic calculator, its core part. Its primary functions are interpreting calculator instructions and processing data within the calculator software. The CPU is the core component of the calculator, rereading, decoding, and executing instructions. The CPU has a unique arithmetic unit, capable of performing arithmetic calculations in a few loops. It can also store large amounts of data. Furthermore, the CPU includes a complex logic control unit that can reduce the complexity of logical operations when a program has multiple branches by providing the ability to predict branches. Therefore, the CPU can efficiently run programs with complex instructions.

[0042] Graphics Processing Unit (GPU): A GPU is a processor designed for high throughput, comprising numerous arithmetic units and minimal cache. GPUs also support a large number of threads running concurrently. If the same data needs to be accessed, the cache will merge these accesses, naturally introducing latency. Despite this latency, the sheer number of arithmetic units allows for extremely high throughput. Therefore, GPUs are capable of efficiently running data-complex programs.

[0043] This invention relates to an accelerator based on a multi-mirror FPGA, comprising: a multi-mirror FPGA, an offline manager, and an online manager, wherein the offline manager and the online manager are respectively connected to the multi-mirror FPGA; the underlying design of the multi-mirror FPGA and the hardware function abstraction method constitute the hardware layer design of the accelerator, wherein the hardware function abstraction method is designed based on the offline manager and the online manager respectively; the offline manager and the online manager together constitute the software layer design of the accelerator.

[0044] This invention also relates to an accelerator implementation method, applied to the aforementioned multi-mirror FPGA-based accelerator. The accelerator implementation method includes: configuring the mirroring of the multi-mirror FPGA and determining the functional load of the multi-mirror FPGA through the offline manager; controlling the switching of FPGA mirrors through the online manager; and implementing the corresponding hardware function abstraction method of the multi-mirror FPGA by calling preset hardware functions through the offline manager and the online manager respectively.

[0045] The present invention also relates to a terminal device, comprising: an accelerator, a memory, a processor, and an accelerator implementation program stored in the memory and executable on the processor, wherein the accelerator implementation program, when executed by the processor, implements the steps of the accelerator implementation method as described above.

[0046] The present invention also relates to a computer storage medium storing an accelerator implementation program, which, when executed by a processor, implements the steps of the accelerator implementation method as described above.

[0047] Reference Figure 1 The system architecture diagram shown is for a multi-mirror FPGA-based accelerator according to a preferred embodiment of the present invention. The multi-mirror FPGA-based accelerator 1 of the present invention includes a multi-mirror FPGA 101, an offline manager 102, and an online manager 103, wherein the offline manager and the online manager are respectively connected to the multi-mirror FPGA. The underlying design of the multi-mirror FPGA 101 and the hardware function abstraction method module 104 constitute the hardware layer design of the accelerator 1, wherein the hardware function abstraction method module is designed based on the offline manager and the online manager respectively. The offline manager 102 and the online manager 103 together constitute the software layer design of the accelerator.

[0048] In this embodiment, the accelerator based on a multi-mirror FPGA includes a hardware layer design and a software layer design. The software layer design includes an online manager and an offline manager. In the hardware layer design, the accelerator employs a multi-mirror FPGA underlying design and a hardware function abstraction method. This hardware function abstraction method aims to address issues such as difficulties in software program implementation during upper-layer design, excessive time consumption when covering FPGA resource load details, and the difficulty in handling code for FPGA vendor-specific services. Using the accelerator of this embodiment, users can utilize the multi-mirror FPGA in the upper layer via function calls, invoking hardware resources through function calls. Furthermore, the multi-mirror FPGA underlying design (hardware layer design) includes solutions to the problem of multi-mirror FPGAs not being able to automatically switch between different mirrors.

[0049] like Figure 1As shown, the offline manager 102 includes: an image library 1021, a network function image decision-maker 1024, and a video analysis image decision-maker 1023. Optionally, it also includes a video analysis requirement module 1022 and a network method requirement module. The image library 1021 includes a video analysis library and a network method library. Before actual deployment, the network function image decision-maker 1024 and the video analysis image decision-maker 1023 can select one or more images from the image library according to the network method requirements and video analysis requirements, respectively. For example, the network function image decision-maker 1024 selects a first image from the network method library of the image library 1021 according to the network method requirements, and the video analysis image decision-maker 1023 selects a second image from the video analysis library of the image library 1021 according to the video analysis requirements, wherein the number of the first image and the second image is greater than or equal to one. The processor 1001 can be used to call the accelerator implementation program stored in the memory 1005 and perform the following operations: configure and store the image of the multi-image FPGA through the image library 1021; determine the functional load of the multi-image FPGA through the network function image decision 1024 and the video analysis image decision 1023.

[0050] The online manager 103 includes a mirror scheduler 1032, a network method process manager 1034, and a video analysis process manager 1035. Optionally, it also includes a network real-time data module 1031 and a video real-time data module 1033. Preferably, the mirror scheduler 1032 is used to control the switching of FPGA mirroring to allocate FPGA resources between the network method process manager 1034 and the video analysis process manager 1035. In actual deployment, the mirror scheduler 1032 can control the switching of FPGA mirroring based on the real-time input video stream file, such as the video stream file from the video real-time data module 1033, and the network conditions, such as the network conditions from the network real-time data module 1031, allocating FPGA resources to network functions and video analysis; the network method process manager 1034 is responsible for calling and processing specific data packets; and the video analysis process manager 1035 is responsible for analyzing and verifying specific video data. The processor 1001 can be used to call the accelerator implementation program stored in the memory 1005 and perform the following operations: switch the FPGA mirrors by controlling the mirror scheduler to allocate FPGA resources between the network method process manager and the video analysis process manager.

[0051] In this embodiment, the multi-mirror FPGA-based accelerator of this invention, through hardware-to-software design, realizes a novel accelerator that leverages the characteristics of multi-mirror FPGAs to accelerate video analytics and network functions. This maximizes the computational power of the hardware design with minimal impact on network processing. Furthermore, the software design of the multi-mirror FPGA-based accelerator specifically incorporates a computational resource capacity assessment algorithm and an efficient resource allocation algorithm to load video analytics and other functions onto the hardware design. This significantly improves throughput in video analytics and network functions, enabling real-time analysis of computationally intensive videos and support for cutting-edge network functions.

[0052] Reference Figure 2 The terminal device hardware operating environment according to a preferred embodiment of the present invention. The terminal device 2 of the preferred embodiment of the present invention may be a terminal device including the above-mentioned multi-mirror FPGA-based accelerator, for example, the terminal device may be a gateway device in the Internet of Things, etc.

[0053] like Figure 2 As shown, the terminal device 2 may include: a processor 201, such as a CPU, a communication bus 202, a user interface 203, a network interface 204, and a memory 205. The processor 201 is connected to the user interface 203, the network interface 204, and the memory 205 via the communication bus 202, thereby enabling communication between these components.

[0054] User interface 203 can connect to a client and communicate with the client for data exchange. It is used to receive user input and / or provide output to the user. The user interface may include or be connected to a display screen (e.g., a touch screen), input units (e.g., a mouse, keyboard, handwriting tablet), etc. Optionally, user interface 203 may also include standard wired interfaces, wireless interfaces, etc. Network interface 204 can be used to connect to a backend server and communicate with the backend server for data exchange. It is used to connect to a network (e.g., the Internet) via wired or wireless means. Optionally, it may include standard wired interfaces, wireless interfaces (e.g., Wi-Fi interfaces), etc. Processor 201 can be used to call the accelerator implementation program stored in memory 205 and execute the accelerator implementation method. Memory 205 can be high-speed RAM or stable non-volatile memory, such as disk storage. Memory 205 may also optionally be a storage device independent of the aforementioned processor 201. Figure 2 As shown, the memory 205, which serves as a computer storage medium, may include an operating system 2051, a network communication module 2052, a user interface module 2053, and an accelerator implementation program module 2054.

[0055] Those skilled in the art will understand that Figure 2 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0056] Based on the above hardware structure, various embodiments of the accelerator implementation method of the present invention are proposed.

[0057] The flow of an accelerator implementation method according to a preferred embodiment of the present invention is as follows: Figure 3 As shown, in this embodiment, the accelerator implementation method is applied to the aforementioned multi-mirror FPGA-based accelerator. The accelerator implementation method of the present invention includes the following steps.

[0058] First, in step S10, the offline manager configures the mirroring of the multi-mirror FPGA and determines the functional load of the multi-mirror FPGA; then, in step S20, the online manager controls the switching of FPGA mirrors; finally, in step S30, the offline manager and the online manager respectively call preset hardware functions to implement the corresponding hardware function abstraction method of the multi-mirror FPGA.

[0059] In step S10, the mirroring of the multi-mirror FPGA is configured and the functional load of the multi-mirror FPGA is determined through the offline manager. In this embodiment, the accelerator determines which functions should be loaded onto the FPGA and how to pre-configure the FPGA mirroring based on the offline manager in the software design, which aims to decouple developers from time-consuming FPGA programming and complex FPGA mirroring decisions.

[0060] In this embodiment, the accelerator implementation method of the present invention can provide hardware resource acceleration for multiple functional methods based on the accelerator, which means that multiple functions need to share limited FPGA resources. Therefore, the accelerator implementation method of the present invention achieves the time-domain multiplexing of the accelerator through a multi-functional acceleration mechanism of accelerator time-domain multiplexing. That is, each method has a mirror image, and these mirror images are programmed onto logic units to form different functional modules. According to a specified order, priority, etc., the corresponding methods are accelerated in sequence.

[0061] Further, in step S10, configuring the images of the multi-image FPGA through the offline manager may include configuring and storing the images of the multi-image FPGA through the image library. In this embodiment, in order to configure the images of the multi-image FPGA, the accelerator can use the offline manager to implement one or more image libraries to pre-configure and store the images of the multi-image FPGA.

[0062] Further, in step S10, determining the functional load of the multi-mirror FPGA through the offline manager may include determining the functional load of the multi-mirror FPGA through the network function mirroring decision-maker and the video analytics mirroring decision-maker. For this purpose, the offline manager provides an image library to store pre-configured images. In this embodiment, the offline manager in the accelerator implements one or more network function mirroring decision-makers to determine which network functions are loaded onto the FPGA, and designs one or more video analytics mirroring decision-makers to determine the neural network model mirroring used for video analytics.

[0063] In this embodiment, step S20 specifies that FPGA image switching is controlled by the online manager. The accelerator controls FPGA image switching based on the online manager in the software design, determining how to switch images at runtime to allocate FPGA resources to network functions and video analytics. The runtime switching adapts to runtime changes, maximizing the utilization of hardware resources, with the goal of maximizing FPGA resource utilization.

[0064] Step S20 may include switching FPGA mirrors via the mirror scheduler to allocate FPGA resources between the network method process manager and the video analysis process manager. In this embodiment, the mirror scheduler is the core module of the online manager. The mirror scheduler first allocates resources for network functions and then utilizes the remaining computing power to accelerate video analysis.

[0065] The mirror scheduler prioritizes network functions when allocating FPGA resources because Ethernet packets range in size from 64 to 1500 bytes, and FPGA-based network function acceleration solutions for processing such small data require only ultra-low microsecond latency. In contrast, video data sizes vary widely, from tens to hundreds of megabytes, requiring millisecond-level latency. The processing latency of network functions is significantly lower than that of video analytics.

[0066] Specifically, the mirror scheduler estimates network traffic and allocates sufficient FPGA resources to the network function based on the estimation results. This ensures that switching to the network function mirror allows enough time for video analytics to be performed on the remaining FPGA resources. The network traffic estimation and remaining resource calculation are performed when the mirror scheduler sends the allocation results to the network method process manager and the video analytics process manager, and invokes hardware switching to the corresponding mirror when resources are allocated to the function.

[0067] The network method process manager is designed to schedule network packet processing after receiving resource location results from the mirror scheduler. When resources are allocated to network functions, the network method process manager operates on all unprocessed network packets.

[0068] Another function of the network method process manager is to send a preemption signal to the mirror scheduler when an existing network packet is about to verify the delay tolerance. When the mirror scheduler receives the preemption signal, the network method process manager immediately switches the FPGA to the network function mirror.

[0069] The video analytics process manager schedules the processing of video analytics tasks when allocating FPGA resources for video analytics. A video analytics task is to process a single frame of video using a given DNN (Deep Neural Networks) model. Video analytics tasks can be executed on a local CPU or FPGA.

[0070] It should be noted that in this embodiment, not all types of DNN layers exhibit higher performance when processed by an FPGA; processing certain types of layers on a CPU is more efficient than on an FPGA. For example, the performance of processing fully connected layers with small data and large models on a CPU is not significantly worse than that on an FPGA. The video analytics process manager runs analytics task load optimization algorithms to minimize the overall latency of processing video analytics tasks using allocated FPGA resources.

[0071] In this embodiment, the accelerator implementation method of the present invention is also based on the accelerator running a multi-functional acceleration mechanism with time-domain multiplexing through software design (specifically as described in steps S10 to S20 above). This solves the problems of existing accelerators having single functions and accelerator resources that cannot be shared with other functions, resulting in low utilization, high cost, and bulky size. This solution adopts a method of hardware multiplexing in the time domain, enabling accelerator resources to achieve hardware acceleration between different functions, such as acceleration of network functions and video analysis functions, based on the designed computing resource capability evaluation algorithm and efficient resource allocation algorithm. Furthermore, it allows for free switching between functions to achieve multi-functionality, thereby increasing accelerator utilization and reducing cost and size. Through flexible and automatic accelerator resource sharing, accelerator resources can be shared between different functions, and automatic switching between different functions is achieved through hardware circuits and logic programs.

[0072] Furthermore, in this embodiment, the accelerator implementation method of the present invention is also based on the accelerator running accelerator hardware abstraction method mechanism to solve the problem that in the existing acceleration schemes, each function requires the use of a dedicated customized FPGA. Through the designed hardware function abstraction method, the upper layer can use multiple mirrored FPGAs in the form of function calls, and users can conveniently achieve the acceleration target by calling hardware resources through function calls.

[0073] In other words, the accelerator implementation method of this invention loads the workload of multiple types of acceleration tasks onto multiple mirrored FPGAs and allocates FPGA resources to functions by controlling the mirroring time. To address the difficulties and time-consuming nature of upper-level design programming, which involves detailed FPGA resource loads and requires careful handling of FPGA vendor-specific service code, this invention's accelerator implementation method incorporates a hardware function abstraction method in the hardware design portion of the accelerator. This includes a mirroring loading hardware method corresponding to the network function mirroring decision-maker 1024 and the video analysis mirroring decision-maker 1023, a mirroring programming hardware method corresponding to the mirroring scheduler 1032, a network process hardware method corresponding to the network method process manager 1034, and a video analysis hardware method corresponding to the video analysis process manager 1035. This allows the upper layer to use the multiple mirrored FPGAs via function calls, and users can invoke hardware resources through function calls. Accelerator users can also invoke the hardware API (Application Programming Interface) via function calls.

[0074] In this implementation, the actual implementation of the hardware abstraction method is stored in the FPGA memory. The mapping between function names and their corresponding implementation memory addresses is called the function address map, which is built and stored in the edge device's memory when the accelerator is installed. When the accelerator calls a hardware function, it first looks up the function's memory address in the FPGA memory through the function address map, then accesses the specific memory address, and runs the function's implementation. In this way, the function address map provides accelerator users with a clean logical abstraction, allowing them to easily call these hardware functions.

[0075] The hardware function abstraction method provides a programming model for the upper layers that fully encapsulates the low-level details of the multi-image FPGA code. Using multiple hardware APIs, application developers can easily work with the multi-image FPGA without needing to understand the underlying code details. By combining these APIs, application developers can control the FPGA's operation to achieve rich and complex functionalities. For example, the function `Img_query_programmed` checks if the image programmed in the logic cell is the required image for video processing. If not, the functions `Img_load` and `Img_program` are called to load the required image into the configuration flash memory and program it into the logic cell. Then, the function `Data_load` loads the video frames into the FPGA's user flash memory, `Task_process` is called to process the video parsing task, and finally, the results are retrieved from the user flash memory and stored in the flash memory.

[0076] Furthermore, in this embodiment, the accelerator implementation method of the present invention also addresses the problem in existing acceleration schemes where existing multi-image FPGAs cannot automatically switch between different images according to needs or settings, lacking flexibility, based on the accelerator's multi-image fast switching mechanism. This is achieved through new hardware circuits and logic programs, enabling multi-image FPGAs to automatically switch between different images according to algorithms. Specifically, the accelerator implementation method of the present invention implements a hardware abstraction method for switching programming images in logic units within the hardware abstraction method of the accelerator's hardware design. In particular, in the accelerator implementation method of the present invention, the switching between FPGA images is automatically completed by instructions. This results in a multi-functional accelerator in the time domain, maximizing the utilization of the accelerator's hardware resources.

[0077] In order to overcome such Figure 4 The present invention addresses the shortcomings of the manual switch by incorporating an accelerator in the hardware design, using methods such as... Figure 5 The relays shown replace the manual switches SW1 and SW2; that is, a relay is fixed in the original positions of SW1 and SW2. When switching to the CPU image stored in flash memory region one, this scheme switches the CONFIG SEL pin via relay 0, and then triggers reconfiguration by pulling the RU nCONFIG pin low via relay 1, and vice versa. This enables fast image switching on the FPGA via instructions or hardware functions.

[0078] Furthermore, a preferred embodiment of the accelerator implementation method of the present invention is proposed.

[0079] In this embodiment, an Intel MAX10 dual-mirror FPGA serves as the FPGA hardware board for the multi-mirror FPGA-based accelerator of this invention. The MAX10 has a USB Blaster, which can be used to connect to the USB port of a smart camera using a USB-to-serial cable. Communication between the accelerator and the smart camera is implemented using the open-source RPC interface Thrift, with RPC messages transmitted via the USB-to-serial cable. By connecting the MAX10 to the smart camera, the workload of the media intervention control layer and physical layer in processing network packets is offloaded to the MAX10. Post-processing and network packets can be directly transmitted to the cloud, eliminating the need to waste time returning processing results to the network card. The other network card on the smart camera can be disabled by editing the file "ifcfg-eth0" in the Linux system.

[0080] In the Max10, only one image can be used for network method acceleration, while the other is allocated to video analytics acceleration, because the Max10 FPGA's logic cells are large enough to implement these methods. Network methods DES and Gzip are implemented in a single image, specifically using partial reconfiguration technology to achieve both functions simultaneously within a single image.

[0081] By processing network packets at the Media Intervention Control (MIC) layer and the Physical Layer, the MIC layer constructs Ethernet frames, including realigning the payload, modifying the source address, calculating and inserting packet gap bytes, while the Physical Layer converts digital signals into analog signals. The MIC layer is implemented in the FPGA area of ​​the MAX10 with sufficient computing power, and the Physical Layer is implemented in the Marvell 88E111 chip on the MAX10 development board. The MIC layer communicates with the Physical Layer using an RGMII interface, and the Physical Layer connects to an Ethernet cable via a modular RJ-45 connector to send data packets.

[0082] PipeCNN was chosen as the neural network model for implementing video analysis mirroring. PipeCNN is a high-efficiency convolutional neural network accelerator based on OpenCL (Open Computing Language) on FPGAs, providing a faster hardware development cycle and a user-friendly software interface. Using PipeCNN, the development cycle is significantly shortened by exploring the design space to find the optimal design that maximizes throughput or minimizes execution time.

[0083] The results based on this optimal embodiment show that the present invention can be applied to the implementation method of a multi-mirror FPGA-based accelerator. The network transmission and video analysis gateway accelerator based on multi-mirror FPGA can improve the performance of data analysis and transmission by 1.49 times and 2.33 times respectively compared with the current method. The utilization rate of system hardware resources can reach 92.51%. Compared with the previous method, the throughput of video analysis and network processing is significantly improved.

[0084] Furthermore, this invention can solve the problem that each function in existing acceleration solutions requires a dedicated custom FPGA through hardware function abstraction, thereby enabling the upper layer to use multiple mirrored FPGAs in the form of function calls, which facilitates the achievement of acceleration goals and improves the flexibility and ease of use of the solution.

[0085] Furthermore, the software layer design of the accelerator in this invention, which consists of an offline manager and an online manager, can also achieve rapid switching between multiple images, thereby solving the problem that existing FPGA-based acceleration solutions cannot automatically switch images according to needs or settings and lack flexibility, thus improving the reliability and advancement of the solution.

[0086] The present invention also provides a computer storage medium storing an accelerator implementation program, which, when executed by a processor, implements the steps of the accelerator implementation method as described in any of the above embodiments.

[0087] The specific embodiments of the computer storage medium of the present invention are basically the same as the embodiments of the above-described accelerator implementation method, and will not be described in detail here.

[0088] The present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the accelerator implementation method as described in any of the above embodiments.

[0089] The specific embodiments of the computer program product of the present invention are basically the same as the embodiments of the above-described accelerator implementation method, and will not be described in detail here.

[0090] This invention proposes a novel accelerator that uses multi-mirror FPGAs to accelerate video analytics and network functions. By leveraging the characteristics of multi-mirror FPGAs and through hardware-to-software design, the computational power of the hardware design is maximized with minimal impact on network processing. We designed a computational resource capacity assessment algorithm and an efficient resource allocation algorithm to load video analytics and other functions onto the hardware design, which significantly improves the throughput of video analytics and network functions, thereby enabling real-time analysis of computationally intensive videos and support for cutting-edge network functions.

[0091] The present invention employs a method of hardware reuse in the time domain, enabling accelerator resources to freely switch functions in the time domain according to the designed computing resource capability assessment algorithm and efficient resource allocation algorithm, thereby increasing accelerator utilization and reducing cost and size.

[0092] This solution addresses the need for a dedicated custom FPGA for each function. Through a designed hardware function abstraction method, the upper layer can use multiple mirrored FPGAs in the form of function calls. Users can easily achieve acceleration goals by calling hardware resources through function calls.

[0093] This solution addresses the inflexibility issue of multi-mirror FPGAs failing to automatically switch between different mirrors based on requirements or settings. By designing new hardware circuits and logic programs, the solution enables multi-mirror FPGAs to automatically switch between different mirrors according to an algorithm.

[0094] In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0095] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0096] As used in this specification, the term "comprising" means "at least partially comprising". In interpreting each statement in this specification that contains the word "comprising", there may also be other features or features beginning with that word. Related terms such as "comprising" and "including" should be interpreted in the same manner.

[0097] Many structural variations and a wide range of different embodiments and applications of the invention will be apparent to those skilled in the art without departing from the scope of the invention as defined by the appended claims. The disclosure and description herein are purely illustrative and are not intended to be limiting in any way. When specific integers with known equivalents in the art related to this invention are mentioned herein, these known equivalents are considered to be incorporated herein as if they were set forth separately.

[0098] As used in this article, the term "and / or" means "and" or "or" or both.

[0099] In the description of this specification, reference may be made to subject matter that is not within the scope of the appended claims. Such subject matter should be readily recognized by those skilled in the art and may help to put the invention into practice as defined in the appended claims.

[0100] Although the invention is generally as defined above, those skilled in the art will understand that the invention is not limited thereto, and that the invention also includes exemplary embodiments given in the following examples.

[0101] The foregoing description of the present invention includes its preferred form. Modifications may be made thereto without departing from the scope of the invention.

[0102] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An accelerator based on a multi-mirror FPGA, characterized in that, The accelerator includes: A multi-mirror FPGA, comprising multiple mirrors pre-stored in the mirror flash memory of an FPGA, utilizes the hardware resources of the multi-mirror FPGA in the form of function calls through time-domain multiplexing using a hardware function abstraction method. An offline manager, connected to the multi-mirror FPGA, configures multiple mirrors within the multi-mirror FPGA and determines the functional load of the multi-mirror FPGA; and An online manager, connected to the multi-mirror FPGA, is configured to control the switching of FPGA mirrors for FPGA resource allocation. The offline manager and the online manager are configured to respectively call preset hardware functions to implement the corresponding hardware function abstraction methods of the multi-mirror FPGA. The online manager includes a mirror scheduler, a network method process manager, and a video analytics process manager. The mirror scheduler controls the switching of FPGA mirrors to allocate FPGA resources between the network method process manager and the video analytics process manager. Specifically, when existing network data packets are verifying the delay tolerance, a preemption signal is sent to the mirror scheduler. When the mirror scheduler receives the preemption signal, the network method process manager switches the FPGA mirror to the network function mirror.

2. The accelerator based on a multi-mirror FPGA as described in claim 1, characterized in that, The hardware function abstraction method includes a function address mapping between function names and corresponding implementation memory addresses, so as to load the required image into the configuration flash memory through function calls.

3. The accelerator based on a multi-mirror FPGA as described in claim 1 or 2, characterized in that, The offline manager includes a mirror library, a network function mirror decision-maker, and a video analysis mirror decision-maker. The mirror library is configured with and stores images of multiple mirrored FPGAs. The network function mirror decision-maker selects a first image from the mirror library to be loaded onto the FPGA according to network method requirements. The video analysis mirror decision-maker selects a second image from the mirror library for video analysis according to video analysis requirements.

4. The accelerator based on a multi-mirror FPGA as described in claim 3, characterized in that, The number of the first image and the second image is greater than or equal to one.

5. The accelerator based on a multi-mirror FPGA as described in claim 1, characterized in that, The mirror scheduler prioritizes allocating FPGA resources for network functions, while utilizing the remaining FPGA resources for video analytics.

6. An accelerator implementation method based on a multi-mirror FPGA as described in any one of claims 1 to 5, characterized in that, The accelerator implementation method includes: The offline manager configures the mirroring of the multi-mirror FPGA and determines the functional load of the multi-mirror FPGA. Switching between FPGA images is controlled via the online manager; and The offline manager and the online manager respectively call preset hardware functions to implement the corresponding hardware function abstraction method of the multi-mirror FPGA. Among them, the hardware resources of multi-mirror FPGAs are used in the form of function calls through time-domain multiplexing by utilizing hardware function abstraction methods.

7. The accelerator implementation method as described in claim 6, characterized in that, The functional load of the multi-mirror FPGA is determined by the network function mirror decision-maker and the video analysis mirror decision-maker.

8. The accelerator implementation method as described in claim 6, characterized in that, A mirror scheduler is provided to control the switching of FPGA mirrors to allocate FPGA resources between the network method process manager and the video analytics process manager.

9. A terminal device, characterized in that, The terminal device includes: an accelerator, a memory, and a processor based on a multi-mirror FPGA according to any one of claims 1-5, wherein the accelerator is configured to execute the accelerator implementation method according to any one of claims 6 to 8.

10. A computer-readable storage medium storing an accelerator implementation program thereon, characterized in that, When the accelerator implementation program is executed by the processor, it implements the steps of the accelerator implementation method as described in any one of claims 6 to 8.

Images