FPGA (Field Programmable Gate Array) hardware system based on intelligent sensing algorithm and deployment method thereof

Abstract

The invention discloses an FPGA (Field Programmable Gate Array) hardware system based on an intelligent sensing algorithm and a deployment method thereof, and relates to the technical field of intelligent video image processing, and the FPGA hardware system comprises a reconfigurable sensing module, a compatible FPGA module, an expandable external storage module and a customizable information processing module, through the technical means of a configurable hardware system scheme, a pipelined deployment process method, an extensible external storage module, a customized information processing module and the like, the efficient, flexible and extensible FPGA hardware system based on the intelligent sensing algorithm is provided. The system can meet the deployment requirements of different intelligent sensing algorithms, improves the performance and processing efficiency of the algorithms, and can be widely applied to the fields of industry, aerospace, deep space exploration and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent video image processing, and in particular to an FPGA hardware system based on an intelligent perception algorithm and a deployment method thereof. Background Art

[0002] Intelligent perception technology refers to mapping the signals of the physical world to the digital world through hardware devices such as cameras, microphones or other sensors, with the help of cutting-edge technologies such as voice recognition and image recognition, and then further elevating these digital information to a cognitive level. The intelligent perception algorithm is the core of intelligent perception technology, and it serves as the hub for complex processing from data input to output. Since the perception algorithm is the key to linking the physical world and the digital world. Based on the latest research results, it is shown that the indicators of the perception algorithm implemented by deep learning are significantly improved compared with the traditional method. Therefore, deploying intelligent perception algorithms based on deep learning is the trend of the future. The intelligent perception algorithm discussed in this article is implemented by deep learning, rather than the traditional implementation of specific object algorithms. Among them, the intelligent perception algorithm based on deep learning is that the machine obtains information from part of the sample data by itself, and after a certain period of training, it finds general rules from it and predicts the information of unknown data. Deep learning to implement intelligent perception algorithms is different from traditional methods. It is mainly based on the method of deep neural network to implement the model architecture, and combines the back propagation and gradient descent methods to reconstruct the parameters of related components in the model, thereby meeting the needs of algorithm implementation. As we all know, the mainstream platform for implementing deep learning algorithms is to use GPU for training and implementation, so most intelligent perception algorithms are also deployed based on GPU. Among them, NVIDIA's GPU graphics card, with its rich scalability and good CUDA ecosystem, has created its current market position. In addition, due to the high price of GPU, and its size, cost, power consumption, latency and other aspects, there are certain shortcomings. However, FPGA has certain advantages in these indicators, so it is forward-looking to deploy intelligent perception algorithms based on deep learning on FPGA. In addition, due to the reconfigurability of FPGA and the superiority of edge deployment, especially in the context of AIOT, it can directly connect to various digital sensors, obtain relevant data, and process them. However, there is a problem in the industry: how to transplant intelligent perception algorithms to FPGA and run them efficiently. The reason is that FPGA itself is closer to the underlying circuit, and its description ability at the abstract level still has shortcomings. Although high-level synthesis tools such as HLS have appeared, deploying intelligent perception algorithms on FPGA is still not as convenient as writing a standalone software. Most software implementations only require a Visual Studio integrated software development environment with a Windows system. The deployment of FPGA requires the cooperation of multiple software integration environments and actual circuit-level operation testing, which is indeed very different.

[0003] In exploring how to deploy intelligent perception algorithms on FPGA, companies such as Labview, MATLAB, Google, AMD, and Intel have conducted a lot of research and attempts. In summary, platform companies such as Labview and MATLAB will combine their own software with FPGA and build a large number of libraries to meet the needs of designers to customize or directly call them on their basis, but the disadvantage is that they are limited by the device selection and driver version specified by the developer. If designers do not choose the corresponding hardware platform or library, they will not be able to develop subsequent perception algorithms. Device manufacturers such as AMD and Intel have developed corresponding underlying platforms. Intel has OpenCL, and AMD has Vitis. This platform is compatible with most of its FPGA models and basically meets the needs of developers. However, it also has the disadvantage that the underlying platform is too large, and design and development personnel need to go through a lot of development and learning, and have the ability to be compatible with software and hardware before they can deploy intelligent perception algorithms. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide an FPGA hardware system based on an intelligent perception algorithm and a deployment method thereof, which can better deploy and apply intelligent perception technology at the edge.

[0005] In the first aspect, the purpose of the present invention can be achieved by the following technical solution: an FPGA hardware system based on an intelligent perception algorithm, comprising:

[0006] Reconfigurable sensor module: used to collect data from the intelligent perception signals of the camera, IMU and radar, obtain the collected data, and input the collected data into a compatible FPGA module for processing using a unified protocol format;

[0007] Compatible FPGA module: used to perform different signal preprocessing and data screening on the received collected data according to different collected data signals, obtain processed data, input the processed data into the pre-established intelligent perception algorithm network model, output the final information result, and input the final information result into the customizable information processing module;

[0008] Expandable external storage module: used to pre-set and store the relevant weight parameters and instructions in the data collection processing process of the compatible FPGA module, and interactively process the collected data;

[0009] Customizable information processing module: used to package the final information results according to the standard protocol or the custom protocol after receiving the final information results, and then output and display the processed final information results.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the system further includes: the compatible FPGA module is capable of performing mounting and replacement on FPGAs of different models and capacities.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the system further includes: the compatible FPGA module closes and opens some components of the subordinate reconfigurable sensing module, or discards data, according to information from the host computer.

[0012] In combination with the first aspect, in some implementations of the first aspect, the system further includes: the expandable external storage module mounts different types of storage devices.

[0013] In combination with the first aspect, in some implementations of the first aspect, the system further includes: the customizable information processing module customizes the format of the output information to meet the requirements of different external interface devices.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the system further includes: the external interfaces equipped with the customizable information processing module include: multiple standard communication interfaces Ethernet, Fiber, USB3.0 and PCIE, and a configurable LVDS interface is also reserved.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the system further includes: the process of inputting the processed data into a pre-established intelligent perception algorithm network model in the compatible FPGA module, and outputting the final information result includes the following steps:

[0016] Build a pre-processing IP processing chain and a reconfigurable deep learning accelerator on a compatible FPGA module. The pre-processing IP processing chain includes the signal pre-processing of different smart sensor devices.

[0017] The obtained weight parameters and instruction parameters of model disassembly are stored in an expandable external storage module and arranged in a certain storage order to meet the requirements of fast reading and loading;

[0018] The compatible FPGA module deduces the intelligent perception algorithm network model layer by layer. Before completing the calculation of each layer of the network, it will place the network weights and instructions of the next layer in the high-bandwidth memory HBM closest to the FPGA within a set cycle, while other unloaded instructions and weights are placed in DDR3 or FLASH memory and flow into HBM.

[0019] The intelligent perception algorithm network model completes the calculation and processing of weights and data through instructions, outputs the final information results, and inputs the information results into a customizable information processing module.

[0020] In combination with the first aspect, in some implementations of the first aspect, the system further includes: the intelligent perception algorithm network model reduces the consumption of FPGA resources through operator fusion optimization and floating point to fixed point weight quantization:

[0021] The calculation formula of convolution is expressed as:

[0022] z=ω*x+b

[0023] The calculation formula of BN is:

[0024]

[0025] Substitute the formula of the convolutional layer into the equation, perform fusion operations and simplify it. The simplified formula is:

[0026]

[0027] After combining the above formulas, we can get new values ​​of ω and b:

[0028]

[0029] The model quantization method used is linear quantization after training, and the quantization formula is:

[0030]

[0031] Where s is the scaling factor and z represents the zero point.

[0032] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a deployment method of an FPGA hardware system based on an intelligent perception algorithm, the method comprising the following steps:

[0033] According to the actual application scenario, the training set, test set and hyperparameters of the perception algorithm are established, and the deep learning network of the perception algorithm is built. The perception algorithm is trained through a high-performance GPU, and the weight parameters are converged using a reasonable training strategy to obtain the training weights of the FPGA model. Then, the floating-point number conversion of the weight parameters is completed using the quantization principle.

[0034] Add intelligent perception algorithm network model and use calculation number and other compilation methods to realize the generation of instruction set, and save the relevant instruction data format;

[0035] Implement operator functions through high-level tools such as HLS, Spinal hdl or modelsim, and cascade related operators according to the network level to implement the network;

[0036] According to the model construction of deep learning network algorithm, the relevant core operators are cascaded to realize the synthesizable network engineering, and the synthesizable network engineering is simulated and realized after synthesis to obtain the bit stream program that meets the test requirements;

[0037] The collected data is input into the FPGA model, and the corresponding weight parameters can be stored in an expandable external memory. The corresponding weight parameters are loaded through instruction calls according to the requirements of the FPGA model operation to obtain the final information results.

[0038] Run the corresponding model on the FPGA and process the data to obtain the expected perception results, and output the information flow to the corresponding interface.

[0039] In another aspect of the present invention, in order to achieve the above object, a device is disclosed, comprising:

[0040] one or more processors;

[0041] A memory for storing one or more programs;

[0042] When one or more of the programs are executed by one or more of the processors, one or more of the processors

[0043] Beneficial effects of the present invention:

[0044] The present invention provides a configurable hardware system solution: a reconfigurable sensor module and a compatible FPGA module, so that the hardware system can be flexibly configured and expanded as needed. Such configurability makes the system applicable to FPGAs of different models and capacities, solving the limitation that the traditional hardware system only supports one type of FPGA. At the same time, the compatible FPGA module can also control and manage the sensor module according to the information of the host computer, thereby enhancing the flexibility and controllability of the system.

[0045] Pipeline deployment process method: Provides a pipeline deployment process method, which effectively improves the processing efficiency and response speed of the system. By loading the weights and instructions of the next layer in the high bandwidth memory (HBM), data transmission and loading time can be reduced, and computing efficiency can be improved. This pipeline deployment process method enables the system to complete the processing tasks of the intelligent perception algorithm more efficiently.

[0046] FPGA hardware system based on intelligent perception algorithm Scalable external storage module: Provides scalable external storage module, supports different types of storage devices, and can flexibly configure storage according to the needs of designers. This enables the system to meet the storage bandwidth requirements of different intelligent algorithms, providing greater freedom and scalability.

[0047] Customized information processing module: A customizable information processing module is provided, which can customize the output and display of processed information according to the designer's needs. By supporting multiple standard communication interfaces and configurable LVDS interfaces, it can meet the needs of different external interface devices and improve the compatibility and scalability of the system.

[0048] Improve the performance of the intelligent perception algorithm: A new deep learning accelerator is provided, which can extract data features and calculate weight parameters through self-learning, thereby improving the performance of the intelligent perception algorithm. The application of this accelerator enables the system to extract features and predict more quickly, improving the accuracy and efficiency of the intelligent perception algorithm.

[0049] In summary, the present invention provides an efficient, flexible and scalable FPGA hardware system based on intelligent perception algorithm through configurable hardware system solutions, pipeline deployment process methods, expandable external storage modules and customized information processing modules. The system can meet the deployment requirements of different intelligent perception algorithms and improve the performance and processing efficiency of the algorithms. It can be widely used in the fields of industry, aerospace, deep space exploration, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 It is a schematic diagram of the composition of the FPGA hardware system based on the intelligent perception algorithm of the present invention;

[0052] Figure 2 This is a flow chart of the FPGA-based intelligent perception algorithm deployment method in the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Embodiment 1:

[0055] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0056] FPGA: FPGA (Field Programmable Gate Array) is a product further developed on the basis of programmable devices such as PAL (Programmable Array Logic) and GAL (General Array Logic). It appears as a semi-custom circuit in the field of Application Specific Integrated Circuit (ASIC), which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices.

[0057] Protocol packaging: The protocol packaging and unpacking process refers to the process of encapsulating and parsing original data (such as messages, files, etc.) according to the format specified by the protocol in network communication.

[0058] The process includes the following steps:

[0059] 1. Data segmentation: The original data is segmented according to the maximum transmission unit (MTU) specified by the protocol to meet the requirements of network transmission. Each segment is usually called a data packet or frame.

[0060] 2. Add header information: Add the header information required by the protocol to each data packet. The header information includes source address, destination address, data packet type, sequence number, etc. This information helps the receiver to correctly parse and process the data packet.

[0061] 3. Calculate the checksum: To ensure the integrity of data during transmission, a checksum field is usually added to the header or tail of the data packet. The sender calculates the checksum of the data packet, and the receiver can check it after receiving the data to determine whether the data has been modified or lost.

[0062] 4. Packing: Pack the data packets with added header information and checksum according to the format specified by the protocol to form a complete data message.

[0063] like Figure 1 As shown, an FPGA hardware system based on intelligent perception algorithm includes:

[0064] Reconfigurable sensor module: used to collect data from the intelligent perception signals of the camera, IMU and radar, obtain the collected data, and input the collected data into a compatible FPGA module for processing using a unified protocol format;

[0065] Among them, the reconfigurable sensor module can acquire and transmit the information of different sensor modules to the compatible FPGA module according to system requirements;

[0066] Compatible FPGA module: used to perform different signal preprocessing and data screening on the received collected data according to different collected data signals, obtain processed data, input the processed data into the pre-established intelligent perception algorithm network model, output the final information result, and input the final information result into the customizable information processing module;

[0067] Among them, the compatible FPGA module can be mounted and replaced with FPGAs of different models and capacities, solving the problem that a hardware system can only have one model or capacity of FPGA, and avoiding the problem of developing a larger capacity FPGA and needing to redevelop the hardware system;

[0068] Furthermore, the compatible FPGA module can close and open some components of the subordinate reconfigurable sensor module, or discard data, etc. according to the information of the host computer;

[0069] The compatible FPGA module can perform different signal preprocessing and data screening on the collected data according to the different signals, and send the processed data to the intelligent perception algorithm network (a deep learning network) for processing, and obtain useful information of the sensor module after processing;

[0070] The process of inputting the processed data into the pre-established intelligent perception algorithm network model in the compatible FPGA module and outputting the final information result includes the following steps:

[0071] Step 1: Build a pre-processing IP processing chain and a reconfigurable deep learning accelerator on a compatible FPGA module. The processing IP processing chain includes signal pre-processing of different smart sensor devices. For example, the camera module must include de-mosaicing, automatic white balance, automatic exposure / focus, image noise reduction, and color correction.

[0072] Step 2: The deep learning accelerator is a tool for implementing the intelligent perception algorithm. It can extract the features of relevant data through self-learning, obtain relevant weight parameters through feature matching, and finally load the relevant parameters for prediction.

[0073] Step 3: The obtained weight parameters and instruction parameters of the model disassembly can be stored in an expandable external storage module and arranged in a certain storage order to meet the needs of fast reading and loading.

[0074] Step 4: The compatible FPGA module can deduce the network of the perception model layer by layer. Before completing the calculation of each layer of the network, it will place the network weights and instructions of the next layer in the high-bandwidth memory (HBM) closest to the FPGA within a certain period of time. Other unloaded instructions and weights can be placed in other memories such as DDR3 or FLASH and piped into the HBM.

[0075] Step 5: The perception network completes the calculation and processing of weights and data through instructions, outputs the final information results, and outputs the information results to a customizable information processing module. The customizable information processing module can further organize the relevant information, integrate it, output it, and display it.

[0076] The intelligent perception algorithm network model can reduce the consumption of FPGA resources through operator fusion optimization and floating-point to fixed-point weight quantization. As follows:

[0077] The calculation formula of convolution can be expressed as:

[0078] z=ω*x+b

[0079] The calculation formula of BN is:

[0080]

[0081] The formula of the convolutional layer can be brought in to perform fusion operations and simplify. The simplified formula is:

[0082]

[0083] After merging the above formulas, we can get new values ​​of ω and b, namely:

[0084]

[0085] Replacement is performed during inference to complete the operator fusion operation. This is equivalent to making certain modifications to the convolution kernel, but does not increase the computational complexity of the convolution layer, and also saves the computational complexity of the BN operation. For model quantization, the main idea is to quantize Float32-bit data to INT8-bit data, which can compress the model parameters to one-fourth of the original. The model quantization method used is post-training linear quantization, and the quantization formula is:

[0086]

[0087] Where s is the scaling factor and z represents the zero point.

[0088] Expandable external storage module: used to pre-set and store the relevant weight parameters and instructions in the data collection processing process of the compatible FPGA module, and interactively process the collected data;

[0089] Among them, the expandable external storage module mounts different types of storage devices; such as DDR3, DDR4, FLASH, EMMC and other memories, which can add and reduce storage configurations according to the different needs of designers to meet the storage bandwidth requirements of different intelligent algorithms.

[0090] Customizable information processing module: used to package the final information results according to the standard protocol or the custom protocol after receiving the final information results, and then output and display the processed final information results.

[0091] Among them, the customizable information processing module customizes the format of the output information to meet the needs of different external interface devices. It is equipped with a variety of standard communication interfaces such as Ethernet, Fiber, USB3.0 and PCIE, etc., and also has a configurable LVDS interface to meet the designer's customized needs.

[0092] Specifically, the present invention is further described below by way of embodiments:

[0093] 1. Applied in aerospace and deep space exploration fields;

[0094] The aerospace and deep space exploration fields have high requirements for hardware systems with high performance, reliability and real-time performance. The FPGA hardware system and deployment method of the present invention can be applied to the following scenarios:

[0095] (1) Aerospace Navigation System: This hardware system can be integrated into the aerospace navigation system to achieve high-precision navigation and positioning functions. Through intelligent perception algorithms and customized information processing modules, the system can perceive and analyze the aircraft's environmental information in real time, including weather, airspace conditions, etc., thereby providing more accurate navigation guidance and flight path planning. The configurability and scalability of the system make it suitable for different types of aerospace vehicles, including aircraft, satellites, and rockets.

[0096] (2) Deep space exploration missions: In deep space exploration missions, this hardware system can be applied to intelligent perception and autonomous decision-making tasks. By deploying the system on a probe or drone, it is possible to perceive and analyze target celestial bodies, including image processing, target recognition, and trajectory prediction. The high performance and low power consumption of the system make it suitable for long-term operation and autonomous decision-making deep space exploration missions, providing strong support for scientific research and exploration.

[0097] 2. Applied to industry, automotive navigation, intelligent sensing prototype development, etc.;

[0098] In the fields of industrial, automotive navigation and intelligent perception prototyping, the patented FPGA hardware system and deployment method can be applied to the following scenarios:

[0099] (1) Industrial Automation: This hardware system can be applied to the field of industrial automation to achieve intelligent perception and control. By deploying the system on industrial equipment, it can perceive environmental information in real time, monitor equipment status, and perform data analysis and decision-making. For example, using the system for quality inspection and fault diagnosis on a production line can improve production efficiency and product quality and reduce production costs. The system's configurability and high performance make it suitable for different types of industrial equipment and automation tasks.

[0100] (2) Car navigation and driving assistance: In car navigation and driving assistance systems, this hardware system can be applied to intelligent perception and road condition analysis. By deploying this system in the car, it can perceive and analyze the information of roads, traffic signals and other vehicles in real time, and provide accurate navigation guidance and driving assistance functions. For example, through intelligent perception algorithms, it can identify traffic signs, predict traffic congestion, and provide optimal route planning and driving suggestions to improve driving safety and driving efficiency. The real-time and reliability of this system make it suitable for real-time decision-making and control needs in the automotive field.

[0101] (3) Intelligent perception prototype development: The hardware system and deployment method can also be applied to intelligent perception prototype development. Researchers and engineers can use the system to quickly build intelligent perception prototypes and perform algorithm verification and performance evaluation. Through flexible hardware configuration and customized information processing modules, they can quickly deploy and debug different intelligent perception algorithms, accelerate the prototype development process, and provide support for the research and application of intelligent perception technology. The programmability and high performance of the system make it suitable for a variety of intelligent perception prototype development needs, including image processing, speech recognition, target detection and other fields.

[0102] Embodiment 2: A deployment method of an FPGA hardware system based on an intelligent perception algorithm, the method comprising the following steps:

[0103] Step 1: First, we establish the training set, test set, and hyperparameters of the perception algorithm according to the actual application scenario, and build a deep learning network for the perception algorithm. We implement the training of the algorithm through a high-performance GPU, and use a reasonable training strategy to achieve the convergence of the weight parameters to obtain the model training weights. Then, we use the quantization principle to convert the floating-point number of the weight parameter to a fixed-point number, such as converting 32-bit float to 8-bit int.

[0104] Step 2: Load the model and use compilation methods such as calculation numbers to generate the instruction set, and save the relevant instruction data format to meet the needs of subsequent instruction loading.

[0105] Step 3: Since there are a large number of operators in the model, operators are the basis for realizing deep learning algorithm networks. Operator functions can be implemented through high-level tools such as HLS, Spinal hdl or modelsim (verilog), and the relevant operators can be cascaded according to the network level to realize the network.

[0106] Step 4: According to the model construction of the deep learning network algorithm, the relevant core operators are cascaded to realize a synthesizable network project, and the network project is simulated and implemented after synthesis to obtain a bit stream program that meets the test requirements.

[0107] In step 5, data is acquired by the reconfigurable sensing module and input into the FPGA model. The corresponding weight parameters can be stored in an expandable external memory and loaded through instruction calls according to the requirements of the model operation.

[0108] Step 6: Run the corresponding model on the FPGA and process the data to obtain the expected perception results, and output the information flow to the corresponding interface.

[0109] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0110] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0112] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. An FPGA hardware system based on intelligent perception algorithm, characterized in that: include: Reconfigurable sensor module: used to collect data from the intelligent perception signals of the camera, IMU and radar, obtain the collected data, and input the collected data into a compatible FPGA module for processing using a unified protocol format; Compatible FPGA module: used to perform different signal preprocessing and data screening on the received collected data according to different collected data signals, obtain processed data, input the processed data into the pre-established intelligent perception algorithm network model, output the final information result, and input the final information result into the customizable information processing module; Expandable external storage module: used to pre-set and store the relevant weight parameters and instructions in the data collection processing process of the compatible FPGA module, and interactively process the collected data; Customizable information processing module: used to package the final information results according to the standard protocol or the custom protocol after receiving the final information results, and then output and display the processed final information results.

2. The FPGA hardware system based on the intelligent perception algorithm according to claim 1, characterized in that: The compatible FPGA module can be mounted and replaced on FPGAs of different models and capacities.

3. The FPGA hardware system based on the intelligent perception algorithm according to claim 2 is characterized in that: The compatible FPGA module closes and opens some components of the subordinate reconfigurable sensor module, or discards data according to the information of the host computer.

4. The FPGA hardware system based on the intelligent perception algorithm according to claim 1, characterized in that: The expandable external storage module mounts different types of storage devices.

5. The FPGA hardware system based on the intelligent perception algorithm according to claim 1, characterized in that: The customizable information processing module customizes the format of the output information to meet the requirements of different external interface devices.

6. The FPGA hardware system based on the intelligent perception algorithm according to claim 5, characterized in that: The customizable information processing module is equipped with external interfaces including: multiple standard communication interfaces Ethernet, Fiber, USB3.0 and PCIE, and a configurable LVDS interface.

7. The FPGA hardware system based on intelligent perception algorithm according to claim 3 is characterized in that: The process of inputting the processed data into the pre-established intelligent perception algorithm network model in the compatible FPGA module and outputting the final information result includes the following steps: Build a pre-processing IP processing chain and a reconfigurable deep learning accelerator on a compatible FPGA module. The pre-processing IP processing chain includes the signal pre-processing of different smart sensor devices. The obtained weight parameters and instruction parameters of model disassembly are stored in an expandable external storage module and arranged in a certain storage order to meet the requirements of fast reading and loading; The compatible FPGA module deduces the intelligent perception algorithm network model layer by layer. Before completing the calculation of each layer of the network, it will place the network weights and instructions of the next layer in the high-bandwidth memory HBM closest to the FPGA within a set cycle, while other unloaded instructions and weights are placed in DDR3 or FLASH memory and flow into HBM. The intelligent perception algorithm network model completes the calculation and processing of weights and data through instructions, outputs the final information results, and inputs the information results into a customizable information processing module.

8. The FPGA hardware system based on the intelligent perception algorithm according to claim 7, characterized in that: The intelligent perception algorithm network model reduces the consumption of FPGA resources through operator fusion optimization and floating-point to fixed-point weight quantization: The calculation formula of convolution is expressed as: z=ω*x+b The calculation formula of BN is: Substitute the formula of the convolutional layer into the equation, perform the fusion operation and simplify it. The simplified formula is: After combining the above formulas, we can get new values ​​of ω and b: The model quantization method used is linear quantization after training, and the quantization formula is: Where s is the scaling factor and z represents the zero point.

9. A deployment method for an FPGA hardware system based on an intelligent perception algorithm, characterized in that: The method comprises the following steps: According to the actual application scenario, the training set, test set and hyperparameters of the perception algorithm are established, and the deep learning network of the perception algorithm is built. The perception algorithm is trained through a high-performance GPU, and the weight parameters are converged using a reasonable training strategy to obtain the training weights of the FPGA model. Then, the floating-point number conversion of the weight parameters is completed using the quantization principle. Add intelligent perception algorithm network model and use calculation number and other compilation methods to realize the generation of instruction set, and save the relevant instruction data format; Implement operator functions through high-level tools such as HLS, Spinal hdl or modelsim, and cascade related operators according to the network level to implement the network; According to the model construction of deep learning network algorithm, the relevant core operators are cascaded to realize the synthesizable network engineering, and the synthesizable network engineering is simulated and realized after synthesis to obtain the bit stream program that meets the test requirements; The collected data is input into the FPGA model, and the corresponding weight parameters can be stored in an expandable external memory. The corresponding weight parameters are loaded through instruction calls according to the requirements of the FPGA model operation to obtain the final information results. Run the corresponding model on the FPGA and process the data to obtain the expected perception results, and output the information flow to the corresponding interface.

10. A device, characterized in that: include: one or more processors; A memory for storing one or more programs; When one or more of the programs are executed by one or more of the processors, one or more of the processors implement an FPGA hardware system based on an intelligent perception algorithm as described in any one of claims 1-8.

Images