A FPGA deployment system and method for a robot's six-degree-of-freedom visual target detection DETR model

Abstract

A FPGA deployment system and method for a robot six-degree-of-freedom visual target detection DETR model, the DETR model in the system includes a ResNet50 feature extraction part and a multi-head attention encoding and decoding part, and the FPGA module includes 2 main modules and 11 slave modules, a total of 13 modules; in this method, the PC first obtains the image data and stores it in the FPGA board card and then obtains the image data from it; preprocessing is performed, and then feature extraction based on ResNet50 is performed; multi-head attention encoding is performed; multi-head attention decoding is performed to obtain the rotation and translation data of the six-degree-of-freedom posture of the target; the above six-degree-of-freedom posture rotation and translation data are stored in the FPGA board card and then sent to the host computer to complete the target detection and guide the robot to perform the six-degree-of-freedom grasping of the target. The present invention expands the target detection model DETR to the robot six-degree-of-freedom grasping task, and deploys the deep learning model part on the FPGA, improves the customizability and easy layout of the entire grasping system, reduces resource consumption, and saves production costs.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision and AI chips, and specifically relates to an FPGA deployment system and method for a robot six-degree-of-freedom visual target detection DETR model. Background Art

[0002] With the rise and development of deep learning in recent years, the robot's six-degree-of-freedom grasping method has been increasingly integrated with deep learning models. However, the training of deep learning models consumes a lot of GPU resources. The operation of GPUs requires a lot of electricity and generates a lot of heat, which requires supporting cooling equipment for cooling. In a factory environment, a large number of GPUs may be inconvenient to arrange and maintain. At the same time, the environmental factors in the factory may be more complicated, and the hardware may need to overcome environmental factors such as dust, humidity, movement, and power distribution.

[0003] As a customizable processor, FPGA is flexible in application and can be programmed according to the actual requirements of the factory. At the same time, it has lower power requirements than GPU, and provides good performance in terms of computing speed and throughput, longer service life, stronger adaptability to special environments, more portable, and easier to replace. Therefore, it is of great prospect and significance to arrange deep learning models in FPGA to meet the production needs of factories. Summary of the invention

[0004] Inspired by the target detection model DETR, this invention extends it to the robot's six-degree-of-freedom grasping task while meeting the factory robot's grasping needs. The deep learning model is partially deployed on the FPGA to improve the customizability and ease of deployment of the entire grasping system, reduce resource consumption, and save production costs.

[0005] An FPGA deployment system for a robot six-degree-of-freedom visual target detection DETR model is proposed. The DETR model includes a ResNet50 feature extraction part and a multi-head attention encoding and decoding part. The DETR six-degree-of-freedom detection FPGA module includes 2 main modules and 11 slave modules, a total of 13 modules. The modules are connected through an AXI-4 bus matrix; among them, the two main modules are a PCI-E main module with DMA working mode and an Ultrascale+MPSOC (DETR Core) main module; the slave modules include CONV 7×7,2, CONV 3×3,1, CONV 1×1,S, BRAM, MaxPooling (CONV 3×3,2), RELU, Softmax, ROOT, ADD, Mult, DDR4, a total of 11 slave modules.

[0006] A FPGA deployment method for a robot six-degree-of-freedom visual target detection DETR model includes the following steps:

[0007] Step 1, the PC first obtains the image data and stores it in the FPGA board, and then obtains the image data from the FPGA board;

[0008] Step 2: preprocess the image data and then perform feature extraction based on ResNet50;

[0009] Step 3: Perform multi-head attention encoding on the extracted feature data;

[0010] Step 4: Based on the encoding results, multi-head attention decoding is performed to obtain the rotation and translation data of the six-degree-of-freedom posture of the target;

[0011] Step 5: The above six-degree-of-freedom posture rotation and translation data are stored in the FPGA board and then sent to the host computer to complete the target detection and guide the robot to grasp the target with six degrees of freedom.

[0012] The beneficial effects achieved by the present invention are:

[0013] (1) Part of the deep learning model is deployed on the FPGA, which improves the customizability and ease of deployment of the overall system. The program can be adjusted on-site according to actual needs to achieve better targeted recognition effects and robot grasping performance.

[0014] (2) The overall system uses FPGA combined with a deep learning model to reduce the use of physical devices, reduce resource consumption, and save production costs.

[0015] (3) Through designs such as convolution and multi-head self-attention mechanisms, the efficiency and effectiveness of the algorithm are improved, and the target recognition effect is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a design diagram of the DETR six-degree-of-freedom detection FPGA module in an embodiment of the present invention.

[0017] Figure 2 This is a functional block diagram of FPGA in an embodiment of the present invention.

[0018] Figure 3 This is a functional block diagram of the Master main module in an embodiment of the present invention.

[0019] Figure 4 This is a functional block diagram of the Slave module in an embodiment of the present invention.

[0020] Figure 5 It is the DETR detection model scheduling model in the embodiment of the present invention.

[0021] Figure 6 Schematic diagram of the BottleNeck scheduling model in an embodiment of the present invention.

[0022] Figure 7 Schematic diagram of Multi-Head Self-Attention in an embodiment of the present invention.

[0023] Figure 8 Schematic diagram of FFN in an embodiment of the present invention.

[0024] Fig. 9 This is the convolution operation flow in an embodiment of the present invention.

[0025] Fig.10 It is a diagram of an actual overall structure example in an embodiment of the present invention.

[0026] Fig.11 This is a physical picture of the FPGA board in the embodiment of the present invention.

[0027] Fig.12 The figure is an overall flow chart of the system operation in an embodiment of the present invention.

[0028] Fig.13 1 is a diagram of the system hardware architecture in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0030] The embodiment of the present invention is mainly a field programmable gate array (hereinafter referred to as FPGA) deployment system of a robot six-degree-of-freedom visual target detection model (hereinafter referred to as DETR model) based on a self-attention encoding and decoding mechanism. The present invention patent implements the FPGA deployment method of each module of the six-degree-of-freedom visual target detection DETR model on FPGA. Figure 1 As shown in Figure 2, the scheduling process of each module is as follows: Figure 2 and Figure 3 shown.

[0031] The DETR model includes the residual neural network (hereinafter referred to as ResNet50) feature extraction part and the multi-head attention encoding and decoding part. Processing data requires a certain amount of computing power, while taking the cost into consideration. Therefore, the XCZU7EG chip in Xilinx ZynqUltraScale+ is selected. The design diagram of the DETR six-degree-of-freedom detection FPGA module is as follows: Figure 1 shown.

[0032] The FPGA part includes 2 master modules and 11 slave modules, a total of 13 modules, and the modules are connected through the Advanced Scalable Interface (AXI-4) bus matrix. Among them, the two master modules are the high-speed serial computer expansion bus (PCI-E) master module with direct memory access (DMA) working mode and the Ultrascale+MPSO C (DETR Core) master module; the slave modules include CONV 7×7,2 convolution slave module, CONV 3×3,1 convolution slave module, CONV1×1,S convolution slave module, BRAM block random access storage slave module, MaxPooling (CONV 3×3,2) maximum pooling slave module, RELU activation function slave module, Softmax normalization slave module, ROOT square root slave module, ADD addition slave module, Mult multiplication slave module, DDR4 storage slave module, a total of 11 slave modules. Among them, CONV represents the convolution operation, BRAM is the block random access memory inside the FPGA chip, RELU is the linear rectification activation function, and DDR4 is the FPGA onboard double rate synchronous dynamic random access memory.

[0033] The functions of the above 13 modules are as follows:

[0034] (1) The main module PCI-E DMA provides the communication function between the FPGA board and the PC. After the PC obtains the image data, the image data on the PC is stored in the DDR slave module of the FPGA board. After the FPGA processes the image data, the detection result data in the DDR slave module is stored in the PC.

[0035] (2) The main module Ultrascale+MPSOC (DETR Core) is used to implement the DETR detection model algorithm, and calls each slave module in time-sharing to achieve the following Figure 3-Figure 4 The DETR detection model shown.

[0036] (3) The slave module “CONV 7×7, 2” is a convolution kernel with a size of 7×7 and a stride of 2. Stage 0 in the ResNet50 feature extraction part calls this slave module.

[0037] (4) The slave module "CONV 3×3, 3" is a convolution kernel of size 3×3 and stride 1. The bottleneck feature extraction (hereinafter referred to as BottleNeck) operation calls this slave module.

[0038] (5) The slave module "CONV 1×1, S" is a convolution kernel of size 1×1 and stride s (s=1 or 2). The BottleNeck1 and BottleNeck2 operations call this slave module.

[0039] (6) The slave module “MaxPooling(CONV 3×3,2)” is a maximum pooling operation, which is essentially a convolution kernel of size 3×3 and stride 2. Stage 0 in the ResNet50 feature extraction part calls this slave module. The module’s input data size is 2×64×320×240, and the output data size is 2×64×160×120.

[0040] (7) The module ROOT is the square root operation, and the input and output data sizes of the module are the same. This module is called by Stage0 to Stage4 in the ResNet50 feature extraction step and the Add&Norm operation in the multi-head attention encoding and decoding steps.

[0041] (8) The module RELU is the activation function operation, and the input and output data sizes of the module are the same. Stage 0 to Stage 4 in the ResNet50 feature extraction part call this module.

[0042] (9) The module Softmax is a normalization operation, and the input and output data sizes of the module are the same. The Multi-Head Self-Attention step and the FFN feedforward network step in the multi-head attention encoding and decoding call this module.

[0043] (10) The slave module ADD is an addition operation, which performs the addition operation on each pixel of the image data. The slave module is called for the addition of data in the BottleNeck step, multi-head attention encoding, and decoding. Among them: BottleNeck implements the addition of data of five sizes: 2×64×160×120, 2×256×160×120, 2×512×80×60, 2×1024×40×30, and 2×2048×20×15; the multi-head attention encoding and decoding process implements the addition of data of two sizes: 300×2×256 and 100×2×256.

[0044] (11) The slave module Mult module is a multiplication operation that performs multiplication operations on the data. The multiplication of data in the multi-head attention encoding and decoding steps calls this slave module to implement data multiplication operations of four sizes: 300×2×256, 100×2×256, 16×300×32, and 16×100×32.

[0045] (12) DDR4 slave module is a storage module. The image data and six-degree-of-freedom detection results transmitted by PCI-E are stored in this module.

[0046] (13) The BRAM slave module is a block RAM module, which is used to read the image data stored in DDR4 in blocks and store the intermediate variables of the model running process.

[0047] The FPGA functional block diagram based on the above deployment is as follows Figure 2 As shown, it includes an AXI Interconnect bus interconnection module, two Master modules mounted on the left side of the AXI Interconnect module, 11 Slave modules mounted on the right side of the AXI Interconnect module, a Utility Buffer data buffer module, a Clocking Wrizard timer module, an ILA integrated logic analyzer module, and a Processor System Reset processor system reset module.

[0048] like Figure 3 The main module part is shown. Among them: Figure 2 The DMA / Bridge Subsy stem for PCIExpress (PCIe) module in Figure 1 PCI-E DMA master module; Figure 3 The Zynq UltraScale+ MPSoC module in Figure 1 The Ultrascale+MPSOC (DETR Core) main module. The Utility Buffer is a cache, and the Clocking Writer is used for clock division and multiplication. The DMA / Bridge Subsystem for PCI Express (PCIe) module and the ZynqUltraScale+MPSoC module are connected to the AXI matrix bus connector AXI Interconnect.

[0049] like Figure 4 The slave module part is shown, which includes 11 slave modules, namely DDR4 SDRAM (MIG), BlockMemory Generator, Conv7x7S2_v1.0, Conv3x3Ss_v1.0, Conv1x1Ss_v1.0, MaxPoolingCore_v1.0, RELUCore_v1.0, ROOT_v1.0, MatrixAddCore_v1.0, MatrixMultCore_v1.0, SoftmaxCore_v1.0, corresponding to Figure 1The DDR4 storage slave module, BRAM block random access storage slave module, CONV 7×7,2 convolution slave module, CONV 3×3,1 convolution slave module, CONV 1×1,S convolution slave module, MaxPooling (CONV 3×3,2) maximum pooling slave module, RELU activation function slave module, ROOT square root slave module, ADD addition slave module, Mult multiplication slave module, and Softmax normalization slave module. Figure 4 In the ILA (Integrated Log Analyzer) module, it is an integrated logic analyzer module used for FPGA debugging; Figure 4 In the Processor System Reset module, the system is used to control the system initialization reset operation; Figure 4 In the example, the Block Memory Generator module is connected to the AXIInterconnect module through the AXI BRAM Controller module.

[0050] Figure 2 The Zynq UltraScale+MPSoC master module is used to schedule the slave modules to implement the six-degree-of-freedom detection DETR algorithm. The scheduling method of each module of the six-degree-of-freedom detection DETR model FPGA is as follows Figure 5 As shown in the figure, each module is divided into two steps according to the data flow direction, namely the ResNet50 feature extraction step and the multi-head attention encoding and decoding step. Among them, the ResNet50 feature extraction step consists of stage0, stage1, stage2, stage3, and stage4 sub-steps; the multi-head attention encoding and decoding step consists of two sub-steps: Encoder encoding and Decoder decoding.

[0051] The BottleNeck1 and BottleNeck2 processes in the stage1-stage4 substeps of the above ResNet50 are as follows Figure 6 (a) and Figure 6 (b) as shown.

[0052] The Multi-Head Self-Attention step in the above multi-head attention encoding and decoding steps is as follows Figure 7 As shown, the FFN feedforward network steps are as follows Figure 8 shown.

[0053] Input image data once, FPGA implements six-degree-of-freedom detection DETR algorithm, and Zynq UltraScale+MPSoC module calls Figure 4The number of times each Slave module function block (except the storage slave module) is performed is shown in Table 1. The specific steps are as follows.

[0054] Table 1 Statistics of the number of calls to each slave module function block

[0055]

[0056] (1) Obtain image data from the DDR module

[0057] The PC side stores the image data in the DDR4 SDRAM (MIG) slave module of the FPGA board through the DMA / Bridge Subsystem for PCI Express (PCIe) slave module. Fig. 9 The sliding window method reads the image data in the DDR4SDRAM (MIG) storage slave module, and sends the read pixel blocks to the Block Memory Generator, i.e., the BRAM block random access storage slave module. Fig. 9 , the pixel block data size is N×N, where N=7.

[0058] (2) ResNet50 feature extraction steps

[0059] according to Figure 5 In the ResNet50 feature extraction part, the above pixel block data is first sent to the Stage0 step: the ZynqUltraScale+MPSoC main module calls the Conv7x7S2_v1.0 slave module, BN normalization operation, RELUCore_v1.0 slave module, and MaxPoolingCore_v1.0 slave module operation for preprocessing, and outputs data of size 2×64×160×120. The above BN normalization operation requires calling MatrixAddCore_v1.0, MatrixMultCore_v1.0, and ROOT_v1.0 slave modules to obtain the input data x i The mean μ B and variance If the input is 2×64×160×120, it means processing data of size i=2, 3×160×120, and then substitute it into the normalization formula Complete the BN normalization operation. Performing a BN normalization operation requires calling 10 MatrixAddCore_v1.0, 8 MatrixMultCore_v1.0, and 2 ROOT_v1.0 slave modules.

[0060] The output data is sent to stage1, stage2, stage3, and stage4. The above stage1, stage2, stage3, and stage4 are composed of 1 BottleNeck1 and multiple BottleNeck2 steps. Among them, the number of BottleNeck2 data processing times of stage1, stage2, stage3, and stage4 are 2, 3, 5, and 2 times respectively. BottleNeck1 calls Conv1x1Ss_v1.0 from the module twice, Conv1x1Ss_v1.0(s=1) from the module once, Conv3x3Ss_v1.0 from the module once, BN normalization operations 4 times, and RELUCore_v1.0 from the module 3 times; BottleNeck2 calls Conv1x1Ss_v1.0 from the module twice, Conv3x3Ss_v1.0 from the module once, BN normalization operations 3 times, and RELUCore_v1.0 from the module 3 times. It should be noted that, in the Conv1x1Ss_v1.0 slave module, s=1 in Stage1, and s=2 in Stage2, Stage3, and Stage4.

[0061] Figure 2 The Zynq UltraScale+ MPSoC main module is based on Figure 5 In the ResNet50 feature extraction steps, the Conv1x1Ss_v1.0 slave module, BN normalization operation, RELUCore_v1.0 slave module, Conv3x3Ss_v1.0 slave module, Conv1x1Ss_v1.0 slave module, and MatrixAddCore_v1.0 slave module are called to obtain feature data of size 300×2×256 and realize the feature extraction function.

[0062] (3) Multi-head attention encoding steps

[0063] like Figure 5 As shown in the figure, the multi-head attention encoding part includes the Encoder encoding sub-step. The 6× in the Encoder encoding sub-step means that there are 6 identical Encoder encoders. The data first passes through the Encoder encoding sub-step in the multi-head attention encoding part. Figure 5 The operations corresponding to the Encoder encoding sub-steps are described in order as follows:

[0064] ①: If Figure 5 As shown in the Encoder encoding part, data with a size of 300×2×256 is sent to the Encoder encoding part.

[0065] ②: The Zynq UltraScale+MPSoc main module calls the MatrixMultCore_v1.0 slave module to perform multiplication operations on the data of size 300×2×256 in ①, and then calls the MatrixAddCore_v1.0 slave module to add the positional encoding data of size 300×2×256, and obtains the feature data sizes of Q, K, and V, all of which are 300×2×256. The obtained Q, K, and V feature data are sent to Multi-Head Self-Attention, and the Zynq UltraScale+MPSoc main module calls the MatrixMultCore_v1.0 slave module once to reconstruct the data into (2×8)×300×32, that is, the size of 16×300×32, and then calls the Matrix MultCore_v1.0 slave module 24 times and the SoftmaxCore_v1.0 slave module 8 times to process the data of size 16×300×32, and outputs feature data of size 300×2×256.

[0066] ③: The output data of step ② is sent to the Add&Norm addition and standardization layer. The Zynq UltraScale+MPSoc main module calls the MatrixAddCore_v1.0 slave module once to add the feature data of 300×2×256 output from ② to the data of 300×2×256 input from ①, and performs the Norm standardization operation. One Norm operation requires the ZynqUltraScale+MPSoc main module to call the MatrixAddCore_v1.0 slave module 5 times, the MatrixMultCore_v1.0 slave module 6 times, and the ROOT_v1.0 slave module once. The output feature data of 300×2×256 size is obtained.

[0067] ④: The output data of step ③ is sent to the FFN feedforward network step. After passing through the Linear layer, the ZynqUltraScale+MPSoc main module calls the MatrixMultCore_v1.0 slave module once and the MatrixAddCore_v1.0 slave module once to obtain the feature data with an output size of 300×2×256.

[0068] ⑤: The output data of step ④ and step ③ are sent to the same Add&Norm addition and normalization layer as ③. As in ③, the Zynq UltraScale+MPSoc main module calls the MatrixAddCore_v1.0 slave module 6 times, the MatrixMultCore_v1.0 slave module 6 times, and the ROOT_v1.0 slave module once. The output feature data of size 300×2×256 is obtained.

[0069] ⑥: Input the output of ⑤ obtained by this encoder into the next identical encoder, and repeat the above process 5 times to obtain the output of the final encoder encoding sub-step, which is still feature data of size 300×2×256.

[0070] (4) Multi-head attention decoding steps

[0071] like Figure 5 As shown, the multi-head attention decoding part includes a Decoder decoding sub-step. The 6× in the upper left corner of the Decoder decoding sub-step indicates that there are 6 identical Decoder decoders. The data obtained by the Encoder encoding sub-step is sent to the Decoder decoding sub-step in the multi-head attention decoding part. Figure 5 The operations corresponding to the Decoder decoding sub-steps are described in order as follows:

[0072] ①: If Figure 5 As shown in the Decoder decoding part, the position encoding data of the preset detection target learned through the offline training network, that is, the target query data with a size of 100×2×256, is sent to the Decoder decoding part.

[0073] ②: The Zynq UltraScale+MPSoc main module calls the MatrixMultCore_v1.0 slave module to perform multiplication operations on the target query data of size 100×2×256 in ①, and then calls the MatrixAddCore_v1.0 slave module to add the target query data of size 100×2×256, and obtains the feature data of Q, K, and V of size 100×2×256. The obtained Q, K, and V feature data are sent to the Multi-Head Self-Attention step, and the MatrixMultCore_v1.0 slave module is called once to reshape the data to (2×8)×100×32, that is, the size of 16×100×32. Then, the MatrixMultCore_v1.0 slave module is called 24 times and the SoftmaxCore_v1.0 slave module is called 8 times to process the size of 16×100×32 data, and the feature data of size 100×2×256 is output and sent to step ③.

[0074] ③: The output data of step ② is sent to the Add&Norm addition and standardization layer. The Zynq UltraScale+MPSoc main module calls the MatrixAddCore_v1.0 slave module once, adds the 100×2×256 feature data output by ② to the target query data with an input size of 100×2×256 in ①, and performs the Norm standardization operation. One Norm operation also requires the Zynq UltraScale+MPSoc main module to call the MatrixAddCore_v1.0 slave module 5 times, the MatrixMultCore_v1.0 slave module 6 times, and the ROOT_v1.0 slave module once. The output feature data with a size of 100×2×256 is obtained.

[0075] ④: The Zynq UltraScale+MPSoc main module calls the MatrixAddCore_v1.0 slave module twice, adds the output data of ③ to the target query data in ① to obtain Q, with a size of 100×2×256; adds the output data of ③ to the final output of the Encoder encoding substep in (3)⑥ and the position code to obtain K, with a size of 300×2×256; the final output of the Encoder encoding substep in (3)⑥ is used as V, with a size of 300×2×256. The Q, K, and V obtained this time are sent to Multi-HeadAttention. The Zynq UltraScale+MPSoc main module calls the MatrixMultCore_v1.0 slave module three times to reshape the data Q into (2×8)×100×32, that is, the size of 16×100×32; K and V are reshaped into (2×8)×300×32, that is, the size of 16×300×32. Then call the MatrixMultCore_v1.0 slave module 24 times and the SoftmaxCore_v1.0 slave module 8 times to process the above-mentioned data of sizes 16×100×32 and 16×300×32, and output feature data of size 100×2×256.

[0076] ⑤: The output data of step ④ is sent to the Add&Norm addition and standardization layer. The Zynq UltraScale+MPSoc main module calls the MatrixAddCore_v1.0 slave module once to add the feature data of 100×2×256 output from ③ to the target query data of 100×2×256 input from ④, and performs the Norm standardization operation. One Norm operation also requires the Zynq UltraScale+MPSoc main module to call the MatrixAddCore_v1.0 slave module 5 times, the MatrixMultCore_v1.0 slave module 6 times, and the ROOT_v1.0 slave module once. The output feature data of 100×2×256 size is obtained.

[0077] ⑥: The output data of step ⑤ is sent to the FFN feedforward network step. After passing through the Linear layer, the Zynq UltraScale+MPSoc main module calls the MatrixMultCore_v1.0 slave module once and the MatrixAddCore_v1.0 slave module once to obtain the feature data with an output size of 100×2×256.

[0078] ⑦: The output data of step ⑥ is sent to the Add&Norm addition and standardization layer. The Zynq UltraScale+MPSoc main module calls the MatrixAddCore_v1.0 slave module once to add the feature data of 100×2×256 output in ⑤ to the feature data of 100×2×256 output in ⑥, and performs the same Norm standardization operation. The Zynq UltraScale+MPSoc main module calls the MatrixAddCore_v1.0 slave module 5 times, the MatrixMul tCore_v1.0 slave module 6 times, and the ROOT_v1.0 slave module once. The output feature data of 100×2×256 size is obtained.

[0079] ⑧: The output of ⑦ obtained by the current decoder is input into the next identical decoder, and the same process is repeated 5 times. The outputs of the 6 decoders are stacked to obtain the final output feature data of 6×100×2×256. The final output feature data of 6×100×2×256 is sent to the two FFN feedforward network steps respectively. The Zynq UltraScale+MPSoc main module calls the MatrixMultCore_v1.0 slave module 3 times and the MatrixAddCore_v1.0 slave module 3 times to predict the rotation and translation data of the six-degree-of-freedom pose of the target.

[0080] The above six-degree-of-freedom posture rotation and translation data are stored in the DDR4 SDRAM (MIG) storage slave module, and then sent to the host computer through the PCI-E DMA master module, thereby guiding the robot to perform six-degree-of-freedom grasping of the target.

[0081] This invention is aimed at factory workshop scenarios. It deploys the deep learning model of processing and learning data in robot grasping on the FPGA processor. Under the premise of ensuring the robot grasping accuracy, it solves the problem of difficulty in placing GPU processors in the workshop for deep learning model deployment, improves the reliability and customizability of the processor under complex factory conditions, reduces energy consumption, and saves production costs. The main improvements include:

[0082] (1) A DETR 6D detection algorithm was designed for the robot's six-degree-of-freedom grasping task in a factory workshop scenario;

[0083] (2) A method for deploying the deep learning model of the detection algorithm to FPGA is proposed;

[0084] (3) Designed a custom IP Core called DETR Core to call each slave module;

[0085] (4) Designed the overall module structure inside the FPGA.

[0086] The present invention is mainly used in factory workshops, and the required target detection deep learning model is deployed on FPGA. The FPGA processor is small in size, easy to arrange and replace, which solves the problem that it is difficult to deploy GPU processors in factory workshops, improves the reliability and stability of the processor, and can customize the FPGA at any time according to the actual needs of the factory. Under the premise of ensuring the grasping accuracy, it reduces energy consumption to a certain extent and saves production costs.

[0087] During the manufacturing process of industrial parts, there may be dust or high temperature environments in the factory. The customized FPGA processor that deploys the deep learning model is small in size, has more options for spatial layout, has stronger adaptability to special environments such as dust and high temperature, and is more convenient to inspect and maintain, ensuring that the FPGA processor that deploys the target detection model processes the image data normally and guides the parts sorting robot to perform a certain degree of precision in grasping the parts. At the same time, the FPGA processor will not take up too much power resources, and will reduce energy consumption without affecting workshop production, thereby reducing production costs.

[0088] like Fig.10 The following is an example of the actual overall structure. Fig.13The hardware structure in the figure corresponds to each other. ① is the camera part, ② is the robot, ③ is the host computer, ④ is the robot controller, and ⑤ is the object to be grasped. The FPGA board is located in the robot controller ④. Fig.11 shown in the box.

[0089] The overall working process of the system is described as follows: Fig.12 As shown. The information data between the host computer and the robot is transmitted through the TCP / IP protocol, and the data between the host computer and the FPGA is transmitted through PCI E. First, the robot collects a set of required image data through the visual system and sends it to the host computer. Then, the host computer maps the image data into the FPGA memory through PCIE, and then the FPGA module equipped with the deep learning model processes the image data. The processed data is then mapped back to the host computer memory through PCIE. The host computer predicts based on the returned data and regresses the six-degree-of-freedom posture of the target. The regressed six-degree-of-freedom posture information is transmitted to the robot to guide the robot to perform six-degree-of-freedom grasping. After the robot grasps, it sends a signal to the host computer to indicate that the grasping is completed, and then reinitializes to perform the next grasping task. Repeat the above operations until all grasping tasks are completed.

[0090] The system hardware architecture is as follows Fig.13 As shown in the figure, the data between the camera and the host computer is transmitted via USB, the data between the host computer and the FPGA is transmitted via PCI-E, and the data between the host computer and the robot controller is transmitted via TCP / IP. The data between the two modules are transmitted bidirectionally.

[0091] The camera collects image data and transmits it to the host computer via USB. The host computer then maps the data to the FPGA via PCI-E. After processing the image data, the FPGA obtains the pose information data of the target object and maps it back to the host computer memory via PCI-E. The host computer that receives the pose information data then transmits the data to the robot controller via TCP / IP. After receiving the pose signal, the robot controller converts it into a double floating-point array and transmits it to the robot to control the robot to grasp the target with six degrees of freedom. After the robot completes the grasping, the robot controller receives the grasping completion signal and then transmits the grasping completion signal to the host computer via TCP / IP. After receiving the grasping completion signal, the host computer transmits the information to the camera for the next image data acquisition.

[0092] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A method for deploying a FPGA system for a robot six-degree-of-freedom visual target detection DETR model, characterized in that: The DETR model includes the ResNet50 feature extraction part and the multi-head attention encoding and decoding part. The DETR six-degree-of-freedom detection FPGA module includes 2 main modules and 11 slave modules, a total of 13 modules, and the modules are connected through the AXI-4 bus matrix; among them, the two main modules are the PCI-E main module with DMA working mode and the Ultrascale+ MPSOC main module; the slave modules include CONV 7×7,2, CONV 3×3,1, CONV 1×1,S, BRAM, MaxPooling CONV 3×3,2, RELU, Softmax, ROOT, ADD, Mult, DDR4, a total of 11 slave modules; The main module PCI-E DMA provides the communication function between the FPGA board and the PC. After the PC obtains the image data, the image data of the PC is stored in the DDR slave module of the FPGA board. After the FPGA processes the image data, the detection result data in the DDR slave module is stored in the PC. The main module Ultrascale+ MPSOC is used to implement the DETR detection model algorithm, and the DETR detection model is implemented by calling each slave module in time-sharing; The module ROOT is a square root operation, and the input and output data sizes of the module are the same; the module ADD is an addition operation, and the sum operation is performed on each pixel of the image data; the module Mult is a multiplication operation, and the data is multiplied; The DDR4 slave module is a storage module, in which the image data and six-degree-of-freedom detection results transmitted by PCI-E are stored; the BRAM slave module is a block RAM module, which is used to read the image data stored in DDR4 in blocks and store the intermediate variables of the model operation process; The method comprises the following steps: Step 1: The PC first obtains the image data and stores it in the FPGA board, and then obtains the image data from the FPGA board; Step 2: preprocess the image data and then perform feature extraction based on ResNet50; Step 3: Perform multi-head attention encoding on the extracted feature data; In step 3, the multi-head attention encoding part includes the Encoder sub-step. There are 6 identical Encoders in the Encoder sub-step. The data first passes through the first Encoder sub-step in the multi-head attention encoding part. The specific steps are as follows: Step 3-1, send the data to the Encoder encoding part; Step 3-2: Multiply the data in step 3-1 and add it to the positional encoding data to obtain three feature data: Q, K, and V. The obtained Q, K, and V feature data are sent to the Multi-Head Self-Attention mechanism. The UltraScale+ MPSoc main module reconstructs the data and then performs a size transformation on the data. Step 3-3, adding the feature data output from step 3-2 to the data in step 3-1, and performing a Norm normalization operation to obtain output feature data; Step 3-4: The output data of step 3-3 is sent to FFN, and the output feature data is obtained through the Linear layer; Step 3-5, the data output from step 3-4 and step 3-3 are subjected to the same adder normalization operation as step 3-3 to obtain output feature data; Step 3-6, the output feature data of step 3-5 finally obtained by the current encoder is input into the next identical encoder, and the same process is repeated 5 times to obtain the output feature data of the final encoder part; Step 4: Based on the encoding results, multi-head attention decoding is performed to obtain the rotation and translation data of the six-degree-of-freedom posture of the target; Step 5: The above six-degree-of-freedom posture rotation and translation data are stored in the FPGA board and then sent to the host computer to complete the target detection and guide the robot to grasp the target with six degrees of freedom.

2. The method of deploying a FPGA system for a robot six-degree-of-freedom visual target detection DETR model according to claim 1, characterized in that: In step 1, the PC stores the image data in the DDR4 slave module of the FPGA board; then reads the image data in the DDR4 slave module in a sliding window manner, and sends the read pixel blocks to the BRAM slave module of the FPGA board.

3. The method of deploying a FPGA system for a robot six-degree-of-freedom visual target detection DETR model according to claim 1, characterized in that: In step 4, the multi-head attention encoding part includes a Decoder sub-step, which contains 6 identical Decoders. The data obtained by the Encoder is sent to the Decoder sub-step in the multi-head attention encoding part. The specific steps are as follows: Step 4-1, sending the target query data to the Decoder decoding part; Step 4-2: Multiply the data in step 4-1 and add it to the target query data to obtain three feature data: Q, K, and V. The obtained Q, K, and V feature data are sent to the Multi-Head Self-Attention mechanism to reshape the data and then resize it. Step 4-3, adding the feature data of step 4-2 to the target query data of step 4-1, and performing a Norm normalization operation to obtain output feature data; Step 4-4, add the output data of step 4-3 to the target query data in step 4-1 to obtain Q; add the output data of step 4-3 to the final output of the encoder in step 3-6 and the position code to obtain K; the final output of the encoder in step 3-6 is used as V; send the Q, K, and V obtained this time to Multi-Head Attention, reshape the data Q, K, and V, and then resize the data; Step 4-5, adding the feature data outputted in step 4-3 to the target query data in step 4-4, and performing a Norm normalization operation to obtain output feature data; Step 4-6: The output data of step 4-5 is sent to FFN, and after passing through the Linear layer, the output feature data is obtained; Step 4-7, adding the feature data outputted in step 4-5 and the feature data outputted in step 4-6, and performing the same Norm normalization operation to obtain the output feature data; In step 4-8, the output feature data of step 4-7 obtained by the current decoder is input into the next identical decoder, and the above process is repeated 5 times. The outputs obtained by 6 decoders are stacked to obtain the final output feature data; the final output feature data is sent to FFN to predict the rotation and translation data of the six-degree-of-freedom posture of the target.

Images