An integrated sensing, computing, and control system for ocean target recognition, tracking, and ranging

Abstract

The present invention discloses a multi-functional sensing, computing and control integrated system based on FPGA, belonging to the field of FPGA parallel computing. The system uses cameras and lidar as sensing means, and realizes the calculations related to the recognition, tracking and ranging of marine ships through three models deployed on the FPGA. The control object is the metasurface; the parameters of the marine target detection model are compressed from Float32 to Int8 to adapt to the bus bit width of the FPGA development board. The marine target tracking model is compiled through a programming language. Finally, based on DPU and HLS, the construction, fusion and connection of the deep learning acceleration operator for the point cloud target ranging model are realized, and the engineering conversion of the model is achieved by adjusting the time delay to maximize resource utilization; according to the IO output of the FPGA, the metasurface state can be switched between absorbing or transmitting the electromagnetic signals of the marine ship communication channel. The present invention realizes the efficient deployment of software algorithms to the FPGA hardware platform, and has the advantages of strong migratability, low power consumption, less resource occupation and high precision.

Description

Technical Field

[0001] The present invention belongs to the field of FPGA (Field-Programmable Gate Array) parallel computing, and more specifically, relates to a sensing, computing, and control integrated system for realizing the recognition, tracking, and ranging of marine targets. Background Art

[0002] Detecting, tracking, and positioning marine targets can provide support for work such as marine resource development and marine ecological protection.

[0003] In recent years, the development of deep learning has promoted the development of target detection, tracking, and positioning technologies. Most convolutional neural network models achieve powerful computing capabilities based on cloud server computing. However, when actually applied in complex and changeable marine scenarios, it is difficult for ships to establish a stable connection with cloud servers, and high-complexity models are difficult to deploy in existing ship systems. Therefore, the development of a detection, tracking, and positioning system for marine targets has become an inevitable requirement. However, the convolutional neural network inference implemented by traditional platforms (such as CPUs and GPUs) has high energy consumption and large volume, and the systems deployed at the terminal often have problems such as too high power consumption of computing devices, insufficient computing power, and poor real-time performance. Summary of the Invention

[0004] In view of the above defects or improvement requirements of the prior art, the present invention provides a sensing, computing, and control integrated system for realizing the recognition, tracking, and ranging of marine targets, which is used to solve the problems of single function, insufficient computing power, and poor real-time performance of existing systems, and has the advantages of lightweight, low power consumption, intelligence, fast response, etc., and can be better applied to the auxiliary decision-making on ships.

[0005] To achieve the above object, according to the first aspect of the present invention, there is provided a sensing, computing, and control integrated system for realizing the recognition, tracking, and ranging of marine targets, including:

[0006] A metasurface for transmitting electromagnetic wave signals emitted by known ships or reflecting electromagnetic wave signals emitted by unknown marine ships;

[0007] A camera for collecting videos to be detected;

[0008] A lidar for collecting point cloud data to be recognized;

[0009] A marine target detection model, a marine target tracking model, and a point cloud target ranging model deployed on an FPGA, which are respectively used to realize the detection, tracking, and ranging of marine ships;

[0010] Among them, the model parameters of the marine target detection model are fixed-point quantized to the int8 type and stored in the SD card of the FPGA; the model parameters of the marine target tracking model are compiled by a programming language and stored in the SD card of the FPGA; the point cloud target ranging model includes a plurality of sequentially connected sparse convolution units, and the sparse convolution unit includes a sequentially connected sparse convolution module, a max pooling module, and an activation function module; the sparse convolution module, the max pooling module, and the activation function module are compiled into corresponding sparse convolution acceleration IP cores, max pooling acceleration IP cores, and activation function acceleration IP cores, and are connected in sequence according to the order of the zynq kernel, the sparse convolution acceleration IP, the max pooling acceleration IP, and the activation function acceleration IP and then connected back to the zynq kernel to form the hardware acceleration logic of the sparse convolution unit, which is stored in the Flash memory of the FPGA;

[0011] Each frame of image in the video to be detected is transmitted to the on-chip storage of the FPGA. After image preprocessing is performed by the PS side, the PS side calls the model parameters of the marine target detection model, and the calculation acceleration of marine ship target detection is performed on the PL side to obtain the detection result; the PS side calls the compiled marine target tracking model and performs marine target tracking according to the detection result to obtain the tracking result; when the tracking result is a known marine ship, the IO interface of the FPGA is controlled to output a control signal to the metasurface to make it transmit the electromagnetic wave signal emitted by the known marine ship, otherwise a control signal is output to the metasurface to reflect the electromagnetic wave signal emitted by the unknown marine ship;

[0012] The point cloud data to be recognized is transmitted to the on-chip storage of the FPGA. After point cloud data preprocessing is performed by the PS side, the PS side repeatedly calls the hardware acceleration logic of the sparse convolution unit to implement point cloud target ranging on the PL side.

[0013] According to the second aspect of the present invention, a design method of a multifunctional sensing, computing, and control integrated system based on FPGA as described in the first aspect is provided, including:

[0014] S1, fix-point quantize the parameters of the marine target detection model to the int8 type and load them into the SD card stored in the FPGA;

[0015] S2, compile the marine target tracking model using a programming language and store it in the SD card of the FPGA

[0016] S3. In HLS, use shift registers to implement the sparse convolution acceleration operator, max pooling acceleration operator, and activation function acceleration operator corresponding to the sparse convolution module, max pooling module, and activation function module of the point cloud target ranging model respectively, and convert them into corresponding sparse convolution acceleration IP cores, max pooling acceleration IP cores, and activation function acceleration IP cores. Connect them in sequence according to the order of the zynq kernel, sparse convolution acceleration IP, max pooling acceleration IP, and activation function acceleration IP through the AXI-Stream interface in Vivado and then connect back to the zynq kernel to form the hardware acceleration logic of the sparse convolution unit, and download it to the Flash memory of the FPGA through the data line;

[0017] S4. Set two encoding sequences in advance according to the electromagnetic characteristics of the metasurface, corresponding to the reflection and transmission functions of the electromagnetic wave signal respectively, and establish a one-to-one correspondence with the tracking result. Both of the two encoding sequences are transmitted through the IO ports of the FPGA.

[0018] Generally speaking, compared with the prior art through the above technical solutions conceived by the present invention, the following beneficial effects can be obtained:

[0019] Based on the lower power consumption and stronger parallel computing ability of the FPGA, the present invention proposes a multi-functional sensing, computing, and control integrated system with low power consumption, intelligence, and fast response based on the FPGA. Using cameras and lidars as sensing means, three models deployed on the FPGA are used to implement calculations related to the recognition, tracking, and ranging of marine ships, and the control object is the metasurface; compress the parameters of the marine target detection model from Float32 to Int8 to adapt to the bus bit width of the FPGA development board, compile the marine target tracking model through a programming language, and finally realize the construction, fusion, and connection of the deep learning acceleration operator of the point cloud target ranging model based on DPU and HLS, adjust the time delay to maximize resource utilization to achieve the engineering conversion of the model; according to the IO output of the FPGA, the metasurface state can be switched between absorbing or transmitting the electromagnetic signal of the marine ship communication channel. The present invention realizes the efficient deployment of software algorithms to the FPGA hardware platform, and has the advantages of strong portability, low power consumption, less resource occupation, and high precision, specifically reflected in:

[0020] 1. Compress the data type from float32 to int8. While the accuracy loss is only 10%, the operating speed of the entire system is increased by 4 times, significantly improving the system operation efficiency;

[0021] 2. Use HLS and DPU to complete the design of the hardware acceleration operator of the FPGA, and optimize the operator structure for the algorithm to implement multiple operators including sparse convolution. By building hardware platforms with different requirements, the requirements for time delay, accuracy, resource occupation, and power consumption are met;

[0022] 3. For the ranging task, sparse convolution can reduce the computational workload by 90% and accelerate the operation speed.

[0023] 4. Using the metasurface as the controlled end of the system, the reconstruction of the channel environment can be realized by controlling the processing methods of different electromagnetic signals, so as to intelligently filter the communication information of known and unknown ships.

[0024] In summary, the integrated sensing, computing and control system provided by the present invention has strong portability (a complete set of algorithm acceleration kits are developed based on DPU and HLS, with low demand for hardware resources), low power consumption (a standard 18650 lithium battery can supply the system for 2.5 hours), fast system response speed (the processing time for each picture is 0.04 s), high accuracy (the recognition accuracy reaches 96%) and other advantages compared with the traditional system.

[0025] As a preferred solution, the present invention makes improvements on Yolov3-tiny, incorporates optimization strategies such as K-means++, Focal IOU, and Merge NMS, and uses it as an ocean target detection model; makes improvements on the Bytetrack model, incorporates the SWIF motion compensation algorithm, and uses it as an ocean target tracking model; can improve the detection and positioning accuracy of targets of different sizes, can accurately determine the contours of different ships, and reduce feature loss; and the above models are all miniaturized algorithms, which are convenient for the subsequent construction of low-power and portable frameworks. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 FIG. is a schematic diagram of the physical construction of the multi-functional integrated sensing, computing and control system based on FPGA provided by the embodiment of the present invention.

[0027] Figure 2 FIG. is a schematic diagram of the software and hardware architecture of the multi-functional integrated sensing, computing and control system based on FPGA provided by the embodiment of the present invention.

[0028] Figure 3 FIG. is a schematic diagram of processing an image into a video sequence provided by the embodiment of the present invention.

[0029] Figure 4 FIG. is a calculation flow chart of the improved Bytetrack model provided by the embodiment of the present invention.

[0030] Figure 5 FIG. is a flow chart of sparse convolution operation provided by the embodiment of the present invention.

[0031] Figure 6 FIG. is a schematic diagram of the design connection based on DPU provided by the embodiment of the present invention.

[0032] Figure 7 FIG. is a schematic diagram of the metasurface provided by the embodiment of the present invention.

[0033] Figure 8 It is a schematic diagram of the overall power consumption after the hardware deployment provided by the embodiments of the present invention.

[0034] Figure 9 It is a flowchart of the design method of a multi-functional sensing, computing, and control integrated system based on FPGA provided by the embodiments of the present invention. Detailed implementation manners

[0035] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0036] The embodiments of the present invention provide a multi-functional sensing, computing, and control integrated system based on FPGA, as Figures 1 - 2 shown, including:

[0037] A metasurface for transmitting electromagnetic wave signals emitted by known ships or reflecting electromagnetic wave signals emitted by unknown ocean ships;

[0038] A camera for collecting the video to be detected;

[0039] A lidar for collecting the point cloud data to be recognized;

[0040] An ocean target detection model, an ocean target tracking model, and a point cloud target ranging model deployed on the FPGA, which are respectively used to implement the detection, tracking, and ranging of ocean ships;

[0041] Among them, the model structure and model parameters of the ocean target detection model are quantized to int8 type by fixed-point quantization and stored in the SD card of the FPGA; the model structure and model parameters of the ocean target tracking model are compiled by a programming language and stored in the SD card of the FPGA; the point cloud target ranging model includes a plurality of sequentially connected sparse convolution units, and the sparse convolution unit includes a sequentially connected sparse convolution module, a max pooling module, and an activation function module; the sparse convolution module, the max pooling module, and the activation function module are compiled into corresponding sparse convolution acceleration IP cores, max pooling acceleration IP cores, and activation function acceleration IP cores, and are connected in sequence according to the order of the zynq kernel, sparse convolution acceleration IP, max pooling acceleration IP, and activation function acceleration IP and then connected back to the zynq kernel to form the hardware acceleration logic of the sparse convolution unit, which is stored in the Flash memory of the FPGA;

[0042] Each frame of the video to be detected is transmitted to the on-chip memory of the FPGA. After image preprocessing on the PS side, the PS side calls the model parameters of the marine target detection model, and performs computational acceleration for marine ship target detection on the PL side to obtain the detection result; the PS side calls the compiled marine target tracking model and performs marine target tracking based on the detection result to obtain the tracking result; when the tracking result is a known marine ship, the PS controls the IO interface of the FPGA to output a control signal to the metasurface to transmit the electromagnetic wave signal emitted by the known marine ship, otherwise it outputs a control signal to the metasurface to reflect the electromagnetic wave signal emitted by the unknown marine ship;

[0043] The point cloud data to be recognized is transmitted to the on-chip memory of the FPGA. After point cloud data preprocessing on the PS side, the PS side repeatedly calls the hardware acceleration logic of the sparse convolution unit to implement point cloud target ranging on the PL side.

[0044] Preferably, the metasurface is a window absorber, and its unit pattern is loaded with active diodes for realizing the absorption and transmission functions of electromagnetic wave signals.

[0045] The marine target detection model can be any existing target detection model.

[0046] To improve the detection progress, preferably, the marine target detection model is an improved Yolov3-tiny model, including:

[0047] The Yolov3-tiny model for obtaining the ship detection boxes in each frame of the image;

[0048] The Merge NMS module for merging similar ship detection boxes;

[0049] The loss function of the improved Yolov3-tiny model includes the Focal Loss function and the Wise IOU loss function.

[0050] To improve the training efficiency and model accuracy, preferably, in the training stage of the marine target detection model, the basic anchor boxes in the training dataset are obtained through the K-means++ clustering module.

[0051] The training processes of the above-mentioned improved Yolov3-tiny model and the point cloud target ranging model are as follows:

[0052] 1. Obtain the marine ship image dataset and the marine ship point cloud dataset. After data augmentation of the image dataset and merging of the point cloud dataset, all image data are synthesized into several short video sequences according to the time series.

[0053] For example, the marine ship image dataset uses the open-source dataset Seaship7000, which contains information about the type of ship, the target box, and the serial number information (i.e., the serial number of the same target in consecutive videos).

[0054] The point cloud dataset uses an open-source lidar dataset, which contains sparse point cloud maps (features) and dense point cloud maps (labels); the video segments used for tracking are synthesized based on the time information scanned and read from the Seaship7000 dataset.

[0055] The ship image data augmentation scheme, that is, the image preprocessing includes: scaling the image, converting the RGB channels, and adjusting the exposure.

[0056] The preprocessing of the point cloud dataset includes: merging the RGB three channels of the sparse point cloud map and the dense point cloud map into a single grayscale channel, and the merging relationship can be expressed using the following equation:

[0057] ;

[0058] where R value 、 G value 、 B value represent the specific values of channels R, G, and B respectively.

[0059] For the synthesis of video sequences, first use OCR recognition to read the location and time watermarks on the pictures. After preliminary division according to the shooting time, then divide them into several short video sequences according to the chronological order of time and the appearance order of ships from different perspectives.

[0060] 2. Using the ship image as a sample and its corresponding target box as a sample, input the data-augmented image into the improved Yolov3-tiny model for training to obtain a trained improved Yolov3-tiny model for marine ship target recognition; using the sparse point cloud map as a sample and its corresponding dense point cloud map as a label, input the point cloud dataset into the sparse convolution model to obtain a trained sparse convolution model for marine ship point cloud ranging.

[0061] Among them, the improved Yolov3-tiny recognition algorithm is optimized by using K-means++ target box clustering, Focal Loss and wise-IOU loss functions, and Merge NMS (Merge Non-Maximum Suppression) non-maximum suppression method.

[0062] Before feature recognition, the K-means++ clustering method was used to pre-determine the basic anchor boxes adapted to the target size of the dataset. During the iterative process of the recognized anchor boxes, in order to deeply distinguish the hull features and environmental features, a scheme combining the Focal Loss function and Wise IOU was used, and a regulation factor of the sample recognition difficulty was added before the loss function. The higher the recognition difficulty of the sample, the larger the regulation factor, artificially amplifying the proportion of the loss function of difficult samples. In addition, the Merge NMS non-maximum suppression method was also introduced. For the sample boxes with high overlap during the iterative process, a threshold function including IOU, aspect ratio, and center point position was set, and the sample boxes reaching the threshold were merged.

[0063] For the marine target detection task, the K-means++ object box clustering can obtain the recognition range approximate to the hull feature size. The loss function combining Focal Loss and wise-IOU can deeply distinguish the hull features and environmental features, ensuring the accurate extraction of the ship's contour in complex environments. The MergeNMS non-maximum suppression can improve the confidence of target recognition, comprehensively enhancing the adaptability and positioning ability of the improved Yolov3-tiny for targets of different sizes.

[0064] For the marine ship target ranging task, the adopted sparse convolution model is trained based on the grayscale channel map obtained from the above processing. To reduce the redundant operations of the deep network, a mask matrix containing only 0 and 1 is introduced. The mask matrix of the first layer of the network is directly obtained according to the values of the grayscale image, where the mask matrix corresponding to the pixel value of 0 has a value of 0, and vice versa. The subsequent mask matrices are all obtained by performing a max pooling operation with padding on the previous layer's mask matrix. The loss function used during the operation is the mean square error loss.

[0065] Preferably, the marine target tracking model is an improved Bytetrack model;

[0066] Among them, before matching the high-confidence ship detection box of the Nth frame image with the target detection box predicted according to the high-confidence ship detection box of the (N - 1)th frame image, the improved Bytetrack model further includes:

[0067] Multiplying the target detection box by the affine matrix for motion compensation;

[0068] Among them, the affine matrix is calculated using the SIFT algorithm.

[0069] Specifically, an improved Bytetrack tracking model obtained by using SIFT (Scale-Invariant Feature Transform) motion compensation and Kalman filter optimization is adopted. For the tracking task, SIFT motion compensation can enhance the system's adaptability to camera movement, and the Kalman filter can accurately predict the subsequent trajectory of the ship without losing the number of occluded targets.

[0070] For the marine ship target tracking task, the improved Yolov3-tiny is used as a high-performance target detector, and the improved Bytetrack model is adopted to achieve target tracking.

[0071] For the input image of the Nth frame, the feature points of the Nth frame image and the (N - 1)th frame image are calculated by SIFT, the affine matrix H is calculated according to the feature points, and the predicted trajectory passing through the Kalman filter is motion-compensated through the affine matrix (that is, multiplying it by the affine matrix). The compensated motion trajectory is matched with the target box obtained from the current frame once and then twice. After two matches, the high-confidence target boxes that are not matched are added as new trajectories, the low-confidence target boxes that are not matched are discarded, and the unmatched trajectories are set as inactive. If the inactive trajectories still do not have a matching box after a certain caching time, then the trajectories are discarded.

[0072] That is, for the input image of the Nth frame, the detector calculates all the detection boxes and divides them into two categories according to the confidence level. The high-confidence detection boxes retained in the (N - 1)th frame are input into the Kalman filter to predict the possible positions of the detection boxes in the next frame. In this process, the SIFT motion compensation algorithm is introduced. By extracting the local feature points in the above two frames and calculating their descriptors (128-dimensional vectors), and then calculating the Euclidean distance of the descriptors between the two frames of images for feature point matching. After successful matching, 500 corresponding points are used to calculate the affine transformation, and the transformation matrix is calculated through the RANSAC algorithm to eliminate false matches. After the transformation matrix is calculated, the current frame image is affine-transformed using this matrix to align it with the reference frame. Since the pixel positions after transformation may be non-integers, bilinear interpolation is used to estimate the pixel values at the new positions to ensure image continuity. By comparing the key points of the front and back frames, which are usually the contour points of the ship target, the characteristics of the front and back frames are matched, and the coincidence degree between the high-confidence boxes calculated by the Kalman filter and the high-confidence boxes of the actual Nth frame is found. The detection box with a higher coincidence degree is used as the tracking mark box.

[0073] Taking the improved Yolov3-tiny model as an example of the marine target detection model, the quantization and compression process of the marine target detection model will be described below. The quantization and compression of the marine target detection model are used to achieve the lightweight design of the network and facilitate fitting with hardware resources.

[0074] The above algorithm parameters are compressed into int8 type by using the TensorFlow framework and fixed-point quantization method, so as to conform to the bit width of the hardware transmission bus and can be deployed on an FPGA at the same time; the fixed-point quantization adopts the method of layer-by-layer quantization and asymmetric quantization. The layer-by-layer quantization ensures that each layer of the neural network uses the same scaling factor and zero point, and the asymmetric quantization allows the quantization zero point to be non-zero, which is suitable for the asymmetric data distribution situation of the present invention. The combination of the two can reduce the quantization error and improve the accuracy of the quantized model.

[0075] The lightweight design mainly focuses on the processing of weights and biases. The process is as follows: for each layer of the neural network in the model, the weights and biases are separated and fixed-point quantized respectively, including determining the 8-bit quantization bit width, calculating the scaling factor and zero point, quantizing the parameters and activation values, dequantizing and fine-tuning; among them, when determining the 8-bit quantization bit width, the quantization process of mapping the floating-point number x to the fixed-point number q can be expressed by the following equation:

[0076] ;

[0077] where s is the scaling factor, z is the relative zero point, round represents rounding; since this design adopts layer-by-layer quantization, the s , z values of each layer are the same, and the result of each round of quantization needs to be used as the initial value to be retrained and calibrated under the network framework defined by TensorFlow. The quantization values in the int8 format are finally determined through multiple rounds of fixed-point quantization for subsequent acceleration

[0078] The following describes the deployment process of the marine target detection model, marine target tracking model and point cloud target ranging model on the FPGA.

[0079] Based on the FPGA for hardware acceleration design, based on the DPU (Deep Learning Unit) for deploying and realizing marine point cloud recognition and tracking; based on the HLS (High-Level Synthesis) for designing and accelerating the module and packaging it as an IP core to realize point cloud ranging;

[0080] The PYNQ-Z2 development board, with its integrated design of FPGA and ARM processor, has the characteristics of low power consumption, small volume and high real-time performance. At the same time, it supports Python development, which reduces the development threshold and speeds up the algorithm implementation speed. Compared with other development boards, it has more advantages in performance, cost and flexibility, and is suitable for the high-speed and low-power requirements of the target perception and tracking system. Based on this, preferably, the system provided by the present invention is designed based on the PYNQ-Z2 development board.

[0081] The DPU-based acceleration solution and the acceleration module-based acceleration solution are used for identification and tracking, and ranging, respectively. The DPU-based acceleration solution relies on the DPU IP core and enables it to run normally on the PYNQ-Z2 by modifying the underlying parameter configuration. By loading the xxx.elf, xxx.weights, and xxx.biases binary network configuration files on it, efficient reasoning is achieved for identification and tracking tasks.

[0082] The design of the acceleration module is based on HLS software implementation. For the sparse convolution module, maximum pooling module, and activation function module of the sparse convolution unit, the corresponding sparse convolution acceleration operator, maximum pooling acceleration operator, and activation function acceleration operator are respectively implemented in HLS using shift registers according to the principle of window function. After the simulation test is correct, the built-in tools of HLS are used to convert them into independent, highly optimized custom IP cores that can be used in Vivado, namely sparse convolution acceleration IP core, maximum pooling acceleration IP core, and activation function acceleration IP core. In the block design of Vivado, the AXI-Stream interface is used to connect the Zynq core, sparse convolution acceleration IP, maximum pooling acceleration IP, and activation function acceleration IP in the order of connection and then back to the Zynq core, forming the hardware acceleration logic of sparse single-layer convolution. After downloading it to the board through the data cable, the underlying logic is repeatedly called on the software side to form a complete sparse convolution acceleration solution.

[0083] Furthermore, the image data and network parameters read by the camera are stored in the on-board DDR (Double Data Rate Synchronous Dynamic Random Access Memory). When performing calculations, the image data and network parameters are entered into the module for calculation together.

[0084] The hardware acceleration design is based on the DPU and the designed acceleration module, both of which run on the PL side of the board. The DPU-based design directly uses the improved Yolov3-tiny and Bytetrack network frameworks, while the HLS-based module acceleration implements data transmission through top-down interface calls. The weights and biases used by both are stored in DDR, facilitating fast data interaction.

[0085] The FPGA-based accelerated computing process is as follows:

[0086] The structure of the marine target recognition model is described as xxx.elf, which contains information about the number of convolutional layers and channels. The weight parameter file is xxx.weights, and the bias parameter file is xxx.biases. All files are stored in the on-board SD card, on which PetaLinux is loaded to facilitate quick call to implement the entire operation process. Parallel operations are all carried out on the PL (Programmable Logic) side, and the PS (Processing System) side is mainly responsible for mirror loading, SD card file reading, serial communication, and network loading;

[0087] When the system provided by the present invention works, the video data will be segmented into image data and sequentially transmitted into the DDR memory. First, image preprocessing is performed on the PS side, including channel conversion, image compression, and cropping. Then, according to requirements, the above-mentioned several types of network description files are read in to construct the corresponding network; subsequently, the parameters are transmitted into the internal BRAM (Block RAM) of the FPGA, that is, on-chip storage, and a series of operations including image-to-matrix conversion, matrix block division, window shifting operation, matrix combination, bias accumulation, and activation are completed on the PL side to complete the detection of marine ship targets. In this process, the binary number of each parameter after quantization is 8, which matches the AXI bus width of 8 between the PL and the PS, and the data only needs to be transmitted once. Then, the PS side calls the compiled marine target tracking model and performs marine target tracking based on the target detection result to obtain the tracking result.

[0088] The system provided by the present invention uses the FPGA as the control core and controls the level output according to the hardware acceleration result of the FPGA, so as to realize the control of the active tunable metasurface.

[0089] The metasurface is a window absorber. This design can realize the functions of absorbing and transmitting electromagnetic signals in a certain frequency band by loading active diodes on the unit pattern of the metasurface. The switching of the functions is realized based on the level signal output by the FPGA. Level 1 represents turning on the active diode, and at this time it has the ability to absorb electromagnetic signals. Level 0 represents turning off the active diode, and at this time it has the ability to transmit electromagnetic signals.

[0090] The signals output by the FPGA are 0 and 1 signals, which are used as switching signals to control the opening and closing states of the PIN diodes on the metasurface. The content of the output signals is determined by the system according to the tracking results. If the recognized object is a known object, a digital signal sequence for controlling electromagnetic wave transmission is generated and transmitted to the FPGA for output; if it is an unknown object, a digital signal sequence for controlling electromagnetic wave absorption is generated and transmitted to the FPGA for output. According to this signal, the metasurface can regulate the reception and transmission of electromagnetic signals in the resonant frequency band. The 0 signal represents closing, and at this time the electromagnetic signal passes through the metasurface. The 1 signal represents opening, and at this time the electromagnetic signal is resonantly absorbed by the metasurface.

[0091] In view of the characteristics of deep learning network features and the structure of the FPGA platform, the present invention develops and designs a multi-functional integrated sensing, computing, and control system with low power consumption, portability, and fast operation. The optimal weights and bias parameters of various marine target detection models and point cloud target ranging models are obtained through pre-training using a GPU (Graphics Processing Unit). Lightweight deployment is carried out using methods such as quantization. Then, a deep learning accelerator is designed according to the distribution of hardware resources. Finally, the state of the metasurface is controlled through this accelerator, realizing the efficient integration of sensing, computing, and control of a transferable hardware acceleration algorithm.

[0092] An embodiment of the present invention provides a design method for a multi-functional integrated sensing, computing, and control system based on FPGA as described in any of the above embodiments, including:

[0093] S1, fix-point quantize the parameters of the marine target detection model to the int8 type and load them into the SD card stored in the FPGA;

[0094] S2, compile the marine target tracking model using a programming language and store it in the SD card of the FPGA;

[0095] S3, use shift registers in HLS to implement the sparse convolution acceleration operator, max pooling acceleration operator, and activation function acceleration operator corresponding to the sparse convolution module, max pooling module, and activation function module of the point cloud target ranging model respectively, and convert them into corresponding sparse convolution acceleration IP cores, max pooling acceleration IP cores, and activation function acceleration IP cores. Connect them in sequence according to the order of the zynq kernel, sparse convolution acceleration IP, max pooling acceleration IP, and activation function acceleration IP through the AXI-Stream interface in Vivado and then connect back to the zynq kernel to form the hardware acceleration logic of the sparse convolution unit, and download it to the Flash memory of the FPGA through the data line;

[0096] S4, preset two coding sequences in advance according to the electromagnetic characteristics of the metasurface, corresponding to the reflection and transmission functions of the electromagnetic wave signal respectively, and establish a one-to-one correspondence with the tracking result. Both of the two coding sequences are transmitted through the IO port of the FPGA.

[0097] The method provided by the present invention will be further described below, taking the improved Yolov3-tiny model for the marine target detection model and the improved Bytetrack model for the marine target tracking model as examples.

[0098] 1. Train the marine target detection model and the point cloud target ranging model, and construct the Bytetrack model.

[0099] (1)Obtain the marine ship image dataset and the point cloud dataset. After performing data augmentation and merging on the datasets, all the image data are synthesized into several short video sequences according to the time series.

[0100] In the example, the sizes of all ship images are compressed to 416×416 and divided into a training set, a validation set, and a test set according to the ratio of 7:2:1; the video sequences are arranged in the order of time from the upper left corner of the image and constructed by sorting after using ORC to scan the text, as Figure 3 shown.

[0101] (2)Input the images after data augmentation into the improved Yolov3-tiny model for training to obtain the trained improved Yolov3-tiny model for realizing marine ship target recognition; input the point cloud dataset into the sparse convolution model to obtain the trained sparse convolution model for realizing marine ship point cloud ranging;

[0102] In this embodiment, for the Yolov3-tiny model, first, the data obtained in the above (1) are subjected to K-means++ clustering, and anchor boxes for large targets and small targets can be obtained. Set the above anchor box parameters as the initial anchor box sizes of Yolov3-tiny, and their sizes are shown in Table 1:

[0103]

[0104] Before starting training, for the class loss, adopt the strategy of Focal Loss with a monotonic focusing mechanism, and introduce a modulation factor , where pt is the predicted probability of the model for the target class. This factor can reduce the weight of easy-to-classify samples and increase the attention to difficult samples, thus effectively dealing with the problem of sample imbalance. Its calculation formula is as follows:

[0105] ;

[0106] In Focal Loss, as pt increases (that is, the sample is easier to be correctly classified), the loss value will monotonically decrease. That is to say, the loss value is monotonic. When ptWhen it is closer to 1, the loss FL is closer to 0. This monotonically decreasing characteristic enables the Focal Loss to effectively reduce the loss contribution of easily classified samples, thereby allowing the model to focus more on those difficult samples; subsequently, Merge-NMS maximum suppression is introduced in front of the detection head of Yolov3-tiny, and similar target boxes are merged in advance according to the level of IOU interpolation, where the interpolation is defined as 0.01; after the above processing for training, the improved Yolov3-tiny model adopted by the present invention can be obtained;

[0107] The specific calculation flow chart of the Bytetrack model is as Figure 4 shown. The basis of the tracker is the above improved Yolov3-tiny algorithm. In this example, the Kalman filter and SIFT motion compensation are introduced in the process of processing the tracking box of the (n - 1)-th frame. The Kalman filter mainly adjusts the estimation parameter P and the measurement error R. P is set as a diagonal matrix , and the mean square error is used for the definition of the measurement error;

[0108] For the point cloud ranging algorithm, the sparse convolution process after introducing the mask layer is as Figure 5 shown. The training process can refer to the Sparsity Invariant CNNs sparse convolution model.

[0109] 2. Model deployment.

[0110] (1) Quantize and compress the trained improved Yolov3-tiny model to achieve lightweight design of the network, which is convenient for fitting hardware resources;

[0111] In this example, the prerequisite for quantization and compression is to convert the Pytorch framework model trained in the previous example into a float32-type TensorFlow framework model. After obtaining the xxx.pt model file based on the TensorFlow framework, the parameter iteration is carried out layer by layer through the following expression:

[0112] ;

[0113] where s is the scaling factor, z is the relative zero point, round represents rounding, s and z are calculated according to the following expression:

[0114] ;

[0115] After introducing the dataset and performing multiple rounds of parameter iteration, the computational complexity and the number of parameters of the weights and biases files in the quantized int8 data format are greatly reduced, while reducing the DSP's consumption of floating-point numbers.

[0116] (2)Perform hardware acceleration design based on FPGA. Deploy based on DPU (Deep Learning Unit) to achieve acceleration of ocean point cloud recognition and tracking; design an acceleration module based on HLS and package it as an IP core to achieve acceleration of point cloud ranging;

[0117] In this implementation, the input images read by the camera are stored in the DDR for preprocessing, including size scaling and channel conversion. After the picture data is ready, it is read into the network parameters for acceleration;

[0118] The acceleration of the sparse convolution model is implemented based on the window function. The sparse convolution model includes multiple sequentially connected sparse convolution units. The sparse convolution unit includes a sparse convolution operator and a max-pooling operator with padding. Each operator is used separately and the parallel calculations do not affect each other. There is an arithmetic relationship between the output results of the two. To avoid repeated memory calls, the sparse convolution operator and the max-pooling operator in the corresponding area are calculated simultaneously, and after the operation is completed, the subsequent operations are directly completed and then sent back to the memory.

[0119] When calling the DPU to accelerate convolution, it is necessary to use Vivado's block design for design. The connections of each IP are as Figure 6 shown. After downloading to the board, read the binary network structure xxx.elf file. By writing the control code on the PS side and sending a running instruction from the PC to the FPGA, the real-time output of the camera input picture can be achieved.

[0120] (3)Use the FPGA as the control core and control the level output according to the hardware acceleration result of the FPGA, so as to achieve the control of the active tunable metasurface;

[0121] Add IO control of the FPGA to the established hardware acceleration framework, which can control the IO to output corresponding digital signals according to different recognition and tracking results. Here, corresponding IO combinations are set for six classifications of ships; the metasurface adopted in this example is as Figure 7 shown. The metasurface has switchable wave absorption and wave transmission characteristics. When the IO output is 1, it represents switching to the wave absorption effect of the metasurface. When the IO output is 0, it represents switching to the wave transmission effect of the metasurface, and the wave absorption and wave transmission switching effect can be completed at a frequency point of 4.5 GHz.

[0122] The resource consumption and power consumption of the entire system are as Figure 8As shown, it can be found from this figure that the power consumption of the entire system is as low as 4.35W, meeting the platform's requirements for low power consumption. At the same time, through the process of the system from receiving an image message to feeding back the digital signal output, the test result only takes 0.04s, meeting the requirements for high speed. Through multiple rounds of accuracy tests on the input images, the calculated recognition accuracy is as high as 96%, meeting the requirements for high precision.

[0123] In this example, the integration of ocean target recognition, tracking, and ranging functions is used as the design benchmark. Through the path of algorithm design optimization, hardware quantization compression, and FPGA deployment, an integrated sensing, computing, and control system with intelligence, low power consumption, high precision, and fast response is realized. The implementation flowchart of the system is as Figure 9 shown.

[0124] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-functional sensing, computing and control integrated system based on FPGA, characterized in that, Including: A metasurface for transmitting electromagnetic wave signals emitted by known ships or reflecting electromagnetic wave signals emitted by unknown ocean ships; A camera for collecting the video to be detected; A lidar for collecting the point cloud data to be recognized; An ocean target detection model, an ocean target tracking model, and a point cloud target ranging model deployed on an FPGA, which are respectively used to implement the detection, tracking, and ranging of ocean ships; Among them, the model parameters of the ocean target detection model are fixed-point quantized to the int8 type and stored in the SD card of the FPGA; the model parameters of the ocean target tracking model are compiled by a programming language and stored in the SD card of the FPGA; the point cloud target ranging model includes a plurality of sequentially connected sparse convolution units, and the sparse convolution unit includes a sequentially connected sparse convolution module, a max pooling module, and an activation function module; the sparse convolution module, the max pooling module, and the activation function module are compiled into corresponding sparse convolution acceleration IP cores, max pooling acceleration IP cores, and activation function acceleration IP cores, and are connected in sequence according to the order of the zynq kernel, the sparse convolution acceleration IP, the max pooling acceleration IP, and the activation function acceleration IP and then connected back to the zynq kernel to form the hardware acceleration logic of the sparse convolution unit, which is stored in the Flash memory of the FPGA; Each frame of image in the video to be detected is transmitted to the on-chip storage of the FPGA, and after image preprocessing by the PS side, the PS side calls the model parameters of the ocean target detection model, and performs computational acceleration for ocean ship target detection on the PL side to obtain the detection result; The PS side calls the compiled ocean target tracking model and performs ocean target tracking according to the detection result to obtain the tracking result; when the tracking result is a known ocean ship, it controls the IO interface of the FPGA to output a control signal to the metasurface to make it transmit the electromagnetic wave signal emitted by the known ocean ship, otherwise it outputs a control signal to the metasurface to reflect the electromagnetic wave signal emitted by the unknown ocean ship; The point cloud data to be recognized is transmitted to the on-chip storage of the FPGA, and after point cloud data preprocessing by the PS side, the PS side repeatedly calls the hardware acceleration logic of the sparse convolution unit to implement point cloud target ranging on the PL side.

2. The system according to claim 1, wherein The metasurface is a window absorber, and an active diode is loaded on its unit pattern for realizing the absorption and transmission functions of electromagnetic wave signals.

3. The system according to claim 1, wherein The ocean target detection model is an improved Yolov3-tiny model, including: The Yolov3-tiny model for obtaining the ship detection frame in each frame of image; The Merge NMS module for merging similar ship detection frames; The loss function of the improved Yolov3-tiny model includes the Focal Loss function and the Wise IOU function.

4. The system according to claim 1, wherein In the training stage of the ocean target detection model, the basic anchor boxes in the training dataset are obtained through the K-means++ clustering module.

5. The system according to claim 1, wherein The ocean target tracking model is an improved Bytetrack model; Among them, before the improved Bytetrack model matches the high-confidence ship detection box of the Nth frame image with the target detection box predicted according to the high-confidence ship detection box of the (N-1)th frame image, it further includes: Multiplying the target detection box by the affine matrix for motion compensation; Among them, the affine matrix is calculated using the SIFT algorithm.

6. The system according to claim 1, wherein The image preprocessing includes: image scaling, RGB channel conversion, and exposure adjustment; The point cloud data preprocessing includes: merging the RGB three channels into one gray channel.

7. The system according to claim 1, wherein The system is designed based on the PYNQ-Z2 development board.

8. A design method for a multi-functional sensing, computing and control integrated system based on FPGA according to any one of claims 1-7, characterized in that, It includes: S1, fixing-point quantizing the parameters of the marine target detection model to the int8 type and loading them into the SD card stored in the FPGA; S2, compiling the marine target tracking model using a programming language and storing it in the SD card of the FPGA S3, using shift registers in HLS to implement the sparse convolution acceleration operator, max pooling acceleration operator, and activation function acceleration operator corresponding to the sparse convolution module, max pooling module, and activation function module of the point cloud target ranging model respectively, and converting them into corresponding sparse convolution acceleration IP cores, max pooling acceleration IP cores, and activation function acceleration IP cores. Connect the AXI-Stream interfaces in Vivado in the order of zynq kernel, sparse convolution acceleration IP, max pooling acceleration IP, and activation function acceleration IP and then connect back to the zynq kernel to form the hardware acceleration logic of the sparse convolution unit, and download it to the Flash memory of the FPGA through the data line; S4, presetting two coding sequences in advance according to the electromagnetic characteristics of the metasurface, corresponding to the reflection and transmission functions of the electromagnetic wave signal respectively, and establishing a one-to-one correspondence with the tracking result. Both of the two coding sequences are transmitted through the IO port of the FPGA.

Images