Drowning person recognition method based on FPGA and YOLO

Abstract

This invention discloses a drowning person identification and detection method based on FPGA and YOLO, relating to the fields of image recognition and machine vision technology. The purpose of this invention is to improve the YOLOv5 algorithm to enhance the network's ability to represent target features, while using FPGA as hardware for high-speed computation processing to achieve real-time target detection in different scenarios. This reduces detection latency, improves production and recognition efficiency, while balancing detection accuracy and hardware resource utilization, and lowering system costs and maintenance expenses.

Description

Technical Field

[0001] This invention pertains to drowning victim identification technology, specifically a drowning victim identification method based on FPGA and YOLO. Background Technology

[0002] With economic and social development and improved living standards, water-based recreational activities are becoming increasingly popular in summer, with a significant increase in the frequency of activities in open water areas such as swimming pools, rivers, and oceans. However, due to the complex and changeable aquatic environment and imperfect supervision and early warning mechanisms, drowning accidents occur frequently. These accidents are characterized by their suddenness and high degree of concealment; drowning victims are often unable to effectively call for help and are therefore difficult to detect in time, missing the best rescue opportunity and ultimately leading to tragedy.

[0003] Traditional drowning prevention mainly relies on manual patrols and safety education, but it has obvious limitations in practical applications: human observation is easily affected by factors such as fatigue, light, and obstruction of vision; traditional monitoring relies on wearable devices, which have disadvantages such as high cost and poor universality. Summary of the Invention

[0004] This invention proposes a drowning person identification method based on FPGA and YOLO.

[0005] The technical solution to achieve the purpose of this invention is: a method for identifying drowning victims based on FPGA and YOLO, the specific steps of which are as follows:

[0006] Step 1: Obtain images of drowning victims and swimmers to create an image set, label them as drowning and normal swimming categories, and construct a dataset;

[0007] Step 2: Construct a recognition network. The recognition network adopts the Yolov5n network, and the Switch activation function in the Yolov5n network is adjusted to the Hard6-Swish function, and an attention mechanism is added.

[0008] Step 3: Use the dataset to train the recognition network to obtain model weights, and convert the model weights into fixed-point numbers;

[0009] Step 4: Construct a hardware-accelerated network, which is used to design the convolution, pooling, and upsampling operations in the recognition network as corresponding data processing modules.

[0010] Step 5: Based on the recognition network, complete the design of hierarchical inference, store the weight data of the corresponding layer in the memory of the FPGA device, and have the ARM hard core system schedule the FPGA to execute hierarchical inference through the AXI bus;

[0011] Step 6: The FPGA reads the feature map and weights to be detected, completes layer-by-layer inference, and obtains the detection results of drowning victims.

[0012] Compared with existing technologies, the significant advantages of this invention are as follows: This invention deploys a neural network in the form of an embedded device. If abnormal behavior by swimmers is observed in the deployed water area, such as struggling or choking on water, an alarm can be triggered immediately. Utilizing the high-speed parallel processing capabilities of FPGAs effectively reduces detection latency. This invention can promptly detect and handle crises, improving personnel safety. The low power consumption of FPGA devices allows for stable operation over extended periods, further enhancing the system's economy and applicability. Compared to traditional PC- and server-based detection systems, this hardware implementation not only improves system performance and stability but also significantly reduces hardware costs and maintenance expenses.

[0013] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0014] Figure 1 This is a flowchart of the drowning person identification method based on FPGA and YOLO of the present invention.

[0015] Figure 2 A schematic diagram of the improved activation function Hard6-Swish.

[0016] Figure 3 This is a diagram of the Yolov5 network structure in this invention.

[0017] Figure 4 This is a schematic diagram of the attention mechanism module in this invention.

[0018] Figure 5 This is a structural diagram of the convolution module in the FPGA of this invention;

[0019] Figure 6 This is a diagram of the ping-pong cache structure in this invention.

[0020] Figure 7 This is a diagram illustrating the production line operation in this invention. Detailed Implementation

[0021] A drowning person identification method based on FPGA and YOLO is disclosed. The detection algorithm is implemented through FPGA hardware, and various technical means are employed to reduce the loss of detection accuracy, improve the resource utilization efficiency of the equipment, and reduce downtime caused by detection errors. Compared with traditional solutions, this invention can reduce hardware costs and maintenance expenses. The specific steps are as follows:

[0022] Step 1: Obtain images of drowning victims and swimmers to create an image set, label the image set, and construct a relevant dataset;

[0023] Based on raw images of people in different situations in the water collected from video and image websites, LabelImg software was used to manually label the situations of the people in the water, distinguishing them as drowning victims or swimmers, resulting in labeled images and corresponding XML tag files.

[0024] Step 2: Construct a recognition network. The recognition network adopts the Yolov5n network, and the Switch activation function in the Yolov5n network is adjusted to the Hard6-Swish function, and an attention mechanism is added.

[0025] Before training, the Switch activation function in the training network is adjusted to the Hard6-Swish function:

[0026] The Hard-Swish activation function takes the form of: Figure 2 As shown. This function, through simulation, approximates the Swish function as closely as possible. At the same time, it eliminates the exponential calculation method, saving computational power. The Hard6-Swish function is shown below.

[0027]

[0028]

[0029] Then, in the Yolov5 pre-trained network, as Figure 3 As shown, the YOLOv5 model has a structure consisting of four parts: input, backbone, neck, and output. The input receives a three-channel image with a size of 640×640. The backbone is the main feature extraction part of the model and adopts the Dacknet-53 structure, which includes Conv convolutional modules, C3 modules, Bottleneck residual modules, and SPPF (Spatial Pyramid Pooling Fast Structure).

[0030] The Conv convolutional module is the most basic building block of the entire network, responsible for feature extraction from the feature map. The C3 module is the core feature extraction and fusion module of YOLOv5, a lightweight module designed based on the CSP (Cross-Stage Local Network) concept, entirely built from Conv modules. The Bottleneck residual module consists of residual blocks composed of two Conv modules, responsible for addressing the problem of decreasing accuracy as the network deepens. The Spatial Pyramid Pooling Fast Structure (SPPF) can transform images of arbitrary size into feature maps of fixed size.

[0031] The neck section, located between the backbone and the last output layer, repeatedly fuses and extracts shallow and deep information from the three feature layers to fully utilize contextual information. The upsampling module is responsible for enlarging small, deep feature maps to facilitate concatenation with larger, shallow feature maps. The concatenation module, on the other hand, concatenates two feature maps of the same size along the channel dimension, completing the feature concatenation.

[0032] The output includes three detection heads of different sizes: 20×20, 40×40, and 80×80 feature maps. Each head is responsible for detecting small, medium, and large targets, respectively. The predicted bounding boxes are directly positioned in each feature map and their probabilities for each category are obtained. Low-probability predicted bounding boxes are filtered out using non-maximum suppression, ultimately yielding the position and category of the predicted bounding boxes in the image.

[0033] To improve model accuracy, an attention mechanism module, SENET, is added after the backbone network to enhance the model's recognition accuracy. Figure 4 As shown. Its core principle is to compress the spatial information of each channel into a single value (representing the global feature of the channel) by performing global average pooling on the feature map through compression; then, by activation, the two-layer MLP learns the channel weights, assigning high weights to important channels and low weights to redundant channels; finally, after weighting, the weights are multiplied with the original feature map channel by channel to strengthen the features of key channels.

[0034] Step 3: Train the dataset in the Yolov5n network to obtain the model weights. For ease of deployment, since the model weights are floating-point numbers, which are difficult to compute on FPGAs, we perform 8-bit fixed-point quantization to convert them into fixed-point numbers.

[0035] During training of the Yolov5 network, the images in the dataset were divided into training, validation, and test sets in a ratio of 7:2:1. After preprocessing the images in the dataset to a uniform width and height of 640×640, the PyTorch environment was launched and model training began. Relevant parameters were adjusted based on the images in the training set, with a batch size of 32 and 100 training iterations. After training, the model weights were obtained.

[0036] In YOLOv5 network operations, the computational weights related to the convolutional computation model are floating-point numbers. For FPGA hardware computation, the required arithmetic resources are too large, so they need to be quantized and converted into easily computed 8-bit fixed-point numbers.

[0037] In the PyTorch environment, the quantization formula is:

[0038]

[0039] In the formula, q represents the quantized fixed-point number, r represents the floating-point number to be quantized, s represents the scaling factor, and round represents rounding. z represents the number of zeros after quantization, ensuring hardware calculation alignment.

[0040] Step 4: Build a hardware-accelerated network. Design the relevant operations in the YOLOv5 neural network, such as convolution, pooling, and upsampling, into corresponding data processing modules. They are designed in the same direction. They control the output and input of data through ping-pong modules and control the speed and accuracy of calculation through pipelines. Therefore, only the calculation modules need to be adjusted to adapt to different calculations.

[0041] Therefore, in the data processing module, taking the convolution module mentioned above as an example, such as... Figure 5 As shown.

[0042] For data input and output, a ping-pong buffer is used, employing two buffers of the same size as the input and output buffers to reduce the overall computation latency. The pattern is as follows: Figure 6 As shown in the diagram. The specific operation of the ping-pong function is as follows: In the first clock cycle, data is written to buffer 1; subsequently, in the second buffer cycle, data is written to buffer 2, while data is read from output buffer 1; in the third clock cycle, data is written to buffer 1, while data is read from output buffer 2. In the ping-pong operation, the two buffers switch back and forth according to the clock, allowing data input, data processing, and output to be performed in parallel, effectively realizing continuous data processing and transmission.

[0043] During computation, for the input feature map, the input channels N are split into... Each block is processed in one go. One channel. The output channel M is split into... Each block, calculated each time Each channel. Feature map Split into The spatial block is K, where K is the size of the convolution kernel, so each time an input feature map block is read... The weights are then adjusted as follows: In convolution computation, local convolution is performed on each output block.

[0044] Finally, the output block size is oc × r × c. After calculation, it is written to the global output feature map. Then, the next block of local data is loaded for calculation, and so on, until the entire input feature map is calculated. Then, the global output feature map is output. Then, the global output feature map is used as the input feature map for the next layer's calculation, completing the iterative calculation until the entire neural network is calculated. This is the idea behind the convolution module's calculation. Other modules are similar, requiring only slight modifications to the calculation module.

[0045] During data computation, pipelined operations are used to improve system throughput by overlapping the execution of operations from different loop iterations. For example... Figure 7 As shown, the pipelined design breaks down the complex computational structure into T independent computational stages, with the output of each stage temporarily stored in a register before being passed to the next stage. Within each clock cycle, the current pipeline stage computes new data while simultaneously passing intermediate results from the previous stage to the next. After T+1 clock cycles, the final output of the logic circuit can be continuously obtained. Computational tasks at different pipeline stages can be executed in parallel, avoiding resource idleness. Compared to repeatedly building independent computational units, pipelined design only requires a small number of registers, reducing the overall latency to 1 / T (ideally) of the original circuit without significantly increasing hardware costs. Through continuous pipeline cycles, the system can complete more computations per unit time, significantly improving computational efficiency.

[0046] Step 5: Based on the Yolov5 network structure, complete the design of layered retrieval and store the corresponding layer weight data in the DDR3 of the FPGA device;

[0047] The specific process of performing a hierarchical operation on the weights of the trained model is as follows: Figure 3 As shown, the entire network has a total of 60 convolutional modules, including 42 layers of 1x1 convolutions and 18 layers of 3x3 convolutions, accounting for the majority of the computational load in the YOLO V5 algorithm. These modules are numbered from conv1 to conv60, thus creating a hierarchical structure. The Zynq7000 device consists of a PS (ARM hard-core system) for software scheduling and a PL (FPGA) for hardware computation. The PS and PL are connected via an AXI bus. The network structure is created on the PS based on the model described above, and the data is saved to a specified path via the ARM's DDR, thereby enabling the PL to perform accelerated computation.

[0048] Step 6: The FPGA reads the feature map data and weights to be detected and passes the feature map to the FPGA. At this time, the FPGA will allocate the weights to the corresponding layered modules according to the address assigned by the algorithm. Then, the feature map is processed layer by layer. In the corresponding module, the corresponding weight data is invoked for data processing. The completed intermediate feature map result is returned to DDR. The host computer manages these data and controls the FPGA to execute the next calculation until all calculations are completed.

[0049] The quantization formula in step 1 can be used to derive the inverse quantization formula.

[0050]

[0051] In the formula, r represents the floating-point number to be quantized, s represents the scaling factor, q represents the quantized result, and z represents the quantized zero point.

[0052] For convolution calculations, the desired output image is obtained by multiplying the image selected by the relevant convolution kernel with the corresponding model weights, as shown in the following formula:

[0053]

[0054] Where r1 is the input image, r2 is the model weight, and r3 is the output image.

[0055] Therefore, by substituting the above dequantization formula into the convolution calculation, where the weights are symmetrically quantized, z2 is 0, and z represents the number of zeros after quantization, ensuring hardware calculation alignment.

[0056]

[0057] In the formula, q3 is the final output fixed-point number after quantization, q1 is the fixed-point input data of the feature map after quantization, q2 is the data weight after quantization, s1, s2, and s3 are the quantization scaling factors corresponding to q1, q2, and q3, respectively, and z1 and z3 are the quantization zero points corresponding to q1 and q3, which are used to align with the true value 0 and eliminate quantization offset error.

[0058] Finally, the quantified results are obtained.

[0059]

[0060] This operation quantizes the input image and weight data into 8 bits for computation on the FPGA.

[0061] The PL (Programmer) reads the model weights and feature maps to be detected from the PS (Programmer) and stores them in input buffers 1 and 2 respectively using ping-pong buffers. Simultaneously, the weight data is stored in the weight module. Then, the feature map data and weight data undergo convolution to accelerate the computation and obtain the output result, which is stored in output buffers 1 and 2 via the data output module. The completed intermediate feature map results are returned to DDR (Data Memory Memory). The PS continues to manage these data and retrieve modules for the next calculation. This process is repeated until the entire network structure is complete. Finally, the detection results are output and returned to the host computer, where they are saved and displayed.

[0062] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for identifying drowning victims based on FPGA and YOLO, characterized in that, The specific steps are as follows: Step 1: Obtain images of drowning victims and swimmers to create an image set, label them as drowning and normal swimming categories, and construct a dataset; Step 2: Construct a recognition network. The recognition network adopts the Yolov5n network, and the Switch activation function in the Yolov5n network is adjusted to the Hard6-Swish function, and an attention mechanism is added. Step 3: Use the dataset to train the recognition network to obtain model weights, and convert the model weights into fixed-point numbers; Step 4: Construct a hardware-accelerated network, which is used to design the convolution, pooling, and upsampling operations in the recognition network as corresponding data processing modules. Step 5: Based on the recognition network, complete the design of hierarchical inference, store the weight data of the corresponding layer in the memory of the FPGA device, and have the ARM hard core system schedule the FPGA to execute hierarchical inference through the AXI bus; Step 6: The FPGA reads the feature map and weights to be detected, completes layer-by-layer inference, and obtains the detection results of drowning victims.

2. The FPGA and YOLO-based drowning person recognition method according to claim 1, characterized in that, The Hard6-Swish function is specifically... 。 3.The method of claim 1, wherein, The Yolov5n network includes an input terminal, a backbone, a neck, and an output terminal. The input terminal receives three-channel images, and the backbone is the main feature extraction part, which adopts the Dacknet-53 structure.

4. The FPGA and YOLO-based drowning person recognition method according to claim 1, characterized in that, The model weights are converted to The formula is: In the formula, q represents the quantized fixed-point number, r represents the floating-point number to be quantized, s represents the scaling factor, round represents rounding, and z represents the number of zeros after quantization, ensuring hardware calculation alignment.

5. The FPGA and YOLO-based drowning person recognition method according to claim 4, characterized in that, The data processing module uses a ping-pong buffer for data input and output. In the first clock cycle, data is written to buffer 1; then, in the second buffer cycle, data is written to buffer 2, and data in output buffer 1 is read at the same time; in the third clock cycle, data is written to buffer 1, and data in output buffer 2 is read at the same time. In the ping-pong operation, the two buffers switch with each other according to the clock, so that data input, data processing and output can be performed in parallel.

6. The drowning person identification method based on FPGA and YOLO according to claim 1, characterized in that, The data processing module splits the input channel N into blocks, processing channels at a time; Splitting the output channel M into blocks, computing channels at a time; splitting a feature map of size into spatial blocks, K being the size of the convolution kernel, reading the input feature map block at a time as , and adjusting the weights to: ; in the convolution computation, performing a local convolution for each output block: In the formula, out[oc][r][c] represents the value of the oc-th output channel of the output feature map at position (r,c). W[oc][ic][i][j] represents the convolution kernel weights, where oc is the output channel, ic is the input channel, i and j are the coordinates within the convolution kernel, and in[ic][r+i][c+j] represents the pixel value at the window position (r+i, c+j) of the ic-th input channel of the input feature map. The core computation of convolution is achieved by performing multiplication and summation on all elements within the convolution kernel window.

Images