A railway crossing train recognition and warning system based on monocular vision
Through the multi-parallel processing architecture of FPGA and ARM, combined with Mobilenet_v1 SSD and Koggle Stone algorithm, the cost, speed and power consumption problems of the train identification and early warning system in the existing technology are solved, and fast and low-power train recognition is achieved to avoid There is a risk of a train hitting a person.
Patent Information
- Application Number
- CN202111098925.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-09-18
AI Technical Summary
The existing railway intersection train identification warning system based on monocular vision has problems in terms of cost, speed and power consumption. The CPU processing speed is slow, the GPU costs are high and the power consumption is large, and it is not suitable for large-scale deployment.
The multi-parallel processing architecture of FPGA and ARM ends is adopted, combined with the Mobilenet_v1 SSD algorithm and Koggle Stone addition optimization algorithm, to achieve fast object detection, and ensure data accuracy and streaming processing efficiency through an input memory structure similar to ping-pong operation.
A train identification and warning system with low power, small size and fast recognition speed is realized, which avoids the occurrence of train hitting people and reduces processing time and storage space requirements.
Smart Images

Figure CN113837054B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of computers, and in particular relates to a railway crossing train identification and warning system based on monocular vision. Background Art
[0002] The railway crossing train recognition and warning system based on monocular vision mainly relies on the target detection algorithm to realize its function. The target detection algorithm is the basis of many computer vision tasks. It can identify specific targets in the image and provide the coordinate position information of the identified specific targets in the image. In the railway crossing train recognition and warning system based on monocular vision, the target detection algorithm identifies the specific position of the train in the camera image and obtains the coordinates of the train in the image. Through a simple monocular vision ranging algorithm, the distance between the train and the camera can be calculated. Once the distance between the train and the camera is less than the set threshold, the system will alarm.
[0003] At present, in most areas of my country, the train identification and warning system still relies mainly on manpower to observe trains. Of course, a small number of areas have adopted CPU or GPU to deploy neural network algorithms to implement train identification and warning systems. However, both CPU and GPU face three major problems in actual deployment: cost, speed, and power consumption. For GPU, although the processing speed of GPU is very fast, it is not conducive to large-scale deployment due to the huge power consumption required by GPU and the high cost. For CPU, although the cost and power consumption of CPU are lower than GPU, the processing speed is too slow to accurately detect, track, and identify ordinary trains and high-speed trains in real time and efficiently. Summary of the invention
[0004] To solve the above problems, the present invention discloses a railway crossing train identification and warning system based on monocular vision. The railway crossing train identification and warning system of the present invention has the advantages of low power, small size and fast recognition speed, and it can effectively avoid the occurrence of train collisions; in the hardware circuit architecture, the input memory adopts a structure similar to ping-pong operation to ensure the accuracy of input data. A streaming processing architecture that transmits and processes at the same time is realized, saving processing time and storage space. Multi-parallel processing is performed, which is equivalent to 16-core computing, thereby saving waiting time. The system uses mobilenet_v1SSD algorithm for target detection, and uses koggle stone algorithm to optimize the addition calculation, reducing the addition calculation time.
[0005] To achieve the above object, the technical solution of the present invention is:
[0006] A railway crossing train identification and warning system based on monocular vision, comprising a camera, the camera is communicatively connected to a data processing system, and the data processing system is communicatively connected to an alarm device;
[0007] The camera is used to take pictures, the data processing system is used to process the pictures taken by the camera, and the alarm device is used to give an alarm when the data processing system determines that a train is passing by;
[0008] The data processing system includes a PL side and a PS side, where the PL is the logic part, which is the FPGA side, and the PS part is the ARM side;
[0009] Among them, the PL side includes the AccSystem unit. The PS side mainly includes the fpgadrv unit, ssd_detection unit, Paddlelite unit, mobilenet_v1 SSD unit, lib unit and opencv unit;
[0010] The AccSystem unit communicates with the fpgadrv unit, the fpgadrv unit communicates with the ssd_detection unit, the ssd_detection unit communicates with the lib unit, the opencv unit, and the Paddlelite unit
[0011] The AccSystem unit is used to accelerate the convolution part;
[0012] The fpgadrv unit is the driver for the FPGA accelerator on the Linux side;
[0013] Mobilenet_v1 SSD unit English storage SSD model and parameters;
[0014] The Lib unit is used to store the library files required to run the demo;
[0015] The Ssd_detection unit is used to store the demo source code;
[0016] The Opencv unit is used to store the compiled Opencv library;
[0017] The Paddlelite unit is used to mount the paddle frame.
[0018] As a further improvement, the ConvCore module includes a din_fifo unit, a ConvCompute unit and a dout_fifo unit; wherein the din_fifo unit is used to cache input data, the ConvCompute unit performs multiplication and addition operations on the input data of the convolution kernel, and the dout_fifo unit is used to cache the data after calculation.
[0019] A further improvement is that the din_fifo unit includes an Input fifo module, a Kernal fifo module and a Convheight counter; the Input fifo module caches the input image data, and the Kernal fifo module caches the weight data; when the data enters, the Input fifo unit performs ping-pong storage on the input image data; the Convheight counter counts the number of kernel rows in each batch, packs each 32 bits into a unit, and then copies the 32-bit unit to make it 512 bits, and finally splices the weight data and the image data, and inputs the spliced result into the convolution calculation unit.
[0020] A further improvement is that the ConvCompute unit performs multiplication and addition operations on the input image data and weight data; the ConvCompute unit includes an 8*8 multiplier unit and an adder array unit; the weight data and image data are multiplied by the 8X8 multiplier unit, and then the result enters the adder array unit for addition calculation; the ConvHeght counter counts the number of kernel rows in each batch to control the output.
[0021] As a further improvement, the ConvCompute unit adopts a multi-core parallel computing method, that is, 64 multipliers are used to complete the calculation of 16 points in parallel.
[0022] As a further improvement, when the adder array unit performs addition calculation, the koggle stone algorithm is used for optimization; and the railway crossing train identification and warning system based on monocular vision adopts mobilenet_v1 SSD algorithm for target detection.
[0023] As a further improvement, the data processing flow of the railway crossing train identification and warning system based on monocular vision is as follows:
[0024] The ARM side fuses and assembles the data according to the same batch of data from different channels, and then each row of data is 64 bits. A row of data contains data of 16 points. The data required for each point is divided and assembled according to 4 bytes. The ConvHeight unit calculates the amount of data required for a row of results and takes the quotient of 4 bytes upward. Then the result is output to the output memory according to 16 points per row. Only the calculation depth and result depth are configured in the configuration parameters. The ARM side needs to configure the kernel of each batch first, and then slice the data and transfer it to the acceleration core of the FPAG side through the AXI bus to the Avalon bus. The acceleration core receives the data while performing convolution calculations. After the transmission is completed, it is known whether the data has been processed by reading the status register, and then reads it back to the ARM side to reorganize the data.
[0025] Further improvements: the input memory on the ARM side is a 2-line storage structure with a total of 128 bytes of memory, while the Kernel memory is a 512-line memory and the output memory is a 4096-line memory, which can store 65536 result points; in the Avalon bus selector, the Avalon address line is extended to 18 bits in length, and the mapping address on the ARM side is extended to 4194304 bytes), while the input memory on the FPGA side only occupies 128 bytes, thereby optimizing the Avalon bus address mapping selection rules, and the input memory adopts a structure similar to ping-pong operation to ensure the accuracy of the input data, and realizes a streaming processing architecture that transmits and processes at the same time. In the driver, the data that needs to be convolved is channel-fused and the convolution kernel is normalized, that is, any convolution calculation is regarded as a 1*1* multi-channel convolution calculation, which improves the versatility of the convolution acceleration kernel. .
[0026] The present invention has innovated the convolution kernel calculation of the PL end, redesigned the storage and calculation system, and rewritten the configurable convolution kernel and storage system. The specific innovations are as follows:
[0027] 1. Putting the input and output cache and convolution calculation together realizes pipeline storage and calculation. The efficiency of the convolution kernel is greatly improved by calculating while transmitting, and the time wasted by frequent data interaction on the bus is reduced.
[0028] 2. The input cache is stored in only two steps, and the data of the input convolution kernel is cached in a ping-pong manner. At the same time, the capacity of the output cache is expanded to improve storage and transmission efficiency.
[0029] 3. Multi-parallel processing is performed, and data is stored and convolution operations are performed in multiple channels. The weight data and image data enter the convolution kernel at the same time, saving some waiting time.
[0030] 4. Adjusted the synthesis strategy of the synthesizer so that its placement and routing is more inclined to improve system speed, and performed some timing optimizations based on the timing report.
[0031] 5. Optimize the multiplication and addition operations of convolution. Multiplication fully utilizes DSP resources to speed up. One DSP resource is used for every two multiplications. Addition uses an addition matrix with balanced clock paths, a tree adder with high-speed Koggle Stone algorithm, and inserts registers to generate pipelines.
[0032] 6. The algorithm performs channel fusion, processes a batch of data in one convolution operation, normalizes the convolution kernel, and converts the 3x3 convolution kernel into a 1x1 convolution kernel, which improves the operation speed and has strong versatility.
[0033] 7. The system uses mobilenet_v1 SSD algorithm for target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is a schematic diagram of the overall architecture of the FPGA; Figure 2 This is a schematic diagram of the Accsystem architecture;
[0035] Figure 2 This is a schematic diagram of the architecture of the ConvCore module;
[0036] Figure 3 This is a schematic diagram of the architecture of the din_fifo unit;
[0037] Figure 4 This is a schematic diagram of the architecture of the ConvCompute unit;
[0038] Figure 5 This is a schematic diagram of the architecture of the convolutional computing module;
[0039] Figure 6 is a schematic diagram of the architecture of a multiplication calculation unit;
[0040] Figure 7 is a schematic diagram of the architecture of an adder unit;
[0041] Figure 8 Schematic diagram of the parallel computing architecture of the convolution computing unit. DETAILED DESCRIPTION
[0042] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0043] This system is mainly composed of a camera, FPGA, and an alarm device. The camera is mainly used to identify trains and people on the railway. The FPGA is used to process and calculate the video data from the camera and obtain the recognition results. The alarm module will sound an alarm after receiving the signal that it recognizes people and trains, which is used to alert people around. The FPGA development board model is Altera's C5SOC.
[0044] like Figure 1 As shown in the figure, the FPGA is divided into two parts, namely the PL part and the PS part. The PL is the logic part, which is the FPGA side, and the PS part is the ARM side. Among them, the PL side mainly includes the AccSystem unit, the operator unit and the storage unit, and the PS side mainly includes the fpgadrv unit, the ssd_detection unit, the Paddlelite unit, the mobilenet_v1 SSD unit, the lib unit, and the opencv unit.
[0045] The PL end mainly realizes the acceleration of convolution; the PS end mainly transmits image data to the PL and also needs to further process the convolution result. The overall system block diagram is as follows Figure 1 The specific functions of each unit are shown below.
[0046] AccSystem: The unit mainly accelerates the convolution part. Data enters the convolution kernel through the Avalon bus, and the convolution kernel performs convolution operations on the weight data and image data.
[0047] Fpgadrv: Fpgadrv is the driver for the FPGA accelerator on the Linux side.
[0048] Mobilenet_v1 SSD: Mobilenet_v1 SSD stores the SSD model and parameters.
[0049] Lib: Lib contains the library files required to run the demo.
[0050] Ssd_detection_src: Contains the demo source code.
[0051] Opencv: compiled opencv library.
[0052] like Figure 2 As shown in Figure 1, Accsystem consists of AvalonBusUpSizer ABUS module, AvalonBusMatrix MTX module, ConvCore module and sysCtrl module. The sysctrl module completes the configuration of the convolution kernel register by configuring the mode register.
[0053] The AvalonBusUpsizer ABUS module performs bus adaptation on the data and expands the 128-bit bus to 512-bit.
[0054] The AvalonBusMartix MTX module judges and arbitrates the data entering the module, adapts according to the address bit, determines whether it is image data or convolution kernel data, and places the convolution kernel calculation result on the bus. The ConvCore module mainly performs convolution operations on the input data.
[0055] like Figure 3 As shown in the figure, the ConvCore module consists of din_fifo unit, ConvCompute unit and dout_fifo unit. Among them, din_fifo is mainly used to cache input data, ConvCompute is mainly used to perform multiplication and addition operations on the input data of the convolution kernel, and dout_fifo is mainly used to cache the data after calculation.
[0056] din_fifo:such as Figure 4 As shown in the figure, din_fifo mainly caches the input weight data and image data and splices them into the convolution calculation module. Figure 5 As shown in the figure, this is the structure diagram of din_fifo. Input fifo completes the cache of input image data, and kernel fifo completes the cache of weight data. When data enters, the input fifo unit performs ping-pong storage. Convheight counts the number of kernel rows in each batch of kernel, packs each 32-bit into a unit, and then copies the 32-bit unit to make it 512-bit. Finally, the weight data and image data are spliced, and the spliced result is input into the convolution calculation unit.
[0057] ConvCompute: Figure 5 As shown in the figure, the function of the convolution calculation module is to perform multiplication and addition operations on the input image data and weight data. The weight data and image data are multiplied by the 8X8 multiplier unit, and the result enters the adder array unit for addition calculation. The ConvHeght counter counts the number of kernel rows in each batch and controls the output enable.
[0058] Multiplier unit: For the multiplication calculation unit, such as Figure 6 As shown. We use 4 multipliers to perform 8-bit fixed-point multiplication on the input image data and weight data, and then input the result into the addition matrix module for addition operation, so that we have completed a point operation.
[0059] Adder Matrix Unit: Figure 7 As shown in the figure, the adder unit consists of three layers of adders. The data first passes through 32 adders to complete an addition operation, then the calculation result is added again, and then the data is stored and then passed through a layer of adders to complete the self-addition operation. For the adder, we chose the Koggle-Stone tree adder parallel algorithm, which greatly improves the speed of addition operations.
[0060] In order to increase the computing speed of the convolution calculation unit, we adopt multi-core parallel computing, that is, using 64 multipliers to complete the calculation of 16 points in parallel, so as to maximize the computing power of the convolution calculation unit.
[0061] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the embodiments. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.
Claims
1. A railway crossing train identification and warning system based on monocular vision, characterized in that: The camera is connected to the data processing system, and the data processing system is connected to the alarm device; The camera is used to take pictures, the data processing system is used to process the pictures taken by the camera, and the alarm device is used to sound an alarm when the data processing system determines that a train is passing by; The data processing system includes the PL side and the PS side, where the PL is the logic part, which is the FPGA side, and the PS part is the ARM side; Among them, the PL side includes the AccSystem unit, and the PS side includes the fpgadrv unit, ssd_detection unit, Paddlelite unit, mobilenet_v1 SSD unit, lib unit, and opencv unit; The AccSystem unit is connected to the fpgadrv unit, the fpgadrv unit is connected to the ssd_detection unit, and the ssd_detection unit is connected to the lib unit, opencv unit, and Paddlelite unit. The AccSystem unit is used to accelerate the convolution part; The fpgadrv unit is the driver for the FPGA accelerator on the Linux side; Mobilenet_v1 SSD unit is used to store SSD models and parameters; The Lib unit is used to store the library files required to run the demo; The Ssd_detection unit is used to store the demo source code; The Opencv unit is used to store the compiled Opencv library; The Paddlelite unit is used to load the paddle frame; It also includes a ConvCore module, which includes a din_fifo unit, a ConvCompute unit, and a dout_fifo unit; the din_fifo unit is used to cache input data, the ConvCompute unit performs multiplication and addition operations on the input data of the convolution kernel, and the dout_fifo unit is used to cache the data after calculation; the din_fifo unit includes an Inputfifo module, a Kernal fifo module, and a Convheight counter; the Input fifo module caches the input image data, and the Kernal fifo module caches the weight data; when the data enters, the input The fifo unit performs ping-pong storage on the input image data; the Convheight counter counts the number of kernel rows in each batch, packs each 32 bits into a unit, and then copies the 32-bit unit to make it 512 bits. Finally, the weight data and image data are spliced, and the spliced result is input to the convolution calculation unit; the ConvCompute unit performs multiplication and addition operations on the input image data and weight data; the ConvCompute unit includes an 8*8 multiplier unit and an adder array unit; the weight data and image data are multiplied by the 8X8 multiplier unit, and then the result enters the adder array unit for addition calculation; the ConvHeght counter counts the number of kernel rows in each batch and controls the output; the ConvCompute unit uses multi-core parallel computing, that is, 64 multipliers are used to complete the calculation of 16 points in parallel; when the adder array unit performs addition calculation, the koggle stone algorithm is used for optimization; the railway crossing train recognition and warning system based on monocular vision adopts mobilenet_v1 The SSD algorithm is used for target detection. The data processing flow of the railway crossing train identification and warning system based on monocular vision is as follows: the ARM side fuses and assembles the data according to the same batch of data from different channels. Then each row of data is 64 bits, and a row of data contains data of 16 points. The data required for each point is split and assembled according to 4 bytes. The ConvHeight unit calculates the amount of data required for a row of results and takes the quotient of 4 bytes upward. The result is then output to the output memory according to 16 points per row. Only the calculation depth and result depth are configured in the configuration parameters. The ARM side needs to configure the kernel for each batch first, and then slice the data and transfer it to the acceleration core on the FPAG side via the AXI bus to the Avalon bus. The acceleration core performs convolution calculations while receiving data. After the transmission is completed, it is determined whether the data processing is completed by reading the status register, and then the data is read back to the ARM side for reorganization.The ARM-side input memory uses a two-line storage structure, totaling 128 bytes of memory, while the kernel memory uses 512 lines of memory and the output memory uses 4096 lines of memory, storing 65,536 result points. In the Avalon bus selector, the Avalon address lines are extended to 18 bits, extending the mapping address on the ARM side to 4,194,304 bytes, while the input memory on the FPGA side only occupies 128 bytes. This optimizes the Avalon bus address mapping selection rules. The input memory uses a ping-pong-like structure to ensure input data accuracy and implement a streaming processing architecture that processes data while it is being transmitted. The driver performs channel fusion and kernel normalization on data requiring convolution. This means that any convolution calculation is treated as a 1x1x multi-channel convolution, improving the versatility of the convolution accelerator kernel.
