Image recognition system based on FPGA

Abstract

The invention provides an image recognition system based on an FPGA, and the system comprises a sensor which is used for collecting image information; the FPGA integrated processor is connected with the sensor, and is used for carrying out image segmentation on the image information, extracting at least one target object in the image information, respectively scanning each pixel of each target object based on an FPGA image algorithm, and determining the type of each target object according to the angular point number of each target object obtained by scanning; and the upper computer is connected with the FPGA integrated processor and is used for displaying the type of the target object.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and particularly to an FPGA-based image recognition system. Background Art

[0002] The importance of a high-speed camera image recognition system in modern technological applications is self-evident. Such systems are widely used in numerous fields, such as industrial automation, intelligent transportation systems, aerospace, military reconnaissance, etc. In these applications, accurately and quickly identifying high-speed moving targets is crucial. However, traditional image processing methods face many challenges when dealing with the recognition of high-speed moving targets. First of all, due to the extremely fast speed of the target movement, traditional processing methods often struggle to process and recognize images in real time. This leads to a rapid accumulation of computational workload when processing consecutive high-speed images, making the processing time become longer and longer. This long processing and computational delay will further cause other problems. For example, due to the long processing time, the target may have moved out of the camera's field of view during the processing, resulting in the loss of the target. In addition, the recognition accuracy of traditional processing methods will also drop significantly when facing high-speed movement, because traditional algorithms often do not take into account the characteristics of high-speed movement of the target. Therefore, how to design a real-time, efficient, and accurate image recognition algorithm according to the characteristics of high-speed moving targets is an urgent problem to be solved in the current field of high-speed camera image recognition. Only by solving this problem can the high-speed camera image recognition system be better applied to various modern technological fields and play its due value. Summary of the Invention

[0003] (1) Technical Problems to be Solved

[0004] In view of the above problems, the present disclosure provides an FPGA-based image recognition system to at least partially solve the problems of slow processing speed, poor real-time performance, low recognition accuracy and precision in traditional image processing.

[0005] (2) Technical Solutions

[0006] The present disclosure provides an FPGA-based image recognition system, including: a sensor for collecting image information; an FPGA integrated processor connected to the sensor for performing image segmentation on the image information, extracting at least one target object from the image information, scanning each pixel of each target object based on an FPGA-based image algorithm, and determining the type of each target object according to the number of corner points of each target object obtained by the scanning; and a host computer connected to the FPGA integrated processor for displaying the type of the target object.

[0007] According to an embodiment of the present disclosure, the FPGA integrated processor is further configured to: receive the image information of the sensor; perform binarization processing on the image information, scan the segmented image information through a sliding window, extract each target object in the image information, and respectively perform pixel-by-pixel scanning on each target object in the X direction and the Y direction by using the image algorithm, and obtain the type of each target object according to the number of corner points of each scanned target object; and upload the type of each target object to the host computer in real time.

[0008] According to an embodiment of the present disclosure, the FPGA integrated processor is further configured to: receive the configuration data preset by the host computer, and perform an initialization operation on the sensor according to the configuration data.

[0009] According to an embodiment of the present disclosure, the FPGA integrated processor is further configured to: receive the delay signal preset by the host computer, convert the delay signal into a trigger control signal, and control the sensor to collect the image information after a preset time delay.

[0010] According to an embodiment of the present disclosure, it further includes: an external light source for providing illumination when the sensor collects the image information; a trigger source for emitting a trigger signal, transmitting the trigger signal to the external light source and the sensor, triggering the external light source to emit light, and triggering the sensor to collect the image information with a time delay under the action of the delay signal.

[0011] According to an embodiment of the present disclosure, it further includes: a trigger input and output interface provided in the FPGA integrated processor for receiving the trigger signal and controlling the exposure amount of the external light source by setting the trigger source.

[0012] According to an embodiment of the present disclosure, it further includes: a DDR memory provided in the FPGA integrated processor for caching the image information.

[0013] According to an embodiment of the present disclosure, the host computer is connected to the FPGA integrated processor through a serial port and a USB communication library.

[0014] According to an embodiment of the present disclosure, the sensor is a PYTHON 300 sensor.

[0015] According to an embodiment of the present disclosure, the sensor, the FPGA integrated processor, and the host computer are respectively distributed on a PCB board in a tower structure.

[0016] (III) Beneficial effects

[0017] The FPGA-based image recognition system provided by the present disclosure uses an FPGA integrated processor as the core processing unit, making full use of the characteristics of high-speed parallel computing of the FPGA. It can achieve real-time processing of a video stream with a size of 256*256 and a frame rate of 2000 frames, enabling fast image processing and feature extraction while the image data is being transmitted. Thus, complex image processing tasks can be completed in an extremely short time, and the rapid changes of high-speed moving targets can be handled, improving the accuracy and precision of target recognition. Description of the Drawings

[0018] To more fully understand the present disclosure and its advantages, reference will now be made to the following description in conjunction with the accompanying drawings, where:

[0019] Figure 1 Schematically shows a block diagram of an FPGA-based image recognition system provided according to an embodiment of the present disclosure;

[0020] Figure 2 Schematically shows an application block diagram of an FPGA-based image recognition system provided according to an embodiment of the present disclosure. Detailed Embodiments

[0021] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present disclosure.

[0022] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0024] Some block diagrams and / or flowcharts are shown in the accompanying drawings. It should be understood that some of the blocks or combinations of blocks in the block diagrams and / or flowcharts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when executed by the processor, these instructions can create a device for implementing the functions / operations illustrated in these block diagrams and / or flowcharts.

[0025] Therefore, the technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). Additionally, the technology of the present disclosure can take the form of a computer program product on a computer-readable medium storing instructions, which can be used by or in conjunction with an instruction execution system. In the context of the present disclosure, a computer-readable medium can be any medium that can contain, store, transmit, propagate, or transport instructions. For example, a computer-readable medium can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, components, or propagation media. Specific examples of computer-readable media include: magnetic storage devices, such as magnetic tapes or hard disk drives (HDDs); optical storage devices, such as compact discs (CD-ROMs); memories, such as random access memories (RAMs) or flash memories; and / or wired / wireless communication links.

[0026] Figure 1 The structural block diagram of an FPGA-based image recognition system provided according to an embodiment of the present disclosure is schematically shown.

[0027] As Figure 1 shown, the FPGA-based image recognition system includes a sensor, an FPGA integrated processor, and a host computer. Among them, the FPGA integrated processor mainly includes a sensor interface module, a DDR memory, a core processing module, and 4 main interfaces.

[0028] The sensor is used to collect image information; the FPGA integrated processor is connected to the sensor and is used to perform image segmentation on the image information, extract at least one target object from the image information, scan each pixel of each target object based on the FPGA-based image algorithm, and determine the type of each target object according to the number of corner points of each target object obtained by the scan; the host computer is connected to the FPGA integrated processor and is used to display the type of the target object.

[0029] Specifically, the sensor is connected to the FPGA integrated processor via LVDS, with a communication rate of 576 Mbps. The FPGA needs to complete functions such as image acquisition, caching, processing, transmission, and trigger signal output. Through the evaluation of the sensor LVDS signal rate, image cache size, and algorithm scale, the Artix-7 series FPGA of Xilinx is selected for development. The FPGA accelerates operations mainly in two directions: increasing the frequency and parallel computing. For the embodiments of the present disclosure, the image algorithm operates on each pixel of the image. The pixel transmission rate of the image is 288 MHz. For the FPGA, a clock frequency exceeding 200 MHz will bring complex timing problems. Therefore, in the present disclosure, the idea of increasing parallel computing is mainly adopted to accelerate the algorithm. The processing flow of the image algorithm includes three major steps: image segmentation, target extraction, and target recognition. Among them, target recognition needs to complete two scans in the X direction and the Y direction respectively. At the same time, since the exposure time is the same and the targets to be recognized are also the same, image transmission and image segmentation can be executed in a pipeline. The size of the window selected for target extraction is 30*30. Therefore, when the image segmentation module outputs the 31st row of the image, the target extraction operation can be performed. During target extraction, the original image and an image rotated 90 degrees are stored respectively to prepare for the next parallel scan.

[0030] The horizontal working distance of the FPGA-based image recognition system is 100 cm, and the horizontal and vertical distances of the field of view are not less than 100 cm. After power-on, the FPGA integrated processor first configures the sensor and initializes the exposure area and other basic sensor parameters. After the external trigger signal is input, the FPGA integrated processor notifies the sensor to start exposure. After the preset exposure duration, such as 0.12 ms, the image is output. While the FPGA integrated processor caches the image information into the DDR memory, it performs the operation of the image algorithm. Then, the image and the result information are output to the host computer through the USB3.0 interface.

[0031] The present disclosure uses the logic of the FPGA to complete all positioning algorithm calculations and interface implementations, can perform real-time recognition and processing of images, and upload the recognition results to the host computer in real time. It makes full use of the high-speed parallel computing characteristics of the FPGA, and can realize the real-time processing of a video stream with a size of 256*256 and a frame rate of 2000 frames. Thus, through hardware acceleration technology, the acquisition, processing, and analysis of high-speed image data are realized. At the same time, the parallel processing architecture of the FPGA enables fast image processing and feature extraction while the image data is being transmitted, achieving the completion of complex image processing tasks in an extremely short time.

[0032] Furthermore, the FPGA integrated processor is further configured to: receive the image information of the sensor by using the sensor interface module; the core processing module performs binarization processing on the image information, scans the segmented image information through a sliding window, extracts each target object in the image information, and uses image algorithms to scan each target object pixel by pixel in the X direction and the Y direction respectively, and obtains the type of each target object according to the number of corner points of each scanned target object; and upload the type of each target object to the host computer in real time.

[0033] In the embodiment of the present disclosure, the difference between the target image and the background is obvious, and the targets to be recognized have consistency in gray scale, and the method of binarization is applicable to perform image segmentation. The threshold of binarization segmentation can be determined by using the median value of the maximum gray scale and the minimum gray scale of the image in the case of simple image content. Then, the binarized image is scanned through a window of 30*30. When all the values around the 30*30 matrix are 0 and the sum of the matrix elements is greater than 10, it is considered that the window completely encloses the target. After completely scanning an image, multiple 30*30 targets can be obtained to complete the target extraction operation. Using the basic principle of Harris corner detection, sliding detection is performed in the X and Y directions respectively by using a 3*3 window. When there is a large change in the number of target pixels in both the X and Y directions in the window, it is considered that there is a corner point in the detection window. By calculating the number of corner points, the type of the target object can be confirmed. For example, when the number of corner points is 0, the image is considered to be circular; when the number of corner points is 3, the image is considered to be triangular; when the number of corner points is 4, the image is considered to be square.

[0034] By processing the image through binarization, based on a suitable threshold value, the interference of the background in the image to the target image can be greatly eliminated. And the adopted Harris corner detection algorithm is relatively simple, the number of parameters is small, and all are integer operations, which is convenient to be deployed on the FPGA. At the same time, binarization processing and sliding window scanning segmentation can accurately distinguish each target object in the image information. And, using the number of corner points to judge the type of the target object can avoid the influence of problems such as image distortion or uneven illumination on the recognition result and improve the recognition accuracy.

[0035] Furthermore, the FPGA integrated processor is further configured to: receive the configuration data preset by the host computer and perform an initialization operation on the sensor according to the configuration data.

[0036] Receive the sensor initialization data sent by the host computer through the serial port, perform an initialization operation on the sensor, and initialize the exposure area and other basic sensor parameters.

[0037] Receiving the configuration data preset by the host computer, the FPGA integrated processor can perform customized initialization operations on the sensor. In this way, according to the actual application scenarios and requirements, the configuration of the sensor can be flexibly adjusted, improving the adaptability and flexibility of the system. At the same time, by using the configuration data preset by the host computer, the function of the FPGA integrated processor can be extended without changing its hardware design.

[0038] Furthermore, the FPGA integrated processor is also configured to: receive the delay signal preset by the host computer, convert the delay signal into a trigger control signal, and control the sensor to collect image information after a preset delay duration.

[0039] In order to wait for the external light source to emit light with a rated power, the camera will delay for a preset duration before performing exposure. By receiving the delay signal preset by the host computer, the FPGA integrated processor can accurately control the time point at which the sensor collects image information, ensuring image acquisition at the correct time point, thereby obtaining more complete and accurate image information.

[0040] In the embodiment of the present disclosure, it further includes an external light source and a trigger source. The external light source is used to provide illumination when the sensor collects image information. The trigger source is used to emit a trigger signal and transmit the trigger signal to the external light source and the sensor, triggering the external light source to emit light, and triggering the sensor to delay the collection of image information under the action of the delay signal.

[0041] Through the cooperation of the external light source and the trigger source, synchronous triggering among the sensor, the external light source, and the trigger source can be achieved. When the trigger source emits a trigger signal, the external light source will emit light synchronously, and at the same time, the sensor will also delay the collection of image information under the action of the delay signal. This synchronous triggering mechanism can improve the accuracy and stability of image acquisition.

[0042] In the embodiment of the present disclosure, it further includes a trigger input and output interface, which is provided in the FPGA integrated processor and is used to receive the trigger signal and control the exposure amount of the external light source by setting the trigger source.

[0043] The FPGA integrated processor not only includes a USB3.0 output interface for outputting the parallel data transmitted by the FPGA to the host computer, and an RS422 interface, i.e., a serial port, for transmitting instructions and configuration data. The specific function of the serial port is first to transmit configuration data, with the data direction from the host computer to the FPGA, so as to initialize the sensor. Secondly, it is to transmit an enable signal. When the FPGA completes the calculation, it will send a flag signal indicating the completion of the calculation, and this signal needs to be transmitted to the host computer and the oscilloscope to check whether the calculation time meets the standard. In addition, it also includes trigger input and output interfaces (reserved Camera link interfaces). Through the trigger input and output interfaces, the FPGA integrated processor can set the trigger source according to the preset logic or algorithm, so as to control the exposure of the external light source, better adapt to different ambient light and acquisition requirements, and improve the quality and stability of image acquisition. At the same time, through the trigger input and output interfaces, the FPGA integrated processor can communicate and control with external devices. In this way, the linkage and interaction with the external light source can be realized, enhancing the flexibility and scalability of the system.

[0044] In the embodiment of the present disclosure, it further includes a DDR memory, which is disposed in the FPGA integrated processor and is used for caching image information.

[0045] The DDR memory has a high read and write speed and can quickly cache a large amount of image information. By storing the image information in the DDR memory, the rapid reading and writing of the image information can be realized, meeting the requirements of real-time processing and transmission. Since the read and write speed of the DDR memory is relatively fast, the image information is first cached in the DDR memory and then processed and analyzed by the FPGA integrated processor. This can avoid the problem of reduced processing efficiency caused by the read and write delay of external storage devices and improve the overall processing efficiency. At the same time, since the DDR memory is integrated in the FPGA integrated processor, the dependence on external storage devices can be reduced. This can reduce the connection and failure risks of external devices and improve the stability and reliability of the system.

[0046] In the embodiment of the present disclosure, the host computer is connected to the FPGA integrated processor through the serial port and the USB communication library.

[0047] The host computer software is designed using the QT development environment, calls the serial port and the USB communication library to connect to the board, and has functions such as camera register initialization, trigger source signal setting, recognition target selection (not used temporarily), and recognition information feedback display.

[0048] The serial port and USB communication library support high-speed data transmission, which can meet the requirements of real-time image information transmission. Through high-speed data transmission, the data interaction speed between the host computer and the FPGA integrated processor can be ensured, and the response speed and performance of the overall system can be improved. At the same time, through the serial port and USB communication library, the host computer can establish a stable and reliable communication connection with the FPGA integrated processor. This general connection method enables the host computer to flexibly communicate with the FPGA integrated processor, realizing a wider range of applications.

[0049] In the embodiment of the present disclosure, the sensor is a PYTHON 300 sensor.

[0050] The sensor selected is the PYTHON 300 sensor of ON Semiconductor. In the ZROT mode with a resolution of 256*256, the frame rate can reach 2681 frames.

[0051] In the embodiment of the present disclosure, the sensor, the FPGA integrated processor, and the host computer are respectively distributed on the PCB board of the tower structure. This highly integrated design can improve the compactness and maintainability of the system, reducing the risk of connection lines and interface failures between components. Moreover, since they are distributed on the same PCB board, the signal transmission path between them can be optimized, thereby reducing signal delay and loss, and improving the signal quality and reliability of the overall system.

[0052] Figure 2 The application block diagram of the FPGA-based image recognition system provided by the embodiment of the present disclosure is schematically shown.

[0053] As Figure 2 shown, the working process is as follows: The host computer writes the configuration data to the camera through RS422 to complete the initialization configuration. The trigger source sends a trigger signal to the high-speed camera and the external light source (which can be given to the oscilloscope to verify whether the calculation is completed within 500 us). After waiting for a fixed delay (50 us), the camera starts to expose and collect images, and sends the collected images to the FPGA for image recognition calculation through LVDS, and sends the calculation result to the host computer. Among them, the assessment computer is used to assess whether the operation time of the FPGA integrated processor for image information is less than or equal to the first threshold, and the imaging computer is used to display the final result. The host computer software is designed using the QT development environment and calls the serial port and USB communication library to connect to the FPGA integrated processor.

[0054] The parallel processing architecture based on the FPGA integrated processor of the present disclosure can complete the calculation of one frame of image within 500 μs, and the real-time frame rate can reach more than 2000 frames, realizing the acquisition, processing and analysis of high-speed image data, so as to realize complex image processing in an extremely short time, and being able to cope with the rapid changes of high-speed moving targets, improving the accuracy and precision of target recognition. At the same time, the high-speed integrated module is equipped with external trigger input and output interfaces, and can perform exposure control on external light sources. This intelligent control method can improve the self-adjusting ability and adaptability of the system, reduce manual intervention and incorrect operations, and thus improve the intelligent level of the system.

[0055] Those skilled in the art can understand that the features recited in the various embodiments and / or claims of the present disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly recited in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features recited in the various embodiments and / or claims of the present disclosure can be combined and combined in various ways. All such combinations and / or combinations fall within the scope of the present disclosure.

[0056] Although the present disclosure has been shown and described with reference to specific exemplary embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. An FPGA-based image recognition system, characterized in that, it includes: a sensor for collecting image information; an FPGA integrated processor connected to the sensor, configured to perform image segmentation on the image information, extract at least one target object from the image information, scan each pixel of each target object based on an FPGA-based image algorithm, and determine the type of each target object according to the number of corner points of each target object obtained by the scan; a host computer connected to the FPGA integrated processor for displaying the type of the target object.

2. The FPGA-based image recognition system according to claim 1, characterized in that, the FPGA integrated processor is further configured to: receive the image information from the sensor; perform binarization processing on the image information, scan the segmented image information through a sliding window, extract each target object from the image information, and scan each pixel of each target object in the X direction and Y direction respectively using the image algorithm, and obtain the type of each target object according to the number of corner points of each target object scanned; upload the type of each target object to the host computer in real time.

3. The FPGA-based image recognition system according to claim 1, characterized in that, the FPGA integrated processor is further configured to: receive the configuration data preset by the host computer and perform an initialization operation on the sensor according to the configuration data.

4. The FPGA-based image recognition system according to claim 3, characterized in that, the FPGA integrated processor is further configured to: receive the delay signal preset by the host computer, convert the delay signal into a trigger control signal, and control the sensor to collect the image information after a preset time delay.

5. The FPGA-based image recognition system according to claim 4, characterized in that, it further includes: an external light source for providing illumination when the sensor collects the image information; a trigger source for emitting a trigger signal, transmitting the trigger signal to the external light source and the sensor, triggering the external light source to emit light, and triggering the sensor to collect the image information with a time delay under the action of the delay signal.

6. The FPGA-based image recognition system according to claim 5, characterized in that, it further includes: a trigger input and output interface provided in the FPGA integrated processor for receiving the trigger signal and controlling the exposure amount of the external light source by setting the trigger source.

7. The FPGA-based image recognition system according to claim 1, characterized in that, it further includes: a DDR memory provided in the FPGA integrated processor for caching the image information.

8. The FPGA-based image recognition system according to claim 1, characterized in that, the host computer is connected to the FPGA integrated processor through a serial port and a USB communication library.

9. The FPGA-based image recognition system according to claim 1, characterized in that, The sensor is a PYTHON300 sensor.

10. The FPGA-based image recognition system according to claim 1, characterized in that, the sensor, the FPGA integrated processor and the host computer are respectively distributed on the PCB boards of the tower structure.

Images