Hardware acceleration-based real-time license plate recognition system

Abstract

The application relates to a hardware acceleration-based real-time license plate recognition system and belongs to the technical field of computer vision, which is used for solving the problem that in complex environments such as heavy fog weather, the data amount to be processed is extremely huge in the process of recognizing license plate images for a large-view lens, time consumption of detection is long, and real-time detection of high-frame-rate and high-resolution images is difficult to realize. The system is provided with the image defogging processing and license plate number area positioning by adopting an FPGA processing board, and then the number recognition is performed by a license plate recognition module in the host computer software, so that the license plate number recognition precision is improved, and the image data real-time processing speed is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a license plate real-time recognition system based on hardware acceleration. BACKGROUND

[0002] License plate recognition detection is an important research topic in the field of intelligent transportation. Nowadays, vehicles have become a part of our life, which greatly facilitates travel, but also brings many problems. License plate recognition plays an important role in traffic detection, congestion control, violation monitoring, and suspicious vehicle detection. However, in heavy fog weather, the visibility is very low, and the detailed information of the vehicle is seriously lost, which brings great difficulty to license plate recognition. Moreover, for a large field of view lens, the license plate image only occupies a small part of the whole image. Using the whole image for license plate recognition, the recognition accuracy is difficult to guarantee. In the recognition process, the amount of data to be processed is also extremely large, which directly leads to long detection time and difficulty in realizing real-time detection of high frame rate and high resolution images. In view of the development trend of the field of computer vision in the new era, there is an urgent need for a method that can adapt to complex environments, large fields of view, high frame rates, and high resolution video license plate real-time detection. SUMMARY

[0003] In complex environments such as heavy fog weather, for a large field of view lens, the license plate image only occupies a small part of the whole image. Using the whole image for license plate recognition, there is a problem of extremely large amount of data to be processed in the recognition process, leading to long detection time and difficulty in realizing real-time detection of high frame rate and high resolution images. In view of the foregoing problem, the present application proposes a license plate real-time recognition system based on hardware acceleration. The FPGA processing board is used for image defogging processing and license plate number region positioning. After that, the number recognition is performed by the license plate recognition module in the host computer software. This not only improves the license plate number recognition accuracy, but also improves the image data real-time processing speed.

[0004] The technical scheme of the present application is: a license plate real-time recognition system based on hardware acceleration, the system comprising an FPGA processing board and a license plate recognition module.

[0005] The FPGA processing board uses a defogging module to perform hardware defogging processing on the image according to the video image input source, and transmits the defogged image to the license plate positioning module for license plate position preselection.

[0006] The position coordinates and the region image centered on the coordinates are transmitted to the license plate recognition module in the host computer software, and the characters of the license plate are recognized by the license plate recognition module.

[0007] In the above technical solution, the FPGA processing board comprises 1 HDMI input interface, 2 optical fiber input / output interfaces, 1 HDMI output interface, 1 gigabit Ethernet output interface, 1 FPGA processing chip and 4 DDR3 particles.

[0008] Among them:

[0009] The HDMI input interface is used for connecting an HDMI line and inputting a computer device projection;

[0010] The optical fiber input / output interface is connected to a camera end through an optical fiber line, and the camera video image is input by the input interface, and the camera is instructed by the output interface, such as adjusting exposure, resolution and other camera parameters;

[0011] The HDMI output interface outputs the video image after fog removal to a display to view the fog removal effect;

[0012] The Ethernet output interface is used for uploading the position coordinates and the specified area image to the upper computer software, and using the license plate recognition module in the upper computer software to recognize the license plate characters;

[0013] The FPGA processing chip is used for completing the calculation required for fog removal processing and license plate position preselection;

[0014] The DDR3 particle is used for buffering the image and the calculation data.

[0015] In the above technical solution, the specified area image is a rectangular area surrounded by extending X1 and Y1 pixels in the positive and negative directions of the X axis and the Y axis, respectively, with the geometric center of the located license plate area as the coordinate origin and the Cartesian coordinate system as the reference coordinate system. The set values of X1 and Y1 are related to the calculation capability of the FPGA processing chip.

[0016] In the above technical solution, the fog removal module comprises the following processing:

[0017] Each pixel of the input image is split into 3 8-bit data, a comparator is used to compare the 3 data, the minimum value is taken as the pixel value of the dark channel image of the input video image, and the original image timing is output;

[0018] The pixel value of the output dark channel image is used to build a 9x9 filter window as a basic template, and a 5x5 sub-window is used to traverse the basic template. First, the average value of each sub-region pixel is calculated, then the difference between the average value and the center point pixel value is calculated, and then the sub-region with the minimum difference value is selected for minimum value filtering to obtain the processed dark channel, which is used to calculate the transmittance;

[0019] The accumulated operation is performed on the dark channel pixel value, and the accumulated result is stored in each address of the BRAM, each address corresponding to the histogram statistical result of a gray value, the mean value of about 0.1% pixels of the previous frame dark channel image is estimated as a threshold value through the histogram statistics, the data of the previous frame and the current frame are buffered in the DDR unit, and then the atmospheric light value is estimated by performing linear adjustment on the difference between the pixels and the threshold value in the three channels of the current frame respectively;

[0020] Based on the transmittance and the atmospheric light value, a foggy image is obtained by using a foggy imaging model.

[0021] In the above technical solution, the license plate positioning module includes the following processing:

[0022] The RGB image after the defogging processing is converted to the YCbCr color space;

[0023] The FPGA is used for parallel processing, and the Cb and Cr components are used for extracting the gray image, and then the target information enhanced gray image is obtained;

[0024] The target information enhanced gray image is binarized and segmented, and then morphological processing is performed to restore the geometric size of the target region and filter out the speckle-like interference.

[0025] In the above technical solution, the target information enhanced gray image acquisition step includes:

[0026] The Cr component gray image is traversed pixel by pixel according to the row field timing, and two registers are set to store the target pixel range value and the pixel mean value;

[0027] After the traversal is completed, the difference between the pixel mean value and the target pixel range value in the Cr component is taken as a threshold C, the threshold C is compared with the Cr component to obtain the processed Cr component gray image, and the processed Cr component gray image is stored in the DDR;

[0028] The Cb component gray image and the processed Cr component gray image under the same frame are read out from the DDR buffer unit, and the DSP resource is used for pixel-by-pixel difference operation to obtain the target information enhanced gray image.

[0029] In the above technical solution, the morphological processing includes:

[0030] First, the binarized gray image is dilated to connect the holes in the license plate region, and a row buffer structure is constructed using the BRAM; after three rows of data are buffered, the image is read into a register array of a required size to construct a convolution unit, and the convolution kernel size constructed by the register is fixed at 9x9;

[0031] Then, two times of corrosion are performed, and the convolution kernel is 9*9 and 3*3 respectively, so as to restore the geometric size of the target region and filter out the speckle interference.

[0032] In the technical scheme, after the binarization segmentation, the improved bounding box target detection algorithm is used to recognize the binarization image containing the license plate region.

[0033] Firstly, a BRAM memory is defined, and the area of the line with overlapping area in the image read out by the DDR is accumulated in a row-by-row scanning manner, and the accumulation result is stored in the minimum storage unit in the BRAM address.

[0034] Then, the target license plate coordinates are outputted and smoothed, the smoothing operation is to set a fixed limit difference value between the two results, and the subtraction result and the limit difference value are sent to the comparator.

[0035] In the technical scheme, the filter condition is stored in the ROM resource of the FPGA.

[0036] In the technical scheme, the license plate recognition module includes the following processing:

[0037] Based on the accepted position coordinates, the license plate coordinates are processed for coordinate offset to obtain actual position information.

[0038] A rectangular structure element is constructed for the license plate image, and the binarization image is sequentially eroded and expanded, and the constructed structure element and the processed binarization image are subtracted to obtain an image with redundant parts removed.

[0039] For the image with the redundant parts removed, each pixel point is traversed to obtain the target pixel number of each column of the image.

[0040] Step S: compare the target pixel number of each column with a set threshold A to determine whether the column is a character.

[0041] If it is not a character, the continuous column number with the target pixel number greater than the set threshold A is compared with a set threshold B to update the character position information.

[0042] If it is a character, the character width is accumulated, and when the character width is no longer accumulated, the current character position positioning is ended, and the next character positioning is started;

[0043] Return to step S until the 8 character positions on the license plate are all positioned and ended;

[0044] The character recognition is performed by using template matching, an affine transformation target image is created, an affine transformation matrix is calculated, an affine transformation operation is performed on the loaded image, and then the affine transformed image is compared with the character library, and the recognized license plate number is output. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0046] Figure 1 , the hardware acceleration-based license plate real-time recognition system in a complex environment provided by the embodiment of the present application is shown in the schematic diagram;

[0047] Figure 2 , the hardware acceleration-based license plate real-time recognition system working process schematic diagram provided by the embodiment of the present application in a complex environment is shown in the schematic diagram;

[0048] Figure 3 , the FPGA processing board algorithm flow schematic diagram for license plate region positioning in a complex environment provided by the embodiment of the present application is shown in the schematic diagram;

[0049] Figure 4 , the network data transmission content schematic diagram provided by the embodiment of the present application is shown in the schematic diagram;

[0050] Figure 5 , the host computer software effect schematic diagram provided by the embodiment of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments.

[0052] The hardware acceleration-based license plate real-time recognition system in a complex environment is composed of Figure 1 , mainly including a video image source, an FPGA processing board, a computer device, a display and host computer software. The video image source is divided into a camera image picture, a computer device projection picture and a test image generated inside the FPGA processing board, mainly providing video input for the FPGA processing board.

[0053] like Figure 2 As shown, the FPGA processing board integrates multiple modules and input / output interfaces. During operation, it first determines the input status of the video image source, selectively inputting video sources in descending order of priority: camera input, HDMI projection, and test images. Camera images are transmitted to the FPGA processing board via fiber optic interface. The fiber optic transmission bandwidth is 5Gbps, enabling bidirectional data transmission and allowing remote configuration of camera parameters while receiving video feeds. HDMI projection from the computer device is transmitted to the FPGA processing board via HDMI input interface connected by an HDMI cable. Test images are connected via internal wiring within the FPGA, requiring no external physical connection. The image size of the aforementioned video image sources is 4096×2160, with a frame rate of approximately 100 frames per second.

[0054] After the FPGA completes the selection of the image input source, it implements a hardware dehazing algorithm for the image. The clear image after dehazing is then output and displayed via HDMI and sent to the next-level algorithm module. The license plate positioning module implements the pre-selection of the license plate position on the FPGA and uploads the position coordinates and the image of the area centered on the coordinates to the computer via the network. The software then completes the subsequent license plate character recognition and user interface operations.

[0055] The FPGA processing board, connected to the video image source, is used for real-time dehazing of the input image or video image. To improve recognition speed, this invention employs a dark channel image dehazing algorithm and combines it with targeted algorithm improvements designed using the FPGA processing board. The steps include:

[0056] (1) Extract the dark channel image from the input video image.

[0057] Each pixel of the input image is split into three 8-bit data points, representing the R, G, and B channels respectively. A comparator is used to compare the three data points and find the minimum value; this minimum value is then used as the pixel value of the dark channel image and output according to the original image timing.

[0058] (2) Estimate the transmittance value t(x) and atmospheric light value A based on the dark channel image.

[0059] (2.1) Transmittance value t(x)

[0060] The input video image is filtered using a dual-scale minimum difference filtering algorithm, and a 9×9 filtering window is constructed using BRAM resources. This filtering window is then used as a base template, and 5×5 sub-windows are used to traverse the base template.

[0061] The traversal method primarily utilizes the DSP resources within the FPGA to perform basic mathematical operations. First, the average pixel value of each sub-region is calculated. Then, the difference between the average value and the center pixel value is calculated. Next, the sub-region with the smallest difference is selected for minimum value filtering to obtain a refined dark channel image. This refined dark channel is used to calculate the transmittance, which can be obtained based on a general fog imaging model.

[0062] (2.2) Estimating atmospheric light value A

[0063] The pixel values ​​in the dark channel are accumulated, and the accumulated results are stored in various addresses in the BRAM. Each address corresponds to a histogram statistical result of a grayscale value. By statistically analyzing the histogram, the average value of approximately the top 0.1% of pixels in the dark channel image of the previous frame is used as a threshold, and the data from the previous frame and the current frame are cached in DDR units. Then, the atmospheric light value A is estimated by linearly adjusting the three channels of the current frame based on the difference between the pixel values ​​and the threshold.

[0064] (3) Substitute the estimated parameters into the fog imaging model to obtain the defogging image.

[0065] The FPGA processing board searches the dehazed image to determine the coordinates of the license plate within the entire image frame.

[0066] The workflow of license plate location algorithm is as follows: Figure 3 As shown, the implementation is as follows:

[0067] First, the RGB image processed by the dehazing algorithm is converted to the YCbCr color space.

[0068] Secondly, by utilizing the parallel processing capabilities of the FPGA, grayscale images can be extracted from both the Cb and Cr components simultaneously, which doubles the processing speed compared to software implementation.

[0069] Next, the Cr component grayscale image is traversed pixel by pixel according to row and field time sequences, with two registers used to store the target pixel range value and the pixel mean. The maximum grayscale value and 0.85 times that grayscale value are used as the range boundary value. Since the range containing the target information includes the minimum pixel value, this range is defined as the target pixel range value. After the traversal, a background suppression design is adopted to perform nonlinear compensation on the minimum value obtained from the Cr component grayscale image. The difference between the pixel mean and the target pixel range value is used as the threshold C. The threshold C is compared with the Cr component to obtain the processed Cr component grayscale image, which is then stored in DDR. If the Cr component is greater than or equal to the threshold C, the threshold C is used as the processed Cr component grayscale image; otherwise, the Cr component itself is used as the processed Cr component grayscale image. Then, the Cb component grayscale image and the processed Cr component grayscale image in the same frame are read from the DDR cache unit, and the DSP resources are used to perform interpolation operation pixel by pixel to obtain a grayscale image with enhanced target information, thereby suppressing background interference caused by the car body color and achieving the purpose of suppressing unnecessary noise.

[0070] Subsequently, the grayscale image is binarized and segmented using an adaptive binarization module, followed by morphological processing. Specifically, the binarized grayscale image is first dilated to connect the holes in the license plate region, and a row buffer structure is constructed using BRAM. After completing the three rows of data buffering, the image is read into a register array of the required scale to construct convolutional units. The convolutional kernels constructed from the registers are of a fixed size of 9×9. Then, two erosion processes are performed, with erosion kernels of 9×9 and 3×3 sizes, respectively, to restore the geometric dimensions of the target region and filter out speckle interference.

[0071] Next, the binarized image containing the license plate region is identified using an improved bounding box target detection algorithm. Specifically, a BRAM memory is first defined, and then the overlapping lines in the DDR readout image are accumulated line by line. The accumulated results are stored sequentially in the smallest memory unit within the BRAM address. During the search process, closed regions are marked, and the marking results are placed in a register, recording the area and coordinates of the two vertices of each closed region. At the end of the image search, the areas of all closed regions are compared in a comparator, and area range and aspect ratio are set as filtering conditions. These conditions reference the actual aspect ratio of the license plate, and the specific data is stored in on-chip ROM resources. The closed region with the highest probability is selected as the target license plate, and then the target license plate coordinates are smoothed with a smoothing step size of 50 pixels. The smoothing operation involves setting a fixed limit difference between two detected results, subtracting the two results using the DSP, and sending the subtraction result and the limit difference to the comparator. If the comparison result is within the set limit, the detected license plate coordinates are used; if it exceeds the limit, the limit difference is assigned to the coordinate result. Then, the RGB image of the corresponding frame is read from the DDR3 cache, and the license plate coordinates and a rectangular area bounded by the geometric center of the license plate area as the origin, using a Cartesian coordinate system as the reference coordinate system, and extended 370 and 300 pixels in the positive and negative X-axis directions respectively, are uploaded to the host computer software of the computer device via the network port. The transmission format uses a custom encoding method, using FPGA logic units to construct the Ethernet frame format and fill the data segment with valid data. After being sent to the MAC layer with the RGMII interface, the onboard PHY chip completes subsequent network communication, such as... Figure 4 .

[0072] After receiving the video stream data sent by the FPGA processing board via the Ethernet interface, the host computer software performs the following two tasks simultaneously:

[0073] 1. Display the received video feed in real time in the video display area of ​​the host computer software.

[0074] Second, after receiving the license plate coordinate information, the host computer software performs coordinate offset processing on the license plate coordinates because the license plate coordinates are different between the FPGA processing board and the reference coordinate system of the uploaded screen, so as to obtain the actual position information of the license plate coordinates in the uploaded video image.

[0075] After obtaining the license plate coordinates, the license plate image is binarized, and the following operations are performed: a rectangular structuring element is constructed, and the binarized image is sequentially eroded and dilated. The constructed structuring element is subtracted from the processed binarized image to obtain the image with redundant parts removed. Further character localization is performed by traversing every pixel of the license plate number image to obtain the target pixel count for each column. The target pixel count for each column is compared with a set threshold A to determine if the column contains a character. If not, the number of consecutive columns with a target pixel count greater than threshold A is compared with threshold B to update the character position information. If it contains a character, the character width is accumulated. When the character width no longer accumulates, the current character position localization ends, and the next character is located. This process is repeated until all eight characters on the license plate are located. Finally, template matching is used for character recognition. An affine transformation target image is created, the affine transformation matrix is ​​calculated, the loaded image is subjected to an affine transformation operation, and then compared with a character library. The recognized license plate characters are output to the license plate character display area of ​​the host computer software, such as... Figure 5 .

[0076] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. A hardware acceleration based real-time license plate recognition system, characterized in that, The system comprises an FPGA processing board and a license plate recognition module; The FPGA processing board uses a defogging module to perform hardware defogging processing on the image according to a video image input source, and transmits the defogged image to the license plate positioning module for license plate position preselection, including: performing color space conversion on the RGB image after defogging processing to convert to YCbCr color space; using FPGA parallel processing to extract a grayscale image from the Cb and Cr components; traversing the Cr component grayscale image pixel by pixel according to row field timing, setting two registers to store target pixel range values and pixel mean values, taking the difference between the pixel mean values and the target pixel range values as a threshold C, comparing the threshold C with the Cr component to obtain a processed Cr component grayscale image, and storing the processed Cr component grayscale image in a DDR; reading the Cb component grayscale image and the processed Cr component grayscale image under the same frame from the DDR cache unit, using DSP resources to perform pixel-by-pixel difference operation to obtain a target information enhanced grayscale image; performing binaryzation segmentation on the target information enhanced grayscale image, and then performing morphological processing, including performing inflation processing on the binaryzation segmented grayscale image, connecting the holes in the license plate region, and using a BRAM to build a row buffer structure; after completing three row data buffering, reading the image to a convolution unit built in a register array to perform twice erosion, completing restoration of the target region geometric size and filtering of speckle-like interference; performing recognition on the binaryzation image, including: defining a BRAM memory, performing area accumulation on the lines with overlapping areas in the DDR readout image, storing the accumulation result in the smallest storage unit in the BRAM address in turn, marking the closed region in the search process, placing the marking result in a register, and recording the area of the closed region and the coordinate information of the two side vertices, comparing all the closed region areas in the comparator at the end of the image search, using the filter conditions pre-stored in the ROM to screen out the most probable closed region as the target license plate; performing smoothing processing on the target license plate coordinates, which is to set a fixed limit difference value between the two detected results, perform subtraction operation on the two results by DSP, and send the subtraction result and the limit difference value to the comparator, if the comparison result is within the set limit range, the coordinate result of the detected license plate is adopted; if the range is exceeded, the limit difference value is assigned to the coordinate result; The RGB image corresponding to the frame is read out from the DDR3 cache area, the license plate position coordinates and the region image centered on the coordinates are transmitted to the license plate recognition module in the host computer software through the network port, and the characters on the license plate are recognized through the license plate recognition module.

2. The system of claim 1, wherein, The FPGA processing board comprises one HDMI input interface, two optical fiber input / output interfaces, one HDMI output interface, one gigabit Ethernet output interface, one FPGA processing chip and four DDR3 particles; The HDMI input interface is used for connecting an HDMI line and inputting a computer device projection; The optical fiber input / output interface is connected to a camera end through an optical fiber line. ​ An HDMI output interface is configured to output the video image after the fog removal to a display to view the fog removal effect. An Ethernet output interface is configured to upload the position coordinates and the specified area image to an upper computer, and use a license plate recognition module in the upper computer to recognize the license plate characters. An FPGA processing chip is configured to complete the calculation required by the fog removal processing and the license plate position pre-selection. A DDR3 particle is configured to cache the image and the calculation data.

3. The system of claim 2, wherein, The specified area image is a rectangular area surrounded by extending X1 and Y1 pixels in the positive and negative directions of the X and Y axes respectively, with the geometric center of the located license plate area as the coordinate origin and a Cartesian coordinate system as the reference coordinate system. The set values of X1 and Y1 are related to the calculation capability of the FPGA processing chip.

4. The system of claim 1, wherein, The fog removal module includes the following processing: Each pixel of the input image is split into three 8-bit data, and a comparator is used to compare the three data, with the minimum value as the pixel value of the dark channel image of the input video image, and output according to the original image timing; The pixel value of the output dark channel image is used to build a 9x9 filter window as a basic template, and a 5x5 sub-window is used to traverse the basic template. The mean value of each sub-region pixel is calculated first, then the difference between the mean value and the center point pixel value is calculated, and then the sub-region with the minimum difference value is selected for minimum value filtering to obtain the processed dark channel image for calculating the transmittance; The dark channel pixel value is accumulated, and the accumulation result is stored in each address of the BRAM. Each address corresponds to a histogram statistical result of a gray value. The mean value of the top 0.1% pixels of the previous frame dark channel image is estimated as a threshold value by statistical histogram. The data of the previous frame and the current frame are cached in the DDR unit, and then the atmospheric light value is estimated by linear adjustment according to the difference between the pixel and the threshold value in the three channels of the current frame. Based on the transmittance and the atmospheric light value, the foggy image is obtained by using the foggy imaging model.

5. The system of claim 1, wherein, The morphological processing includes: The convolution unit is built by registers, and the convolution kernel size is fixed at 9x9; The erosion convolution kernel is 9x9 and 3x3 in size.

6. The system of claim 1, wherein, The license plate recognition module includes the following processing: Based on the accepted position coordinates, the license plate coordinates are processed for coordinate offset to obtain the actual position information; A rectangular structure element is constructed for the license plate image, and the binary image is sequentially eroded and expanded. The constructed structure element and the processed binary image are subtracted to obtain an image with redundant parts removed; For the image with redundant parts removed, each pixel point is traversed to obtain the target pixel number of each column of the image; Step S: Compare the target pixel number of each column with a set threshold value A to determine whether the column is a character; If it is not a character, compare the number of consecutive columns with a target pixel number greater than the set threshold value A with a set threshold value B to update the character position information; If it is a character, the character width is accumulated. When the character width is no longer accumulated, the current character position positioning is completed, and the next character is positioned. Return to step S until the positions of the eight characters on the license plate are all positioned. The character recognition is carried out by template matching, an affine transformation target image is created, an affine transformation matrix is calculated, an affine transformation operation is carried out on the loaded image, and the recognized license plate number is output by comparison with a character library.

Images