A fuzzy license plate recognition method based on event camera
Through the combination of event cameras and multi-input deep neural networks, the problem of license plate recognition in harsh environments is solved, the accuracy of fuzzy license plate recognition is improved, and it is adapted to low-illumination and high-speed motion environments, providing new recognition ideas.
Patent Information
- Application Number
- CN202210530060.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-16
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-05-16
AI Technical Summary
The prior art cannot accurately identify fuzzy license plate characters when the lighting conditions are poor or the vehicle is driving at high speed, and traditional optical cameras have poor imaging quality and are severely blurred dynamically, making it difficult to identify license plates in harsh environments.
Use event cameras to capture license plates, obtain traditional optical images and event stream data, and fuse traditional optical images with multiple event frame images through multi-input deep neural networks, build detection modules and identification modules, and output bounding box position information and recognition results of license plates , utilizes the high dynamic range and low latency characteristics of event cameras to adapt to harsh environments.
It improves the accuracy of fuzzy license plate recognition, can effectively identify license plates in low illumination and high-speed motion environments, provides recognition capabilities in extreme environments, and provides new ideas for intelligent driving and engineering applications.
Smart Images

Figure CN114998879B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer image processing, and more specifically, relates to a fuzzy license plate recognition method based on an event camera. Background Art
[0002] In the field of computer image detection and license plate recognition, deep neural networks can distinguish different license plate characters after good training and show excellent performance. However, most deep neural networks can only recognize license plate characters based on good lighting conditions and clear optical images. They cannot accurately recognize license plate characters in special lighting conditions or when the vehicle is driving at high speed and the images taken by the optical camera are blurred. Summary of the invention
[0003] The present invention provides a fuzzy license plate recognition method based on an event camera to solve the problem in the prior art that license plate characters cannot be accurately recognized under poor lighting conditions or when a vehicle is traveling at high speed.
[0004] The present invention provides a fuzzy license plate recognition method based on an event camera. The method comprises the following steps: photographing a license plate with an event camera to obtain a traditional optical image and event stream data corresponding to the license plate; preprocessing the event stream data to obtain a plurality of event frame images having the same resolution as the traditional optical image; constructing a multi-input deep neural network, wherein the multi-input deep neural network is divided into a detection module and a recognition module; fusing the traditional optical image and the plurality of event frame images and inputting them into the detection module, which outputs the bounding box position information of the license plate; and extracting feature information based on the bounding box position information by the recognition module, and obtaining a license plate recognition result based on the feature information.
[0005] Preferably, when an event camera is used to photograph a license plate, a time threshold for photographing is set according to an event point that is excited by the event camera, and event stream data within the time threshold is obtained; the event stream data includes the X coordinate, Y coordinate, polarity and timestamp of each event point; when preprocessing the event stream data, the event stream data is equally divided into multiple groups according to the timestamp within the time threshold, each group of event points is superimposed according to the coordinates, the background is set to white, and the superimposed event points are set to different brightness according to the polarity to obtain multiple event frame images.
[0006] Preferably, the detection module includes multiple convolutional layers and a boundary predictor; different horizontal feature maps are extracted from the input image through multiple convolutional layers, and the horizontal feature map of the last convolutional layer is output to the boundary predictor; the boundary predictor is composed of multiple fully connected layers of the same level, and the boundary predictor outputs the bounding box position information of the license plate; the recognition module includes multiple ROI pooling layers and a classification component; the extracted multiple horizontal feature maps are converted into multiple feature maps with different sizes through multiple ROI pooling layers, and the feature map obtained after merging the multiple feature maps is input to the classification component, the classification component includes multiple classifiers, and the classification component outputs the license plate recognition result.
[0007] Preferably, in the training of the multi-input deep neural network, the localization loss with respect to the detection performance is expressed as:
[0008]
[0009] Among them, L loc is the positioning loss function, pb represents the predicted box, gb represents the ground truth box, and smooth L1 is the smooth L1 loss function, N is the size of a small batch in training, cx is the x-coordinate of the center point of the license plate bounding box, cy is the y-coordinate of the center point of the license plate bounding box, w is the width of the license plate bounding box, and h is the height of the license plate bounding box;
[0010]
[0011] Among them, b x is the x coordinate of the center point of the bounding box, b y is the y coordinate of the center point of the bounding box, b w is the width of the bounding box, b h is the height of the bounding box; W is the width of the input image, and H is the height of the input image;
[0012] In the training of the multi-input deep neural network, the classification loss with respect to recognition performance is expressed as:
[0013]
[0014] Among them, L cls is the classification loss function, pn represents the predicted value of the license plate character, gn represents the true value of the license plate character; k represents the number of digits of the license plate character, pn i is the predicted value of the i-th license plate character, gn i is the true value of the i-th license plate character, [] indicates rounding operation; nc i is the total number of floating point numbers, each floating point number represents the possibility of belonging to a specific character class, and the classification loss is the cross loss;
[0015] The unified loss function in the multi-input deep neural network training is expressed as:
[0016]
[0017] Among them, L is the unified loss function.
[0018] Preferably, the detection module includes 10 convolutional layers and a boundary predictor, and the boundary predictor is composed of 3 fully connected layers of the same level; the recognition module includes 3 ROI pooling layers and a classification component, and the classification component includes 7 classifiers; a single channel is extracted from a traditional optical image, and merged with 10 event frame images into 11 channels, which are input into the detection module after normalization. The detection module extracts horizontal feature maps in the 2nd convolutional layer, the 4th convolutional layer and the 6th convolutional layer, and the 3 ROI pooling layers convert the extracted 3 horizontal feature maps into 3 feature maps with different sizes, which are merged and input into the classification component, and the 7 classifiers output the recognition results of the 7-bit license plate characters.
[0019] Preferably, one traditional optical image and ten event frame images are deformed to a resolution of 480×480, and the grayscale single-channel data of each image is extracted to obtain 11 480×480 two-dimensional arrays, which are stored as grayscale image pixel brightness information with a value range of [0,255], and then fused into a 11×480×480 three-dimensional array, with the traditional optical image data as the first item, and the three-dimensional array is normalized; the sizes of the three feature maps obtained after conversion by three ROI pooling layers are 8×16×64, 8×16×160, and 8×16×192, respectively, and the three feature maps are connected to a feature map of 8×16×416 for license plate character classification.
[0020] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:
[0021] In the present invention, an event camera is used to shoot the license plate to obtain a traditional optical image and event stream data, and then the event stream data is preprocessed to obtain multiple event frame images; the multi-input deep neural network constructed by the present invention is divided into a detection module and a recognition module, and the traditional optical image and multiple event frame images are fused and input into the detection module, and the bounding box position information of the license plate is output through the detection module, and the license plate recognition result is obtained through the recognition module. The present invention introduces the event camera into the field of license plate recognition, and utilizes the high dynamic range and low latency characteristics of the event camera to enable the license plate detection task to adapt to more severe environments, such as highways (vehicles move at high speeds, which usually leads to poor imaging quality of traditional optical cameras and dynamic blur), and the license plate is illuminated by the driving lights of the rear vehicle (strong exposure) when driving. At the same time, the present invention proposes a multi-input structure fusion deep neural network, which fuses the traditional optical camera image with multiple event frames of the event camera. The network can fuse multi-dimensional inputs and extract features, and its recognition effect is better than other existing deep neural networks for recognizing license plate characters. The present invention can effectively improve the accuracy of fuzzy license plate recognition, and can effectively handle situations where traditional optical cameras cannot recognize license plates in extreme environments such as low illumination and high-speed vehicle movement, providing new ideas for subsequent license plate recognition, intelligent driving, and engineering applications in extreme environments. In addition, the event camera only takes pictures when the light changes and the object moves, and can be combined with solar cells to enable license plate detection tasks to be carried out on unplanned roads. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 An overall framework diagram of a multi-input deep neural network in a fuzzy license plate recognition method based on an event camera provided by an embodiment of the present invention;
[0023] Figure 2 Schematic diagram of event camera event stream and frame compression;
[0024] Figure 3 This is the experimental effect diagram. DETAILED DESCRIPTION
[0025] Event cameras are different from traditional optical cameras, and their advantages include: event cameras do not use shutters to capture images like traditional optical cameras. Instead, each pixel within the event camera operates independently and asynchronously, reporting brightness changes when they occur and remaining silent otherwise. Each pixel stores a reference brightness level and constantly compares it to the current brightness level. If the brightness difference exceeds a threshold, the pixel resets its reference level and generates an event: a discrete data packet containing a pixel address and a timestamp. Events may also contain the polarity of brightness changes (increase or decrease), or instantaneous measurements of illumination levels, so event cameras output asynchronous event streams triggered by changes in scene illumination. Event cameras have microsecond temporal resolution, 120dB dynamic range, and less underexposure / overexposure and motion blur. This enables them to more accurately track the movement of objects and cameras (optical flow) and produce grayscale information.
[0026] The present invention utilizes an event camera in combination with an improved deep neural network to provide a fuzzy license plate recognition method to solve the current license plate recognition problems caused by low illumination and high-speed vehicle movement, and can improve the accuracy of fuzzy license plate recognition.
[0027] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0028] The present embodiment provides a fuzzy license plate recognition method based on an event camera, which uses an event camera to shoot the license plate to obtain a traditional optical image and event stream data corresponding to the license plate; pre-processes the event stream data to obtain multiple event frame images with the same resolution as the traditional optical image; constructs a multi-input deep neural network, which is divided into a detection module and a recognition module; the traditional optical image and the multiple event frame images are fused and input into the detection module, which outputs the bounding box position information of the license plate; the recognition module extracts feature information based on the bounding box position information, and obtains the license plate recognition result based on the feature information.
[0029] Among them, when an event camera is used to shoot the license plate, a shooting time threshold is set according to the event point excited by the event camera, and event stream data within the time threshold is obtained; the event stream data includes the X coordinate, Y coordinate, polarity and timestamp of each event point; when the event stream data is pre-processed, the event stream data is equally divided into multiple groups according to the timestamp within the time threshold, each group of event points is superimposed according to the coordinates, the background is set to white, and the superimposed event points are set to different brightness according to the polarity to obtain multiple event frame images.
[0030] See also Figure 1The detection module includes multiple convolutional layers and a boundary predictor; different horizontal feature maps are extracted from the input image through multiple convolutional layers, and the horizontal feature map of the last convolutional layer is output to the boundary predictor; the boundary predictor is composed of multiple fully connected layers of the same level, and the boundary predictor outputs the bounding box position information of the license plate. The recognition module includes multiple ROI pooling layers and a classification component; the multiple horizontal feature maps extracted are converted into multiple feature maps with different sizes through multiple ROI pooling layers, and the feature map obtained by merging multiple feature maps is input to the classification component, the classification component includes multiple classifiers, and the classification component outputs the license plate recognition result.
[0031] In the training of the multi-input deep neural network, the localization loss with respect to the detection performance is expressed as:
[0032]
[0033] Among them, L loc is the positioning loss function, pb represents the predicted box, gb represents the ground truth box, and smooth L1 is the smooth L1 loss function, N is the size of a small batch in training, cx is the x-coordinate of the center point of the license plate bounding box, cy is the y-coordinate of the center point of the license plate bounding box, w is the width of the license plate bounding box, and h is the height of the license plate bounding box.
[0034]
[0035] Among them, b x is the x coordinate of the center point of the bounding box, b y is the y coordinate of the center point of the bounding box, b w is the width of the bounding box, b h is the height of the bounding box; W is the width of the input image, and H is the height of the input image.
[0036] In the training of the multi-input deep neural network, the classification loss with respect to recognition performance is expressed as:
[0037]
[0038] Among them, L cls is the classification loss function, pn represents the predicted value of the license plate character, gn represents the true value of the license plate character; k represents the number of digits of the license plate character, pn i is the predicted value of the i-th license plate character, gn i is the true value of the i-th license plate character, [] indicates rounding operation; nc i is the total number of floating point numbers, each floating point number represents the possibility of belonging to a specific character class, and the classification loss is the cross loss;
[0039] The unified loss function in the multi-input deep neural network training is expressed as:
[0040]
[0041] Among them, L is the unified loss function.
[0042] In order to better understand the present invention, an example is given below assuming that the number of characters of the license plate to be recognized is 7 and the number of event frame images is 10.
[0043] The present invention provides a fuzzy license plate recognition method based on an event camera, comprising the following steps:
[0044] Step 1: Use an event camera to photograph the license plate to obtain the traditional optical image and event stream data corresponding to the license plate.
[0045] According to the event point excited by the event camera, the time threshold for each shooting is set, usually 0.6 seconds, and the event camera outputs the grayscale traditional optical image and the event stream data within the time threshold. The event stream data includes the two-dimensional X-coordinate and Y-coordinate of each event point, the polarity of the excitation and the timestamp.
[0046] Step 2: pre-process the event stream data and compress it into 10 event frame images according to time.
[0047] The obtained event stream data is divided into 10 groups according to the timestamp within the time threshold set in step 1, and each group of event points is superimposed according to the coordinates. The present invention proposes setting the background to white, setting the superimposed event points to different brightness according to polarity, and obtaining 10 grayscale event frame images with the same resolution as the traditional optical image obtained in step 1.
[0048] In order to obtain event frames, the event stream needs to be cut. Common cutting methods are cutting by the number of event points and cutting by time. In order to quickly perform license plate recognition, the present invention cuts the event stream by time. Figure 2 , the event stream data obtained in step 1 is divided into 10 groups according to the timestamp within the time threshold set in step 1, and each group of event points is superimposed according to the coordinates. Different from the prior art method of setting the background of the event frame to gray and setting the brightness according to the polarity of the event point for superposition, in order to improve the accuracy of recognizing license plate characters and remove redundant information, the present invention proposes to set the background to white, set the superimposed event points to different brightness according to the polarity, and form 10 grayscale event frame images with the same resolution as the traditional optical image obtained in step 1.
[0049] Step 3: Use a multi-input deep neural network to obtain the license plate recognition result.
[0050] See also Figure 2The multi-input deep neural network is divided into a detection module and a recognition module; the detection module includes 10 convolutional layers and a boundary predictor, and the boundary predictor is composed of 3 fully connected layers of the same level; the recognition module includes 3 ROI pooling layers and a classification component, and the classification component includes 7 classifiers.
[0051] Among them, the convolution layer is a convolution layer with a linear rectification function (ReLU) and batch normalization, and the ROI pooling layer is a maximum pooling (MaxPooling) layer with a dropout method (Dropout).
[0052] A single channel is extracted from one traditional optical image and merged with 10 event frame images into 11 channels, which are normalized and input into the detection module. The detection module extracts horizontal feature maps at the second convolution layer, the fourth convolution layer and the sixth convolution layer. The three ROI pooling layers convert the three extracted horizontal feature maps into three feature maps with different sizes, which are merged and input into the classification component. The seven classifiers output the recognition results of the 7-bit license plate characters.
[0053] Specifically, one traditional optical image and ten event frame images are deformed to a resolution of 480×480. The grayscale single-channel data of each image is extracted to obtain 11 480×480 two-dimensional arrays, which are stored as grayscale image pixel brightness information with a value range of [0,255]. They are then fused into a 11×480×480 three-dimensional array, in which the traditional optical image data is the first item. Finally, the three-dimensional array is normalized, that is, the data type of the three-dimensional array member values is converted to a normalized value range of [0,1].
[0054] Ten convolutional layers are used to extract different horizontal feature maps from the input license plate image. The detection module provides the feature map output of the last convolutional layer to three fully connected layers of the same level, which we name "box predictor", i.e., boundary predictor, for bounding box prediction.
[0055] The license plate bounding box position satisfies:
[0056]
[0057] Where cx is the x-coordinate of the center point of the license plate bounding box, cy is the y-coordinate of the center point of the license plate bounding box, w is the width of the license plate bounding box, and h is the height of the license plate bounding box; b x is the x coordinate of the center point of the bounding box, b y is the y coordinate of the center point of the bounding box, b w is the width of the bounding box, b his the height of the bounding box; W is the width of the input image, and H is the height of the input image.
[0058] Since the feature maps of different layers in the network have different receptive field sizes, and the feature maps from the lower layers can improve the quality of semantic segmentation, the lower layers can capture more details of the input object. The area of the license plate is very small relative to the entire image. After the detection module completes all the convolutional layer calculations, the boundary predictor will output the bounding box position (cx, cy, w, h). See Figure 1 , the recognition module extracts feature maps in a bounding box region of size (m×h)×(n×w) with p channels. The multi-input deep neural network extracts feature maps in the second, fourth, and sixth convolutional layers, with sizes of (122×h)×(122×w)×64, (63×h)×(63×w)×160, and (33×h)×(33×w)×192. In practical applications, extracting feature maps from higher convolutional layers slows down the recognition process and does little to improve recognition accuracy.
[0059] In multi-input deep neural network training, the localization loss (loc) for detection performance is defined as:
[0060]
[0061] Among them, L loc is the positioning loss function, pb represents the predicted box, gb represents the ground truth box, and smooth L1 is the smooth L1 loss function, and N is the size of a mini-batch in training.
[0062] The multi-input deep neural network first uses the detection module to predict the bounding box, and then uses the recognition module to extract the region of interest (ROI) from several generated feature maps according to the relative position of the bounding box in each feature map. After merging them to the same width and height (16×8), the merged feature map is provided to the subsequent classifier to finally obtain the license plate character.
[0063] Using a popular term in neural networks, “attention”, the detection module acts as the “attention” of this unified network, which tells the recognition module where to look, and the recognition module then extracts the ROI from the shared feature map and predicts the license plate characters.
[0064] After extracting the feature maps, the multi-input deep neural network uses the ROI pooling layer to convert each extracted feature into a feature map with a fixed spatial range of 8 × 16. Then, the three resized feature maps (8 × 16 × 64, 8 × 16 × 160, and 8 × 16 × 192, respectively) are concatenated into a feature map of size 8 × 16 × 416 for license plate character classification.
[0065] In multi-input deep neural network training, the classification loss (cls) for recognition performance is defined as:
[0066]
[0067] Among them, L cls is the classification loss function, pn i (1≤i≤7) is the predicted value of the 7 license plate characters, and the predicted value of each license plate character pn i Contains nc i Floating point numbers, each of which represents the probability of belonging to a specific character class, gn i (1≤i≤7) is the true value of the 7 license plate characters.
[0068] Combining the positioning loss and classification, we can get the unified loss function for model training:
[0069]
[0070] Among them, L is the unified loss function.
[0071] The present invention jointly optimizes the positioning loss and the classification loss, and the extracted feature information will have richer license plate feature information. Experiments show that jointly optimizing these two losses can improve detection and recognition performance.
[0072] In order to illustrate the effect of the present invention, an experimental verification is carried out below.
[0073] During the experiment, due to the lack of event frame license plate dataset based on event camera, the present invention adopts the ESIM method to simulate the existing Chinese license plate dataset CCPD, sets random path motion for each photo in the dataset in ROS, uses a simulated event camera to shoot, and outputs traditional optical grayscale images and event stream data with a resolution of 720×1160.
[0074] The analysis indicators used in this method to measure the accuracy of license plate recognition are: standard protocol intersection over union (IoU) and the number of correctly recognized license plate digits. License plate recognition is considered correct only when IoU is greater than 0.6 and all characters in the license plate digits are correctly recognized.
[0075] The experimental results are as follows Figure 3 As shown in Table 1, Figure 3Table 1 and Table 1 both include the original image and the corresponding recognition results of the multi-input deep neural network EMRP-net (the present invention), LPRnet (the license plate recognition network published by Intel in 2018), and RP-net (the license plate detection and recognition network published by the University of Science and Technology of China in 2019). It can be seen that only the present invention can correctly recognize the license plate under poor lighting conditions or high-speed driving of the vehicle, and other methods have recognition errors. It should be pointed out that LPRnet cannot identify the specific position of the license plate in the data set. This experiment provides the license plate position to LPRnet before performing license plate recognition. From the quantitative analysis and qualitative analysis of license plate recognition, it can be seen that the license plate recognition accuracy of the present invention under extreme environment dynamic blur has reached the industry-leading level.
[0076] Table 1 Experimental results of the license plate recognition method on the fuzzy license plate data set established by ESIM of the present invention
[0077]
[0078] Finally, it should be noted that the above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.
Claims
1. A fuzzy license plate recognition method based on event camera, characterized in that: An event camera is used to photograph a license plate to obtain a traditional optical image and event stream data corresponding to the license plate; the event stream data is preprocessed to obtain a plurality of event frame images having the same resolution as the traditional optical image; a multi-input deep neural network is constructed, and the multi-input deep neural network is divided into a detection module and a recognition module; The traditional optical image and a plurality of event frame images are fused and input into the detection module, and the detection module outputs the bounding box position information of the license plate; The recognition module extracts feature information based on the bounding box position information, and obtains a license plate recognition result based on the feature information; When an event camera is used to photograph a license plate, a shooting time threshold is set according to the event point triggered by the event camera, and event stream data within the time threshold is obtained; the event stream data includes the X coordinate, Y coordinate, polarity and timestamp of each event point; When preprocessing the event stream data, the event stream data is equally divided into multiple groups according to the timestamp within the time threshold, each group of event points is superimposed according to the coordinates, the background is set to white, and the superimposed event points are set to different brightness according to the polarity to obtain multiple event frame images.
2. The method for fuzzy license plate recognition based on event camera according to claim 1 is characterized in that: The detection module includes multiple convolutional layers and a boundary predictor; different horizontal feature maps are extracted from the input image through multiple convolutional layers, and the horizontal feature map of the last convolutional layer is output to the boundary predictor; the boundary predictor is composed of multiple fully connected layers of the same level, and the boundary predictor outputs the bounding box position information of the license plate; The recognition module includes multiple ROI pooling layers and a classification component; the multiple horizontal feature maps extracted are converted into multiple feature maps with different sizes through the multiple ROI pooling layers, and the feature maps obtained after merging the multiple feature maps are input into the classification component. The classification component includes multiple classifiers, and the classification component outputs the license plate recognition result.
3. The method for fuzzy license plate recognition based on event camera according to claim 1, characterized in that: In the training of the multi-input deep neural network, the localization loss with respect to the detection performance is expressed as: Among them, L loc is the positioning loss function, pb represents the predicted box, gb represents the ground truth box, and smooth L1 is the smooth L1 loss function, N is the size of a small batch in training, cx is the x-coordinate of the center point of the license plate bounding box, cy is the y-coordinate of the center point of the license plate bounding box, w is the width of the license plate bounding box, and h is the height of the license plate bounding box; Among them, b x is the x coordinate of the center point of the bounding box, b y is the y coordinate of the center point of the bounding box, b w is the width of the bounding box, b h is the height of the bounding box; W is the width of the input image, and H is the height of the input image; In the training of the multi-input deep neural network, the classification loss with respect to recognition performance is expressed as: Among them, L cls is the classification loss function, pn represents the predicted value of the license plate character, gn represents the true value of the license plate character; k represents the number of digits of the license plate character, pn i is the predicted value of the i-th license plate character, gn i is the true value of the i-th license plate character, [] indicates rounding operation; nc i is the total number of floating point numbers, each floating point number represents the possibility of belonging to a specific character class, and the classification loss is the cross loss; The unified loss function in the multi-input deep neural network training is expressed as: Among them, L is the unified loss function.
4. The method for fuzzy license plate recognition based on event camera according to claim 2 is characterized in that: The detection module includes 10 convolutional layers and a boundary predictor, and the boundary predictor is composed of 3 fully connected layers of the same level; the recognition module includes 3 ROI pooling layers and a classification component, and the classification component includes 7 classifiers; A single channel is extracted from one traditional optical image and merged with 10 event frame images into 11 channels, which are normalized and input into the detection module. The detection module extracts horizontal feature maps at the second convolution layer, the fourth convolution layer and the sixth convolution layer. The three ROI pooling layers convert the three extracted horizontal feature maps into three feature maps with different sizes, which are merged and input into the classification component. The seven classifiers output the recognition results of the 7-bit license plate characters.
5. The method for fuzzy license plate recognition based on event camera according to claim 4 is characterized in that: One traditional optical image and 10 event frame images are deformed to a resolution of 480×480. The grayscale single channel data of each image is extracted to obtain 11 480×480 two-dimensional arrays, which are stored as the brightness information of grayscale image pixels with a value range of [0,255]. They are then fused into a 11×480×480 three-dimensional array, with the traditional optical image data as the first item, and the three-dimensional array is normalized. The sizes of the three feature maps obtained after conversion by three ROI pooling layers are 8×16×64, 8×16×160, and 8×16×192 respectively. The three feature maps are connected to a feature map of size 8×16×416 for license plate character classification.
