Image super-resolution reconstruction method and device for license plate recognition in scenic spots

By using image super-resolution reconstruction network in scenic spot license plate recognition, combined with multi-level calibration and feature fusion of spatial attention module, the problem of low image resolution of scenic spot license plates is solved, significantly improving the accuracy and noise resistance of license plate recognition .

CN114419612BActive Publication Date: 2024-11-08CHONGQING RUIYUN TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202210023720.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-10
Publication Date
2024-11-08
Estimated Expiration
2042-01-10

AI Technical Summary

Technical Problem

In modern intelligent transportation systems, the low resolution of license plate images in scenic spots leads to low accuracy of license plate recognition, especially due to the large area and complex terrain of scenic spots, the resolution of license plate images collected by cameras is usually low, which affects the accuracy of license plate recognition.

Method used

Using an image super-resolution reconstruction method, the network is constructed by training the image super-resolution reconstruction, using the pre-stage convolution layer, the post-stage convolution module, the multi-stage fusion module and the amplification module, combined with the LSA, GSA and MSA spatial attention module , super-resolution reconstruction of low-resolution license plate images, output high-resolution license plate images, and improve the accuracy of license plate recognition.

Benefits of technology

Through the multi-level calibration and feature fusion of the spatial attention module, the resolution and recognition accuracy of license plate images are significantly improved, the probability of misjudgment is reduced, and the noise resistance of the network is improved, which significantly improves the accuracy of license plate recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114419612B_ABST
    Figure CN114419612B_ABST
Patent Text Reader

Abstract

The present invention discloses an image super-resolution reconstruction method and device for license plate recognition in scenic spots. The reconstruction method includes model training, segmentation to obtain license plate images, and using super-resolution reconstruction network to reconstruct and output high-resolution license plate images. The image super-resolution reconstruction network includes a front-stage convolution layer, a rear-stage convolution module, a multi-stage fusion module, and an amplification module. The front-stage convolution layer is connected to multiple rear-stage convolution modules in sequence. The rear-stage convolution module simultaneously receives the feature map output by its upstream end and the primary feature map as input, and extracts and outputs the secondary feature map of the image. The present invention uses the LSA spatial attention module to perform spatial calibration on the feature map of a small field of view, and uses the GSA spatial attention module and the MSA spatial attention module to perform spatial calibration on the feature maps of all different sizes of fields of view at the same time, so that the information calibration is more targeted, and the model has the advantages of high efficiency in extracting edge features and strong anti-noise ability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular, relates to an image super-resolution reconstruction method and device for license plate recognition in scenic spots. Background Art

[0002] In modern intelligent transportation systems, license plate recognition technology based on computer vision has been widely adopted. Combined with cloud computing and big data technologies, it can realize automatic toll collection, vehicle positioning, crime prevention, etc. Accurate license plate recognition depends on clear license plate image input, but for many scenic spots, the area is large and the terrain is complex. The resolution of license plate images collected by cameras in scenic spots is usually low, resulting in low final recognition accuracy. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides an image super-resolution reconstruction method and device for license plate recognition in scenic spots, which improves the resolution of license plate images by super-resolution reconstruction, thereby improving the accuracy of license plate recognition.

[0004] In order to achieve the above object, the solution adopted by the present invention is: an image super-resolution reconstruction method for license plate recognition in scenic spots, comprising the following steps:

[0005] S1. Acquire an image super-resolution reconstruction network, acquire a training data set, and use the training data set to train the image super-resolution reconstruction network;

[0006] S2, obtaining a low-resolution license plate image, wherein the low-resolution license plate image is obtained by segmenting the scene image;

[0007] S3, obtaining the image super-resolution reconstruction network trained in step S1, inputting the low-resolution license plate image obtained in step S2 into the image super-resolution reconstruction network, and after super-resolution reconstruction, the image super-resolution reconstruction network outputs a high-resolution license plate image with a resolution greater than that of the low-resolution license plate image;

[0008] The image super-resolution reconstruction network includes a front-stage convolution layer, a back-stage convolution module, a multi-stage fusion module and an amplification module; the front-stage convolution layer is arranged at the front end of the image super-resolution reconstruction network, and the front-stage convolution layer is connected to a plurality of the back-stage convolution modules in sequence end to end, and the front-stage convolution layer is used to extract the primary feature map of the image. The back-stage convolution module simultaneously receives the feature map output by its upstream end and the primary feature map as input, extracts and outputs the secondary feature map of the image. For the first back-stage convolution module in the network, the feature maps input to its two input ends are both primary feature maps, and for the subsequent back-stage convolution modules, the feature map input to the left side comes from the output of the previous back-stage convolution module, and the feature map input to the intermediate GSA spatial attention module is the primary feature map.

[0009] The multi-level fusion module receives the secondary feature maps output by each of the post-convolution modules as input, and outputs a deep feature map after feature fusion; the amplification module receives the deep feature map as input, and then outputs the high-resolution license plate image;

[0010] The post-convolution module can be represented by the following mathematical model:

[0011] M1=σ1(f 13 (K n ))

[0012] M2=σ2(f 15 (K n ))

[0013] M3=f LSA ([M1,M2]) · [M1,M2]

[0014] M4=[M3,σ3(f 251 (M1)),σ4(f 252 (M2))]

[0015] M5=(f GSA (K0)·M4)+(f MSA (M4)

[0016] K n+1 =K n +σ5(f 31 (M5)

[0017] Among them, K0 represents the primary feature map, Kn and K0 are simultaneously used as the feature map of the input post-convolution module, and f 13 represents a convolution operation with a convolution kernel size of 3*3, f 15 、f 251 and f 252All represent convolution operations with a convolution kernel size of 5*5, σ1, σ2, σ3, σ4 and σ5 all represent nonlinear activation functions ReLU, f 31 represents a convolution operation with a convolution kernel size of 1*1, [·] represents the concatenation operation of the feature maps, and f LSA () represents the LSA spatial attention module, f LSA ([M1,M2]) represents the spatial calibration map output by the LSA spatial attention module after the [M1,M2] feature map is input into the LSA spatial attention module. The calibration map is multiplied by the feature map after the concatenation of M1 and M2, and weight parameters of different sizes are assigned to different positions in the feature map after the concatenation of M1 and M2. GSA () represents the GSA spatial attention module, f MSA () represents the MSA spatial attention module, and Kn+1 is the feature map output by the subsequent convolution module.

[0018] Furthermore, in step S1, when the image super-resolution reconstruction network is trained using the training data set, the loss function used is the L1 loss function.

[0019] Furthermore, the convolution kernel size of the previous convolution layer is 3*3.

[0020] Furthermore, the LSA spatial attention module can be represented by the following mathematical model:

[0021] JF=[LCaxP(A),LCenP(A),LCvaP(A),LCaxP(A)-LCvaP(A)]LA=δ L (f L (JF))

[0022] Wherein, A represents the feature map input to the LSA spatial attention module, LA represents the spatial calibration map output by the LSA spatial attention module, LCenP() represents the global average pooling operation on the feature map, LCaxP() represents the global maximum pooling operation on the feature map, LCvaP() represents the global variance pooling operation on the feature map (that is, the variance value in the channel direction), the directions of the global average pooling operation, the global maximum pooling operation and the global variance pooling operation are all in the channel direction, [·] represents the splicing operation of the calibration map, and f L Represents a convolution operation, whose convolution kernel size is 1*1, and δL represents the sigmoid activation function.

[0023] Furthermore, the GSA spatial attention module can be represented by the following mathematical model:

[0024] G=δ G (f G([GCenP(K0),GCaxP(K0)]))

[0025] Wherein, K0 represents the primary feature map, K0 is used as the feature map of the GSA spatial attention module as input, G represents the spatial calibration map output by the GSA spatial attention module, GCenP() represents the global average pooling operation on the feature map, GCaxP() represents the global maximum pooling operation on the feature map, and the directions of the global average pooling operation and the global maximum pooling operation are both in the channel direction, [·] represents the splicing operation on the feature maps, and f G Indicates the convolution operation with a convolution kernel size of 5*5, δ G Represents the sigmoid activation function.

[0026] Furthermore, the mathematical model of the MSA spatial attention module is:

[0027] MA=δ M (f M ([MCenP(M4),MCaxP(M4),MCmeP(M4)]))

[0028] Wherein, M4 represents the feature map input to the MSA spatial attention module, MA represents the spatial calibration map output by the MSA spatial attention module, MCenP() represents the global average pooling operation on the feature map, MCaxP() represents the global maximum pooling operation on the feature map, MCmeP() represents the global median pooling operation on the feature map (that is, finding the median in the channel direction), the directions of the global average pooling operation, the global maximum pooling operation and the global median pooling operation are all in the channel direction, [·] represents the splicing operation on the feature map in the channel direction, and f M Indicates the convolution operation with a convolution kernel size of 5*5, δ M Represents the sigmoid activation function.

[0029] The present invention also provides an image super-resolution reconstruction device for scenic area license plate recognition, comprising a processor and a memory, wherein the memory stores a computer program, and the processor loads the computer program to execute the image super-resolution reconstruction method for scenic area license plate recognition as described above.

[0030] The beneficial effects of the present invention are:

[0031] (1) In the post-convolution module, the convolution operation is first performed using the 3*3 convolution and 5*5 convolution with smaller field of view. The feature map of the small field of view is spatially calibrated through the LSA spatial attention module. Then, the 5*5 convolution is connected in series with the previous 3*3 convolution and 5*5 convolution respectively to form a large field of view equivalent to 7*7 and 9*9. The GSA spatial attention module and the MSA spatial attention module are used to simultaneously perform spatial calibration on the feature maps of all different sizes of field of view. In this way, the final calibration methods of the large field of view feature map and the small field of view feature map are different. The two calibrations are coordinated to make the calibration of different information more targeted.

[0032] (2) The study found that the GSA spatial attention module and the MSA spatial attention module provided by the present invention have a good calibration effect on the large field of view feature map, and the GSA spatial attention module and the MSA spatial attention module cooperate with the LSA spatial attention module to also have efficient modulation calibration capabilities for the small field of view feature map;

[0033] (3) Since the ultimate goal is to classify and identify license plate information, what needs to be obtained is Chinese characters, letters, and numbers. Different from other super-resolution reconstruction goals, clearly reconstructing the edge information of the font is important for improving the subsequent license plate recognition accuracy. Therefore, median pooling is set in the MSA spatial attention module, so that the network focuses on learning to extract edge features and improve the final license plate recognition accuracy.

[0034] (4) The experiment also shows that by inputting the primary feature map into the GSA spatial attention module and adding the feature map modulated by the GSA spatial attention module and the feature map modulated by the MSA spatial attention module, the network's anti-noise ability is significantly improved, which can also reduce the probability of misjudgment of license plate recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the overall structure of the image super-resolution reconstruction network of the present invention;

[0036] Figure 2 for Figure 1 Schematic diagram of the internal structure of the latter convolution module in the image super-resolution reconstruction network shown;

[0037] Figure 3 for Figure 2 Schematic diagram of the internal structure of the LSA spatial attention module in the subsequent convolution module;

[0038] Figure 4 for Figure 2 Schematic diagram of the internal structure of the GSA spatial attention module in the subsequent convolution module;

[0039] Figure 5 for Figure 2 Schematic diagram of the internal structure of the MSA spatial attention module in the subsequent convolution module;

[0040] In the attached figure:

[0041] 1- low-resolution license plate image, 2- front-end convolution layer, 3- back-end convolution module, 31- LSA spatial attention module, 32- GSA spatial attention module, 33- MSA spatial attention module, 34- residual connection, 4- multi-level fusion module, 5- magnification module, 6- high-resolution license plate image. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings:

[0043] Embodiment 1:

[0044] In this embodiment, the overall architecture of the super-resolution reconstruction network is as follows: Figure 1 As shown in the figure, the step size of all convolution operations in the model is 1, and the length and width of the feature map remain unchanged before and after convolution. During the training process, the loss function is the L1 loss function, and the learning rate is fixed. The model is first pre-trained using a public dataset, and then further trained using the license plate image dataset.

[0045] The convolution kernel size of the front-stage convolution layer 2 is 3*3. After the low-resolution license plate image 1 passes through the front-stage convolution layer 2, the output is a primary feature map with 48 channels. In order to strike a balance between model performance and complexity, the back-stage convolution module 3 is set to six. The internal structure of the back-stage convolution module 3 is as follows: Figure 2 As shown. In order to avoid feature disappearance and gradient explosion during network training, a residual connection 34 is set in the post-convolution module 3. When the feature map is input into the post-convolution module 3, the front-end 3*3 convolution and 5*5 convolution extract small field of view features, and both output feature maps with 48 channels. After concatenation and calibration by the LSA spatial attention module 31, a feature map M3 with 96 channels is obtained. On the other hand, the feature maps output by the front-end 3*3 convolution and 5*5 convolution are also subjected to the subsequent 5*5 convolution operation to extract features on a larger field of view. Figure 1 The feature map M4 with 192 channels is concatenated together. After calibration by the GSA spatial attention module 32 and the MSA spatial attention module 33, the number of channels is reduced to 48 using a 1*1 convolutional layer.

[0046] In the LSA spatial attention module 31, global maximum pooling, global variance pooling and global average pooling operations are performed on each channel of the feature map. During the experiment, the inventor found that adding the calibration map after global maximum pooling and the calibration map after global variance pooling as a difference branch, the final calibration map is combined with the GSA spatial attention module 32 and the MSA spatial attention module 33, which has a better calibration effect. The operation process inside the GSA spatial attention module 32 and the MSA spatial attention module 33 is similar to that of the LSA spatial attention module 31.

[0047] The multi-level fusion module 4 and the amplification module 5 are both implemented using existing technologies. Specifically, in this embodiment, the multi-level fusion module 4 uses a convolution layer with a convolution kernel size of 1*1, and the number of channels of the input feature map is 6*48. After feature fusion, the number of channels of the output feature map is 48. The amplification module 5 includes two 3*3 convolution layers and a pixelshuffle layer. The pixelshuffle layer is arranged in the middle of the two 3*3 convolution layers. The amplification module 5 finally outputs a high-resolution license plate image 6 with a number of channels of 3.

[0048] Using exactly the same training and testing process (the data set and other conditions are also exactly the same), the present invention is compared with the existing advanced networks SAN (from the article Second-order Attention Network for Single Image Super-Resolution) and CSAR (from the article Channel-wise and Spatial Feature Modulation Networkfor Single Image Super-Resolution) in an experiment, and the results are shown in the following table:

[0049] Model Magnification PSNR SSIM Recognition rate SAN 4 26.53 0.8013 83.59% CSAR 4 26.78 0.8054 86.72% Example 1 4 27.15 0.8096 92.44%

[0050] The classification recognition network used in the test process is the same trained network. From the above results, it can be seen that the method provided by the present invention has obvious progress compared with the prior art.

[0051] Embodiment 2:

[0052] In order to prove the efficiency of the overall network architecture and the structure of the post-convolution module 3 provided by the present invention, the internal structure of the LSA spatial attention module 31 and the internal structure of the MSA spatial attention module 33 are changed to Figure 4 The structure of the GSA spatial attention module 32 is shown. The same experimental method and experimental process as in Example 1 are used, and the comparison results are shown in the following table:

[0053] Model Magnification PSNR SSIM Recognition rate Example 1 4 27.15 0.8096 92.44% Example 2 4 26.91 0.8079 89.23%

[0054] It can be seen from the results that when the spatial attention modules in the post-stage convolution module 3 adopt different internal structures, the image super-resolution reconstruction network still has relatively excellent super-resolution reconstruction capabilities, indicating that the overall network architecture and the post-stage convolution module 3 structure provided by the present invention have better feature extraction capabilities compared with the prior art.

[0055] The above-mentioned embodiments only express the specific implementation of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. An image super-resolution reconstruction method for license plate recognition in scenic spots, characterized by: The following steps are involved: S1. Acquire an image super-resolution reconstruction network, acquire a training data set, and use the training data set to train the image super-resolution reconstruction network; S2, obtaining a low-resolution license plate image, wherein the low-resolution license plate image is obtained by segmenting the scene image; S3, obtaining the image super-resolution reconstruction network trained in step S1, inputting the low-resolution license plate image obtained in step S2 into the image super-resolution reconstruction network, and after super-resolution reconstruction, the image super-resolution reconstruction network outputs a high-resolution license plate image with a resolution greater than that of the low-resolution license plate image; The image super-resolution reconstruction network includes a front-stage convolution layer, a rear-stage convolution module, a multi-level fusion module and an amplification module; the front-stage convolution layer is arranged at the front end of the image super-resolution reconstruction network, and the front-stage convolution layer is connected to a plurality of the rear-stage convolution modules in sequence end to end, and the front-stage convolution layer is used to extract the primary feature map of the image; the rear-stage convolution module simultaneously receives the feature map output by its upstream end and the primary feature map as input, extracts and outputs the secondary feature map of the image; the multi-level fusion module receives the secondary feature map output by each of the rear-stage convolution modules as input, and outputs a deep feature map after feature fusion; the amplification module receives the deep feature map as input, and then outputs the high-resolution license plate image; The post-convolution module can be represented by the following mathematical model: M1=σ1(f 13 (K n )) M2=σ2(f 15 (K n )) <h2 style=";text-align:left;direction:ltr">M3=f<h2 style=";text-align:left;direction:ltr"> LSA <h2 style=";text-align:left;direction:ltr"> ([M1,M2])·[M1,M2] M4=[M3,σ3(f 251 (M1)),σ4(f 252 (M2))] M5=(f GSA (K0)·M4)+(f MSA (M4)·M4) K n+1 =K n +σ5(f 31 (M5)) Among them, K0 represents the primary feature map, Kn and K0 are simultaneously used as the feature map of the input post-convolution module, and f 13 represents a convolution operation with a convolution kernel size of 3*3, f 15 、f 251 and f 252 All represent convolution operations with a convolution kernel size of 5*5, σ1, σ2, σ3, σ4 and σ5 all represent nonlinear activation functions ReLU, f 31 represents a convolution operation with a convolution kernel size of 1*1, [·] represents the concatenation operation of the feature maps, and f LSA () represents the LSA spatial attention module, f GSA () represents the GSA spatial attention module, f MSA () represents the MSA spatial attention module, and Kn+1 is the feature map output by the latter convolution module; The LSA spatial attention module can be represented by the following mathematical model: JF=[LCaxP(A),LCenP(A),LCvaP(A),LCaxP(A)-LCvaP(A)]LA=δ L (f L (JF)) Wherein, A represents the feature map input to the LSA spatial attention module, LA represents the spatial calibration map output by the LSA spatial attention module, LCenP() represents the global average pooling operation on the feature map, LCaxP() represents the global maximum pooling operation on the feature map, LCvaP() represents the global variance pooling operation on the feature map, the directions of the global average pooling operation, the global maximum pooling operation and the global variance pooling operation are all in the channel direction, [·] represents the splicing operation of the calibration map, and f L Represents the convolution operation, the convolution kernel size is 1*1, and δL represents the sigmoid activation function; The GSA spatial attention module can be represented by the following mathematical model: G=δ G (f G ([GCenP(K0),GCaxP(K0)])) Wherein, K0 represents the primary feature map, K0 is used as the feature map of the GSA spatial attention module as input, G represents the spatial calibration map output by the GSA spatial attention module, GCenP() represents the global average pooling operation on the feature map, GCaxP() represents the global maximum pooling operation on the feature map, and the directions of the global average pooling operation and the global maximum pooling operation are both in the channel direction, [·] represents the splicing operation on the feature maps, and f G Indicates the convolution operation with a convolution kernel size of 5*5, δ G Represents the sigmoid activation function; The mathematical model of the MSA spatial attention module is: MA=δ M (f M ([MCenP(M4),MCaxP(M4),MCmeP(M4)])) Wherein, M4 represents the feature map input to the MSA spatial attention module, MA represents the spatial calibration map output by the MSA spatial attention module, MCenP() represents the global average pooling operation on the feature map, MCaxP() represents the global maximum pooling operation on the feature map, MCmeP() represents the global median pooling operation on the feature map, the directions of the global average pooling operation, the global maximum pooling operation and the global median pooling operation are all in the channel direction, [·] represents the splicing operation on the feature map in the channel direction, and f M Indicates the convolution operation with a convolution kernel size of 5*5, δ M Represents the sigmoid activation function.

2. The image super-resolution reconstruction method for license plate recognition in scenic spots according to claim 1 is characterized by: In step S1, when the image super-resolution reconstruction network is trained using the training data set, the loss function used is the L1 loss function.

3. The image super-resolution reconstruction method for license plate recognition in scenic spots according to claim 1 is characterized by: The convolution kernel size of the previous convolution layer is 3*3.

4. An image super-resolution reconstruction device for license plate recognition in scenic spots, comprising a processor and a memory, wherein the memory stores a computer program, and is characterized in that: The processor is used to execute the image super-resolution reconstruction method for scenic area license plate recognition as described in any one of claims 1 to 3 by loading the computer program.

Citation Information

Cited By

  • Low-resolution license plate recognition method based on multi-scale fusion attention

    CN121861638A