License plate recognition method and device, electronic equipment and storage medium

Abstract

The invention provides a license plate recognition method and device, electronic equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a to-be-recognized license plate image; inputting the to-be-recognized license plate image into a shared feature layer module in the end-to-end license plate recognition model to obtain a shared feature map corresponding to the to-be-recognized license plate image output by the shared feature layer module; inputting the shared feature map into a license plate detection module in an end-to-end license plate recognition model to obtain position information which is output by the license plate detection module and is used for positioning a license plate area; based on the position information, a license plate area in the shared feature map is segmented, and a feature area matrix corresponding to the license plate area is obtained; and inputting the feature region matrix into a license plate recognition module in the end-to-end license plate recognition model to obtain a license plate recognition prediction result output by the license plate recognition module. A unified network structure can be used, and the license plate recognition efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a license plate recognition method, device, electronic equipment and storage medium. Background Art

[0002] As an important part of the development of smart city transportation, license plate recognition technology has provided great convenience for traffic management and safety. With the continuous increase in the number of vehicles, traffic safety issues have also received increasing attention, and the requirements for license plate recognition technology have become increasingly stringent, that is, under the premise of ensuring recognition accuracy, the requirements for recognition efficiency have also become increasingly stringent.

[0003] In the prior art, license plate recognition technology is generally based on multi-stage recognition, and the recognition stage mainly includes: license plate detection, license plate correction and license plate recognition, and the license plate recognition result is obtained by cascading multiple models. However, the existing license plate recognition technology is often a pipeline serial processing. When the image data is transmitted between the cascaded models, the data transmission structure from the GPU (Graphics Processing Unit) to the CPU (Central Processing Unit) is complex and the computational overhead is large, which greatly reduces the processing speed of the image data, thereby resulting in low efficiency of license plate recognition. Summary of the invention

[0004] The present invention provides a license plate recognition method, device, electronic equipment and storage medium, which are used to solve the defect of low license plate recognition efficiency in the prior art, use a unified network structure, and improve the license plate recognition efficiency.

[0005] The present invention provides a license plate recognition method, comprising:

[0006] Get the license plate image to be recognized;

[0007] Input the license plate image to be recognized into a shared feature layer module in an end-to-end license plate recognition model, and obtain a shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module;

[0008] Inputting the shared feature map into a license plate detection module in the end-to-end license plate recognition model to obtain position information output by the license plate detection module for locating the license plate area;

[0009] Based on the position information, segmenting the license plate area in the shared feature map to obtain a feature area matrix corresponding to the license plate area;

[0010] The feature region matrix is ​​input into a license plate recognition module in the end-to-end license plate recognition model to obtain a license plate recognition prediction result output by the license plate recognition module.

[0011] According to the license plate recognition method provided by the present invention, the inputting the shared feature map into the license plate detection module in the end-to-end license plate recognition model to obtain the position information of the license plate area output by the license plate detection module includes:

[0012] The shared feature map is input into the license plate detection module, and a convolution operation is performed on the shared feature map to obtain position information of at least four coordinate points output by the license plate detection module for locating the license plate area, wherein the at least four coordinate points constitute a rectangular frame corresponding to the license plate area.

[0013] According to the license plate recognition method provided by the present invention, the license plate detection module is trained based on the following steps:

[0014] Inputting the sample feature map into the initial license plate detection module, performing a first convolution operation on the sample feature map, and determining a confidence loss function corresponding to the first convolution operation;

[0015] Perform a second convolution operation on the sample feature map to determine a rotation box geometry loss function corresponding to the second convolution operation;

[0016] Perform a third convolution operation on the sample feature map to determine a rotation box angle loss function corresponding to the third convolution operation;

[0017] Performing a fourth convolution operation on the sample feature map, and determining a key point regression loss function corresponding to the fourth convolution operation;

[0018] Determine a license plate detection loss function based on the confidence loss function, the rotation box geometry loss function, the rotation box angle loss function and the key point regression loss function;

[0019] Based on the license plate detection loss function, the initial license plate detection module is trained until the initial license plate detection module converges to obtain a trained license plate detection module.

[0020] According to the license plate recognition method provided by the present invention, based on the position information, the license plate area in the shared feature map is segmented to obtain a feature area matrix corresponding to the license plate area, including:

[0021] S1. Taking any coordinate point among the at least four coordinate points as a starting point;

[0022] S2. Determine a first coordinate point distributed in the horizontal direction and a second coordinate point distributed in the vertical direction adjacent to the starting point, wherein the angle formed by the starting point, the first coordinate point and the second coordinate point is a right angle;

[0023] S3, based on the position information corresponding to the first coordinate point and the second coordinate point, segmenting the area with the same position information in the shared feature map to determine the feature area corresponding to the starting point;

[0024] S4, taking the first coordinate point as a new starting point, repeatedly executing steps S2 to S4, and stopping iteration when the first coordinate point and the second coordinate point do not exist, thereby obtaining at least two feature areas corresponding to the license plate area;

[0025] S5. Determine a feature region matrix corresponding to the license plate region based on at least two feature regions corresponding to the license plate region.

[0026] According to the license plate recognition method provided by the present invention, the inputting of the license plate image to be recognized into the shared feature layer module in the end-to-end license plate recognition model to obtain the shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module comprises:

[0027] Input the license plate image to be identified into the shared feature layer module, compress the license plate to be identified at least twice, and determine at least two first semantic feature graphs corresponding to the license plate to be identified, the number of which is the same as the number of compression times;

[0028] Upsampling the at least two first semantic feature maps to determine at least two second semantic feature maps corresponding to the license plate to be identified;

[0029] The minimum first semantic feature map and the at least two second semantic feature maps are fused to obtain a shared feature map corresponding to the license plate image to be identified output by the shared feature layer module.

[0030] According to the license plate recognition method provided by the present invention, the end-to-end license plate recognition model is trained based on the following steps:

[0031] Pre-training an initial shared feature layer module based on a first training data set;

[0032] Based on the deep learning optimizer and the second training data set, the initial license plate detection module and the pre-trained initial shared feature layer module are trained to obtain a trained shared feature layer module and a license plate detection module whose evaluation index is greater than a preset threshold;

[0033] Freeze the model parameters corresponding to the shared feature layer module, train the initial license plate recognition module based on the second training data set, and fine-tune the license plate detection module to obtain a trained license plate recognition module and a license plate detection module.

[0034] According to the license plate recognition method provided by the present invention, the initial license plate recognition module is trained based on the second training data set, and the license plate detection module is fine-tuned to obtain the trained license plate recognition module and license plate detection module, including:

[0035] Determine a license plate recognition loss function based on the second training data set using an aggregate cross entropy algorithm;

[0036] Determine a total license plate loss function by performing a weighted summation based on the license plate recognition loss function and the license plate detection loss function;

[0037] Based on the total license plate loss function, the initial license plate recognition module is trained, and the license plate detection module is fine-tuned to obtain a trained license plate recognition module and a license plate detection module.

[0038] The present invention also provides a license plate recognition device, comprising:

[0039] An acquisition unit, used for acquiring a license plate image to be recognized;

[0040] A sharing unit, used for inputting the license plate image to be recognized into a shared feature layer module in an end-to-end license plate recognition model, and obtaining a shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module;

[0041] A detection unit, used to input the shared feature map into a license plate detection module in the end-to-end license plate recognition model to obtain position information output by the license plate detection module for locating the license plate area;

[0042] A segmentation unit, configured to segment the license plate region in the shared feature map based on the position information to obtain a feature region matrix corresponding to the license plate region;

[0043] The recognition unit is used to input the feature region matrix into the license plate recognition module in the end-to-end license plate recognition model to obtain the license plate recognition prediction result output by the license plate recognition module.

[0044] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned license plate recognition methods is implemented.

[0045] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the license plate recognition method as described above is implemented.

[0046] The license plate recognition method, device, electronic device and storage medium provided by the present invention, after obtaining the license plate image to be recognized, first input the image to be recognized into the shared feature layer module in the end-to-end license plate recognition model, extract the shared feature map corresponding to the license plate image to be recognized, and input the shared feature map into the license plate detection module to determine the position information of the license plate area, after segmenting the license plate area in the shared feature map according to the position information of the license plate area, input the feature area matrix corresponding to the obtained license plate area into the license plate recognition module to obtain the license plate recognition prediction result, by using a unified end-to-end license plate recognition model, reduce the complexity of the network structure and the computational overhead, avoid the transmission delay of image data between each cascade model, and thus improve the license plate recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 is a flow chart of a license plate recognition method provided by an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of the structure of an end-to-end license plate recognition model provided by an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of a process for determining a shared feature graph provided by an embodiment of the present invention;

[0051] Figure 4 is a schematic diagram of segmenting a characteristic region corresponding to a license plate provided by an embodiment of the present invention;

[0052] Figure 5 is a schematic diagram of the structure of a license plate recognition device provided by an embodiment of the present invention;

[0053] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Combine the following Figure 1-Figure 4 A license plate recognition method provided by an embodiment of the present invention is described.

[0056] In view of the problem of low efficiency of license plate recognition in the prior art, an embodiment of the present invention provides a license plate recognition method. Figure 1 is a flow chart of a license plate recognition method provided by an embodiment of the present invention, such as Figure 1 As shown, the license plate recognition method includes:

[0057] Step 110: Obtain the license plate image to be recognized.

[0058] Optionally, the license plate image to be identified may be an image of the license plate to be identified that complies with relevant legal provisions and is obtained from a traffic management department, or may be captured by a camera with user authorization. The license plate image to be identified may include only the license plate area, or may include a partial vehicle image and a vehicle logo image, etc., which is not limited in the embodiment of the present invention.

[0059] Furthermore, before recognizing the license plate image, the constructed initial end-to-end license plate recognition model needs to be trained. Figure 2 is a schematic diagram of the structure of the initial end-to-end license plate recognition model provided by an embodiment of the present invention, such as Figure 2 As shown, the initial end-to-end license plate recognition model includes an initial shared feature layer module, an initial license plate detection module and an initial license plate recognition module. The end-to-end license plate recognition model is trained based on the following steps:

[0060] Pre-training an initial shared feature layer module based on a first training data set;

[0061] Based on the deep learning optimizer and the second training data set, the initial license plate detection module and the pre-trained initial shared feature layer module are trained to obtain a trained shared feature layer module and a license plate detection module whose evaluation index is greater than a preset threshold;

[0062] Freeze the model parameters corresponding to the shared feature layer module, train the initial license plate recognition module based on the second training data set, and fine-tune the license plate detection module to obtain a trained license plate recognition module and a license plate detection module.

[0063] Specifically, the initial shared feature layer module is pre-trained by the first training data set to obtain the initial weights corresponding to the shared feature layer. Afterwards, the second training data set and deep learning optimization are used, the learning rate is set to 0.01, the training rounds are 500 rounds, the pre-trained initial shared feature layer module and the initial license plate detection module are trained, and the evaluation index corresponding to the initial license plate detection module is determined. The evaluation index is used to characterize the detection accuracy of the initial license plate detection module. When the training rounds are reached and the evaluation index is greater than the preset threshold, the trained shared feature layer module and license plate detection module can be obtained. Afterwards, since the shared feature layer module has been trained, in order to avoid the subsequent training of the initial license plate recognition module affecting the model parameters of the initial license plate recognition module, at the same time, in order to improve the training efficiency, after the shared feature layer module is trained, the model parameters corresponding to the shared feature layer module need to be frozen, and the learning rate of the license plate detection module is set to be much smaller than that of the initial license plate recognition module. For example, the learning rate of the license plate detection module is set to 0.0001, and the learning rate of the license plate recognition branch is set to 0.01, so as to train the initial license plate recognition module through the second training data set, and fine-tune the license plate detection module, so as to obtain the trained license plate recognition module and license plate detection module. Based on the above method, a trained end-to-end license plate recognition model is obtained.

[0064] Optionally, the first training data set may be an Imagenet public data set. After determining the Imagenet public data set, data augmentation may be performed on the Imagenet public data set to improve the robustness of the initial shared feature layer. The data augmentation specifically includes: first, determining the longer side of each first sample image in the Imagenet public data set, and extending the longer side of each first sample image from 640 to 2560; second, randomly rotating each extended first sample image within an angle range, which may be [-10°, 10°]; then, keeping the width of each first sample image unchanged, and randomly scaling the height within a multiple range, which may be [0.8, 1.2]; finally, randomly cutting each first sample image along a fixed size from the above-mentioned transformation, which may be 640*640.

[0065] Optionally, the second training data set may be an open source data set of CCPD (Chinese City Parking Dataset), which includes about 250,000 independent license plate images, each of which includes different backgrounds, different shooting angles, different times, and different backlighting levels, and the samples are relatively rich. In addition, a data augmentation strategy may be implemented on CCPD to improve the robustness of the end-to-end license plate recognition model. The specific data augmentation strategy is the same as that of the first training data, and the embodiments of the present invention will not be described in detail here.

[0066] Optionally, the above-mentioned deep learning optimizer can be an Adam optimizer. In addition, the evaluation indicators corresponding to the above-mentioned license plate detection module may include recall rate and accuracy rate, etc. The recall rate is used to measure the ability of the license plate detection module to detect license plates. The higher the recall rate, the higher the probability that the license plate detection module can detect the license plate area, reducing the possibility of missed detection. The accuracy rate is used to measure the ability of the license plate detection module to correctly detect license plates. The higher the accuracy rate, the higher the probability that the license plate area detected by the license plate detection module is actually a license plate, reducing the possibility of false detection. The embodiment of the present invention does not limit the evaluation indicators.

[0067] Further, the initial license plate recognition module is trained based on the second training data set, and the license plate detection module is fine-tuned to obtain a trained license plate recognition module and a license plate detection module, including:

[0068] Determine a license plate recognition loss function based on the second training data set using an aggregate cross entropy algorithm;

[0069] Determine a total license plate loss function by performing a weighted summation based on the license plate recognition loss function and the license plate detection loss function;

[0070] Based on the total license plate loss function, the initial license plate recognition module is trained, and the license plate detection module is fine-tuned to obtain a trained license plate recognition module and a license plate detection module.

[0071] Specifically, the license plate recognition module adopts the CNN (Convolutional Neural Networks) + BLSTM (Bi-directional Long Short-Term Memory) encoding and decoding structure, and uses the aggregate cross entropy algorithm to calculate the license plate recognition loss function. The license plate recognition loss function is shown in formula (1). Formula (1) is:

[0072]

[0073] Among them, L rec represents the license plate recognition loss function, S represents the annotation information of the text line, L represents the length of S, S l represents the character at the first position, I represents the image corresponding to the text line, Q represents the second training data set, and ω represents the network weight.

[0074] At the same time, the license plate detection loss function is determined according to the second training data set. After that, the weighted sum of the license plate recognition loss function and the license plate detection loss function is calculated to determine the total license plate loss function. The total license plate loss function is shown in formula (2). Formula (2) is:

[0075] L t =L det +λL rec

[0076] Among them, L t Represents the total loss function of the license plate, L det represents the license plate detection loss function, and λ represents the weight of the license plate recognition loss function.

[0077] After determining the total license plate loss function, the initial license plate recognition module is trained according to the total license plate loss function, and the license plate detection module is fine-tuned to obtain the trained license plate recognition module and license plate detection module. Combined with the trained shared feature layer module, a trained end-to-end license plate recognition model can be obtained.

[0078] Step 120: input the license plate image to be recognized into a shared feature layer module in an end-to-end license plate recognition model, and obtain a shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module.

[0079] Specifically, in the end-to-end license plate recognition model, the shared feature layer module is the common part of the license plate detection module and the license plate recognition module. The shared feature layer module can use ResNet50 as the backbone feature extraction network. After the license plate image to be recognized is input into the shared feature layer module, the height and width of the license plate image to be recognized can be compressed multiple times, and then multiple upsampling and feature fusion can be performed to obtain a more detailed shared feature map.

[0080] Furthermore, Figure 3 is a schematic diagram of a process for determining a shared feature graph provided by an embodiment of the present invention, such as Figure 3 As shown, the step of inputting the license plate image to be recognized into the shared feature layer module in the end-to-end license plate recognition model to obtain the shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module includes:

[0081] Input the license plate image to be identified into the shared feature layer module, compress the license plate to be identified at least twice, and determine at least two first semantic feature graphs corresponding to the license plate to be identified, the number of which is the same as the number of compression times;

[0082] Upsampling the at least two first semantic feature maps to determine at least two second semantic feature maps corresponding to the license plate to be identified;

[0083] The minimum first semantic feature map and the at least two second semantic feature maps are fused to obtain a shared feature map corresponding to the license plate image to be identified output by the shared feature layer module.

[0084] For example, after the license plate image to be recognized is input into the shared feature layer module, the input license plate image to be recognized is continuously compressed in height and width, that is, the first semantic feature maps corresponding to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the license plate image to be recognized are obtained, and then the four first semantic feature maps obtained by the compression are used to construct a feature pyramid. Networks, FPN), that is, after the first semantic feature map corresponding to 1 / 32 of the license plate image to be identified is deconvolved, it is upsampled with the first semantic feature map corresponding to 1 / 16 of the license plate image to be identified to obtain the first second semantic feature map, then the first second semantic feature map is upsampled with the first semantic feature map corresponding to 1 / 8 of the license plate image to be identified to obtain the second second semantic feature map, then the second second semantic feature map is upsampled with the first semantic feature map corresponding to 1 / 4 of the license plate image to be identified to obtain the third second semantic feature map, and the height and width are restored by upsampling, and finally, the first semantic feature map corresponding to 1 / 32 of the license plate image to be identified and the three second semantic feature maps are feature fused to obtain the shared feature map of the license plate image to be identified output by the shared feature layer module, and the size of the shared feature map is 1 / 4 of the license plate image to be identified. Specifically, it includes the following steps:

[0085] 1) The license plate image to be recognized is cropped, and the size of the cropped license plate image to be recognized is 512*512, and the cropped license plate image to be recognized is zero-padding, that is, additional zero elements are added around the cropped license plate image to be recognized to adjust the size or shape of the license plate image to be recognized. After that, the license plate image to be recognized after the zero-padding operation passes through a convolution block with a stride of 2, which includes a convolution layer (Conv), a batch normalization (Batch Normalization, BN) layer and a rectified linear unit (Rectified Linear Unit, ReLU). The size of the license plate image to be recognized after the convolution block is 256*256*64, and then after the maximum pooling (Maxpooling) operation with a stride of 2 in the maximum pooling layer (Pool1), the license plate image to be recognized with a size of 128*128*64 is obtained. Afterwards, the license plate image to be recognized after maximum pooling passes through the first residual block, which includes a convolution block (Conv Block) and two identity blocks (Identity Block). The step size of the convolution in the convolution block is stride=1. Therefore, the height and width of the license plate image to be recognized after passing through the convolution block remain unchanged, that is, the size of the license plate image to be recognized output by the convolution block is 128*128*256. Afterwards, the license plate image to be recognized output by the convolution block remains unchanged after passing through two identity blocks, and the number of channels of the input license plate image to be recognized and the output first semantic feature map Res2 remains unchanged. Therefore, the size of the output first semantic feature map Res2 is still 128*128*256, and the size of the output first semantic feature map Res2 is 1 / 4 of the original license plate image to be recognized.

[0086] 2) After that, the feature layer Res2 enters the second residual block. The second residual block includes a convolution block and three identity blocks. The convolution step size stride=2 in the convolution block. Therefore, the height and width of the feature layer Res2 are compressed by 1 / 2 respectively. The size of the feature layer output by the convolution block is 64*64*512. After that, the height and width in the three identity blocks remain unchanged, and the number of channels of the input and output license plate images to be recognized remains unchanged. Therefore, the size of the first semantic feature map Res3 output by the three identity blocks is still 64*64*512, that is, the size of the first semantic feature map Res3 output by the second residual block is 64*64*512, and the size of the first semantic feature map Res3 is 1 / 8 of the original license plate image to be recognized.

[0087] 3) Afterwards, the first semantic feature map Res3 enters the third residual block. The third residual block includes a convolution block and five identity blocks. The convolution step size stride=2 in the convolution block. Therefore, the height and width of the first semantic feature map Res3 are compressed by 1 / 2 respectively. The size of the feature layer output by the convolution block is 32*32*1024. Afterwards, the height and width in the five identity blocks remain unchanged, and the number of channels of the input and output license plate images to be recognized remains unchanged. Therefore, the size of the first semantic feature map Res4 output by the five identity blocks is still 32*32*1024, that is, the size of the first semantic feature map Res4 output by the third residual block is 32*32*1024, and the size of the first semantic feature map Res4 is 1 / 16 of the original license plate image to be recognized.

[0088] 4) Afterwards, the first semantic feature map Res4 enters the fourth residual block. The fourth residual block includes a convolution block and two identity blocks. The convolution step size stride=2 in the convolution block. Therefore, the height and width of the first semantic feature map Res4 are compressed by 1 / 2 respectively. The size of the feature layer output by the convolution block is 16*16*2048. Afterwards, the height and width in the two identity blocks remain unchanged, and the number of channels of the input and output license plate images to be recognized remains unchanged. Therefore, the size of the first semantic feature map Res5 output by the two identity blocks is still 16*16*2048, that is, the size of the first semantic feature map Res5 output by the fourth residual block is 16*16*2048, and the size of the first semantic feature map Res5 is 1 / 32 of the original license plate image to be recognized.

[0089] 5) Afterwards, based on the first semantic feature map Res2, the first semantic feature map Res3, the first semantic feature map Res4 and the first semantic feature map Res5, a feature pyramid is constructed. First, the first semantic feature map Res5 is upsampled (Deconv), and the upsampling operation can be linear interpolation, etc. The size of the first upsampled image obtained after upsampling the first semantic feature map Res5 is 32*32*2048. Afterwards, the first upsampled image is concatenated with the first semantic feature map Res4 by the number of channels to perform feature fusion. At this time, the size of the first second semantic feature map obtained is 32*32*(2048+1024). Afterwards, the first second semantic feature map is input into a convolution block with a convolution kernel size of 1*1. The convolution block includes a convolution layer, a batch normalization layer, and a rectified linear unit. The convolution block is used to adjust the number of channels of the first second semantic feature map, and the size of the adjusted first second semantic feature map is 32*32*128. Afterwards, the adjusted first second semantic feature map is input into a convolution block with a convolution kernel size of 3*3. The convolution block is used to extract features and continue upsampling. The size of the second upsampled image obtained after upsampling is 64*64*128. Afterwards, the second upsampled image is connected with the first semantic feature map Res3 by the number of channels to perform feature fusion. At this time, the size of the second second semantic feature map obtained is 64*64*(128+512). Afterwards, the second second semantic feature map is input into a convolution block with a convolution kernel size of 1*1. The convolution block includes a convolution layer, a batch normalization layer, and a rectified linear unit. The convolution block is used to adjust the number of channels of the second second semantic feature map, and the size of the adjusted second second semantic feature map is 64*64*64. Afterwards, the adjusted second second semantic feature map is input into a convolution block with a convolution kernel size of 3*3. The convolution block is used to extract features and continue upsampling. The size of the third upsampled image obtained after upsampling is 128*128*64. Afterwards, the third upsampled image is connected to the first semantic feature map Res2 by the number of channels for feature fusion. At this time, the size of the third second semantic feature map obtained is 128*128*(64+256).Afterwards, the third second semantic feature map is input into a convolution block with a convolution kernel size of 1*1. The convolution block includes a convolution layer, a batch normalization layer and a rectified linear unit. The convolution block is used to adjust the number of channels of the third second semantic feature map, and the adjusted third second semantic feature map has a size of 128*128*32. Afterwards, the adjusted third second semantic feature map is further input into a convolution block with a convolution kernel size of 3*3. The convolution block is used to extract features and fuse the minimum first semantic feature map with the three second semantic feature maps to obtain a shared feature map. The size of the shared feature map is 128*128*32 and is 1 / 4 of the original license plate image to be identified.

[0090] Step 130: input the shared feature map into the license plate detection module in the end-to-end license plate recognition model to obtain position information output by the license plate detection module for locating the license plate area.

[0091] Specifically, after the shared feature map is determined, the shared feature map is input into the license plate detection module, and the license plate detection module is used to analyze the key points of the license plate area in the shared feature map to determine the position information of the license plate area.

[0092] Optionally, the position information may be the row and column number of each pixel in the shared feature map, or the coordinate information of the pixel constituting the license plate area in the coordinate system after the coordinate system is constructed in the shared feature map. This embodiment of the present invention is not limited to this.

[0093] Furthermore, the step of inputting the shared feature map into a license plate detection module in the end-to-end license plate recognition model to obtain location information of the license plate area output by the license plate detection module includes:

[0094] The shared feature map is input into the license plate detection module, and a convolution operation is performed on the shared feature map to obtain position information of at least four coordinate points output by the license plate detection module for locating the license plate area, wherein the at least four coordinate points constitute a rectangular frame corresponding to the license plate area.

[0095] Specifically, after the shared feature map is input into the license plate detection module, the shared feature map is convolved to analyze the key points in the shared feature map, thereby determining at least four coordinate points constituting the license plate area. When the number of coordinate points is four, the four coordinate points are respectively located at the four vertices of the rectangular frame of the license plate area. When the number of coordinate points is greater than four, the at least four coordinate points include the four vertices of the rectangular frame of the license plate area and the coordinate points on the longer sides of the rectangular frame.

[0096] For example, after a convolution operation is performed on the shared feature map in an embodiment of the present invention, position information of 18 coordinate points is obtained, wherein the 18 coordinate points include four vertices in a rectangular box and 14 coordinate points on the longer sides of the rectangular box, each longer side includes 7 coordinate points, and the coordinate points on the two longer sides are divided into 7 groups, and each group of coordinate points is on the same vertical axis.

[0097] Furthermore, before the license plate image to be identified is input into the license plate detection module, the constructed initial license plate detection module needs to be trained first, thereby obtaining a trained license plate detection module, which is trained based on the following steps:

[0098] Inputting the sample feature map into the initial license plate detection module, performing a first convolution operation on the sample feature map, and determining a confidence loss function corresponding to the first convolution operation;

[0099] Perform a second convolution operation on the sample feature map to determine a rotation box geometry loss function corresponding to the second convolution operation;

[0100] Perform a third convolution operation on the sample feature map to determine a rotation box angle loss function corresponding to the third convolution operation;

[0101] Performing a fourth convolution operation on the sample feature map, and determining a key point regression loss function corresponding to the fourth convolution operation;

[0102] Determine a license plate detection loss function based on the confidence loss function, the rotation box geometry loss function, the rotation box angle loss function and the key point regression loss function;

[0103] Based on the license plate detection loss function, the initial license plate detection module is trained until the initial license plate detection module converges to obtain a trained license plate detection module.

[0104] Specifically, after the second training data set is input into the shared feature layer module, multiple sample feature maps are obtained. For each sample feature map, four convolution operations are performed on the sample feature map in parallel. Among them:

[0105] 1) When the first convolution operation is performed on the sample feature map, the number of channels of the first convolution operation is 1, and the convolution kernel size is 1*1. Through the first convolution operation, each pixel in the sample feature map is classified to obtain a confidence map corresponding to the sample feature map. The size of the confidence map is 128*128*1. The confidence map includes the confidence corresponding to each pixel, that is, the probability value of each pixel belonging to the text area. Afterwards, the confidence loss function corresponding to the first convolution operation is calculated using class balanced cross entropy. The confidence loss function is shown in formula (3), which is:

[0106]

[0107] Among them, L s represents the confidence loss function, Y * Indicates the label value. The pixel value label of the license plate area in the confidence map is 1, and the pixel value label of the background area is 0. represents the predicted value, β represents the balance coefficient, and y* represents the label value corresponding to each pixel.

[0108] 2) When performing the second convolution operation on the sample feature map, the number of channels of the second convolution operation is 4, and the convolution kernel size is 1*1, the distance from each pixel in the sample feature map to the boundary of the prediction box is obtained, that is, the predicted value of the distance from each pixel to the upper boundary of the prediction box is determined The predicted value of the distance from each pixel to the right edge of the prediction box The predicted value of the distance from each pixel to the lower boundary of the prediction box The predicted value of the distance from each pixel to the left boundary of the prediction box After determining the four distance prediction values, the width of the overlapping area is determined according to formula (4), and the height of the overlapping area is determined according to formula (5). Formula (4) is:

[0109]

[0110] Where wi represents the width of the overlapping area, Indicates the distance label value from each pixel to the right boundary, Indicates the distance label value from each pixel to the left boundary.

[0111] Formula (5) is:

[0112]

[0113] Among them, h i Indicates the height of the overlapping area, Indicates the distance label value from each pixel to the upper boundary, Indicates the distance label value from each pixel to the lower boundary.

[0114] The overlapping area is the intersection of the prediction box and the label box, and both the prediction box and the label box are rotated boxes. Then, according to the intersection ratio of the prediction box and the label box, the rotation box geometric loss function can be determined. The rotation box geometric loss function is shown in formula (6), which is:

[0115]

[0116] Among them, L AABB represents the rotation box geometry loss function, R * Represents a label box, represents the prediction box, represents the overlapping area. The width of the overlapping area can be calculated based on formula (4). The height of the overlapping area can be calculated based on formula (5).

[0117] 3) When the third convolution operation is performed on the sample feature map, the number of channels of the third convolution operation is 1, and the convolution kernel size is 1*1, and a feature map of size 128*128*1 is obtained, which is used to represent the direction of the license plate area, that is, the predicted rotation angle.

[0118] Then, using formula (7), the rotation box angle loss function is calculated, formula (7) is:

[0119]

[0120] in, represents the rotation box angle loss function, Represents the rotation angle of the prediction box, θ * Indicates the rotation angle of the label frame.

[0121] 4) When the fourth convolution operation is performed on the sample feature map, the number of channels of the fourth convolution operation is 36, and the convolution kernel size is 1*1, the size after convolution is 128*128*36, and the position information of at least four key points is obtained, that is, the position information of at least four coordinate points of the rectangular frame used to locate the license plate area in the sample feature map. After determining the position information of at least four coordinate points, smoothed-L1 is used to construct the key point regression loss function, which is shown in formula (8), formula (8):

[0122]

[0123] Where: L q represents the key point regression loss function, C Q Represents the location information set corresponding to at least four coordinate points corresponding to the prediction box, CQ ={x 1 ,y 1 ,x 2 ,y 2 ,…x i ,y i …,x n ,y n}, n represents the number of coordinate points, c i Indicates the position information corresponding to the i-th coordinate point of the prediction box; Q * Represents the coordinate point set of the label box. Indicates all * A set of equivalent polygons, the order of the coordinate points of these polygons is different. for A coordinate point order in express The set of corresponding coordinate point sequences; express The position information corresponding to the i-th coordinate point in ; represents the maximum edge distance, and in, y=p i , p i Represents the position information of the i-th coordinate point, that is, p i ={x i ,y i}, after constructing the coordinate system based on the sample feature map, x i Indicates the x-axis coordinate corresponding to the i-th coordinate point, y i Indicates the y-axis coordinate corresponding to the i-th coordinate point; z = p (imodn)+1 , in,

[0124] 5) After determining the rotating box geometry loss function and the rotating box angle loss function, the rotating box regression loss function can be calculated according to the weighted sum of the rotating box geometry loss function and the rotating box angle loss function. The rotating box regression loss function is shown in formula (9), which is:

[0125] L g =L AABB +λ θ L θ

[0126] Among them, L g represents the rotation box regression loss function, λ θ Represents the weight corresponding to the rotation box angle loss function.

[0127] After determining the rotation box regression loss function, the weighted sum of the confidence loss function, the rotation box regression loss function and the key point regression loss function is calculated to obtain the license plate detection loss function. The license plate detection loss function is shown in formula (10):

[0128] L det =L s +λ g L g +λ q L q

[0129] Among them, L det represents the license plate detection loss function, λ g represents the weight of the rotation box regression loss function, λ q Represents the weight of the keypoint regression loss function.

[0130] 6) According to the license plate detection loss function, the initial license plate detection module is trained until the initial license plate detection module converges to obtain a trained license plate detection module.

[0131] Step 140: Based on the position information, segment the license plate area in the shared feature map to obtain a feature area matrix corresponding to the license plate area.

[0132] Specifically, after determining the position information, the license plate area is located in the shared feature map according to the position information, and the license plate area is segmented according to the position information to obtain multiple feature areas, each feature area corresponds to a character, which can be a number, a Chinese character, a letter or a space. Based on the multiple feature areas obtained by segmentation, a feature area matrix corresponding to the license plate area is constructed.

[0133] Optionally, a total of 70 characters may be included in the license plate area, including 10 numbers, 35 Chinese characters for provinces, 24 letters, and 1 space.

[0134] Furthermore, based on the position information, segmenting the license plate area in the shared feature map to obtain a feature area matrix corresponding to the license plate area includes:

[0135] S1. Taking any coordinate point among the at least four coordinate points as a starting point;

[0136] S2. Determine a first coordinate point distributed in the horizontal direction and a second coordinate point distributed in the vertical direction adjacent to the starting point, wherein the angle formed by the starting point, the first coordinate point and the second coordinate point is a right angle;

[0137] S3, based on the position information corresponding to the first coordinate point and the second coordinate point, segmenting the area with the same position information in the shared feature map to determine the feature area corresponding to the starting point;

[0138] S4, taking the first coordinate point as a new starting point, repeatedly executing steps S2 to S4, and stopping iteration when the first coordinate point and the second coordinate point do not exist, thereby obtaining at least two feature areas corresponding to the license plate area;

[0139] S5. Determine a feature region matrix corresponding to the license plate region based on at least two feature regions corresponding to the license plate region.

[0140] Specifically, after obtaining at least four coordinate points, any coordinate point is taken as the starting point, and the first coordinate point adjacent to the starting point in the horizontal direction and the second coordinate point adjacent to the starting point in the vertical direction are determined respectively, and a rectangular frame is formed by the starting point, the first coordinate point and the second coordinate point, or the rectangular frame is formed by the first coordinate point and the second coordinate point, that is, the width and height of the rectangular frame are determined according to the difference between the x-axis coordinates of the first coordinate point and the second coordinate point, and the difference between the y-axis coordinates, and then the area with the same position information is determined in the shared feature map, and the area is segmented to obtain the feature area formed by the starting point, the first coordinate point and the second coordinate point. After that, the first coordinate point is taken as the starting point, and the two adjacent coordinate points are continuously searched in the horizontal direction and the vertical direction to form the next rectangular frame, and the execution is repeated in sequence, and all coordinate points are traversed until the first coordinate point and the second coordinate point do not exist at the starting point at the same time, and the iteration is stopped, and at least two feature areas corresponding to the license plate area can be obtained. Based on the at least two feature areas corresponding to the license plate area, the pixel point information corresponding to each feature area is normalized and integrated to obtain at least two feature vectors of the same dimension, and then the feature matrix corresponding to the license plate area is formed.

[0141] For example, Figure 4 is a schematic diagram of segmenting the characteristic area corresponding to the license plate provided by an embodiment of the present invention, such as Figure 4As shown, 18 position points corresponding to the license plate area of ​​128*128 size are obtained, and the 18 coordinate points form a rectangular frame, including coordinate points A1, A9, B1 and B9 at the four vertices of the rectangular frame, coordinate points A2 to A8 on the first longer side, and coordinate points B2 to B8 on the second longer side. Afterwards, all coordinate points are traversed, starting from coordinate point A1 in the upper left corner, and the first coordinate point A2 adjacent to coordinate point A1 is searched in the horizontal direction, and the second coordinate point B1 adjacent to coordinate point A1 is searched in the vertical direction, and A1, B1 and A2 form a rectangular area, that is, the first feature area. Afterwards, the area with the same position information as the rectangular area is determined in the shared feature map, and the area is segmented to obtain the first feature area. After that, take coordinate point A2 as the new starting point, continue to search for coordinate point A3 adjacent to coordinate point A2 in the horizontal direction, and search for coordinate point B2 adjacent to coordinate point A2 in the vertical direction, to form a second rectangular area, segment the shared feature map according to the second rectangular area, segment to obtain the second feature area, repeat the above operation, traverse the remaining coordinate points in turn, and stop the iteration until the starting point does not have the first coordinate point and the second coordinate point at the same time, to obtain 8 feature areas after segmentation. The pixel points corresponding to each feature area are normalized and integrated to obtain the feature vector corresponding to each feature area, and then form a feature matrix of 8*64*32 size.

[0142] Step 150: input the feature region matrix into a license plate recognition module in the end-to-end license plate recognition model to obtain a license plate recognition prediction result output by the license plate recognition module.

[0143] Specifically, after the feature area matrix is ​​input into the license plate recognition module, the license plate recognition module can use the CNN+BLSTM encoding and decoding structure to extract the features corresponding to the feature area matrix, and use softmax to generate a Ne*T posterior probability matrix, where Ne represents the length of the character set to be recognized and T represents the sequence length, so as to predict the characters in the license plate and obtain the license plate recognition prediction result.

[0144] Optionally, the CNN+BLSTM encoding and decoding structure has a total of 6 layers, including: 1 input layer, into which the regional feature matrix generated by the license plate detection module is input; 1 anti-pooling layer for alleviating fuzzy features; 1 convolution layer; 1 pooling layer; 1 BLSTM layer and 1 fully connected layer. Among them, the kernel size of the anti-pooling layer is 2*1, and the span is 2*1; the kernel size of the convolution layer is 3*3, and the span is 1*1; the number of kernels in the convolution layer can be 64; the kernel size of the pooling layer is 1*2, and the span is 1*2; the number of kernels of the BLSTM can be 96, and the sequence length T=26.

[0145] The license plate recognition method provided by the embodiment of the present invention, after obtaining the license plate image to be recognized, first inputs the image to be recognized into the shared feature layer module in the end-to-end license plate recognition model, extracts the shared feature map corresponding to the license plate image to be recognized, and inputs the shared feature map into the license plate detection module to determine the position information of the license plate area. After segmenting the license plate area in the shared feature map according to the position information of the license plate area, the feature area matrix corresponding to the obtained license plate area is input into the license plate recognition module to obtain the license plate recognition prediction result. By using a unified end-to-end license plate recognition model, the complexity of the network structure and the computational overhead are reduced, the transmission delay of the image data between each cascade model is avoided, and the license plate recognition efficiency is improved.

[0146] The license plate recognition device provided by the present invention is described below. The license plate recognition device described below and the license plate recognition method described above can be referenced to each other.

[0147] The embodiment of the present invention further provides a license plate recognition device, Figure 5 is a schematic diagram of the structure of a license plate recognition device provided by an embodiment of the present invention, such as Figure 5 As shown, the license plate recognition device 500 includes: an acquisition unit 510, a sharing unit 520, a detection unit 530, a segmentation unit 540 and a recognition unit 550, wherein:

[0148] An acquisition unit 510 is used to acquire a license plate image to be recognized;

[0149] A sharing unit 520 is used to input the license plate image to be recognized into a shared feature layer module in an end-to-end license plate recognition model, and obtain a shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module;

[0150] A detection unit 530, configured to input the shared feature map into a license plate detection module in the end-to-end license plate recognition model, and obtain position information output by the license plate detection module for locating a license plate area;

[0151] A segmentation unit 540 is used to segment the license plate area in the shared feature map based on the position information to obtain a feature area matrix corresponding to the license plate area;

[0152] The recognition unit 550 is used to input the feature region matrix into the license plate recognition module in the end-to-end license plate recognition model to obtain the license plate recognition prediction result output by the license plate recognition module.

[0153] The license plate recognition device provided by the embodiment of the present invention, after acquiring the license plate image to be recognized, first inputs the image to be recognized into the shared feature layer module in the end-to-end license plate recognition model, extracts the shared feature map corresponding to the license plate image to be recognized, and inputs the shared feature map into the license plate detection module to determine the position information of the license plate area. After segmenting the license plate area in the shared feature map according to the position information of the license plate area, the feature area matrix corresponding to the obtained license plate area is input into the license plate recognition module to obtain the license plate recognition prediction result. By using a unified end-to-end license plate recognition model, the complexity of the network structure and the computational overhead are reduced, the transmission delay of image data between each cascade model is avoided, and the license plate recognition efficiency is improved.

[0154] Optionally, the sharing unit 520 is specifically configured to:

[0155] Input the license plate image to be identified into the shared feature layer module, compress the license plate to be identified at least twice, and determine at least two first semantic feature graphs corresponding to the license plate to be identified, the number of which is the same as the number of compression times;

[0156] Upsampling the at least two first semantic feature maps to determine at least two second semantic feature maps corresponding to the license plate to be identified;

[0157] The minimum first semantic feature map and the at least two second semantic feature maps are fused to obtain a shared feature map corresponding to the license plate image to be identified output by the shared feature layer module.

[0158] Optionally, the detection unit 530 is specifically configured to:

[0159] The shared feature map is input into the license plate detection module, and a convolution operation is performed on the shared feature map to obtain position information of at least four coordinate points output by the license plate detection module for locating the license plate area, wherein the at least four coordinate points constitute a rectangular frame corresponding to the license plate area.

[0160] Optionally, the segmentation unit 540 is specifically configured to:

[0161] S1. Taking any coordinate point among the at least four coordinate points as a starting point;

[0162] S2. Determine a first coordinate point distributed in the horizontal direction and a second coordinate point distributed in the vertical direction adjacent to the starting point, wherein the angle formed by the starting point, the first coordinate point and the second coordinate point is a right angle;

[0163] S3, based on the position information corresponding to the first coordinate point and the second coordinate point, segmenting the area with the same position information in the shared feature map to determine the feature area corresponding to the starting point;

[0164] S4, taking the first coordinate point as a new starting point, repeatedly executing steps S2 to S4, and stopping iteration when the first coordinate point and the second coordinate point do not exist, thereby obtaining at least two feature areas corresponding to the license plate area;

[0165] S5. Determine a feature region matrix corresponding to the license plate region based on at least two feature regions corresponding to the license plate region.

[0166] Optionally, the license plate recognition device 500 further includes a first training unit, and the first training unit is specifically used for:

[0167] Inputting the sample feature map into the initial license plate detection module, performing a first convolution operation on the sample feature map, and determining a confidence loss function corresponding to the first convolution operation;

[0168] Perform a second convolution operation on the sample feature map to determine a rotation box geometry loss function corresponding to the second convolution operation;

[0169] Perform a third convolution operation on the sample feature map to determine a rotation box angle loss function corresponding to the third convolution operation;

[0170] Performing a fourth convolution operation on the sample feature map, and determining a key point regression loss function corresponding to the fourth convolution operation;

[0171] Determine a license plate detection loss function based on the confidence loss function, the rotation box geometry loss function, the rotation box angle loss function and the key point regression loss function;

[0172] Based on the license plate detection loss function, the initial license plate detection module is trained until the initial license plate detection module converges to obtain a trained license plate detection module.

[0173] Optionally, the license plate recognition device 500 further includes a second training unit, and the second training unit is specifically used for:

[0174] Pre-training an initial shared feature layer module based on a first training data set;

[0175] Based on the deep learning optimizer and the second training data set, the initial license plate detection module and the pre-trained initial shared feature layer module are trained to obtain a trained shared feature layer module and a license plate detection module whose evaluation index is greater than a preset threshold;

[0176] Freeze the model parameters corresponding to the shared feature layer module, train the license plate recognition module based on the second training data set, and fine-tune the license plate detection module to obtain a trained license plate recognition module and a license plate detection module.

[0177] Optionally, the second training unit is specifically used for:

[0178] Determine a license plate recognition loss function based on the second training data set using an aggregate cross entropy algorithm;

[0179] Determine a total license plate loss function by performing a weighted summation based on the license plate recognition loss function and the license plate detection loss function;

[0180] Based on the total license plate loss function, the license plate recognition module is trained, and the license plate detection module is fine-tuned to obtain a trained license plate recognition module and a license plate detection module.

[0181] Figure 6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the license plate recognition method, which includes:

[0182] Get the license plate image to be recognized;

[0183] Input the license plate image to be recognized into a shared feature layer module in an end-to-end license plate recognition model, and obtain a shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module;

[0184] Inputting the shared feature map into a license plate detection module in the end-to-end license plate recognition model to obtain position information output by the license plate detection module for locating the license plate area;

[0185] Based on the position information, segmenting the license plate area in the shared feature map to obtain a feature area matrix corresponding to the license plate area;

[0186] The feature region matrix is ​​input into a license plate recognition module in the end-to-end license plate recognition model to obtain a license plate recognition prediction result output by the license plate recognition module.

[0187] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0188] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the license plate recognition method provided by the above methods, the method includes:

[0189] Get the license plate image to be recognized;

[0190] Input the license plate image to be recognized into a shared feature layer module in an end-to-end license plate recognition model, and obtain a shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module;

[0191] Inputting the shared feature map into a license plate detection module in the end-to-end license plate recognition model to obtain position information output by the license plate detection module for locating the license plate area;

[0192] Based on the position information, segmenting the license plate area in the shared feature map to obtain a feature area matrix corresponding to the license plate area;

[0193] The feature region matrix is ​​input into a license plate recognition module in the end-to-end license plate recognition model to obtain a license plate recognition prediction result output by the license plate recognition module.

[0194] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the license plate recognition method provided by the above methods, the method comprising:

[0195] Get the license plate image to be recognized;

[0196] Input the license plate image to be recognized into a shared feature layer module in an end-to-end license plate recognition model, and obtain a shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module;

[0197] Inputting the shared feature map into a license plate detection module in the end-to-end license plate recognition model to obtain position information output by the license plate detection module for locating the license plate area;

[0198] Based on the position information, segmenting the license plate area in the shared feature map to obtain a feature area matrix corresponding to the license plate area;

[0199] The feature region matrix is ​​input into a license plate recognition module in the end-to-end license plate recognition model to obtain a license plate recognition prediction result output by the license plate recognition module.

[0200] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0201] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A license plate recognition method, characterized in that: include: Get the license plate image to be recognized; Input the license plate image to be recognized into a shared feature layer module in an end-to-end license plate recognition model, and obtain a shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module; Inputting the shared feature map into a license plate detection module in the end-to-end license plate recognition model to obtain position information output by the license plate detection module for locating the license plate area; Based on the position information, segmenting the license plate area in the shared feature map to obtain a feature area matrix corresponding to the license plate area; The feature region matrix is ​​input into a license plate recognition module in the end-to-end license plate recognition model to obtain a license plate recognition prediction result output by the license plate recognition module.

2. The license plate recognition method according to claim 1, characterized in that: The step of inputting the shared feature map into a license plate detection module in the end-to-end license plate recognition model to obtain location information of the license plate area output by the license plate detection module includes: The shared feature map is input into the license plate detection module, and a convolution operation is performed on the shared feature map to obtain position information of at least four coordinate points output by the license plate detection module for locating the license plate area, wherein the at least four coordinate points constitute a rectangular frame corresponding to the license plate area.

3. The license plate recognition method according to claim 2, characterized in that: The license plate detection module is trained based on the following steps: Inputting the sample feature map into the initial license plate detection module, performing a first convolution operation on the sample feature map, and determining a confidence loss function corresponding to the first convolution operation; Perform a second convolution operation on the sample feature map to determine a rotation box geometry loss function corresponding to the second convolution operation; Perform a third convolution operation on the sample feature map to determine a rotation box angle loss function corresponding to the third convolution operation; Performing a fourth convolution operation on the sample feature map, and determining a key point regression loss function corresponding to the fourth convolution operation; Determine a license plate detection loss function based on the confidence loss function, the rotation box geometry loss function, the rotation box angle loss function and the key point regression loss function; Based on the license plate detection loss function, the initial license plate detection module is trained until the initial license plate detection module converges to obtain a trained license plate detection module.

4. The license plate recognition method according to claim 3, characterized in that: The step of segmenting the license plate region in the shared feature map based on the position information to obtain a feature region matrix corresponding to the license plate region includes: S1. Taking any coordinate point among the at least four coordinate points as a starting point; S2. Determine a first coordinate point distributed in the horizontal direction and a second coordinate point distributed in the vertical direction adjacent to the starting point, wherein the angle formed by the starting point, the first coordinate point and the second coordinate point is a right angle; S3, based on the position information corresponding to the first coordinate point and the second coordinate point, segmenting the area with the same position information in the shared feature map to determine the feature area corresponding to the starting point; S4, taking the first coordinate point as a new starting point, repeatedly executing steps S2 to S4, and stopping iteration when the first coordinate point and the second coordinate point do not exist, thereby obtaining at least two feature areas corresponding to the license plate area; S5. Determine a feature region matrix corresponding to the license plate region based on at least two feature regions corresponding to the license plate region.

5. The license plate recognition method according to any one of claims 1 to 4, characterized in that: The step of inputting the license plate image to be recognized into a shared feature layer module in an end-to-end license plate recognition model to obtain a shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module includes: Input the license plate image to be identified into the shared feature layer module, compress the license plate to be identified at least twice, and determine at least two first semantic feature graphs corresponding to the license plate to be identified, the number of which is the same as the number of compression times; Upsampling the at least two first semantic feature maps to determine at least two second semantic feature maps corresponding to the license plate to be identified; The minimum first semantic feature map and the at least two second semantic feature maps are fused to obtain a shared feature map corresponding to the license plate image to be identified output by the shared feature layer module.

6. The license plate recognition method according to claim 3 or 4, characterized in that: The end-to-end license plate recognition model is trained based on the following steps: Pre-training an initial shared feature layer module based on a first training data set; Based on the deep learning optimizer and the second training data set, the initial license plate detection module and the pre-trained initial shared feature layer module are trained to obtain a trained shared feature layer module and a license plate detection module whose evaluation index is greater than a preset threshold; Freeze the model parameters corresponding to the shared feature layer module, train the initial license plate recognition module based on the second training data set, and fine-tune the license plate detection module to obtain a trained license plate recognition module and a license plate detection module.

7. The license plate recognition method according to claim 6, characterized in that: The initial license plate recognition module is trained based on the second training data set, and the license plate detection module is fine-tuned to obtain a trained license plate recognition module and a license plate detection module, including: Determine a license plate recognition loss function based on the second training data set using an aggregate cross entropy algorithm; Determine a total license plate loss function by performing a weighted summation based on the license plate recognition loss function and the license plate detection loss function; Based on the total license plate loss function, the initial license plate recognition module is trained, and the license plate detection module is fine-tuned to obtain a trained license plate recognition module and a license plate detection module.

8. A license plate recognition device, characterized in that: include: An acquisition unit, used for acquiring a license plate image to be recognized; A sharing unit, used for inputting the license plate image to be recognized into a shared feature layer module in an end-to-end license plate recognition model, and obtaining a shared feature map corresponding to the license plate image to be recognized output by the shared feature layer module; A detection unit, used to input the shared feature map into a license plate detection module in the end-to-end license plate recognition model to obtain position information output by the license plate detection module for locating the license plate area; A segmentation unit, configured to segment the license plate region in the shared feature map based on the position information to obtain a feature region matrix corresponding to the license plate region; The recognition unit is used to input the feature region matrix into the license plate recognition module in the end-to-end license plate recognition model to obtain the license plate recognition prediction result output by the license plate recognition module.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the license plate recognition method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the license plate recognition method as described in any one of claims 1 to 7 is implemented.

Images