Methods, apparatus, storage media and electronic devices for processing license plate images

Abstract

This application provides a method, apparatus, storage medium, and electronic device for processing license plate images. The method includes: acquiring a training set of license plate images and a tag license plate image corresponding to each training license plate image in the training set, wherein each training license plate image and the tag license plate image corresponding to each training license plate image are different license plate images obtained by capturing the same license plate, and the clarity of each training license plate image is less than the clarity of the tag license plate image corresponding to each training license plate image; and training an initial license plate deblurring model using each training license plate image and the tag license plate image corresponding to each training license plate image to obtain a trained license plate deblurring model, wherein the license plate deblurring model is a generative adversarial network-based model for image deblurring.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation, and more specifically, to a method, apparatus, storage medium, and electronic device for processing license plate images. Background Technology

[0002] Most existing license plate recognition methods are limited by complex weather conditions. For example, lighting conditions, rain, snow, dust, and image noise caused by extreme weather can blur license plate images and reduce the accuracy of license plate recognition. To improve the accuracy of license plate recognition, blurred license plates can be deblurred, for example, by using an end-to-end convolutional neural network.

[0003] However, due to the overemphasis on local characteristics by convolutional neural networks and the sensitivity of end-to-end networks to input, the deblurring effect on license plates is poor. Therefore, it is evident that related methods for processing license plate images suffer from poor deblurring performance due to the excessive focus of convolutional neural networks on local characteristics. Summary of the Invention

[0004] This application provides a method, apparatus, storage medium, and electronic device for processing license plate images, in order to at least solve the problem that the license plate deblurring model has poor deblurring effect due to the convolutional neural network's excessive focus on local characteristics in related art processing methods.

[0005] According to one embodiment of this application, a method for processing license plate images is provided, comprising: acquiring a training set of license plate images and a label license plate image corresponding to each training license plate image in the training set, wherein each training license plate image and the label license plate image corresponding to each training license plate image are different license plate images obtained by taking pictures of the same license plate, and the clarity of each training license plate image is less than the clarity of the label license plate image corresponding to each training license plate image; using each training license plate image and the label license plate image corresponding to each training license plate image to train an initial license plate deblurring model to obtain a trained license plate deblurring model, wherein the license plate deblurring model is a generative adversarial network-based model for image deblurring.

[0006] In an exemplary embodiment, the license plate deblurring model includes a generator; the method further includes: inputting a license plate image to be recognized into the generator of the license plate deblurring model to obtain a deblurred license plate image output by the generator of the license plate deblurring model; and performing license plate recognition processing on the input deblurred license plate image to obtain a license plate recognition result corresponding to the license plate image to be recognized.

[0007] In an exemplary embodiment, training an initial license plate deblurring model using each training license plate image and a corresponding labeled license plate image to obtain a trained license plate deblurring model includes: using each training license plate image as a current training license plate image to perform multiple rounds of model training on the license plate deblurring model to be trained, thereby obtaining the trained license plate deblurring model. The license plate deblurring model includes a generator and a discriminator. When training the model using the current training license plate image, the generator of the current license plate deblurring model is the current generator, and the discriminator of the current license plate deblurring model is the current discriminator. Discriminator: The current training license plate image is input into the current generator to obtain the deblurred license plate image output by the current generator; the deblurred license plate image and the label license plate image corresponding to the current training license plate image are input into the current discriminator to obtain the image discrimination result output by the current discriminator, wherein the image discrimination result is used to represent the image similarity between the deblurred license plate image and the label license plate image; at least one of the parameters of the current generator and the current discriminator is adjusted according to the deblurred license plate image, the label license plate image and the image discrimination result to obtain the adjusted license plate deblurring model.

[0008] In an exemplary embodiment, adjusting the parameters of at least one of the current generator and the current discriminator based on the deblurred license plate image, the tagged license plate image, and the image discrimination result to obtain an adjusted license plate deblurring model includes: determining the model loss of the current license plate deblurring model based on the deblurred license plate image, the tagged license plate image, and the image discrimination result; and adjusting the parameters of at least one of the current generator and the current discriminator based on the model loss of the current license plate deblurring model to obtain the adjusted license plate deblurring model, wherein the model loss of the adjusted license plate deblurring model is less than the model loss of the current license plate deblurring model.

[0009] In an exemplary embodiment, determining the model loss of the current license plate deblurring model based on the deblurred license plate image, the labeled license plate image, and the image discrimination result includes: determining the Manhattan distance between the deblurred license plate image and the labeled license plate image as a first model loss corresponding to the current generator; determining the binary cross-entropy loss between the image discrimination result and the labeled license plate image as a second model loss corresponding to the current discriminator; and performing a weighted summation of the first model loss and the second model loss to obtain the model loss of the current license plate deblurring model.

[0010] In an exemplary embodiment, the step of inputting the current training license plate image into a current generator to obtain a deblurred license plate image output by the current generator includes: inputting the current training license plate image into the input convolutional layer of the current generator to obtain a shallow feature map of the current training license plate image extracted by the input convolutional layer; extracting deep features of the current training license plate image from the shallow feature map of the current training license plate image through multiple cascaded rotational attention layers of the current generator to obtain a deep feature map of the current training license plate image; adding the shallow feature map of the current training license plate image and the deep feature map of the current training license plate image through residual connections to obtain a feature map addition result of the current training license plate image; and inputting the feature map addition result of the current training license plate image into the output convolutional layer of the current generator for feature fusion to obtain the deblurred license plate image corresponding to the current training license plate image.

[0011] In an exemplary embodiment, the step of extracting deep features of the current training license plate image from the shallow feature map of the current training license plate image through multiple cascaded rotational attention layers of the current generator to obtain the deep feature map of the current training license plate image includes: sequentially processing the input of each rotational attention layer through a first normalization layer, a multi-head rotational attention layer, a second normalization layer, and a fully connected layer in each of the multiple cascaded rotational attention layers to obtain the deep feature map of the current training license plate image output by the fully connected layer of the last rotational attention layer of the multiple rotational attention layers; wherein, the input of the first rotational attention layer in the multiple cascaded rotational attention layers is the shallow feature map of the current training license plate image, and the inputs of the other rotational attention layers are the outputs of the previous rotational attention layer of the other rotational attention layers.

[0012] In an exemplary embodiment, acquiring a training license plate image set and a labeled license plate image corresponding to each training license plate image in the training license plate image set includes: acquiring multiple license plate image sets, wherein each license plate image set contains multiple license plate images obtained by acquiring license plate images of the same vehicle at different acquisition distances, and all license plate images in each license plate image set correspond to the same preset license plate; performing license plate recognition on each license plate image in each license plate image set to obtain a license plate recognition result corresponding to each license plate image in each license plate image set; and determining the license plate recognition result based on each license plate image. For each license plate image in the set, a training license plate image and a corresponding tag license plate image are selected from each set of license plate images to obtain the training license plate image set and the tag license plate image corresponding to each training license plate image. The license plate indicated by the license plate recognition result of each training license plate image is different from the preset license plate corresponding to each training license plate image. The tag license plate image corresponding to each training license plate image is a license plate image in the set of license plate images to which the corresponding license plate recognition result is the same as the corresponding preset license plate.

[0013] According to another embodiment of this application, a license plate image processing apparatus is provided, comprising: an acquisition unit, configured to acquire a training license plate image set and a tag license plate image corresponding to each training license plate image in the training license plate image set, wherein each training license plate image and the tag license plate image corresponding to each training license plate image are different license plate images obtained by capturing the same license plate, and the clarity of each training license plate image is less than the clarity of the tag license plate image corresponding to each training license plate image; and a training unit, configured to train an initial license plate deblurring model using each training license plate image and the tag license plate image corresponding to each training license plate image to obtain a trained license plate deblurring model, wherein the license plate deblurring model is a generative adversarial network-based model for image deblurring.

[0014] In one exemplary embodiment, the license plate deblurring model includes a generator; the apparatus further includes: an input unit for inputting a license plate image to be recognized into the generator of the license plate deblurring model to obtain a deblurred license plate image output by the generator of the license plate deblurring model; and a processing unit for performing license plate recognition processing on the input deblurred license plate image to obtain a license plate recognition result corresponding to the license plate image to be recognized.

[0015] In an exemplary embodiment, the training unit includes: a training module, configured to perform multiple rounds of model training on a license plate deblurring model to be trained, using each training license plate image as a current training license plate image, to obtain a trained license plate deblurring model, wherein the license plate deblurring model includes a generator and a discriminator; when using the current training license plate image for model training, the generator of the current license plate deblurring model is the current generator, and the discriminator of the current license plate deblurring model is the current discriminator; the current training license plate image is input into the current generator to obtain the... The deblurred license plate image output by the current generator is described; the deblurred license plate image and the labeled license plate image corresponding to the current training license plate image are input into the current discriminator to obtain the image discrimination result output by the current discriminator, wherein the image discrimination result is used to represent the image similarity between the deblurred license plate image and the labeled license plate image; at least one of the parameters of the current generator and the current discriminator is adjusted according to the deblurred license plate image, the labeled license plate image and the image discrimination result to obtain the adjusted license plate deblurring model.

[0016] In an exemplary embodiment, the training module includes: a determination submodule, configured to determine the model loss of the current license plate deblurring model based on the deblurred license plate image, the labeled license plate image, and the image discrimination result; and an adjustment submodule, configured to adjust the parameters of at least one of the current generator and the current discriminator based on the model loss of the current license plate deblurring model to obtain an adjusted license plate deblurring model, wherein the model loss of the adjusted license plate deblurring model is less than the model loss of the current license plate deblurring model.

[0017] In an exemplary embodiment, the determining submodule includes: a first determining subunit, configured to determine the Manhattan distance between the deblurred license plate image and the labeled license plate image as a first model loss corresponding to the current generator; a second determining subunit, configured to determine the binary cross-entropy loss between the image discrimination result and the labeled license plate image as a second model loss corresponding to the current discriminator; and a summing subunit, configured to perform a weighted summation of the first model loss and the second model loss to obtain the model loss of the current license plate deblurring model.

[0018] In an exemplary embodiment, the training module includes: an input submodule, configured to input the current training license plate image into the input convolutional layer of the current generator to obtain a shallow feature map of the current training license plate image extracted by the input convolutional layer; an extraction submodule, configured to extract deep features of the current training license plate image from the shallow feature map of the current training license plate image through multiple cascaded rotational attention layers of the current generator to obtain a deep feature map of the current training license plate image; a processing submodule, configured to add the shallow feature map of the current training license plate image and the deep feature map of the current training license plate image through residual connections to obtain a feature map addition result of the current training license plate image; and a fusion submodule, configured to input the feature map addition result of the current training license plate image into the output convolutional layer of the current generator for feature fusion to obtain the deblurred license plate image corresponding to the current training license plate image.

[0019] In an exemplary embodiment, the extraction submodule includes: a processing unit, configured to process the input of each rotation attention layer sequentially through a first normalization layer, a multi-head rotation attention layer, a second normalization layer, and a fully connected layer in each of the cascaded plurality of rotation attention layers, to obtain a deep feature map of the currently trained license plate image output by the fully connected layer of the last rotation attention layer of the plurality of rotation attention layers; wherein, the input of the first rotation attention layer in the cascaded plurality of rotation attention layers is a shallow feature map of the currently trained license plate image, and the inputs of the other rotation attention layers besides the first rotation attention layer are the outputs of the previous rotation attention layer of the other rotation attention layers.

[0020] In an exemplary embodiment, the acquisition unit includes: an acquisition module, configured to acquire multiple license plate image sets, wherein each license plate image set contains multiple license plate images obtained by acquiring license plate images of the same vehicle at different acquisition distances, and all license plate images in each license plate image set correspond to the same preset license plate; an identification module, configured to perform license plate recognition on each license plate image in each license plate image set to obtain a license plate recognition result corresponding to each license plate image in each license plate image set; and a selection module, configured to select based on each license plate image in each license plate image set. The license plate recognition result corresponding to the license plate image is obtained by selecting training license plate images and corresponding tag license plate images from each license plate image set, thus obtaining the training license plate image set and the tag license plate image corresponding to each training license plate image; wherein, the license plate indicated by the license plate recognition result corresponding to each training license plate image is different from the preset license plate corresponding to each training license plate image, and the tag license plate image corresponding to each training license plate image is the same as the corresponding preset license plate indicated by the license plate recognition result in the license plate image set to which each training license plate image belongs.

[0021] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0022] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0023] This application trains a license plate deblurring model using a training set of license plate images and corresponding label license plate images for each training license plate image in the training set. The training set of license plate images and corresponding label license plate images are obtained by acquiring different license plate images of the same license plate, where each training license plate image and its corresponding label image are different images of the same license plate, and the sharpness of each training license plate image is less than that of its corresponding label image. The initial license plate deblurring model is trained using each training license plate image and its corresponding label image to obtain a trained license plate deblurring model. This license plate deblurring model is a generative adversarial network-based model for image deblurring. The model is designed to be based on the sharpness of the license plate images. This method divides different license plate images obtained from image acquisition of the same license plate into training license plate images and corresponding label images, resulting in a training license plate image set and a label image corresponding to each training license plate image in the training license plate image set. This improves the convenience and effectiveness of acquiring the dataset for model training. Furthermore, the trained license plate deblurring model is based on a generative adversarial network (GAN) image processing model, which avoids the problem of convolutional neural networks focusing too much on local characteristics and the instability of output results due to the sensitivity of end-to-end networks to input. This improves the accuracy and reliability of license plate image deblurring, achieving a better deblurring effect. This solves the problem of poor deblurring performance in related license plate image processing methods due to convolutional neural networks focusing too much on local characteristics. Attached Figure Description

[0024] Figure 1 This is a hardware structure block diagram of a mobile terminal for a license plate image processing method according to an embodiment of this application.

[0025] Figure 2 This is a flowchart illustrating an optional license plate image processing method according to an embodiment of this application.

[0026] Figure 3 This is a schematic diagram of an optional license plate image processing method according to an embodiment of this application;

[0027] Figure 4 This is a schematic diagram of another optional license plate image processing method according to an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of another optional license plate image processing method according to an embodiment of this application;

[0029] Figure 6This is a flowchart illustrating another optional method for processing license plate images according to an embodiment of this application;

[0030] Figure 7 This is a structural block diagram of an optional license plate image processing apparatus according to an embodiment of this application;

[0031] Figure 8 This is a structural block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0032] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0034] According to one aspect of the embodiments of this application, a method for processing license plate images is provided. Optionally, in this embodiment, the above-described license plate image processing method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102, image acquisition device 104, and server 106. Figure 1 As shown, the terminal device 102 can connect to the image acquisition device 104 and / or the server 106 (e.g., an IoT platform or a cloud server) via a network to control the image acquisition device 104, for example, by binding to the image acquisition device 104 and configuring the image acquisition function of the image acquisition device 104.

[0035] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network (WAN), metropolitan area network (MAN), local area network (LAN). The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth, infrared. The network used by the terminal device 102 to communicate with the image acquisition device 104 and / or the server 106 may be the same as or different from the network used by the image acquisition device 104 to communicate with the server 106. The terminal device 102 is not limited to PCs, mobile phones, tablets, etc., the image acquisition device 104 may include, but is not limited to, cameras, camcorders, etc., and the server 106 may be a server of an IoT platform.

[0036] The license plate image processing method of this application embodiment can be executed by a single terminal device 102, image acquisition device 104, or server 106, or it can be executed jointly by at least two of the terminal device 102, image acquisition device 104, and server 106. The area map processing method of this application embodiment executed by the terminal device 102 or image acquisition device 104 can also be executed by a client installed on it.

[0037] Taking the license plate image processing method in this embodiment, executed by server 106, as an example, Figure 2 This is a flowchart illustrating an optional license plate image processing method according to an embodiment of this application, as shown below. Figure 2 As shown, the process of this method may include the following steps:

[0038] Step S202: Obtain a training license plate image set and a label license plate image corresponding to each training license plate image in the training license plate image set. Each training license plate image and the label license plate image corresponding to each training license plate image are different license plate images obtained by taking pictures of the same license plate. The clarity of each training license plate image is less than the clarity of the label license plate image corresponding to each training license plate image.

[0039] The license plate image processing method in this embodiment can be applied to scenarios involving deblurring captured license plate images. Image deblurring can be a preprocessing step before license plate recognition, and can also be applied to other processes that require improving the clarity of license plate images. Deblurring of license plate images can be accomplished using a trained license plate deblurring model. Before using the license plate deblurring model to deblurr the license plate image, the initial license plate deblurring model can be trained.

[0040] With the rapid development of intelligent transportation technology, intelligent transportation systems can be installed to perform tasks such as traffic monitoring and traffic enforcement, making them an indispensable part of the intelligent transportation field. Vehicle recognition technology can be widely used in areas such as illegal parking detection, violation identification, and electronic toll collection, and it also plays an important role in searching for missing vehicles or locating specific vehicles. However, most license plate recognition technologies are limited by the complex weather conditions in real-world applications. For example, lighting conditions, rain, snow, dust, and some extreme weather conditions can lead to high noise levels in the captured license plate images, making the images blurry and reducing the recognition accuracy of the license plate recognition system.

[0041] To improve the success rate of license plate recognition in complex weather conditions, enhance its accuracy and robustness, and build and improve intelligent transportation systems, the accuracy of license plate recognition technology in complex situations can be improved by enhancing the quality of images captured by cameras. This involves deblurring blurred license plate images. Currently, a common approach to license plate image deblurring is to train a convolutional neural network (CNN) end-to-end (i.e., using a CNN-based license plate deblurring model) and then use the trained CNN to deblur the license plate images. However, because CNNs focus too much on local close-ups and end-to-end networks are sensitive to input, the output of the CNN is unstable, resulting in poor deblurring performance.

[0042] To at least partially solve the aforementioned technical problems, in this embodiment, based on the clarity of the license plate images, different license plate images obtained from image acquisition of the same license plate are divided into training license plate images and corresponding label license plate images. By acquiring images of multiple vehicles, a training license plate image set and a label license plate image corresponding to each training license plate image in the training license plate image set can be obtained. The initial license plate deblurring model is trained using the training license plate image set and the label license plate images corresponding to each training license plate image in the training license plate image set, resulting in a trained license plate deblurring model. The trained license plate deblurring model can automatically deblurr license plate images of different distances, different lighting intensities, and different degrees of blur. It can also achieve good deblurring effects for license plate images where the license plate information is difficult for the human eye to identify, thereby improving the accuracy of the license plate recognition system and solving the problem of unclear license plate images captured by cameras in special traffic scenarios. This saves manpower costs and improves work efficiency, playing a crucial role in maintaining safe and civilized driving and punishing traffic violations.

[0043] In this embodiment, license plates of multiple vehicles can be photographed using an image acquisition device at different acquisition distances. The image acquisition device can be a camera, video camera, or other device with image acquisition capabilities. The acquisition distance is the distance between the image acquisition device and the vehicle. Among the different license plate images obtained from photographing the same license plate, the license plate image with lower resolution is used as the training license plate image, while the license plate image with higher resolution is used as the label license plate image corresponding to the training license plate image. Here, the resolution can be determined manually. For example, by manually examining different license plate images obtained from photographing the same license plate, the license plate image that is illegible, the license plate image that is legible but has low resolution, or the license plate image other than the one with the highest resolution is selected as the training license plate image. The license plate image that is legible or the license plate image with the highest resolution is selected as the label license plate image corresponding to the training license plate image.

[0044] Here, a training license plate image and its corresponding label license plate image are different license plate images obtained by taking pictures of the same license plate. The training license plate image can be a license plate image that cannot be identified (by human or machine) or a license plate image that can be identified. The label license plate images corresponding to different training license plate images can be the same or different. In this embodiment, there is no limitation on the training license plate image and the label license plate image.

[0045] For example, by taking pictures of the same vehicle at different distances using a camera, the obtained vehicle photos are preprocessed: first, the license plate image is extracted using a deep learning detection framework, and then the blurriness / clarity of the license plate images of the same vehicle is classified by manual annotation. The image that is closer to the camera and clearer (img_H) is used as the training label for the license plate deblurring model, and the image that is farther away from the camera and blurry (img_L) is used as the training input for the license plate deblurring model, thus obtaining the training set, and training a license plate deblurring model based on a transformer (self-attention) generative adversarial network.

[0046] Optionally, for the same vehicle, capturing license plate images of the vehicle at different acquisition distances using an image acquisition device can be done by using a single image acquisition device at different acquisition distances. For example, capturing the license plate of the same vehicle at different distances from the camera using a camera. Alternatively, it can be done by using multiple image acquisition devices at different acquisition distances from the vehicle to capture license plate images of the vehicle.

[0047] Step S204: Use each training license plate image and the corresponding labeled license plate image to train the initial license plate deblurring model to obtain the trained license plate deblurring model. The license plate deblurring model is a generative adversarial network-based model for image deblurring.

[0048] After obtaining a set of training license plate images and corresponding labeled license plate images for each training license plate image, the initial license plate deblurring model is trained using these images. The resulting trained model can then deblur license plate images. Here, the license plate deblurring model is a model used to deblur images and can be based on Generative Adversarial Networks (GANs).

[0049] A Generative Adversarial Network (GAN) can consist of a generator and a discriminator. The generator deblurs the training license plate image to obtain a deblurred license plate image corresponding to the training license plate image. The discriminator distinguishes whether the input is real data or a generated sample; that is, it determines whether the input image is the labeled license plate image corresponding to the training license plate image or the deblurred license plate image obtained by the generator after deblurring the training license plate image.

[0050] During model training, the generator updates its parameters using a loss function with the condition that "the deblurred license plate image corresponding to the training license plate image is true." In other words, the generator's training objective is for the discriminator to classify the deblurred license plate image corresponding to the training license plate image as the label license plate image corresponding to the training license plate image. The discriminator, on the other hand, updates its parameters using a loss function with the condition that "the label license plate image corresponding to the training license plate image is true, and the deblurred license plate image corresponding to the training license plate image is false." In other words, the discriminator's training objective is to accurately distinguish whether the input image is the label license plate image corresponding to the training license plate image or the deblurred license plate image obtained by the generator after deblurring the training license plate image. Therefore, the generator and discriminator form an adversarial relationship during model training.

[0051] The discriminator's main function is to assist in the training of the generator; its input is the generator's output, such as... Figure 3 As shown, the discriminator can be a Unet network structure (a U-shaped network structure), consisting of an upsampling layer, a downsampling layer, an input convolutional layer, an output convolutional layer, and a residual connection module. The number of upsampling layers and downsampling layers can both be two.

[0052] Optionally, the trained license plate deblurring model can be deployed on different devices, such as servers, terminal devices, or other electronic devices, so that license plate images can be deblurred using the deployed license plate deblurring model. The device deploying the trained license plate deblurring model and the training device for the license plate deblurring model can be the same device or different devices. The above image processing procedure can also be applied to other scenarios besides license plate deblurring. In this case, the license plate image can be other object images, the license plate deblurring model can be an image deblurring model, the training license plate image can be a training object image, and the corresponding labeled license plate image can be a labeled object image. The training license plate image and the corresponding labeled license plate image are object images of different resolutions obtained by capturing the same object. Other processes are similar to those in the previous embodiments and will not be repeated here.

[0053] Through the above steps, a training license plate image set and a corresponding label license plate image for each training license plate image in the training license plate image set are obtained. Each training license plate image and its corresponding label license plate image are different license plate images obtained by capturing the same license plate. The clarity of each training license plate image is less than that of its corresponding label license plate image. The initial license plate deblurring model is trained using each training license plate image and its corresponding label license plate image to obtain a trained license plate deblurring model. This model is based on a generative adversarial network and is used for image deblurring. It solves the problem in related technologies where license plate image processing methods suffer from poor deblurring performance due to convolutional neural networks focusing too much on local characteristics, thus improving the deblurring effect of the license plate deblurring model.

[0054] In one exemplary embodiment, the license plate deblurring model includes a generator; the method further includes:

[0055] S11, Input the license plate image to be recognized into the generator of the license plate deblurring model to obtain the deblurred license plate image to be recognized output by the generator of the license plate deblurring model;

[0056] S12, perform license plate recognition processing on the deblurred license plate image to obtain the license plate recognition result corresponding to the license plate image.

[0057] For a trained license plate deblurring model, the image to be recognized can be deblurred using only the generator of the license plate deblurring model. It is not necessary to use the discriminator of the license plate deblurring model to discriminate the deblurred license plate image. Therefore, the image of the license plate to be recognized can be input into the generator of the license plate deblurring model to obtain the deblurred license plate image output by the generator of the license plate deblurring model.

[0058] For example, such as Figure 4 As shown, during the model inference stage, the blurry license plate image captured by the camera or webcam is input into the trained license plate deblurring model. Only the deblurring module G (i.e., the generator) in the license plate deblurring model is needed to output a clear license plate image. At this time, the discriminator module used to assist training can be discarded.

[0059] For the deblurred license plate image to be identified, license plate recognition processing can be performed to obtain the license plate recognition result corresponding to the license plate image to be identified. Here, license plate recognition processing can include license plate number recognition, license plate color recognition, license plate layer recognition, etc. The deblurred license plate image to be identified can be input into the corresponding license plate recognition processing model to obtain the license plate recognition result corresponding to the license plate image to be identified.

[0060] In this embodiment, by inputting the license plate image into the generator of the trained license plate deblurring model for deblurring, there is no need to use a discriminator to distinguish the deblurred license plate image, which can reduce resource consumption.

[0061] In one exemplary embodiment, training an initial license plate deblurring model using each training license plate image and a corresponding labeled license plate image to obtain a trained license plate deblurring model includes:

[0062] S21, each training license plate image is used as the current training license plate image to train the license plate deblurring model for multiple rounds, resulting in a trained license plate deblurring model. The license plate deblurring model includes a generator and a discriminator. When training the model using the current training license plate image, the generator of the current license plate deblurring model is called the current generator, and the discriminator of the current license plate deblurring model is called the current discriminator.

[0063] Input the current training license plate image into the current generator to obtain the deblurred license plate image output by the current generator;

[0064] The deblurred license plate image and the labeled license plate image corresponding to the current training license plate image are input into the current discriminator to obtain the image discrimination result output by the current discriminator. The image discrimination result is used to represent the image similarity between the deblurred license plate image and the labeled license plate image.

[0065] Based on the deblurred license plate image, the labeled license plate image, and the image discrimination result, at least one of the parameters of the current generator and the current discriminator is adjusted to obtain the adjusted license plate deblurring model.

[0066] When training the initial license plate deblurring model in multiple rounds using each training license plate image and the corresponding labeled license plate image, in each round of model training, the license plate deblurring model being trained is the current license plate deblurring model, each training license plate image is used as the current training license plate image, the generator of the current license plate deblurring model is used as the current generator, and the discriminator of the current license plate deblurring model is used as the current discriminator.

[0067] The current training license plate image is input into the current generator for deblurring, resulting in a deblurred license plate image output by the current generator. The deblurred license plate image and the corresponding label license plate image are then input into the current discriminator. The current discriminator distinguishes between the deblurred license plate image and the corresponding label license plate image, obtaining the image discrimination result output by the current discriminator. Here, the image discrimination result represents the image similarity between the deblurred license plate image corresponding to the current training license plate image and the corresponding label license plate image.

[0068] Based on the deblurred license plate image, the labeled license plate image, and the image discrimination result, the parameters of the current generator or the current discriminator can be adjusted. That is, the parameters of only one of the current generator and the current discriminator can be adjusted, or the parameters of the current generator and the current discriminator can be adjusted simultaneously.

[0069] In this embodiment, by adjusting the parameters of the license plate deblurring model based on the deblurred license plate image, the labeled license plate image, and the image discrimination results during multi-round model training, the deblurring effect of the license plate deblurring model can be improved.

[0070] In an exemplary embodiment, adjusting the parameters of at least one of the current generator and the current discriminator based on the deblurred license plate image, the tagged license plate image, and the image discrimination result to obtain an adjusted license plate deblurring model includes:

[0071] S31. Based on the deblurred license plate image, the labeled license plate image, and the image discrimination results, determine the model loss of the current license plate deblurring model.

[0072] S32, based on the model loss of the current license plate deblurring model, adjust the parameters of at least one of the current generator and the current discriminator to obtain the adjusted license plate deblurring model, wherein the model loss of the adjusted license plate deblurring model is less than the model loss of the current license plate deblurring model.

[0073] When training a license plate deblurring model multiple times using each training license plate image as the current training license plate image, the model loss of the current license plate deblurring model can be determined based on the deblurred license plate image, the labeled license plate image, and the image discrimination result. Based on the model loss of the current license plate deblurring model, at least one of the parameters of the current generator and the current discriminator is adjusted to obtain the adjusted license plate deblurring model. To ensure improved deblurring performance of the adjusted model, the model loss of the adjusted license plate deblurring model should be less than the model loss of the current license plate deblurring model.

[0074] In this embodiment, by adjusting the parameters of the license plate deblurring model through the model loss of the license plate deblurring model, the deblurring effect of the license plate deblurring model can be improved.

[0075] In an exemplary embodiment, the model loss of the current license plate deblurring model is determined based on the deblurred license plate image, the tagged license plate image, and the image discrimination result, including:

[0076] S41, the Manhattan distance between the deblurred license plate image and the labeled license plate image is determined as the first model loss corresponding to the current generator;

[0077] S42, the binary cross-entropy loss between the image discrimination result and the labeled license plate image is determined as the second model loss corresponding to the current discriminator;

[0078] S43, the first model loss and the second model loss are weighted and summed to obtain the model loss of the current license plate deblurring model.

[0079] In this embodiment, the model loss of the license plate deblurring model can be composed of a first model loss and a second model loss. Here, the first model loss is the model loss corresponding to the current generator, i.e., the generator loss, and the second model loss is the model loss corresponding to the current discriminator, i.e., the discriminator loss.

[0080] The first model loss can be the Manhattan distance (L-1 distance) between the deblurred license plate image and the labeled license plate image, as shown in Equation (1). The generator's loss function (G_loss) is the L-1 distance between the generator's output (i.e., the deblurred license plate image) and the labeled license plate image.

[0081] G_loss = ||G(img_L)-img_H||1 (1)

[0082] Where G(img_L) is the output of the generator.

[0083] The second model loss can be the binary cross-entropy loss between the image discrimination result and the labeled license plate image, as shown in formula (2). The discriminator's loss function (D_loss) is the BCE logistic loss function (a binary classification loss function) between the discriminator's output (i.e., the image discrimination result) and the labeled license plate image.

[0084] D_loss ={-,y*logD(G(img_L))+(1-y)*log(1-D(G(img_L)))]} (2)

[0085] Where D(G(img_L)) is the image discrimination result, and y is the matrix obtained by normalizing the pixel values ​​of each pixel in the license plate image to the 0-1 interval.

[0086] The loss function of the entire generative adversarial network consists of two parts: the generator loss and the discriminator loss. Parameters are used to measure the contribution of these two losses to the total loss function. The loss function of the license plate deblurring model can be expressed as shown in formula (3):

[0087] Loss = G_loss + γ * D_loss (3)

[0088] Here, γ is a hyperparameter controlling the discriminator loss. The larger the value, the greater the weight of the discriminator loss in the total loss, and vice versa. The value of γ can be determined manually through experiments. Optionally, the value of γ can be 0.1.

[0089] Optionally, the loss function shown in Equation (3) can be optimized using the Adam (Adaptive Moment Estimation) optimizer, thereby continuously learning and adjusting the network parameters until the loss function meets the preset threshold or reaches the maximum number of training rounds, the network converges, and the training of the model is completed.

[0090] In this embodiment, by optimizing the license plate deblurring model using the loss function of the license plate deblurring model, the reliability of the license plate deblurring model in deblurring processing can be improved.

[0091] In one exemplary embodiment, the currently trained license plate image is input into the current generator to obtain the deblurred license plate image output by the current generator, including:

[0092] S51, input the current training license plate image into the input convolutional layer of the current generator to obtain the shallow feature map of the current training license plate image extracted by the input convolutional layer;

[0093] S52, extract the deep features of the current training license plate image from the shallow feature map of the current training license plate image through multiple rotation attention layers cascaded by the current generator, and obtain the deep feature map of the current training license plate image.

[0094] S53, the shallow feature map and the deep feature map of the current training license plate image are added together through residual connection to obtain the feature map addition result of the current training license plate image;

[0095] S54, the sum of the feature maps of the current training license plate image is input into the output convolutional layer of the current generator for feature fusion, so as to obtain the deblurred license plate image corresponding to the current training license plate image.

[0096] The license plate deblurring model generator includes an input convolutional layer, an output convolutional layer, a residual connection module, and multiple cascaded rotational attention layers. The current training license plate image is input into the generator. The input convolutional layer extracts shallow features from the current training license plate image, resulting in a shallow feature map. This shallow feature map is then input into the cascaded rotational attention layers in the generator to extract deep features, resulting in a deep feature map.

[0097] For example, such as Figure 3 As shown, for the license plate deblurring model based on transformer generative adversarial network, its generator is an image restoration network based on Rotate-transformer (rotational self-attention). This network consists of an input convolutional layer, an output convolutional layer, a residual connection module, and several cascaded residual Rotate-transformer layers. The input convolutional layer is used to extract shallow features of the license plate image, and the cascaded residual Rotate-transformer layers are used to extract deep semantic features of the license plate image.

[0098] By adding the shallow feature map of the current training license plate image to the deep feature map of the current training license plate image through residual connection, the feature map sum of the current training license plate image can be obtained. The feature map sum of the current training license plate image is then input into the output convolutional layer of the current generator for feature fusion, which yields the deblurred license plate image corresponding to the current training license plate image.

[0099] For example, such as Figure 3 As shown, the shallow and deep feature maps of the license plate image are added together through a residual connection module to obtain a feature map addition structure. The feature map addition result is then input into the output convolutional layer for feature fusion, which yields the generator's output, i.e., the deblurred license plate image corresponding to the currently trained license plate image.

[0100] In this embodiment, the license plate image is deblurred by a generator, which improves the clarity of the license plate image.

[0101] In an exemplary embodiment, deep features of the current training license plate image are extracted from the shallow feature map of the current training license plate image through multiple rotational attention layers cascaded with the current generator, resulting in a deep feature map of the current training license plate image, including:

[0102] S61, the input of each rotation attention layer is processed sequentially through the first normalization layer, the multi-head rotation attention layer, the second normalization layer, and the fully connected layer in each of the cascaded rotation attention layers, to obtain the deep feature map of the current training license plate image output by the fully connected layer of the last rotation attention layer of the multiple rotation attention layers; wherein, the input of the first rotation attention layer in the cascaded multiple rotation attention layers is the shallow feature map of the current training license plate image, and the input of the other rotation attention layers is the output of the previous rotation attention layer of the other rotation attention layers.

[0103] In the process of inputting the shallow feature map of the current training license plate image into multiple rotation attention layers cascaded in the current generator, the input of the first rotation attention layer in the multiple cascaded rotation attention layers is the shallow feature map of the current training license plate image, and the input of the other rotation attention layers is the output of the previous rotation attention layer.

[0104] By sequentially processing the input of each rotation attention layer through the first normalization layer, the second rotation multi-head attention layer, the third normalization layer, and the fourth fully connected layer in each of the cascaded rotation attention layers, the deep feature map of the current training license plate image can be obtained from the output of the third fully connected layer of the last rotation attention layer.

[0105] Each rotational attention layer consists of a first normalization layer, a rotational multi-head attention layer, a second normalization layer, a fully connected layer, and a residual connection module, such as... Figure 5 As shown, taking the first rotating multi-head attention layer as an example, the shallow feature map extracted from the input convolutional layer is processed sequentially through the layer normalization module, the rotating multi-head attention module, the layer normalization module, and the fully connected module in the first rotating attention layer.

[0106] Optionally, during the model training phase, the process of back-derivative can be optimized through residual connections. That is, after obtaining the generator loss, the parameters of the generator can be adjusted through residual connections.

[0107] It should be noted that the rotating multi-head self-attention module in the Rotate-transformer layer can perform attention calculations by rotating the window. That is, the image is divided into different sub-blocks, and attention calculations are performed between different sub-blocks in a clockwise order. This makes full use of the similarity between adjacent pixels and also has a good modeling effect on pixels that are far apart.

[0108] In this embodiment, the deep feature map of the license plate image is extracted from the shallow feature map of the license plate image by cascading multiple rotating attention layers, and then the blurry license plate images are deblurred, which can improve the deblurring effect of the license plate image.

[0109] In one exemplary embodiment, obtaining a training license plate image set and a labeled license plate image corresponding to each training license plate image in the training license plate image set includes:

[0110] S71, acquire multiple license plate image sets, wherein each license plate image set contains multiple license plate images obtained by acquiring license plate images of the same vehicle at different acquisition distances, and all license plate images in each license plate image set correspond to the same preset license plate;

[0111] S72, perform license plate recognition on each license plate image in each license plate image set to obtain the license plate recognition result corresponding to each license plate image in each license plate image set;

[0112] S73, based on the license plate recognition result corresponding to each license plate image in each license plate image set, select training license plate images and label license plate images corresponding to the selected training license plate images from each license plate image set to obtain a training license plate image set and a label license plate image corresponding to each training license plate image; wherein, the license plate indicated by the license plate recognition result corresponding to each training license plate image is different from the preset license plate corresponding to each training license plate image, and the label license plate image corresponding to each training license plate image is the same as the license plate indicated by the corresponding license plate recognition result in the license plate image set to which the training license plate image belongs.

[0113] In this embodiment, license plate images of the same vehicle can be captured at different acquisition distances for each vehicle in a group of vehicles. The multiple license plate images obtained from capturing the same license plate image are combined into a license plate image set. All license plate images in each license plate image set correspond to the same preset license plate. Here, the same preset license plate corresponding to all license plate images in each license plate image set is the license plate of the same vehicle corresponding to all license plate images in each license plate image set, thus obtaining multiple license plate image sets.

[0114] For each license plate image in each license plate image set, license plate recognition is performed to obtain the license plate recognition result corresponding to each license plate image in each license plate image set. Here, license plate recognition can be license plate number recognition. Accordingly, the license plate recognition result corresponding to each license plate image in each license plate image set is the license plate number recognition result, which is used to identify the license plate number obtained by performing license plate image recognition on each license plate image.

[0115] Based on the license plate recognition result corresponding to each license plate image in each license plate image set, license plate images whose license plates indicated by the corresponding license plate recognition result in each license plate image set are different from the preset license plates corresponding to each training license plate image can be used as training license plate images. License plate images whose license plates indicated by the corresponding license plate recognition result in the license plate image set to which each training license plate image belongs are the same as the preset license plates corresponding to each training license plate image can be used as the label license plate images corresponding to each training license plate image.

[0116] Optionally, license plate number recognition is performed on each license plate image in each license plate image set to obtain the license plate number of each license plate image. The license plate number corresponding to each license plate image is compared with the license plate number of a preset license plate. The license plate images in each license plate image set whose corresponding license plate number is different from the preset license plate number are used as training license plate images, and the license plate images whose corresponding license plate number is the same as the preset license plate number are used as the training license plate images corresponding to each training license plate image.

[0117] In this embodiment, based on the license plate recognition results of each license plate image in each license plate image set, each license plate image is determined to be either a training license plate image or a labeled license plate image, which can improve the reliability of the license plate deblurring model.

[0118] The processing method for license plate images in the embodiments of this application will be explained below with reference to optional examples.

[0119] To determine how to deblur blurry license plate images, this optional example provides a license plate deblurring scheme based on a transformer generative adversarial network. Combined with... Figure 6 The process of processing the license plate image in this optional example may include the following steps:

[0120] Step S602: Collect clear and blurry license plate data to construct a training set.

[0121] Step S604: Use the blurred license plate as input and the clear license plate as label to train the license plate deblurring model.

[0122] Step S606: Training of the discriminator-assisted license plate deblurring model.

[0123] This optional example demonstrates how a license plate deblurring model can be trained by using a blurred license plate as input and a clear license plate as a label. By inputting the blurred license plate image into the trained model, a clearer image can be obtained. This can solve the problem of difficult recognition of highly blurred license plate data in complex scenarios, thereby improving the accuracy of the vehicle recognition system.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0125] This embodiment also provides a license plate image processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0126] Figure 7 This is a structural block diagram of an optional license plate image processing apparatus according to an embodiment of this application, such as... Figure 7 As shown, the device includes:

[0127] The acquisition unit 702 is used to acquire a training license plate image set and a label license plate image corresponding to each training license plate image in the training license plate image set. Each training license plate image and the label license plate image corresponding to each training license plate image are different license plate images obtained by taking pictures of the same license plate. The resolution of each training license plate image is less than the resolution of the label license plate image corresponding to each training license plate image.

[0128] The training unit 704, connected to the acquisition unit 702, is used to train the initial license plate deblurring model using each training license plate image and the corresponding labeled license plate image to obtain the trained license plate deblurring model. The license plate deblurring model is a generative adversarial network-based model for image deblurring.

[0129] In one exemplary embodiment, the license plate deblurring model includes a generator; the above-described apparatus further includes:

[0130] The input unit is used to input the license plate image to be recognized into the generator of the license plate deblurring model, and obtain the deblurred license plate image to be recognized output by the generator of the license plate deblurring model;

[0131] The processing unit is used to perform license plate recognition processing on the deblurred license plate image input to obtain the license plate recognition result corresponding to the license plate image.

[0132] In one exemplary embodiment, the training unit includes:

[0133] The training module is used to train the license plate deblurring model multiple times using each training license plate image as the current training license plate image, resulting in a trained license plate deblurring model. This model includes a generator and a discriminator. When training the model using the current training license plate image, the generator of the current license plate deblurring model is called the current generator, and the discriminator is called the current discriminator.

[0134] Input the current training license plate image into the current generator to obtain the deblurred license plate image output by the current generator;

[0135] The deblurred license plate image and the labeled license plate image corresponding to the current training license plate image are input into the current discriminator to obtain the image discrimination result output by the current discriminator. The image discrimination result is used to represent the image similarity between the deblurred license plate image and the labeled license plate image.

[0136] Based on the deblurred license plate image, the labeled license plate image, and the image discrimination result, at least one of the parameters of the current generator and the current discriminator is adjusted to obtain the adjusted license plate deblurring model.

[0137] In one exemplary embodiment, the training module includes:

[0138] The determination submodule is used to determine the model loss of the current license plate deblurring model based on the deblurred license plate image, the tagged license plate image, and the image discrimination results.

[0139] The adjustment submodule is used to adjust the parameters of at least one of the current generator and the current discriminator based on the model loss of the current license plate deblurring model, so as to obtain the adjusted license plate deblurring model, wherein the model loss of the adjusted license plate deblurring model is less than the model loss of the current license plate deblurring model.

[0140] In one exemplary embodiment, determining the submodule includes:

[0141] The first determining subunit is used to determine the Manhattan distance between the deblurred license plate image and the labeled license plate image as the first model loss corresponding to the current generator;

[0142] The second determining subunit is used to determine the binary cross-entropy loss between the image discrimination result and the labeled license plate image as the second model loss corresponding to the current discriminator.

[0143] The summation subunit is used to perform a weighted summation of the first model loss and the second model loss to obtain the model loss of the current license plate deblurring model.

[0144] In one exemplary embodiment, the training module includes:

[0145] The input submodule is used to input the current training license plate image into the input convolutional layer of the current generator to obtain the shallow feature map of the current training license plate image extracted by the input convolutional layer;

[0146] The extraction submodule is used to extract the deep features of the current training license plate image from the shallow feature map of the current training license plate image through multiple rotation attention layers cascaded by the current generator, so as to obtain the deep feature map of the current training license plate image.

[0147] The processing submodule is used to add the shallow feature map and the deep feature map of the current training license plate image through residual connection to obtain the feature map addition result of the current training license plate image.

[0148] The fusion submodule is used to input the sum of the feature maps of the current training license plate image into the output convolutional layer of the current generator for feature fusion, so as to obtain the deblurred license plate image corresponding to the current training license plate image.

[0149] In one exemplary embodiment, the extraction submodule includes:

[0150] The processing unit is used to process the input of each rotation attention layer sequentially through the first normalization layer, the rotation multi-head attention layer, the second normalization layer, and the fully connected layer in each of the cascaded rotation attention layers, to obtain the deep feature map of the current training license plate image output by the fully connected layer of the last rotation attention layer of the cascaded rotation attention layers; wherein, the input of the first rotation attention layer in the cascaded rotation attention layers is the shallow feature map of the current training license plate image, and the input of the other rotation attention layers is the output of the previous rotation attention layer of the other rotation attention layers.

[0151] In one exemplary embodiment, the acquisition unit includes:

[0152] The acquisition module is used to acquire multiple license plate image sets. Each license plate image set contains multiple license plate images obtained by acquiring license plate images of the same vehicle at different acquisition distances. All license plate images in each license plate image set correspond to the same preset license plate.

[0153] The recognition module is used to perform license plate recognition on each license plate image in each license plate image set, and obtain the license plate recognition result corresponding to each license plate image in each license plate image set;

[0154] The selection module is used to select training license plate images and corresponding label license plate images from each license plate image set based on the license plate recognition result corresponding to each license plate image in each license plate image set, thereby obtaining a training license plate image set and a label license plate image corresponding to each training license plate image. The license plate indicated by the license plate recognition result corresponding to each training license plate image is different from the preset license plate corresponding to each training license plate image. The label license plate image corresponding to each training license plate image is the same license plate indicated by the corresponding license plate recognition result in the license plate image set to which the training license plate image belongs as the corresponding preset license plate.

[0155] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0156] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute program code for any of the license plate image processing methods described above in the embodiments of this application.

[0157] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0158] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0159] S1, acquire a training license plate image set and a label license plate image corresponding to each training license plate image in the training license plate image set, wherein each training license plate image and the label license plate image corresponding to each training license plate image are different license plate images obtained by taking pictures of the same license plate, and the resolution of each training license plate image is less than the resolution of the label license plate image corresponding to each training license plate image.

[0160] S2, use each training license plate image and the corresponding labeled license plate image to train the initial license plate deblurring model to obtain the trained license plate deblurring model, wherein the license plate deblurring model is a generative adversarial network-based model for image deblurring.

[0161] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0162] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0163] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described license plate image processing method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0164] Figure 8 This is a structural block diagram of an optional electronic device according to an embodiment of this application, such as... Figure 8 As shown, it includes a processor 802, a communication interface 804, a memory 806, and a communication bus 808. The processor 802, communication interface 804, and memory 806 communicate with each other via the communication bus 808.

[0165] Memory 806 is used to store computer programs;

[0166] When processor 802 executes a computer program stored in memory 806, it performs the following steps:

[0167] S1, acquire a training license plate image set and a label license plate image corresponding to each training license plate image in the training license plate image set, wherein each training license plate image and the label license plate image corresponding to each training license plate image are different license plate images obtained by taking pictures of the same license plate, and the resolution of each training license plate image is less than the resolution of the label license plate image corresponding to each training license plate image.

[0168] S2, use each training license plate image and the corresponding labeled license plate image to train the initial license plate deblurring model to obtain the trained license plate deblurring model, wherein the license plate deblurring model is a generative adversarial network-based model for image deblurring.

[0169] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 8The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices.

[0170] The aforementioned memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0171] As an example, the memory 806 described above may include, but is not limited to, the acquisition unit 702 and the training unit 704 in the control device of the aforementioned device. Furthermore, it may include, but is not limited to, other module units in the control device of the aforementioned device, which will not be elaborated upon in this example.

[0172] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0173] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0174] Those skilled in the art will understand that Figure 8 The structure shown is for illustrative purposes only. The device that implements the above-described license plate image processing method can be a terminal device, such as a smartphone (e.g., an Android phone, an iOS phone), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal devices. Figure 8 This does not limit the structure of the aforementioned electronic device. For example, the electronic device may also include components that are more... Figure 8 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 8 The different configurations shown.

[0175] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0176] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0177] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0178] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0182] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method of processing a license plate image, characterized by, The method comprises: obtaining a set of training license plate images and a labeled license plate image corresponding to each training license plate image in the set of training license plate images, wherein the each training license plate image and the labeled license plate image corresponding to the each training license plate image are different license plate images taken of the same license plate, and the clarity of the each training license plate image is less than that of the labeled license plate image corresponding to the each training license plate image; training an initial license plate deblurring model using the each training license plate image and the labeled license plate image corresponding to the each training license plate image to obtain a trained license plate deblurring model, comprising: performing multiple rounds of model training on the license plate deblurring model to be trained using the each training license plate image as a current training license plate image, respectively, to obtain the trained license plate deblurring model, wherein the license plate deblurring model comprises a generator and a discriminator, and when the current training license plate image is used for model training, the generator of the current license plate deblurring model is a current generator, and the discriminator of the license plate deblurring model is a current discriminator; inputting the current training license plate image into the current generator to obtain a deblurred license plate image output by the current generator; inputting the deblurred license plate image and the labeled license plate image corresponding to the current training license plate image into the current discriminator to obtain an image discrimination result output by the current discriminator, wherein the image discrimination result is used to represent the image similarity between the deblurred license plate image and the labeled license plate image; and adjusting the parameters of at least one of the current generator and the current discriminator according to the deblurred license plate image, the labeled license plate image, and the image discrimination result to obtain an adjusted license plate deblurring model; wherein the license plate deblurring model is a model based on a generative adversarial network and used for image deblurring; the inputting the current training license plate image into the current generator to obtain a deblurred license plate image output by the current generator comprises: inputting the current training license plate image into an input convolutional layer of the current generator to obtain a shallow feature map of the current training license plate image extracted by the input convolutional layer; extracting deep features of the current training license plate image from the shallow feature map of the current training license plate image through a plurality of rotation attention layers cascaded in the current generator to obtain a deep feature map of the current training license plate image; performing addition processing on the shallow feature map of the current training license plate image and the deep feature map of the current training license plate image through residual connection to obtain a feature map addition result of the current training license plate image; and inputting the feature map addition result of the current training license plate image into an output convolutional layer of the current generator for feature fusion to obtain the deblurred license plate image corresponding to the current training license plate image.

2. The method of claim 1, wherein, The license plate deblurring model comprises a generator; and the method further comprises: input the to-be-recognized license plate image into the license plate deblurring model generator to obtain a deblurred to-be-recognized license plate image output by the license plate deblurring model generator; perform license plate recognition processing on the deblurred to-be-recognized license plate image to obtain a license plate recognition result corresponding to the to-be-recognized license plate image.

3. The method of claim 1, wherein, The adjusting the parameters of at least one of the current generator and the current discriminator according to the deblurred license plate image, the labeled license plate image, and the image discrimination result to obtain an adjusted license plate deblurring model includes: determining a model loss of the current license plate deblurring model according to the deblurred license plate image, the labeled license plate image, and the image discrimination result; adjusting the parameters of at least one of the current generator and the current discriminator according to the model loss of the current license plate deblurring model to obtain an adjusted license plate deblurring model, wherein the model loss of the adjusted license plate deblurring model is less than the model loss of the current license plate deblurring model.

4. The method of claim 3, wherein, The determining the model loss of the current license plate deblurring model according to the deblurred license plate image, the labeled license plate image, and the image discrimination result includes: determining a first model loss corresponding to the current generator as a Manhattan distance between the deblurred license plate image and the labeled license plate image; determining a second model loss corresponding to the current discriminator as a binary cross-entropy loss between the image discrimination result and the labeled license plate image; performing weighted summation on the first model loss and the second model loss to obtain the model loss of the current license plate deblurring model.

5. The method of claim 1, wherein, The extracting the deep-level feature of the current training license plate image from the shallow-level feature map of the current training license plate image through the plurality of rotation attention layers cascaded by the current generator includes: processing an input of each rotation attention layer in the plurality of rotation attention layers cascaded in sequence through a first layer normalization layer, a rotation multi-head attention layer, a second layer normalization layer, and a full connection layer of the each rotation attention layer to obtain a deep-level feature map of the current training license plate image output by a full connection layer of a last rotation attention layer of the plurality of rotation attention layers, wherein an input of a first rotation attention layer in the plurality of rotation attention layers cascaded is the shallow-level feature map of the current training license plate image, and an input of each rotation attention layer other than the first rotation attention layer is an output of a previous rotation attention layer of the each rotation attention layer.

6. The method according to any one of claims 1 to 5, characterized in that, The obtaining a training license plate image set and a labeled license plate image corresponding to each training license plate image in the training license plate image set includes: obtaining a plurality of license plate image sets, wherein each license plate image set in the plurality of license plate image sets contains a plurality of license plate images of a same vehicle obtained by license plate image acquisition at different acquisition distances, and all license plate images in each license plate image set correspond to a same preset license plate. perform license plate recognition on each license plate image in each of the license plate image sets respectively to obtain a license plate recognition result corresponding to each license plate image in each of the license plate image sets; select a training license plate image and a label license plate image corresponding to the training license plate image from each of the license plate image sets according to the license plate recognition result corresponding to each license plate image in each of the license plate image sets, to obtain the training license plate image set and the label license plate image corresponding to each training license plate image; wherein the license plate indicated by the license plate recognition result corresponding to each training license plate image is different from a preset license plate corresponding to each training license plate image, and the label license plate image corresponding to each training license plate image is a license plate image in the license plate image set to which the training license plate image belongs, for which the license plate indicated by the license plate recognition result is the same as the corresponding preset license plate.

7. An apparatus for processing a license plate image, characterized by comprising: comprise: an acquisition unit configured to acquire a training license plate image set and a label license plate image corresponding to each training license plate image in the training license plate image set, wherein the training license plate image and the label license plate image corresponding to the training license plate image are different license plate images obtained by photographing the same license plate, and the clarity of the training license plate image is less than the clarity of the label license plate image corresponding to the training license plate image; a training unit configured to perform model training on an initial license plate deblurring model using the training license plate image and the label license plate image corresponding to the training license plate image to obtain a trained license plate deblurring model, comprising a training module configured to perform multiple rounds of model training on the license plate deblurring model to be trained using the training license plate image as a current training license plate image respectively to obtain the trained license plate deblurring model, wherein the license plate deblurring model comprises a generator and a discriminator, when the current training license plate image is used for model training, the generator of the current license plate deblurring model is a current generator, and the discriminator of the license plate deblurring model is a current discriminator; inputting the current training license plate image into the current generator to obtain a deblurred license plate image output by the current generator; inputting the deblurred license plate image and the label license plate image corresponding to the current training license plate image into the current discriminator to obtain an image discrimination result output by the current discriminator, wherein the image discrimination result is used to represent the image similarity between the deblurred license plate image and the label license plate image; and adjusting the parameters of at least one of the current generator and the current discriminator according to the deblurred license plate image, the label license plate image, and the image discrimination result to obtain an adjusted license plate deblurring model; wherein the license plate deblurring model is a model based on a generative adversarial network and used for image deblurring. The training module comprises: an input submodule configured to input the current training license plate image to an input convolutional layer of the current generator to obtain a shallow feature map of the current training license plate image extracted by the input convolutional layer; an extraction submodule configured to extract deep features of the current training license plate image from the shallow feature map of the current training license plate image through a plurality of rotation attention layers cascaded by the current generator to obtain a deep feature map of the current training license plate image; a processing submodule configured to add the shallow feature map of the current training license plate image and the deep feature map of the current training license plate image through residual connection to obtain a feature map addition result of the current training license plate image; and a fusion submodule configured to input the feature map addition result of the current training license plate image to an output convolutional layer of the current generator for feature fusion to obtain the deblurred license plate image corresponding to the current training license plate image.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6. 9.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the computer program to perform the method in any one of claims 1 to 6.

Images