License plate deblurring method, device, electronic device and storage medium

Abstract

The present invention discloses a license plate deblurring method, device, electronic device and storage medium. The method comprises acquiring a license plate image in real time; judging whether the license plate image is blurred, if so, entering a primary deblurring mode, if not, outputting the license plate image; in the primary deblurring mode, inputting the license plate image into a preset cyclic generative adversarial network model to obtain an optimized license plate image; judging whether the optimized license plate image is clear, if so, outputting the optimized license plate image, if not, entering a secondary deblurring mode; in the secondary deblurring mode, inputting the optimized license plate image into a preset super-resolution reconstruction network model to obtain a clear license plate image; and outputting a clear license plate image. The license plate deblurring method, device, electronic device and storage medium provided by the embodiments of the present invention can perform corresponding effective processing for license plates with different blur levels, thereby reducing the resources and time required for license plate deblurring and promoting the intelligent process of license plate deblurring processing.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a license plate deblurring method, device, electronic equipment and storage medium. Background Art

[0002] With the continuous improvement of living standards, vehicles have gradually become one of the indispensable means of transportation for people to travel. License plates, as important identification credentials of vehicles, play a very important role in the recognition of information. Therefore, whether license plates can be effectively identified has been one of the key research areas of traffic management departments in recent years.

[0003] In practical applications, due to the high-speed movement of vehicles, the vehicle images collected by road cameras often show motion blur, which brings many inconveniences to traffic management, such as license plate positioning and recognition. However, the current method for dealing with blurred license plates is to directly input the collected multiple license plates into the algorithm model one by one for processing. The relevant algorithm model consumes a lot of memory, resources and time for the processing of each license plate, resulting in slow data processing, insufficient intelligent optimization, and delayed information acquisition, which is not conducive to the effective implementation of traffic law enforcement. Summary of the invention

[0004] The present invention provides a license plate deblurring method, device, electronic device and storage medium to solve the technical problem that the existing license plate deblurring method will lead to waste of resources. By configuring a multi-level fuzzy judgment and fuzzy processing method strategy, corresponding effective processing can be performed for different types of license plates, thereby reducing the resources and time required for license plate deblurring and promoting the intelligent process of license plate deblurring processing.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a license plate deblurring method, comprising:

[0006] Acquire license plate images in real time;

[0007] Determine whether the license plate image is blurred, if so, enter the first level deblurring mode, if not, output the license plate image;

[0008] In the primary deblurring mode, the license plate image is input into a preset recurrent generative adversarial network model to obtain an optimized license plate image;

[0009] Determine whether the optimized license plate image is clear, if so, output the optimized license plate image, if not, enter the secondary deblurring mode;

[0010] In the secondary deblurring mode, the optimized license plate image is input into a preset super-resolution reconstruction network model to obtain a clear license plate image;

[0011] Output the clear license plate image.

[0012] As one of the preferred solutions, the determining whether the license plate image is blurred is specifically as follows:

[0013] Based on convolutional neural network technology, a license plate fuzzy classification model is constructed;

[0014] Inputting the license plate image into the trained license plate fuzzy classification model to obtain an initial classification result;

[0015] When the initial classification result is motion blur and out-of-focus blur, the license plate image is determined to be blurred; otherwise, the license plate image is determined to be not blurred.

[0016] As one of the preferred solutions, the determination of whether the optimized license plate image is clear is specifically as follows:

[0017] Inputting the optimized license plate image into the trained license plate fuzzy classification model to obtain a binary classification result;

[0018] When the binary classification results are motion blur and out-of-focus blur, it is determined that the optimized license plate image is not clear; otherwise, it is determined that the optimized license plate image is clear.

[0019] As one preferred solution, the method further comprises:

[0020] Obtaining a primary classification loss function value and a secondary classification loss function value based on the loss function calculation method of the license plate fuzzy classification model respectively;

[0021] Updating the cyclic generative adversarial network model with the initial classification loss function value;

[0022] The super-resolution reconstruction network model is updated with the binary classification loss function value.

[0023] As one of the preferred solutions, the loss function calculation method based on the license plate fuzzy classification model obtains the initial classification loss function value and the second classification loss function value, specifically:

[0024] Respectively extracting each character in the license plate image and the optimized license plate image;

[0025] Input each character in the license plate image into the R network, calculate the loss corresponding to each character, and determine the initial classification loss function value based on the sum of the losses of each character in the license plate image;

[0026] Each character in the optimized license plate image is input into the R network, the loss corresponding to each character is calculated, and the binary classification loss function value is determined based on the sum of the losses of each character in the optimized license plate image.

[0027] As one of the preferred solutions, the structure of the R network is a Lenet5 network.

[0028] As one preferred solution, in the secondary deblurring mode, before inputting the optimized license plate image into a preset super-resolution reconstruction network model, the method further includes:

[0029] The optimized license plate image is preprocessed, wherein the preprocessing includes cropping and rotation transformation.

[0030] Another embodiment of the present invention provides a license plate deblurring device, comprising:

[0031] An acquisition module is used to acquire license plate images in real time;

[0032] A primary judgment module, used to judge whether the license plate image is blurred, if so, enter the primary deblurring mode, if not, output the license plate image;

[0033] An optimization module, used for inputting the license plate image into a preset recurrent generative adversarial network model in the primary deblurring mode to obtain an optimized license plate image;

[0034] A secondary judgment module is used to judge whether the optimized license plate image is clear, and if so, output the optimized license plate image; if not, enter the secondary deblurring mode;

[0035] A reconstruction module, used for inputting the optimized license plate image into a preset super-resolution reconstruction network model in the secondary deblurring mode to obtain a clear license plate image;

[0036] An output module is used to output the clear license plate image.

[0037] Yet another embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the license plate deblurring method as described above when executing the computer program.

[0038] Yet another embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the license plate deblurring method as described above.

[0039] Compared with the prior art, the beneficial effects of the embodiments of the present invention lie in at least one of the following: a method strategy of configuring mutually coordinated multi-level fuzzy judgment and fuzzy processing, effectively judging whether a license plate needs to be deblurred by introducing a judgment step, directly outputting the license plate image when it is judged that the input license plate image does not need to be blurred, without calling the relevant algorithm model for processing, thereby saving a large amount of memory, resources and time, and performing fuzzy judgment on the input license plate image through a multi-level fuzzy recognition strategy, respectively combining a cyclic generative adversarial network model and a super-resolution reconstruction network model, and processing the license plate image in steps, thereby optimizing the deblurring degree of the license plate image, achieving a better deblurring effect, and promoting the intelligent process of license plate deblurring processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of a license plate deblurring method in one embodiment of the present invention;

[0041] Figure 2 is a logic block diagram of a license plate deblurring method in one embodiment of the present invention;

[0042] Figure 3 is a schematic structural diagram of a license plate deblurring device in one embodiment of the present invention;

[0043] Figure 4 It is a schematic diagram of the structure of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0046] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0047] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0048] An embodiment of the present invention provides a license plate deblurring method. For details, see Figure 1 , Figure 1 The flowchart of the license plate deblurring method in one embodiment of the present invention is shown, which includes:

[0049] S1, real-time acquisition of license plate image;

[0050] S2, determining whether the license plate image is blurred, if so, entering the primary deblurring mode, if not, outputting the license plate image;

[0051] S3, in the first-level deblurring mode, inputting the license plate image into a preset cyclic generative adversarial network model (i.e., a GAN network also called a BEGAN network) to obtain an optimized license plate image;

[0052] S4, judging whether the optimized license plate image is clear, if so, outputting the optimized license plate image, if not, entering the secondary deblurring mode;

[0053] S5. In the secondary deblurring mode, inputting the optimized license plate image into a preset super-resolution reconstruction network model to obtain a clear license plate image;

[0054] S6. Output the clear license plate image.

[0055] It should be noted that the existing license plate deblurring method cannot distinguish and judge whether the license plate is blurred. In actual application scenarios, there are a large number of clear license plates that do not need to be deblurred, which will undoubtedly cause a lot of waste of resources and memory. The embodiment of the present invention is based on the BEGAN network to achieve the deblurring effect. In the prior art, the calculation of R-loss in the BEGAN network needs to first determine the coordinate information of each character in the license plate image, input each character into the R network to calculate the loss of each character, and then add the loss of all characters on the license plate as the loss function value R-loss. It can be seen from the above that the implementation of this process requires further consumption of a large amount of resources, and it is also difficult to accurately determine the position of the characters under the premise of blurring the license plate, resulting in a lot of waste of computing time. In view of this, in this embodiment, a logical judgment step is first configured, and an effective judgment is made on whether the license plate needs to be deblurred by introducing a judgment step. When it is judged that the input license plate image does not need to be blurred, the license plate image is directly output without calling the relevant algorithm model for processing, thereby saving a lot of memory, resources and time.

[0056] In addition, in practical applications, since the blur degree of license plates is relatively scattered and diverse, and the blur degree cannot be uniformly defined, the effect of removing blur by a single GAN network is not stable. Among them, the blur of some license plate images is caused by too small image size, and the GAN network cannot effectively process these blurred license plates. In this embodiment, for blurred license plates caused by too small size, a super-resolution reconstruction network model is introduced, thereby achieving effective processing of specific blur types, making up for the shortcomings of GAN network deblurring, and achieving better license plate deblurring effect.

[0057] Further, in the above embodiment, for step S2: determining whether the license plate image is blurred, specifically:

[0058] S21. Construct a license plate fuzzy classification model based on convolutional neural network technology;

[0059] S22, inputting the license plate image into the trained license plate fuzzy classification model to obtain an initial classification result;

[0060] S23. When the initial classification result is motion blur and out-of-focus blur, determine that the license plate image is blurred; otherwise, determine that the license plate image is not blurred.

[0061] In the embodiment of the present invention, a license plate blur classification model based on a small convolutional neural network is added to remove license plate blur by using a GAN network, so as to realize rapid judgment of the blurriness of the license plate. Only when the model judges that the result is blurred will the subsequent deblurring operation be performed. Otherwise, there is no need to process the license plate image and the license plate image can be directly output.

[0062] Of course, in this embodiment, the classification model of license plate blur based on the small convolutional neural network can also be used as a judgment tool multiple times to make multiple judgments on the processed vehicle image. Preferably, in the above embodiment, for step S4: judging whether the optimized license plate image is clear, specifically:

[0063] S41, inputting the optimized license plate image into the trained license plate fuzzy classification model to obtain a binary classification result;

[0064] S42. When the binary classification results are motion blur and out-of-focus blur, it is determined that the optimized license plate image is not clear; otherwise, it is determined that the optimized license plate image is clear.

[0065] It should be noted that in the above embodiment, the loss of the license plate blur classification model will also participate in the training of the GAN network model and the super-resolution reconstruction network model as a supervisory signal of the two networks, so as to more intuitively reflect the degree of license plate blur and better supervise the training of the deblurring network model. Preferably, the license plate deblurring method also includes:

[0066] Obtaining a primary classification loss function value and a secondary classification loss function value based on the loss function calculation method of the license plate fuzzy classification model respectively;

[0067] Updating the cyclic generative adversarial network model with the initial classification loss function value;

[0068] The super-resolution reconstruction network model is updated with the binary classification loss function value.

[0069] Preferably, in the above embodiment, the loss function calculation method based on the license plate fuzzy classification model obtains the initial classification loss function value and the second classification loss function value, specifically:

[0070] Respectively extracting each character in the license plate image and the optimized license plate image;

[0071] Input each character in the license plate image into the R network, calculate the loss corresponding to each character, and determine the initial classification loss function value based on the sum of the losses of each character in the license plate image;

[0072] Each character in the optimized license plate image is input into the R network, the loss corresponding to each character is calculated, and the binary classification loss function value is determined based on the sum of the losses of each character in the optimized license plate image.

[0073] For details, see Figure 2 , Figure 2Shown is a logical block diagram of a license plate deblurring method in one of the embodiments of the present invention. Of course, the cyclic generative adversarial network model, the super-resolution reconstruction network model, and the calculation method of the loss function in the embodiment of the present invention can all refer to the prior art. The setting of relevant training parameters in the model needs to be determined in combination with actual needs and specific vehicle images. The above is only one of the preferred embodiments. Without departing from the principles of the embodiments of the present invention, any other improvements and modifications made are also regarded as the scope of protection of the present invention and will not be repeated here.

[0074] Furthermore, in the above embodiment, the structure of the R network is a Lenet5 network.

[0075] As described above, in this embodiment, for the blurred license plate caused by too small size, considering that the GAN network has limited effect in removing the blur of the license plate, a small super-resolution reconstruction network model is added to further remove the blur based on the output of the GAN network. Preferably, in order to make up for the shortcomings of the small-sized license plate image, in the secondary deblurring mode, before the optimized license plate image is input into the preset super-resolution reconstruction network model, the license plate deblurring method also includes:

[0076] The optimized license plate image is preprocessed, wherein the preprocessing includes clipping and rotation transformation. Of course, the specific clipping scale and rotation transformation angle need to be combined with the actual license plate image and the adaptability of the subsequent deblurring algorithm model, which will not be repeated here.

[0077] Another embodiment of the present invention provides a license plate deblurring device. For details, see Figure 3 , Figure 3 The structure diagram of the license plate deblurring device in one embodiment of the present invention is shown, which includes:

[0078] An acquisition module 11 is used to acquire a license plate image in real time;

[0079] The primary judgment module 12 is used to judge whether the license plate image is blurred, and if so, enter the primary deblurring mode, and if not, output the license plate image;

[0080] An optimization module 13 is used to input the license plate image into a preset recurrent generative adversarial network model in the primary deblurring mode to obtain an optimized license plate image;

[0081] The secondary judgment module 14 is used to judge whether the optimized license plate image is clear, and if so, output the optimized license plate image, and if not, enter the secondary deblurring mode;

[0082] Reconstruction module 15, used for inputting the optimized license plate image into a preset super-resolution reconstruction network model in the secondary deblurring mode to obtain a clear license plate image;

[0083] The output module 16 is used to output the clear license plate image.

[0084] Another embodiment of the present invention provides an electronic device. For details, see Figure 4 , Figure 4 It is a schematic diagram of the structure of an electronic device in one embodiment of the present invention, which includes a processor 10, a memory 20, and a computer program A stored in the memory 20 and configured to be executed by the processor, and when the processor 10 executes the computer program A, the license plate deblurring method as described above is implemented.

[0085] Yet another embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the license plate deblurring method as described above.

[0086] The license plate deblurring method, device, electronic device and storage medium provided by the embodiments of the present invention have the beneficial effects of at least one of the following: it realizes the judgment of whether the input license plate is blurred or not, avoiding redundant deblurring operations; because the loss of the license plate blur classification network model intuitively reflects whether the license plate is blurred or not and the degree of blur, the loss of the license plate blur classification network model is used as a part of the GAN network model and the super-resolution reconstruction network model for supervised training and network training, thereby achieving better deblurring effect; the super-resolution reconstruction network model is added to make up for the deficiencies of the GAN network model in deblurring, thereby achieving better license plate deblurring effect.

[0087] In summary, the present invention is configured with a method strategy of multi-level fuzzy judgment and fuzzy processing that cooperates with each other. By introducing a judgment step, an effective judgment is made on whether the license plate needs to be deblurred. When it is judged that the input license plate image does not need to be blurred, the license plate image is directly output without calling the relevant algorithm model for processing, thereby saving a large amount of memory, resources and time. Fuzzy judgment is performed on the input license plate image through a multi-level fuzzy recognition strategy, and the cyclic generative adversarial network model and the super-resolution reconstruction network model are respectively combined to process the license plate image in steps, thereby optimizing the deblurring degree of the license plate image, achieving a better deblurring effect, and promoting the intelligent process of license plate deblurring processing.

[0088] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A license plate deblurring method, characterized in that: include: Acquire license plate images in real time; Determine whether the license plate image is blurred, if so, enter the first level deblurring mode, if not, output the license plate image; In the primary deblurring mode, the license plate image is input into a preset recurrent generative adversarial network model to obtain an optimized license plate image; Determine whether the optimized license plate image is clear, if so, output the optimized license plate image, if not, enter the secondary deblurring mode; In the secondary deblurring mode, the optimized license plate image is input into a preset super-resolution reconstruction network model to obtain a clear license plate image; Outputting the clear license plate image; in, The determining whether the license plate image is blurred is specifically as follows: Based on convolutional neural network technology, a license plate fuzzy classification model is constructed; Inputting the license plate image into the trained license plate fuzzy classification model to obtain an initial classification result; When the initial classification result is motion blur and out-of-focus blur, the license plate image is judged to be blurred; otherwise, the license plate image is judged to be not blurred; The determining whether the optimized license plate image is clear is specifically as follows: Inputting the optimized license plate image into the trained license plate fuzzy classification model to obtain a binary classification result; When the binary classification results are motion blur and out-of-focus blur, it is determined that the optimized license plate image is not clear; otherwise, it is determined that the optimized license plate image is clear.

2. The license plate deblurring method according to claim 1, characterized in that: The method further comprises: Obtaining a primary classification loss function value and a secondary classification loss function value based on the loss function calculation method of the license plate fuzzy classification model respectively; Updating the cyclic generative adversarial network model with the initial classification loss function value; The super-resolution reconstruction network model is updated with the binary classification loss function value.

3. The license plate deblurring method according to claim 2, characterized in that: The loss function calculation method based on the license plate fuzzy classification model obtains the initial classification loss function value and the second classification loss function value, which are specifically: Respectively extracting each character in the license plate image and the optimized license plate image; Input each character in the license plate image into the R network, calculate the loss corresponding to each character, and determine the initial classification loss function value based on the sum of the losses of each character in the license plate image; Each character in the optimized license plate image is input into the R network, the loss corresponding to each character is calculated, and the binary classification loss function value is determined based on the sum of the losses of each character in the optimized license plate image.

4. The license plate deblurring method according to claim 3, characterized in that: The structure of the R network is a Lenet5 network.

5. The license plate deblurring method according to claim 1, characterized in that: In the secondary deblurring mode, before inputting the optimized license plate image into a preset super-resolution reconstruction network model, the method further includes: The optimized license plate image is preprocessed, wherein the preprocessing includes cropping and rotation transformation.

6. A license plate deblurring device, characterized in that: include: An acquisition module is used to acquire license plate images in real time; A primary judgment module, used to judge whether the license plate image is blurred, if so, enter the primary deblurring mode, if not, output the license plate image; An optimization module, used for inputting the license plate image into a preset recurrent generative adversarial network model in the primary deblurring mode to obtain an optimized license plate image; A secondary judgment module is used to judge whether the optimized license plate image is clear, and if so, output the optimized license plate image; if not, enter the secondary deblurring mode; A reconstruction module, used for inputting the optimized license plate image into a preset super-resolution reconstruction network model in the secondary deblurring mode to obtain a clear license plate image; An output module, used for outputting the clear license plate image; in, The first-level judgment module is specifically used for: Based on convolutional neural network technology, a license plate fuzzy classification model is constructed; Inputting the license plate image into the trained license plate fuzzy classification model to obtain an initial classification result; When the initial classification result is motion blur and out-of-focus blur, the license plate image is judged to be blurred; otherwise, the license plate image is judged to be not blurred; The secondary judgment module is specifically used for: Inputting the optimized license plate image into the trained license plate fuzzy classification model to obtain a binary classification result; When the binary classification results are motion blur and out-of-focus blur, it is determined that the optimized license plate image is not clear; otherwise, it is determined that the optimized license plate image is clear.

7. An electronic device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the license plate deblurring method as claimed in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the license plate deblurring method according to any one of claims 1 to 5.

Images