A license plate recognition method, device and storage medium

Abstract

The disclosure provides a license plate recognition method, device and storage medium, the method comprising: acquiring a license plate target image with a vehicle logo; region delimiting a single character of the acquired target image; performing multi-region processing on the delimited region of the single character to obtain multi-region character recognition of the single character; comprehensively voting the multi-region character recognition result of the single character, and obtaining a recognition result of the single character according to the comprehensive voting result; obtaining a recognition result of license plate characters according to the recognition result of the single character; generating a recognized license plate result according to the obtained license plate character recognition result; recognizing the vehicle logo to obtain a vehicle brand, and associating the vehicle brand with the recognized license plate result to generate license plate information with the vehicle brand; training a license plate recognition model according to each original license plate information with the vehicle logo and the generated license plate information with the vehicle brand; and obtaining the final license plate information. The recognition method improves the accuracy of license plate character recognition.

Description

Technical Field

[0001] This disclosure relates to the field of image recognition, and more particularly to a license plate recognition method, device, and storage medium. Background Technology

[0002] With the rapid development of information technology, its application in the transportation sector is gradually increasing. License plate character recognition is a relatively basic and common task, representing a successful application of computer vision in practice. Improving the accuracy of license plate character recognition is of great significance for more complex applications in reality.

[0003] License plate recognition using artificial intelligence is a relatively mature field, utilizing convolutional neural networks for single-character extraction and recognition, as described in Chinese patent publication CN110348448A. Figure 2 As shown, Chinese license plates consist of seven characters in total, including Chinese characters, English letters, and numbers. The first character is a Chinese character, the second is an English letter, and the third to seventh characters are a combination of English letters and numbers. Figure 1 As shown, existing license plate character recognition methods first extract the region of a single character through a neural network, and then recognize the characters in that region. Different neural networks can be trained for different character types.

[0004] Currently, the accuracy rate of single-character recognition can reach 99.9% under ideal conditions, which can meet the needs of license plate character recognition in certain ideal scenarios. However, in practical applications, some special cases and scenarios cannot achieve the ideal result: 1. License plates in real-world scenarios are often dirty or worn, making them prone to misidentification; 2. For example... Figure 3 , 4 As shown, in the step of extracting a single character region, if the extracted region is not ideal, some easily confused characters cannot be accurately recognized. In practical applications, various scenarios are encountered, and ideal conditions are often not achieved, significantly reducing the accuracy of character recognition. The accuracy of single character recognition can drop to 88.7%, meaning the lower limit of single character recognition accuracy is 88.7%. Therefore, since Chinese license plates consist of 7 individual characters, the lower limit of license plate recognition accuracy is 88.7%. 7 =43.2%. Summary of the Invention

[0005] The disclosure provides a license plate recognition method, device and storage medium, which can solve the problems of misrecognition of dirty and worn license plates, unsatisfactory extraction of single character regions, misrecognition of easily confused characters and the like in actual license plate character recognition scenarios, and reduce the influence of the extracted single character regions on the final character recognition accuracy. In addition, the disclosure increases the accuracy of license plate recognition by increasing the recognition of vehicle brand information and the association with license plates. The disclosure provides the following technical solutions:

[0006] In a first aspect, the disclosure provides a license plate character recognition method, characterized in that the method comprises:

[0007] obtaining a target license plate image with a vehicle logo;

[0008] delimiting a single character of the obtained target image;

[0009] performing multi-region processing on the delimited region of the single character to obtain multi-region character recognition of the single character;

[0010] performing comprehensive voting on the multi-region character recognition result of the single character, and obtaining a recognition result of the single character according to the comprehensive voting result;

[0011] obtaining a recognition result of license plate characters according to the recognition result of the single character;

[0012] generating a recognized license plate result according to the obtained license plate character recognition result;

[0013] recognizing a vehicle brand from the vehicle logo, and associating the vehicle brand with the recognized license plate result to generate license plate information with the vehicle brand;

[0014] training a license plate recognition model according to each original license plate information with a vehicle logo and the generated license plate information with a vehicle brand;

[0015] inputting a target license plate image into the license plate recognition model for license plate recognition to obtain the final license plate information.

[0016] Preferably, the multi-region processing comprises scaling and translation of the delimited region of the single character.

[0017] Preferably, the scaling of the delimited region comprises expanding and reducing the bounding box range by a pixel distance in a certain proportion of the pixel value of the length of the bounding box side.

[0018] Preferably, the proportion is 2%-4%.

[0019] Preferably, the translation of the demarcated region comprises translation of the position of the border in multiple directions by a pixel distance in a certain proportion of the pixel value of the side length of the border.

[0020] Preferably, the proportion is 2%-4%.

[0021] Preferably, the translation in multiple directions is translation in 8 directions.

[0022] Preferably, the comprehensive voting of the multi-region character recognition result of the single character comprises:

[0023] When the multi-region character recognition result has only one category, the category is the recognition result of the single character.

[0024] When the multi-region character recognition result has n categories, the average confidence of the character result of each category is calculated The mth category character with the maximum value is taken as the recognition result of the single character, wherein n is an integer greater than 1, and m = 1, 2, …, n.

[0025] Preferably, the calculation method of the average confidence is: for the mth category character in the recognition result, the average confidence of the category in all character recognition results is calculated respectively. wherein C i represents the confidence of all character recognition results being m, and i = 1, 2, …, n.

[0026]

[0027] In a second aspect, the present disclosure provides an electronic device, comprising a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement the license plate recognition method in the first aspect.

[0028] In a second aspect, the present disclosure provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the license plate recognition method in the first aspect.

[0029] The beneficial effects of the present disclosure at least include: the method adopts the strategy of time and space for accuracy, effectively improves the accuracy of license plate character recognition under acceptable time and space loss, and has very important significance for license plate character recognition in specific scenarios, such as license plate material resolution, license plate with dirt in muddy road sections, etc.

[0030] (1) Through the method, the lower limit of the accuracy rate of single character recognition can be improved from 88.7% to 96.9%, and the lower limit of the accuracy rate of the entire license plate recognition is 96.9%. 7 = 80.2%, and the method effectively improves the lower limit of the license plate character recognition accuracy rate from 43.2% to 80.2%.

[0031] (2) The single character recognition time of the common method is 1.0ms-1.5ms, and the single character is extracted and recognized 11 times in the method. If the same network parameter setting as the common neural network is used, 11 times of character extraction and recognition require 11 times of inference calculation, and the cumulative time is 11.0ms-16.5ms. Multiple extraction and recognition of single characters will increase the time loss. In order to ensure the real-time performance of license plate character recognition, the method increases the number of samples selected by a single inference of the neural network, that is, increases the parameter Batch Size. By using the parallel computing characteristics of the neural network, the 11 times of extraction and recognition of single characters are calculated in one batch during inference, and 11 times of extraction and recognition results can be obtained by one inference calculation, so that the final result of single character recognition can be obtained, and the efficiency of character recognition is improved. The cumulative time of single character recognition of the common method is 11.0ms-16.5ms, and the recognition time of single character is 3.5ms-4.8ms under the premise of improving the accuracy, which can meet the needs of practical application.

[0032] (3) The vehicle brand information is introduced into the license plate recognition model, and the vehicle brand information and the license plate information are associated with each other, so as to improve the accuracy of license plate recognition. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description, explain the principles of the present disclosure, wherein the drawings are included to provide further understanding of the present disclosure and are incorporated in and constitute a part of the specification.

[0034] Figure 1 is a flowchart of the existing license plate character recognition process.

[0035] Figure 2 is a schematic diagram of the existing Chinese license plate.

[0036] Figure 3 is a schematic diagram of unreasonable single character region demarcation in the prior art.

[0037] Figure 4 is a schematic diagram of unreasonable single character region demarcation in the prior art.

[0038] Figure 5 is a flowchart of the license plate character recognition method according to one embodiment of the present disclosure.

[0039] Figure 6 is a schematic diagram of the expansion and scaling of the bounding box of an extracted single character according to one embodiment of the present disclosure.

[0040] Figure 7 is a schematic diagram of the 8-direction translation of the bounding box of an extracted single character according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] The present disclosure will be further described in details with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related content, and not limit the present disclosure. In addition, it should be noted that, for the convenience of description, only parts related to the present disclosure are shown in the drawings.

[0042] It should be noted that the embodiments and features in the embodiments of the present disclosure can be combined with each other without conflict. The technical solutions of the present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0043] Unless otherwise specified, the exemplary embodiments / instances shown will be understood as providing exemplary features of various details that can implement the technical concepts of the present disclosure in practice. Therefore, unless otherwise specified, the features of various embodiments / instances can be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of the present disclosure.

[0044] In the drawings, cross-hatching and / or shading are generally used to make the boundaries of adjacent components clear. Thus, unless otherwise specified, the presence of cross-hatching or shading does not convey or imply any preference or requirement for specific material, material properties, dimensions, proportions, commonality of the illustrated components, and / or any other characteristic, attribute, property, etc. of the components. In addition, in the drawings, the size and relative sizes of components can be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence can be performed in a different order than described. For example, two consecutively described processes can be performed substantially simultaneously or in reverse order to the described order. In addition, the same reference numbers represent the same components.

[0045] When a component is referred to as being "on", "connected to", or "coupled to" another component, it can be directly on, connected, or coupled to the other component, or intervening components can be present. When a component is referred to as being "directly on", "directly connected to", or "directly coupled to" another component, there are no intervening components present. For example, the term "connected" can refer to physical or electrical connection, etc., with or without intervening components.

[0046] For purposes of the description hereinafter, spatial or directional terms, such as "below", "lower", "bottom", "bottomed", "above", "upper", "top", "up", "over", and "side" (e.g., as in "sidewall") are used with reference to the orientation of the component as shown in the drawings. The spatial or directional terms are intended to encompass different orientations of the device in use, operation, and / or manufacture, depending on the particular orientation of the device in use, operation, and / or manufacture. For example, if the device in the drawings is turned over, then components described as "below" or "under" other components or features would then be oriented "above" the other components or features. Thus, the exemplary term "below" can encompass both an orientation of above and below. Moreover, the device can be otherwise oriented (e.g., rotated 90 degrees or at other orientations) and the spatial or directional terms used herein interpreted accordingly.

[0047] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, to the extent that the terms "including", "includes", "having", "has", "with", or variants thereof are used in either the detailed description and / or the claims, such terms are intended to be inclusive in a manner similar to the term "comprising". It is also noted that the terms "substantial", "approximately", and other similar terms, as used herein, are used in a relative sense and not in an absolute sense. Thus, they are used to explain approximations, measurement values, and / or provided values that will be recognized by those of ordinary skill in the art as being inherently inexact.

[0048] In the process of implementing the existing license plate recognition, the inventors find that in license plate recognition, due to the existence of license plate pollution, wear and tear and other conditions, the license plate recognition accuracy is seriously affected. Among them, the area demarcation of a single license plate character is a key factor affecting the accuracy of vehicle recognition, but there will inevitably be some errors between the selected character area and the actual position of the character in the existing character area demarcation method. These errors directly affect the accuracy of license plate image acquisition. Therefore, the current license plate recognition through conventional image preprocessing and recognition cannot meet the high-accuracy license plate recognition demand, seriously affecting the development of related license plate-based businesses.

[0049] In view of the above shortcomings, the license plate recognition method, system, device and storage medium of the present disclosure first acquire a target image; then, the single character of the acquired target image is regionally demarcated; next, the demarcated region of the single character is subjected to multi-region processing to obtain multi-region character recognition of the single character; then, the multi-region character recognition result of the single character is subjected to comprehensive voting, and according to the comprehensive voting result, the recognition result of the single character is obtained; finally, according to the recognition result of the single character, the recognition result of the license plate character is obtained. The license plate recognition method of the present disclosure can be better applied to license plate recognition tasks in complex scenes and increase the accuracy of license plate recognition.

[0050] Further, the present disclosure first subjects the character demarcated region to multi-region processing, then recognizes the characters in the multi-region, obtains the recognition result of a single character according to the voting result of multiple recognition results, and the specific implementation is as follows:

[0051] The demarcated region of the single character is subjected to expansion, reduction and translation multi-region processing to obtain multi-region character recognition of the single character; then, the multi-region character recognition result of the single character is subjected to comprehensive voting, and according to the comprehensive voting result, the recognition result of the single character is obtained;

[0052] As shown in Figure 5 The license plate recognition method of the present embodiment specifically includes the following steps:

[0053] (1) acquiring a license plate target image with a license plate;

[0054] (2) regionally demarcating a single character of the acquired target image;

[0055] (3) next, the demarcated region of the single character is subjected to expansion, reduction and translation multi-region processing to obtain multi-region character recognition of the single character;

[0056] (4) the multi-region character recognition result of the single character is subjected to comprehensive voting, and according to the comprehensive voting result, the recognition result of the single character is obtained;

[0057] (5) obtaining a license plate character recognition result according to the recognition result of the single character.

[0058] (6) generating a recognized license plate result according to the obtained license plate character recognition result;

[0059] (7) recognizing a vehicle brand to obtain a vehicle brand, and associating the vehicle brand with the recognized license plate result to generate license plate information with the vehicle brand;

[0060] (8) training a license plate recognition model according to each original license plate information with a vehicle brand and the generated license plate information with the vehicle brand;

[0061] (9) inputting a license plate target image into the license plate recognition model to perform license plate recognition, and obtaining a final license plate information.

[0062] The present disclosure takes a blue background white character license plate used by a small civilian vehicle as an example.

[0063] To reduce the influence of the extracted single character area on the final character recognition accuracy, the license plate recognition method in the present disclosure optimizes the single character recognition process based on the existing convolutional neural network for single character extraction and character recognition, specifically including:

[0064] As shown in Figure 6 , for the recognition process after the single character area is demarcated, first, the single character extraction frame is expanded and reduced. For the single character frame that has been extracted, the frame range is expanded and reduced by a pixel distance of 2%, 3% or 4% of the frame side length pixel value, respectively, to obtain two frames larger and smaller than the original frame, and the characters in the two regions are recognized again.

[0065] At the same time, as shown in Figure 7 , after the single character area is demarcated, the single character extraction frame is translated in the 8 directions of directly above, directly below, directly left, directly right, upper left, lower left, upper right, and lower right. For the single character frame that has been extracted, the position of the frame is translated in 8 directions by a pixel distance of 2%, 3% or 4% of the frame side length pixel value, to obtain 8 new frames, and the characters in the 8 frames after position translation are recognized.

[0066] By scaling and translating the single character frame area, 10 new frames can be obtained, together with the original demarcation of the single character frame area, the characters in the 11 frame areas obtained are recognized, and 11 character recognition results can be obtained.

[0067] For the above 11 single character multi-region recognition result comprehensive voting, the final character recognition result is obtained by multi-region recognition result comprehensive voting, and the specific comprehensive voting method can adopt the following method:

[0068] Let the total number of character categories in the 11 character recognition results be N (N≥1), n represent the mth (m=1, 2, …, N) category in all recognized character categories, and the final character recognition result is represented as the nth character in it. According to the value of N, there are two cases:

[0069] When N=1, it represents that there is only one category in all character recognition results, that is, n=1, and the category is the final recognition result: that is, R m =n

[0070] When N>1, it represents that there are multiple categories in all character recognition results, and the average confidence of the nth category character result needs to be calculated for decision. For the mth category character in the recognition result, the average confidence of the category in all character recognition results is calculated Where C i represents the confidence of all character recognition results for m, i=1, 2, …, n.

[0071]

[0072] N results are obtained, that is, Take the m value that can take the maximum value, that is, the final character recognition result:

[0073]

[0074] After accurately recognizing the license plate characters, the license plate information can be obtained, and the vehicle brand information generated by the license plate information is associated with each other. According to the license plate information associated with the vehicle brand information, the license plate recognition model is trained, and the target license plate information is input into the license plate recognition model for license plate recognition to obtain the final license plate information.

[0075] As can be known by those skilled in the art, the existing neural network training mode can meet the training requirements of the license plate recognition model. Taking the deep learning neural network in the prior art as an example, a deep convolutional generative adversarial network can be constructed and trained as the optimal construction method, such as constructing a deep convolutional neural network containing 3 layers as a generation model, constructing a convolutional neural network containing 3 layers as a discrimination model, using a separate alternating training method, training the generation model and the discrimination model, and forming a deep convolutional generative adversarial network for generating license plate pictures with vehicle brands by using the trained generation model and discrimination model. Of course, the above training method is only an exemplary description, and other neural network training methods in the prior art can be applied to the training and formation of the license plate recognition model.

[0076] In addition, in this embodiment, as the optimal mode, a 3-layer deep convolutional neural network is constructed, and other neural network construction modes and construction depths that can meet deep learning can also be selected and set. However, the 3-layer deep convolutional neural network is the most matched mode with the license plate recognition model with vehicle brand information.

[0077] Compared with the prior art, the present disclosure reduces the influence of the extracted single character region on the final character recognition accuracy, ensures the accuracy of the license plate recognition model through the accurate and effective character recognition method, and further increases the accuracy of the license plate recognition on the basis of not excessively increasing the recognition calculation amount by adding the additional parameter of vehicle brand information. The license plate recognition method of the present disclosure can be better applied to the license plate recognition task in the scene of dirty and worn license plates, overcome the problem of misrecognition of easily confused characters due to the unideal extraction of single character regions, and increase the accuracy of license plate recognition.

[0078] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0079] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0080] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0081] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable devices provide a process for implementing the functions described in the flowcharts Figure 1 one flowchart or multiple flowcharts and / or blocks Figure 1 one flowchart or multiple flowcharts and / or blocks

[0082] In the description of the present disclosure, the description of the terms "one embodiment / way", "some embodiments / ways", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present disclosure. In the present description, the illustrative description of the above terms does not necessarily refer to the same embodiment / way or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments / ways or examples. In addition, the person skilled in the art can combine and combine the different embodiments / ways or examples described in the present description and the features of the different embodiments / ways or examples without contradiction and combination.

[0083] In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0084] The person skilled in the art should understand that the above-mentioned embodiments are only for clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. Based on the above disclosure, other changes or modifications can also be made by those skilled in the art, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A license plate recognition method, characterized in that, The method includes: Obtain the target image of the license plate containing the car logo; Region delineation is performed on individual characters in the acquired target image; The defined region of the single character is subjected to multi-region processing, including scaling and translation, to obtain multi-region character recognition of the single character; The recognition results of the multi-region character of the single character are comprehensively voted on, and the recognition result of the single character is obtained based on the comprehensive voting result; Based on the recognition result of the single character, the recognition result of the license plate character is obtained; Based on the obtained license plate character recognition results, generate the recognized license plate result; The vehicle brand is obtained by recognizing the vehicle logo, and the vehicle brand is associated with the recognized license plate result to generate license plate information with the vehicle brand. A license plate recognition model is trained based on the original license plate information with vehicle logos and the generated license plate information with vehicle brand. The target image of the license plate is input into the license plate recognition model for license plate recognition, and the final license plate information is obtained.

2. The license plate recognition method as described in claim 1, characterized in that, Scaling the defined area includes expanding and shrinking the border range by a pixel distance of 2%-4% of the border side length pixel value.

3. The license plate recognition method as described in claim 2, characterized in that, The translation of the defined area includes translating the position of the border in multiple directions by a pixel distance of 2%-4% of the border side length pixel value.

4. The license plate recognition method as described in claim 3, characterized in that, The translation in multiple directions refers to translation in eight directions.

5. The license plate recognition method as described in claim 4, characterized in that, The process of comprehensively voting on the multi-region character recognition results of the single character, and obtaining the recognition result of the single character based on the comprehensive voting result, includes: When the multi-region character recognition result has only one category, that category is the recognition result of the single character. When the multi-region character recognition results have n categories, calculate the average confidence level of the character results for each category. The m-th category character that reaches the maximum value is taken as the recognition result of the single character, where n is an integer greater than 1, and m = 1, 2, ..., n.

6. The license plate recognition method as described in claim 5, characterized in that, The average confidence score is calculated as follows: for the m-th category character in the recognition results, calculate the average confidence score of that category in all character recognition results. Among them, C i This represents the confidence level for all character recognition results being m, where n is an integer greater than 1, m = 1, 2, ..., n, and i = 1, 2, ..., n.

7. An electronic device, characterized in that, The device includes a processor and a memory; the memory stores a program that is loaded and executed by the processor to implement the license plate recognition method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement the license plate recognition method as described in any one of claims 1 to 6.

Images