License plate recognition method and device and storage medium

Abstract

The invention relates to a license plate recognition method and device and a storage medium. The license plate recognition method comprises the following steps: acquiring to-be-recognized license plate data; and calling a license plate recognition model, and based on a plurality of recognition sub-models in the license plate recognition model, recognizing the to-be-recognized license plate data to obtain a plurality of license plate recognition results, the plurality of license plate recognition results being recognition results of the same license plate in different dimensions. According to the invention, the license plate recognition results of multiple dimensions can be obtained according to the same input license plate data, and the precision and efficiency of license plate recognition are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition, and in particular to a license plate recognition method, device and storage medium. Background Art

[0002] The license plate, as the "identity card" of the car, is a credential that can confirm the identity of the car. In daily life, there are many scenarios where the license plate information needs to be confirmed, such as scanning the license plate information to confirm the vehicle entry and exit when entering and exiting the parking lot, capturing illegal driving on the highway, and capturing illegal parking on the roadside. Therefore, license plate recognition technology is a very important technology.

[0003] In the related art, during the process of license plate recognition, only the license plate number can be extracted. When faced with different types of license plates, the license plate information cannot be accurately obtained. For incomplete license plates or objects similar to license plates, the license plate recognition technology in the related art is prone to misjudgment and the license plate information recognition is not accurate enough. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a license plate recognition method, device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a license plate recognition method is provided, comprising: obtaining license plate data to be recognized; calling a license plate recognition model, and recognizing the license plate data to be recognized based on multiple recognition sub-models in the license plate recognition model to obtain multiple license plate recognition results, wherein the multiple license plate recognition results are recognition results of different dimensions of the same license plate.

[0006] In one embodiment, the license plate data to be identified based on the multiple recognition sub-models in the license plate recognition model to obtain multiple license plate recognition results includes: inputting the license plate data to be identified into the license plate recognition model, and extracting the first feature map of the license plate data to be identified by the first sub-network model of the license plate recognition model; inputting the first feature map into the multiple recognition sub-models in the second sub-network model of the license plate recognition model, and identifying the license plate data to be identified based on the multiple recognition sub-models in the second sub-network model to obtain multiple license plate recognition results.

[0007] In one embodiment, the first feature map is input into multiple recognition sub-models in the second sub-network model of the license plate recognition model to obtain multiple recognition results respectively, and the license plate recognition result is determined, including: inputting the first feature map into the first recognition sub-model among the multiple recognition sub-models to obtain a license plate number recognition result; inputting the first feature map into the second recognition sub-model among the multiple recognition sub-models to obtain a license plate type recognition result; inputting the first feature map into the third recognition sub-model among the multiple recognition sub-models to obtain a license plate validity recognition result, the license plate validity recognition result characterizing whether the license plate is a real and complete license plate; using the license plate number recognition result, the license plate type recognition result and the license plate validity recognition result as the multiple license plate recognition results.

[0008] In one embodiment, the step of inputting the first feature map into the first recognition sub-model among the multiple recognition sub-models to obtain a license plate number recognition result includes: inputting the first feature map into the first recognition sub-model among the multiple recognition sub-models to obtain a second feature map corresponding to multiple characters; obtaining multiple confidence averages corresponding to the multiple license plate characters based on the second feature map; determining the highest confidence average from the multiple confidence averages; determining the multiple license plate characters corresponding to the highest confidence average to obtain the license plate number recognition result.

[0009] In one embodiment, the first feature map is input into the second recognition sub-model among the multiple recognition sub-models to obtain a license plate type recognition result, including: inputting the first feature map into the second recognition sub-model among the multiple recognition sub-models to obtain a third feature map; determining the confidence of the corresponding license plate color category and the license plate character combination category according to the third feature map; determining the highest confidence of the corresponding license plate color category and the license plate character combination category from the confidences of the corresponding license plate color category and the license plate character combination category; determining the license plate color category and the license plate character combination category corresponding to the highest confidence as the license plate type recognition result.

[0010] In one embodiment, the first feature map is input into the third recognition sub-model among the multiple recognition sub-models to obtain a license plate validity recognition result, including: inputting the first feature map into the third recognition sub-model among the multiple recognition sub-models to obtain a fourth feature map; determining validity confidence and invalidity confidence based on the fourth feature map; and using the license plate recognition result corresponding to the higher confidence between the validity confidence and the invalidity confidence as the license plate validity recognition result.

[0011] According to the second aspect of an embodiment of the present disclosure, a license plate recognition device is provided, comprising: a receiving module for acquiring license plate data to be recognized; a processing module for calling a license plate recognition model, recognizing the license plate data to be recognized based on multiple recognition sub-models in the license plate recognition model, and obtaining multiple license plate recognition results, wherein the multiple license plate recognition results are recognition results of different dimensions of the same license plate.

[0012] In one embodiment, the processing module identifies the license plate data to be identified based on the multiple recognition sub-models in the license plate recognition model to obtain multiple license plate recognition results in the following manner: inputting the license plate data to be identified into the license plate recognition model, extracting the first feature map of the license plate data to be identified by the first sub-network model of the license plate recognition model; inputting the first feature map into the multiple recognition sub-models in the second sub-network model of the license plate recognition model, identifying the license plate data to be identified based on the multiple recognition sub-models in the second sub-network model, and obtaining multiple license plate recognition results.

[0013] In one embodiment, the processing module inputs the first feature map into multiple recognition sub-models in the second sub-network model of the license plate recognition model to obtain multiple recognition results respectively and determine the license plate recognition result in the following manner: inputs the first feature map into the first recognition sub-model among the multiple recognition sub-models to obtain a license plate number recognition result; inputs the first feature map into the second recognition sub-model among the multiple recognition sub-models to obtain a license plate type recognition result; inputs the first feature map into the third recognition sub-model among the multiple recognition sub-models to obtain a license plate validity recognition result, the license plate validity recognition result characterizing whether the license plate is a real and complete license plate; and uses the license plate number recognition result, the license plate type recognition result and the license plate validity recognition result as the multiple license plate recognition results.

[0014] In one embodiment, the processing module obtains a license plate number recognition result by inputting the first feature map into the first recognition sub-model among the multiple recognition sub-models in the following manner: inputting the first feature map into the first recognition sub-model among the multiple recognition sub-models to obtain a second feature map corresponding to multiple characters; obtaining multiple confidence average values ​​corresponding to the multiple license plate characters based on the second feature map; determining the highest confidence average value from the multiple confidence average values; determining the multiple license plate characters corresponding to the highest confidence average value to obtain the license plate number recognition result.

[0015] In one embodiment, the processing module inputs the first feature map into the second recognition sub-model among the multiple recognition sub-models to obtain a license plate type recognition result in the following manner: inputs the first feature map into the second recognition sub-model among the multiple recognition sub-models to obtain a third feature map; according to the third feature map, respectively determines the confidence of the corresponding license plate color category and the license plate character combination category; from the confidences of the corresponding license plate color category and the license plate character combination category, respectively determines the highest confidence of the corresponding license plate color category and the license plate character combination category; determines the license plate color category and the license plate character combination category corresponding to the highest confidence as the license plate type recognition result.

[0016] In one embodiment, the processing module inputs the first feature map into the third recognition sub-model among the multiple recognition sub-models to obtain a license plate validity recognition result in the following manner: inputs the first feature map into the third recognition sub-model among the multiple recognition sub-models to obtain a fourth feature map; determines the validity confidence and the invalidity confidence based on the fourth feature map; and uses the license plate recognition result corresponding to the higher confidence between the validity confidence and the invalidity confidence as the license plate validity recognition result.

[0017] According to a third aspect of an embodiment of the present disclosure, a license plate recognition device is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: execute the method described in the first aspect or any one of the embodiments of the first aspect.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the method described in the first aspect or any one of the embodiments of the first aspect.

[0019] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: through multiple branch networks, different categories of recognition results can be obtained according to the feature map of the same input license plate data to be recognized, thereby improving the accuracy of license plate recognition.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] Figure 1 The figure is a flow chart of a license plate recognition method according to an exemplary embodiment.

[0023] Figure 2 The present invention is a flowchart showing a method of determining multiple license plate recognition results based on a first feature map according to an exemplary embodiment.

[0024] Figure 3a The present invention is a flowchart of determining multiple license plate recognition results according to an exemplary embodiment.

[0025] Figure 3b The diagram is a schematic diagram showing a method of determining multiple license plate recognition results according to an exemplary embodiment.

[0026] Figure 4a The present invention is a flowchart showing a method for determining a license plate number recognition result according to an exemplary embodiment.

[0027] Figure 4b The present invention is a schematic diagram of a process for determining a license plate number recognition result according to an exemplary embodiment.

[0028] Figure 5a The present invention is a flowchart of determining a license plate type recognition result according to an exemplary embodiment.

[0029] Figure 5b The present invention is a schematic diagram of a process for determining a license plate type recognition result according to an exemplary embodiment.

[0030] Figure 6a The present invention is a flowchart showing a method for determining a license plate validity recognition result according to an exemplary embodiment.

[0031] Figure 6b The present invention is a schematic diagram of a process for determining a license plate validity recognition result according to an exemplary embodiment.

[0032] Figure 7 The figure is a block diagram of a vehicle license plate recognition device according to an exemplary embodiment.

[0033] Figure 8 The invention is a block diagram of a device for license plate recognition according to an exemplary embodiment. DETAILED DESCRIPTION

[0034] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0035] The traditional license plate recognition algorithm first locates the license plate position, cuts the license plate image, and determines the license plate number by comparing the features of the license plate number. This method has poor robustness, and the accuracy of license plate recognition is very low for large-angle license plates and in bad weather. With the development of deep learning, the convolutional neural network method is now basically used for license plate recognition, and the recognition accuracy has been greatly improved. However, the current method still does not handle incomplete license plates and fake license plates well, and when recognizing license plate numbers, the efficiency of different license plate recognition is different due to different calculation processes. For example, in the related art, a neural network with a CRNN (Convolutional Recurrent Neural Network, a convolutional recurrent neural network structure) architecture can be used for license plate recognition. However, RNN (Recurrent Neural Network, recurrent neural network) is used in CRNN. ​​When recognizing license plate numbers, the license plate number recognition starts from the first digit, and then each digit is recognized in turn. When recognizing each license plate number, iterative calculations are performed based on the information of the recognized digits, resulting in reduced efficiency in license plate number recognition. Moreover, CRNN can only recognize the license plate number, but cannot effectively judge the type of license plate and whether it is a complete license plate. In the related art, CCPD (a license plate recognition algorithm model) can also be used to recognize license plates, but it cannot effectively recognize when multiple license plates appear in the picture, and it cannot accurately judge the incomplete license plates.

[0036] In view of this, a method for license plate recognition is proposed in an embodiment of the present disclosure, which recognizes the license plate to be recognized based on the feature map corresponding to the image to be recognized through multiple branch network sub-models to obtain a multi-dimensional recognition result of the license plate to be recognized.

[0037] Through the license plate recognition method proposed in the embodiment of the present disclosure, only one network model is used to meet the recognition of multiple license plate information, and a multi-dimensional recognition result of the license plate to be recognized is obtained. The multi-dimensional license plate recognition result contains more information about the license plate, which can make the license plate recognition more accurate and improve the accuracy of license plate recognition.

[0038] The following is an explanation of the process of how to perform license plate recognition.

[0039] Figure 1 FIG. 1 is a flow chart of a license plate recognition method according to an exemplary embodiment. Figure 1 As shown, the license plate recognition method includes the following steps.

[0040] In step S11, the license plate data to be identified is obtained.

[0041] In the disclosed embodiment, the license plate data to be identified is the processed cropped license plate area image data.

[0042] In step S12, the license plate recognition model is called, and based on multiple recognition sub-models in the license plate recognition model, the license plate data to be recognized is recognized to obtain multiple license plate recognition results, which are recognition results of different dimensions of the same license plate.

[0043] In the disclosed embodiment, the license plate recognition model is a trained model including a plurality of network sub-modules.

[0044] In the disclosed embodiment, the second sub-network model includes a plurality of recognition sub-models. The plurality of recognition sub-models of the second network model can be used to recognize the same license plate data to be recognized, and obtain a plurality of license plate recognition results corresponding to the license plate data to be recognized.

[0045] In one implementation, the plurality of license plate recognition results may include recognition results of license plate type, license plate number, license plate validity, or other dimensions.

[0046] In one implementation, the same license plate data to be identified is identified through multiple identification sub-models to obtain identification results of multi-dimensional license plate information of the same license plate data, thereby improving the accuracy of license plate recognition by the license plate recognition model.

[0047] In the above embodiment, multiple recognition sub-network models are used to obtain recognition results of different dimensions of the same license plate according to the first feature graph of the license plate data to be recognized. The following is a description of the process of determining multiple license plate recognition results based on the first feature graph.

[0048] Figure 2 The present invention is a flowchart showing a method of determining multiple license plate recognition results based on a first feature map according to an exemplary embodiment.

[0049] In step S21, the license plate data to be recognized is input into the license plate recognition model, and the first feature map of the license plate data to be recognized is extracted by the first sub-network model of the license plate recognition model.

[0050] The first feature map includes the color features, texture features, shape features and spatial relationship features of the license plate image data to be recognized. The first feature map corresponds to the input license plate data to be recognized, and the first feature map is used to obtain multiple license plate recognition results in the following embodiments.

[0051] In step S22, the first feature map is input into multiple recognition sub-models in the second sub-network model of the license plate recognition model, and the license plate data to be recognized is recognized based on the multiple recognition sub-models in the second sub-network model to obtain multiple license plate recognition results.

[0052] In the disclosed embodiment, the first feature map corresponding to the same license plate data to be identified is identified through multiple recognition sub-models in a single license plate recognition model, so as to obtain the recognition result of multi-dimensional license plate information of the same license plate data, thereby improving the efficiency of license plate recognition.

[0053] In one embodiment, the processed license plate data of the cropped license plate area is obtained and input into the license plate recognition model. The first sub-network model of the license plate recognition model outputs a first feature map according to the input data to be recognized, and inputs the first feature map into multiple recognition sub-models in the second network sub-model.

[0054] In one implementation, the license plate data to be identified may be obtained by processing an open source algorithm. The license plate data to be identified may be processed by using an open source CenterNet, Yolo v8 or other open source detection algorithms to detect the location of the license plate, and crop the license plate area to obtain a processed license plate area image as the license plate data to be identified. The size of the cropped license plate area is determined according to actual task requirements.

[0055] In one implementation, the license plate recognition model may be composed of a first sub-network model and three recognition sub-models. The first sub-network model may use ResNet18, MobileNet or other lightweight networks. The number of recognition sub-models may be determined according to task requirements.

[0056] In the above embodiment, the first sub-network model outputs a first feature map corresponding to the input license plate data according to the input license plate data, and inputs the first feature map into multiple sub-recognition models in the second sub-network model to obtain multiple license plate recognition results. The following describes the process of determining multiple license plate recognition results.

[0057] Figure 3a The present invention is a flowchart of determining multiple license plate recognition results according to an exemplary embodiment.

[0058] In step S31, the first feature map is input into the first recognition sub-model among the plurality of recognition sub-models to obtain a license plate number recognition result.

[0059] In the disclosed embodiment, the license plate number recognition result includes the number of digits in the license plate number and the character corresponding to each digit.

[0060] In step S32, the first feature map is input into a second recognition sub-model among the plurality of recognition sub-models to obtain a license plate type recognition result.

[0061] In the disclosed embodiment, the license plate type recognition result includes information such as the license plate character combination and the license plate color distribution.

[0062] In step S33, the first feature map is input into the third recognition sub-model among the multiple recognition sub-models to obtain a license plate validity recognition result, which represents whether the license plate is a real and complete license plate.

[0063] In the disclosed embodiment, the license plate validity recognition result is obtained to determine whether the license plate recognition result is valid. If it is valid, it is determined to be a real complete license plate. If it is invalid, it is determined not to be a real complete license plate.

[0064] In step S34, the license plate number recognition result, the license plate type recognition result and the license plate validity recognition result are taken as a plurality of license plate recognition results.

[0065] In the disclosed embodiment, by integrating the license plate number recognition result, the license plate type recognition result and the license plate validity recognition result, the license plate recognition result contains more information, making the license plate recognition result more accurate.

[0066] In one implementation, after receiving the first feature graph output by the first sub-network model, multiple recognition sub-models in the second sub-network model can obtain the license plate number recognition result, the license plate type recognition result, and the license plate validity recognition result respectively according to the same first feature graph. By synchronously outputting recognition results of multiple dimensions according to the input of the same feature graph, the efficiency and accuracy of license plate recognition are improved.

[0067] Figure 3b The diagram is a schematic diagram showing a method of determining multiple license plate recognition results according to an exemplary embodiment.

[0068] In one embodiment, if Figure 3b As shown, for example, an image of a 6-digit B-type real license plate "AAAAAA" to be identified is input into the first sub-network model to obtain a first feature map. The first feature map is input into the first recognition sub-model in the second sub-network model for license plate number recognition to obtain a recognition result of "AAAAAA". The first feature map is input into the second recognition sub-model in the second sub-network model for license plate type recognition to obtain a recognition result of "B type". The first feature map is input into the third recognition sub-model in the second sub-network model for license plate type recognition to obtain a recognition result of "real license plate". Among them, A can be a license plate number of any characters, the license plate to be identified can also be a license plate to be identified with any number of digits other than 6 digits, and type B is a random type. Among them, "real license plate" represents the result of the valid judgment of the license plate, which is not a fixed judgment output, and other descriptions can also be used. When the validity of the license plate is judged to be invalid, the optional output "invalid license plate" indicates an invalid judgment result, and other descriptions can also be used.

[0069] The above embodiment describes the contents of multiple license plate recognition results. The following describes a process of determining a license plate number recognition result among multiple license plate recognition results.

[0070] Figure 4a The present invention is a flowchart showing a method for determining a license plate number recognition result according to an exemplary embodiment.

[0071] In step S41, the first feature map is input into a first recognition sub-model among a plurality of recognition sub-models to obtain a second feature map corresponding to a plurality of characters.

[0072] In the disclosed embodiment, the second feature map is the output of the first recognition sub-model based on the first feature map, including the color features, texture features, shape features and spatial relationship features of the license plate number data image to be recognized. The second feature map contains different information from the first feature map.

[0073] In step S42, a plurality of confidence average values ​​corresponding to a plurality of license plate characters are obtained according to the second feature map.

[0074] In the disclosed embodiment, the multiple confidences corresponding to the individual license plate characters include the confidences corresponding to the multiple character categories in the individual license plate characters. Since the multiple license plate characters include multiple individual license plate characters, the multiple confidences corresponding to the multiple license plate characters include multiple confidences corresponding to the multiple license plate characters. Based on the multiple confidences corresponding to the multiple license plate characters, the multiple confidence averages corresponding to the multiple license plate characters can be obtained. The confidence average represents the similarity between each digit in the license plate number and each digit of the license plate number to be identified.

[0075] In step S43, the highest confidence average value is determined from the plurality of confidence average values.

[0076] In the disclosed embodiment, the higher the average confidence value, the closer it is to the license plate number to be identified. Determining the highest average confidence value is to determine the confidence value corresponding to the license plate number recognition result that is most similar to the license plate data to be identified.

[0077] In step S44, a plurality of license plate characters corresponding to the highest confidence average value are determined to obtain a license plate number recognition result.

[0078] In the disclosed embodiment, the license plate number with the highest average confidence value is output as the license plate recognition result. At this time, the output license plate number is the most accurate, that is, the license plate number is closest to the license plate number of the license plate to be recognized.

[0079] In one embodiment, for example, when the license plate to be identified consists of 8 characters, the 8 characters of the license plate to be identified are predicted by the first recognition sub-model according to the second feature map. Each character of the license plate to be identified can include 71 categories of numbers, letters, and the abbreviations of the corresponding provinces, so the recognition sub-model can output 71x8 dimensions of confidence, which is used to predict which character category each of the 8 positions belongs to, thereby obtaining the number of the overall license plate. Among them, the number of characters of the license plate to be identified can be other numbers of characters, and the character category of each character of the license plate to be identified can be other types of categories.

[0080] Figure 4b The present invention is a schematic diagram of a process for determining a license plate number recognition result according to an exemplary embodiment.

[0081] like Figure 4b As shown, Reshape is a matrix conversion function, shape <2> Indicates that the matrix conversion function outputs a two-dimensional matrix, shape <3> Indicates that the matrix conversion function outputs a three-dimensional matrix. B<568×512> indicates a 568×512 B matrix. <568> Represents a 568×1 C matrix.

[0082] In one embodiment, if Figure 4b As shown, for example, when recognizing an 8-digit license plate number, after obtaining the first feature map through the first sub-network model, the first feature map is input into the second recognition sub-model. The hidden layer output is obtained through the activation function, the hidden layer output is globally averaged pooled, and then the result of the global average pooling is matrix-converted to obtain the B matrix. The B matrix and the C matrix are subjected to a general matrix multiplication calculation, and the calculation result is matrix-converted to obtain a 1×71×8 matrix. Each 71×1 represents the classification of a license plate character, and 8 represents that the number of license plate characters is 8. Among them, the B matrix and the C matrix can select other types of matrices according to actual needs, and the 1×71×8 matrix can also use other types of matrices according to task requirements to adapt to the recognition of different license plate numbers. The confidence of 71x8 dimensions is determined according to the 1×71×8 matrix, and multiple license plate characters corresponding to the highest confidence average value are determined as the recognition result of the license plate number.

[0083] In addition to determining the license plate number recognition result mentioned in the above embodiment, it is also necessary to determine the license plate type recognition result among the multiple license plate recognition results. How to determine the license plate type recognition result is described below.

[0084] Figure 5a The present invention is a flowchart of determining a license plate type recognition result according to an exemplary embodiment.

[0085] In step S51, the first feature map is input into a second recognition sub-model among the plurality of recognition sub-models to obtain a third feature map.

[0086] In the disclosed embodiment, the third feature map is used for license plate type recognition, including color features, texture features, shape features and spatial relationship features of the image data of the license plate type to be recognized. The features of the third feature map are different from those of the second feature map and the first feature map.

[0087] In step S52, the confidence levels of the corresponding license plate color categories and license plate character combination categories are determined respectively according to the third feature map.

[0088] In the disclosed embodiment, the confidence level of the license plate color category represents the similarity between the recognition result and the color of the license plate to be recognized, and the confidence level of the license plate character combination category represents the similarity between the recognition result and the character combination of the license plate to be recognized. The character combination includes information such as the arrangement position and size of numbers, letters, symbols, Chinese characters, and other characters.

[0089] In step S53, the highest confidences of the corresponding license plate color categories and license plate character combination categories are determined respectively from the confidences of the corresponding license plate color categories and license plate character combination categories.

[0090] In the disclosed embodiment, the license plate color category and license plate character combination category corresponding to the highest confidence level are most similar to the license plate color and license plate character combination of the license plate to be identified.

[0091] In step S54, the license plate color category and the license plate character combination category corresponding to the highest confidence are determined as the license plate type recognition result.

[0092] In the disclosed embodiment, the license plate color category and the license plate character combination category with the highest similarity to the license plate type of the input license plate data to be identified are used as the license plate type recognition result to achieve recognition of the license plate type.

[0093] Figure 5b The present invention is a schematic diagram of a process for determining a license plate type recognition result according to an exemplary embodiment.

[0094] like Figure 5b As shown, Reshape is a matrix conversion function, shape <2> Indicates that the matrix conversion function outputs a two-dimensional matrix.

[0095] In one embodiment, if Figure 5bAs shown, for example, when the license plate type is identified, the license plate types include 6 categories: ordinary blue license plate, ordinary yellow license plate, new energy license plate, double-layer yellow license plate, white license plate, and black license plate. After the first feature map is obtained through the first sub-network model, the first feature map is input into the third recognition sub-model. The hidden layer output is obtained through the activation function, the hidden layer output is globally averaged pooled, and then the result of the global average pooling is matrix-converted to obtain a 1×6 matrix. Among them, 6 represents the number of categories of license plate types. The confidence of each license plate type category is determined according to the 1×6 matrix, and multiple license plate characters corresponding to the highest confidence are determined as the recognition result of the license plate type. The 1×6 matrix can also use other types of matrices according to the task requirements to adapt to the recognition of different license plate types.

[0096] The following is an explanation of the process of determining the license plate validity recognition result among the multiple license plate recognition results.

[0097] Figure 6a The present invention is a flowchart showing a method for determining a license plate validity recognition result according to an exemplary embodiment.

[0098] In step S61, the first feature map is input into a third recognition sub-model among the plurality of recognition sub-models to obtain a fourth feature map.

[0099] In the disclosed embodiment, the fourth feature map includes color features, texture features, shape features and spatial relationship features of the image data of the license plate validity to be identified, and the features of the fourth feature map are different from those of the first feature map, the second feature map and the third feature map.

[0100] In step S62, the validity confidence and the invalidity confidence are determined based on the fourth characteristic graph.

[0101] In the disclosed embodiment, the validity confidence represents the probability that the license plate to be identified is a real complete license plate, and the invalidity confidence represents the probability that the license plate to be identified is not a real complete license plate.

[0102] In step S63, the license plate recognition result with a higher confidence level among the corresponding validity confidence level and invalidity confidence level is used as the license plate validity recognition result.

[0103] In the disclosed embodiment, when the validity confidence is high, it indicates that the recognized license plate is a real complete license plate. When the invalidity confidence is high, it indicates that the recognized license plate is not a real complete license plate. The license plate validity recognition result is the determination result of whether it is a real complete license plate.

[0104] Figure 6b The present invention is a schematic diagram of a process for determining a license plate validity recognition result according to an exemplary embodiment.

[0105] like Figure 6bAs shown, Reshape is a matrix conversion function, shape <2> Indicates that the matrix conversion function outputs a two-dimensional matrix.

[0106] In one embodiment, if Figure 6b As shown, for example, when the validity of a license plate is identified, when the validity of the license plate includes two categories, valid and invalid, after the first feature map is obtained through the first sub-network model, the first feature map is input into the third recognition sub-model. The hidden layer output is obtained through the activation function, the hidden layer output is globally averaged pooled, and then the result of the global average pooling is matrix-converted to obtain a 1×2 matrix. Among them, 2 represents the number of categories of license plate validity. The confidence of the validity and invalidity of each license plate is determined according to the 1×2 matrix, and the license plate recognition result corresponding to the higher confidence is determined from the validity confidence and the invalidity confidence as the recognition result of the license plate validity.

[0107] In one embodiment, during the training of the license plate recognition model, the loss functions of the three recognition sub-models all use the cross entropy loss function. Since the training difficulty of the license plate number is more difficult than the other two tasks, a weight coefficient is multiplied in front of the loss function. For example, the weight coefficient can be 3 or other values. The three recognition sub-models are optimized according to the cross entropy loss.

[0108] In one embodiment, an image containing a license plate to be identified is obtained, and the area of ​​the license plate to be identified in the image is cropped to obtain an image of the license plate to be identified. The image of the license plate to be identified is input into the first network sub-model in the license plate recognition model, and a first feature map is obtained through the first network character model. The first feature map is input into the first recognition sub-model in the second network sub-model to obtain a second feature map corresponding to multiple characters. According to the second feature map, multiple confidence averages corresponding to multiple license plate characters are obtained. Among them, the number of multiple license plate characters is determined according to actual needs. The highest confidence average is determined from the multiple confidence averages corresponding to the multiple license plate characters. Determine the multiple license plate characters corresponding to the highest confidence average, and obtain a license plate number recognition result. The first feature map is input into the second recognition sub-model in the second network sub-model to obtain a third feature map. According to the third feature map, the confidence of the corresponding license plate color category and the license plate character combination category is determined respectively. From the confidence of the corresponding license plate color category and the license plate character combination category, the highest confidence of the corresponding license plate color category and the license plate character combination category is determined respectively. Determine the license plate color category and license plate character combination category corresponding to the highest confidence as the license plate type recognition result. The number of license plate color categories and license plate character combination categories is determined according to actual needs. Input the first feature map into the third recognition sub-model among the multiple recognition sub-models to obtain a fourth feature map. According to the fourth feature map, determine the validity confidence and the invalidity confidence. The license plate recognition result with a higher confidence among the corresponding validity confidence and invalidity confidence is used as the license plate validity recognition result. In the license plate validity recognition process, there are two categories: valid or invalid.

[0109] Through the license plate recognition method in the embodiment of the present disclosure, the same backbone network model is used to obtain the license plate image features, which are then input into multiple recognition branch models, so that the license plate recognition model can extract different license plate information at the same time, thereby enhancing the robustness and accuracy of license plate recognition. And compared with using a one-to-one license plate recognition model for license plate recognition, the one-to-many license plate recognition model proposed in the embodiment of the present disclosure occupies less resources and is more efficient. By simultaneously determining the confidence of all characters in the license plate, the license plate number is recognized according to the average confidence value corresponding to all characters, and the recognition speed is faster. The license plate type is determined according to the license plate color distribution and character combination, so that the license plate type result is more accurate.

[0110] Based on the same concept, the embodiment of the present disclosure also provides a license plate recognition device.

[0111] It is understandable that the license plate recognition device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0112] Figure 7 is a block diagram of a license plate recognition device according to an exemplary embodiment. Figure 7 The device includes a receiving module 101 and a processing module 102.

[0113] The receiving module 101 is configured to obtain license plate data to be recognized.

[0114] The processing module 102 is configured to call a license plate recognition model, identify the license plate data to be recognized based on multiple recognition sub-models in the license plate recognition model, and obtain multiple license plate recognition results, which are recognition results of different dimensions of the same license plate.

[0115] In one implementation provided by the embodiments of the present disclosure, the processing module 102 identifies the license plate data to be identified based on multiple recognition sub-models in the license plate recognition model to obtain multiple license plate recognition results in the following manner: the license plate data to be identified is input into the license plate recognition model, and the first sub-network model of the license plate recognition model extracts the first feature map of the license plate data to be identified; the first feature map is input into the multiple recognition sub-models in the second sub-network model of the license plate recognition model, and the license plate data to be identified is identified based on the multiple recognition sub-models in the second sub-network model to obtain multiple license plate recognition results.

[0116] In one implementation provided by the embodiments of the present disclosure, the processing module 102 inputs the first feature map into multiple recognition sub-models in the second sub-network model of the license plate recognition model in the following manner, obtains multiple recognition results respectively, and determines the license plate recognition result: inputs the first feature map into the first recognition sub-model among the multiple recognition sub-models to obtain the license plate number recognition result; inputs the first feature map into the second recognition sub-model among the multiple recognition sub-models to obtain the license plate type recognition result; inputs the first feature map into the third recognition sub-model among the multiple recognition sub-models to obtain the license plate validity recognition result, the license plate validity recognition result characterizes whether the license plate is a real and complete license plate; the license plate number recognition result, the license plate type recognition result and the license plate validity recognition result are used as multiple license plate recognition results.

[0117] In one implementation provided by an embodiment of the present disclosure, the processing module 102 obtains a license plate number recognition result by inputting a first feature map into a first recognition sub-model among multiple recognition sub-models in the following manner: inputting the first feature map into the first recognition sub-model among multiple recognition sub-models to obtain a second feature map corresponding to multiple characters; obtaining multiple confidence averages corresponding to multiple license plate characters based on the second feature map; determining the highest confidence average from the multiple confidence averages; determining multiple license plate characters corresponding to the highest confidence average to obtain a license plate number recognition result.

[0118] In one implementation provided by the embodiments of the present disclosure, the processing module 102 inputs the first feature map into the second recognition sub-model among multiple recognition sub-models to obtain a license plate type recognition result in the following manner: inputs the first feature map into the second recognition sub-model among multiple recognition sub-models to obtain a third feature map; according to the third feature map, respectively determines the confidence of the corresponding license plate color category and the license plate character combination category; from the confidences of the corresponding license plate color category and the license plate character combination category, respectively determines the highest confidence of the corresponding license plate color category and the license plate character combination category; determines the license plate color category and the license plate character combination category corresponding to the highest confidence as the license plate type recognition result.

[0119] In one implementation provided by an embodiment of the present disclosure, the processing module 102 inputs the first feature map into a third recognition sub-model among multiple recognition sub-models to obtain a license plate validity recognition result in the following manner: inputs the first feature map into a third recognition sub-model among multiple recognition sub-models to obtain a fourth feature map; determines the validity confidence and the invalidity confidence based on the fourth feature map; and uses the license plate recognition result with a higher confidence among the corresponding validity confidence and invalidity confidence as the license plate validity recognition result.

[0120] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0121] Figure 8 2 is a block diagram of a device 200 for data classification according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0122] Reference Figure 8, the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0123] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0124] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0125] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0126] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0127] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0128] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0129] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0130] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0131] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0132] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0133] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0134] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0135] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0136] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0137] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0138] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A license plate recognition method, characterized in that: include: Get the license plate data to be recognized; The license plate recognition model is called, and based on a plurality of recognition sub-models in the license plate recognition model, the license plate data to be recognized is recognized to obtain a plurality of license plate recognition results, wherein the plurality of license plate recognition results are recognition results of different dimensions of the same license plate.

2. The method according to claim 1, characterized in that The method of identifying the license plate data to be identified based on the multiple identification sub-models in the license plate recognition model to obtain multiple license plate recognition results includes: Inputting the to-be-recognized license plate data into the license plate recognition model, and extracting a first feature graph of the to-be-recognized license plate data by a first subnetwork model of the license plate recognition model; The first feature map is input into multiple recognition sub-models in the second sub-network model of the license plate recognition model, and the license plate data to be recognized is recognized based on the multiple recognition sub-models in the second sub-network model to obtain multiple license plate recognition results.

3. The method according to claim 2, characterized in that Inputting the first feature graph into a plurality of recognition sub-models in the second sub-network model of the license plate recognition model to obtain a plurality of license plate recognition results, including: Inputting the first feature map into a first recognition sub-model among the plurality of recognition sub-models to obtain a license plate number recognition result; Inputting the first feature map into a second recognition sub-model among the plurality of recognition sub-models to obtain a license plate type recognition result; Inputting the first feature graph into a third recognition sub-model among the plurality of recognition sub-models to obtain a license plate validity recognition result, wherein the license plate validity recognition result indicates whether the license plate is a real and complete license plate; The license plate number recognition result, the license plate type recognition result and the license plate validity recognition result are used as the multiple license plate recognition results.

4. The method according to claim 3, characterized in that The step of inputting the first feature graph into a first recognition sub-model among the plurality of recognition sub-models to obtain a license plate number recognition result includes: Inputting the first feature map into a first recognition sub-model among the plurality of recognition sub-models to obtain a second feature map corresponding to a plurality of characters; According to the second feature map, obtaining a plurality of confidence average values ​​corresponding to the plurality of license plate characters; determining a highest confidence average from the plurality of confidence averages; The plurality of license plate characters corresponding to the highest confidence average value are determined to obtain the license plate number recognition result.

5. The method according to claim 3, characterized in that: Inputting the first feature map into a second recognition sub-model among the plurality of recognition sub-models to obtain a license plate type recognition result, including: Inputting the first feature map into a second recognition sub-model among the plurality of recognition sub-models to obtain a third feature map; According to the third characteristic graph, respectively determine the confidence of the corresponding license plate color category and the license plate character combination category; From the confidences of the corresponding license plate color categories and license plate character combination categories, respectively determine the highest confidences of the corresponding license plate color categories and license plate character combination categories; The license plate color category and the license plate character combination category corresponding to the highest confidence are determined as the license plate type recognition result.

6. The method according to claim 3, characterized in that: Inputting the first feature graph into a third recognition sub-model among the plurality of recognition sub-models to obtain a license plate validity recognition result, including: Inputting the first feature map into a third recognition sub-model among the plurality of recognition sub-models to obtain a fourth feature map; Determining validity confidence and invalidity confidence according to the fourth characteristic graph; The license plate recognition result corresponding to the higher confidence level between the validity confidence level and the invalidity confidence level is used as the license plate validity recognition result.

7. A license plate recognition device, characterized in that: include: A receiving module, used to obtain the license plate data to be identified; The processing module is used to call the license plate recognition model, identify the license plate data to be recognized based on multiple recognition sub-models in the license plate recognition model, and obtain multiple license plate recognition results, which are recognition results of different dimensions of the same license plate.

8. The device according to claim 7, characterized in that The processing module identifies the license plate data to be identified based on the multiple identification sub-models in the license plate recognition model to obtain multiple license plate recognition results in the following manner: Inputting the to-be-recognized license plate data into the license plate recognition model, and extracting a first feature graph of the to-be-recognized license plate data by a first subnetwork model of the license plate recognition model; The first feature map is input into multiple recognition sub-models in the second sub-network model of the license plate recognition model, and the license plate data to be recognized is recognized based on the multiple recognition sub-models in the second sub-network model to obtain multiple license plate recognition results.

9. The device according to claim 8, characterized in that The processing module inputs the first feature graph into a plurality of recognition sub-models in the second sub-network model of the license plate recognition model to obtain a plurality of recognition results respectively and determine the license plate recognition result in the following manner: Inputting the first feature map into a first recognition sub-model among the plurality of recognition sub-models to obtain a license plate number recognition result; Inputting the first feature map into a second recognition sub-model among the plurality of recognition sub-models to obtain a license plate type recognition result; Inputting the first feature graph into a third recognition sub-model among the plurality of recognition sub-models to obtain a license plate validity recognition result, wherein the license plate validity recognition result indicates whether the license plate is a real and complete license plate; The license plate number recognition result, the license plate type recognition result and the license plate validity recognition result are used as the multiple license plate recognition results.

10. The device according to claim 9, characterized in that The processing module inputs the first feature map into the first recognition sub-model among the multiple recognition sub-models to obtain the license plate number recognition result in the following manner: Inputting the first feature map into a first recognition sub-model among the plurality of recognition sub-models to obtain a second feature map corresponding to a plurality of characters; According to the second feature map, obtaining a plurality of confidence average values ​​corresponding to the plurality of license plate characters; determining a highest confidence average from the plurality of confidence averages; The plurality of license plate characters corresponding to the highest confidence average value are determined to obtain the license plate number recognition result.

11. The device according to claim 9, characterized in that The processing module inputs the first feature map into the second recognition sub-model among the multiple recognition sub-models to obtain a license plate type recognition result in the following manner: Inputting the first feature map into a second recognition sub-model among the plurality of recognition sub-models to obtain a third feature map; According to the third characteristic graph, respectively determine the confidence of the corresponding license plate color category and the license plate character combination category; From the confidences of the corresponding license plate color categories and license plate character combination categories, respectively determine the highest confidences of the corresponding license plate color categories and license plate character combination categories; The license plate color category and the license plate character combination category corresponding to the highest confidence are determined as the license plate type recognition result.

12. The device according to claim 9, characterized in that The processing module inputs the first feature map into the third recognition sub-model among the multiple recognition sub-models to obtain a license plate validity recognition result in the following manner: Inputting the first feature map into a third recognition sub-model among the plurality of recognition sub-models to obtain a fourth feature map; Determining validity confidence and invalidity confidence according to the fourth characteristic graph; The license plate recognition result corresponding to the higher confidence level between the validity confidence level and the invalidity confidence level is used as the license plate validity recognition result.

13. A data classification device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the method described in any one of claims 1 to 6.

14. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the method according to any one of claims 1 to 6.

Images