License plate recognition method and device, and computer-readable storage medium

Abstract

This invention provides a license plate recognition method and apparatus, and a computer-readable storage medium. The method includes: acquiring a license plate image to be recognized; performing character recognition on the license plate image using a target detection model to determine the character-related information of each of the recognized characters in the license plate image; identifying abnormal characters by filtering different categories of characters for abnormal categories and abnormal regions, and removing abnormal characters from multiple characters to determine candidate license plate characters; performing linear fitting based on the position information of the candidate license plate characters to determine the permutation layer number of each candidate character; and sequentially concatenating the candidate license plate characters at different permutation layers according to the order of preset permutation layers and the position information of the candidate license plate characters to determine the license plate number corresponding to the license plate image. In the above scheme, by filtering and removing abnormal characters from the recognized multiple characters, the accuracy of license plate number recognition can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a license plate recognition method and apparatus, and a computer-readable storage medium. Background Technology

[0002] Currently, most mature automatic license plate recognition systems on the market are used to identify the front license plates of vehicles, with cameras positioned relatively close to the plates. The scenarios are also relatively simple, such as parking lot license plate recognition systems. In loading and unloading areas like platforms, vehicles often reverse into the space for easier loading and unloading. Therefore, cameras primarily capture the rear license plates. Compared to the uniform single-line format of front license plates, rear license plates can be either single-line or double-line (double-line for yellow plates). Furthermore, rear license plates in logistics scenarios are often dirtier than front license plates, with the characters frequently obscured by dust. Additionally, in real logistics scenarios, because cameras are often fixed at a higher position, the captured license plates are generally small and easily obscured, and affected by lighting and angle. Consequently, conventional automatic license plate recognition systems have low accuracy in these scenarios. Summary of the Invention

[0003] This invention provides a license plate recognition method and apparatus, as well as a computer-readable storage medium, which can improve the accuracy of license plate recognition.

[0004] The technical solution of the present invention is achieved as follows:

[0005] This invention provides a license plate recognition method, the method comprising: acquiring a license plate image to be recognized; performing character recognition on the license plate image using a target detection model to determine the character-related information of each of the multiple characters identified in the license plate image; the character-related information including the position information and category of each character; identifying abnormal characters by filtering characters of different categories for abnormal categories and abnormal regions, and removing the abnormal characters from the multiple characters to determine candidate license plate characters; performing linear fitting based on the position information of the candidate license plate characters to determine the permutation layer number of each candidate character; and sequentially concatenating the candidate license plate characters at different permutation layers according to the order of the preset permutation layers and the position information of the candidate license plate characters to determine the license plate number corresponding to the license plate image.

[0006] This invention provides a license plate recognition device, which includes an acquisition unit and a determination unit, wherein...

[0007] The acquisition unit is used to acquire the license plate image to be identified;

[0008] The determining unit is used to perform character recognition on the license plate image using a target detection model, and determine the character-related information of each of the multiple characters identified in the license plate image; the character-related information includes the position information and category of each character; by filtering characters of different categories for abnormal categories and abnormal regions, abnormal characters are determined, and the abnormal characters are removed from the multiple characters to determine candidate license plate characters; based on the position information of the candidate license plate characters, linear fitting is performed to determine the permutation layer number of each candidate character; according to the order of the preset permutation layer number and the position information of the candidate license plate characters, the candidate license plate characters in different permutation layers are sequentially concatenated to determine the license plate number corresponding to the license plate image.

[0009] This invention provides a wireless network adjustment device, the wireless network adjustment device comprising:

[0010] Memory, used to store executable data instructions;

[0011] A processor is configured to execute executable instructions stored in the memory, and when the executable instructions are executed, the processor executes the license plate recognition method.

[0012] This invention provides a computer-readable storage medium, characterized in that it stores executable instructions, which, when executed by one or more processors, enable the processors to perform the license plate recognition method.

[0013] This invention provides a license plate recognition method and apparatus, and a computer-readable storage medium. The method includes: acquiring a license plate image to be recognized; performing character recognition on the license plate image using a target detection model to determine the character-related information of each of the multiple characters identified in the license plate image; the character-related information includes the position information and category of each character; identifying abnormal characters by filtering characters of different categories for abnormal categories and abnormal regions, and removing the abnormal characters from the multiple characters to determine candidate license plate characters; performing linear fitting based on the position information of the candidate license plate characters to determine the permutation layer number of each candidate character; and sequentially concatenating the candidate license plate characters at different permutation layers according to the order of the preset permutation layers and the position information of the candidate license plate characters to determine the license plate number corresponding to the license plate image. In the above scheme, by performing anomaly screening on multiple identified characters and their respective character-related information to remove abnormal characters and determine candidate license plate characters, the accuracy of candidate license plate characters can be improved. By using the position information of candidate license plate characters to determine the permutation layer number of each candidate character, and by sequentially splicing candidate license plate characters at different permutation layers to determine the license plate number corresponding to the license plate image, the accuracy of license plate number recognition can be improved. Attached Figure Description

[0014] Figure 1 This invention provides an optional flowchart of a license plate recognition method. Figure 1 ;

[0015] Figure 2a An optional recognition effect of the license plate recognition method provided in this embodiment of the invention. Figure 1 ;

[0016] Figure 2b Figure 2 shows an optional recognition effect of a license plate recognition method provided in an embodiment of the present invention.

[0017] Figure 2c An optional recognition effect of the license plate recognition method provided in this embodiment of the invention. Figure 3 ;

[0018] Figure 3 This is a schematic diagram of an optional process for a license plate recognition method according to an embodiment of the present invention;

[0019] Figure 4 An optional recognition effect of the license plate recognition method provided in this embodiment of the invention. Figure 4 ;

[0020] Figure 5 This invention provides an optional flowchart of a license plate recognition method. Figure 3 ;

[0021] Figure 6 An optional recognition effect of the license plate recognition method provided in this embodiment of the invention. Figure 4 ;

[0022] Figure 7 This invention provides an optional flowchart of a license plate recognition method. Figure 4 ;

[0023] Figure 8 This invention provides a schematic diagram of the structure of a license plate recognition device according to an embodiment of the present invention. Figure 1 ;

[0024] Figure 9 The second schematic diagram shows the structure of a license plate recognition device provided in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0026] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 This is an optional flowchart of a license plate recognition method provided by an embodiment of the present invention. Figure 1 , will combine Figure 1 The steps shown are explained.

[0027] S101. Obtain the image of the license plate to be identified.

[0028] In some embodiments of the present invention, the terminal can acquire the license plate image to be identified through monitoring video.

[0029] S102. Using the target detection model, perform character recognition on the license plate image to determine the character-related information of each of the multiple characters identified in the license plate image.

[0030] In some embodiments of the present invention, character-related information includes the position information of each character and the category of each character. The position information may be coordinates, and the embodiments of the present invention do not limit this.

[0031] In some embodiments of the present invention, the object detection model used is Yolov5, which is trained with license plate training data. Other object detection algorithms can also be used, such as faster RCNN, center_NET, etc. The embodiments of the present invention do not limit this.

[0032] In some embodiments of the present invention, the categories of characters are the abbreviations of 31 provinces, 24 letters (excluding O and I in license plates), 10 digits, and the character '挂', totaling 66 categories.

[0033] In some embodiments of the present invention, the terminal uses the object detection model to perform character recognition on the license plate image, and obtains the character-related information of each of the multiple characters recognized in the license plate image.

[0034] Exemplarily, Figure 2a This is an optional recognition effect of a license plate recognition method provided by an embodiment of the present invention Figure 1 , such as Figure 2a shown, 7 characters are recognized, namely 'X', 'Y', '1', '2', '3', '4', and '5'; each character has a corresponding position box, as shown by the rectangular box in Figure 2a , and there is also the probability of the character category. For example, the probability of the 'Y' character is 0.84, and the probability of the '5' character is 0.92. Figure 2b This is an optional recognition effect diagram two of a license plate recognition method provided by an embodiment of the present invention. As shown in Figure 2b , 7 characters are recognized, namely 'X', 'Y', '1', '2', '3', '4', '5', and '6'; each character has a corresponding position box, as shown by the rectangular box in Figure 2b , and there is also the probability of the character category. For example, the probability of the 'X' character is 0.88, and the probability of the '6' character is 0.87. Figure 2c This is an optional recognition effect of a license plate recognition method provided by an embodiment of the present invention Figure 3 , such as Figure 2c shown, 7 characters are recognized, namely 'X', 'Y', '1', '2', '3', '4', and '5'; each character has a corresponding position box, as shown by the rectangular box in Figure 2c , and there is also the probability of the character category. For example, the probability of the 'X' character is 0.88, and the probability of the '1' character is 0.90. Here, X represents the abbreviation of the province, and Y represents the prefecture-level city.

[0035] S103. By screening the abnormal categories and abnormal regions of different categories of characters, determine the abnormal characters, and eliminate the abnormal characters from the multiple characters to determine the candidate license plate characters.

[0036] In some embodiments of the present invention, abnormal characters include duplicate characters and characters in abnormal regions.

[0037] In some embodiments of the present invention, the terminal can perform abnormal category filtering on different categories of characters to determine duplicate characters; perform abnormal category filtering on different categories of characters to determine duplicate characters; remove abnormal characters from multiple characters to determine candidate license plate characters.

[0038] In some embodiments of the present invention Figure 3 This is a schematic diagram of an optional process for a license plate recognition method according to an embodiment of the present invention, as shown in Figure 2. Figure 3 As shown, by filtering different categories of characters by anomaly category and anomaly region, the abnormal characters can be identified through S201-S202, as follows:

[0039] S201. Filter characters of different categories for abnormal categories to identify duplicate characters.

[0040] In some embodiments of the present invention, character-related information includes the position box of each character and the probability of each character.

[0041] In some embodiments of the present invention, the terminal can determine the overlapping area and total area between pairs of character position boxes by using the position box of each character; perform a quotient operation on the overlapping area and the total area to obtain the repetition rate between pairs of character position boxes; determine a repetition rate matrix based on the repetition rate between pairs of character position boxes; filter the elements in the repetition rate matrix according to a preset first threshold to obtain the repetition rate corresponding to the preset first threshold; determine the two characters corresponding to the repetition rate corresponding to the repetition rate corresponding to the preset first threshold; compare the probabilities corresponding to the two characters, and select the character with the smallest probability to determine the repetitive character.

[0042] In some embodiments of the present invention, S201 can be implemented by S2011-S2015, as follows:

[0043] S2011. Based on the position box of each character, determine the overlap area and total area between any two character position boxes.

[0044] In some embodiments of the present invention, the intersection-union ratio (IOU) is used to measure the overlap between two boxes, that is, the repetition rate between the two boxes.

[0045] In some embodiments of the present invention, the terminal determines the overlapping area and total area between pairs of character position frames by using the position frame of each character and the position information of each character.

[0046] S2012. Calculate the quotient between the overlapping area and the total area to obtain the repetition rate between each pair of character position boxes.

[0047] In some embodiments of the present invention, the terminal performs a quotient calculation on the overlapping area between pairs of character position boxes and the total area of ​​the pairs of character position boxes to obtain the repetition rate between the pairs of character position boxes.

[0048] For example, IOU = overlap area / total area, where IOU is the repetition rate between pairs of character position boxes.

[0049] S2013. Determine the repetition rate matrix based on the repetition rate between pairs of character position boxes.

[0050] In some embodiments of the present invention, the elements of the repetition rate matrix are repetition rates.

[0051] In some embodiments of the present invention, the terminal can arrange the characters by the repetition rate between each pair of character position boxes to obtain a repetition rate matrix.

[0052] For example, the repetition rate matrix can be represented by the following formula (1).

[0053]

[0054] Where, d ij Let D represent the repetition rate of bounding boxes i and j, and let D be the repetition rate matrix.

[0055] S2014. Filter the elements in the repetition rate matrix according to the preset first threshold to obtain the repetition rate corresponding to the first threshold.

[0056] In some embodiments of the present invention, the terminal can traverse the elements in the repetition rate matrix according to a preset first threshold to obtain the repetition rate corresponding to a value greater than the preset first threshold.

[0057] S2015. Based on the repetition rate corresponding to the target repetition rate, determine the two characters corresponding to the target repetition rate; compare the probabilities of the two characters, and select the character with the lowest probability as the repetition character.

[0058] In some embodiments of the present invention, the target repetition rate is the repetition rate that is greater than a preset first threshold.

[0059] In some embodiments of the present invention, the terminal can determine two characters corresponding to the target repetition rate based on the repetition rate corresponding to a repetition rate greater than a preset first threshold; compare the probabilities corresponding to the two characters, and select the character with the lowest probability among the two as the repetition character.

[0060] For example, the first threshold is preset to thresh, d ijrep_rate represents the repetition rate of position box i and position box j; confidence represents the probability that this position box is the current character category (equivalent to the probability of the character); when d ij >thresh, it is determined that the overlapping range of these two position boxes is relatively large; one of them is a repeated recognition, and the one with the smallest confidence corresponding to the two position boxes is removed. Assuming confidence(i) < confidence(j), that is, the i-th row and the i-th column are deleted in D, and then the D matrix is traversed again until all values in the D matrix are less than thresh.

[0061] It can be understood that by calculating the repetition rate between pairwise position boxes through the IOU index and removing the characters that are repeatedly recognized, the accuracy of the characters can be improved.

[0062] S202. Screen the abnormal regions of characters of different categories and determine the characters in the abnormal regions.

[0063] In some embodiments of the present invention, the terminal can calculate the distance between each character and its adjacent characters according to the position information of each character to obtain the density corresponding to each character and its adjacent characters; screen the density through a preset second threshold, and determine the characters corresponding to the density less than the preset second threshold as the characters in the abnormal region.

[0064] In some embodiments of the present invention, S202 can be implemented through S2021 and S2022, as follows:

[0065] S2021. Calculate the distance between each character and its adjacent characters according to the position information of each character to obtain the density corresponding to each character and its adjacent characters.

[0066] In some embodiments of the present invention, the Local Outlier Factor (LOF) algorithm mainly determines whether a point is an outlier by comparing the density of each point p and its neighborhood points. If the density of point p is lower, it is more likely to be identified as an outlier.

[0067] In some embodiments of the present invention, the terminal can calculate the distance between each character and its adjacent characters according to the position information of each character through the Local Outlier Factor (LOF) algorithm to obtain the density corresponding to each character and its adjacent characters.

[0068] S2022. Screen the density through a preset second threshold, and determine the characters corresponding to the density less than the preset second threshold as the characters in the abnormal region.

[0069] In some embodiments of the present invention, the terminal can screen the density through a preset second threshold, and determine the characters corresponding to the density less than the preset second threshold as the characters in the abnormal region.

[0070] For example, characters in the abnormal region are as follows: Figure 4 As shown, the model incorrectly identified the "1" in the lower left corner, whose position is significantly different from other regions in the character set.

[0071] Understandably, by using the Loop Detection (LOF) algorithm to calculate the density of each character and its neighboring characters, and then removing characters from abnormal regions based on the density, the accuracy of character recognition can be improved.

[0072] S104. Based on the position information of the candidate license plate characters, perform linear fitting to determine the permutation layer number of each candidate character.

[0073] In some embodiments of the present invention, the candidate license plate characters include multiple characters.

[0074] In some embodiments of the present invention, the terminal can calculate the center position information corresponding to each candidate character based on the position information of each candidate character, thereby obtaining the center position information of each candidate character; perform linear fitting based on the center position information of each candidate character to determine the fitted line. Based on the center position information of each candidate character, calculate the first distance information from the center of all candidate characters to the fitted line; determine the number of arrangement layers based on the first distance information and a preset third threshold; when the number of arrangement layers is single-layered, each candidate license plate character belongs to the second layer of the preset arrangement layer; when the number of arrangement layers is double-layered, perform linear fitting based on the position information of the candidate license plate characters to determine the arrangement layer number of each candidate character.

[0075] In some embodiments of the present invention Figure 5 This invention provides an optional flowchart of a license plate recognition method. Figure 3 ,like Figure 5 As shown, S104 can be implemented through S1041-S1046, as follows:

[0076] S1041. Based on the position information of each candidate character, calculate the center position information corresponding to each candidate character to obtain the center position information of each candidate character.

[0077] In some embodiments of the present invention, the center position information is the position information of the center point of the character. The center position information can be the center coordinates. The embodiments of the present invention do not limit this.

[0078] In some embodiments of the present invention, the terminal can calculate the center position information corresponding to each candidate character based on the position information of each candidate character, thereby obtaining the center position information of each candidate character.

[0079] For example, the terminal can calculate the center coordinates of each candidate character by using the coordinates of each candidate character.

[0080] S1042. Based on the center position information of each candidate character, perform linear fitting to determine the fitting line.

[0081] In some embodiments of the present invention, the terminal can fit the center position information of all characters onto a straight line based on the center position information of each candidate character, wherein the center position information of each candidate character is near this straight line, and this straight line is used as the fitting line.

[0082] For example, the fitted straight line is as follows Figure 6 The straight line shown.

[0083] S1043. Based on the center position information of each candidate character, calculate the first distance information from the center of all candidate characters to the fitted line.

[0084] In some embodiments of the present invention, the terminal can calculate the first distance information from the center of each candidate character to the fitted straight line based on the center position information of each candidate character. The first distance information can be the first sum of squared errors. The embodiments of the present invention do not limit this.

[0085] For example, the terminal can calculate the first sum of squared errors from the center of all candidate characters to the fitted line based on the center coordinates of each candidate character, thus obtaining the first sum of squared errors.

[0086] S1044. Determine the number of arrangement layers based on the first distance information and the preset third threshold.

[0087] In some embodiments of the present invention, the number of layers includes single layer and double layer.

[0088] In some embodiments of the present invention, the terminal compares the first distance information with a preset third threshold; if the first distance information is not greater than the preset third threshold, the number of layers is a single layer; if the first distance information is greater than the preset third threshold, the number of layers is a double layer.

[0089] S1045. When the number of arrangement layers is a single layer, each candidate license plate character belongs to the second layer of the preset arrangement layer.

[0090] In some embodiments of the present invention, when the terminal determines that the number of arrangement layers is a single layer, all candidate license plate characters are treated as the second layer of the preset arrangement layer.

[0091] S1046. When the number of arrangement layers is two, linear fitting is performed based on the position information of the candidate license plate characters to determine the number of arrangement layers for each candidate character.

[0092] In some embodiments of the present invention, when the number of arrangement layers is two, the terminal can arbitrarily select a preset number of license plate characters from the candidate license plate characters as a combination to obtain multiple combinations; perform linear fitting on each combination, calculate the second error sum of squares corresponding to each combination, and obtain multiple second distance information; select the combination with the smallest second distance information from the multiple second distance information, and the license plate characters corresponding to it all belong to the second layer of the preset arrangement layer; the characters in the candidate license plate characters other than those belonging to the second layer of the preset arrangement layer all belong to the first layer of the preset arrangement layer.

[0093] Understandably, by performing linear fitting based on the positional information of the candidate license plate characters, the number of permutation layers for each candidate character can be determined, thereby accurately locating the position of the candidate license plate characters and improving the recognition accuracy.

[0094] In some embodiments of the present invention, S1046 can be implemented by S301-S304, as follows:

[0095] S301. When the number of layers is two, a preset number of license plate characters are randomly selected from the candidate license plate characters to form a combination, resulting in multiple combinations.

[0096] In some embodiments of the present invention, the preset quantity represents the rules for setting license plates.

[0097] In some embodiments of the present invention, when the terminal determines that the number of arrangement layers is two layers, it arbitrarily selects a preset number of license plate characters from all candidate license plate characters as a combination, thereby obtaining multiple combinations containing a preset number of license plate characters.

[0098] For example, the preset number is 5. Generally, a double-layer license plate contains 7 license plate characters. Five license plate characters are randomly selected from all candidate license plate characters as a combination, thus obtaining 21 combinations.

[0099] S302. Perform linear fitting on each combination and calculate the second distance information corresponding to each combination to obtain multiple second distance information.

[0100] In some embodiments of the present invention, the terminal performs linear fitting on each combination based on the center position information of the candidate license plate characters corresponding to each combination, and obtains an initial fitting line corresponding to each combination; based on the center position information of the candidate license plate characters corresponding to each combination, it calculates the second distance information from the center of the corresponding candidate license plate character in each combination to the corresponding initial fitting line, and obtains multiple second distance information; the second distance information can be the second error sum of squares, and the embodiments of the present invention do not limit this.

[0101] S303. From multiple second distance information, select the combination with the smallest second distance information. The license plate characters corresponding to the combination with the smallest second distance information all belong to the second layer of the preset arrangement layer.

[0102] In some embodiments of the present invention, the terminal may select the combination with the smallest second distance information from a plurality of second distance information, and use the license plate character corresponding to the combination with the smallest second distance information as the second layer of a preset arrangement layer.

[0103] For example, the license plate characters in the second layer are themselves on a straight line, such as... Figure 6 As shown, the license plate characters "12345" are the second layer of the preset arrangement.

[0104] S304. Among the candidate license plate characters, all characters except those belonging to the second layer of the preset arrangement layer belong to the first layer of the preset arrangement layer.

[0105] In some embodiments of the present invention, the terminal may use all characters in the candidate license plate characters except those belonging to the second layer of the preset arrangement layer as the first layer of the preset arrangement layer.

[0106] For example, such as Figure 6 As shown, the remaining two characters "X" and "Y" are used as the first layer of the preset arrangement.

[0107] S105. According to the preset arrangement layer order and the position information of the candidate license plate characters, the candidate license plate characters in different arrangement layers are sequentially spliced ​​together to determine the license plate number corresponding to the license plate image.

[0108] In some embodiments of the present invention, the terminal can sort the candidate license plate characters belonging to the first layer of a preset arrangement according to the position information of the candidate license plate characters to obtain a first sorting result, and then concatenate the candidate license plate characters belonging to the first layer of the preset arrangement according to the first sorting result to obtain a first-layer license plate number; sort the candidate license plate characters belonging to the second layer of the preset arrangement according to the position information of the candidate license plate characters to obtain a second sorting result, and then concatenate the candidate license plate characters belonging to the second layer of the preset arrangement according to the second sorting result to obtain a second-layer license plate number; and determine the license plate number corresponding to the license plate image based on the first-layer license plate number and the second-layer license plate number.

[0109] Understandably, the terminal can improve the accuracy of candidate license plate characters by filtering out abnormal characters from multiple identified characters and their respective character-related information, removing abnormal characters, and determining the candidate license plate characters. Furthermore, by determining the position information of the candidate license plate characters, the terminal can determine the arrangement layer of each candidate character, and then sequentially concatenate the candidate license plate characters at different arrangement layers to determine the license plate number corresponding to the license plate image, thereby improving the accuracy of license plate number recognition.

[0110] In some embodiments of the present invention, S105 can be implemented by S1051, S1052 and S1053, as follows:

[0111] S1051. For the candidate license plate characters belonging to the first layer of the preset arrangement layer, sort them according to the position information of the candidate license plate characters to obtain the first sorting result. Then, according to the first sorting result, concatenate the candidate license plate characters belonging to the first layer of the preset arrangement layer to obtain the first layer license plate number.

[0112] In some embodiments of the present invention, the terminal sorts the candidate license plate characters belonging to the first layer of the preset arrangement layer according to the first position information in the position information of the candidate license plate characters in the first layer to obtain a first sorting result, and splices the candidate license plate characters belonging to the first layer of the preset arrangement layer according to the first sorting result, and after splicing, obtains the first layer license plate number.

[0113] It should be noted that the location information can be coordinates; the first location information can be the horizontal coordinate in the coordinate system, and the embodiments of the present invention do not impose any restrictions on this.

[0114] S1052. For the candidate license plate characters belonging to the second layer of the preset arrangement layer, sort them according to the position information of the candidate license plate characters to obtain a second sorting result. Then, concatenate the candidate license plate characters belonging to the second layer of the preset arrangement layer according to the second sorting result to obtain the second layer license plate number.

[0115] In some embodiments of the present invention, the terminal sorts the candidate license plate characters belonging to the second layer of the preset arrangement layer according to the first position information in the position information of the candidate license plate characters in the second layer to obtain a second sorting result, and splices the candidate license plate characters belonging to the second layer of the preset arrangement layer according to the second sorting result to obtain the second layer license plate number after splicing.

[0116] S1053. Based on the first-layer license plate number and the second-layer license plate number, determine the license plate number corresponding to the license plate image.

[0117] In some embodiments of the present invention, the terminal can output the license plate number corresponding to the license plate image in sequence according to the first layer license plate number and the second layer license plate number.

[0118] Understandably, the terminal can recognize dual license plates, improving the diversity and accuracy of recognition.

[0119] In some embodiments of the present invention, S105 can also be implemented by S401 and S402, as follows:

[0120] S401. When the number of arrangement layers is a single layer, the characters in the first layer of the preset arrangement layer are determined to be empty. For the candidate license plate characters belonging to the second layer of the preset arrangement layer, they are sorted according to the position information of the candidate license plate characters to obtain the third sorting result.

[0121] In some embodiments of the present invention, when the terminal obtains that the number of arrangement layers is a single layer, the characters of the first layer of the preset arrangement layer are set to empty; all candidate license plate characters belonging to the second layer of the preset arrangement layer are sorted according to the first position information in the position information of the candidate license plate characters to obtain a third sorting result; the first position information can be the horizontal coordinate in the coordinate system, and the embodiments of the present invention do not limit this.

[0122] For example, all candidate license plate characters in the second layer of the preset arrangement layer are sorted according to the horizontal coordinate of the candidate license plate characters to obtain a third sorting result; the third sorting result is from smallest to largest.

[0123] It should be noted that since characters located on the same layer have the same vertical coordinate in their position information, the first position information can be a position point.

[0124] S402. According to the third sorting result, the candidate license plate characters belonging to the second layer of the preset arrangement layer are concatenated to determine the license plate number corresponding to the license plate image.

[0125] In some embodiments of the present invention, the terminal can splice candidate license plate characters belonging to the second layer of a preset arrangement layer according to the third sorting result, and after splicing, obtain the license plate number corresponding to the license plate image.

[0126] Understandably, concatenating the candidate license plate characters according to the sorting results improves the accuracy of recognition.

[0127] In some embodiments of the present invention, S105 can also be implemented by S501, S502 and S503, as follows:

[0128] S501. When the number of arrangement layers is two, the candidate license plate characters belonging to the first layer of the preset arrangement layer are sorted according to the position information of the candidate license plate characters to obtain the fourth sorting result. The candidate license plate characters belonging to the first layer of the preset arrangement layer are concatenated according to the fourth sorting result to obtain the first layer license plate number.

[0129] In some embodiments of the present invention, when the terminal obtains a double-layer arrangement, the candidate license plate characters belonging to the first layer of the preset arrangement are sorted according to the first position information in the position information of the candidate license plate characters in the first layer to obtain a fourth sorting result. The candidate license plate characters belonging to the first layer of the preset arrangement are then concatenated according to the fourth sorting result. After concatenation, the first layer license plate number is obtained.

[0130] S502. For the candidate license plate characters belonging to the second layer of the preset arrangement layer, sort them according to the position information of the candidate license plate characters to obtain the fifth sorting result. Then, according to the fifth sorting result, concatenate the candidate license plate characters belonging to the second layer of the preset arrangement layer to obtain the second layer license plate number.

[0131] In some embodiments of the present invention, the terminal sorts the candidate license plate characters belonging to the second layer of the preset arrangement layer according to the first position information in the position information of the candidate license plate characters in the second layer, and obtains a fifth sorting result. The terminal then splices the candidate license plate characters belonging to the second layer of the preset arrangement layer according to the fifth sorting result, and obtains the second layer license plate number after splicing.

[0132] S503. Based on the first-layer license plate number and the second-layer license plate number, determine the license plate number corresponding to the license plate image.

[0133] In some embodiments of the present invention, the terminal can output the license plate number corresponding to the license plate image in sequence according to the first layer license plate number and the second layer license plate number.

[0134] Understandably, the terminal can recognize dual license plates, improving the diversity and accuracy of recognition.

[0135] In some embodiments of the present invention, S106 and S107 are executed before S102, as follows:

[0136] S106. Obtain license plate training data.

[0137] In some embodiments of the present invention, the terminal can acquire license plate training data.

[0138] In some embodiments of the present invention, S601-S603 are executed before S106, as follows:

[0139] S601. Obtain standard license plate images and corresponding real license plate images for various types of license plates.

[0140] In some embodiments of the present invention, the terminal can acquire standard license plate images corresponding to a certain type of license plate and corresponding real license plate images; the standard license plate images are produced in accordance with national standards, and the real license plate images are taken in actual use.

[0141] S602. Generate a simulated license plate image based on the standard license plate image and the real license plate image.

[0142] In some embodiments of the present invention, the terminal can map the license plate information in the standard license plate image to the license plate information corresponding to the real license plate image, thereby generating a simulated license plate image.

[0143] S603. Use simulated license plate images and real license plate images as license plate training data.

[0144] In some embodiments of the present invention, the terminal uses both simulated license plate images and real license plate images as license plate training data.

[0145] Understandably, using simulated license plate images and real license plate images as training data makes the training data more diverse, which can improve the recognition accuracy of the target detection model.

[0146] S107. Continuously train the initial target detection model based on the license plate training data until the probability is greater than the preset fourth threshold, and determine the target detection model.

[0147] In some embodiments of the present invention, the terminal can continuously train the initial target detection model based on license plate training data, compare the probability of the output character with a preset fourth threshold, and determine the target detection model until the probability of the character is greater than the preset fourth threshold.

[0148] It is understandable that training an initial target detection model using license plate training data can improve the recognition accuracy of the target detection model, thereby increasing the accuracy of license plate recognition results.

[0149] An optional flowchart of a license plate recognition method provided in this embodiment of the invention. Figure 6 ,like Figure 7 As shown, the terminal can acquire the license plate area image (equivalent to the license plate image to be recognized) through monitoring video stream. In the character detection module, the position and category information of each character are obtained based on the object detection model. In the abnormal character removal module, repeatedly recognized characters are removed based on IOU (equivalent to duplicate characters); characters in abnormal areas are removed based on the LOF algorithm. In the arrangement layer character judgment module, a straight line is fitted to the remaining characters (equivalent to candidate license plate characters), the first distance information is calculated, and the single or double arrangement layer is determined. If the arrangement layer is single, all characters are second-layer characters, and the first-layer characters are empty. If the arrangement layer is double, a straight line is fitted for every 5 characters, the second distance information is calculated, and the combination with the smallest second distance information is the second-layer character, and the rest are the first-layer characters. In the character splicing module, characters are spliced ​​from left to right, first the first layer and then the second layer, to form the license plate number (equivalent to the license plate number corresponding to the license plate image), and the license plate recognition result is output.

[0150] Understandably, by using the abnormal character removal module to filter out multiple identified characters and their respective character-related information, and removing repeatedly identified characters and characters in abnormal areas, the accuracy of candidate license plate characters can be improved. By using the arrangement layer character judgment module to determine the arrangement layer number of each candidate character, and then sequentially concatenating the candidate license plate characters at different arrangement layers to determine the license plate number corresponding to the license plate image, the accuracy of license plate number recognition can be improved.

[0151] Based on the license plate recognition method described in the above embodiments, this invention also provides a license plate recognition device, such as... Figure 8 As shown, Figure 8 A schematic diagram of the structure of a license plate recognition device provided in an embodiment of the present invention. Figure 1 The license plate recognition device 8 includes: an acquisition unit 801 and a determination unit 802; wherein,

[0152] The acquisition unit 801 is used to acquire the license plate image to be identified;

[0153] The determining unit 802 is used to perform character recognition on the license plate image using a target detection model, and determine the character-related information of each of the multiple characters identified in the license plate image; the character-related information includes the position information and category of each character; by filtering characters of different categories for abnormal categories and abnormal regions, abnormal characters are determined, and the abnormal characters are removed from the multiple characters to determine candidate license plate characters; based on the position information of the candidate license plate characters, linear fitting is performed to determine the permutation layer number of each candidate character; according to the order of the preset permutation layer number and the position information of the candidate license plate characters, the candidate license plate characters in different permutation layers are sequentially concatenated to determine the license plate number corresponding to the license plate image.

[0154] In some embodiments of the present invention, the abnormal characters include duplicate characters and characters in abnormal regions; the determining unit 802 is further configured to filter characters of different categories for abnormal categories and determine the duplicate characters; and to filter characters of different categories for abnormal regions and determine the characters in the abnormal regions.

[0155] In some embodiments of the present invention, the character-related information includes the position box of each character and the probability of each character; the determining unit 802 is further configured to determine the overlapping area and the total area between any two character position boxes based on the position box of each character.

[0156] The acquisition unit 801 is further configured to perform a quotient operation on the overlapping area and the total area to obtain the repetition rate between the two character position boxes;

[0157] The determining unit 802 is further configured to determine a repetition rate matrix based on the repetition rate between the pairs of character position boxes; the elements of the repetition rate matrix are the repetition rates.

[0158] The acquisition unit 801 is further configured to filter the elements in the repetition rate matrix according to a preset first threshold to obtain a repetition rate greater than the preset first threshold.

[0159] The determining unit 802 is further configured to determine two characters corresponding to the target repetition rate based on the repetition rate corresponding to the repetition rate greater than the preset first threshold; compare the probabilities corresponding to the two characters, and select the character with the lowest probability among the two as the repetition character; the target repetition rate is the repetition rate corresponding to the preset first threshold.

[0160] In some embodiments of the present invention, the acquisition unit 801 is further configured to calculate the distance between each character and its adjacent characters based on the position information of each character, and obtain the density corresponding to each character and its adjacent characters;

[0161] The determining unit 802 is further configured to filter the density by a preset second threshold, and determine the characters whose density is less than the preset second threshold as characters in the abnormal region.

[0162] In some embodiments of the present invention, the candidate license plate characters include multiple characters; the acquisition unit 801 is further configured to calculate the center position information corresponding to each candidate character based on the position information of each candidate character, thereby obtaining the center position information of each candidate character; the center position information is the position information of the center point of the character;

[0163] The determining unit 802 is further configured to perform linear fitting based on the center position information of each candidate character to determine the fitting line;

[0164] The acquisition unit 801 is further configured to calculate the first distance information from the center of all candidate characters to the fitted straight line based on the center position information of each candidate character;

[0165] The determining unit 802 is further configured to determine the number of arrangement layers based on the first distance information and a preset third threshold; the number of arrangement layers includes single layer and double layer;

[0166] The determining unit 802 is further configured to, when the number of arrangement layers is a single layer, determine that each candidate license plate character belongs to the second layer of the preset arrangement layer; and when the number of arrangement layers is a double layer, determine the number of arrangement layers for each candidate character by performing linear fitting based on the position information of the candidate license plate characters.

[0167] In some embodiments of the present invention, the number of arrangement layers includes single layer and double layer; the determining unit 802 is further configured to: if the first distance information is not greater than the preset third threshold, then the number of arrangement layers is single layer; if the first distance information is greater than the preset third threshold, then the number of arrangement layers is double layer.

[0168] In some embodiments of the present invention, the acquisition unit 801 is further configured to, when the number of arrangement layers is two layers, arbitrarily select a preset number of license plate characters from the candidate license plate characters as a combination to obtain multiple combinations; the preset number indicates that it conforms to the setting rules of the license plate; perform linear fitting on each combination, calculate the second distance information corresponding to each combination, and obtain multiple second distance information;

[0169] The determining unit 802 is further configured to select the combination with the smallest second distance information from the plurality of second distance information, wherein the license plate characters corresponding to the combination with the smallest second distance information all belong to the second layer of the preset arrangement layer; and the candidate license plate characters, except those belonging to the second layer of the preset arrangement layer, all belong to the first layer of the preset arrangement layer.

[0170] In some embodiments of the present invention, the acquisition unit 801 is further configured to sort the candidate license plate characters belonging to the first layer of the preset arrangement layer according to the position information of the candidate license plate characters to obtain a first sorting result, and to concatenate the candidate license plate characters belonging to the first layer of the preset arrangement layer according to the first sorting result to obtain a first layer license plate number; and to sort the candidate license plate characters belonging to the second layer of the preset arrangement layer according to the position information of the candidate license plate characters to obtain a second sorting result, and to concatenate the candidate license plate characters belonging to the second layer of the preset arrangement layer according to the second sorting result to obtain a second layer license plate number;

[0171] The determining unit 802 is further configured to determine the license plate number corresponding to the license plate image based on the first layer license plate number and the second layer license plate number.

[0172] In some embodiments of the present invention, before the target detection model is used to perform character recognition on the license plate image and determine the character-related information of each of the multiple characters identified in the license plate image, the acquisition unit 801 is further used to acquire license plate training data.

[0173] The determining unit 802 is further configured to continuously train the initial target detection model based on the license plate training data until the probability is greater than a preset fourth threshold, and then determine the target detection model.

[0174] In some embodiments of the present invention, before acquiring the license plate training data, the acquisition unit 801 is further configured to acquire standard license plate images and corresponding real license plate images corresponding to various types of license plates; and generate simulated license plate images based on the standard license plate images and the real license plate images.

[0175] The determining unit 802 is further configured to use the simulated license plate image and the real license plate image as the license plate training data.

[0176] It should be noted that the above-described division of program modules is only used as an example for license plate recognition. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the license plate recognition device and license plate recognition method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process and beneficial effects are detailed in the method embodiments, which will not be repeated here. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiments of this invention for understanding.

[0177] Based on the license plate recognition method described in the above embodiments, this invention also provides a license plate recognition device, such as... Figure 9 As shown, Figure 9 The second schematic diagram of a license plate recognition device provided in an embodiment of the present invention includes a processor 901 and a memory 902. The memory 902 stores one or more programs executable by the processor. When one or more programs are executed, the processor 901 executes any of the license plate recognition methods described in the previous embodiments.

[0178] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0179] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0182] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A license plate recognition method, characterized in that, include: Obtain the image of the license plate to be identified; The license plate image is subjected to character recognition using a target detection model to determine the character-related information of each of the multiple characters identified in the license plate image; the character-related information includes the position information of each character and the category of each character; By filtering characters of different categories based on their abnormality type and region, abnormal characters are identified, and then the abnormal characters are removed from the list of characters to determine candidate license plate characters. The candidate license plate characters include multiple characters; Based on the position information of each candidate character, the center position information corresponding to each candidate character is calculated to obtain the center position information of each candidate character; The center position information is the position information of the center point of the character; Based on the center position information of each candidate character, a linear fit is performed to determine the fitted line; Based on the center position information of each candidate character, calculate the first distance information from the center of all candidate characters to the fitted straight line; The number of arrangement layers is determined based on the first distance information and a preset third threshold; the number of arrangement layers includes single layer and double layer. When the number of arrangement layers is a single layer, each candidate license plate character belongs to the second layer of the preset arrangement layer; When the number of arrangement layers is two, linear fitting is performed based on the position information of the candidate license plate characters to determine the number of arrangement layers for each candidate character; According to the preset arrangement layer order and the position information of the candidate license plate characters, the candidate license plate characters in different arrangement layers are sequentially spliced ​​together to determine the license plate number corresponding to the license plate image.

2. The method according to claim 1, characterized in that, The abnormal characters include duplicate characters and characters in abnormal regions; The process involves filtering characters of different categories based on their anomaly type and region to identify anomalous characters, including: The system filters characters of different categories to identify abnormal categories and determines the duplicate characters. Filter out abnormal regions from different categories of characters to identify the characters in the abnormal regions.

3. The method according to claim 2, characterized in that, The character-related information includes the position box of each character and the probability of each character; The process of filtering characters of different categories to determine the duplicate characters includes: Based on the position frame of each character, determine the overlapping area and total area between any two character position frames; The overlap area and the total area are used to calculate the repetition rate between the pairs of character position boxes. Based on the repetition rate between the pairwise character position boxes, a repetition rate matrix is ​​determined; the elements of the repetition rate matrix are the repetition rates. The elements in the repetition rate matrix are filtered according to a preset first threshold to obtain the repetition rate corresponding to the value greater than the preset first threshold; Based on the repetition rate corresponding to the preset first threshold, two characters corresponding to the target repetition rate are determined; the probabilities corresponding to the two characters are compared, and the character with the lowest probability is selected as the repetitive character; the target repetition rate is the repetition rate corresponding to the preset first threshold.

4. The method according to claim 2, characterized in that, The process of filtering abnormal regions for different categories of characters to determine the characters in the abnormal regions includes: Based on the position information of each character, the distance between each character and its neighboring characters is calculated to obtain the density corresponding to each character and its neighboring characters; By filtering the density using a preset second threshold, characters with a density less than the preset second threshold are identified as characters in the abnormal region.

5. The method according to claim 1, characterized in that, The number of arrangement layers includes single-layer and double-layer; The step of determining the number of arrangement layers based on the first distance information and a preset third threshold includes: If the first distance information is not greater than the preset third threshold, then the number of arrangement layers is a single layer; If the first distance information is greater than the preset third threshold, then the number of arrangement layers is two layers.

6. The method according to claim 1, characterized in that, When the number of arrangement layers is two, linear fitting is performed based on the position information of the candidate license plate characters to determine the number of arrangement layers for each candidate character, including: When the number of arrangement layers is two, a preset number of license plate characters are randomly selected from the candidate license plate characters to form a combination, resulting in multiple combinations; the preset number indicates that the license plate setting rules are met. For each combination, a linear fit is performed, and the second distance information corresponding to each combination is calculated to obtain multiple second distance information; From the plurality of second distance information, select the combination with the smallest second distance information. The license plate characters corresponding to the combination with the smallest second distance information all belong to the second layer of the preset arrangement layer. All characters in the candidate license plate, except those belonging to the second layer of the preset arrangement layer, belong to the first layer of the preset arrangement layer.

7. The method according to any one of claims 1-4, characterized in that, The step of sequentially concatenating candidate license plate characters at different arrangement levels according to a preset arrangement layer order and the position information of the candidate license plate characters to determine the license plate number corresponding to the license plate image includes: For the candidate license plate characters belonging to the first layer of the preset arrangement layer, sort them according to the position information of the candidate license plate characters to obtain a first sorting result. Then, concatenate the candidate license plate characters belonging to the first layer of the preset arrangement layer according to the first sorting result to obtain the first layer license plate number. For the candidate license plate characters belonging to the second layer of the preset arrangement layer, a second sorting result is obtained according to the position information of the candidate license plate characters. The candidate license plate characters belonging to the second layer of the preset arrangement layer are then concatenated according to the second sorting result to obtain the second layer license plate number. Based on the first layer license plate number and the second layer license plate number, the license plate number corresponding to the license plate image is determined.

8. The method according to any one of claims 1-4, characterized in that, Before performing character recognition on the license plate image using a target detection model to determine the character-related information of each of the multiple characters identified in the license plate image, the method further includes: Obtain license plate training data; The initial target detection model is continuously trained based on the license plate training data until the probability is greater than a preset fourth threshold, and then the target detection model is determined.

9. The method according to claim 8, characterized in that, Before acquiring the license plate training data, the method further includes: Obtain standard license plate images and corresponding real license plate images for various types of license plates; Based on the standard license plate image and the real license plate image, a simulated license plate image is generated; The simulated license plate image and the real license plate image are used as the license plate training data.

10. A license plate recognition device, characterized in that, It includes an acquisition unit and a determination unit; wherein, The acquisition unit is used to acquire the license plate image to be identified; The determining unit is used to perform character recognition on the license plate image using a target detection model, and determine the character-related information of each of the multiple characters identified in the license plate image; the character-related information includes the position information of each character and the category of each character; by filtering characters of different categories for abnormal categories and abnormal regions, abnormal characters are determined, and the abnormal characters are removed from the multiple characters to determine candidate license plate characters; the candidate license plate characters include multiple characters; The acquisition unit is further configured to calculate the center position information corresponding to each candidate character based on the position information of each candidate character, thereby obtaining the center position information of each candidate character; the center position information is the position information of the center point of the character; The determining unit is further configured to perform linear fitting based on the center position information of each candidate character to determine the fitting line; The acquisition unit is further configured to calculate the first distance information from the center of all candidate characters to the fitted straight line based on the center position information of each candidate character; The determining unit is further configured to determine the number of arrangement layers based on the first distance information and a preset third threshold; the number of arrangement layers includes single layer and double layer; when the number of arrangement layers is single layer, each candidate license plate character belongs to the second layer of the preset arrangement layer; when the number of arrangement layers is double layer, linear fitting is performed based on the position information of the candidate license plate characters to determine the arrangement layer number of each candidate character; according to the order of the preset arrangement layer number and the position information of the candidate license plate characters, the candidate license plate characters in different arrangement layers are sequentially concatenated to determine the license plate number corresponding to the license plate image.

11. A license plate recognition device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the license plate recognition method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The storage medium stores executable instructions, which, when executed, cause the processor to perform the license plate recognition method as described in any one of claims 1-9.

Images