License plate recognition method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a license plate recognition method and device, electronic equipment and storage medium, and relates to the technical field of image recognition. The method comprises: performing license plate correction on a to-be-detected image to obtain a corrected image; extracting image features of the corrected image to obtain a target feature map, splitting the target feature map to obtain a first feature vector and a second feature vector; performing different type feature fusion on the first feature vector and the second feature vector to obtain a third feature vector, a fourth feature vector and a fifth feature vector; inputting the third feature vector, the fourth feature vector and the fifth feature vector into different branch networks of a trained license plate recognition model respectively to obtain a target license plate recognition result, which is determined by single-layer license plate recognition results, double-layer license plate recognition results and license plate type recognition results output by the different branch networks of the trained license plate recognition model. The present disclosure improves the efficiency of license plate recognition.

Description

Technical Field

[0001] This disclosure relates to the field of image recognition technology, and in particular to a license plate recognition method, device, electronic device, and storage medium. Background Technology

[0002] With the rapid development of the economy and society, the number of motor vehicles is constantly increasing. Automatic vehicle identification can improve vehicle management efficiency and reduce labor costs. Therefore, license plate recognition technology has become a research hotspot in recent years. License plate category recognition methods are a crucial part of license plate recognition technology, capable of identifying the specific category of a given license plate image, thereby supplementing and improving license plate recognition information. This allows for further applications in more complex scenarios such as license plate counterfeiting detection and vehicle access control.

[0003] Traditional license plate recognition methods cannot accurately segment characters in relatively complex backgrounds. Existing deep learning-based license plate recognition methods cannot simultaneously solve single-layer and double-layer license plate recognition with a single model. Generally, two or more models need to be loaded simultaneously for single-layer and double-layer license plate recognition, resulting in redundancy in speed and resources, and a complex inference process.

[0004] Therefore, improving the efficiency of license plate recognition has become an urgent technical problem to be solved.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] This disclosure provides a license plate recognition method, apparatus, electronic device, and storage medium, which at least to some extent overcomes the problem of low license plate recognition efficiency in related technologies.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to one aspect of this disclosure, a license plate recognition method is provided, comprising: acquiring an image to be detected; performing license plate detection on the image to be detected to obtain a detection result of the image to be detected, wherein the detection result includes category information, the category information being license plate or non-license plate; if the category information of the image to be detected is license plate, performing license plate correction on the image to be detected to obtain a corrected image; extracting image features of the corrected image to obtain a target feature map, splitting the target feature map to obtain a first feature vector and a second feature vector; performing different types of feature fusion on the first feature vector and the second feature vector to obtain a third feature vector, a fourth feature vector, and a fifth feature vector; inputting the third feature vector, the fourth feature vector, and the fifth feature vector into different branch networks of a pre-trained license plate recognition model to obtain a target license plate recognition result, wherein the target license plate recognition result is determined by single-layer license plate recognition result, double-layer license plate recognition result, and license plate type recognition result output by different branch networks of the pre-trained license plate recognition model.

[0009] In one embodiment of this disclosure, before performing license plate detection on the image to be detected and obtaining the detection result of the image to be detected, the method further includes: identifying the image size of the image to be detected; if the image size of the image to be detected does not meet the preset size, patching the image to be detected into a square to obtain a patched image to be detected; placing the patched image to be detected at the center point of a preset size base plate for training to obtain a preset size image to be detected.

[0010] In one embodiment of this disclosure, performing license plate detection on the image to be detected to obtain a detection result of the image to be detected includes: inputting the image to be detected into the pre-trained license plate recognition model and outputting the detection result of the image to be detected, wherein the pre-trained license plate recognition model is also used to output the detection result of the image.

[0011] In one embodiment of this disclosure, when the category information of the image to be detected is a license plate, license plate correction is performed on the image to be detected to obtain a corrected image, including: performing license plate correction on the image to be detected based on projection mapping to obtain a corrected image.

[0012] In one embodiment of this disclosure, the image features of the corrected image are extracted to obtain a target feature map, and the target feature map is split to obtain a first feature vector and a second feature vector, including: downsampling the corrected image using CNN convolution to obtain the target feature map.

[0013] In one embodiment of this disclosure, different types of feature fusion are performed on the first feature vector and the second feature vector to obtain a third feature vector, a fourth feature vector, and a fifth feature vector. This includes: normalizing the first feature vector and the second feature vector; performing a merge operation on the normalized first feature vector and the second feature vector to obtain the third feature vector; performing a convolution operation on the first feature vector and the second feature vector; performing a horizontal feature connection on the convolutional first feature vector and the second feature vector to obtain the fourth feature vector; and performing a vertical feature connection on the first feature vector and the second feature vector to obtain the fifth feature vector.

[0014] In one embodiment of this disclosure, the third feature vector, the fourth feature vector, and the fifth feature vector are respectively input into different branch networks of a pre-trained license plate recognition model to obtain a target license plate recognition result. This includes: inputting the third feature vector into the first branch network of the pre-trained license plate recognition model to output a single-layer license plate recognition result; inputting the fourth feature vector into the second branch network of the pre-trained license plate recognition model to output a double-layer license plate recognition result; inputting the fifth feature vector into the third branch network of the pre-trained license plate recognition model to output a license plate type recognition result; and determining the target license plate recognition result based on the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result.

[0015] In one embodiment of this disclosure, the trained license plate recognition model is obtained by training the following steps: acquiring a license plate image sample set, wherein the license plate image sample set contains multiple license plate image samples and labels corresponding to each license plate image sample; extracting image features of each license plate image sample in the license plate image sample set; using the image features of each license plate image sample in the license plate image sample set as input to the license plate recognition model, using the labels corresponding to each license plate image sample in the license plate image sample set as output, and training the license plate recognition model using a multi-task loss function to obtain the trained license plate recognition model.

[0016] According to another aspect of this disclosure, a license plate recognition device is provided, comprising: an image acquisition module for acquiring an image to be detected; a license plate detection module for performing license plate detection on the image to be detected to obtain a detection result of the image to be detected, wherein the detection result includes category information, the category information being license plate or non-license plate; a license plate correction module for performing license plate correction on the image to be detected when the category information of the image to be detected is license plate, to obtain a corrected image; and a license plate recognition module for extracting image features of the corrected image to obtain a target feature map, and splitting the target feature map to obtain a first... The license plate recognition module is further configured to perform different types of feature fusion on the first and second feature vectors to obtain a third, fourth, and fifth feature vector; the license plate recognition module is further configured to input the third, fourth, and fifth feature vectors into different branch networks of the pre-trained license plate recognition model to obtain the target license plate recognition result, wherein the target license plate recognition result is determined by the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result output by the different branch networks of the pre-trained license plate recognition model.

[0017] In one embodiment of this disclosure, the apparatus further includes an image processing module, which is used to identify the image size of the image to be detected; if the image size of the image to be detected does not meet the preset size, the image to be detected is padded to form a square to obtain a padded image to be detected; the padded image to be detected is placed on the center point of a preset size base plate for training to obtain a preset size image to be detected.

[0018] In one embodiment of this disclosure, the license plate detection module is further configured to input the image to be detected into the trained license plate recognition model and output the detection result of the image to be detected, wherein the trained license plate recognition model is further configured to output the detection result of the image.

[0019] In one embodiment of this disclosure, the license plate correction module is further configured to perform license plate correction on the image to be detected based on projection mapping to obtain a corrected image. In another embodiment of this disclosure, the corrected image is downsampled using CNN convolution to obtain the target feature map.

[0020] In one embodiment of this disclosure, the license plate recognition module is further configured to normalize the first feature vector and the second feature vector, perform a merge operation on the normalized first feature vector and the second feature vector to obtain the third feature vector; perform a convolution operation on the first feature vector and the second feature vector, perform a horizontal feature connection on the convolutional first feature vector and the second feature vector to obtain the fourth feature vector; and perform a vertical feature connection on the first feature vector and the second feature vector to obtain the fifth feature vector.

[0021] In one embodiment of this disclosure, the license plate recognition module is further configured to input the third feature vector into the first branch network of the trained license plate recognition model and output a single-layer license plate recognition result; input the fourth feature vector into the second branch network of the trained license plate recognition model and output a double-layer license plate recognition result; input the fifth feature vector into the third branch network of the trained license plate recognition model and output a license plate type recognition result; and determine the target license plate recognition result based on the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result.

[0022] In one embodiment of this disclosure, the apparatus further includes a model training module, which is used to acquire a license plate image sample set, wherein the license plate image sample set contains multiple license plate image samples and labels corresponding to each license plate image sample; extract image features of each license plate image sample in the license plate image sample set; use the image features of each license plate image sample in the license plate image sample set as input to a license plate recognition model, use the labels corresponding to each license plate image sample in the license plate image sample set as output, and train the license plate recognition model using a multi-task loss function to obtain a trained license plate recognition model.

[0023] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described license plate recognition method by executing the executable instructions.

[0024] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described license plate recognition method.

[0025] The present disclosure provides a license plate recognition method, apparatus, electronic device, and storage medium. The license plate recognition method includes: acquiring an image to be detected; performing license plate detection on the image to be detected to obtain a detection result, wherein the detection result includes category information, which is either a license plate or a non-license plate; if the category information of the image to be detected is a license plate, performing license plate correction on the image to obtain a corrected image; extracting image features from the corrected image to obtain a target feature map; splitting the target feature map to obtain a first feature vector and a second feature vector; fusing different types of features from the first and second feature vectors to obtain a third feature vector, a fourth feature vector, and a fifth feature vector; inputting the third, fourth, and fifth feature vectors into different branch networks of a pre-trained license plate recognition model to obtain a target license plate recognition result, wherein the target license plate recognition result is determined by single-layer license plate recognition results, double-layer license plate recognition results, and license plate type recognition results output by different branch networks of the pre-trained license plate recognition model. This disclosure improves the efficiency and accuracy of license plate recognition, and can simultaneously recognize single-layer and double-layer license plates, thus enhancing the applicability of the license plate recognition method.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0028] Figure 1 This diagram illustrates a communication system architecture according to an embodiment of the present disclosure;

[0029] Figure 2 This diagram illustrates a license plate recognition method according to an embodiment of the present disclosure.

[0030] Figure 3 This diagram illustrates a detection result according to an embodiment of the present disclosure.

[0031] Figure 4 This diagram illustrates an image correction method according to an embodiment of the present disclosure.

[0032] Figure 5 This diagram illustrates another license plate recognition method according to an embodiment of the present disclosure.

[0033] Figure 6This diagram illustrates another license plate recognition method according to an embodiment of the present disclosure.

[0034] Figure 7 This diagram illustrates a license plate recognition method according to an embodiment of the present disclosure.

[0035] Figure 8 A schematic diagram of a license plate recognition device according to an embodiment of this disclosure is shown; and

[0036] Figure 9 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0037] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0038] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] As mentioned in the background section, common methods for license plate recognition under surveillance cameras include LPR, CRNN, and seq2seq. However, for simultaneous recognition of single and double-layer license plates, at least 2 to 3 models (license plate type recognition, single-layer license plate recognition, double-layer license plate recognition, etc.) are usually cascaded or run in parallel. Such methods are complex to deploy and consume a lot of computing power.

[0040] Based on this, this disclosure provides a license plate recognition method, device, electronic device, and storage medium. Based on multi-task learning and feature-level fusion, this disclosure solves the general recognition problem of single and double-layer license plates, shortens the inference process of the model, and improves inference efficiency.

[0041] Figure 1 A schematic diagram of an exemplary system architecture that can be applied to the license plate recognition method or license plate recognition device in the embodiments of this disclosure is shown.

[0042] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105.

[0043] Network 104 is a medium used to provide a communication link between terminal devices 101, 102, 103 and server 105, and can be a wired network or a wireless network.

[0044] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0045] Terminal devices 101, 102, and 103 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0046] Optionally, the client applications installed on different terminal devices 101, 102, and 103 may be the same, or clients of the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client may also differ; for example, the application client may be a mobile client, a PC client, etc.

[0047] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0048] Optionally, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0049] Those skilled in the art will know that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; any number of terminal devices, networks, and servers can be included depending on actual needs. This disclosure does not limit the scope of the embodiments.

[0050] The following detailed description of this exemplary implementation method is provided in conjunction with the accompanying drawings and embodiments.

[0051] First, this disclosure provides a license plate recognition method, which can be executed by any electronic device with computing power.

[0052] Figure 2 This diagram illustrates a license plate recognition method according to an embodiment of the present disclosure, such as... Figure 2 As shown, the license plate recognition method provided in this embodiment includes the following steps:

[0053] S202, Obtain the image to be detected;

[0054] It should be noted that the image to be detected can be an image taken by a camera above the highway. The image to be detected can be an image that includes a license plate or an image that does not include a license plate. Step S204 is required to identify whether the image to be detected contains license plate information.

[0055] S204, Perform license plate detection on the image to be detected to obtain the detection result of the image to be detected, wherein the detection result includes category information, which is license plate or non-license plate;

[0056] It should be noted that because the images captured by cameras above the highway have diverse angles, including large tilts such as looking up or shifting left and right, the effectiveness of license plate recognition is affected. Therefore, it is necessary to detect whether the image contains license plate information before performing license plate recognition.

[0057] In one embodiment of this disclosure, corner detection techniques from face recognition can be used. A retina-face model is employed to determine the detection result of the image by identifying whether it contains corner information of a license plate. See also... Figure 3 The diagram shows a detection result. The detection result can be a 1×14 dimensional sequence. For example, the expression of the detection result can be [corner box confidence, rect_top, rect_left, rect_down, rect_right, category information, x1, y1, x2, y2, x3, y3, x4, y4], where rect_top is the top edge of the rectangular object, i.e., the y coordinate of the top left vertex; rect_left is the left edge of the rectangular object, i.e., the x coordinate of the top left vertex; rect_down is the bottom edge of the rectangular object, i.e., the y coordinate of the bottom left vertex; rect_right is the right edge of the rectangular object, i.e., the x coordinate of the top right vertex; and the category information can be either 0 (non-license plate) or 1 (license plate).

[0058] In one embodiment of this disclosure, license plate detection is performed on an image to be detected to obtain a detection result. This includes: inputting the image to be detected into a pre-trained license plate recognition model and outputting the detection result. The pre-trained license plate recognition model is also used to output the image detection result. The pre-trained license plate recognition model uses a MobileV1 backbone network, with the neck portion employing an FPN feature pyramid structure, followed by context modules. During license plate detection, the pre-trained license plate recognition model outputs a 1×14-dimensional detection result.

[0059] In one embodiment of this disclosure, before performing license plate detection on the image to be detected and obtaining the detection result, the method further includes: identifying the image size of the image to be detected; if the image size of the image to be detected does not meet a preset size, padding the image to be detected into a square to obtain a padded image to be detected; and placing the padded image to be detected at the center point of a preset-size base plate for training to obtain a detection image of the preset size. The preset size can be any size, and can be in the form of an N×M resolution, where N and M are arbitrary numbers, such as 640×640 resolution. When the size of the image to be detected does not meet the 640×640 resolution, the image to be detected needs to be padded into a square, and the padded image to be detected needs to be placed at the center point of a 640×640 resolution base plate for training to obtain a 640×640 resolution detection image.

[0060] S206, If the category information of the image to be detected is license plate, perform license plate correction on the image to be detected to obtain the corrected image;

[0061] In one embodiment of this disclosure, when the category information of the image to be detected is a license plate, license plate correction is performed on the image to be detected to obtain a corrected image. This includes: performing license plate correction on the image to be detected based on projection mapping to obtain a corrected image. Projection mapping, also known as perspective transformation, refers to projecting the image (original license plate image) onto a new viewing plane (corrected license plate image). In other words, by projecting the image to be detected, the viewing plane of the image to be detected is changed, thereby obtaining a corrected image. See also... Figure 4 The diagram shows an image correction process. The left image is the image to be detected, and the right image is the corrected image. The four points in the left image are the four points detected by the license plate in step S204, and the four points in the right image are the corrected license plate.

[0062] S208, extract the image features of the corrected image to obtain the target feature map, and split the target feature map to obtain the first feature vector and the second feature vector;

[0063] It should be noted that the corrected image can be a 32×128 resolution image.

[0064] In one embodiment of this disclosure, image features of the corrected image are extracted to obtain a target feature map. The target feature map is then split to obtain a first feature vector and a second feature vector. This includes: downsampling the corrected image using CNN convolution to obtain the target feature map. For example, CNN (Convolutional Neural Networks) convolution is used to downsample the corrected image to obtain the target feature map. This target feature map can be a 2×33-dimensional feature map, where 33 is the maximum character length for license plate recognition. The target feature map can be split into two 1×n-dimensional feature vectors, where n is a positive integer, namely the first feature vector and the second feature vector. The first feature vector and the second feature vector represent the features of the upper and lower parts of the license plate image, respectively, and can be used for feature fusion and multi-task input processing in subsequent steps.

[0065] S210, perform different types of feature fusion on the first feature vector and the second feature vector to obtain the third feature vector, the fourth feature vector and the fifth feature vector;

[0066] It should be noted that feature fusion can include summation, feature connections in the horizontal dimension, and feature connections in the vertical dimension.

[0067] In one embodiment of this disclosure, see Figure 5 The flowchart of another license plate recognition method shown illustrates the fusion of different types of features from the first and second feature vectors to obtain a third, fourth, and fifth feature vector, including:

[0068] S502, normalize the first feature vector and the second feature vector, and perform a merge operation on the normalized first feature vector and the second feature vector to obtain the third feature vector.

[0069] It should be noted that, for example, if the first and second feature vectors are both 1×33 dimensional feature vectors, after normalizing the first and second feature vectors, a merge operation is performed on the normalized first and second feature vectors to obtain a 1×33 dimensional third feature vector.

[0070] S504: Perform a convolution operation on the first feature vector and the second feature vector, and perform feature connection on the horizontal dimension of the first feature vector and the second feature vector after the convolution operation to obtain the fourth feature vector.

[0071] It should be noted that, for example, the first feature vector and the second feature vector are both 1×33 dimensional feature vectors. First, convolution operation is performed on the first feature vector and the second feature vector respectively to ensure that the vector length ratio of the first feature vector to the second feature vector is approximately 2:5 (since the license plate of a double-layer license plate is generally 2 characters for the upper layer and 5 characters for the lower layer). Then, feature concatenation is performed on the first feature vector and the second feature vector after the convolution operation in the horizontal dimension to obtain a 1×33 dimensional fourth feature vector.

[0072] S506 performs a vertical feature connection on the first and second feature vectors to obtain the fifth feature vector.

[0073] It should be noted that, for example, the first feature vector and the second feature vector are both 1×33 dimensional feature vectors. By performing a vertical feature connection on the first feature vector and the second feature vector, a fifth feature vector of 1×66 dimensional can be obtained.

[0074] S212, the third feature vector, the fourth feature vector, and the fifth feature vector are respectively input into different branch networks of the pre-trained license plate recognition model to obtain the target license plate recognition result. The target license plate recognition result is determined by the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result output by the different branch networks of the pre-trained license plate recognition model.

[0075] It should be noted that the trained license plate recognition model has a first branch network, a second branch network, and a third branch network. The first branch network is used to output the single-layer license plate recognition result, the second branch network is used to output the double-layer license plate recognition result, and the third branch network is used to output the license plate type recognition result.

[0076] In one embodiment of this disclosure, participants Figure 6 The flowchart of another license plate recognition method shown illustrates that the third, fourth, and fifth feature vectors are input into different branch networks of a pre-trained license plate recognition model to obtain the target license plate recognition result, including:

[0077] S602, input the third feature vector into the first branch network of the trained license plate recognition model, and output the single-layer license plate recognition result;

[0078] S604, input the fourth feature vector into the second branch network of the trained license plate recognition model, and output the dual-layer license plate recognition result;

[0079] It should be noted that the single-layer license plate recognition result refers to the license plate number recognition result under a single-layer license plate, and the double-layer license plate recognition result refers to the license plate number recognition result under a double-layer license plate. The first and second branch networks of the trained license plate recognition model can both accept a 1×33-dimensional feature vector as input. The first and second branch networks can respectively connect to the RNN network output and perform decoding (using the grey search method). Based on the license plate type recognition result output by the third branch network, the selection of either the single-layer or double-layer license plate recognition result is determined. The RNN network output is the result of the third branch network.

[0080] S606, input the fifth feature vector into the third branch network of the trained license plate recognition model, and output the license plate type recognition result;

[0081] It should be noted that the license plate type recognition result is either a single-layer license plate or a double-layer license plate. The third branch network of the pre-trained license plate recognition model can take a 1×66-dimensional feature vector as input. This third branch network is followed by a 1×3-dimensional CNN layer, and finally a fully connected layer. The fully connected layer outputs two probabilities: the probability of a single-layer license plate and the probability of a double-layer license plate. The sum of these probabilities is 1. When the probability of a single-layer license plate is greater than the probability of a double-layer license plate, the license plate type recognition result is determined to be a single-layer license plate; when the probability of a single-layer license plate is less than the probability of a double-layer license plate, the license plate type recognition result is determined to be a double-layer license plate. For example, if the outputs are 0.6 and 0.4, where 0.6 is the probability of a single-layer license plate and 0.4 is the probability of a double-layer license plate, and 0.6 is greater than 0.4, then the license plate type recognition result is a single-layer license plate.

[0082] S608 determines the target license plate recognition result based on the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result.

[0083] It should be noted that the final output of the trained license plate recognition model can be determined based on the license plate type recognition result. For example, when the license plate type recognition result is a single-layer license plate, the single-layer license plate recognition result is output as the target license plate recognition result; when the license plate type recognition result is a double-layer license plate, the double-layer license plate recognition result is output as the target license plate recognition result.

[0084] The license plate recognition method provided in this disclosure performs license plate detection on the image to be detected. If the category information of the image to be detected is license plate, license plate correction is performed to obtain a corrected image. Image features of the corrected image are extracted to obtain a target feature map. The target feature map is split to obtain a first feature vector and a second feature vector. Different types of feature fusion are performed on the first and second feature vectors to obtain a third, fourth, and fifth feature vector. The third, fourth, and fifth feature vectors are respectively input into different branch networks of a pre-trained license plate recognition model to obtain single-layer license plate recognition results, double-layer license plate recognition results, and license plate type recognition results output by different branch networks. Based on the single-layer, double-layer, and license plate type recognition results, the target license plate recognition result is determined. This disclosure improves the efficiency and accuracy of license plate recognition, and can simultaneously recognize single-layer and double-layer license plates, thus improving the applicability of the license plate recognition method.

[0085] In one embodiment of this disclosure, a trained license plate recognition model is obtained through the following steps:

[0086] Obtain a license plate image sample set, which contains multiple license plate image samples and the corresponding labels for each license plate image sample;

[0087] Extract image features from each license plate image sample in the license plate image sample set;

[0088] The image features of each license plate image sample in the license plate image sample set are used as the input of the license plate recognition model, and the labels corresponding to each license plate image sample in the license plate image sample set are used as the output. The license plate recognition model is trained using a multi-task loss function to obtain a trained license plate recognition model.

[0089] It should be noted that the labels can include single-layer license plate recognition results, double-layer license plate recognition results, and license plate type recognition results for license plate image samples. Each branch network of the license plate recognition model corresponds to a loss function; for example, the first branch network corresponds to the first loss function, the second branch network corresponds to the second loss function, and the third branch network corresponds to the third loss function. The image features of each license plate image sample in the license plate image sample set, as well as the corresponding labels of each license plate image sample, can be input into the respective branch networks of the license plate recognition model to train the model. When the first, second, and third loss functions all converge, the trained license plate recognition model is obtained.

[0090] In one embodiment of this disclosure, participants Figure 7The diagram illustrates a license plate recognition method. After obtaining the corrected license plate image, this disclosure utilizes a CNN feature extraction module to extract image features from the corrected license plate image. The image features are then split into two 1×n-dimensional feature vectors. Different types of feature fusion are performed on these two 1×n-dimensional feature vectors, such as add, concat_h, and concat_v, to obtain multiple fused features. These multiple fused features are then input into the corresponding single-row image features, double-row image features, and license plate type prediction, respectively, to obtain single-layer license plate recognition results, double-layer license plate recognition results, and license plate type recognition results. At the license plate number decision point, based on the license plate type recognition result, either the single-layer license plate recognition result or the double-layer license plate recognition result is selected as the target license plate recognition result and output.

[0091] Based on the same inventive concept, this disclosure also provides a license plate recognition device, as shown in the following embodiments. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiments, the implementation of this device embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0092] Figure 8 This diagram illustrates a license plate recognition device according to an embodiment of the present disclosure, such as... Figure 8 As shown, the device includes:

[0093] Image acquisition module 810 is used to acquire the image to be detected;

[0094] The license plate detection module 820 is used to detect license plates in the image to be detected and obtain the detection result of the image to be detected. The detection result includes category information, which is either license plate or non-license plate.

[0095] The license plate correction module 830 is used to correct the license plate of the image to be detected when the category information of the image to be detected is license plate, so as to obtain the corrected image;

[0096] The license plate recognition module 840 is used to extract image features from the corrected image to obtain a target feature map, and to split the target feature map to obtain a first feature vector and a second feature vector.

[0097] The license plate recognition module 840 is also used to perform different types of feature fusion on the first feature vector and the second feature vector to obtain the third feature vector, the fourth feature vector and the fifth feature vector;

[0098] The license plate recognition module 840 is also used to input the third feature vector, the fourth feature vector and the fifth feature vector into different branch networks of the pre-trained license plate recognition model to obtain the target license plate recognition result. The target license plate recognition result is determined by the single-layer license plate recognition result, the double-layer license plate recognition result and the license plate type recognition result output by the different branch networks of the pre-trained license plate recognition model.

[0099] In one embodiment of this disclosure, the above-mentioned device further includes an image processing module, which is used to identify the image size of the image to be detected; when the image size of the image to be detected does not meet the preset size, the image to be detected is padded to a square to obtain the padded image to be detected; the padded image to be detected is placed on the center point of the preset size base plate for training to obtain the preset size image to be detected.

[0100] In one embodiment of this disclosure, the license plate detection module 820 is further configured to input the image to be detected into a pre-trained license plate recognition model and output the detection result of the image to be detected, wherein the pre-trained license plate recognition model is further configured to output the detection result of the image.

[0101] In one embodiment of this disclosure, the license plate correction module 830 is further configured to perform license plate correction on the image to be detected based on projection mapping, thereby obtaining a corrected image. In another embodiment of this disclosure, CNN convolution is used to downsample the corrected image to obtain a target feature map.

[0102] In one embodiment of this disclosure, the license plate recognition module 840 is further configured to normalize the first feature vector and the second feature vector, perform a merge operation on the normalized first feature vector and the second feature vector to obtain a third feature vector; perform a convolution operation on the first feature vector and the second feature vector, perform a feature connection on the first feature vector and the second feature vector after the convolution operation in the horizontal dimension to obtain a fourth feature vector; and perform a feature connection on the first feature vector and the second feature vector in the vertical dimension to obtain a fifth feature vector.

[0103] In one embodiment of this disclosure, the license plate recognition module 840 is further configured to input a third feature vector into the first branch network of the trained license plate recognition model and output a single-layer license plate recognition result; input a fourth feature vector into the second branch network of the trained license plate recognition model and output a double-layer license plate recognition result; input a fifth feature vector into the third branch network of the trained license plate recognition model and output a license plate type recognition result; and determine the target license plate recognition result based on the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result.

[0104] In one embodiment of this disclosure, the apparatus further includes a model training module, which is used to acquire a license plate image sample set, wherein the license plate image sample set contains multiple license plate image samples and labels corresponding to each license plate image sample; extract image features of each license plate image sample in the license plate image sample set; use the image features of each license plate image sample in the license plate image sample set as input to the license plate recognition model, use the labels corresponding to each license plate image sample in the license plate image sample set as output, and train the license plate recognition model using a multi-task loss function to obtain a trained license plate recognition model.

[0105] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0106] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0107] like Figure 9 As shown, the electronic device 900 is manifested in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0108] The storage unit stores program code, which can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure. For example, the processing unit 910 can execute the following steps of the above method embodiment: acquiring an image to be detected; performing license plate detection on the image to be detected to obtain a detection result of the image to be detected, wherein the detection result includes category information, which is license plate or non-license plate; if the category information of the image to be detected is license plate, performing license plate correction on the image to be detected to obtain a corrected image; extracting image features of the corrected image to obtain a target feature map, splitting the target feature map to obtain a first feature vector and a second feature vector; performing different types of feature fusion on the first feature vector and the second feature vector to obtain a third feature vector, a fourth feature vector, and a fifth feature vector; inputting the third feature vector, the fourth feature vector, and the fifth feature vector into different branch networks of the trained license plate recognition model to obtain a target license plate recognition result, wherein the target license plate recognition result is determined by the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result output by the different branch networks of the trained license plate recognition model.

[0109] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 9201 and / or cache memory 9202, and may further include read-only memory (ROM) 9203.

[0110] Storage unit 920 may also include a program / utility 9204 having a set (at least one) program module 9205, such program module 9205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0111] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0112] Electronic device 900 can also communicate with one or more external devices 940 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0113] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0114] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0115] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0117] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0118] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0119] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0120] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0121] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0122] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A license plate recognition method, characterized by, include: Acquire the image to be detected; The image to be detected is subjected to license plate detection to obtain the detection result of the image to be detected, wherein the detection result includes category information, and the category information is license plate or non-license plate; If the category information of the image to be detected is a license plate, the image to be detected is corrected for license plate information to obtain a corrected image. Extract the image features of the corrected image to obtain a target feature map, and split the target feature map to obtain a first feature vector and a second feature vector; Different types of feature fusion are performed on the first feature vector and the second feature vector to obtain the third feature vector, the fourth feature vector and the fifth feature vector; The third feature vector, the fourth feature vector, and the fifth feature vector are respectively input into different branch networks of the trained license plate recognition model to obtain the target license plate recognition result. The target license plate recognition result is determined by the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result output by the different branch networks of the trained license plate recognition model. The process of fusing different types of features from the first and second feature vectors to obtain a third, fourth, and fifth feature vector includes: normalizing the first and second feature vectors; performing a merge operation on the normalized first and second feature vectors to obtain the third feature vector; performing a convolution operation on the first and second feature vectors; performing a horizontal feature connection on the convolutional first and second feature vectors to obtain the fourth feature vector; and performing a vertical feature connection on the first and second feature vectors to obtain the fifth feature vector. The process involves inputting the third, fourth, and fifth feature vectors into different branch networks of a pre-trained license plate recognition model to obtain the target license plate recognition result. This includes: inputting the third feature vector into the first branch network of the pre-trained license plate recognition model to output a single-layer license plate recognition result; inputting the fourth feature vector into the second branch network of the pre-trained license plate recognition model to output a double-layer license plate recognition result; inputting the fifth feature vector into the third branch network of the pre-trained license plate recognition model to output a license plate type recognition result; and determining the target license plate recognition result based on the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result.

2. The license plate recognition method according to claim 1, characterized in that, Before performing license plate detection on the image to be detected and obtaining the detection result of the image to be detected, the method further includes: Identify the image size of the image to be detected; If the image size of the image to be detected does not meet the preset size, the image to be detected is padded to form a square, thus obtaining the padded image to be detected; The edge-padded image to be detected is placed at the center point of a base plate of a preset size for training, thereby obtaining an image to be detected of a preset size.

3. The license plate recognition method according to claim 1, characterized in that, Perform license plate detection on the image to be detected to obtain the detection result of the image to be detected, including: The image to be detected is input into the pre-trained license plate recognition model, and the detection result of the image to be detected is output. The pre-trained license plate recognition model is also used to output the detection result of the image.

4. The license plate recognition method according to claim 1, characterized in that, If the category information of the image to be detected is a license plate, license plate correction is performed on the image to be detected to obtain a corrected image, including: The license plate is corrected based on projection mapping to obtain the corrected image.

5. The license plate recognition method according to claim 1, characterized in that, Extracting image features from the corrected image to obtain a target feature map, and then splitting the target feature map to obtain a first feature vector and a second feature vector, including: The corrected image is downsampled using CNN convolution to obtain the target feature map.

6. The license plate recognition method according to claim 1, characterized in that, The trained license plate recognition model is obtained through the following steps: Obtain a license plate image sample set, wherein the license plate image sample set contains multiple license plate image samples and a label corresponding to each license plate image sample; Extract image features from each license plate image sample in the license plate image sample set; The image features of each license plate image sample in the license plate image sample set are used as the input of the license plate recognition model, and the labels corresponding to each license plate image sample in the license plate image sample set are used as the output. The license plate recognition model is trained using a multi-task loss function to obtain a trained license plate recognition model.

7. A license plate recognition device, characterized in that, include: The image acquisition module is used to acquire the image to be detected; The license plate detection module is used to perform license plate detection on the image to be detected and obtain the detection result of the image to be detected. The detection result includes category information, which is either a license plate or a non-license plate. The license plate correction module is used to correct the license plate of the image to be detected when the category information of the image to be detected is a license plate, so as to obtain a corrected image; The license plate recognition module is used to extract the image features of the corrected image to obtain a target feature map, and to split the target feature map to obtain a first feature vector and a second feature vector. The license plate recognition module is also used to perform different types of feature fusion on the first feature vector and the second feature vector to obtain a third feature vector, a fourth feature vector and a fifth feature vector; The license plate recognition module is further configured to input the third feature vector, the fourth feature vector, and the fifth feature vector into different branch networks of the pre-trained license plate recognition model to obtain the target license plate recognition result, wherein the target license plate recognition result is determined by the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result output by the different branch networks of the pre-trained license plate recognition model; The license plate recognition module is further configured to normalize the first feature vector and the second feature vector, perform a merge operation on the normalized first feature vector and the second feature vector to obtain the third feature vector; perform a convolution operation on the first feature vector and the second feature vector, perform a horizontal feature connection on the convolutional first feature vector and the second feature vector to obtain the fourth feature vector; and perform a vertical feature connection on the first feature vector and the second feature vector to obtain the fifth feature vector. The license plate recognition module is further configured to input the third feature vector into the first branch network of the trained license plate recognition model and output a single-layer license plate recognition result; input the fourth feature vector into the second branch network of the trained license plate recognition model and output a double-layer license plate recognition result; input the fifth feature vector into the third branch network of the trained license plate recognition model and output a license plate type recognition result; and determine the target license plate recognition result based on the single-layer license plate recognition result, the double-layer license plate recognition result, and the license plate type recognition result.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the license plate recognition method of any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the license plate recognition method according to any one of claims 1 to 6.

Images