License plate recognition method, device and equipment and storage medium

Abstract

The application discloses a license plate recognition method, device and equipment and a storage medium. The method comprises the following steps: acquiring an image to be recognized; using a convolution layer of a trained license plate recognition model to extract a first character feature sequence corresponding to each target license plate region from at least one target license plate region in the image to be recognized; using a recurrent layer of the license plate recognition model to determine a probability matrix corresponding to each first character feature sequence; and using a conversion layer of the license plate recognition model to determine a license plate number of a corresponding license plate according to each first character feature sequence and the corresponding probability matrix.

Description

Technical Field

[0001] This application relates to the field of license plate recognition technology, and includes, but is not limited to, a license plate recognition method, apparatus, device, and storage medium. Background Technology

[0002] In related technologies, license plate recognition processes are quite complex, involving multiple network models, including target detection, target correction, and target recognition network models. This greatly increases the difficulty of deploying the algorithm on the platform. At the same time, cascading multiple models also results in a huge number of algorithm parameters and low recognition efficiency. Summary of the Invention

[0003] In view of this, embodiments of this application provide a license plate recognition method, apparatus, device, and storage medium.

[0004] In a first aspect, embodiments of this application provide a license plate recognition method, the method comprising: acquiring an image to be recognized; extracting a first character feature sequence corresponding to each of at least one target license plate region from the image to be recognized using a convolutional layer of a trained license plate recognition model; determining a probability matrix corresponding to each of the first character feature sequences using a recurrent layer of the license plate recognition model; and determining the license plate number of the corresponding license plate using a transformation layer of the license plate recognition model based on each of the first character feature sequences and the corresponding probability matrix.

[0005] Secondly, embodiments of this application provide a license plate recognition device, comprising: a first acquisition module for acquiring an image to be recognized; a first extraction module for extracting, using a convolutional layer of a trained license plate recognition model, a first character feature sequence corresponding to each target license plate region from the image to be recognized; a second extraction module for determining, using a recurrent layer of the license plate recognition model, a probability matrix corresponding to each first character feature sequence; and a determination module for determining, using a transformation layer of the license plate recognition model, the license plate number of the corresponding license plate based on each first character feature sequence and the corresponding probability matrix.

[0006] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the steps in the license plate recognition method described in the first aspect of this application.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the license plate recognition method described in the first aspect of embodiments of this application.

[0008] In this embodiment, the image to be recognized can be input into the license plate recognition model, which then outputs the license plate number. This integrates multiple recognition processes into a single license plate recognition model, directly identifying the license plate number from the input image. This allows for faster and more efficient license plate recognition, reducing the engineering complexity of the license plate recognition model. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating a license plate recognition method according to an embodiment of this application;

[0010] Figure 2 This is a schematic diagram of a first original image according to an embodiment of this application;

[0011] Figure 3 This is a schematic diagram of a cross-stage local multi-residual scale fusion module according to an embodiment of this application;

[0012] Figure 4 This is a schematic diagram of a license plate key point recognition network according to an embodiment of this application;

[0013] Figure 5 This is a schematic diagram of a bidirectional long short-term memory network according to an embodiment of this application;

[0014] Figure 6 This is a schematic diagram of an image reconstruction method according to an embodiment of this application;

[0015] Figure 7 This is a schematic diagram illustrating a method for determining a license plate area based on a target point according to an embodiment of this application;

[0016] Figure 8 This is a schematic diagram illustrating a method for extracting a first character feature sequence according to an embodiment of this application;

[0017] Figure 9 This is a schematic diagram of a license plate number generation method according to an embodiment of this application;

[0018] Figure 10 This is a schematic diagram of a license plate recognition model according to an embodiment of this application;

[0019] Figure 11 This is a schematic diagram of another license plate recognition model according to an embodiment of this application;

[0020] Figure 12 This is a schematic diagram of the composition structure of a license plate recognition device according to an embodiment of this application;

[0021] Figure 13 This is a schematic diagram of a hardware entity of an electronic device according to an embodiment of this application. Detailed Implementation

[0022] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0023] Figure 1 This is a flowchart illustrating a license plate recognition method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0024] Step 102: Obtain the image to be recognized;

[0025] The image to be identified can be an image including a license plate captured by a camera device, or an image corresponding to any frame in a video including a license plate captured by a camera device.

[0026] Step 104: Using the convolutional layer of the trained license plate recognition model, extract the first character feature sequence corresponding to each target license plate region from the image to be recognized;

[0027] The target license plate region can be the area where the license plate is located in the image to be identified. The image to be identified may contain one or more license plates, and each license plate corresponds to a target license plate region.

[0028] The image to be recognized can be feature-extracted by the convolutional layer of the license plate recognition model to obtain a feature map, the target license plate region can be determined from the feature map, and the target license plate region can be converted into a first character feature sequence.

[0029] Step 106: Using the recurrent layer of the license plate recognition model, determine the probability matrix corresponding to each of the first character feature sequences;

[0030] The recurrent layer of the license plate recognition model can be used as a classifier to output the probability matrix corresponding to the first character feature sequence.

[0031] Step 108: Using the conversion layer of the license plate recognition model, determine the license plate number of the corresponding license plate based on the first character feature sequence and the corresponding probability matrix corresponding to each target license plate area.

[0032] Specifically, the target character features that make up the corresponding license plate can be determined based on the first character feature sequence and the corresponding probability matrix corresponding to each target license plate area, and then the target character features can be spliced ​​together to form the license plate number of the corresponding license plate.

[0033] In this embodiment, the image to be recognized can be input into the license plate recognition model, which then outputs the license plate number. This integrates multiple recognition processes into a single license plate recognition model, directly identifying the license plate number from the input image. This allows for faster and more efficient license plate recognition, reducing the engineering complexity of the license plate recognition model.

[0034] This application also provides a license plate recognition method, the method comprising the following steps:

[0035] Step S202: Obtain at least one first original image and at least one second original image; wherein the resolution of the first original image is less than the resolution of the second original image;

[0036] Among them, such as Figure 2 As shown, the first original image can be a blurry image including the license plate taken in a natural, unconstrained scene, such as poor lighting conditions or extreme weather conditions, or a distorted or warped image including the license plate taken under the influence of physical factors such as the shooting angle. The first original image can also be an image of a license plate consisting of special characters (such as the symbols of certain countries). The first original image can be considered as a difficult sample in the training of the license plate recognition model.

[0037] The second original image can be an image including the license plate taken in a scene where the camera and the license plate capture position are fixed. Alternatively, the second original image can be an image of the license plate that does not include special characters.

[0038] Step S204: Using at least one of the first original images and at least one of the second original images, train the initial license plate recognition model to obtain the trained license plate recognition model;

[0039] Step S206: Obtain the image to be recognized;

[0040] Step S208: Using the convolutional layer of the trained license plate recognition model, perform feature extraction on the image to be recognized to obtain the feature map corresponding to the image to be recognized;

[0041] like Figure 3 As shown, the convolutional layer may include a multi-residual-scale feature fusion (RRSFF) module based on a cross-stage partial (CSP) network structure, referred to as CSP-RRSFF (Cross-Stage Local Multi-Residual Scale Fusion) module 30. The image to be recognized can be input into the CSP-RRSFF module 30, which extracts features from the image and outputs the corresponding feature map.

[0042] The CSP-RRSFF module 30 may include an RRSFF module 31, a transition layer 32, and a concatenation layer 33. The RRSFF module 31 is composed of multiple CBM layers 311 connected by residual connections and hierarchical connections (upsampling or downsampling operations). A CBM consists of a convolutional layer 3111, a batch normalization layer 3112 (which normalizes the features), and an activation layer 3113 (which makes the features non-linear). The RRSFF module 31 also includes an upsampling layer 312 and a downsampling layer 313, where β can be a constant between 0 and 1.

[0043] Connecting multiple CBM layers 311 through residual dense blocks increases the complexity of the license plate recognition model, allowing residual learning to be applied to different layers and increasing network capacity. Furthermore, a residual scaling strategy is employed, multiplying the residual by a constant β between 0 and 1 before each residual connection to prevent unstable training of the residual structure and facilitate training. Simultaneously, by upsampling and downsampling the feature maps, and fusing feature maps at different scales, the loss of feature information is reduced, and the utilization of feature information is enhanced.

[0044] By using CSP across stages on the RRSFF module 31, the gradient propagation path is increased, improving the network's learning ability. Furthermore, dimensionality reduction is achieved by using a 1x1 convolutional kernel in the transition layer 32, truncating the gradient flow and preventing the learning of duplicate gradient information between different layers. The concatenation layer 33 concatenates two or more feature maps along the channel or number dimension. In addition, CSP has the advantages of eliminating computational bottlenecks and reducing memory costs, making the license plate recognition model applicable to more hardware resources and facilitating its deployment.

[0045] Step S210: Identify target points in the feature map, the target points being used to determine the boundaries of the license plate area;

[0046] The target point is also known as the key point, extreme point, or boundary point of the license plate. The boundary point includes the points at the upper left, upper right, lower right, and lower left corners of the license plate. The boundary of the license plate area can be determined based on the target point, thereby determining the license plate area.

[0047] The convolutional layer may also include an RRPDB structure. In this embodiment, a license plate pole recognition network can be designed based on the RRPDB structure (Residual in Residual Point Detection Block, multiple residual keypoint detection module), such as... Figure 4 As shown in the dashed box, the RRPDB structure may include a CSP-RRSFF module 41, a heat map 42, and three convolutional neural networks 431, 432, and 433. It should be noted that before using the RRPDB structure to identify each keypoint, a convolutional neural network and a PReLU activation layer (Parametric Rectified Linear Unit, parameterized ReUL) can be used first.

[0048] First, the feature map is processed through a CSP-RRSFF structure 41. The extracted hybrid features are then passed through a first 1x1 fully convolutional neural network 431, splitting into upper and lower branches. The upper branch continues through a second 1x1 fully convolutional network 432 before entering the next layer; the lower branch generates a keypoint heatmap 42. Since the depth of the heatmap matrix is ​​the same as the number of poles in the training data (only 4), while other convolutional layers have higher depths (e.g., 256), the heatmap 42 needs to pass through a third 1x1 fully convolutional network 433 to adjust its depth to match that of the upper branch. Finally, the upper and lower branches are merged and used together as the input to the next layer. By sequentially processing four RRPDB structures, the four pole positions of the license plate can be obtained, thus determining the target point positions in the target feature map.

[0049] Step S212: Determine at least one initial license plate area based on the location of each target point;

[0050] Once the location of the target point is determined, the area enclosed by connecting the target points is the initial license plate area.

[0051] Step S214: Perform an affine transformation on each of the initial license plate regions to obtain the corresponding target license plate region.

[0052] The convolutional layer may also include an affine layer, which is used to perform an affine transformation on the initial license plate area to correct deformed or distorted license plates and obtain the target license plate area, which can be a front view of the license plate. An affine transformation, also known as an affine mapping, refers to the transformation of a vector space into another vector space by performing a linear transformation followed by a translation in geometry.

[0053] It should be noted that the RRPDB structure of the convolutional layer can be followed by an RRSFF module and a convolutional neural network for further feature extraction processing of the target license plate region.

[0054] Step S216: Convert each target license plate region into a corresponding character feature sequence;

[0055] Step S218: Using the recurrent layer of the license plate recognition model, determine the probability matrix corresponding to each of the first character feature sequences;

[0056] The recurrent layer may include a recurrent neural network, which may be a bidirectional long-short-term memory (Bi-LSTM) network.

[0057] like Figure 5 As shown, the bidirectional long short-term memory network includes an embedding layer, an encoding layer, a decoding layer, and a softmax layer.

[0058] At time t, the LSTM has three inputs: the current input value x of the network. t The output value h of the LSTM at the previous time step t-1 The cell state c at the previous moment t-1 The Bi-LSTM output has two parts: the current LSTM output value h. t The current cell state c t .

[0059] Three control switches are used: a forget gate, which determines the cell state c of the previous moment. t-1 How much is retained up to the current time c? t The input gate determines the network's input value x at the current moment. t How much is saved to cell state c? t The output gate determines the state c of the control unit. t How much output is given to the current LSTM output value h? t .

[0060] Step S220: Using the conversion layer of the license plate recognition model, determine the license plate number of the corresponding license plate based on the first character feature sequence and the corresponding probability matrix corresponding to each target license plate area.

[0061] In this embodiment, by training the license plate recognition model using a low-resolution first original image and a high-resolution second original image, the recognition accuracy of the license plate recognition model in handling blurry or low-resolution input images or video frames is improved. This allows the license plate recognition model to perform hard example mining, enhancing its generalization ability. The multi-residual scale fusion module based on the cross-stage local structure of convolutional layers offers several advantages: it strengthens the learning ability of the convolutional neural network, eliminates computational bottlenecks, and reduces memory costs. Furthermore, by fusing dense residual blocks, residual scaling factors, and feature maps at different scales (high resolution, low resolution), the network capacity is increased, improving the robustness of the license plate recognition model. By performing keypoint detection on four fixed points of the license plate, the initial license plate region can be determined more efficiently.

[0062] This application also provides a license plate recognition method, the method comprising the following steps:

[0063] Step S302: Acquire at least one first original image and at least one second original image; wherein the resolution of the first original image is less than the resolution of the second original image;

[0064] Step S304: Select at least one second processed image from at least one second original image;

[0065] In general, by adding a first original image with a lower resolution, the license plate recognition model can perform hard example mining to improve the generalization ability of the license plate recognition model. However, it is time-consuming and laborious to spend manpower and resources to collect a large number of first original images with a lower resolution. Therefore, data augmentation can be performed on the second original image with a higher resolution to expand the data and enrich the data diversity. A certain number of second processed images can be randomly selected from the second original image for data augmentation to obtain the second reconstructed image.

[0066] Step S306: Compress each of the second processed images using the encoder of the autoencoder model to obtain the feature map of the latent space of the corresponding second processed image;

[0067] like Figure 6 As shown, the encoder 601 of the autoencoder model can be used to compress the second processed image 602 to obtain the feature map 603 of the latent space of the second processed image 602. The relationship between the feature map 603 of the latent space of the second processed image 602 and the second processed image 602 can be expressed by the following formula (1):

[0068] s=E(x) (1);

[0069] Where x is the second processed image 602, E(x) represents the compression processing (also known as encoding processing) of the second processed image 602, and s represents the feature map 603 of the latent space of the second processed image 602.

[0070] Step S308: Using the decoder of the autoencoder model, the feature map of the latent space of each second processed image is mapped to the Gaussian distribution space to obtain the second reconstructed image corresponding to each second processed image;

[0071] like Figure 6 As shown, the feature map 603 of the latent space of the second processed image 602 can be mapped to the Gaussian distribution space using the decoder 604 of the autoencoder model to obtain the second reconstructed image 605 corresponding to the second processed image 602. The relationship between the feature map 603 of the latent space of the second processed image 602 and the second reconstructed image 605 can be expressed by the following formula (2):

[0072] o=D(s=D(E(x)) (2);

[0073] Where D(s) represents the mapping of the feature map 603 (i.e. s) of the latent space of the second processed image 602 to the Gaussian distribution space (also known as decoding processing), and o is the second reconstructed image 605. In some embodiments, each pixel in the feature map 603 of the latent space of the second processed image 602 can be added to Gaussian noise as shown in the following formula (3) to obtain the second reconstructed image 605.

[0074]

[0075] Here, S' can also represent the second reconstructed image 605, P ij The pixel values ​​in the i-th row and j-th column of the feature map matrix of feature map 603; μ and σ 2 represents the expectation and variance of the Gaussian distribution, respectively; W and H represent the width and height of feature map 603, and exp represents the exponential function with the natural constant e as the base.

[0076] Step S310: Using at least one of the first original image, the second reconstructed image, and other images in the second original image excluding the second processed image, train the initial license plate recognition model to obtain the trained license plate recognition model;

[0077] Step S312: Obtain the image to be recognized;

[0078] Step S314: Using the convolutional layer of the trained license plate recognition model, perform feature extraction on the image to be recognized to obtain the feature map corresponding to the image to be recognized;

[0079] Step S316: Identify target points in the feature map, the target points being used to determine the boundaries of the license plate area;

[0080] Since multiple license plates may exist in the images to be identified generated through video surveillance and other means, it is necessary to perform key point matching for the same license plate in order to determine multiple license plate areas.

[0081] Step S318: Determine the position attribute of each target point, the position attribute being used to characterize the position of the target point on the license plate area, the position attribute including a first position attribute, a second position attribute, a third position attribute and a fourth position attribute;

[0082] The first location attribute, the second location attribute, the third location attribute, and the fourth location attribute can respectively represent that the target point is located in one of the upper left, upper right, lower right, and lower left corners of the license plate area.

[0083] In some embodiments, the first position attribute indicates that the target point is the upper left corner of the license plate area, the second position attribute indicates that the target point is the upper right corner of the license plate area, the third position attribute indicates that the target point is the lower right corner of the license plate area, and the fourth position attribute indicates that the target point is the lower left corner of the license plate area.

[0084] In other embodiments, the first position attribute indicates that the target point is the upper left corner of the license plate area, the second position attribute indicates that the target point is the upper right corner of the license plate area, the third position attribute indicates that the target point is the lower left corner of the license plate area, and the fourth position attribute indicates that the target point is the lower right corner of the license plate area.

[0085] The four key points of a license plate number can be labeled in the order of top left, top right, bottom right, and bottom left. The license plate recognition model then identifies these key points in four stages according to the labeled order. The first to fourth stages correspond to the target points of the first to fourth positional attributes, respectively. When detecting multiple license plates, the key point recognition in the same stage identifies the coordinates of key points at the same location. For example, the first stage simultaneously identifies the coordinates of the top left key point of all license plates, the second stage identifies the coordinates of the top right key point, the third stage identifies the coordinates of the bottom right key point, and the fourth stage identifies the coordinates of the bottom left key point.

[0086] Step S320: Determine the confidence level of the line connecting any two target points of the first and second position attributes;

[0087] Step S322: Determine the first connection relationship between the target points of the first location attribute and the second location attribute based on the confidence level;

[0088] Wherein, the first connection relationship is either connected or disconnected; the confidence level between a target point of a certain first position attribute and the target point of each second position attribute can be determined, and the first connection relationship between the target point of the second position attribute with the highest confidence level and the target point of the first position attribute is determined to be connected, while the first connection relationship between the target points of other second position attributes with lower confidence levels and the target points of the first position attribute is determined to be disconnected.

[0089] Step S324: The first connection relationship is the line connecting the target points of the first position attribute and the second position attribute, which is the first connection line, and the target points of the first position attribute and the second position attribute are the target points of the same license plate area.

[0090] After identifying all key points in step S316 and determining the location attributes of all key points in step S318, a confidence matching algorithm can be used to determine which license plate these key points belong to. The confidence matching algorithm can be summarized as follows: if the line connecting two points intersects perpendicularly with the perpendicular line from a third point to that line, then this third point overlaps with any rectangle formed by the line connecting the two points; if the extension of the line connecting two points intersects perpendicularly with the perpendicular line from the third point to that extension, then this third point does not overlap with any rectangle formed by the line connecting the two points. In this scenario, since license plates do not overlap, the confidence matching algorithm can be used to determine the ownership of the license plate key points.

[0091] First, we can use A, B, C, and D to represent the key points of the first to fourth position attributes respectively (e.g., top left, top right, bottom right, bottom left, etc.). We can use numbers like 1, 2, and x to represent the number of license plates. Therefore, A1 represents the A-th (top left) key point of the first license plate, B... x This represents the B-th (top right) key point of the x-th license plate.

[0092] Since the coordinates of the four target points have been identified during key point recognition, it is possible to determine which position of the license plate each target point is on (top left, top right, bottom right, or bottom left). In other words, the positional attributes of the target points have been determined, but it is impossible to determine whether the target points belong to the same license plate.

[0093] Assuming there are two license plates (if the number of target points with the same location attribute identified is two, then there are definitely two license plates), the first stage identifies the target point with the first location attribute, which can be represented as A. x and A y The points identified in the second stage are target points with the second location attribute, which can be represented as B. u and B v Then it is necessary to determine A. xIs it with B? u and B v Which one is on the same license plate? Let's assume we determine A. x and B u On the same license plate, since there are two license plates, each license plate includes four points with different positional attributes. That is, each license plate cannot include two points with the same positional attribute. Therefore, A y and B v On another license plate.

[0094] like Figure 7 As shown, calculate A respectively. y To A x and B v The perpendicular distance h between the lines yv The horizontal distance is d yv A x and B v The length of the connection is P xv Because d yv In A x B v Between the straight lines, it indicates A y To A x B v The perpendicular point of the perpendicular line is A x B v Intersecting and overlapping, thus indicating that A x and B v It is impossible to connect them, i.e., d yv It is 0, therefore Figure 7 It is not marked in the text.

[0095] A y To A x and B u The perpendicular distance h between the lines yu The horizontal distance is d yu A x and B u The length of the connection is P xu A x To A y and B u The perpendicular distance h between the lines xu The horizontal distance is d xu A y and B u The length of the connection is P yu A x To A y and B v The perpendicular distance h between the lines xv The horizontal distance is d xv The length of the line connecting Ay and Bv is P. yv .

[0096] Use Q xv Represent edge A x B v The confidence level is then edge A x B v The confidence level can be expressed by the following formula (4):

[0097] Q xv =0 (4);

[0098] Among them, A y To A x B v The perpendicular point of the perpendicular line is at A. x B v Above, not in A x B v On the extension line, according to the confidence matching algorithm, if the line connecting the two points (i.e., A) is... x B v ) and the third point (i.e. A) y If the line connecting these two points intersects the perpendicular line from the third point, then this third point will overlap with any rectangle formed by the line connecting these two points. However, license plates will not overlap, indicating that A... y A x and B v On the same license plate, it is impossible for two target points A with the same location attributes to exist simultaneously on the same license plate. y and A x Therefore, A x and B v The first connection is no connection.

[0099] Use Q xu Represent edge A x B u The confidence level is then edge A x B u The confidence level can be expressed by the following formula (5):

[0100]

[0101] Among them, A y To A x B u The perpendicular point of the perpendicular line is at A. x B u On the extension line.

[0102] Use Q yu Represent edge A y B u The confidence level is then edge A y B u The confidence level can be expressed by the following formula (6):

[0103]

[0104] Among them, A x To A y B u The perpendicular point of the perpendicular line is at A. y B u On the extension line.

[0105] Use Q yv Represent edge A y B v The confidence level is then edge A y B v The confidence level can be expressed by the following formula (7):

[0106]

[0107] Among them, A x To A y B v The perpendicular point of the perpendicular line is at A. y B v On the extension line.

[0108] It allows comparing the confidence of a line connecting a target point with one location attribute to any target point with another location attribute. For example, it allows comparing the confidence of a target point A with the first location attribute. x Target point B with the second position attribute u Confidence Q of the connection between them xu Then compare A x Target point B with the second position attribute v Confidence Q of the connection between them xv , assuming Q xu Greater than Q xv Then you can choose to connect to A. x and B u A x and B u On the same license plate, A y and B v On the same license plate.

[0109] Similarly, assuming there are four license plates, the target point with the first location attribute identified in the first stage can also include A. e and A f The target points identified in the second stage based on the second location attribute may also include B. m and B n Q can then be calculated using the method described above. xm Q xn Q ym Q em Q ev and Qfn The confidence level of equilateral connections is used, and then the connection of key points is selected by comparing the confidence levels from the same key point in the previous stage to the key point in the next stage. For example, starting from key point A in the first stage... x Depart and proceed to the key point B of the second phase. u B v B m and B n Confidence level Q in xu Q xv Q xm and Q xn The size of the second-stage keypoint with the highest confidence level is used as A. x The connection object.

[0110] Step S326: If the target point of the third position attribute or the fourth position attribute satisfies the first position relationship with the first connecting line, the target point of the third position attribute and the fourth position attribute is determined as the target point of the same license plate area as the first position attribute and the second position attribute.

[0111] The license plate area is generally rectangular. If the target point of the third or fourth position attribute belongs to the same license plate as the first line, then the target point of the third or fourth position attribute is perpendicular to the first line. One perpendicular point coincides with the target point of the first position attribute, and the other perpendicular point coincides with the target point of the second position attribute.

[0112] Therefore, the first positional relationship can be that the point perpendicular to the first line drawn from the target point of the third positional attribute coincides with the target point of the first positional attribute or the second positional attribute. Similarly, the point perpendicular to the first line drawn from the target point of the fourth positional attribute coincides with the target point of the second positional attribute or the first positional attribute.

[0113] Step S328: Perform an affine transformation on each of the initial license plate regions to obtain the corresponding target license plate region.

[0114] Step S330: Determine the character feature vector corresponding to at least one sub-region of each target license plate region;

[0115] In this process, a sliding window of fixed unit length can be used to slide on the target license plate area to obtain multiple sliding areas; each sliding area is mapped to a character vector; each character vector includes multiple first character features, the unit length can be 1 pixel, the sliding step size can be 1 pixel, and the sliding areas on all target license plate areas are mapped to a set of character feature vectors with a length equal to the number of target license plate areas. This character feature vector can then be used to represent the feature vector of the character in the corresponding sub-region on the target license plate area.

[0116] Step S332: Generate a first character feature sequence corresponding to each target license plate region based on at least one character feature vector of each target license plate region;

[0117] The first character feature sequence can be a license plate feature matrix of size M×T, x = x1, x2, ..., x T This license plate feature matrix can be understood as consisting of M T-dimensional row vectors, or M T-dimensional license plate feature vectors. The M T-dimensional character feature vectors can include x1 = x 11 ,x 12 ,…,x 1T x2 = x 21 ,x 22 ,…,x 2T ... and x M =x M1 ,x M2 ,…,x MT Since license plates generally consist of a maximum of 8 characters, T can be 25, 26, or 27, etc., providing sufficient redundancy.

[0118] In addition, such as Figure 8 As shown, this license plate feature matrix 801 can also be understood as consisting of T M-dimensional column vectors, or T M-dimensional character feature vectors, where each column vector represents the feature vector of a character. The T M-dimensional character feature vectors may include x1 = x 11 ,x 21 ,…,x M1 x2 = x 12 ,x 22 ,…,x M2 ... and x T =x 1T ,x 2T ,…,x MT Since the types of characters generally include the Chinese characters for the province identifier, 24 letters, 10 numbers and 1 space, M can be 512, 520, etc., which also provides sufficient redundancy.

[0119] Step S334: Using the recurrent layer of the license plate recognition model, determine the probability matrix corresponding to each of the first character feature sequences;

[0120] The probability matrix, also known as the attribution probability matrix, is used to describe the first character feature sequence, which can have a corresponding attribution probability matrix y = y1, y2, ..., y T The attribution probability matrix can also be understood as being composed of M T-dimensional row vectors: y1 = y 11 ,y 12 ,…,y 1Ty2=y 21 ,y 22 ,…,y 2T and y M =y M1 ,y M2 ,…,y MT Etc. Or composed of T M-dimensional column vectors: y1 = y 11 ,y 21 ,…,y M1 y2=y 12 ,y 22 ,…,y M2 , ... and y T =y 1T ,y 2T ,…,y MT wait.

[0121] When the license plate feature matrix is ​​understood as consisting of M T-dimensional row vectors, x1 = x 11 ,x 12 ,…,x 1T The corresponding probability vector could be y1 = y 11 ,y 12 ,…,y 1T x2 = x 21 ,x 22 ,…,x 2T The corresponding probability vector could be y2 = y 21 ,y 22 ,…,y 2T , ..., x M =x M1 ,x M2 ,…,x MT The corresponding probability vector could be y M =y M1 ,y M2 ,…,y MT .

[0122] When the license plate feature matrix is ​​understood as consisting of T M-dimensional column vectors, x1 = x 11 ,x 21 ,…,x M1 The corresponding probability vector could be y1 = y 11 ,y 21 ,…,y M1 x2 = x 12 ,x 22 ,…,x M2 The corresponding probability vector could be y2 = y 12 ,y 22 ,…,y M2 , ..., x M =x 1T,x 2T ,…,x MT The corresponding probability vector could be y T =y 1T ,y 2T ,…,y MT .

[0123] Step S336: Using the conversion layer of the license plate recognition model, determine the first character feature with the highest probability in the corresponding probability vector of each character feature vector as the second character feature;

[0124] The conversion layer is used to convert the character feature sequence into a license plate number, with the character feature vector x1 = x 11 ,x 21 ,…,x M1 Taking an example, the corresponding probability vector is y1 = y 11 ,y 21 ,…,y M1 Assume that the probability with the highest probability in the probability vector is y. 21 Then, the first character feature x with the highest probability in the character feature vector can be selected. 21 Once the second character feature is identified, similarly, the second character feature can be determined sequentially from other character feature vectors, and a second character feature sequence can be generated from the second character feature.

[0125] Step S338: Filter out duplicate and invalid character features in the second character features to obtain the target character features;

[0126] The invalid character features can be spaces, hyphens, etc. In this embodiment, the Connectionist Temporal Classification (CTC algorithm) conditional probability function can be introduced to delete duplicate and invalid character features. The CTC algorithm is used to handle the alignment problem between input and output labels in sequence labeling. The CTC algorithm processes the license plate label sequence generated by the network. The probability distribution y is used to predict values ​​after removing duplicate and blank labels, and then mapped to the license plate label sequence l to output the license plate number. This requires a function B that maps sequences to each other, establishing the mapping relationship between the two sequences. Therefore, the conditional probability function mapping to sequence l can be defined as the sequence... The sum of probabilities is shown in formulas (8) and (9):

[0127]

[0128]

[0129] in, It refers to the probability of the label in the license plate label sequence at the time stamp t of the recurrent neural network. In short, the CTC conditional probability function is used to filter out the repeated and invalid license plate number labels generated by the network and generate complete and valid license plate number labels. As described above, the second character feature sequence may be "-Anhui-A A-P600-90-", and the target character features may include "Anhui", "A", "P", "6", "0", "9", and "0".

[0130] For example Figure 9 As shown, the second character feature sequence can be "-Wan-A A-P600-90-", and the target character features can include "Wan", "A", "P", "6", "0", "9", and "0".

[0131] Step S340: Determine the license plate number corresponding to the license plate based on the target character features.

[0132] For example Figure 9 As shown, the license plate number corresponding to the license plate can be "Wan AP6090".

[0133] In the embodiments of the present application, data can be augmented by means of data augmentation based on existing data to enrich the diversity of data. In addition, while identifying key points, the confidence matching algorithm is used to match the key points of the same license plate, which greatly improves the license plate recognition efficiency and recognition accuracy, and can identify multiple license plate numbers.

[0134] Automatic License Plate Recognition (ALPR) is of great significance in traffic management, digital security monitoring, vehicle identification, parking management, etc. in big cities, but it is also challenging. Affected by many natural factors, the accuracy of the model network recognition is not high. For example: blurred images, poor lighting conditions, changes in license plate numbers (including special characters, such as Chinese and Japanese signs), physical effects (deformation), extreme weather conditions, etc.; the above natural factors have a greater impact on the license plate detection and license plate character recognition stages.

[0135] In related technologies, license plate recognition methods include: first, automatic license plate recognition in scenarios where the camera and license plate acquisition positions are fixed; and second, automatic license plate recognition in natural, unconstrained scenarios. In relatively fixed and constrained scenarios, license plate recognition is less difficult. Traditional image processing operations (such as binarization, filtering, and edge contour detection) can yield relatively accurate license plate numbers, which are then recognized using OCR (Optical Character Recognition) technology. However, in natural, unconstrained scenarios, traditional image processing methods are unsuitable. Due to complex background factors, deep learning networks are required to recognize license plate numbers. The typical license plate recognition process involves: image preprocessing, license plate detection, license plate correction, and license plate recognition. This involves multiple deep learning networks (object detection network, object angle correction network, and recognition network), each performing its specific function to complete the license plate recognition task.

[0136] The disadvantages of the first license plate recognition method are: poor accuracy in segmenting license plate characters during license plate OCR recognition; susceptibility to external adverse factors such as noise, low resolution, blurring, and distortion of license plate images, resulting in poor robustness; additional space required for character segmentation (OCR); and a large workload and time-consuming process for annotating individual characters.

[0137] The disadvantages of the second license plate recognition method mentioned above are: the license plate recognition process is relatively complex, involving multiple network models, including target detection, target angle correction, and target recognition network models, which greatly increases the difficulty of deploying the algorithm on the platform; at the same time, the cascading of multiple models also results in a huge number of algorithm parameters and low inference efficiency; the automatic license plate recognition system has multiple sub-tasks, which seriously slows down the recognition rate; in unconstrained scenarios, the license plate recognition algorithm solves the problem of license plate angle correction, but does not handle abnormal situations such as image blurring, reflection, and dimness caused by extreme weather conditions.

[0138] Both of the above methods can only perform license plate recognition tasks for specific scenarios and unique vehicles. The models can only handle a single scenario and cannot recognize license plates of multiple vehicles, which increases the additional time and financial costs.

[0139] To enable the algorithm to be deployed in any scenario, this application's embodiments design the entire license plate recognition system as an end-to-end network structure, reducing the complexity of model engineering. In the license plate recognition network model of this application, the entire algorithm implementation process, from image input to model inference and then to outputting license plate number information, differs significantly from license plate recognition systems in related technologies. For example... Figure 10As shown, this embodiment integrates multiple processes (vehicle detection network, license plate detection network, and license plate character recognition network) into a single network, directly identifying the license plate number from the input image. The entire network abandons the traditional method of vehicle or license plate detection, instead employing a method of keypoint detection at the four vertices of the license plate, which is faster and more efficient. Then, an affine transformation layer is used to perform a frontal view transformation on the four vertices of the license plate. Finally, a Bi-LSTM network from the RNN (Recurrent Neural Network) is used to recognize the license plate characters.

[0140] In addition, to enrich the data samples for model training and make them more consistent with complex and realistic scenarios, this application also proposes a data augmentation method based on an autoencoder model.

[0141] Autoencoded data augmentation networks:

[0142] To improve the recognition accuracy of license plate recognition models when encountering blurry or low-resolution input images or video frames, data from such cases can be added during model training to allow the model to perform hard example mining and improve its generalization ability. However, collecting such data is time-consuming and labor-intensive. Therefore, to achieve this goal, this application expands the data based on existing data using data augmentation techniques to enrich the data diversity.

[0143] Data augmentation methods typically include image rotation, scaling, cropping, and color alteration, as well as more advanced operations such as image subtraction, image averaging, histogram equalization, and filtering. However, these common image augmentation operations are not used in this application embodiment. Instead, this application embodiment provides a novel data augmentation strategy: using an autoencoder model to automatically generate images to achieve the purpose of image augmentation. Moreover, the autoencoder model is an unsupervised learning model, which does not require further data collection for training.

[0144] An autoencoder model typically consists of an encoder and a decoder. The encoder compresses the input data x into a latent representation s, which can be represented by the encoding function s = E(x). The decoder then reconstructs the input from the latent representation and outputs the data, which can be represented by the function o = D(s). Therefore, o = D(E(x)).

[0145] License plate recognition model:

[0146] like Figure 11As shown, the entire license plate recognition system can complete the license plate recognition task through a single license plate recognition model. The license plate recognition model proposed in this application modifies the traditional license plate number recognition method, not only detecting the position of the license plate frame based on key points, but also directly recognizing the license plate number. The entire license plate recognition model can be divided into three main parts: 1) a convolutional layer, which extracts character feature sequences from the input image to be recognized; 2) a recurrent layer, which predicts the label distribution (i.e., probability distribution) of the character feature sequences in each frame; and 3) a transformation layer, which converts the predicted label distribution of the character feature sequences in each frame into the final license plate sequence (i.e., license plate number).

[0147] Convolutional layers in a license plate recognition model:

[0148] In the convolutional layer, this application proposes a multi-residual in residual scale feature fusion (RRSFF) module based on the cross-stage partial (CSP) network structure, referred to as CSP-RRSFF.

[0149] Using the proposed CSP-RRSFF network structure as the basic module, after extracting image features to obtain the feature map corresponding to the image to be identified, unlike traditional target detection algorithms for license plate detection, this embodiment uses key point recognition technology to identify the four poles of the license plate, and then uses the positions of the four poles to determine the position of the license plate, thereby completely extracting the license plate and feeding it into the subsequent recurrent neural network. Furthermore, to prevent deformation of the license plate in the image due to the shooting angle, this embodiment uses an affine transformation layer to handle the deformation problem.

[0150] This application's embodiments can design a license plate pole recognition network based on the RRPDB structure (Residual in Residual Point DetectionBlock, a multi-residual key point detection module), such as... Figure 4 As shown in the dashed box, the RRPDB structure may include a CSP-RRSFF structure, a heat map, and three convolutional networks. It should be noted that before the RRPDB structure, the input feature map can be passed through convolutional layers and then activated by PReLU (Parametric Rectified Linear Unit) activation layers.

[0151] First, the extracted hybrid features, obtained through a CSP-RRSFF structure, are processed by a first 1x1 fully convolutional network and split into two branches. The upper branch continues through a second 1x1 fully convolutional network before entering the next layer; the lower branch first passes through a first 1x1 convolution to generate a heat map. Since the depth of the heat map matrix is ​​the same as the number of poles in the training data (only 4), while other convolutional layers have a higher depth (256), the heat map needs to pass through a third 1x1 fully convolutional network to adjust the depth to match that of the upper branch. Finally, the upper and lower branches are merged and used together as the input to the next layer. By sequentially processing four RRPDB structures, the four pole positions of the license plate can be obtained. After obtaining the coordinates of the four poles, it is necessary to consider how to perform key point matching for the same license plate if multiple license plates exist in the video surveillance, thereby determining the multiple license plate regions.

[0152] Therefore, this application proposes a method for matching key points from the same license plate, called the confidence matching algorithm. First, the four key points of the license plate number can be labeled in the order of top left, top right, bottom right, and bottom left. The license plate recognition model then identifies the key points in four stages according to the labeling order. When detecting multiple license plates, key point identification in the same stage finds the coordinates of key points at the same location. For example, in the first stage, the coordinates of the top left key point of all license plates are identified simultaneously; in the second stage, the coordinates of the top right key point are identified; in the third stage, the coordinates of the bottom right key point are identified; and in the fourth stage, the coordinates of the bottom left key point are identified. After all key points are identified, the confidence matching algorithm proposed in this application can be used to determine which license plate these key points belong to.

[0153] The confidence matching algorithm can be summarized as follows: if the line connecting two points intersects perpendicularly with the perpendicular line from a third point to that line, then this third point overlaps with any rectangle formed by the line connecting the two points; if the extension of the line connecting two points intersects perpendicularly with the perpendicular line from the third point to that extension, then this third point does not overlap with any rectangle formed by the line connecting the two points. In this scenario, since license plates do not overlap, the confidence matching algorithm can be used to determine the ownership of key points on the license plate.

[0154] After matching key points of different license plates to determine the initial license plate region corresponding to each license plate in at least one license plate, an affine transformation can be performed on each initial license plate region to correct the license plate and obtain a front view. Finally, the target license plate region is captured. After converting the target license plate region into at least one character feature sequence, it is fed into a recurrent neural network through a multi-residual scale fusion structure RRSFF and a convolutional layer.

[0155] After detecting the target license plate region in the image using convolutional layers and extracting features, the feature map needs to be converted into at least one character feature sequence, which is then used as input to the recurrent neural network Bidirectional Long Short-Term Memory (Bi-LSTM). The specific operation process is as follows:

[0156] The license plate feature maps are obtained after multiple convolutions; a sliding window of fixed unit length (1 pixel) is used to slide on the feature maps with a stride of 1 pixel; the sliding region on all feature maps is mapped into a set of character feature vectors with a length equal to the number of feature maps. These character feature vectors can then be used to represent the feature vectors of the characters in the corresponding region on the feature map.

[0157] After obtaining the first character feature sequence of the license plate, the first character feature sequence is then input into a recurrent neural network: Bi-LSTM (Bi-directional Long-Short Term Memory) network, in order to extract the first character feature sequence from each set of character feature sequences (x = x1, x2, ..., x...). T Predict character label y in ) T Recurrent Neural Networks (RNNs) possess a powerful ability to capture contextual information within character feature sequences. In image-based character feature sequence recognition, using contextual cues is more stable and advantageous than processing each character individually. RNNs can backpropagate the loss to the input convolutional layers, and both RNN and CNN convolutional neural network layers can be trained simultaneously in a single network. RNNs can handle character feature sequences of arbitrary length.

[0158] After passing through the Bi-LSTM network, each input feature sequence vector outputs a probability value y, thus yielding a probability distribution y = y1, y2, ..., y of length T. T Then, this probability distribution is used to determine the labels of the license plate characters, and finally, after filtering, the complete license plate number can be obtained. Considering the cumbersome and inefficient nature of labeling individual license plate characters, this embodiment of the application abandons the labeling of individual characters, and only requires providing the image to be identified and its corresponding license plate label sequence (i.e., character feature sequence).

[0159] This application embodiment uses a license plate key point and character recognition network to recognize license plate numbers in real environments, which can improve the speed and accuracy of license plate number detection and recognition, and can be deployed end-to-end on low-computing-power edge AI (Artificial Intelligence) devices.

[0160] This application embodiment can process license plate image anomalies caused by extreme weather conditions, enabling accurate identification of license plate numbers even under such weather conditions, and completing license plate recognition with fewer modules.

[0161] This application proposes a network structure that integrates an image feature extraction network, a key point extraction structure, a key point matching algorithm, and a license plate sequence feature extraction network. The gradient of the sequence feature extraction network is backpropagated to the image feature extraction network to reversely regulate the effect of image feature extraction, making the information flow of the entire network more stable, reducing the multi-task process of traditional license plate recognition, and forming an end-to-end solution.

[0162] This application presents a multi-residual scale fusion module based on a cross-stage local structure. Firstly, this module possesses the advantages of a cross-stage local structure: it enhances the learning ability of the convolutional neural network; it eliminates computational bottlenecks; and it reduces memory costs. Furthermore, by fusing dense residual blocks, residual scaling factors, and feature maps at different scales (high resolution, low resolution), it increases network capacity and improves the model's robustness.

[0163] This application proposes a key point matching method, namely a confidence-based matching algorithm. By matching key points of the same license plate while identifying key points, the recognition efficiency is greatly improved, and multiple license plate numbers can be identified.

[0164] In related technologies, automatic license plate recognition systems involve multiple sub-tasks (vehicle detection, license plate detection, license plate correction and transformation, license plate character recognition, license plate number recognition, etc.), which severely slows down the recognition speed and makes deployment difficult. However, the embodiments of this application only require the construction of a single network to solve the task of recognizing multiple types of license plates, demonstrating their advanced nature.

[0165] The confidence matching algorithm proposed in this application can identify multiple license plates based on license plate key points. While identifying key points, it matches key points of the same license plate, which greatly improves the recognition efficiency.

[0166] This application addresses the issue of license plate image anomalies caused by extreme weather conditions, which affect license plate recognition performance. It designs an autoencoder model to automatically enhance the richness and diversity of data, thereby improving the accuracy of license plate recognition under abnormal conditions.

[0167] It should be noted that, in the embodiments of this application, if the above-described license plate recognition method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a mobile phone, tablet computer, desktop computer, personal digital assistant, navigator, digital phone, video phone, television, sensor device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0168] Based on the foregoing embodiments, this application provides a license plate recognition device, which includes various modules that can be implemented by a processor in the device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0169] Figure 12 This is a schematic diagram of the composition structure of a license plate recognition device according to an embodiment of this application, as shown below. Figure 12 As shown, the device 1200 includes a first acquisition module 1201, a first extraction module 1202, a second extraction module 1203, and a determination module 1204, wherein:

[0170] The first acquisition module 1201 is used to acquire the image to be recognized;

[0171] The first extraction module 1202 is used to extract a first character feature sequence corresponding to each target license plate region from the image to be recognized using the convolutional layer of the trained license plate recognition model.

[0172] The second extraction module 1203 is used to determine the probability matrix corresponding to each first character feature sequence by utilizing the loop layer of the license plate recognition model.

[0173] The determination module 1204 is used to determine the license plate number of the corresponding license plate by utilizing the conversion layer of the license plate recognition model, based on each first character feature sequence and the corresponding probability matrix.

[0174] In some embodiments, the apparatus further includes: a second acquisition module, configured to acquire at least one first original image and at least one second original image; wherein the resolution of the first original image is less than the resolution of the second original image; and a training module, configured to train an initial license plate recognition model using at least one first original image and at least one second original image to obtain a trained license plate recognition model.

[0175] In some embodiments, the training module includes: a selection submodule for selecting at least one second processed image from at least one second original image; a compression submodule for compressing each second processed image using the encoder of an autoencoder model to obtain a feature map of the latent space corresponding to the second processed image; a mapping submodule for mapping the feature map of the latent space of each second processed image to a Gaussian distribution space using the decoder of the autoencoder model to obtain a second reconstructed image corresponding to each second processed image; and a training submodule for training an initial license plate recognition model using at least one first original image, the second reconstructed image, and other images from the second original image excluding the second processed image to obtain a trained license plate recognition model.

[0176] In some embodiments, the first extraction module includes: an extraction submodule, configured to extract features from the image to be recognized using the convolutional layer of a trained license plate recognition model to obtain a feature map corresponding to the image to be recognized; a recognition submodule, configured to recognize target points in the feature map, the target points being used to determine the boundaries of the license plate region; a first determination submodule, configured to determine at least one initial license plate region based on the position of each target point; a transformation submodule, configured to perform an affine transformation on each initial license plate region to obtain a corresponding target license plate region; and a conversion submodule, configured to convert each target license plate region into a corresponding first character feature sequence.

[0177] In some embodiments, the first determining submodule includes: a first determining unit, configured to determine the position attribute of each target point, the position attribute being used to characterize the position of the target point on the license plate area, the position attribute including a first position attribute, a second position attribute, a third position attribute, and a fourth position attribute; a second determining unit, configured to determine the confidence level of the line connecting any two target points of the first position attribute and the second position attribute; a third determining unit, configured to determine a first connection relationship between the target points of the first position attribute and the second position attribute based on the confidence level; a fourth determining unit, configured to determine the line connecting the target points of the first position attribute and the second position attribute that are connected by the first connection relationship as a first connecting line, and to determine the target points of the first position attribute and the second position attribute that are connected as target points of the same license plate area; and a fifth determining unit, configured to determine the target points of the third position attribute and the fourth position attribute as target points of the same license plate area as the target points of the first position attribute and the second position attribute when the target points of the third position attribute or the fourth position attribute satisfy the first position relationship with the first connecting line.

[0178] In some embodiments, the second extraction module includes: a second determining submodule, configured to determine a first character feature vector corresponding to at least one subregion of each target license plate region; and a generating submodule, configured to generate a first character feature sequence corresponding to the target license plate region based on the first character feature vector of each subregion of each target license plate region.

[0179] In one embodiment, the determining module includes: a third determining submodule, used to use the conversion layer of the license plate recognition model to determine the first character feature with the highest probability in the corresponding probability vector of each character feature vector as the second character feature; a filtering submodule, used to filter out duplicate and invalid character features in the second character feature to obtain the target character feature; and a fourth determining submodule, used to determine the license plate number of the corresponding license plate based on the target character feature.

[0180] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0181] Correspondingly, embodiments of this application provide a device, Figure 13 This is a schematic diagram of a hardware entity of an electronic device according to an embodiment of this application, such as... Figure 13As shown, the hardware entity of the device 1300 includes a memory 1301 and a processor 1302. The memory 1301 stores a computer program that can run on the processor 1302. When the processor 1302 executes the program, it implements the steps in the license plate recognition method of the above embodiment.

[0182] The memory 1301 is configured to store instructions and applications executable by the processor 1302, and can also cache data to be processed or already processed by the processor 1302 and the various modules in the device 1300 (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0183] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the license plate recognition method provided in the above embodiments.

[0184] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects as the device embodiments. For technical details not disclosed in the storage medium and method embodiments of this application, please refer to the descriptions of the device embodiments of this application for understanding.

[0185] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0186] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0188] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this application may all be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the integrated unit may be implemented in hardware or in a combination of hardware and software functional units.

[0189] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the related technology, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause computer devices (which may be mobile phones, tablets, desktops, personal digital assistants, navigators, digital phones, video phones, televisions, sensing devices, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0190] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined to obtain new product embodiments without conflict. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined to obtain new method embodiments or device embodiments without conflict.

[0191] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A license plate recognition method, characterized in that, The method includes: Acquire the image to be recognized; The convolutional layer of the trained license plate recognition model is used to extract features from the image to be recognized, thereby obtaining the feature map corresponding to the image to be recognized. Identify target points in the feature map, which are used to determine the boundaries of the license plate area; Based on the location of each target point, at least one initial license plate area is determined; Determine the position attributes of each target point, the position attributes being used to characterize the position of the target point on the license plate area, the position attributes including a first position attribute, a second position attribute, a third position attribute, and a fourth position attribute; Determine the confidence level of the line connecting any two target points with the first and second position attributes; Based on the confidence level, a first connection relationship between the target points of the first location attribute and the second location attribute is determined; The first connection relationship is defined as the line connecting the target points of the first position attribute and the second position attribute, and the connected target points of the first position attribute and the second position attribute are defined as target points of the same license plate area. If the target point of the third position attribute or the fourth position attribute satisfies the first position relationship with the first connecting line, the target point of the third position attribute and the fourth position attribute are determined as the target point of the same license plate area as the first position attribute and the second position attribute; the first position relationship is that the target point of the third position attribute and the target point of the fourth position attribute are respectively perpendicular to the first connecting line, and the perpendicular points respectively coincide with the target point of the first position attribute or the target point of the second position attribute. Perform an affine transformation on each of the initial license plate regions to obtain the corresponding target license plate regions; Each target license plate region is converted into a corresponding first character feature sequence; Using the recurrent layer of the license plate recognition model, the probability matrix corresponding to each first character feature sequence is determined; Using the conversion layer of the license plate recognition model, the license plate number of the corresponding license plate is determined based on the feature sequence of each first character and the corresponding probability matrix.

2. The method according to claim 1, characterized in that, The method further includes: Acquire at least one first original image and at least one second original image; wherein the resolution of the first original image is less than the resolution of the second original image; The initial license plate recognition model is trained using at least one of the first original images and at least one of the second original images to obtain a trained license plate recognition model.

3. The method according to claim 2, characterized in that, The step of training an initial license plate recognition model using at least one first original image and at least one second original image to obtain a trained license plate recognition model includes: Select at least one second processed image from at least one second original image; The encoder of the autoencoder model is used to compress each of the second processed images to obtain the feature map of the latent space of the corresponding second processed image. The feature map of the latent space of each second processed image is mapped to a Gaussian distribution space using the decoder of the autoencoder model to obtain the second reconstructed image corresponding to each second processed image; The initial license plate recognition model is trained using at least one of the first original images, the second reconstructed image, and other images in the second original image excluding the second processed image, to obtain a trained license plate recognition model.

4. The method according to claim 1, characterized in that, The step of converting each target license plate region into a corresponding first character feature sequence includes: Determine the first character feature vector corresponding to at least one sub-region of each target license plate region; Based on the first character feature vector of each sub-region of each target license plate region, a first character feature sequence of the corresponding target license plate region is generated.

5. The method according to claim 4, characterized in that, The step of using the conversion layer of the license plate recognition model to determine the license plate number of the corresponding license plate based on the first character feature sequence and the corresponding probability matrix corresponding to each target license plate region includes: Using the conversion layer of the license plate recognition model, the first character feature with the highest probability in the corresponding probability vector of each character feature vector is determined as the second character feature; Filter out duplicate and invalid character features from the second character features to obtain the target character features; Based on the target character features, the license plate number of the corresponding vehicle is determined.

6. A license plate recognition device, characterized in that, The device includes: The first acquisition module is used to acquire the image to be recognized; The first extraction module is used to extract features from the image to be recognized using the convolutional layer of a trained license plate recognition model to obtain a feature map corresponding to the image to be recognized; identify target points in the feature map, the target points being used to determine the boundaries of the license plate region; determine at least one initial license plate region based on the position of each target point; determine the position attribute of each target point, the position attribute being used to characterize the position of the target point on the license plate region, the position attribute including a first position attribute, a second position attribute, a third position attribute, and a fourth position attribute; determine the confidence level of the line connecting any two target points of the first position attribute and the second position attribute; determine a first connection relationship between the target points of the first position attribute and the second position attribute based on the confidence level; and connect the target points of the first position attribute and the second position attribute to the target points of the second position attribute. The line connecting the punctuation marks is defined as the first line, and the target points of the first and second positional attributes connected by it are defined as target points of the same license plate area. If the target point of the third or fourth positional attribute satisfies a first positional relationship with the first line, the target points of the third and fourth positional attributes are defined as target points of the same license plate area as the first and second positional attributes. The first positional relationship is that the target points of the third and fourth positional attributes are perpendicular to the first line, and the perpendicular points respectively coincide with the target points of the first or second positional attributes. An affine transformation is performed on each initial license plate area to obtain the corresponding target license plate area. Each target license plate area is converted into a corresponding first character feature sequence. The second extraction module is used to determine the probability matrix corresponding to each first character feature sequence by utilizing the loop layer of the license plate recognition model. The determination module is used to determine the license plate number of the corresponding license plate based on the transformation layer of the license plate recognition model and each first character feature sequence and the corresponding probability matrix.

7. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the license plate recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the license plate recognition method according to any one of claims 1 to 5.

Images