Training methods, correction methods, devices, equipment and media for license plate correction models

Abstract

This application discloses a training method, correction method, device, equipment, and medium for a license plate correction model, relating to the field of machine learning. The method includes acquiring a training set, which includes pre-processed license plate images and corresponding real license plate bounding box annotations; training a pre-set model based on the training set to obtain a license plate correction model; wherein the pre-set model includes a backbone network, a neck network, and a head network connected in sequence; the backbone network includes four convolutional network layers connected in sequence; the neck network includes a fully connected layer, a randomly deactivated layer, and an activation function layer connected in sequence; the head network includes a linear output layer; the license plate correction model is used to process the license plate image to be corrected and output the predicted bounding box parameters of the license plate image to be corrected. This application can accurately correct the license plate position while considering hardware resources.

Description

Technical Field

[0001] This application relates to the field of charging technology, and in particular to a training method, correction method, device, equipment and medium for a license plate correction model. Background Technology

[0002] Neural network-based license plate recognition technology has been widely used in daily life and the railway sector. The technology mainly includes license plate location detection and license plate number recognition.

[0003] License plate location detection requires locating the license plate position within the entire image. Commonly used open-source object detection networks include the Faster R-CNN (Region-based Convolutional Neural Networks) series, the SSD (Single Shot MultiBox Detector) series, and the YOLO (YouOnly Look Once) series. With the gradual maturation of object detection technology and the growth of mobile application requirements, lightweight detection networks such as the MobileNet series, the ShuffleNet series, and the EfficientNet-Lite series have emerged, which can balance efficiency and accuracy on mobile devices.

[0004] However, in practical applications, there are some specific and simple mobile application scenarios where even if a lightweight model is deployed, the parameters are too large and the forward inference time is relatively long, making it difficult to adapt to hardware resources. Summary of the Invention

[0005] The purpose of this application is to provide a training method, correction method, device, equipment and medium for a license plate correction model, which can accurately correct the license plate position while taking into account hardware resources.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] Firstly, this application provides a training method for a license plate correction model, including:

[0008] Obtain the training set, which includes preprocessed license plate images and corresponding real license plate frame annotations;

[0009] A license plate correction model is obtained by training a pre-defined model on a training set. The pre-defined model includes a backbone network, a neck network, and a head network connected in sequence. The backbone network is used to acquire high semantic features of the images in the training set. The neck network is used to perform secondary processing and integration of the high semantic features. The head network is used to map the features output by the neck network to the output result. The backbone network includes four convolutional network layers connected in sequence. The neck network includes a fully connected layer, a random deactivation layer, and an activation function layer connected in sequence. The head network includes a linear output layer. The license plate correction model is used to process the license plate image to be corrected and output the predicted bounding box parameters of the license plate image to be corrected.

[0010] Optionally, the linear output layer is used to output the predicted bounding box parameters. Preset anchor boxes are set in the linear output layer, and the correspondence between the preset anchor boxes and the predicted bounding boxes is described by the following formula:

[0011] ;

[0012] ;

[0013] ;

[0014] ;

[0015] in, The coordinates of the center point of the preset anchor box in the x-direction. The coordinates of the center point of the preset anchor point frame in the y-direction. The width of the preset anchor point frame, Preset the height of the anchor box; sigmoid() is the sigmoid function, exp() is the exponential function; The coordinates of the center point of the predicted bounding box in the x-direction are: The coordinates of the center point of the predicted bounding box in the y-direction are: The width of the prediction box. The height of the predicted bounding box, The width of the prediction box is adjusted based on the width of the preset anchor point box. The height of the predicted bounding box is the height scaling factor based on the preset anchor point, and (t_cx, t_cy) is the offset of the center point of the predicted bounding box based on the center point of the preset anchor point bounding box.

[0016] Optionally, during the training of the preset model based on the training set, a function based on CIOU loss and constraint loss is used as the loss function of the preset model, which is expressed by the following formula:

[0017] Loss=CIOULoss+ ContraintLoss;

[0018] Here, CIOULoss is the CIOU base loss function, designed to make the predicted bounding box approximate the actual license plate bounding box, while ContraintLoss is the constraint loss function, designed to satisfy the constraint conditions when the predicted bounding box approximates the actual license plate bounding box. These are the weighting coefficients of the constraint loss function.

[0019] Optionally, the CIOU basic loss function is described by the following formula:

[0020] ;

[0021] Where Iou represents the intersection-union ratio between the predicted bounding box and the actual license plate bounding box; represents the Euclidean distance between the center point of the predicted bounding box and the center point of the actual license plate frame; c represents the diagonal length of the minimum closed region covering the predicted bounding box and the actual license plate frame; This represents the weighting coefficient used to balance the influence of center point distance and aspect ratio; v represents the aspect ratio similarity measure.

[0022] Optionally, the constraint loss function is described by the following formula:

[0023] ;

[0024] in, This represents the i-th constraint quantity that satisfies the corresponding constraint condition. This represents the penalty coefficient of the sigmoid function; where the constraint quantities include... , , and The expressions for the constraint quantities and their corresponding constraint conditions are as follows:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] Where (pred_xmin,pred_ymin) are the coordinates of the top left corner of the predicted bounding box, (pred_xmax,pred_ymax) are the coordinates of the bottom right corner of the predicted bounding box, (gt_xmin,gt_ymin) are the coordinates of the top left corner of the actual license plate frame, and (gt_xmax,gt_ymax) are the coordinates of the bottom right corner of the actual license plate frame.

[0030] Secondly, this application provides a method for correcting license plates, including:

[0031] Obtain the image of the license plate to be corrected using a preset frame;

[0032] The license plate correction model is used to process the license plate image to be corrected to obtain the bounding box parameters of the license plate image; wherein, the license plate correction model is the license plate correction model obtained according to the training method of the above-mentioned license plate correction model;

[0033] The prediction box is obtained based on the prediction box parameters.

[0034] Optionally, the prediction box parameters include the center point offset of the prediction box based on the center point of a preset anchor point and the width and height scaling of the prediction box based on the preset anchor point; obtaining the prediction box according to the prediction box parameters includes:

[0035] The center point, height, and width of the prediction box are obtained based on the center point offset of the prediction box relative to the center point of the preset anchor box, the width and height scaling of the prediction box relative to the preset anchor box, the center coordinates of the preset anchor box, and the width and height of the preset anchor box.

[0036] The coordinates of the upper left and upper right corners of the prediction box are obtained based on the center point, height, and width of the prediction box, and the prediction box is output based on the coordinates of the upper left and upper right corners.

[0037] Thirdly, this application provides a license plate correction device, comprising:

[0038] The deployment module is used to receive the deployment parameters of the license plate correction model, which is a license plate correction model obtained according to the training method of the above-mentioned license plate correction model.

[0039] The correction module is used to obtain the license plate image to be corrected through a preset bounding box, and to process the license plate image to be corrected through a license plate correction model to obtain the predicted bounding box parameters of the license plate image to be corrected.

[0040] The output module is used to obtain the prediction box of the license plate image to be corrected based on the prediction box parameters.

[0041] Fourthly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the training method or license plate correction method of any of the above.

[0042] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the training method for the license plate correction model or the license plate correction method described above.

[0043] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the training method or the license plate correction method of any of the above-mentioned methods.

[0044] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0045] This application provides a training method, correction method, device, equipment, and medium for a license plate correction model. The model is obtained by training a pre-set model using a training set. The pre-set model includes a backbone network, a neck network, and a head network connected in sequence. The backbone network includes four convolutional network layers connected in sequence. The neck network includes a fully connected layer, a randomly deactivated layer, and an activation function layer connected in sequence. The license plate correction model is used to process images of license plates to be corrected and output predicted bounding boxes for those images. The pre-set model has a simple structure, is adaptable to mobile deployment, and can accurately correct license plates. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating a training method for a license plate correction model according to an embodiment of this application.

[0048] Figure 2 This is a schematic diagram of the structure of a preset model provided in an embodiment of this application;

[0049] Figure 3 This is a schematic diagram of a preset frame and a real license plate frame provided in an embodiment of this application;

[0050] Figure 4 A graph of the sigmoid function provided in an embodiment of this application;

[0051] Figure 5 A schematic diagram of the misaligned prediction box output when only the CIOU basic loss function is used as the loss function of the preset model, as provided in another embodiment of this application;

[0052] Figure 6 A schematic diagram of an idealized prediction frame provided for another embodiment of this application;

[0053] Figure 7A graph of the constraint loss function ContraintLoss provided in another embodiment of this application;

[0054] Figure 8 This is a flowchart illustrating a license plate correction method according to one embodiment of this application;

[0055] Figure 9 for Figure 8 A detailed flowchart of step 802;

[0056] Figure 10 This is a schematic diagram of the functional modules of a license plate correction device according to an embodiment of this application;

[0057] Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] In one exemplary embodiment, such as Figure 1 As shown, a training method for a license plate correction model is provided, including the following steps 101 to 102. Wherein:

[0061] Step 101: Obtain the training set, which includes preprocessed license plate images and corresponding real license plate frame annotations;

[0062] Specifically, the process involves acquiring license plate images, preprocessing the license plate images, and annotating them with real license plate bounding boxes to obtain a training set.

[0063] Specifically, preprocessing license plate images refers to normalizing the license plate images to a preset size and removing blurry, truncated, or poor-quality samples.

[0064] As one example, the preset size is set to 96x24.

[0065] Specifically, when capturing license plate images, you can directly set a preset frame size of 96x24, place the license plate within the preset frame, and then capture the image.

[0066] Furthermore, the collected data must meet the following requirements: 1. The training data should contain as many different license plate numbers as possible; 2. It should contain as many different types of license plates as possible; 3. It should contain as many license plates as possible in different scenarios, such as front lighting, backlighting, dusk, morning, rainy days, foggy days, and night; 4. The license plate should be located in as many different positions as possible within the prediction bounding box.

[0067] Step 102: Train the preset model using the training set to obtain the license plate correction model. The preset model includes a backbone network, a neck network, and a head network connected in sequence. The backbone network is used to acquire high semantic features of the images in the training set. The neck network is used to perform secondary processing and integration of the high semantic features. The head network is used to map the features output by the neck network to the output result. The backbone network includes four convolutional network layers connected in sequence. The neck network includes a fully connected layer, a random deactivation layer, and an activation function layer connected in sequence. The head network includes a linear output layer. The license plate correction model is used to process the license plate image to be corrected and output the predicted bounding box parameters of the license plate image to be corrected.

[0068] Specifically, the backbone network is the basic structure of the model, responsible for abstracting the original image layer by layer into high semantic features. By stacking components such as convolutional layers and pooling layers, it captures edges, textures, local structures, and global contextual information in the image.

[0069] Common backbone networks include general-purpose backbones such as ResNet, VGG (Visual Geometry Group Network), and EfficientNet; lightweight backbones such as the MobileNet series, ShuffleNet series, and EfficientNet-Lite series; and novel backbones such as Vision Transformer (based on self-attention mechanism) and ConvNeXt (Convolutional Neural Network Next).

[0070] Current license plate recognition systems are deployed on Android devices. Due to limited resources, if the model is to be deployed on a mobile device, the model parameters need to be limited to 3M. Therefore, the above-mentioned backbone network is not suitable for this task and needs to be implemented using the backbone network designed in the embodiments of this application.

[0071] The methods described in steps 101 to 102 are mainly applied to simple scenarios. While accurately correcting the position of license plates, they can be deployed to mobile devices to adapt to the hardware resources of mobile devices.

[0072] In another exemplary embodiment of this application, in order to improve the accuracy of the predicted bounding box output by the license plate correction model, such as... Figure 2 The following describes the preset model:

[0073] Specifically, such as Figure 2 The preset model structure consists of three parts: the backbone network, the nect network, and the head network.

[0074] The backbone network is the basic structure of the model, responsible for abstracting the original image into high semantic features layer by layer. By stacking components such as convolutional layers and pooling layers, it captures edges, textures, local structures, and global contextual information in the image.

[0075] Due to the simplification of the application scenario, the backbone network design adopts four layers, including a first convolutional network layer, a second convolutional network layer, a third convolutional network layer, and a fourth convolutional network layer connected in sequence. The first, second, and third convolutional network layers each include a convolutional layer, an activation layer, and a max pooling layer connected in sequence, respectively. The fourth convolutional network layer includes a convolutional layer and an activation layer connected in sequence. The specific parameters are shown in Table 1.

[0076] Specifically, since the input image size is fixed at 96x24 and the number of channels for color images is 3, the input data format for the network is channel number x height x width, abbreviated as CHW. Therefore, the data of the image is 3x24x96. The parameters are shown in Table 1.

[0077]

[0078] The neck network, located between the backbone and the head network, is responsible for secondary processing and integration of the raw features extracted by the backbone network. In this preset model, the neck network has only one fully connected layer, used for feature integration.

[0079]

[0080] The head network receives features from the neck network and maps them to specific detection results. In this task, only position correction is performed, so the head output size is 1x4. The definitions of the four output predictions are determined by the loss function.

[0081]

[0082] The specific output consists of four prediction results (t_cx, t_cy, t_w, t_y), where t_cx and t_cy are the offsets of the predicted bounding box's center point relative to the center point of the preset anchor box, and t_w and t_y are the width and height scaling of the predicted bounding box relative to the preset anchor box. From (t_cx, t_cy, t_w, t_y), we can calculate (pred_cx, pred_cy, pred_w, pred_h), (pred_cx, pred_cy, pred_w, pred_h), and further calculate (pred_xmin, pred_ymin, pred_xmax, pred_ymax).

[0083] In another exemplary embodiment of this application, during the training of the preset model, a function based on CIOU loss and constraint loss is used as the loss function of the preset model. Regarding the design of the loss function in this embodiment, it is necessary to start from the optimization problem of the model and design a loss function capable of solving the optimization problem. The following will combine... Figures 3-4 The loss function provided in the embodiments of this application will be explained. The parameters used and their physical meanings are shown in Table 4.

[0084]

[0085] First, the optimization problem is described, such as... Figure 3 Based on the design of the preset bounding box 301, a coordinate system is established, setting the coordinates of the upper left and lower right corners of the preset bounding box 301 to A(0, 0) and B(96, 24), respectively. The coordinates of the upper left and lower right corners of the actual license plate bounding box 302 are A1(gt_xmin, gt_ymin) and B1(gt_xmax, gt_ymax), respectively. Since the preset bounding box 301 contains a large amount of background content, directly using it for license plate recognition will reduce the recognition rate. To improve the recognition rate, feature learning is performed on the content of the preset bounding box 301 to locate the license plate position within the preset bounding box 301, i.e., the predicted bounding box, hereinafter referred to as pred_box(pred_xmin, pred_ymin, pred_max, pred_ymax). The actual license plate bounding box 302 is abbreviated as gt_box(gt_xmin, gt_ymin, gt_max, gt_ymax). The optimization problem is for the pred_box to infinitely approach the gt_box, which can be expressed mathematically as follows:

[0086]

[0087] In another exemplary embodiment of this application, to enable the model to converge stably and quickly, the idea of ​​anchors in the YOLO network is borrowed. The idea of ​​anchors is to pre-define a base box (i.e., the pre-defined anchor box in this embodiment) within a preset bounding box, and then predict the license plate position on this base box. The core of the anchor idea is to introduce prior knowledge.

[0088] Specifically, the preset anchor point box is set in the output layer. After normalizing the license plate positions of all training sets to 0-1, the center point of the preset anchor point box is set to (0, 0). The average width and height of the license plate are used as the width and height of the preset anchor point box.

[0089] As a specific implementation, the prior knowledge based on the training data is: anchor_cx=0, anchor_cy=0, anchor_w=0.6, anchor_h=0.4. The above-mentioned preset anchor boxes are described by their center point, width, and height.

[0090] Specifically, in the linear output layer, the output is 1x1x4, where 1x1 represents the prediction of a bounding box. The four predicted values ​​are t_cx, t_cy, t_w, and t_y. Based on the anchor point concept, the position coordinates of the predicted bounding box are calculated using the following formula:

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] The curve of the sigmoid function is shown below. Figure 4 It can be seen that the sigmoid curve makes the predicted value between 0 and 1. Its expression is:

[0096]

[0097] Based on the center point of the prediction box, the positions of the top left and top right corners of the prediction box are:

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] Specifically, based on the design of the aforementioned preset model, in the process of training the preset model using the training set, this embodiment employs a function based on CIOU loss and constraint loss as the loss function of the preset model. The loss function of the preset model is expressed by the following formula:

[0103] Loss=CIOULoss+ ContraintLoss

[0104] Here, CIOULoss is the CIOU base loss function, designed to make the predicted bounding box approximate the actual license plate bounding box, while ContraintLoss is the constraint loss function, designed to satisfy the constraint conditions when the predicted bounding box approximates the actual license plate bounding box. These are the weighting coefficients of the constraint loss function.

[0105] Specifically, the CIOU basic loss function is described by the following formula:

[0106]

[0107] Where Iou represents the intersection-union ratio (IoU) between the predicted bounding box and the actual license plate bounding box, and its expression is:

[0108]

[0109] in, represents the Euclidean distance between the center point of the predicted bounding box and the center point of the actual license plate frame; c represents the diagonal length of the minimum closed region covering the predicted bounding box and the actual license plate frame; This represents the weighting coefficient used to balance the effects of center point distance and aspect ratio;

[0110] Where v represents the aspect ratio, the similarity metric is calculated as follows:

[0111]

[0112] The purpose of the CIOU base loss function is to make the predicted bounding box 303pred_box infinitely close to the actual license plate bounding box gt_box. However, with only the CIOU base loss function, misalignment often occurs between the actual license plate bounding box 302 and the predicted bounding box 303, such as... Figure 5As shown, the first license plate number is easily missed, leading to subsequent license plate recognition errors. The CIOU basic loss function focuses more on the IOU value between the predicted bounding box 303 and the actual license plate bounding box 302, without restricting how the predicted bounding box 303 approximates the actual license plate bounding box 302. Therefore, the learned optimal solution does not conform to the actual task. In practical applications, to reduce the error rate of license plate recognition, the predicted license plate position must contain the actual license plate. Therefore, what we want is for the predicted bounding box 303 to be infinitely close to the actual license plate bounding box 302 in a way that surrounds it, such as... Figure 6 As shown.

[0113] Therefore, in order to make the obtained prediction frame 303 infinitely close to the real license plate frame, the constraints set in this application embodiment are as follows:

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] The above constraints can be written in the following style:

[0119] ;

[0120] ;

[0121] ;

[0122] ;

[0123] The above constraints are the constraint vectors. As input to the constraint loss function ContraintLoss, the expression for the obtained constraint loss function ContraintLoss is:

[0124]

[0125] in, This represents the i-th constraint quantity that satisfies the corresponding constraint condition. This represents the penalty coefficient of the sigmoid function. For example... Figure 7The graph above shows the constraint loss function, ContraintLoss. Experiments show that when the constraint data in the constraint vector Diff is less than 0, the constraint is satisfied, and according to the curve, the loss function will produce a loss value close to 0. Conversely, when the constraint value is greater than 0, the loss function will produce a large loss value, which indicates that the constraint condition has been broken, thus resulting in a large loss value.

[0126] As one implementation method, Set it to 50.

[0127] Furthermore, the CIOU base loss function and the constraint loss function play different roles at different stages in the entire iterative optimization process. In the early stages of iteration, the goal is to find a local optimum where the predicted bounding box quickly approaches the actual license plate frame. In this early stage, the CIOU base loss function, CIOULoss, plays a dominant role. In the later stages of iteration, the predicted bounding box has already fluctuated around the actual license plate frame. At this point, local adjustments are desired. When the predicted bounding box iterates near the actual license plate frame, the goal is for it to infinitely approach the actual license plate frame in a surrounding manner. At this stage, the constraint loss function gradually comes into play, working together with CIOULoss to bring the model closer to the desired optimal solution. Therefore, the setting of the weight coefficients of the constraint function determines at which stage the constraint loss function takes effect.

[0128] Table 5 shows the magnitudes of the CIOULoss loss and constraint function loss values ​​on the training set in this embodiment of the application. As can be seen from Table 1, during iterations 132-143, CIOULoss fluctuated around 0.05. Although the constraint function loss value was generally smaller, an order of magnitude smaller than CIOULoss, when calculating the standard deviation by normalizing both loss values ​​to the range of 0-1, the standard deviation of CIOULoss was 0.2627, while the standard deviation of the constraint function loss value was 0.2712. This indicates that the constraint function loss value fluctuated significantly. At this point, the predicted bounding box oscillated around the actual license plate frame while continuously breaking the constraints. Therefore, in the case of local adjustments, it is desirable to increase the influence of the constraint function on model optimization. Based on the multiple experimental data in Table 5, the weight coefficient of the constraint function was set to 5.

[0129] Furthermore, during the model training process, a preset model is first loaded with a 96x24 input. Then, the loss function and training set data provided in this embodiment are loaded. The RMSprop optimizer is used with a momentum parameter of 0.9, a weight decay of 0.000001, an initial learning rate of 0.0001, and 300 epochs. The learning rate is 0.0001 for the first 200 epochs and 0.0001 for the remaining 200 epochs. The bath size is 64. Data augmentation is performed only with normalization to a mean of 127 and a bias of 128.

[0130]

[0131] This application discloses a training method for a license plate correction model, which has the following technical effects:

[0132] The pre-designed model has a simple structure, few parameters, and low memory usage, which significantly reduces the consumption of computing resources. It can be efficiently deployed on mobile devices with limited computing power, meets real-time requirements, and has good engineering applicability.

[0133] Based on the anchor point concept, anchor boxes are pre-defined. The model provides prior knowledge, making the initial position and shape of the predicted box closer to the real target. Combined with an optimized loss function, the accuracy of bounding box regression can be further improved.

[0134] The loss function of the preset model is adopted based on CIOU loss and constraint loss. It also takes into account the IOU value between the predicted box and the real license plate box, and satisfies the constraint condition when the predicted box is close to the real license plate box, thereby reducing the error rate of the predicted box.

[0135] This paper proposes a loss function that transforms the license plate localization problem into an optimization problem. The optimization objective is to measure the geometric similarity and positional deviation between the predicted license plate bounding box and the actual license plate bounding box. Reasonable constraints are introduced to effectively suppress false positives and false negatives. This loss function considers the coordinates, size, and confidence level of the bounding box, driving the model's output prediction results to approximate the true label infinitely, significantly improving the accuracy and robustness of localization.

[0136] Experiments show that this solution exhibits excellent positioning performance in complex road environments, under varying lighting conditions, and in multi-angle shooting scenarios, demonstrating high positioning accuracy and strong generalization ability. In summary, this technical solution organically combines a lightweight model with high-precision positioning, providing an efficient and reliable solution for applications such as intelligent transportation and mobile inspection.

[0137] Based on the same inventive concept, this application also provides a license plate correction method. The implementation scheme of this license plate correction method is similar to the implementation scheme described in the training method of the aforementioned license plate correction model, and will not be repeated here. Figure 8 As shown, the license plate correction method includes:

[0138] Step 801: Obtain the image of the license plate to be corrected using a preset frame;

[0139] Specifically, a preset frame is set, and when acquiring the license plate of the vehicle to be tested, the user places the license plate within the preset frame and takes a picture.

[0140] Specifically, when taking a photo, the user places the license plate image into a preset frame.

[0141] Specifically, the obtained license plate image to be corrected is a preset size image containing the license plate;

[0142] Step 802: Process the license plate image to be corrected using the license plate correction model to obtain the prediction box parameters of the license plate image to be corrected; wherein, the license plate correction model is a model obtained according to the training method of the above-mentioned license plate correction model.

[0143] Specifically, the prediction box parameters include the center point offset of the prediction box based on the preset anchor point, and the width and height scaling of the prediction box based on the preset anchor point.

[0144] Step 803: Obtain the prediction box of the license plate image to be corrected based on the prediction box parameters;

[0145] Specifically, step 803 can be derived from steps 901 to 902, such as... Figure 9 Specifically, it includes:

[0146] Step 901: Obtain the center point, height, and width of the prediction box based on the center point offset of the prediction box relative to the center point of the preset anchor box, the width and height scaling of the prediction box relative to the preset anchor point, the center coordinates of the preset anchor point, and the width and height of the preset anchor point.

[0147] Specifically, it can be obtained through the following formula:

[0148] ;

[0149] ;

[0150] ;

[0151] ;

[0152] Step 902: Obtain the coordinates of the upper left and upper right corners of the prediction box based on the center point, height, and width of the prediction box. Output the prediction box using the coordinates of the upper left and upper right corners.

[0153] Specifically, it can be obtained through the following formula:

[0154] ;

[0155] ;

[0156] ;

[0157] ;

[0158] Based on the same inventive concept, this application also provides a license plate correction device, the concept of which is similar to the problem it solves, and will not be described in detail here. Figure 10 As shown, the license plate correction device includes:

[0159] The deployment module is used to receive the deployment parameters of the license plate correction model, which is a license plate correction model obtained according to the training method of the above-mentioned license plate correction model.

[0160] Specifically, during model training, the above-mentioned preset model structure is built in the PyTorch environment, and the corresponding loss function and training script are designed. The training set is input into the preset model for training, and the model and its parameter files are generated and saved.

[0161] Specifically, it can be deployed on Android using the MNN framework. MNN is a lightweight deep learning inference framework developed by Alibaba, designed specifically for mobile and embedded devices. It supports multiple mainstream deep learning model formats (such as TensorFlow, Caffe, ONNX, etc.) and achieves cross-platform model inference through highly optimized computing kernels. MNN's core goals are low barrier to entry, high performance, and lightweight design, enabling developers to easily deploy complex AI models to devices such as mobile phones and tablets.

[0162] Furthermore, during deployment on the Android platform, it is necessary to convert different weight formats. The pth format model and its parameter files are converted to ONNX format files and then to MNN format files. The MNN format model and its parameter files are then deployed on the Android platform under the MNN framework.

[0163] The pth file is the standard file format used in PyTorch to save and load model parameters.

[0164] ONNX files are an open deep learning model exchange format, short for Open Neural Network Exchange, designed to enable model interoperability between different deep learning frameworks.

[0165] MNN files are model file formats for Mobile Neural Networks, a lightweight deep learning inference engine developed by Alibaba. They are primarily used to efficiently run neural network models in resource-constrained environments such as mobile devices and the Internet of Things.

[0166] Based on the designed model and the data collected, after training the model, the model's parameter weights are saved in pth format. In order to deploy it under the MNN framework, the pth format model and its parameter files need to be converted to onnx format and then converted to mnn format.

[0167] The correction module is used to obtain the license plate image to be corrected through a preset bounding box, and to process the license plate image to be corrected through a license plate correction model to obtain the predicted bounding box of the license plate image to be corrected.

[0168] The output module is used to obtain the prediction bounding box of the license plate image to be corrected based on the prediction bounding box parameters.

[0169] Furthermore, the output module is specifically used for:

[0170] The center point, height, and width of the prediction box are obtained based on the center point offset of the prediction box relative to the center point of the preset anchor box, the width and height scaling of the prediction box relative to the preset anchor box, the center coordinates of the preset anchor box, and the width and height of the preset anchor box.

[0171] The coordinates of the top left and top right corners of the prediction box are obtained based on the center point, height, and width of the prediction box. The prediction box is then output using the coordinates of the top left and top right corners.

[0172] This application also provides an application scenario that utilizes the training method or license plate correction method of the aforementioned license plate correction model. Specifically: The license plate correction model is deployed on a mobile device, and an image of the license plate to be corrected is acquired through the mobile device. This image is then processed and a predicted bounding box for the license plate to be corrected is output.

[0173] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 11As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a training method for a license plate correction model or a license plate correction method.

[0174] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0175] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0176] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0177] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0179] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0180] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0181] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. In summary, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for training a license plate rectification model, characterized in that, The training method of the license plate correction model comprises: obtaining a training set, the training set comprising preprocessed license plate images and corresponding real license plate frame annotations; training a preset model according to the training set to obtain a license plate correction model; wherein the preset model comprises a backbone network, a neck network and a head network connected in sequence; the backbone network is used to obtain high semantic features of the images in the training set, the neck network is used to perform secondary processing and integration on the high semantic features, and the head network is used to map the features output by the neck network into an output result; the backbone network comprises four convolutional network layers connected in sequence, the neck network comprises fully connected layers, random inactivation layers and activation function layers connected in sequence, and the head network comprises a linear output layer; the license plate correction model is used to process a license plate image to be corrected to obtain predicted frame parameters of the license plate image to be corrected. In the process of training the preset model according to the training set, a function based on a CIOU loss and a constraint loss is used as a loss function of the preset model, and the loss function of the preset model is expressed by the following formula: Loss = CIoU Loss + ConstraintLoss ContraintLoss; Wherein, CIOULoss is a CIOU basic loss function, which is to make the predicted frame close to the real license plate frame, and ContraintLoss is a constraint loss function, which is to meet the constraint condition when the predicted frame is close to the real license plate frame, is a weight coefficient of the constraint loss function; The constraint loss function is described by the following formula: ； wherein, represents the ith constraint quantity satisfying the corresponding constraint condition, represents a penalty coefficient of a sigmoid function; wherein, the constraint quantity comprises , , and , the expression of the constraint quantity and the corresponding constraint condition are: ； ； ； ； wherein (pred_xmin, pred_ymin) is the left upper corner coordinate of the predicted frame, (pred_xmax, pred_ymax) is the right lower corner coordinate of the predicted frame, (gt_xmin, gt_ymin) is the left upper corner coordinate of the real license plate frame, and (gt_xmax, gt_ymax) is the right lower corner coordinate of the real license plate frame.

2. The method of claim 1, wherein the training of the license plate rectification model is characterized by, The linear output layer is used to output the predicted frame parameters, and a preset anchor frame is set in the linear output layer, and the correspondence between the preset anchor frame and the predicted frame is described by the following formula: ； ； ； ； wherein, is a center point coordinate of the preset anchor box in an x direction, is a center point coordinate of the preset anchor box in a y direction, is a width of the preset anchor box, is a height of the preset anchor box; sigmoid() is a sigmoid function, and exp() is an exponential function; is a center point coordinate of the prediction box in the x direction, is a center point coordinate of the prediction box in the y direction, is a width of the prediction box, is a height of the prediction box, is a width scaling amount of the prediction box based on the width of the preset anchor box, is a height scaling amount of the prediction box based on the height of the preset anchor box, and (t_cx, t_cy) is an offset amount of the center point of the prediction box based on the center point of the preset anchor box.

3. The method of claim 1, wherein the training of the license plate rectification model is characterized by, The CIOU basic loss function is described by the following formula: ； where Iou denotes the intersection over union between the predicted and real license plate bounding boxes; denotes the Euclidean distance between the centers of the predicted and real license plate bounding boxes; c denotes the length of the diagonal of the minimum enclosing region covering the predicted and real license plate bounding boxes; denotes the weight coefficient for balancing the influence of the center distance and the aspect ratio; v denotes the aspect ratio similarity measure.

4. A license plate correction method characterized by, The license plate correction method comprises: obtaining a license plate image to be corrected through a preset frame; processing the license plate image to be corrected through a license plate correction model to obtain predicted frame parameters of the license plate image to be corrected; wherein the license plate correction model is a license plate correction model obtained according to the training method of the license plate correction model in any one of claims 1-3; obtaining a predicted frame of the license plate image to be corrected according to the predicted frame parameters.

5. The license plate correction method of claim 4, wherein, The predicted frame parameters comprise a center point offset of the predicted frame based on a center point of a preset anchor frame and a width and height stretching amount of the predicted frame based on a width and height of a preset anchor; The obtaining of the predicted frame of the license plate image to be corrected according to the predicted frame parameters comprises: obtaining a center point, a height and a width of the predicted frame according to the center point offset of the predicted frame based on the center point of the preset anchor frame, the width and height stretching amount of the predicted frame based on the width and height of the preset anchor, a center coordinate of the preset anchor frame, a width and a height of the preset anchor frame; obtaining a left upper corner coordinate and a right upper corner coordinate of the predicted frame according to the center point of the predicted frame and the height and width of the predicted frame, and outputting the predicted frame according to the left upper corner coordinate and the right upper corner coordinate of the predicted frame.

6. A license plate straightening device, characterized by, The license plate correction device comprises: A deployment module configured to receive deployment parameters of a license plate rectification model, the license plate rectification model being obtained according to the training method of the license plate rectification model of any one of claims 1-3; A rectification module configured to obtain a to-be-rectified license plate image through a preset frame, and obtain predicted frame parameters of the to-be-rectified license plate image by processing the to-be-rectified license plate image through the license plate rectification model; An output module configured to obtain the predicted frame of the to-be-rectified license plate image according to the predicted frame parameters.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the training method of the license plate rectification model of any one of claims 1-3 or the license plate rectification method of any one of claims 4 or 5.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the training method of the license plate rectification model of any one of claims 1-3 or the license plate rectification method of any one of claims 4 or 5.

Images