A nameplate VIN code recognition method and device, a storage medium and equipment

Abstract

The application provides a nameplate VIN code recognition method and device, a storage medium and equipment. In the method, a character detection model is constructed, and the character detection model is reinforced by an auxiliary branch network. Meanwhile, a plurality of sample pictures with specified labels are used as training samples to construct a character recognition model. In this way, for a to-be-recognized automobile nameplate image, the position of the VIN code region is detected based on the character detection model, and the image is cropped and then input into the text recognition model, and then the target VIN code contained in the automobile nameplate image is recognized. In this way, the detection capability of the model for the VIN code is enhanced by using the auxiliary learning mode, so that the recognition accuracy of the nameplate VIN code is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method, apparatus, storage medium, and device for identifying nameplate VIN codes. Background Technology

[0002] A vehicle identification plate is like a vehicle's ID card, recording important information about each vehicle. The VIN (Vehicle Identification Number) code on the plate contains a wealth of vehicle information, such as vehicle type, vehicle structure type, and vehicle technical characteristics. Therefore, automating the extraction of VIN codes facilitates vehicle management.

[0003] The main method for extracting VIN codes from vehicle nameplates in related technologies relies on traditional algorithms, using conventional image processing transformations to separate the foreground and background, and then extracting the VIN code. However, this method has poor robustness and a high false detection rate. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, storage medium and device for identifying VIN codes on vehicle nameplates, aiming to solve the problem of low accuracy in related technologies for extracting VIN codes from vehicle nameplates.

[0005] Firstly, this application provides a method for identifying VIN codes on nameplates, including:

[0006] Obtain the image of the vehicle nameplate to be identified;

[0007] The vehicle nameplate image is processed based on a text detection model to obtain the VIN code location information; during the training process, the text detection model is jointly optimized based on the loss function of the text detection model and the loss function of the auxiliary branch network.

[0008] The vehicle nameplate image is cropped based on the VIN code location information, and then the cropped image is processed based on a text recognition model to identify the target VIN code contained in the vehicle nameplate image; the text recognition model is trained through multiple sample images with specified labels; the specified labels indicate the VIN code contained in the corresponding sample images.

[0009] In the above implementation process, a text detection model is constructed and enhanced through an auxiliary branch network. Simultaneously, a text recognition model is built using multiple sample images with specified labels as training samples. Thus, for a car license plate image to be recognized, the text detection model detects the location of the VIN code, and after cropping the image, it is input into the text recognition model to identify the target VIN code contained in the car license plate image. In this way, by utilizing auxiliary learning, the model's ability to detect VIN codes is enhanced, thereby improving the accuracy of VIN code recognition on license plates.

[0010] Furthermore, in some embodiments, the text detection model is trained based on the YOLOv5 algorithm; the VIN code location information includes VIN code coordinates and confidence level.

[0011] In the above implementation process, a text detection model is constructed based on the YOLOv5 network structure, so that the constructed text detection model can accurately detect the coordinates of the VIN code and the corresponding confidence level in the image.

[0012] Furthermore, in some embodiments, the auxiliary branch network is constructed on the backbone network of the text detection model by stacking units connected by convolutional layers, batch normalization layers, activation layers, and shortcut layers to form a neck network, constructing a first output terminal with a fully connected layer, and then constructing a second output terminal parallel to the first output terminal with a convolutional layer; wherein, the first output terminal is used to determine the car manufacturer information contained in the input image; the second output terminal is used to determine the number of characters contained in the position where characters exist in the input image.

[0013] In the above implementation process, an auxiliary branch network is used to enhance the training of the text detection model. The characteristics of the VIN code, such as the car manufacturer and the number of characters, are transmitted to the text detection model, thereby improving the detection accuracy of the text detection model for the VIN code.

[0014] Furthermore, in some embodiments, the text detection model is trained in the following manner:

[0015] Input the target sample image with classification label and ground truth bounding box coordinate label into the initial text detection model to obtain the predicted classification result and predicted coordinate result;

[0016] A classification loss function is constructed based on the classification identifier and the predicted classification result, and a coordinate loss function is constructed based on the ground truth bounding box coordinate identifier and the predicted coordinate result;

[0017] After training the initial text detection model using the classification loss function and the coordinate loss function to obtain the basic detection model, the backbone network of the text detection model is jointly tuned using the loss function of the auxiliary branch network, the classification loss function, and the coordinate loss function.

[0018] In the above implementation process, a specific method for strengthening the training of the text detection model based on an auxiliary branch network is provided.

[0019] Furthermore, in some embodiments, the text recognition model includes a CNN branch network, an LSTM branch network, and an MLP branch network; the CNN branch network is used to extract text features from the input image; the LSTM branch network is used to recognize text information based on the text features; the latent variable output by the last subunit of the LSTM branch network is input to the MLP branch network; the MLP branch network is used to determine whether the text information constitutes a VIN code based on the latent variable.

[0020] In the above implementation process, the ability of the character recognition model to recognize VIN codes is enhanced through CNN construction, LSTM construction and adaptive improvements.

[0021] Furthermore, in some embodiments, the loss function of the character recognition model is obtained by adding CTC Loss and BCELoss.

[0022] In the above implementation process, during the training of the character recognition model, CTC Loss is used as the loss function, and BCE Loss is introduced to calculate the prediction loss of VIN code, so that the trained character recognition model can accurately identify the VIN code contained in the input image.

[0023] Furthermore, in some embodiments, the method further includes:

[0024] When the confidence level is higher than the first threshold, the target VIN code is output;

[0025] When the confidence level is lower than the second threshold, the target VIN code is removed; the second threshold is less than the first threshold.

[0026] In the above implementation process, the recognition result is directly output or removed according to the confidence level of the detection, so as to ensure the accuracy of the final output VIN code recognition result and improve the user experience.

[0027] Secondly, this application provides a nameplate VIN code recognition device, comprising:

[0028] The acquisition module is used to acquire the image of the vehicle nameplate to be identified;

[0029] The detection module is used to process the vehicle nameplate image based on the text detection model to obtain the VIN code location information; during the training process, the text detection model is jointly optimized based on the loss function of the text detection model and the loss function of the auxiliary branch network.

[0030] The recognition module is used to crop the vehicle nameplate image according to the VIN code location information, and then process the cropped image based on a text recognition model to identify the target VIN code contained in the vehicle nameplate image; the text recognition model is trained through multiple sample images with specified labels; the specified labels indicate the VIN code contained in the corresponding sample images.

[0031] Thirdly, this application provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method described in any of the first aspects.

[0032] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in any of the first aspects.

[0033] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the method described in any of the first aspects.

[0034] Other features and advantages disclosed in this application will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the above-described technology disclosed in this application.

[0035] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating a nameplate VIN code recognition method provided in this application embodiment;

[0038] Figure 2 A schematic diagram illustrating the workflow of a knowledge-assisted branch-based nameplate VIN code recognition scheme provided in this application embodiment;

[0039] Figure 3 A block diagram of a nameplate VIN code recognition device provided in an embodiment of this application;

[0040] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0042] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0043] As described in the background section, existing methods for extracting VIN codes from vehicle nameplates suffer from low accuracy. Therefore, this application provides a nameplate VIN code recognition scheme to address the aforementioned problem.

[0044] The embodiments of this application will be described below:

[0045] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a method for identifying VIN codes on nameplates, provided in an embodiment of this application. The method can be applied to a terminal or a server. The terminal can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers; the server can be a single server or a distributed server cluster composed of multiple servers. The terminal or server provides an image processing environment, which includes software and hardware components. The software component mainly includes an operating system, such as Windows or Linux, while the hardware component mainly includes computing resources and storage resources, such as a CPU (Central Processing Unit), memory, and hard disk. It should be noted that the terminal / server can also be implemented as multiple software programs or software modules, or as a single software program or software module; this application does not impose any limitations on this.

[0046] The method includes:

[0047] Step 101: Obtain the image of the vehicle nameplate to be identified;

[0048] The vehicle identification plate image mentioned in this step can be any image containing a VIN code, such as an image captured by a camera, mobile phone, or other acquisition device, or a video frame extracted from a video captured by the acquisition device. This embodiment can be applied to various scenarios. For example, in a fintech scenario, auto finance institutions can capture images of vehicle identification plates belonging to customers who need to conduct leasing, loan, or other business transactions, and then automatically identify the VIN code on the identification plate to obtain vehicle information and achieve intelligent management.

[0049] In step 102, the vehicle nameplate image is processed based on the text detection model to obtain the VIN code location information; during the training process, the text detection model is jointly optimized based on the loss function of the text detection model and the loss function of the auxiliary branch network.

[0050] In practical applications, vehicle nameplates may contain information other than the VIN code. This makes VIN code recognition on captured vehicle nameplate images susceptible to interference from other information. Therefore, to improve the accuracy of VIN code recognition, this embodiment uses an auxiliary branch network to enhance the text detection model. The trained text detection model is then used to process the vehicle nameplate image to determine the VIN code region, thus facilitating accurate recognition by the subsequent text recognition model.

[0051] Specifically, in some embodiments, the text detection model mentioned in this step can be trained based on the YOLOv5 algorithm; the VIN code location information includes the VIN code coordinates and confidence score. The YOLOv5 algorithm is a deep learning algorithm for object detection and is a member of the YOLO family. YOLOv5's architecture consists of four parts: input, backbone, neck, and head. At the input, YOLOv5 performs adaptive image padding and Mosaic augmentation on the image data. It also integrates adaptive anchor box calculation, allowing the model to obtain initial anchor box sizes by clustering the label boxes of different datasets before training. The backbone extracts features from the images, primarily using the C3 and SPPF modules. The C3 module contains three standard convolutional layers and multiple bottleneck modules, reducing computational cost. The SPPF module, based on a principle similar to spatial pyramid pooling, performs multi-scale feature extraction on the same feature map, improving model accuracy. The neck typically consists of a feature pyramid and path aggregation structure, fusing feature information from feature maps of different sizes. The head, as the model's detection part, predicts objects of different sizes from the extracted multi-scale feature maps. The anchor box mechanism at the output extracts prior box scales through clustering and constrains the position of the predicted bounding boxes. A text detection model is constructed based on the YOLOv5 network structure, enabling the model to accurately detect the coordinates and corresponding confidence levels of VIN codes in images.

[0052] During training, the text detection model undergoes joint optimization based on its own loss function and the loss function of the auxiliary branch network. In other words, the auxiliary branch network is used to enhance the text detection model's training, transmitting the characteristics of VIN codes to the model and thus improving its detection accuracy. In some embodiments, the auxiliary branch network can be constructed by stacking units connected by convolutional layers, batch normalization (BN) layers, activation layers, and shortcut layers on top of the backbone network of the text detection model to form a neck network. A fully connected layer forms the first output, and a convolutional layer forms the second output parallel to the first output. The first output is used to determine the car manufacturer information contained in the input image; the second output is used to determine the number of characters contained at the positions where characters exist in the input image. In other words, the auxiliary branch network can also be trained based on the YOLOv5 algorithm. It can build upon the backbone of the text detection model by constructing units connected by convolutional layers, batch normalization (BN) layers, activation layers, and shortcut layers, stacked in a certain number to form a neck layer. This auxiliary branch network has two heads: one head, constructed with fully connected layers, is used to determine the car manufacturer written on the nameplate; the other head, constructed with convolutional layers, is used to determine the number of characters corresponding to positions containing alphanumeric characters. This allows the auxiliary branch network to focus on more VIN code features, which can then be passed to the text detection model during its reinforcement training. Specifically, when training the auxiliary branch network, multiple nameplate images labeled with car manufacturer and character count can be used as training samples. The output of the first output is cross-entropy calculated with the ground truth to construct the first loss function. The output of the second output is compared with the ground truth to calculate the smooth loss, resulting in the second loss function, thus obtaining the loss function of the auxiliary branch network.

[0053] Furthermore, in some embodiments, the text detection model mentioned in this step can be trained in the following manner: Inputting a target sample image with classification labels and ground truth bounding box coordinate labels into an initial text detection model to obtain predicted classification results and predicted coordinate results; constructing a classification loss function based on the classification labels and the predicted classification results, and constructing a coordinate loss function based on the ground truth bounding box coordinate labels and the predicted coordinate results; after training the initial text detection model using the classification loss function and the coordinate loss function to obtain a basic detection model, jointly optimizing the backbone network of the text detection model using the loss function of the auxiliary branch network, the classification loss function, and the coordinate loss function. In other words, multiple target sample images with classification labels and ground truth bounding box coordinate labels are used as training samples for the text detection model. Here, the classification labels represent the category to which the target sample image belongs, and the ground truth bounding box coordinate labels represent the coordinates of the VIN code in the region where the target sample image is located. The model is trained using these training samples to obtain classification loss functions and coordinate loss functions. The text detection model is gradually trained until convergence, completing the construction of the basic detection model. Then, based on the classification loss function and coordinate loss function, combined with the previously mentioned first and second loss functions, a joint loss function is obtained. This joint loss function is used for optimization, making the backbone network more efficient. By minimizing this joint loss function, the model reaches convergence, thus completing the reinforcement training of the text detection model. The classification loss function can be CE Loss (Cross-Entropy Loss), and the coordinate loss function can be IoU Loss (Intersection over Union Loss). The specific construction process of the loss function can be found in the descriptions in related technologies, and this application will not elaborate on it further.

[0054] In step 103, the vehicle nameplate image is cropped according to the VIN code location information, and then the cropped image is processed based on the text recognition model to identify the target VIN code contained in the vehicle nameplate image; the text recognition model is trained through multiple sample images with specified labels; the specified labels indicate the VIN code contained in the corresponding sample images.

[0055] This step involves: for the vehicle nameplate image to be identified, obtaining the VIN code location information through a text detection model, cropping the image accordingly, and then inputting it into a text recognition model for VIN code recognition, thereby obtaining the target VIN code contained in the vehicle nameplate image. In this process, because VIN code knowledge is effectively utilized, the model's recognition ability is improved, and post-processing steps are reduced, resulting in high accuracy in identifying the nameplate VIN code.

[0056] In some embodiments, the text recognition model mentioned in this step may include a CNN branch network, an LSTM branch network, and an MLP branch network; the CNN branch network is used to extract text features from the input image; the LSTM branch network is used to recognize text information based on the text features; the latent variable output by the last subunit of the LSTM branch network is input to the MLP branch network; the MLP branch network is used to determine whether the text information constitutes a VIN code based on the latent variable. In other words, the training phase of this character recognition model can include the construction of a CNN (Convolutional Neural Network), the construction of an LSTM (Long Short-Term Memory) network, and adaptive improvements. In some scenarios, the CNN branch network can consist of convolutional layers, batch normalization (BN) layers, activation layers, and shortcut layers to extract text features. Sub-modules are constructed through input gates, output gates, and forget gates, and these sub-modules are arranged in a self-loop manner to form an LSTM network. The CNN and LSTM are cascaded to obtain the character recognition network. Based on this, the latent variable output from the last sub-unit of the LSTM is input into an MLP (Multilayer Perceptron) network to determine whether the recognized text information constitutes a VIN code. This enhances the character recognition model's ability to recognize VIN codes.

[0057] Furthermore, the loss function of this character recognition model can be obtained by adding CTC Loss and BCE Loss. During the training process of the character recognition model, CTC Loss (Connectionist Temporal Classification Loss) is used as the loss function. CTC Loss is solved using a dynamic programming algorithm to maximize the likelihood probability of the predicted value composed of the true labels from multiple joint probabilities. Simultaneously, BCELoss (Binary Cross Entropy Loss) is introduced to calculate the prediction loss of the VIN code. The two loss functions are added together to obtain the total loss. The model is trained until convergence, enabling the character recognition model to accurately identify the VIN code contained in the input image. Of course, in other embodiments, this character recognition model can also be trained using other types of loss functions.

[0058] Furthermore, in some embodiments, the above method may further include: outputting the target VIN code when the confidence level is higher than a first threshold; removing the target VIN code when the confidence level is lower than a second threshold; the second threshold is less than the first threshold. That is, after the character recognition model identifies the target VIN code contained in the vehicle nameplate image, if the character detection model detects a confidence level higher than the first threshold, it directly outputs the recognition result of the character recognition model; if the detection confidence level is lower than the second threshold, it indicates that the recognition result may differ from the actual situation, and the recognition result is removed. This ensures the accuracy of the final output VIN code recognition result and improves the user experience. Optionally, the first threshold can be 0.8, and the second threshold can be 0.2. When the detection confidence level is between 0.2 and 0.8, the vehicle nameplate image can be detected again. If the confidence level in the new detection result is higher than 0.5, the identified target VIN code is retained; otherwise, it is removed. Of course, the first and second thresholds can also be set differently according to the needs of different scenarios, and this application does not limit this.

[0059] In this embodiment, a text detection model is constructed and enhanced through an auxiliary branch network. Simultaneously, a text recognition model is built using multiple sample images with specified labels as training samples. Thus, for a vehicle nameplate image to be recognized, the text detection model detects the location of the VIN code. The image is then cropped and input into the text recognition model to identify the target VIN code contained in the vehicle nameplate image. In this way, by utilizing auxiliary learning, the model's ability to detect VIN codes is enhanced, thereby improving the accuracy of VIN code recognition on nameplates.

[0060] To provide a more detailed explanation of the solution in this application, a specific embodiment is described below:

[0061] This embodiment relates to a vehicle identification number (VIN) code recognition scenario. In the fintech field, with the increasing number of car purchases, auto finance institutions aim to promote the intelligentization of vehicle management processes, and VIN code recognition is a crucial step in vehicle management. Current VIN code recognition methods employ a single deep learning approach, which detects all text information and is prone to interference, resulting in low recognition accuracy. Therefore, this embodiment provides a knowledge-assisted branch-based VIN code recognition scheme to address these issues.

[0062] The workflow of this embodiment is as follows: Figure 2As shown in the diagram, black arrows represent the training process, and white arrows represent the testing process. This scheme involves the construction of a text detection model 21, a knowledge-assisted branch 22, and a text recognition model 23. The text recognition model 23 includes an MLP network (denoted as 231 in the diagram). Furthermore, this text recognition model also includes a CNN branch network and an LSTM branch network (not shown in the diagram). The workflow includes:

[0063] S201. Construct a text detection model based on the network structure of YOLOv5;

[0064] Specifically, in the construction of this text detection model, the components consist of convolutional layers, activation layers, and pooling layers. The sub-unit structure adopts the traditional bottleneck structure and the shortcut connection structure, which are stacked layer by layer to form the backbone network. After the features are extracted in the backbone, they enter the bottleneck layer, which consists of a concat layer, upsampling, and a convolutional layer. Features of different resolutions are fused into the prediction layer. The prediction layer consists of a single convolutional layer without BN. Assuming the input image size is 640x640 pixels, after feature extraction, it outputs features of large (80x80), medium (40x40), and small (20x20) resolutions with a depth of c. Here, c consists of a 4-dimensional coordinate dimension (x, y, width, height) and a 1-dimensional confidence (used to indicate whether a target exists).

[0065] The basic loss functions of this text detection model include a classification loss function and a coordinate loss function. The classification loss function uses CE Loss, and the coordinate loss function uses IoU Loss.

[0066] Set batch_zise to 64, LR to 1e-3, and use cosine annealing to change the learning rate. Under the Adam optimizer, gradually train the text detection model until convergence, thus completing the construction of the basic detection model.

[0067] S202. Introduce a knowledge-assisted branch and train the model together with the loss function output by the knowledge-assisted branch and the loss function in S201 to improve the model's feature extraction ability.

[0068] Specifically, the construction process of this knowledge-assisted branch can be based on the backbone of YOLOv5 in S201, building units connected by convolutional layers, BN layers, activation layers and shortcut layers, stacking them in a certain number to form a neck layer, and then building head1 with fully connected layers to determine the car manufacturer written on the nameplate, and building head2 in parallel with head1 with convolutional layers to determine the number of characters corresponding to the positions where English / numeric characters exist;

[0069] The output of head1 is cross-entropy with the true value to obtain the loss function L1, and the output of head2 is cross-entropy with the true value to obtain the loss function L2.

[0070] During training, the Adam optimizer was used to optimize the parameters of YOLOv5 and the two parallel heads. First, based on S201, the YOLOv5 parameters were fixed, and the head1 and head2 parameters were trained separately. Then, the YOLOv5 loss function was jointly tuned with the L1 and L2 loss functions at a ratio of 8:1:1. Finally, to make the backbone network more effective, the training difficulty was increased by changing the ratio to 2:1:1 until convergence, thus completing the reinforcement training of the text detection model.

[0071] S203. Implement the text recognition model training, which mainly includes CNN construction, LSTM construction and adaptive improvement, and completing the training based on the newly constructed loss function;

[0072] Specifically, the training process of this character recognition model may include:

[0073] S2031, similar to the backbone structure of a text detection model, is a CNN constructed from convolutional layers, batch normalization layers, activation layers, and shortcut layers to extract text features;

[0074] S2032. Sub-modules are constructed through input gates, output gates, and forget gates, and the sub-modules are arranged in a self-loop manner to form an LSTM network. The CNN and LSTM are cascaded to obtain a text recognition network. At the same time, unlike the traditional text recognition network, the latent variable output by the last sub-unit of the LSTM is input into an MLP network to determine whether the recognized text constitutes a VIN code.

[0075] S2033. Set the learning rate to 1e-5 and the image resolution to 32x640. Based on the ImageNet pre-trained model, use CTC Loss as the loss function and gradient descent as the iterative algorithm to train the character recognition model. CTC Loss is calculated by using dynamic programming to maximize the likelihood probability of the predicted value composed of the true labels. BCE Loss is introduced to calculate the prediction loss of the VIN code. The losses of CTC Loss and BCE Loss are added in a 1:1 ratio to obtain the total loss. Train the model until convergence to obtain the character recognition model.

[0076] S204. The trained models are cascaded according to certain rules and then used to recognize the VIN code on the nameplate.

[0077] Specifically, this step may include:

[0078] S2041. After removing the knowledge auxiliary branch from the model trained in S202, it is cascaded with the text recognition model obtained in S203 to form a complete nameplate VIN recognition module.

[0079] S2042. Input the nameplate image to be identified into the nameplate VIN recognition module in S2041. During the text detection stage, the relevant VIN code coordinates and confidence scores will be obtained. After cropping the image, input it into the text recognition model for VIN code recognition. When the detection confidence score is higher than 0.8, the recognition result is directly output; when the detection confidence score is lower than 0.2, the recognition result is removed from the VIN code candidate; when the detection confidence score is between 0.2 and 0.8, observe the VIN code value determined by text recognition. If it is higher than 0.5, the recognized VIN code value is retained; otherwise, it is removed. This completes the VIN code detection and recognition.

[0080] The model trained by this embodiment has high recognition accuracy and can efficiently identify and recall difficult-to-identify cases on nameplates. Secondly, this solution effectively utilizes VIN code knowledge, enhancing VIN code detection while reducing post-processing pressure, making nameplate VIN code recognition more efficient, accelerating vehicle management processes, and thus promoting the efficiency of storing vehicle information from the physical world to the digital world.

[0081] Corresponding to the embodiments of the aforementioned methods, this application also provides embodiments of a nameplate VIN code recognition device and a terminal thereof:

[0082] like Figure 3 As shown, Figure 3 This is a block diagram of a nameplate VIN code recognition device provided in an embodiment of this application, comprising:

[0083] The acquisition module 31 is used to acquire the image of the vehicle nameplate to be identified;

[0084] The detection module 32 is used to process the vehicle nameplate image based on the text detection model to obtain the VIN code location information; during the training process, the text detection model is jointly optimized based on the loss function of the text detection model and the loss function of the auxiliary branch network.

[0085] The recognition module 33 is used to crop the vehicle nameplate image according to the VIN code location information, and then process the cropped image based on the text recognition model to identify the target VIN code contained in the vehicle nameplate image; the text recognition model is trained by multiple sample images with specified labels; the specified labels indicate the VIN code contained in the corresponding sample images.

[0086] In some embodiments, the above-mentioned text detection model is trained based on the YOLOv5 algorithm; the VIN code location information includes VIN code coordinates and confidence level.

[0087] In some embodiments, the auxiliary branch network is constructed on the backbone network of the text detection model by building units connected by convolutional layers, batch normalization layers, activation layers and shortcut layers, stacking them to form a neck network, constructing a first output terminal with a fully connected layer, and then constructing a second output terminal parallel to the first output terminal with a convolutional layer; wherein, the first output terminal is used to determine the car manufacturer information contained in the input image; the second output terminal is used to determine the number of characters contained in the position where characters exist in the input image.

[0088] In some embodiments, the above-described apparatus further includes a training module, the training module comprising:

[0089] The input submodule is used to input the target sample image with classification labels and ground truth bounding box coordinate labels into the initial text detection model to obtain the predicted classification results and predicted coordinate results.

[0090] A submodule is constructed to construct a classification loss function based on the classification identifier and the predicted classification result, and to construct a coordinate loss function based on the ground truth bounding box coordinate identifier and the predicted coordinate result;

[0091] The tuning submodule is used to jointly tune the backbone network of the text detection model after training the initial text detection model using the classification loss function and the coordinate loss function to obtain the basic detection model.

[0092] In some embodiments, the above-mentioned text recognition model includes a CNN branch network, an LSTM branch network, and an MLP branch network; the CNN branch network is used to extract text features from the input image; the LSTM branch network is used to recognize text information based on the text features; the latent variable output by the last subunit of the LSTM branch network is input to the MLP branch network; the MLP branch network is used to determine whether the text information constitutes a VIN code based on the latent variable.

[0093] In some embodiments, the loss function of the above character recognition model is obtained by adding CTC Loss and BCE Loss.

[0094] In some embodiments, the above-described apparatus further includes:

[0095] The output module is used to output the target VIN code when the confidence level is higher than the first threshold.

[0096] The removal module is used to remove the target VIN code when the confidence level is lower than a second threshold; the second threshold is less than the first threshold.

[0097] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0098] This application also provides an electronic device, please refer to [link to application]. Figure 4 , Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device may include a processor 410, a communication interface 420, a memory 430, and at least one communication bus 440. The communication bus 440 is used to enable direct communication between these components. In this embodiment, the communication interface 420 of the electronic device is used for signaling or data communication with other node devices. The processor 410 may be an integrated circuit chip with signal processing capabilities.

[0099] The processor 410 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor, or the processor 410 can be any conventional processor.

[0100] The memory 430 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 430 stores computer-readable instructions. When these computer-readable instructions are executed by the processor 410, the electronic device can perform the aforementioned operations. Figure 1 or Figure 2 The various steps involved in the method implementation examples.

[0101] Alternatively, the electronic device may also include a storage controller and an input / output unit.

[0102] The memory 430, storage controller, processor 410, peripheral interface, and input / output unit are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 440. The processor 410 is used to execute executable modules stored in the memory 430, such as software function modules or computer programs included in electronic devices.

[0103] The input / output unit is used to provide users with the ability to create tasks and to set optional start periods or preset execution times for those tasks, thereby enabling user-server interaction. The input / output unit may be, but is not limited to, a mouse and keyboard.

[0104] Understandable. Figure 4 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0105] This application also provides a storage medium storing instructions. When the instructions are run on a computer, the computer program is executed by a processor to implement the method described in the method embodiment. To avoid repetition, the method will not be described again here.

[0106] This application also provides a computer program product that, when run on a computer, causes the computer to perform the method described in the method embodiment.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0108] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0109] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0112] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for identifying VIN codes on nameplates, characterized in that, include: Obtain the image of the vehicle nameplate to be identified; The vehicle nameplate image is processed based on a text detection model to obtain the VIN code location information; During the training process, the text detection model is jointly optimized based on the loss function of the text detection model and the loss function of the auxiliary branch network. The vehicle nameplate image is cropped based on the VIN code location information, and then the cropped image is processed based on a character recognition model to identify the target VIN code contained in the vehicle nameplate image. The text recognition model is trained using multiple sample images with specified labels; the specified labels indicate the VIN code contained in the corresponding sample image. The auxiliary branch network is constructed on the backbone network of the text detection model, with units connected by convolutional layers, batch normalization layers, activation layers, and shortcut layers stacked to form a neck network. A fully connected layer is used to construct the first output terminal, and a convolutional layer is used to construct a second output terminal parallel to the first output terminal. The first output terminal is used to determine the car manufacturer information contained in the input image; the second output terminal is used to determine the number of characters contained at the position of the character in the input image. The auxiliary branch network is trained using multiple nameplate images with car manufacturer labels and character count labels as training samples.

2. The method according to claim 1, characterized in that, The text detection model is trained based on the YOLOv5 algorithm; the VIN code location information includes the VIN code coordinates and confidence level.

3. The method according to claim 2, characterized in that, The text detection model was trained using the following method: Input the target sample image with classification label and ground truth bounding box coordinate label into the initial text detection model to obtain the predicted classification result and predicted coordinate result; A classification loss function is constructed based on the classification identifier and the predicted classification result, and a coordinate loss function is constructed based on the ground truth bounding box coordinate identifier and the predicted coordinate result; After training the initial text detection model using the classification loss function and the coordinate loss function to obtain the basic detection model, the backbone network of the text detection model is jointly tuned using the loss function of the auxiliary branch network, the classification loss function, and the coordinate loss function.

4. The method according to claim 1, characterized in that, The text recognition model includes a CNN branch network, an LSTM branch network, and an MLP branch network. The CNN branch network is used to extract text features from the input image. The LSTM branch network is used to recognize text information based on the text features. The latent variable output by the last subunit of the LSTM branch network is input to the MLP branch network. The MLP branch network is used to determine whether the text information constitutes a VIN code based on the latent variable.

5. The method according to claim 4, characterized in that, The loss function of the character recognition model is obtained by adding CTCLoss and BCE Loss.

6. The method according to claim 2, characterized in that, The method further includes: When the confidence level is higher than the first threshold, the target VIN code is output; When the confidence level is lower than the second threshold, the target VIN code is removed; the second threshold is less than the first threshold.

7. A nameplate VIN code recognition device, characterized in that, include: The acquisition module is used to acquire the image of the vehicle nameplate to be identified; The detection module is used to process the vehicle nameplate image based on a text detection model to obtain the VIN code location information; During the training process, the text detection model is jointly optimized based on the loss function of the text detection model and the loss function of the auxiliary branch network. The recognition module is used to crop the vehicle nameplate image according to the VIN code location information, and then process the cropped image based on the character recognition model to identify the target VIN code contained in the vehicle nameplate image. The text recognition model is trained using multiple sample images with specified labels; the specified labels indicate the VIN code contained in the corresponding sample image. The auxiliary branch network is constructed on the backbone network of the text detection model, with units connected by convolutional layers, batch normalization layers, activation layers, and shortcut layers stacked to form a neck network. A fully connected layer is used to construct the first output terminal, and a convolutional layer is used to construct a second output terminal parallel to the first output terminal. The first output terminal is used to determine the car manufacturer information contained in the input image; the second output terminal is used to determine the number of characters contained at the position of the character in the input image. The auxiliary branch network is trained using multiple nameplate images with car manufacturer labels and character count labels as training samples.

8. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as claimed in any one of claims 1 to 6.

Images