Vehicle nameplate frame number identification method, system and device and storage medium

Abstract

The invention provides a frame number identification method, system and device for a vehicle nameplate, and a storage medium. The method comprises the following steps: training by using a first image identification data set to obtain a vehicle nameplate identification model; inputting the to-be-recognized nameplate image into the vehicle nameplate recognition model to obtain an element set corresponding to the to-be-recognized nameplate image; screening an image with the highest matching degree with the to-be-identified nameplate image from the element set by adopting a local feature matching algorithm, and taking the image as an optimal matching image; acquiring a frame number bounding box of the optimal matching image, mapping the nameplate image to be identified to a coordinate system of the optimal matching image through a coordinate transformation matrix, and performing translation operation on the mapped frame number bounding box to generate a plurality of bounding boxes to be identified; and character recognition is carried out on the image in each bounding box to be recognized, and finally the frame number is obtained. According to the invention, the accuracy and efficiency of frame number identification can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of vehicle frame number recognition, and in particular to a method, system, device and storage medium for recognizing the vehicle frame number of a vehicle nameplate. Background Art

[0002] The vehicle nameplate is an important identification of the car, which generally contains important information about the car, such as brand, engine model, engine displacement, number of passengers, country of manufacture, year of manufacture, and vehicle identification code. The vehicle identification code is also called the vehicle identification number (VIN). In an automobile parts management system, there are many scenarios where the frame number in the picture needs to be identified. When the user needs to extract the frame number in the picture, the user needs to upload a picture, and the automobile parts management system identifies the frame number in the picture and returns it to the user. Vehicle nameplate pictures account for a large proportion of all pictures uploaded by users.

[0003] Frame number recognition, or more generally text recognition, mainly uses the OCR (Optical Character Recognition) algorithm. The OCR algorithm mainly includes two steps, namely text detection and text recognition. The OCR algorithm needs to locate the position of the text in the image through the first step of text detection, and then use the second step of text recognition to identify the text content at the corresponding position. For general vehicle nameplate images, ordinary OCR algorithms can be used for recognition, that is, the method of first performing text detection and then performing text recognition. For images where the frame number position cannot be located by the text detection algorithm, the usual OCR algorithm will not be able to recognize it. For example: in a picture, if the frame number position is not located, the subsequent text recognition will not be able to recognize the frame number.

[0004] In view of this, the present application proposes a method, system, device and storage medium for recognizing the frame number of a vehicle nameplate, which can improve the probability of recognizing the frame number. Summary of the invention

[0005] In order to solve the problem that the existing text detection algorithm cannot locate the frame number position and the image is difficult to recognize the frame number, the present application provides a frame number recognition method, system, device and storage medium for a vehicle nameplate to solve the above-mentioned technical defects.

[0006] According to a first aspect of the present application, a method for recognizing a vehicle frame number of a vehicle nameplate is provided, the method comprising the following steps:

[0007] S1. Collecting a first image recognition data set, and using the first image recognition data set to train a vehicle nameplate recognition model, wherein the first image recognition data set includes a plurality of element sets;

[0008] S2, obtaining a nameplate image to be identified, inputting the nameplate image to be identified into a vehicle nameplate recognition model, determining the brand and subcategory of the nameplate image to be identified, and obtaining an element set corresponding to the nameplate image to be identified;

[0009] S3, using a local feature matching algorithm to select an image with the highest matching degree with the nameplate image to be identified from the element set obtained in step S2 as the optimal matching image;

[0010] S4, obtaining a frame number boundary box of the best matching image, and mapping the nameplate image to be identified to the coordinate system of the best matching image through a coordinate conversion matrix, and performing a translation operation on the mapped frame number boundary box to generate multiple boundary boxes to be identified;

[0011] S5. Perform text recognition on each image within the bounding box to be recognized, and finally obtain the vehicle frame number.

[0012] Preferably, in step S1, collecting a first image recognition data set specifically includes the following sub-steps:

[0013] S11, obtaining a nameplate image, calculating the number of nameplate fields contained in the nameplate image, and if the number of nameplate fields is lower than a preset threshold, removing the corresponding nameplate image to obtain a cleaned nameplate image;

[0014] S12, grouping the cleaned nameplate images according to the frame number prefix, and classifying the grouped nameplate images according to the brand to obtain multiple element sets;

[0015] S13. Perform unsupervised clustering on each element set, classify the element sets according to the nameplate format, and obtain a first image recognition data set.

[0016] Preferably, step S12 further includes: extracting text from the cleaned nameplate image using an OCR model, and classifying the nameplate image according to brand names contained in the text to obtain a plurality of element sets.

[0017] Preferably, in step S3, a local feature matching algorithm is used to select an image with the highest matching degree with the nameplate image to be identified from the element set obtained in step S2 as the best matching image, which specifically includes the following sub-steps:

[0018] S31, using heuristic methods to select multiple representative images from the element set;

[0019] S32, performing local feature matching between the nameplate image to be identified input by the user and the representative image respectively;

[0020] S33. According to the number of matching edges and the confidence level, an image with the highest matching degree with the nameplate image to be identified is obtained as the optimal matching image.

[0021] Preferably, in step S31, it also includes: using a text detection model to detect all text boundary boxes in the representative image, and using a text recognition model to recognize the text in each text boundary box, and filtering out the frame number boundary box from the obtained text boundary boxes according to the text.

[0022] Preferably, in step S4, the nameplate image to be identified is mapped to the coordinate system of the best matching image through a coordinate conversion matrix, and the mapped frame number boundary box is translated to generate multiple boundary boxes to be identified, which specifically includes the following steps:

[0023] S41, obtaining a set of matching edges between the nameplate image to be identified and the optimal matching image, wherein each edge in the set of matching edges is attached with a confidence level;

[0024] S42, filtering the edges whose confidences in the matching edge set are lower than a preset threshold to obtain a filtered matching edge set;

[0025] S43, extracting a matching point set N in the best matching image and a matching point set N′ of the nameplate image to be identified from the filtered matching edge set obtained in step S42;

[0026] S44, using a clustering algorithm to remove noise points from the matching point set N and the matching point set N′, and cutting the optimal matching image and the nameplate image to be identified based on the bounding box of the remaining matching points to obtain the cut optimal matching image and the cut nameplate image to be identified;

[0027] S45, using the filtered matching edge set obtained in step S42 to calculate the coordinate transformation matrix, and mapping the cropped optimal matching image to the coordinate system of the cropped nameplate image to be identified;

[0028] S46, identifying an initial vehicle frame number boundary box in the mapped nameplate image to be identified, and performing a translation operation on the initial vehicle frame number boundary box to generate a plurality of translated boundary boxes to be identified.

[0029] Preferably, in step S46, the initial vehicle frame number boundary box is translated to generate a plurality of translated boundary boxes to be identified, and further comprises: presetting the number of translation operations and the step length; performing a preset number of translation operations on the initial vehicle frame number boundary box in at least one direction according to the preset step length, obtaining a new boundary box to be identified after each translation, and finally generating a list containing a plurality of translated boundary boxes to be identified.

[0030] In a second aspect, the present application provides a vehicle frame number recognition system for a vehicle nameplate, the system comprising:

[0031] An acquisition module is configured to acquire a first image recognition data set and train a vehicle nameplate recognition model using the first image recognition data set, wherein the first image recognition data set includes a plurality of element sets;

[0032] A classification module is configured to obtain a nameplate image to be identified, input the nameplate image to be identified into a vehicle nameplate recognition model, determine the brand and subclass of the nameplate image to be identified, and obtain an element set corresponding to the nameplate image to be identified;

[0033] A matching module is configured to select an image with the highest matching degree with the nameplate image to be identified from the element set obtained from the classification module by using a local feature matching algorithm as the best matching image;

[0034] A mapping module is configured to obtain a frame number boundary frame of the best matching image, and to map the nameplate image to be identified to the coordinate system of the best matching image through a coordinate conversion matrix, and to perform a translation operation on the mapped frame number boundary frame to generate a plurality of boundary frames to be identified;

[0035] The output module is configured to perform text recognition on each image within the bounding box to be recognized, and finally obtain the frame number.

[0036] In a third aspect, the present application provides a terminal device, characterized in that it includes a processor, a memory, and a computer program stored in the memory, and the computer program is executed by the processor to implement the vehicle frame number recognition method of the vehicle nameplate as described in any one of the above.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the medium stores a computer program, and when the computer program is executed by a processor, the vehicle frame number recognition method of a vehicle nameplate as described in any one of the above is implemented.

[0038] Compared with the prior art, the beneficial results of the present invention are:

[0039] (1) Improving the accuracy and efficiency of vehicle frame number recognition: By collecting the first image recognition data set and training the vehicle nameplate recognition model, the brand and subcategory of the nameplate image to be recognized can be automatically and accurately identified, thereby quickly locating the optimal matching image with the highest matching degree with the nameplate image to be recognized, reducing manual intervention and improving recognition efficiency.

[0040] (2) Optimizing dataset quality: When collecting the first image recognition dataset, the dataset was effectively cleaned and the data quality was improved by calculating the number of nameplate fields, classifying according to the vehicle frame number prefix and brand, and performing unsupervised clustering, thereby improving the recognition performance of the model.

[0041] (3) Application of local feature matching algorithm: The local feature matching algorithm is used to screen out the optimal matching image, which can more accurately match the nameplate image to be identified with the images in the element set, thereby improving the accuracy and robustness of the matching.

[0042] (4) Coordinate transformation and translation operation: The nameplate image to be identified is mapped to the coordinate system of the optimal matching image through the coordinate transformation matrix, and the mapped frame number boundary box is translated to generate multiple bounding boxes to be identified, thereby increasing the fault tolerance and flexibility of frame number recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0044] Figure 1 is a flow chart of a method for identifying a vehicle frame number of a vehicle nameplate according to the present application;

[0045] Figure 2 is a structural diagram of a vehicle frame number recognition system for a vehicle nameplate according to the present application;

[0046] Figure 3 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0048] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0049] Assume that a certain number of user-uploaded pictures have been accumulated in the auto parts management system, but these pictures are not necessarily all pictures of vehicle nameplates. They may be pictures of vehicles, auto parts, documents, product labels, mobile phone screenshots, receipts, etc. This application only requires vehicle nameplate pictures, so it is necessary to train an image classification model to identify vehicle nameplate pictures in pictures uploaded by users. After classification by the image classification model, a subset of all pictures uploaded by users is obtained, which are all pictures of vehicle nameplates.

[0050] Secondly, assume that the purpose of users uploading pictures in the automobile parts management system is to extract the frame number from the picture, so when the user uploads a picture in the system, the system will use its built-in OCR algorithm to recognize the picture. But not all pictures can recognize the frame number, such as some blurry pictures. In addition, the frame number is a string of length 17, consisting of English letters and numbers, and the correct frame number string needs to meet the calculation rules of the frame number. This application only retains images whose frame numbers have been identified and whose frame numbers meet the calculation rules, and the image set is recorded as X. An element in the image set X is a tuple (x, y), that is, (x, y)∈X, where x is a vehicle nameplate image and y is a frame number that meets the calculation rules, that is, a set of vehicle nameplate images marked with frame numbers is obtained.

[0051] The main idea of ​​this application is: when the user inputs a vehicle nameplate image (online stage), it is necessary to find one or more images in the image set X that best match the nameplate image to be identified input by the user, and then use these images to help determine the position of the frame number in the nameplate image to be identified.

[0052] Figure 1 The flowchart of the vehicle frame number recognition method of the vehicle nameplate of the present application is shown as follows: Figure 1 As shown, the method comprises the following steps:

[0053] S1. Collect a first image recognition data set, and use the first image recognition data set to train a vehicle nameplate recognition model, wherein the first image recognition data set includes a plurality of element sets. Preferably, ResNet is used to train the vehicle nameplate recognition model. Alternatively, an image retrieval model is trained using the first image recognition data set, and the image retrieval model may use a DELF (DEep Local Features) model or a DELG (DEep Local and Global features) model.

[0054] In this embodiment, collecting the first image recognition data set specifically includes the following sub-steps:

[0055] S11, obtaining a nameplate image, calculating the number of nameplate fields contained in the nameplate image, and if the number of nameplate fields is lower than a preset threshold, removing the corresponding nameplate image to obtain a cleaned nameplate image.

[0056] Image set X contains multiple nameplate images. Since image set X contains some images that have been enlarged and cut by users, such images only retain the local area containing the frame number, which makes the frame number easier to identify. However, the present application hopes that the image used to train the vehicle nameplate recognition model can include the entire vehicle nameplate, so it is necessary to exclude those nameplate images that have been cut in image set X. Specifically, the text information contained in the vehicle nameplate can be used to complete data cleaning. The vehicle nameplate generally contains about ten pieces of vehicle information such as brand, engine model, engine displacement and frame number. These vehicle information are called nameplate fields. When the vehicle nameplate image is cut, it contains only a few nameplate fields. That is, all the text in the nameplate image is identified by an OCR model that can recognize Chinese and English, and then the number of nameplate fields contained in these texts is calculated, and then the number of nameplate fields obtained is compared with the preset threshold to determine whether the nameplate image has been cut and whether it needs to be removed.

[0057] S12, grouping the cleaned nameplate images according to the frame number prefix, and classifying the grouped nameplate images according to the brand to obtain multiple element sets. The OCR model can be used to extract text from the cleaned nameplate images, and classify the nameplate images according to the brand name contained in the text to obtain multiple element sets.

[0058] In this embodiment, the image set X is first grouped according to the vehicle frame number prefix. All prefixes in the comparison table are recorded as set P. For each element p in set P, all images corresponding to element p are taken from the image set X and recorded as S p ={(x,y):(x,y)∈X,g(y)=p}, that is, take out all the elements (x,y) that meet the conditions from the image set X, and the condition that needs to be met is that y is equal to p after taking the prefix. There are a total of |P| image subsets, where |P| represents the number of elements in the set P.

[0059] Second, we need to check each image subset S p ,p∈P corresponds to how many brands. Specifically, the set of all brands is recorded as B, and the set of brands corresponding to the prefix p is recorded as B p , that is, f(p)=B p ,in If the set B p There is only one brand, namely B p = {b}, then the entire image subset S p Recorded as If the set B pIf there are two or more brands, then the image subset S p Classify by brand. Again, use the text contained in the nameplate to classify the nameplate. Assume that set B p Contains two brands a and b, namely B p ={a,b}. For an image (x,y), use the OCR model to extract all the text in this image. Assuming that these texts contain the name of brand b, then put the image (x,y) into the set Right now This results in two image subsets and Through the above operations, all image subsets S p ,p∈P are marked as The subscript p is the prefix corresponding to the image subset, and the superscript is the brand corresponding to the image subset.

[0060] Finally, according to the image subset The superscript of merges all image subsets. Suppose there are two sets S p and S q , p,q∈P, and S p Contains two subsets and S q It also contains two subsets and Combine according to the superscripts a and b, and record and In this way, the image set X can be classified by brand. A total of |B| image subsets are obtained: S a ,S b ,…, where a,b∈B.

[0061] S13. Perform unsupervised clustering on each element set, classify the element sets according to the nameplate format, and obtain a first image recognition data set.

[0062] In this embodiment, not all nameplates of the same brand are in the same format. Nameplates corresponding to different car series, different models or different manufacturers of the same brand may be different, such as the typesetting and color of the nameplate, the color and font of the nameplate text, the fields included in the nameplate, and the size of the nameplate. However, on the other hand, the format of the nameplates of the same brand is relatively fixed. Generally, there are only a few formats of nameplates in a brand. Therefore, it is necessary to perform a nameplate image search on all nameplate images of a brand, i.e., S b , further classified so that the nameplate of each subcategory has the same format.

[0063] For each brand b, we do not know its corresponding nameplate set S b It can be divided into several small categories, so the unsupervised clustering method can be used. First, use the image feature extraction model as the set S b Extract features from each nameplate image in , and then use clustering algorithm to cluster the extracted features. Assume that the clustering algorithm clusters the set S b Divided into k b clusters, where k b is a constant related to brand b, then S b In fact, it is divided into k b subset, denoted as S bi ,i=1,2,…,k b The clustering results of each brand can be manually verified by domain experts. Because the clustering of each brand is completed automatically, and with the help of the user interface to display and interact with the clustering, the manual workload is relatively small. These manual operations can be regarded as the labeling work of the training data set.

[0064] After the above steps, the first image recognition data set of the final image recognition model can be obtained, which is denoted as T = {S bi :b∈B,i=1,2,…,k b}. Each element in the first image recognition dataset T is a set S bi , where the first superscript b∈B is a brand, the second superscript i is the index of a cluster (or subclass) of brand b, and i=1,2,…,k b It means that brand b has a total of k b subcategories. The number of subcategories for each brand is different, so k b It is related to b.

[0065] An element S in the first image recognition dataset T bi represents a category of the model, so there are a total of |T| categories of models. bi The superscript bi of is used as the label of the category, which is interpreted as the i-th subcategory of brand b. The trained models are collectively referred to as vehicle nameplate recognition models.

[0066] It should be understood that any element in the image set X is (x, y), where x is an image and y is the frame number in the image. The first image recognition data set T is grouped according to the frame number. It is known that the first three characters of the frame number are usually related to the brand and series of the vehicle. For example, the frame number starting with "LGX" corresponds to the brand BYD, and the brand represented by "LGB" is Dongfeng Fengxing. However, the first three characters of the frame number and the brand of the vehicle are not one-to-one corresponding. For example, "LGB" and "LGG" both correspond to the brand Dongfeng Fengxing. However, "LVV" corresponds to multiple brands such as Chery and Kaiyi. Therefore, this application is constructed by field experts to have a comparison table of frame number prefixes (specifically prefixes consisting of the first three characters) and vehicle brands. In this comparison table, a frame number prefix may correspond to one or more vehicle brands, and a vehicle brand may correspond to one or more frame number prefixes. The frame number has an obvious long-tail effect, and the frame number prefixes included in the comparison table of this application may not cover all frame numbers on the market, but only the frame number prefixes listed in the comparison table are considered in this application.

[0067] Define a comparison table of frame number prefixes and vehicle brands as a function f. Function f maps a frame number prefix to a set of vehicle brands. For example, f("LGX") = {"BYD"} means that the prefix "LGX" corresponds to a brand set {"BYD"} and there is only one brand in this set, while f("LVV") = {"Chery", "Kaiyi"} means that "LVV" corresponds to two brands. In addition, define a function g that takes the first three characters of the frame number. If you input a frame number into this function, it will output the first three characters of the frame number, such as g("LGBC76C33H0000000") = "LGB".

[0068] Continue to refer Figure 1 The vehicle frame number recognition method provided by the present application also includes the following steps:

[0069] S2. Obtain a nameplate image to be identified, input the nameplate image to be identified into a vehicle nameplate recognition model, determine the brand and subcategory to which the nameplate image to be identified belongs, and obtain an element set corresponding to the nameplate image to be identified.

[0070] The goal of this application is to find several images in the image set X that are closest to the nameplate image to be identified input by the user, in order to narrow the range of images to be found to a certain set S bi middle.

[0071] S3, using a local feature matching algorithm to select the image with the highest matching degree with the nameplate image to be identified from the element set obtained in step S2 as the best matching image. Preferably, the best matching image is selected by a local feature matching model, and the local feature matching model can use a LoFTR (Local Feature TRansformer) model or a SuperGlue model.

[0072] In this embodiment, a local feature matching algorithm is used to select an image with the highest matching degree with the nameplate image to be identified from the element set obtained in step S2 as the best matching image, which specifically includes the following sub-steps:

[0073] S31, using a heuristic method to select multiple representative images from the element set. Preferably, a text detection model is used to detect all text bounding boxes in the representative image, and a text recognition model is used to recognize the text in each text bounding box, and the frame number bounding box is selected from the obtained text bounding boxes according to the text.

[0074] S32, performing local feature matching between the nameplate image to be identified input by the user and the representative image respectively.

[0075] S33. According to the number of matching edges and the confidence level, an image with the highest matching degree with the nameplate image to be identified is obtained as the optimal matching image.

[0076] Specifically, the nameplate image to be identified input by the user online is recorded as x′. In the above, the vehicle nameplate recognition model is used to locate the image to be found in the set S bi Then the nameplate image x′ to be identified is combined with the set S bi The images in the set S are matched and bi The image closest to x′ is selected from the image. Since the vehicle nameplate is attached to the vehicle and photographed by the user with a camera, the user may shoot from different angles and the distance between the camera and the nameplate may also be different, so the nameplate photos may have differences in perspective, distance, or light. In addition to the nameplate, the nameplate photo may also contain interference from other objects, text or background, so the local feature matching method is preferably used when matching two nameplate images.

[0077] The characteristic of local feature matching is that it does not match two photos as a whole, but finds the feature points in the local area of ​​the photo and establishes a comparison relationship between the feature points in the two photos. For example, there are two input images x and x'. If the local feature matching algorithm calculates that the feature point m in image x and the feature point m' in image x' are matched, then the algorithm will output an edge (m, m'), which connects the two feature points m and m'. At the same time, the algorithm will also output a confidence of this edge, which is a decimal between 0 and 1. In addition, a feature point is represented by the two-dimensional pixel coordinates in the photo.

[0078] When two photos are input to the local feature matching model, the model will output a list of edges and the confidence level of each edge. Each edge connects two feature points, one from each input photo. When two photos are input, the matching degree of the two photos can be judged based on the number of edges and confidence levels output by the model. bi Each image in the image set S is matched, and the photo with the highest number of edges and confidence matching the nameplate image x′ to be identified is selected. bi This method will consume a lot of time when there are many images in S. Therefore, a heuristic method can be selected to select bi Select some representative images from the image set S. bi The maximum number of representative images can be set according to the actual situation. bi All the images in are matched pairwise. From the calculation results, we can find that some images have a high degree of matching with each other. These images are regarded as an equivalent class. bi Allowing for n representative images, then S bi Divide into n equivalence classes and select a representative image for each equivalence class. The division of equivalence classes can be achieved by using heuristic algorithms or other algorithms.

[0079] When an image set S bi After the n representative images in the image set S are determined, the bounding box of the vehicle frame number needs to be found in each representative image. For a representative image, the text detection model needs to be used to detect all the text bounding boxes in the image, and then the text recognition model needs to be used to recognize the text in each bounding box. biEach image x in corresponds to a label y, and y is the vehicle number in this image, so the label y can be used to determine which bounding box contains the vehicle number, and the bounding box containing the vehicle number is called the vehicle number bounding box. A bounding box consists of four vertices, and a vertex is represented by a two-dimensional pixel coordinate, so the coordinates of the bounding box are represented as (u1, v1, u2, v2, u3, v3, u4, v4), where u1, v1 represent the coordinates of the first vertex.

[0080] Use image set S bi The n representative images in the image are matched with the nameplate image x′ to be identified input by the user. The n representative images are denoted as x1, x2, …, x n , that is, it is necessary to input n image pairs such as x1 and x′, x2 and x′ into the local feature matching model respectively. Among the n results output by the local feature matching model, select the image with the largest number of matching edges and the highest confidence as the image that best matches the nameplate image x′ to be identified, and record this image as the best matching image x * .

[0081] Continue to refer Figure 1 The vehicle frame number recognition method provided by the present application also includes the following steps:

[0082] S4, obtaining the frame number boundary box of the best matching image, and mapping the nameplate image to be identified to the coordinate system of the best matching image through a coordinate conversion matrix, and performing a translation operation on the mapped frame number boundary box to generate multiple boundary boxes to be identified.

[0083] In this embodiment, the nameplate image to be identified is mapped to the coordinate system of the best matching image through a coordinate conversion matrix, and the mapped frame number boundary box is translated to generate multiple boundary boxes to be identified, which specifically includes the following steps:

[0084] S41. Obtain a set of matching edges between the nameplate image to be identified and the best matching image, wherein each edge in the set of matching edges is attached with a confidence level.

[0085] S42: Filter the edges in the matching edge set whose confidence is lower than a preset threshold to obtain a filtered matching edge set.

[0086] S43 . Extracting the matching point set N in the best matching image and the matching point set N′ of the nameplate image to be identified from the filtered matching edge set obtained in step S42 .

[0087] S44. Use a clustering algorithm to remove noise points from the matching point set N and the matching point set N′ respectively, and cut the optimal matching image and the nameplate image to be identified based on the bounding box of the remaining matching points to obtain the cut optimal matching image and the cut nameplate image to be identified; the clustering algorithm can use the DBSCAN algorithm.

[0088] S45, using the filtered matching edge set obtained in step S42 to calculate the coordinate transformation matrix, and mapping the cropped optimal matching image to the coordinate system of the cropped nameplate image to be identified. The algorithm for calculating the coordinate transformation matrix preferably uses the findHomography method of opencv.

[0089] S46, identifying an initial vehicle frame number boundary box in the mapped nameplate image to be identified, and performing a translation operation on the initial vehicle frame number boundary box to generate a plurality of translated boundary boxes to be identified.

[0090] Preferably, in step S46, the initial vehicle frame number boundary box is translated to generate a plurality of translated boundary boxes to be identified, and further comprises: presetting the number of translation operations and the step length; performing a preset number of translation operations on the initial vehicle frame number boundary box in at least one direction according to the preset step length, obtaining a new boundary box to be identified after each translation, and finally generating a list containing a plurality of translated boundary boxes to be identified.

[0091] S5. Perform text recognition on each image within the bounding box to be recognized, and finally obtain the vehicle frame number.

[0092] Specifically, in step S3, the local feature matching model is used to find the best matching image x that best matches the nameplate image x′ to be identified input by the user. * . And find the best matching image x * The purpose is to use the optimal matching image x * The frame number bounding box in helps determine the position of the frame number in the nameplate image x′ to be recognized. However, the optimal matching image x * There may be a difference in perspective between the nameplate image x′ and the nameplate image to be identified, and the position of the nameplate in the image may also be different, so the optimal matching image x cannot be used directly * The coordinates of the frame number in the image x′ to be identified are used to locate the frame number in the nameplate image to be identified. The following steps need to be performed:

[0093] First, define the set of edges output by the local feature matching model as E = {(m i ,m′ i ):i=1,2,…,L}, assuming there are L edges in total. The point m on the left side of each edge i From the best matching image x * , and the point m′ on the righti Then it comes from the nameplate image x′ to be identified input by the user. Each edge in the set E has a confidence level. The edges with lower confidence levels can be filtered out first, and the filtered set is still recorded as E.

[0094] Secondly, the nameplate image x′ to be identified input by the user is an image of the vehicle nameplate, but the nameplate may only occupy a small part of the nameplate image x′ to be identified, and the rest is the background. In order to reduce the range of searching for the frame number, the nameplate image x′ to be identified can be cropped first. Because the set E is mainly the matching relationship between the feature points on one nameplate and the feature points on another nameplate, the points contained in the set E can be used to crop the image. However, the set E will also contain some mismatched edges, which can be called noise. Let N = {m i :i=1,2,…,L} and N′={m′ i :i=1,2,…,L}, that is, N is the image x * N′ is the set of matching points in image x′. A clustering algorithm that can eliminate noise is used to cluster N. Noise points in set N are removed based on the clustering results. Specifically, RANSAC (RANdom Sampling And Consensus) is used to eliminate the interference of outliers. The bounding box of the remaining points in set N is then used to cluster image x. * The same operation is performed on the set N′, and the image x′ is cut. The cut images are still recorded as the best matching images x * And the nameplate image x′ to be identified.

[0095] Again, the coordinate transformation matrix is ​​calculated using the set E, and then the nameplate image x′ to be identified is mapped to the optimal matching image x by the coordinate transformation matrix. * in the coordinate system of .

[0096] Finally, the best matching image x * In order to improve the probability of detecting the frame number bounding box in the nameplate image x′ to be identified, it is preferred to use the best matching image x * The vehicle number bounding box (u1,v1,u2,v2,u3,v3,u4,v4) in the image is translated at a certain step size. For example, the bounding box is translated down by 5 pixels to obtain (u1,v1+5,u2,v2+5,u3,v3+5,u4,v4+5). The bounding box is translated multiple times in all directions to obtain a list of bounding boxes. For example, 32 bounding boxes are obtained by translation, and then text recognition is performed on the images in these 32 bounding boxes. This can increase the probability of recognizing the vehicle number.

[0097] The main goal of this application is to locate the position of the frame number in the image input by the user, and this goal is achieved mainly by the following steps: first, use image classification or image retrieval methods to find an image from the existing image set that best matches the nameplate image to be identified input by the user, and then use the image matching method to calculate the set of matching point pairs between the nameplate image to be identified and the best matching image, and then calculate the coordinate transformation matrix of the two images based on the set of matching point pairs, and put the nameplate image to be identified and the best matching image into the same coordinate system through the coordinate transformation matrix, and use the coordinates of the frame number in the best matching image to locate the frame number in the nameplate image to be identified input by the user. This application can locate the frame number in the image input by the user without using a text detection model. In other words, this application can process images where the text detection model cannot detect the frame number. This application complements the text detection model, improves the recall rate of text detection, and of course indirectly improves the accuracy of text recognition.

[0098] Further references Figure 2 As an implementation of the above method, in the second aspect, the present application provides an embodiment of a structural diagram of a vehicle nameplate frame number recognition system 200, which can be specifically applied to various electronic devices. The system 200 includes the following modules:

[0099] The acquisition module 210 is configured to acquire a first image recognition data set and train a vehicle nameplate recognition model using the first image recognition data set, wherein the first image recognition data set includes a plurality of element sets;

[0100] The classification module 220 is configured to obtain the nameplate image to be identified, input the nameplate image to be identified into the vehicle nameplate recognition model, determine the brand and subclass of the nameplate image to be identified, and obtain an element set corresponding to the nameplate image to be identified;

[0101] The matching module 230 is configured to select the image with the highest matching degree with the nameplate image to be identified from the element set obtained by the classification module 220 by using a local feature matching algorithm as the best matching image;

[0102] A mapping module 240 is configured to obtain a frame number boundary frame of the best matching image, map the nameplate image to be identified to the coordinate system of the best matching image through a coordinate conversion matrix, and perform a translation operation on the mapped frame number boundary frame to generate a plurality of boundary frames to be identified;

[0103] The output module 250 is configured to perform text recognition on each image within the boundary box to be recognized, and finally obtain the vehicle frame number.

[0104] In a third aspect, the present application provides a terminal device, characterized in that it includes a processor, a memory, and a computer program stored in the memory, and the computer program is executed by the processor to implement the vehicle frame number recognition method of the vehicle nameplate as described in any one of the above.

[0105] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the medium stores a computer program, and when the computer program is executed by a processor, the vehicle frame number recognition method of a vehicle nameplate as described in any one of the above is implemented.

[0106] Reference below Figure 3 , which shows a schematic diagram of the structure of a computer system 300 of a terminal device or server suitable for implementing an embodiment of the present application. Figure 3 The terminal device or server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0107] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the computer system 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0108] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0109] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0110] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for identifying a vehicle frame number of a vehicle nameplate, characterized in that: The method comprises the following steps: S1. Collect a first image recognition data set, and use the first image recognition data set to train a vehicle nameplate recognition model, wherein the first image recognition data set includes a plurality of element sets; S2, obtaining a nameplate image to be identified, inputting the nameplate image to be identified into the vehicle nameplate recognition model, determining the brand and subcategory to which the nameplate image to be identified belongs, and obtaining an element set corresponding to the nameplate image to be identified; S3, using a local feature matching algorithm to select an image with the highest matching degree with the nameplate image to be identified from the element set obtained in step S2 as the best matching image; S4, obtaining a frame number boundary frame of the best matching image, and mapping the nameplate image to be identified to the coordinate system of the best matching image through a coordinate conversion matrix, and performing a translation operation on the mapped frame number boundary frame to generate a plurality of boundary frames to be identified; S5. Perform text recognition on each image within the to-be-recognized boundary box to finally obtain the vehicle frame number.

2. The method for recognizing the vehicle frame number of a vehicle nameplate according to claim 1, characterized in that: In step S1, a first image recognition data set is collected, which specifically includes the following sub-steps: S11, obtaining a nameplate image, calculating the number of nameplate fields contained in the nameplate image, and if the number of nameplate fields is lower than a preset threshold, removing the corresponding nameplate image to obtain a cleaned nameplate image; S12, grouping the cleaned nameplate images according to the frame number prefixes, and classifying the grouped nameplate images according to brands to obtain multiple element sets; S13. Perform unsupervised clustering on each of the element sets, classify the element sets according to nameplate formats, and obtain the first image recognition data set.

3. The method for recognizing the vehicle frame number of a vehicle nameplate according to claim 2, characterized in that: In step S12, it also includes: using the OCR model to extract text from the cleaned nameplate image, and classifying the nameplate image according to the brand name contained in the text to obtain multiple element sets.

4. The method for recognizing the vehicle frame number of a vehicle nameplate according to claim 1, characterized in that: In step S3, a local feature matching algorithm is used to select an image with the highest matching degree with the nameplate image to be identified from the element set obtained in step S2 as the best matching image, which specifically includes the following sub-steps: S31, selecting a plurality of representative images from the element set by using a heuristic method; S32, performing local feature matching between the nameplate image to be identified input by the user and the representative image respectively; S33. According to the number of matching edges and the confidence level, an image with the highest matching degree with the nameplate image to be identified is obtained as the optimal matching image.

5. The method for recognizing the vehicle frame number of a vehicle nameplate according to claim 4, characterized in that: In step S31, it also includes: using a text detection model to detect all text boundary boxes in the representative image, and using a text recognition model to recognize the text in each of the text boundary boxes, and filtering out the frame number boundary box from the obtained text boundary boxes according to the text.

6. The method for recognizing the vehicle frame number of a vehicle nameplate according to claim 1, characterized in that: In step S4, the nameplate image to be identified is mapped to the coordinate system of the optimal matching image through a coordinate conversion matrix, and the mapped frame number boundary box is translated to generate multiple boundary boxes to be identified, which specifically includes the following steps: S41, obtaining a set of matching edges between the nameplate image to be identified and the optimal matching image, wherein each edge in the set of matching edges is attached with a confidence level; S42, filtering the edges whose confidences in the matching edge set are lower than a preset threshold to obtain a filtered matching edge set; S43, extracting a matching point set N in the optimal matching image and a matching point set N′ of the nameplate image to be identified from the filtered matching edge set obtained in step S42; S44, using a clustering algorithm to remove noise points from the matching point set N and the matching point set N′, and cutting the optimal matching image and the nameplate image to be identified based on the bounding box of the remaining matching points to obtain a cut optimal matching image and a cut nameplate image to be identified; S45, using the filtered matching edge set obtained in step S42 to calculate the coordinate transformation matrix, and mapping the cropped optimal matching image to the coordinate system of the cropped nameplate image to be identified; S46, identifying an initial vehicle frame number boundary box in the mapped nameplate image to be identified, and performing a translation operation on the initial vehicle frame number boundary box to generate a plurality of translated boundary boxes to be identified.

7. The method for recognizing the vehicle frame number of a vehicle nameplate according to claim 6, characterized in that: In step S46, the initial vehicle frame number boundary box is translated to generate a plurality of translated boundary boxes to be identified, and the method further includes: presetting the number of translation operations and the step length; performing a preset number of translation operations on the initial vehicle frame number boundary box in at least one direction according to the preset step length, obtaining a new boundary box to be identified after each translation, and finally generating a list containing a plurality of translated boundary boxes to be identified.

8. A vehicle frame number recognition system for a vehicle nameplate, characterized in that: The system comprises: An acquisition module is configured to acquire a first image recognition data set and train a vehicle nameplate recognition model using the first image recognition data set, wherein the first image recognition data set includes a plurality of element sets; a classification module configured to obtain a nameplate image to be identified, input the nameplate image to be identified into the vehicle nameplate recognition model, determine the brand and subcategory to which the nameplate image to be identified belongs, and obtain an element set corresponding to the nameplate image to be identified; A matching module configured to select an image with the highest matching degree with the nameplate image to be identified from the element set obtained from the classification module by using a local feature matching algorithm as the best matching image; A mapping module, configured to obtain a frame number boundary frame of the best matching image, map the nameplate image to be identified to the coordinate system of the best matching image through a coordinate conversion matrix, and perform a translation operation on the mapped frame number boundary frame to generate a plurality of boundary frames to be identified; The output module is configured to perform text recognition on each image within the boundary box to be recognized, and finally obtain the frame number.

9. A terminal device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the vehicle frame number recognition method of the vehicle nameplate as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, wherein a computer program is stored in the medium, and when the computer program is executed by a processor, the method for recognizing the vehicle frame number of a vehicle nameplate according to any one of claims 1 to 7 is implemented.

Images