A single-layer / double-layer license plate augmentation method based on machine vision technology

Abstract

The invention discloses a single-layer / double-layer license plate augmentation method based on machine vision technology, comprising: making a license plate image; performing diversity enhancement on the license plate image; adjusting the vehicle license plate corner point position of the real image; calculating four vertex positions for perspective transformation according to the size of the license plate image and the real license plate, and then performing image filling to obtain an image Img_tmp with only the license plate; making an image with the same size as the license plate and all values ​​being 0, using the image for perspective transformation, transforming to a picture of the size of the real license plate, and filling 1 at the remaining positions of the license plate position to obtain a mask image; performing an AND operation on each pixel point of the mask image and the original real vehicle image Img_org to obtain an image Img_crop; performing an OR operation on the pixels of Img_crop and Img_tmp to obtain a final image. The invention can greatly enrich the license plate detection data set and improve the accuracy of rare license plate detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a single-layer / double-layer license plate augmentation method based on machine vision technology. Background Art

[0002] With the increasing development of convolutional neural networks, they have been widely used in related fields such as text recognition and license plate content recognition.

[0003] With the continuous development of deep learning and convolutional neural networks, neural networks are applied to all walks of life. In the transportation system, vehicle license plate positioning and recognition is one of the important research topics in the application of computer vision and pattern recognition technology in the field of intelligent transportation. In actual projects, we can easily obtain pictures of blue license plates and green license plates, which are more common vehicles on the road, but it is difficult to obtain clearer images of rare license plates under natural road conditions, such as yellow license plates (agricultural vehicles and engineering vehicles), black license plates (ambassadors, consulates, foreign-funded enterprise vehicles), white license plates (military vehicles or public security vehicles), etc. This makes it difficult for us to train license plate detection and recognition models for license plates with small amounts of data. When using convolutional neural networks to solve practical problems, the quantity and quality of training data are a bottleneck that restricts the network effect. For some types of niche or difficult to collect data, data augmentation operations are required to meet the training requirements of convolutional neural networks.

[0004] In the invention patent with patent application number CN201910738162.8, a data augmentation method based on a convolutional neural network is disclosed, including: inputting an image with marked license plate information into a data augmentation network, the data augmentation network includes: a data generator and a data discriminator; using the data generator, respectively obtaining three levels of downsampled images of the input image; inputting each level of downsampled image into the corresponding level CCGAN, obtaining the spliced ​​feature vector of the corresponding level, performing image upsampling operation on the spliced ​​feature vector, and using the obtained image as the output image of the corresponding level CCGAN; using the data discriminator, calculating the loss function value of the augmented data generator through the objective function formula, adjusting the network parameters of each level of CCGAN according to the loss function value until the loss function value converges; inputting the license plate image into the data augmentation network for processing, and obtaining the augmented image of the license plate image. Compared with the prior art, this patent can improve the imbalance in the number of types of license plate augmented images. However, this patent mainly divides the threshold, and actually processes the results obtained by the detection algorithm. Instead of processing the license plate image itself. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a single-layer / double-layer license plate augmentation method based on machine vision technology.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A single-layer / double-layer license plate augmentation method based on machine vision technology comprises the following steps:

[0008] Step 1: According to various license plate formats, select fixed characters and randomly select characters from English letters and Arabic numerals to produce the license plate image Img_plate;

[0009] Step 2: Randomly select a prepared license plate image Img_plate and use computer vision methods to perform diversity enhancement;

[0010] Step 3: According to the four corner point information of the vehicle license plate in the real image, the corner point position of the vehicle license plate in the real image is adjusted;

[0011] Step 4: According to the size of the license plate image Img_plate and the real license plate, calculate the four vertex positions for perspective transformation to obtain a picture of the real license plate size, fill the license plate image Img_plate in the four vertices, and fill the rest of the positions with white to obtain an image Img_tmp with only the license plate;

[0012] Step 5: Create an image that is the same size as the license plate, but with all values ​​set to 0; use the image and the four vertex positions to perform perspective transformation, transform it to a picture the size of the actual license plate, and fill the remaining positions at the license plate position with 1; obtain a mask image Img_mask with 0 inside the license plate and 1 in the remaining positions;

[0013] Step 6: Use the mask image Img_mask and the original real vehicle image Img_org to perform an AND operation on each pixel, remove the information inside the license plate, and turn it black, thereby obtaining the image Img_crop;

[0014] Step 7: Use Img_crop to perform an OR operation with each pixel of Img_tmp to fill the license plate into the image and obtain the final image Img_res.

[0015] Furthermore, the license plates include Hong Kong and Macao license plates, consulate license plates, ambassador license plates, police license plates, new energy license plates, emergency rescue license plates, double-layer white license plates, double-layer yellow license plates and single-layer yellow license plates.

[0016] Furthermore, the step 2 includes the following sub-steps:

[0017] Step 201: adding a stain effect by weighted superposition with the prepared stain image to obtain an updated image;

[0018] Step 202: adding noise, adding Gaussian noise to the three BGR channels respectively;

[0019] Step 203: Add motion blur filtering and use the toolkit imgaug to add motion blur to the license plate image.

[0020] Furthermore, the step 3 includes the following sub-steps:

[0021] Step 301: judging whether the vehicle's license plate type is a new energy license plate, a special emergency rescue license plate, or a double-layer license plate according to the four corner point information of the vehicle's license plate in the real image;

[0022] Step 302: If yes, adjust the position of the corner point of the license plate; if no, do not adjust it.

[0023] Furthermore, when the license plate type is a new energy license plate or a special license plate for emergency rescue, the following adjustment steps are performed:

[0024] Step 3011: Calculate the lengths that need to be added to the two line segments S1 and S3 by using ratio = 480px / 440px-1 = 0.0909;

[0025] Step 3012: Calculate the straight line length d1 of S1 and the straight line length d3 of S3 by the distance formula between two points, and calculate the line segment p1=d1*ratio that S1 needs to pad and the line segment p3=d3*ratio that S3 needs to pad; when padding, the length of padding on the left and right of S1 is p1_half=p1 / 2, and the length of padding on the left and right of S3 is p3_half=p3 / 2;

[0026] Step 3013: Through the straight line formulas of S1 and S3 and y=kx+b, and according to the lengths of p1_half and p3_half, the transformed points (x1', y1'), (x2', y2'), (x3', y3'), (x4', y4') are obtained.

[0027] Furthermore, when the vehicle type is a double-layer license plate, the following adjustment steps are performed:

[0028] Step 3014: Calculate the lengths that need to be added to the two line segments S2 and S4 by using ratio = 440px / 220px-1 = 1;

[0029] Step 3015: Calculate the straight line length d2 of S2 and the straight line length d4 of S4 by the distance formula between two points, and calculate the line segment p2=d2*ratio that S2 needs to pad and the line segment p4=d4*ratio that S4 needs to pad; when padding, the length of padding on the left and right of S2 is p2_half=p2 / 2, and the length of padding on the left and right of S4 is p4_half=p4 / 2;

[0030] Step 3016: Through the straight line formula of S2 and S4 and y=kx+b, and according to the lengths of p2_half and p4_half, the transformed points (x1', y1'), (x2', y2'), (x3', y3'), (x4', y4') are obtained.

[0031] Beneficial effects of the invention: The invention proposes a license plate augmentation method in natural scenes, which can greatly enrich the license plate detection data set and improve the accuracy of rare license plate detection; improve and solve the long tail problem of the data set in deep learning, greatly increase the number of rare license plates (black plates, white plates, double-layer yellow and green plates, etc.) in the license plate data, and use a balanced number of license plates of various colors in the license plate samples. Solve the problem of unbalanced sample categories in training. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0033] Figure 1 It is a flow chart of the method of the present invention.

[0034] Figure 2 This is a data diagram when the license plate is adjusted. DETAILED DESCRIPTION

[0035] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0038] In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0039] In this embodiment, Figure 1 As shown, a single-layer / double-layer license plate augmentation method based on machine vision technology includes the following steps:

[0040] Step 1: According to various license plate formats, select fixed characters and randomly select characters from English letters and Arabic numerals to produce the license plate image Img_plate;

[0041] Step 2: Randomly select a prepared license plate image Img_plate and use computer vision methods to perform diversity enhancement;

[0042] Step 3: According to the four corner point information of the vehicle license plate in the real image, the corner point position of the vehicle license plate in the real image is adjusted;

[0043] Step 4: According to the size of the license plate image Img_plate and the real license plate, calculate the four vertex positions for perspective transformation to obtain a picture of the real license plate size, fill the license plate image Img_plate in the four vertices, and fill the rest of the positions with white to obtain an image Img_tmp with only the license plate;

[0044] Step 5: Create an image that is the same size as the license plate, but with all values ​​set to 0; use the image and the four vertex positions to perform perspective transformation, transform it to a picture the size of the actual license plate, and fill the remaining positions at the license plate position with 1; obtain a mask image Img_mask with 0 inside the license plate and 1 in the remaining positions;

[0045] Step 6: Use the mask image Img_mask and the original real vehicle image Img_org to perform an AND operation on each pixel, remove the information inside the license plate, and turn it black, thereby obtaining the image Img_crop;

[0046] Step 7: Use Img_crop to perform an OR operation with each pixel of Img_tmp to fill the license plate into the image and obtain the final image Img_res.

[0047] Among them, the license plates include Hong Kong and Macao license plates, consulate license plates, ambassador license plates, police license plates, new energy license plates, special license plates for emergency rescue, double-layer white license plates, double-layer yellow license plates and single-layer yellow license plates.

[0048] Wherein, the step 2 includes the following sub-steps:

[0049] Step 201: adding a stain effect by weighted superposition with the prepared stain image to obtain an updated image;

[0050] Step 202: adding noise, adding Gaussian noise to the three BGR channels respectively;

[0051] Step 203: Add motion blur filtering and use the toolkit imgaug to add motion blur to the license plate image.

[0052] Wherein, the step 3 includes the following sub-steps:

[0053] Step 301: judging whether the vehicle's license plate type is a new energy license plate, a special emergency rescue license plate, or a double-layer license plate according to the four corner point information of the vehicle's license plate in the real image;

[0054] Step 302: If yes, adjust the position of the corner point of the license plate; if no, do not adjust it.

[0055] In this embodiment, if Figure 2 As shown, when the license plate type is a new energy license plate or a special license plate for emergency rescue, perform the following adjustment steps:

[0056] Step 3011: Calculate the lengths that need to be added to the two line segments S1 and S3 by using ratio = 480px / 440px-1 = 0.0909;

[0057] Step 3012: Calculate the straight line length d1 of S1 and the straight line length d3 of S3 by the distance formula between two points, and calculate the line segment p1=d1*ratio that S1 needs to pad and the line segment p3=d3*ratio that S3 needs to pad; when padding, the length of padding on the left and right of S1 is p1_half=p1 / 2, and the length of padding on the left and right of S3 is p3_half=p3 / 2;

[0058] Step 3013: Through the straight line formulas of S1 and S3 and y=kx+b, and according to the lengths of p1_half and p3_half, the transformed points (x1', y1'), (x2', y2'), (x3', y3'), (x4', y4') are obtained.

[0059] When the vehicle type is a double-layer license plate, perform the following adjustment steps:

[0060] Step 3014: Calculate the lengths that need to be increased on the two line segments S2 and S4. Through ratio = 440px / 220px - 1 = 1;

[0061] Step 3015: Calculate the straight-line lengths d2 of S2 and d4 of S4 using the distance formula between two points, and calculate the line segments p2 = d2 * ratio that S2 needs to be padded and p4 = d4 * ratio that S4 needs to be padded; when padding, the lengths that need to be padded on both the left and right of S2 are both p2_half = p2 / 2, and the lengths that need to be padded on both the left and right of S4 are both p4_half = p4 / 2;

[0062] Step 3016: Obtain the transformed points (x1’, y1’), (x2’, y2’), (x3’, y3’), (x4’, y4’) through the straight-line formulas of S2 and S4 and y = kx + b, and based on the lengths of p2_half and p4_half.

[0063] In step 1, the requirements for making different types of license plates are as follows:

[0064] Hong Kong and Macau license plates: black background with white characters, with a width and height of 440 pixels and 140 pixels respectively. The last character is "Hong Kong" or "Macau", the first character is "Yue", the second character is a number or letter, the third character is a dot, and the remaining 5 middle characters are letters or numbers.

[0065] Consulate license plates: black background, with a width and height of 440 pixels and 140 pixels respectively. The last character is "Consulate", in red. The first white character is the abbreviation of the province or autonomous region, the second white character is a number or letter, the third white character is a dot, and the remaining 4 middle white characters are letters or numbers.

[0066] Ambassador license plates: black background, with a width and height of 440 pixels and 140 pixels respectively. The first red character is "Envoy". Then 3 white characters are letters or numbers, then a white character is a dot, and the following 3 white characters are letters or numbers.

[0067] Police vehicle license plates: white background, with a width and height of 440 pixels and 140 pixels respectively. The last red character is "Police". The first white character is the abbreviation of the province, the second black character is a dot, and the remaining 5 black characters are all letters or numbers.

[0068] New energy vehicle license plates: yellow-green background with black characters, with a width and height of 480 pixels and 140 pixels respectively. The first character is the abbreviation of the province or autonomous region, and the remaining 7 characters are all letters or numbers.

[0069] Special license plates for emergency rescue: with a white background, the width and height are 480 pixels and 140 pixels respectively. The first black character is the abbreviation of the province or autonomous region, the second black character is a separator, the third red character represents the code of the affiliated rescue team, the following four characters are letters or numbers, and the last two red characters are "Emergency".

[0070] Double - layer white license plates: with a white background, the width and height are 440 pixels and 220 pixels respectively. The first red character in the upper layer is "Ying", the second is a black separator, and the third is "Ji". The first black character in the lower layer is the abbreviation of the province or autonomous region, and the following four black characters are letters or numbers.

[0071] Double - layer yellow license plates: with a yellow background, the width and height are 440 pixels and 220 pixels respectively. The first black character in the upper layer is the abbreviation of the province or autonomous region, the second is a black separator, and the third is a letter or number. All five characters in the lower layer are letters or numbers.

[0072] Single - layer yellow license plates: with a yellow background and black characters, the width and height are 440 pixels and 140 pixels respectively. The first character is the abbreviation of the province or autonomous region, the second character is a letter or number, the third character is a separator, and the 4th - 7th characters are letters or numbers. The last character is "Trailer" or "Learner".

[0073] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, some steps can be carried out in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and units involved are not necessarily essential to this application.

[0074] In the above - mentioned embodiments, the descriptions of each embodiment have their own focuses. For the parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0075] Those of ordinary skill in the art can understand that all or part of the processes of implementing the above - mentioned method embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer - readable storage medium. When the program is executed, it can include the processes of the above - mentioned method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a ROM, a RAM, etc.

[0076] The above - disclosed are only the preferred embodiments of the present invention. Of course, the scope of the rights of the present invention cannot be limited thereby. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope covered by the present invention.

Claims

1. A single-layer / double-layer license plate augmentation method based on machine vision technology, characterized in that: The following steps are involved: Step 1: According to various license plate formats, select fixed characters and randomly select characters from English letters and Arabic numerals to produce the license plate image Img_plate; Step 2: Randomly select a prepared license plate image Img_plate and use computer vision methods to perform diversity enhancement; Step 3: According to the four corner point information of the vehicle license plate in the real image, the corner point position of the vehicle license plate in the real image is adjusted; Step 4: According to the size of the license plate image Img_plate and the real license plate, calculate the four vertex positions for perspective transformation to obtain a picture of the real license plate size, fill the license plate image Img_plate in the four vertices, and fill the rest of the positions with white to obtain an image Img_tmp with only the license plate; Step 5: Create an image that is the same size as the license plate, but with all values ​​set to 0; use the image and the four vertex positions to perform perspective transformation, transform it to a picture the size of the actual license plate, and fill the remaining positions at the license plate position with 1; obtain a mask image Img_mask with 0 inside the license plate and 1 in the remaining positions; Step 6: Use the mask image Img_mask and the original real vehicle image Img_org to perform an AND operation on each pixel, remove the information inside the license plate, and turn it black, thereby obtaining the image Img_crop; Step 7: Use Img_crop to perform an OR operation with each pixel of Img_tmp to fill the license plate into the image and obtain the final image Img_res.

2. According to claim 1, a single-layer / double-layer license plate augmentation method based on machine vision technology is characterized in that: The license plates include Hong Kong and Macao license plates, consulate license plates, ambassador license plates, police license plates, new energy license plates, emergency rescue license plates, double-layer white license plates, double-layer yellow license plates and single-layer yellow license plates.

3. According to claim 1, a single-layer / double-layer license plate augmentation method based on machine vision technology is characterized in that: The step 2 includes the following sub-steps: Step 201: adding a stain effect by weighted superposition with the prepared stain image to obtain an updated image; Step 202: adding noise, adding Gaussian noise to the three BGR channels respectively; Step 203: Add motion blur filtering and use the toolkit imgaug to add motion blur to the license plate image.

4. According to claim 1, a single-layer / double-layer license plate augmentation method based on machine vision technology is characterized in that: The step 3 includes the following sub-steps: Step 301: judging whether the vehicle's license plate type is a new energy license plate, a special emergency rescue license plate, or a double-layer license plate according to the four corner point information of the vehicle's license plate in the real image; Step 302: If yes, adjust the position of the corner point of the license plate; if no, do not adjust it.

5. According to claim 4, a single-layer / double-layer license plate augmentation method based on machine vision technology is characterized in that: When the license plate type is a new energy license plate or a special license plate for emergency rescue, perform the following adjustment steps: Step 3011: Calculate the length that needs to be added to the two line segments S1 and S3 by using ratio = 480px / 440px-1 = 0.0909; Step 3012: Calculate the straight line length d1 of S1 and the straight line length d3 of S3 by the distance formula between two points, and calculate the line segment p1=d1*ratio that S1 needs to pad and the line segment p3=d3*ratio that S3 needs to pad; when padding, the length of padding on the left and right of S1 is p1_half=p1 / 2, and the length of padding on the left and right of S3 is p3_half=p3 / 2; Step 3013: Through the straight line formulas of S1 and S3 and y=kx+b, and according to the lengths of p1_half and p3_half, the transformed points (x1', y1'), (x2', y2'), (x3', y3'), (x4', y4') are obtained.

6. According to claim 4, a method for augmenting a single-layer / double-layer license plate based on machine vision technology is characterized in that: When the vehicle type is a double-layer license plate, perform the following adjustment steps: Step 3014: Calculate the lengths that need to be added to the two line segments S2 and S4 by using ratio = 440px / 220px-1 = 1; Step 3015: Calculate the straight line length d2 of S2 and the straight line length d4 of S4 by the distance formula between two points, and calculate the line segment p2=d2*ratio that S2 needs to pad and the line segment p4=d4*ratio that S4 needs to pad; when padding, the length of padding on the left and right of S2 is p2_half=p2 / 2, and the length of padding on the left and right of S4 is p4_half=p4 / 2; Step 3016: Through the straight line formula of S2 and S4 and y=kx+b, and according to the lengths of p2_half and p4_half, the transformed points (x1', y1'), (x2', y2'), (x3', y3'), (x4', y4') are obtained.

Images