A method and device for image recognition

Abstract

This invention discloses an image recognition method and device for implementing a more universal license plate recognition method. In addition to supporting national standard license plates, it supports the recognition of various customized license plates, reducing development workload and cycle time. The method includes: determining the image to be recognized and the external features of objects in the image; inputting the image into a trained recognition model, extracting features from the image using a backbone network to obtain a feature map, inputting the feature map into a first branch network and a second branch network respectively, and outputting a first prediction result and a second prediction result accordingly; wherein the recognition model includes the backbone network, the first branch network, and the second branch network; and determining the recognition result of the object based on the external features of the object, the first prediction result, and the second prediction result.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an image recognition method and apparatus. Background Technology

[0002] National standard vehicle license plates include blue plates, new energy vehicles, and yellow plates. The character format of the license plates is relatively uniform, generally consisting of the province abbreviation + letters + letters or numbers, with a length of 7 or 8 characters. The total number of characters used on the license plates does not exceed a few dozen. Currently, the national standard vehicle license plate recognition scheme is relatively mature and has a high recognition rate.

[0003] However, in some factories or industrial parks, special customized license plates are usually displayed on internal vehicles for convenient management. Many customized license plates usually have new characters that are not in the character library of national standard license plates. For such customized license plates, a large number of images need to be collected for training and the recognition algorithm for customized license plates needs to be re-optimized. The development cycle is long and consumes a lot of manpower and resources. Summary of the Invention

[0004] This invention provides an image recognition method and device for implementing a more universal license plate recognition method. In addition to supporting national standard license plates, it also supports the recognition of various customized license plates, reducing development workload and development cycle.

[0005] In a first aspect, an image recognition method is provided by an embodiment of the present invention, the method comprising:

[0006] Determine the external features of the image to be identified and the objects in the image;

[0007] The image is input into a trained recognition model, and the backbone network is used to extract features from the image to obtain a feature map. The feature map is then input into a first branch network and a second branch network, and a first prediction result and a second prediction result are output accordingly. The recognition model includes the backbone network, the first branch network, and the second branch network.

[0008] The identification result of the object is determined based on the object's external features, the first prediction result, and the second prediction result.

[0009] The general image recognition method provided in this embodiment can ensure the accuracy of recognition while adopting a multi-tasking approach to achieve the recognition of national standard license plates and other customized license plates with lower computing power.

[0010] As an optional implementation, the completion of the recognition model training is determined by the following method:

[0011] Obtain the first and second labeled training sets;

[0012] The first training set is fed into the first branch network through the backbone network to calculate the first loss value, and the second training set is fed into the second branch network through the backbone network to calculate the second loss value.

[0013] The recognition model is trained alternately using the first loss value and the second loss value respectively. When both the first loss value and the second loss value meet the preset conditions, the recognition model is determined to be trained successfully.

[0014] As an optional implementation, the second training set is determined in the following manner:

[0015] Obtain the labeled real image, and replace at least one second character in the real image with at least one generated first character to obtain the composite image;

[0016] The synthesized image is labeled according to the label of at least one first character that is replaced;

[0017] The second training set is determined based on the labeled synthetic images.

[0018] This embodiment reduces the cost of manual annotation by generating synthetic images, making it easier to generate a large number of synthetic images for training.

[0019] As an optional implementation, determining the object's identification result based on the object's external features, the first prediction result, and the second prediction result includes:

[0020] When the external features of the object satisfy a first preset feature, the recognition result of the object is determined according to the first prediction result, wherein the first preset feature is determined based on the external features of the object in the images of the first training set; or,

[0021] When the external features of the object satisfy the second preset features, the recognition result of the object is determined according to the second prediction result, wherein the second preset features are determined based on the external features of the object in the images of the second training set.

[0022] This embodiment selects either the first or second prediction result based on the external features of the result object to determine the recognition result, which ensures the universality of the recognition model.

[0023] As an optional implementation, determining the object's recognition result based on the first prediction result when the object's external features satisfy a first preset feature includes:

[0024] Determine the first character rule corresponding to the first preset feature;

[0025] When the first prediction result matches the first character rule, the recognition result of the object is determined based on the first prediction result.

[0026] As an optional implementation, determining the object's recognition result based on the second prediction result when the object's external features satisfy the second preset feature includes:

[0027] Determine the second character rule corresponding to the second preset feature;

[0028] When the second prediction result matches the second character rule, the recognition result of the object is determined based on the second prediction result.

[0029] As an optional implementation, when the second prediction result does not conform to the second character rule, the method further includes:

[0030] Determine the first character rule corresponding to the first preset feature;

[0031] When the second prediction result matches the first character rule, the recognition result of the object is determined based on the second prediction result.

[0032] As an optional implementation, the first prediction result includes at least one first prediction label with a confidence level greater than a first threshold; and / or,

[0033] The second prediction result includes at least one second prediction label with a confidence level greater than the second threshold.

[0034] This embodiment reduces the reliance on the robustness of the network model and improves the recognition rate by selecting multiple predicted labels and performing rule matching sequentially.

[0035] Secondly, an image recognition device provided by an embodiment of the present invention includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0036] Determine the external features of the image to be identified and the objects in the image;

[0037] The image is input into a trained recognition model, and the backbone network is used to extract features from the image to obtain a feature map. The feature map is then input into a first branch network and a second branch network, and a first prediction result and a second prediction result are output accordingly. The recognition model includes the backbone network, the first branch network, and the second branch network.

[0038] The identification result of the object is determined based on the object's external features, the first prediction result, and the second prediction result.

[0039] As an optional implementation, the processor is specifically configured to determine that the recognition model training is complete in the following manner:

[0040] Obtain the first and second labeled training sets;

[0041] The first training set is fed into the first branch network through the backbone network to calculate the first loss value, and the second training set is fed into the second branch network through the backbone network to calculate the second loss value.

[0042] The recognition model is trained alternately using the first loss value and the second loss value respectively. When both the first loss value and the second loss value meet the preset conditions, the recognition model is determined to be trained successfully.

[0043] As an optional implementation, the processor is specifically configured to determine the second training set in the following manner:

[0044] Obtain the labeled real image, and replace at least one second character in the real image with at least one generated first character to obtain the composite image;

[0045] The synthesized image is labeled according to the label of at least one first character that is replaced;

[0046] The second training set is determined based on the labeled synthetic images.

[0047] As an optional implementation, the processor is specifically configured to execute:

[0048] When the external features of the object satisfy a first preset feature, the recognition result of the object is determined according to the first prediction result, wherein the first preset feature is determined based on the external features of the object in the images of the first training set; or,

[0049] When the external features of the object satisfy the second preset features, the recognition result of the object is determined according to the second prediction result, wherein the second preset features are determined based on the external features of the object in the images of the second training set.

[0050] As an optional implementation, the processor is specifically configured to execute:

[0051] Determine the first character rule corresponding to the first preset feature;

[0052] When the first prediction result matches the first character rule, the recognition result of the object is determined based on the first prediction result.

[0053] As an optional implementation, the processor is specifically configured to execute:

[0054] Determine the second character rule corresponding to the second preset feature;

[0055] When the second prediction result matches the second character rule, the recognition result of the object is determined based on the second prediction result.

[0056] As an optional implementation, when the second prediction result does not conform to the second character rule, the processor is further configured to execute:

[0057] Determine the first character rule corresponding to the first preset feature;

[0058] When the second prediction result matches the first character rule, the recognition result of the object is determined based on the second prediction result.

[0059] As an optional implementation, the first prediction result includes at least one first prediction label with a confidence level greater than a first threshold; and / or,

[0060] The second prediction result includes at least one second prediction label with a confidence level greater than the second threshold.

[0061] Thirdly, embodiments of the present invention also provide an image recognition apparatus, comprising:

[0062] Image feature units are defined to determine the external features of the image to be identified and the objects in the image;

[0063] A prediction result determination unit is used to input the image into a trained recognition model, extract features from the image using the backbone network to obtain a feature map, and input the feature map into a first branch network and a second branch network respectively, and output a first prediction result and a second prediction result accordingly; wherein the recognition model includes the backbone network, the first branch network and the second branch network.

[0064] The identification result determination unit is used to determine the identification result of the object based on the external features of the object, the first prediction result, and the second prediction result.

[0065] As an optional implementation, the prediction result determination unit is specifically used to determine that the recognition model training is complete in the following manner:

[0066] Obtain the first and second labeled training sets;

[0067] The first training set is fed into the first branch network through the backbone network to calculate the first loss value, and the second training set is fed into the second branch network through the backbone network to calculate the second loss value.

[0068] The recognition model is trained alternately using the first loss value and the second loss value respectively. When both the first loss value and the second loss value meet the preset conditions, the recognition model is determined to be trained successfully.

[0069] As an optional implementation, the prediction result determination unit is specifically used to determine the second training set in the following manner:

[0070] Obtain the labeled real image, and replace at least one second character in the real image with at least one generated first character to obtain the composite image;

[0071] The synthesized image is labeled according to the label of at least one first character that is replaced;

[0072] The second training set is determined based on the labeled synthetic images.

[0073] As an optional implementation, the unit for determining the identification result is specifically used for:

[0074] When the external features of the object satisfy a first preset feature, the recognition result of the object is determined according to the first prediction result, wherein the first preset feature is determined based on the external features of the object in the images of the first training set; or,

[0075] When the external features of the object satisfy the second preset features, the recognition result of the object is determined according to the second prediction result, wherein the second preset features are determined based on the external features of the object in the images of the second training set.

[0076] As an optional implementation, the unit for determining the identification result is specifically used for:

[0077] Determine the first character rule corresponding to the first preset feature;

[0078] When the first prediction result matches the first character rule, the recognition result of the object is determined based on the first prediction result.

[0079] As an optional implementation, the unit for determining the identification result is specifically used for:

[0080] Determine the second character rule corresponding to the second preset feature;

[0081] When the second prediction result matches the second character rule, the recognition result of the object is determined based on the second prediction result.

[0082] As an optional implementation, when the second prediction result does not conform to the second character rule, the unit for determining the recognition result is further used for:

[0083] Determine the first character rule corresponding to the first preset feature;

[0084] When the second prediction result matches the first character rule, the recognition result of the object is determined based on the second prediction result.

[0085] As an optional implementation, the first prediction result includes at least one first prediction label with a confidence level greater than a first threshold; and / or,

[0086] The second prediction result includes at least one second prediction label with a confidence level greater than the second threshold.

[0087] Fourthly, embodiments of the present invention also provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the method described in the first aspect above.

[0088] These or other aspects of this application will become more apparent in the following description of embodiments. Attached Figure Description

[0089] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 This is a flowchart illustrating an image recognition method according to an embodiment of the present invention.

[0091] Figure 2 This is a schematic diagram illustrating the generation of a license plate according to an embodiment of the present invention;

[0092] Figure 3A A schematic diagram of a customized license plate provided in an embodiment of the present invention;

[0093] Figure 3B A flowchart for determining the identification result is provided in an embodiment of the present invention;

[0094] Figure 4 This is a flowchart illustrating a specific implementation of an image recognition method provided in an embodiment of the present invention;

[0095] Figure 5 A schematic diagram of an image recognition device provided in an embodiment of the present invention;

[0096] Figure 6 This is a schematic diagram of an image recognition device provided in an embodiment of the present invention. Detailed Implementation

[0097] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0098] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0099] The application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems. In the description of this invention, unless otherwise stated, "multiple" means two or more.

[0100] Example 1: National standard vehicle license plates include blue plates, new energy vehicles, and yellow plates. The character format of the license plates is relatively uniform, generally consisting of the province abbreviation + letter + letter or number, with a length of 7 or 8 characters. The total number of characters used on the license plate does not exceed a few dozen. Currently, the national standard vehicle license plate recognition scheme is relatively mature and has a high recognition rate. However, in some factories or industrial parks, for the convenience of internal vehicle management, internal vehicles usually display special customized license plates. Many customized license plates often contain new characters that are not in the character library of national standard license plates. For such customized license plates, it is necessary to collect a large number of images for training and re-optimize the recognition algorithm for customized license plates, which has a long development cycle and consumes human and material resources.

[0101] This embodiment provides an image recognition method for implementing a more universal license plate recognition method. In addition to supporting standard national license plates, it also supports the recognition of various customized license plates and can be deployed quickly, reducing development workload and cycle time. While ensuring recognition accuracy, it employs a multi-task approach, using a shared backbone network for feature extraction and branch networks for recognition, thus achieving the recognition of both standard and customized license plates with lower computational power.

[0102] The image recognition method provided in this embodiment is based on the core idea of ​​inputting the image to be recognized into different branch networks after passing it through a backbone network, thereby obtaining different prediction results. Based on the external features of the object, the recognition result is determined from the first and second prediction results. Since the two recognition algorithms share a single backbone network and employ two separate branch networks, this method increases computational resources only slightly without affecting the recognition rate of national standard license plates, thus ensuring accuracy.

[0103] like Figure 1 As shown in the figure, this embodiment provides an image recognition method, and the specific implementation process of the method is as follows:

[0104] Step 100: Determine the image to be identified and the external features of the objects in the image;

[0105] In some embodiments, the objects in this embodiment include, but are not limited to, at least one of license plates, people, animals, and vehicles. This embodiment does not impose excessive limitations.

[0106] It should be noted that the image to be identified in this embodiment is an image containing the specified object. Optionally, the image in this embodiment is obtained by cropping the original image. The cropped image removes irrelevant backgrounds and non-specified objects. It can be understood that the image to be identified in this embodiment only contains the specified object.

[0107] In some embodiments, the external features of the image and object to be identified are determined in the following manner:

[0108] Acquire the original image containing the object, perform object detection on the original image to determine the object's position and external features in the original image, and crop the image containing the object based on the object's position in the original image.

[0109] In practice, when the object is a license plate, the collected vehicle image is detected to determine the location and external features of the license plate, and the image containing the license plate is cropped from the collected vehicle image based on the location of the license plate.

[0110] In some embodiments, the external features in this embodiment include, but are not limited to, color, shape, etc. This embodiment does not impose excessive limitations on these aspects. In practice, when the object is a license plate, the external feature of the object can be the color of the license plate. For example, the color of the license plate can be used to initially determine whether the license plate is a standard national license plate or a customized license plate.

[0111] Step 101: Input the image into the trained recognition model, extract the features of the image using the backbone network to obtain a feature map, and input the feature map into the first branch network and the second branch network respectively, and output the first prediction result and the second prediction result accordingly; wherein the recognition model includes the backbone network, the first branch network and the second branch network;

[0112] In implementation, this embodiment does not impose excessive limitations on the structure of the backbone network. Optionally, the recognition algorithm used by the first branch network in this embodiment includes, but is not limited to, at least one of CRNN (Convolutional Recurrent Neural Network), AttentionOCR (Attention Optical Character Recognition), and multi-classification networks; optionally, the recognition algorithm used by the second branch network in this embodiment includes, but is not limited to, at least one of CRNN, AttentionOCR, and multi-classification networks.

[0113] CRNN is primarily used for end-to-end recognition of text sequences of variable length. Instead of segmenting individual characters first, it transforms text recognition into a time-dependent sequence learning problem—essentially image-based sequence recognition. The entire CRNN network structure comprises three parts, from bottom to top: CNN (Convolutional Layer), using a deep CNN to extract features from the input image and obtain feature maps; RNN (Recurrent Layer), using a bidirectional RNN (BLSTM) to predict the feature sequence, learning each feature vector in the sequence and outputting the predicted label (true value) distribution; and CTC (Connectionist Temporal Classification) loss, using CTC loss to transform the series of label distributions obtained from the recurrent layer into the final label sequence.

[0114] In some embodiments, the network structure of the recognition model in this embodiment adopts a shared backbone network multi-task approach. It outputs a first prediction result and a second prediction result through two branch networks. Taking license plates as an example, the two output results can correspond to the prediction results for national standard license plates and general license plates, respectively. In this embodiment, the first branch network can recognize characters appearing in national standard license plates, typically dozens of character categories. The second branch network can recognize a larger number of characters, such as supporting the recognition of all commonly used Chinese characters and thousands of alphanumeric character categories. Using this network structure, compared to only supporting national standard license plates, the computational resources required in engineering deployment are only slightly increased (it simply adds a general license plate recognition branch, shared by the backbone network), and the use of two separate branch networks has almost no impact on the recognition rate of national standard license plates.

[0115] In some embodiments, this embodiment determines that the recognition model training is complete through the following steps:

[0116] Step 1) Obtain the first labeled training set and the second labeled training set;

[0117] In some embodiments, the first training set in this embodiment is obtained by manually annotating the character content based on real license plate data. Optionally, the images in the first training set include characters of national standard license plates.

[0118] In some implementations, the second training set is determined in this embodiment in the following manner:

[0119] Obtain a labeled real image, replace at least one second character in the real image with at least one generated first character to obtain a synthetic image; label the synthetic image according to the label of the at least one replaced first character; determine a second training set based on the labeled synthetic image.

[0120] In implementation, the second training set in this embodiment is generated based on labeled real images. The second training set can be generated using images from the first training set. The generation process is as follows:

[0121] First, determine the positions and contents of each second character in one or more real images; generate multiple first character images using a standard font library such as boldface, Microsoft YaHei, etc.; randomly paste the generated one or more first character images onto the positions of the corresponding second characters in the real images to obtain synthetic images. Here, generating one first character can replace one second character, and generating multiple first characters can replace multiple second characters. Optionally, when pasting, image fusion (the first character image and the license plate image) can also be performed according to the license plate background to make the generated license plate more realistic. Secondly, since each first character corresponds to a label, label the synthetic image according to the label marked by at least one first character for replacement; finally, repeat the above content to generate multiple labeled synthetic images, and determine the second training set based on the generated labeled synthetic images. As Figure 2 shown, this embodiment provides a schematic diagram of generating a license plate, in which the generated character "zen" is used to replace "E" in the real license plate to generate a synthetic license plate image.

[0122] In implementation, by repeatedly generating characters and pasting them onto the real license plate, a large amount of license plate data containing more characters can be generated, which is used to generate license plate data of more character categories, so as to be able to recognize customized license plates.

[0123] In some embodiments, a generative adversarial network (GAN, Generative Adversarial Networks) can also be used to generate license plate data of more character categories, so as to determine the second training set.

[0124] Optionally, the images in the second training set in this embodiment include national standard characters and more Chinese characters.

[0125] Step 2) Input the first training set into the first branch network through the backbone network, calculate the first loss value, and input the second training set into the second branch network through the backbone network, calculate the second loss value;

[0126] In implementation, the first branch network and the first training set in this embodiment are corresponding, and the second branch network and the second training set are corresponding. Thus, different training sets are used to train the branch networks, and finally the object categories that the two branch networks can recognize are different. Optionally, the national standard license plate images are recognized through the first branch network, and the customized license plate images are recognized through the second branch network.

[0127] Step 3) Use the first loss value and the second loss value respectively to alternately train the recognition model. When both the first loss value and the second loss value meet the preset conditions, it is determined that the training of the recognition model is completed.

[0128] It should be noted that in this embodiment, due to the existence of two branch networks, the training is carried out alternately. That is, the backpropagation gradient is calculated first using the first loss value and the identification network parameters (including the backbone network parameters and the first branch network parameters) are updated. Then, the backpropagation gradient is calculated using the second loss value and the identification network parameters (including the backbone network parameters and the second branch network parameters) are updated. Then, the backpropagation gradient is calculated again using the first loss value and the identification network parameters are updated. Then, the backpropagation gradient is calculated again using the second loss value and the identification network parameters are updated. This alternating training continues until both the first loss value and the second loss value meet the preset conditions. If both the first loss value and the second loss value are lower than the preset threshold, the training stops and the identification model training is considered complete.

[0129] In some embodiments, to balance the weight parameters of the first and second loss values, the backpropagation gradient can be calculated by multiplying the second loss value by the weight λ, and the recognition network parameters can be updated, where λ is used to balance the weight parameters of the first and second loss values. The specific formula is shown below:

[0130] L1 = loss1(yW(x)) Formula (1);

[0131] L1.backward() formula (2);

[0132] W.update() formula (3);

[0133] L2=λ·loss2(y_gen-W(x_gen)) Formula (4);

[0134] L2.backward() formula (5);

[0135] W.update() formula (6);

[0136] Where x represents the image in the first training set, y represents the label corresponding to x, W represents the network parameters; x_gen represents the image in the second training set, y_gen represents the label corresponding to x_gen; and λ represents the weight parameters used to balance L1 and L2.

[0137] In this equation, formula (2) indicates that the first loss value is backpropagated to calculate the backpropagation gradient; formula (3) indicates that the calculated gradient value is used to update the network parameters. Similarly, formula (5) indicates that the second loss value is backpropagated to calculate the backpropagation gradient; formula (6) indicates that the calculated gradient value is used to update the network parameters.

[0138] In practice, after the image to be recognized is input into the recognition model, two recognition results are output. Optionally, the first prediction result represents the output after the image is input into the first branch network used to recognize national standard license plate images, and the second prediction result represents the output after the image is input into the second branch network used to recognize customized license plate images.

[0139] Step 102: Determine the recognition result of the object based on the external features of the object, the first prediction result, and the second prediction result.

[0140] In some embodiments, this embodiment first determines whether to use a first prediction result or a second prediction result to determine the object's recognition result based on the object's external features. The specific determination steps are as follows:

[0141] 1) When the external features of the object satisfy the first preset features, the recognition result of the object is determined according to the first prediction result;

[0142] The first preset feature is determined based on the external features of objects in the images of the first training set;

[0143] For example, when the object is a license plate and the external feature is the license plate color, the first branch network is used to identify national standard license plates and the second branch network is used to identify customized license plates. When the license plate color meets the national standard license plate color (e.g., the license plate color is blue), the identification result is determined from the first prediction result.

[0144] In some embodiments, this embodiment determines the recognition result of the object based on the first prediction result through the following steps:

[0145] Determine the first character rule corresponding to the first preset feature;

[0146] When the first prediction result matches the first character rule, the recognition result of the object is determined based on the first prediction result.

[0147] For example, when the object is a license plate, the external feature is the license plate color, and the first character rule is the national standard license plate rule (e.g., a blue license plate for motor vehicles generally has 7 characters, the first character being the province, and the following characters being letters and numbers, etc.), the first branch network is used to recognize national standard license plates, and the second branch network is used to recognize customized license plates. When the license plate color meets the national standard license plate color, it is determined whether the license plate in the first prediction result conforms to the first character rule. If it does, the first prediction result is output; otherwise, an empty character is output.

[0148] 2) When the external features of the object satisfy the second preset features, the recognition result of the object is determined according to the second prediction result;

[0149] The second preset feature is determined based on the external features of objects in the images of the second training set.

[0150] For example, when the object is a license plate and the external feature is the license plate color, the first branch network is used to identify national standard license plates and the second branch network is used to identify customized license plates. When the license plate color meets the requirements of a customized license plate (e.g., the license plate color is green), the identification result is determined from the second prediction result.

[0151] In some embodiments, this embodiment determines the recognition result of the object based on the second prediction result through the following steps:

[0152] Determine the second character rule corresponding to the second preset feature;

[0153] When the second prediction result matches the second character rule, the recognition result of the object is determined based on the second prediction result;

[0154] When the second prediction result does not conform to the second character rule, the first character rule corresponding to the first preset feature is determined; when the second prediction result conforms to the first character rule, the recognition result of the object is determined according to the second prediction result.

[0155] For example, when the object is a license plate, the external feature is the license plate color, the first character rule is the national standard license plate rule, the second character rule is the customized license plate rule, the first branch network is used to recognize national standard license plates, and the second branch network is used to recognize customized license plates, when the license plate color meets the customized license plate color, it is determined whether the license plate in the second prediction result conforms to the second character rule. If it does, the second prediction result is output; otherwise, it is determined whether the second prediction result conforms to the first character rule. If it does, the second prediction result is output; otherwise, an empty character is output.

[0156] In some embodiments, the first character rule in this embodiment includes character length, character type at each position, etc., and the second character rule includes character length, character type at each position, etc. The specific rules can be defined according to the actual situation, and this embodiment does not impose too many limitations.

[0157] In some embodiments, the first character rule in this embodiment may include one or more rules, and the second character rule may also include one or more rules. This embodiment does not impose too many limitations on this.

[0158] like Figure 3AAs shown in the figure, a schematic diagram of a customized license plate provided in this embodiment is taken as an example of the customized license plate. When the object is a license plate, the external feature is the license plate color, the first character rule is the national standard license plate rule, the second character rule is the customized license plate rule, the first branch network is used to identify the national standard license plate, and the second branch network is used to identify the customized license plate, as follows Figure 3B shown, the process of determining the recognition result is described as follows:

[0159] Step 300: Determine that the color of the object conforms to the customized license plate color;

[0160] Step 301: Judge whether the second prediction result conforms to the second character rule. If yes, execute Step 302; otherwise, execute Step 303;

[0161] Among them, the second character rule includes a length rule and a character type. First, it can be judged whether the character length in the second prediction result conforms to the length rule; when it is determined that the character length in the second prediction result conforms to the length rule, then judge whether the character type at the preset position in the second prediction result conforms to the rule. Optionally, the preset position can be one or more positions, and this embodiment does not limit it too much.

[0162] For example, the rules related to the character type in the second character rule are as follows: it is required that the first character in the second prediction result is "Qing", and the last three characters are numbers; or it is required that the second character in the second prediction result is "Lv", and the last three characters are numbers; or it is required that the first character in the second prediction result is "Qing", the second character is "Lv", and the last three characters are numbers.

[0163] Step 302: Output the second prediction result;

[0164] Step 303: Judge whether the character length in the second prediction result conforms to the first character rule. If yes, execute Step 304; otherwise, execute Step 305;

[0165] Step 304: Output the first prediction result;

[0166] Step 305: Output an empty character.

[0167] It should be noted that since Chinese character recognition is relatively difficult, certain restrictions can also be relaxed for one or several Chinese characters. For example, even if the recognition result of the second character in the second prediction result is not "Lv", as long as the characters in other positions conform to the corresponding rules, the second character can be forced to be output as "Lv". Through the above strategy, it can avoid missed recognition caused by the insufficient robustness of the network model to a certain extent and ensure the correct output of characters as much as possible.

[0168] In some embodiments, the first prediction result in this embodiment includes at least one first prediction label with a confidence level greater than a first threshold; and / or, the second prediction result includes at least one second prediction label with a confidence level greater than a second threshold.

[0169] In some embodiments, when the first prediction result includes multiple first prediction labels with confidence levels greater than a first threshold, and the external features of the object satisfy the first preset features, the first prediction labels are judged sequentially in descending order of confidence level to determine whether they conform to the first character rule. If they do, the first prediction label is output; otherwise, the next first prediction label is judged to determine whether it conforms to the first character rule, and so on. When all the first prediction labels do not conform to the first character rule, an empty character is output.

[0170] In some embodiments, when the second prediction result includes multiple second prediction labels with confidence levels greater than the second threshold, and the external features of the object satisfy the second preset features, the second prediction labels can be judged sequentially in descending order of confidence level to determine whether they conform to the second character rule. If they do, the second prediction label is output; otherwise, the next second prediction label is judged to determine whether it conforms to the second character rule, and so on. When all the second prediction labels do not conform to the second character rule, the second prediction labels are then judged sequentially in descending order of confidence level to determine whether they conform to the first character rule. If they do, the second prediction label is output; otherwise, the next second prediction label is judged to determine whether it conforms to the first character rule, and so on. When all the second prediction labels do not conform to the first character rule, an empty character is output.

[0171] In some embodiments, the first character rule and the second character rule in this embodiment can be one or more. The customized license plate recognition scheme in this embodiment mainly formulates corresponding rules to control the license plate output for customized license plates in a certain region or park. If there are multiple customized license plates in a certain region, multiple rules can also be formulated. In the actual deployment stage of the algorithm, the character rules can be configured through a configuration file. Both the first character rule and the second character rule are stored in the form of a configuration file, separating the recognition algorithm program from the rule configuration. In new customized license plate scenarios, only the configuration file needs to be modified to support new license plates, shortening the algorithm development cycle and workload.

[0172] like Figure 4 As shown, this embodiment provides a detailed explanation of an image recognition method using a license plate as an example:

[0173] Step 400: Acquire vehicle images and determine the location and color of the license plate in the vehicle images;

[0174] Step 401: Crop the license plate image to be identified from the vehicle image based on the license plate position;

[0175] Step 402: Input the license plate image into the trained recognition model and output the first prediction result and the second prediction result;

[0176] Step 403: Determine whether the license plate color meets the first preset feature. If yes, proceed to step 404; otherwise, proceed to step 406.

[0177] Step 404: Determine whether the first prediction result satisfies the first character rule. If yes, proceed to step 405; otherwise, proceed to step 411.

[0178] Step 405: Output the first prediction result;

[0179] Step 406: Determine whether the license plate color satisfies the second prediction feature. If yes, proceed to step 407; otherwise, proceed to step 411.

[0180] Step 407: Determine whether the second prediction result satisfies the second character rule. If yes, proceed to step 408; otherwise, proceed to step 409.

[0181] Step 408: Output the second prediction result;

[0182] Step 409: Determine whether the second prediction result satisfies the first character rule. If yes, proceed to step 410; otherwise, proceed to step 411.

[0183] Step 410: Output the second prediction result;

[0184] Step 411: Output an empty character.

[0185] The image recognition method provided in this embodiment is applied to license plate recognition scenarios. It can support the recognition of license plates containing any Chinese characters. By generating license plates, training samples for license plate recognition containing more fonts are generated. The recognition model supports the recognition of more license plates besides the national standard license plates, realizing the versatility of the algorithm. The recognition model adopts an alternating training method with two losses during training. The recognition model adopts a multi-task approach, which reduces the computational requirements of the algorithm. It supports the recognition of both national standard and other license plates without affecting the recognition rate of national standard license plates. Furthermore, the output of this embodiment considers multiple predicted labels with confidence scores higher than a threshold. By matching character rules, the reliance on the robustness of the network model is reduced, thereby improving the recognition rate. The output of customized license plates can be configured through configuration files. The configuration file is separated from the algorithm program, and different rules can be configured in different regions to achieve rapid deployment of the recognition algorithm.

[0186] Example 2: Based on the same inventive concept, this embodiment of the invention also provides an image recognition device. Since this device is the same as the device in the method of this embodiment of the invention, and the principle of the device in solving the problem is similar to that of the method, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be described again.

[0187] like Figure 5 As shown, the device includes a processor 500 and a memory 501. The memory 501 stores programs executable by the processor 500. The processor 500 reads the programs from the memory 501 and performs the following steps:

[0188] Determine the external features of the image to be identified and the objects in the image;

[0189] The image is input into a trained recognition model, and the backbone network is used to extract features from the image to obtain a feature map. The feature map is then input into a first branch network and a second branch network, and a first prediction result and a second prediction result are output accordingly. The recognition model includes the backbone network, the first branch network, and the second branch network.

[0190] The identification result of the object is determined based on the object's external features, the first prediction result, and the second prediction result.

[0191] As an optional implementation, the processor 500 is specifically configured to determine that the recognition model training is complete in the following manner:

[0192] Obtain the first and second labeled training sets;

[0193] The first training set is fed into the first branch network through the backbone network to calculate the first loss value, and the second training set is fed into the second branch network through the backbone network to calculate the second loss value.

[0194] The recognition model is trained alternately using the first loss value and the second loss value respectively. When both the first loss value and the second loss value meet the preset conditions, the recognition model is determined to be trained successfully.

[0195] As an optional implementation, the processor 500 is specifically configured to determine the second training set in the following manner:

[0196] Obtain the labeled real image, and replace at least one second character in the real image with at least one generated first character to obtain the composite image;

[0197] The synthesized image is labeled according to the label of at least one first character that is replaced;

[0198] The second training set is determined based on the labeled synthetic images.

[0199] As an optional implementation, the processor 500 is specifically configured to perform:

[0200] When the external features of the object satisfy a first preset feature, the recognition result of the object is determined according to the first prediction result, wherein the first preset feature is determined based on the external features of the object in the images of the first training set; or,

[0201] When the external features of the object satisfy the second preset features, the recognition result of the object is determined according to the second prediction result, wherein the second preset features are determined based on the external features of the object in the images of the second training set.

[0202] As an optional implementation, the processor 500 is specifically configured to perform:

[0203] Determine the first character rule corresponding to the first preset feature;

[0204] When the first prediction result matches the first character rule, the recognition result of the object is determined based on the first prediction result.

[0205] As an optional implementation, the processor 500 is specifically configured to perform:

[0206] Determine the second character rule corresponding to the second preset feature;

[0207] When the second prediction result matches the second character rule, the recognition result of the object is determined based on the second prediction result.

[0208] As an optional implementation, when the second prediction result does not conform to the second character rule, the processor 500 is further configured to execute:

[0209] Determine the first character rule corresponding to the first preset feature;

[0210] When the second prediction result matches the first character rule, the recognition result of the object is determined based on the second prediction result.

[0211] As an optional implementation, the first prediction result includes at least one first prediction label with a confidence level greater than a first threshold; and / or,

[0212] The second prediction result includes at least one second prediction label with a confidence level greater than the second threshold.

[0213] Example 3: Based on the same inventive concept, this embodiment of the invention also provides an image recognition device. Since this device is the same as the device in the method of this embodiment of the invention, and the principle of the device in solving the problem is similar to that of the method, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be described again.

[0214] like Figure 6 As shown, the device includes:

[0215] Image feature unit 600 is used to determine the external features of the image to be identified and the objects in the image;

[0216] The prediction result determination unit 601 is used to input the image into the trained recognition model, extract the features of the image using the backbone network to obtain a feature map, input the feature map into the first branch network and the second branch network respectively, and output the first prediction result and the second prediction result accordingly; wherein the recognition model includes the backbone network, the first branch network and the second branch network.

[0217] The identification result determination unit 602 is used to determine the identification result of the object based on the external features of the object, the first prediction result, and the second prediction result.

[0218] As an optional implementation, the prediction result determination unit 601 is specifically used to determine that the recognition model training is complete in the following manner:

[0219] Obtain the first and second labeled training sets;

[0220] The first training set is fed into the first branch network through the backbone network to calculate the first loss value, and the second training set is fed into the second branch network through the backbone network to calculate the second loss value.

[0221] The recognition model is trained alternately using the first loss value and the second loss value respectively. When both the first loss value and the second loss value meet the preset conditions, the recognition model is determined to be trained successfully.

[0222] As an optional implementation, the prediction result determination unit 601 is specifically used to determine the second training set in the following manner:

[0223] Obtain the labeled real image, and replace at least one second character in the real image with at least one generated first character to obtain the composite image;

[0224] The synthesized image is labeled according to the label of at least one first character that is replaced;

[0225] The second training set is determined based on the labeled synthetic images.

[0226] As an optional implementation, the determination and recognition result unit 602 is specifically used for:

[0227] When the external features of the object satisfy a first preset feature, the recognition result of the object is determined according to the first prediction result, wherein the first preset feature is determined based on the external features of the object in the images of the first training set; or,

[0228] When the external features of the object satisfy the second preset features, the recognition result of the object is determined according to the second prediction result, wherein the second preset features are determined based on the external features of the object in the images of the second training set.

[0229] As an optional implementation, the determination and recognition result unit 602 is specifically used for:

[0230] Determine the first character rule corresponding to the first preset feature;

[0231] When the first prediction result matches the first character rule, the recognition result of the object is determined based on the first prediction result.

[0232] As an optional implementation, the determination and recognition result unit 602 is specifically used for:

[0233] Determine the second character rule corresponding to the second preset feature;

[0234] When the second prediction result matches the second character rule, the recognition result of the object is determined based on the second prediction result.

[0235] As an optional implementation, when the second prediction result does not conform to the second character rule, the determination of recognition result unit 602 is further configured to:

[0236] Determine the first character rule corresponding to the first preset feature;

[0237] When the second prediction result matches the first character rule, the recognition result of the object is determined based on the second prediction result.

[0238] As an optional implementation, the first prediction result includes at least one first prediction label with a confidence level greater than a first threshold; and / or,

[0239] The second prediction result includes at least one second prediction label with a confidence level greater than the second threshold.

[0240] Based on the same inventive concept, embodiments of the present invention also provide a computer storage medium storing a computer program, which, when executed by a processor, performs the following steps:

[0241] Determine the external features of the image to be identified and the objects in the image;

[0242] The image is input into a trained recognition model, and the backbone network is used to extract features from the image to obtain a feature map. The feature map is then input into a first branch network and a second branch network, and a first prediction result and a second prediction result are output accordingly. The recognition model includes the backbone network, the first branch network, and the second branch network.

[0243] The identification result of the object is determined based on the object's external features, the first prediction result, and the second prediction result.

[0244] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0245] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0246] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0247] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0248] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for image recognition, characterized in that, The method includes: Determine the external features of the image to be identified and the objects in the image; The image is input into a trained recognition model, and the backbone network is used to extract features from the image to obtain a feature map. The feature map is then input into a first branch network and a second branch network, and a first prediction result and a second prediction result are output accordingly. The recognition model includes the backbone network, the first branch network, and the second branch network. Based on the object's external features, the first prediction result, and the second prediction result, the identification result of the object is determined, including: When the external features of the object satisfy a first preset feature, the recognition result of the object is determined according to the first prediction result, wherein the first preset feature is determined based on the external features of the object in the images of the first training set; or, When the external features of the object satisfy the second preset features, the recognition result of the object is determined according to the second prediction result, wherein the second preset features are determined based on the external features of the object in the images of the second training set.

2. The method according to claim 1, characterized in that, The completion of the recognition model training is determined by the following method: Obtain the first and second labeled training sets; The first training set is fed into the first branch network through the backbone network to calculate the first loss value, and the second training set is fed into the second branch network through the backbone network to calculate the second loss value. The recognition model is trained alternately using the first loss value and the second loss value respectively. When both the first loss value and the second loss value meet the preset conditions, the recognition model is determined to be trained successfully.

3. The method according to claim 2, characterized in that, The second training set is determined as follows: Obtain the labeled real image, and replace at least one second character in the real image with at least one generated first character to obtain the composite image; The synthesized image is labeled according to the label of at least one first character that is replaced; The second training set is determined based on the labeled synthetic images.

4. The method according to claim 1, characterized in that, When the external features of the object satisfy a first preset feature, determining the recognition result of the object based on the first prediction result includes: Determine the first character rule corresponding to the first preset feature; When the first prediction result matches the first character rule, the recognition result of the object is determined based on the first prediction result.

5. The method according to claim 1, characterized in that, When the external features of the object satisfy the second preset features, determining the recognition result of the object based on the second prediction result includes: Determine the second character rule corresponding to the second preset feature; When the second prediction result matches the second character rule, the recognition result of the object is determined based on the second prediction result.

6. The method according to claim 5, characterized in that, When the second prediction result does not conform to the second character rule, the following is also included: Determine the first character rule corresponding to the first preset feature; When the second prediction result matches the first character rule, the recognition result of the object is determined based on the second prediction result.

7. The method according to any one of claims 1 to 6, characterized in that, The first prediction result includes at least one first prediction label with a confidence level greater than a first threshold; and / or, The second prediction result includes at least one second prediction label with a confidence level greater than the second threshold.

8. An image recognition device, characterized in that, The device includes a processor and a memory for storing a program executable by the processor, and the processor for reading the program in the memory and executing the steps of the method according to any one of claims 1 to 7.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images