License plate recognition method, electronic device and storage medium

Abstract

The present application discloses a license plate recognition method, an electronic device and a storage medium. The license plate recognition method includes: obtaining at least one initial image feature, each initial image feature is obtained by extracting the features of the image to be detected collected from the license plate; based on each initial image feature, determining the character offset information about the license plate in each initial image feature and the position information of the area of ​​interest, the area of ​​interest is the area where the license plate is located in the image to be detected, and each character offset information is used to represent the position relationship between at least part of the characters in the image to be detected and the preset position in the license plate; based on each character offset information and the position information of each area of ​​interest, determining the license plate recognition result of the image to be detected, the license plate recognition result includes at least one of the following: the type of the license plate, the area where the license plate is located in the image to be detected, and the character information in the license plate. The above scheme can improve the accuracy of license plate recognition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a license plate recognition method, electronic equipment and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, smart transportation plays an increasingly important role in the field of modern transportation, among which license plate recognition technology is a typical application in smart transportation. The license plate is the vehicle number and registration information. The license plate can be used to obtain the relevant information of the vehicle's location. According to the arrangement of characters, the license plate type can be divided into single-layer license plates and double-layer license plates. The license plate generally contains 7 to 8 characters, which are composed of Chinese characters, uppercase English characters and Arabic numerals. License plate recognition technology mainly refers to the recognition of characters on the license plate in the image through OCR technology. The various arrangements of characters on various types of license plates, and the various changes in fonts and colors may directly affect the accuracy of OCR technology recognition, resulting in low accuracy of license plate recognition results obtained by directly using OCR technology for license plate recognition of various types of license plates.

[0003] In view of the existing technical defects, how to provide an effective license plate recognition solution is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0004] The present application at least provides a license plate recognition method, an electronic device and a storage medium.

[0005] The present application provides a license plate recognition method, comprising: obtaining at least one initial image feature, each initial image feature is obtained by extracting features of an image to be detected collected from a license plate; based on each initial image feature, determining character offset information about the license plate in each initial image feature and position information of an area of ​​interest, the area of ​​interest being the area where the license plate is located in the image to be detected, and each character offset information is used to represent the positional relationship between at least part of the characters in the image to be detected and a preset position in the license plate; based on each character offset information and the position information of each area of ​​interest, determining a license plate recognition result of the image to be detected, the license plate recognition result including at least one of the following: the type of the license plate, the area where the license plate is located in the image to be detected, and the character information in the license plate.

[0006] The present application provides a license plate recognition device, including: an acquisition module, a first determination module and a second determination module; the acquisition module is used to acquire at least one initial image feature, each initial image feature is respectively extracted from the image feature to be detected collected from the license plate; the first determination module is used to determine, based on each initial image feature, character offset information about the license plate and position information of an area of ​​interest in each initial image feature, the area of ​​interest being the area where the license plate is located in the image to be detected, each character offset information is respectively used to indicate the positional relationship between at least part of the characters in the image to be detected and a preset position in the license plate; the second determination module is used to determine, based on each character offset information and the position information of each area of ​​interest, a license plate recognition result of the image to be detected, the license plate recognition result including at least one of the following: the type of the license plate, the area where the license plate is located in the image to be detected, and the character information in the license plate.

[0007] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned license plate recognition method.

[0008] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned license plate recognition method is implemented.

[0009] The above scheme obtains at least one initial image feature, and then determines the character offset information about the license plate and the position information of the area of ​​interest in each initial image feature based on each initial image feature. The character offset information can enrich the overall feature information of the license plate. The combination of character offset information and position information is conducive to enriching the position information and semantic information of the character. Further, combining the character offset information and the position information of each area of ​​interest to determine the license plate recognition result of the image to be detected can improve the recognition ability of the license plate number, that is, improve the accuracy of the license plate recognition result, and in this way, the recognition process of characters in various types of license plates can be unified without the need for special processing of different types of license plates.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 This is a flowchart of an embodiment of the license plate recognition method of the present application. Figure 1 ;

[0013] Figure 2ais a schematic diagram of a single-layer license plate in an embodiment of the license plate recognition method of the present application;

[0014] Figure 2b is a schematic diagram of a double-layer license plate in an embodiment of the license plate recognition method of the present application;

[0015] Figure 3 is a schematic diagram of a feature construction module in an embodiment of a license plate recognition method of the present application;

[0016] Figure 4a is a schematic diagram of a first feature interaction module in an embodiment of a license plate recognition method of the present application;

[0017] Figure 4b is a schematic diagram of a second feature interaction module in an embodiment of a license plate recognition method of the present application;

[0018] Figure 5 is a schematic diagram of a preset decoding layer in an embodiment of the license plate recognition method of the present application;

[0019] Figure 6 This is a flow chart 2 of an embodiment of a license plate recognition method of the present application;

[0020] Figure 7 is a schematic diagram of a license plate recognition network in an embodiment of a license plate recognition method of the present application;

[0021] Figure 8 It is a structural schematic diagram of an embodiment of the license plate recognition device of the present application;

[0022] Fig. 9 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0023] Fig.10 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0024] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0025] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0026] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0027] The present application provides some license plate recognition methods and license plate recognition devices. The application scenarios of the license plate recognition method include but are not limited to the license plate recognition of single-layer license plates and / or the license plate recognition of double-layer license plates. The execution subject of the license plate recognition method may be a license plate recognition device. For example, the license plate recognition device may be arranged in a terminal device or a server or other processing device, wherein the terminal device may be a device for license plate recognition, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, etc. In some possible implementations, the license plate recognition method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0028] See also Figure 1 , Figure 1 This is a flowchart of an embodiment of the license plate recognition method of the present application. Figure 1 Specifically, the license plate recognition method may include the following steps:

[0029] Step S11: Acquire at least one initial image feature.

[0030] Each initial image feature is obtained by extracting features of the image to be detected collected from the license plate.

[0031] Each initial image feature may be obtained by extracting features from an image to be detected collected by a license plate. The image to be detected is an image collected by an image acquisition device from an area related to the license plate of the vehicle to be detected. The area related to the license plate may be a partial area of ​​the rear face or a partial area of ​​the front face of the vehicle to be detected. Each vehicle to be detected may correspond to an image to be detected. In some application scenarios, the method for extracting features from the image to be detected may be to input the image to be detected into a backbone network. The choice of the backbone network is relatively flexible, and may be any backbone network with strong feature extraction capabilities, such as the ResNet series, the Swin Transformer series, and the like. In other application scenarios, the method for extracting features from the image to be detected may be to input the image to be detected into the backbone network and the feature encoder in sequence, so as to learn the global information of the image to be detected, adjust the attention paid to features at different positions in the image to be detected, strengthen key features, weaken irrelevant features, and obtain at least one initial image feature. In some application scenarios, the initial image features may include image features obtained by extracting features from the image to be detected collected by the license plate. The at least one initial image feature may be one initial image feature corresponding to the image to be detected, or may be multiple initial image features corresponding to the image to be detected. It is understandable that at least one refers to one or more.

[0032] Step S12: Based on each initial image feature, determine the character offset information about the license plate and the position information of the region of interest in each initial image feature.

[0033] The region of interest is the region where the license plate is located in the image to be detected, and each character offset information is used to represent the positional relationship between at least part of the characters in the image to be detected and the preset position in the license plate.

[0034] The region of interest may be a region of the above-mentioned region related to the license plate that only includes the entire license plate. Exemplarily, the region of interest is the region of the entire license plate. The region of the entire license plate includes the license plate frame region and the characters included in the license plate frame region. In some application scenarios, the position information of the region of interest includes the specific position coordinates of the license plate frame in the image to be detected. Exemplarily, the position information of the region of interest includes the coordinates of the upper left corner of the license plate frame (that is, the pixel coordinates of the upper left corner of the license plate frame), the coordinates of the lower right corner of the license plate frame (that is, the pixel coordinates of the lower right corner of the license plate frame), and the width and height of the license plate frame. At least some of the characters may be all the characters included in the license plate frame region or some of the characters included in the license plate frame region. The preset position in the license plate may be any position in the license plate frame region. Exemplarily, the preset position in the license plate may be the coordinate position of the center point in the license plate frame region. The license plate frame region is a four-finger frame, and the preset position in the license plate is the coordinate position of the center point of the four-finger frame. Each character offset information is used to indicate the positional relationship between at least some of the characters in the image to be detected and the preset position in the license plate. It is understandable that the accuracy of the parameters corresponding to the different layers contained in the backbone network may affect the accuracy of the initial image features obtained. In some application scenarios, during the training of the license plate recognition network, if the parameters corresponding to the different layers contained in the backbone network are all accurate, the initial image features obtained are all accurate features. Then the character offset information about the license plate obtained by each initial image feature is the same information. And the location information of the region of interest obtained by each initial image feature is the same information. In other application scenarios, during the training of the license plate recognition network, if the parameters corresponding to the different layers contained in the backbone network are inaccurate, some of the initial image features obtained may be inaccurate features. Furthermore, based on each initial image feature, it is determined that the character offset information about the license plate and the location information of the region of interest in each initial image feature may be inaccurate. For example, the offset between any two character offset information is not the same. For example, the license plate frame area may overlap between the location information of any two regions of interest. In some application scenarios, in the above step S12, only the initial image features output by the layer with accurate parameters included in the backbone network can be selected to determine the character offset information about the license plate and the position information of the region of interest in the initial image features. The layer with accurate parameters included in the backbone network can be a layer with parameters greater than a threshold parameter. In other application scenarios, in the above step S12, the initial image features with a confidence greater than or equal to a preset confidence among the initial image features can be selected as at least one feature to be constructed to determine the character offset information of the at least one feature to be constructed and the position information of the region of interest. Then, based on the character offset information of at least one feature to be constructed and the position information of the region of interest, the license plate recognition result of the image to be detected is determined.

[0035] The above step S12 may be to use each initial image feature as an initial image feature set, and the initial image feature set includes each initial image feature. The initial image feature set is directly input into the preset perceptron network at one time or each initial image feature is input into the preset perceptron network in a random order. The preset perceptron network can be obtained by at least one perceptron module. Each perceptron module in at least one perceptron module can be a multi-layer perceptron (MLP). The above step S12 may be to use each initial image feature or the initial image feature set as the input of the preset perceptron network, and obtain the character offset information about the license plate in each initial image feature output by the preset perceptron network and the position information of the area of ​​interest about the license plate in each initial image feature.

[0036] Step S13: Based on the offset information of each character and the position information of each region of interest, the license plate recognition result of the image to be detected is determined.

[0037] The license plate recognition result includes at least one of the following: the type of the license plate, the area where the license plate is located in the image to be detected, and the character information in the license plate.

[0038] The type of the license plate may be whether the license plate in the image to be detected is a single-layer license plate or a double-layer license plate. In some application scenarios, there may be multiple license plates in the image to be detected, and the type of the license plate may be whether each license plate in the image to be detected is a single-layer license plate or a double-layer license plate. The area where the license plate is located in the image to be detected may be a target frame of the license plate. The target frame of the license plate may be the above license plate frame or a target detection frame that is more accurate than the above license plate frame. The character information in the license plate may be the recognition result of all characters contained in the license plate frame area.

[0039] Optionally, the result recognition model can be pre-trained, and the offset information of each character and the position information of each region of interest are used as the input of the result recognition model, and the output of the result recognition model is the license plate recognition result of the image to be detected. Alternatively, the output of the result recognition model is post-processed to obtain the license plate recognition result of the image to be detected. The post-processing can be to select the license plate recognition result with the highest confidence from the license plate recognition results corresponding to each initial image feature as the license plate recognition result of the image to be detected.

[0040] The above scheme obtains at least one initial image feature, and then determines the character offset information about the license plate and the position information of the area of ​​interest in each initial image feature based on each initial image feature. The character offset information can enrich the overall feature information of the license plate. The combination of character offset information and position information is conducive to enriching the position information and semantic information of the character. Further, combining the character offset information and the position information of each area of ​​interest to determine the license plate recognition result of the image to be detected can improve the recognition ability of the license plate number, that is, improve the accuracy of the license plate recognition result, and in this way, the recognition process of characters in various types of license plates can be unified without the need for special processing of different types of license plates.

[0041] In some embodiments, the above-mentioned step S11 may include the following steps: obtaining an image to be detected; performing multi-scale feature extraction on the image to be detected to obtain a multi-scale feature set corresponding to the image to be detected, wherein the multi-scale feature set contains at least two scale features and the scales of each scale feature are different; performing feature fusion on each scale feature in the multi-scale feature set to obtain at least one initial image feature.

[0042] After obtaining the image to be detected, the image to be detected is used as the input of the backbone network, and multi-scale feature extraction of the image to be detected is implemented in the backbone network to obtain a multi-scale feature set corresponding to the image to be detected output by the backbone network. The multi-scale feature set contains at least two scale features and the scales of each scale feature are different.

[0043] In some application scenarios, the method of fusing the scale features in the multi-scale feature set to obtain at least one initial image feature may be to directly combine the scale features in the scale feature set arbitrarily to obtain at least one initial image feature with the same number of scale features. In other application scenarios, the method of fusing the scale features in the multi-scale feature set to obtain at least one initial image feature may be to splice the scale features in the scale feature set to obtain at least one initial image feature with the same number of scale features. In other application scenarios, the method of fusing the scale features in the multi-scale feature set to obtain at least one initial image feature may be to set a weight for each scale feature according to the distance from the current scale feature, and then for each current scale feature, based on the weight of each scale feature, fuse all scale features to obtain the initial image feature corresponding to the current scale feature. In other application scenarios, the method of fusing the scale features in the multi-scale feature set to obtain at least one initial image feature may be to use each scale feature in the scale feature set as an input of a preset feature encoder to obtain the initial image feature corresponding to each scale feature output by the preset feature encoder.

[0044] Exemplarily, after the backbone network is used to perform multi-scale feature extraction on the image to be detected to obtain multiple scale features, the above-mentioned method of fusing the scale features in the multi-scale feature set to obtain at least one initial image feature can be: performing convolution operations on the scale features in the scale feature set to unify the number of channels of the multiple scale features to obtain each processed scale feature, and flattening and splicing the processed scale features to obtain a spliced ​​feature. The spliced ​​feature can be expressed as In the spliced ​​features, B can represent the batch size, S represents the sum of the width and height products of each scale feature, and C represents the number of channels of the spliced ​​features. The spliced ​​features are used as the input of the preset feature encoder to obtain the initial image features corresponding to each scale feature output by the preset feature encoder. Each initial image feature output by the preset feature encoder can be represented as . Each initial image feature In , B may represent a batch size, S represents the sum of the width and height products of each scale feature, and C represents the number of channels of each initial image feature. The preset feature encoder may be a Transformer feature encoder.

[0045] It can be understood that by sending the spliced ​​feature F into the Transformer feature encoder, the global information of the image to be detected can be learned, the attention of features at different positions in the spliced ​​feature F can be adjusted, key features can be strengthened, irrelevant features can be weakened, and each initial image feature can be obtained. Then each initial image feature has the information of the global context, so that the license plate recognition result obtained by using each initial image feature in the subsequent processing is more comprehensive and accurate.

[0046] In some embodiments, the above step S12 may include the following steps: for each initial image feature, using the initial image feature as the input of the first perceptron module to obtain the character offset information output by the first perceptron module; for each initial image feature, using the initial image feature as the input of the second perceptron module to obtain the position information of the area of ​​interest output by the second perceptron module.

[0047] Each initial image feature or initial image feature set is used as an input of a preset perceptron network in a license plate recognition network, and character offset information about the license plate in each initial image feature output by the preset perceptron network and position information about the region of interest of the license plate in each initial image feature are obtained. Specifically, the preset perceptron network includes a first perceptron module and a second perceptron module. The first perceptron module and the second perceptron module are both multi-layer perceptrons (MLPs), and the parameters of the first perceptron module and the second perceptron module are different.

[0048] See also Figure 2a and Figure 2b , Figure 2ais a schematic diagram of a single-layer license plate in an embodiment of the license plate recognition method of the present application. And, Figure 2b It is a schematic diagram of a double-layer license plate in an embodiment of the license plate recognition method of the present application.

[0049] For each initial image feature, the initial image feature is used as the input of the first perceptron module to obtain the character offset information about the license plate in the initial image feature output by the first perceptron module. Specifically, the character offset information about the license plate in all initial image features can be expressed as , that is, the offset information of each character can be expressed as .like Figure 2a As shown, each character offset is used to represent the offset of 8 characters in the license plate frame area of ​​a single-layer license plate relative to the coordinate position of the center point of the license plate frame area, and the offset is from the coordinate position of the center point of the license plate frame area to each character. Figure 2b As shown, each character offset is used to represent the offset of 8 characters in the license plate frame area of ​​the double-layer license plate relative to the coordinate position of the center point of the license plate frame area, and the offset is from the coordinate position of the center point of the license plate frame area to each character. It can be understood that Figure 2a and Figure 2b As shown, the 8 characters in the license plate frame area are respectively in character frames 1 to 8. The center point of the license plate frame area is point p, and the coordinate position of the center point is at any position in the center frame where point P is located. Specifically, the coordinate position of the center point is at the center position of the center frame where point P is located. The offset of each character is pointed from the coordinate position of point p to any position of the character frame where each character is located. In some application scenarios, the offset of each character is pointed from the coordinate position of point p to the coordinate position of the upper right corner of the character frame where each character is located. The offset of each character is pointed from the coordinate position of point p to the coordinate position of the center point corresponding to the character frame where each character is located.

[0050] For each initial image feature, the initial image feature is used as the input of the second perceptron module, and the position information of the region of interest about the license plate in the initial image feature output by the second perceptron module is obtained. The position information of the region of interest about the license plate in all initial image features can be represented as the candidate box predicted by each initial image feature B represents the batch size, and S represents the sum of the width and height products of each initial image feature. The last dimension 4 represents the center coordinates (x, y) and width and height (w, h) of the four-finger frame.

[0051] In some embodiments, the above step S13 may include the following steps: for each initial image feature, using the character offset information related to the initial image feature and the position information of the area of ​​interest, determine a character feature set from the initial image feature, and the character feature set includes the character features of each character in at least some of the characters; for each initial image feature, fuse the initial image feature with each character feature in the corresponding character feature set to obtain a target fusion feature; based on each initial image feature and each target fusion feature, determine the license plate recognition result of the image to be detected.

[0052] The character feature set includes the character features of each character in at least some of the characters. The present application takes the example that each character feature set includes the character features of each character in all the characters in the license plate frame area. Each character feature in each character feature set can carry the position information of the character in the license plate frame area, the order information of the character among all the characters, the semantic information of the character, and the visual information of the character. Among them, the semantic information of the character can be the text information or vocabulary with special meaning to the character in the license plate frame area. Among them, the visual information of the character can be that the character has a specific font and size. It can be understood that each initial image feature corresponds to a character offset information. Each initial image feature corresponds to the position information of an area of ​​interest.

[0053] The above-mentioned step of determining the character feature set from the initial image feature using the character offset information related to the initial image feature and the position information of the area of ​​interest can be, for each initial image feature, using the coordinate position of the center point contained in the character offset information related to the initial image feature and the position information of the area of ​​interest related to the initial image feature, finding the corresponding preset number of character features in the license plate frame area in the initial image feature by bilinear interpolation, and forming the preset number of character features into a character feature set.

[0054] In some application scenarios, for each initial image feature, the initial image feature and each character feature in the character feature set corresponding to the initial image feature are fused to obtain a target fused feature. It is understandable that all initial image features and each character feature in the character feature set corresponding to all initial image features are fused to obtain a target fused feature. In other application scenarios, some initial image features and each character feature in the character feature set corresponding to the some initial image features are selected for fusion to obtain a target fused feature.

[0055] See also Figure 3 , Figure 3 It is a schematic diagram of a feature construction module in an embodiment of the license plate recognition method of the present application.

[0056] like Figure 3As shown, the license plate recognition network includes a feature construction module. The feature construction module includes a linear layer and a first perceptron module in a preset perceptron network and a second perceptron module in a preset perceptron network. The initial image feature set is used as the input of the linear layer, and the linear layer outputs the probability score of each initial image feature belonging to a license plate positive sample, and the index of the top k initial image features ranked from high to low is obtained. According to the index, a license plate query vector composed of the top k initial image features in the initial image feature set is selected. . B can represent the batch size, k is used to represent the number of initial image features in the license plate query vector, 1 is used to represent the width-height product of each of the k initial image features, and C represents the number of channels of the license plate query vector. The license plate query vector can be a collection of at least part of the initial images in the initial image feature set. The linear layer can be a classification linear layer Linear. Then, the initial feature set is used as the input of the first perceptron module to obtain the character offset information about the license plate in all the initial image features output by the first perceptron module. The initial feature set is used as the input of the second perceptron module to obtain the candidate boxes predicted by all the initial image features output by the second perceptron module. Finally, according to the index, The character offsets corresponding to the k initial image features in B can represent the batch size, k is used to represent the number of initial image features in the license plate query vector, 8 represents the 8 character features corresponding to each initial image feature, and 2 is used to represent the area where the character box of the 8 characters corresponding to each initial image feature is located. Specifically, 2 is used to represent the upper left corner coordinates and lower right corner coordinates of the character box of each character in the 8 characters corresponding to each initial image feature. And the center point coordinates of Bbox, in the initial image feature set By bilinear interpolation, we find The 8 character features corresponding to each license plate feature (that is, each initial image feature) are combined into a character query vector B can represent the batch size, k is used to represent the number of initial image features in the license plate query vector, 8 represents the 8 character features corresponding to each initial image feature, and C represents the number of channels of the character query vector. The character query vector can refer to the character feature set corresponding to at least part of the initial image features. and character query vector Concatenate in the third dimension to get the license plate character query vector B can represent the batch size, k is used to represent the number of initial image features in the license plate query vector, 8 represents the 8 character features corresponding to each initial image feature, 1 represents the initial image feature, and C represents the number of channels of the license plate character query vector. Among them, the result after splicing in the third dimension can be referred to as The dimension in is (1+8). It can be understood that the license plate character query vector can refer to the set of target fusion features, that is, the target fusion feature set. Finally, the license plate recognition result of the image to be detected is determined based on the initial image features and the target fusion features.

[0057] It can be considered that determining at least part of the initial image features and the character feature set corresponding to at least part of the initial image features from the index can reduce the dimension of the feature space, remove the initial image features and the character feature set corresponding to the initial image features that contribute less to the license plate recognition process, thereby improving the accuracy and prediction performance of the license plate recognition process. Removing redundant and irrelevant features can help the license plate recognition device focus more on the features that really affect the prediction results.

[0058] In some embodiments, the number of at least one initial image feature is multiple. Before determining the license plate recognition result of the image to be detected based on each initial image feature and each target fusion feature, the method also includes: determining at least one feature to be constructed based on the confidence ranking corresponding to each initial image feature, at least one feature to be constructed is at least part of the initial image features in each initial image feature, and the confidence of each feature to be constructed is greater than or equal to a preset confidence.

[0059] The confidence ranking corresponding to each initial image feature can be the ranking of the probability score of each initial image feature belonging to the positive sample of the license plate. The probability score of each initial image feature belonging to the positive sample of the license plate can refer to the confidence of the initial image feature. The feature to be constructed is the initial image feature that meets the confidence condition. The confidence condition is that if the confidence of the initial image feature is greater than or equal to the preset confidence, the initial image feature is determined as the feature to be constructed. The preset confidence can be a fixed value or a dynamic value. When the preset confidence is a dynamic value, the preset confidence is the confidence corresponding to the initial image feature ranked kth in confidence.

[0060] In some embodiments, based on each initial image feature and each target fusion feature, determining the license plate recognition result of the image to be detected includes: based on each initial image feature and each target fusion feature corresponding to the feature to be constructed, determining the license plate recognition result of the image to be detected.

[0061] Obtain target fusion features related to all initial image features. Each target fusion feature is a fusion result of an initial image feature and each character feature in a character feature set corresponding to the initial image feature. In some application scenarios, after each feature to be constructed is determined from all initial image features, the license plate recognition result of the image to be detected is determined by all initial image features and the target fusion features corresponding to each feature to be constructed. In other application scenarios, after each feature to be constructed is determined from all initial image features, the license plate recognition result of the image to be detected is determined by each feature to be constructed and the target fusion features corresponding to each feature to be constructed. Each feature to be constructed and the target fusion features corresponding to each feature to be constructed can be used as the input of a multi-layer decoder to obtain the license plate recognition result of the image to be detected. This application takes the example of determining the license plate recognition result of the image to be detected by all initial image features and the target fusion features corresponding to each feature to be constructed.

[0062] In some embodiments, the above step of determining the license plate recognition result of the image to be detected based on each initial image feature and each target fusion feature may include the following steps: inputting each initial image feature and each target fusion feature into a multi-layer decoder for feature fusion processing to obtain candidate features corresponding to the image to be detected. Channel splitting of the candidate features to obtain at least one image candidate feature and at least one character candidate feature. Based on each image candidate feature and each character candidate feature, the license plate recognition result is determined.

[0063] The multi-layer decoder may include several preset decoding layers arranged in cascade, each preset decoding layer has the same structure but different parameters, and the features output by the last preset decoding layer are used as the features output by the multi-layer decoder. In some application scenarios, each preset decoding layer may include a feature fusion module that can directly fuse the initial image features and the target fusion features. In other application scenarios, each preset decoding layer may include a single cross-attention module, or may include other attention modules in addition to the cross-attention module. In the case where the preset decoding layer includes a cross-attention module and other attention modules, the cross-attention module and other attention modules may be arranged in series or in parallel. The setting order of the cross-attention module and other attention modules in each preset decoding layer is not limited here.

[0064] In some embodiments, the multi-layer decoder includes several preset decoding layers arranged in cascade, each preset decoding layer has the same structure but different parameters, and the features output by the last preset decoding layer are used as the features output by the multi-layer decoder, each preset decoding layer includes a starting attention module, and the features output by the multi-layer decoder are candidate features corresponding to the image to be detected. The above-mentioned step of inputting each initial image feature and each target fusion feature into the multi-layer decoder for feature fusion processing to obtain the candidate features corresponding to each image to be detected may include the following steps: First, the first query key-value pair is used as the input of the starting attention module in the current preset decoding layer to obtain the first feature output by the starting attention module in the current preset decoding layer, and the query data, key data and value data in the first query key-value pair are respectively obtained from each target fusion feature or from the features output by the previous preset decoding layer, and the current preset decoding layer is one of several preset decoding layers. Then, the first feature output by the starting attention module in the current preset decoding layer is fused with the input of the starting attention module in the current preset decoding layer to obtain the first fused feature. Subsequently, the first fused feature is cross-attended with the initial image feature to obtain the feature output by the current preset decoding layer.

[0065] In some application scenarios, each target fusion feature or the feature output by the previous preset decoding layer is directly used as the query data, key data, and value data in the first query key-value pair. In other application scenarios, each target fusion feature or the feature output by the previous preset decoding layer is mapped to obtain each mapped target fusion feature or the feature output by the previous preset decoding layer. Specifically, the mapping process can be achieved through different linear transformations. This ensures that the query data, key data, and value data in the first query key-value pair are in the same vector space.

[0066] In some application scenarios, when the current preset decoding layer is the first preset decoding layer in a multi-layer decoder, the input of the starting attention module in the current preset decoding layer is the first query key-value pair. The query data, key data and value data in the first query key-value pair are obtained from the fusion features of each target respectively. The first feature output by the starting attention module in the current preset decoding layer is fused with the input of the starting attention module in the current preset decoding layer, that is, the first feature output by the starting attention module in the current preset decoding layer is fused with the fusion features of each target to obtain the first fusion feature. The specific fusion processing method may be to perform cross-layer addition processing, and use the first feature output by the starting attention module in the current preset decoding layer and the fusion features of each target as the input of the third preset interaction module to obtain the first fusion feature output by the third preset interaction module. In other application scenarios, when the current preset decoding layer is not the first preset decoding layer in a multi-layer decoder, the input of the starting attention module in the current preset decoding layer is the first query key-value pair. The query data, key data and value data in the first query key-value pair are obtained from the features output by the previous preset decoding layer respectively. Each target fusion feature may include a license plate query vector Initial image features and character query vectors in The initial attention module is used to fusion the license plate query vector contained in each target feature. The initial image features (that is, the license plate features) and the character query vector The character features (that is, character features) of each character in the character feature set are interactively learned. The starting attention module may be a collaborative self-attention module. The first feature output by the starting attention module in the current preset decoding layer is fused with the input of the starting attention module in the current preset decoding layer, that is, the first feature output by the starting attention module in the current preset decoding layer is fused with the feature output by the previous preset decoding layer to obtain a first fused feature. The specific fusion processing method may be to perform a cross-layer addition process, and use the first feature output by the starting attention module in the current preset decoding layer and the feature output by the previous preset decoding layer as the input of the third preset interaction module to obtain the first fused feature output by the third preset interaction module.

[0067] See also Figure 4a , Figure 4a It is a schematic diagram of the first feature interaction module in an embodiment of the license plate recognition method of the present application.

[0068] In the case where the initial attention module includes the first interaction module, the first interaction module is used to perform feature interaction processing on the character feature set and the initial image feature contained in each feature input by the first interaction module, and obtain the features output by the first interaction module. Specifically, the feature interaction processing can be the interaction between the features contained in the row and the features contained in the column. Figure 4a As shown, A contains the license plate query vector and character query vector of the same initial image feature, that is, A is used to represent any target fusion feature, and the target fusion feature contains the character feature set and the initial image feature belonging to the same initial image feature. Among them, the square part shown in A can represent the initial image feature, and each circular part can represent the character feature set of the initial image feature. a contains the license plate query vector and character query vector of the same initial image feature, that is, a is used to represent any target fusion feature, and the target fusion feature contains the character feature set and the initial image feature belonging to the same initial image feature. Among them, the relationship between A and a is the same target fusion feature, A is the query data in the first query key-value pair, and a is the key data in the first query key-value pair. The target fusion feature pointed to by A and a and the target fusion feature pointed to by B and b are any two target fusion features in each target fusion feature. That is, A and a and B and b are target fusion features corresponding to different initial image features, and the same license plate refers to the target fusion features belonging to the same initial image feature. The relationship between B and b is the same target fusion feature, B is the query data in the first query key-value pair, and b is the key data in the first query key-value pair. The contents of each part of B and b can refer to the contents of each part of A and a, and will not be repeated here.

[0069] See also Figure 4b , Figure 4b It is a schematic diagram of the second feature interaction module in an embodiment of the license plate recognition method of the present application.

[0070] In the case where the starting attention module includes the second interaction module, the second interaction module is used to perform feature interaction processing on the character features of the character feature sets belonging to different initial image features contained in each feature input by the second interaction module and between the different initial image features, to obtain the features output by the second interaction module. Specifically, the feature interaction processing can be the interaction between the features contained in the row and the features contained in the column. Figure 4bAs shown, D1 is used to represent the license plate query vector of the same initial image feature, and D2, D3 and D4 are used to represent the character query vector of the same initial image feature, that is, D1, D2, D3 and D4 are used to represent any target fusion feature, and the target fusion feature includes the character feature set belonging to the same initial image feature and the initial image feature. Among them, the square part shown in D1 can represent the initial image feature, and the circular parts shown in D2, D3 and D4 can represent the character feature set of the initial image feature. d1 is used to represent the license plate query vector of the same initial image feature, and d2, d3 and d4 are used to represent the character query vector of the same initial image feature, that is, d1, d2, d3 and d4 are used to represent any target fusion feature, and the target fusion feature includes the character feature set belonging to the same initial image feature and the initial image feature. Among them, the square part shown in d1 can represent the initial image feature, and the circular parts shown in d2, d3 and d4 can represent the character feature set of the initial image feature. The relationship between the first parameter to be compared (D1, D2, D3 and D4) and the second parameter to be compared (d1, d2, d3 and d4) is the same target fusion feature, the first parameter to be compared is the query data in the first query key-value pair, and the second parameter to be compared is the key data in the first query key-value pair. The third parameter to be compared (E1, E2, E3 and E4) and the fourth parameter to be compared (e1, e2, e3 and e4) can refer to the first parameter to be compared and the second parameter to be compared. The relationship between the third parameter to be compared and the fourth parameter to be compared is the same target fusion feature, the third parameter to be compared is the query data in the first query key-value pair, and the fourth parameter to be compared is the key data in the first query key-value pair. The target fusion feature pointed to by the first parameter to be compared and the second parameter to be compared, and the target fusion features pointed to by the third parameter to be compared and the fourth parameter to be compared are any two target fusion features in each target fusion feature. That is, the first parameter to be compared and the second parameter to be compared, and the third parameter to be compared and the fourth parameter to be compared are target fusion features corresponding to different initial image features. Different license plates refer to different initial image features between the target fusion features, that is, target fusion features corresponding to different initial image features.

[0071] The starting attention module includes a first interaction module and / or a second interaction module. When the starting attention module includes the first interaction module and the second interaction module, the first interaction module and the second interaction module can be set in parallel or in series. When the first interaction module and the second interaction module are set in parallel, each target fusion feature is used as the input of the first interaction module and the second interaction module respectively to obtain the features output by the first interaction module and the features output by the second interaction module. Then, the features output by the first interaction module and the features output by the second interaction module are fused to obtain the first feature. When the first interaction module and the second interaction module are set in series, each target fusion feature can be input into the first interaction module and the second interaction module in sequence, or each target fusion feature can be input into the second interaction module and the first interaction module in sequence. This application takes the example of inputting each target fusion feature into the first interaction module and the second interaction module in sequence.

[0072] For example, the fusion features of each target after position encoding by the position encoding module First, the license plate features and character features are interactively learned through the starting attention module (also known as the collaborative self-attention module) to obtain the first feature output by the starting attention module in the current preset decoding layer. The process of obtaining the first feature can refer to the following content: Figure 4aAs shown, first, the license plate query vector and character query vector belonging to the same license plate (that is, belonging to the same initial image feature) in each target fusion feature are input into the first feature interaction module for feature interaction, that is, the character feature set and the initial image feature contained in each target fusion feature input into the first interaction module are subjected to feature interaction processing, and the attention is calculated to obtain the first interaction feature output by the first feature interaction module, so that the relevant features of the characters on the license plate further enrich the features of the license plate target, and the distinction between different character features is further improved. Then, the first interaction feature output by the first feature interaction module is cross-layer added with each target fusion feature to obtain the processed first interaction feature. The specific cross-layer addition process can be to use the first interaction feature and each target fusion feature as the input of the first preset interaction module to obtain the processed first interaction feature output by the first preset interaction module. The first preset interaction module can be an Add&Norm module. The first preset interaction module is used to perform addition operations and layer normalization operations on each input feature. The license plate feature and the character feature work together, that is, the initial image feature and the character feature of each character in the character feature set corresponding to the initial image feature work together, effectively improving the attention of the multi-layer decoder to the license plate feature and the attention to each character feature on the license plate. It can be understood that the relationship between the license plate feature and each character feature on the license plate is that the license plate feature is the initial image feature, and each character feature on the license plate is the character feature of each character in the character feature set corresponding to the initial image feature. In some application scenarios, the license plate query vector and the character query vector belonging to different license plates (that is, belonging to different initial image features) in the processed first interactive feature are input into the second feature interactive module for interactive calculation of attention to obtain the second interactive feature output by the second feature interactive module, so that the relevant features of the characters on the license plate further enrich the features of the license plate target, and the distinction between different character features is further improved. Then, the second interactive feature output by the second feature interactive module is cross-layer added with the processed first interactive feature to obtain the processed second interactive feature. The specific cross-layer addition process can be to use the second interaction feature and the processed first interaction feature as the input of the second preset interaction module to obtain the processed second interaction feature output by the second preset interaction module. Calculate the attention of the overall features of different initial image features to improve the network's ability to distinguish different license plates. Finally, directly use the processed second interaction feature as the first feature output by the starting attention module in the current preset decoding layer.

[0073] It is understandable that the above-mentioned other attention modules can be the starting attention modules.

[0074] In some embodiments, each preset decoding layer includes a cross-attention module, and the above-mentioned step of cross-attention processing the first fused feature and the initial image feature to obtain the feature output by the current preset decoding layer may include the following steps: First, the second query key-value pair is used as the input of the cross-attention module in the current preset decoding layer to obtain the second feature output by the cross-attention module in the current preset decoding layer, and the query data in the second query key-value pair is obtained by the first fused feature, and the key data and value data in the second query key-value pair are respectively obtained by each initial image feature. Then, the second feature output by the cross-attention module in the current preset decoding layer is fused with the first fused feature to obtain the second fused feature. Finally, based on the second fused feature, the feature output by the current preset decoding layer is obtained.

[0075] In some application scenarios, the first fused feature is directly used as the query data in the second query key-value pair, and all the initial image features or the features to be constructed are directly used as the key data and value data in the second query key-value pair. In other application scenarios, the first fused feature is mapped to obtain the mapped first fused feature. And, all the initial image features or the features to be constructed are mapped to obtain the mapped initial image features or the mapped features to be constructed. Specifically, the mapping process can be achieved through different linear transformations. Thereby ensuring that the query data, key data and value data in the second query key-value pair are in the same vector space.

[0076] The second feature output by the cross-attention module in the current preset decoding layer is fused with the first fused feature to obtain the second fused feature, which can be obtained by cross-layer addition of the feature output by the cross-attention module and the input of the cross-attention module to obtain the processed feature output by the cross-attention module. It can be understood that the feature output by the cross-attention module can be the second feature, the input of the cross-attention module can be the first fused feature, and the fusion of the second feature and the first fused feature can be a cross-layer addition process.

[0077] See also Figure 5 , Figure 5 It is a schematic diagram of a preset decoding layer in an embodiment of the license plate recognition method of the present application.

[0078] Each preset decoding layer includes a position encoding module, a collaborative self-attention module, a third preset interaction module, a cross attention module, a fourth preset interaction module, a feedforward neural network, and a fifth preset interaction module. The first preset interaction module, the second preset interaction module, the third preset interaction module, the fourth preset interaction module, and the fifth preset interaction module are all the Add&Norm modules, and the parameters of each module may be the same or different. The Add&Norm module is used to perform addition operations and layer normalization operations on each input feature.

[0079] The process of performing cross-attention processing on the first fusion feature and the initial image feature to obtain the feature output by the current preset decoding layer may be to use the first fusion feature and the initial image feature as the input of the cross-attention module, so that the cross-attention processing is performed in the cross-attention module to obtain the feature output by the cross-attention module. Based on the feature output by the cross-attention module, the feature output by the current preset decoding layer is determined. Specifically, based on the feature output by the cross-attention module, determining the feature output by the current preset decoding layer may include the following steps: in some application scenarios, the feature output by the cross-attention module is directly used as the feature output by the current preset decoding layer. In other application scenarios, the feature output by the cross-attention module is cross-layer added with the input of the cross-attention module to obtain the feature output by the processed cross-attention module. The specific cross-layer addition processing may be to use the feature output by the cross-attention module and the input of the cross-attention module as the input of the fourth preset interaction module to obtain the feature output by the fourth preset interaction module. The feature output by the fourth preset interaction module is determined as the feature output by the current preset decoding layer. It can be understood that the feature output by the processed cross-attention module may be the above-mentioned second fusion feature. In some application scenarios, based on the second fused feature, a method for obtaining the feature output by the current preset decoding layer may be to directly determine the second fused feature as the feature output by the current preset decoding layer.

[0080] In other application scenarios, based on the second fusion feature, the method for obtaining the feature output by the current preset decoding layer can also be that the feature output by the cross-attention module can be a target cross feature. The feature output by the processed cross-attention module can be a processed target cross feature, that is, the second fusion feature. The processed target cross feature is preset processed to obtain the feature output by the current decoding layer. The specific process of performing preset processing on the processed target cross feature can be to use the processed target cross feature as the input of the feedforward neural network to obtain the feature output by the feedforward neural network. The processed target cross feature and the feature output by the feedforward neural network are used as the input of the fifth preset interaction module to obtain the feature output by the fifth preset interaction module. The feature output by the fifth preset interaction module is determined as the feature output by the current preset decoding layer. The feedforward neural network can refer to an FFN network. It can be understood that the features related to license plates and characters in the initial image features can be further extracted using the feedforward neural network and the fifth preset interaction module. The feature output by the last preset decoding layer, that is, the feature output by the multi-layer decoder can be recorded as . Wherein, B can represent the batch size, k is used to represent the number of candidate features in the features output by the multi-layer decoder, 8 represents 8 candidate character features corresponding to each initial image feature, 1 represents the candidate image feature, and C represents the number of channels of the features output by the multi-layer decoder. It can be understood that the features output by the multi-layer decoder can be candidate features corresponding to the image to be detected.

[0081] The process of performing channel splitting on the candidate features to obtain at least one image candidate feature and at least one character candidate feature can be that the feature output by the multi-layer decoder is the candidate feature corresponding to the image to be detected, and the feature output by the multi-layer decoder is Split in the third dimension to obtain at least one candidate image feature , get at least one character candidate feature B can represent the batch size, k is used to represent the number of candidate features in the features output by the multi-layer decoder, and C represents the number of channels of at least one image candidate feature or at least one character candidate feature. The 8 in the third dimension is used to represent the 8 license plate character features Based on each image candidate feature and each character candidate feature, the process of determining the license plate recognition result can be to After inputting the Linear layer of each preset decoder, at least one first result is obtained, that is, the license plate category , where the license plate type The 2 in the third dimension is used to represent the single-layer license plate and double-layer license plate in the license plate category. In some application scenarios, After inputting into the Linear layer of each preset decoder, the number of first results obtained may be K. After inputting the feedforward layer FFN of each preset decoder, at least one second result is obtained, that is, the coordinates of the center point of the license plate frame and the width and height .in, The 4 in the dimension can represent the center coordinates (x, y) and width and height (w, h) of the four-finger frame. After inputting the feedforward layer FFN of each preset decoder, at least one third result is obtained, that is, the license plate character result , where m is the total number of character categories in the license plate plus 2, where the 2 in the addition 2 refers to the blank character "#" and invalid characters. It can be understood that based on each image candidate feature and each character candidate feature, at least one first result, at least one second result and at least one third result can be obtained. That is, the number of first results is the same as the number of image candidate features.

[0082] In some application scenarios, based on each image candidate feature and each character candidate feature, determining the license plate recognition result can be to directly use the image candidate feature with the highest confidence in the third dimension in at least one first result as the target candidate feature. The second result corresponding to the target candidate feature is used as the area where the license plate is located in the license plate recognition result of the image to be detected. The third result corresponding to the target candidate feature is used as the character information in the license plate in the license plate recognition result of the image to be detected. In other application scenarios, the image to be detected is input into the license plate recognition network, and the corresponding license plate category (that is, the type of license plate), license plate frame (that is, the area where the license plate is located in the image to be detected) and character recognition result (that is, the character information in the license plate) are obtained through post-processing. Specifically, at least one first result obtained after the image to be detected is input into the license plate recognition network , at least one second result , at least one third result Perform simple post-processing. The specific steps of post-processing are as follows: The last dimension (i.e. 2 on the third dimension) performs a Softmax operation to obtain the confidence of the license plate type. The 2 on the third dimension is used for the license plate type, specifically representing a single-layer license plate and a double-layer license plate. The confidence of the license plate type includes the confidence of a single-layer license plate and the confidence of a double-layer license plate. For each license plate type confidence between the confidence of a single-layer license plate and the confidence of a double-layer license plate, the license plate type with a larger confidence is determined as the license plate type corresponding to the candidate image feature. The results whose confidence (the confidence of each candidate image feature corresponding to the license plate type) is greater than the threshold are retained and the corresponding index is recorded. At this time, at least part of the candidate image features and the candidate character features corresponding to at least part of the candidate image features are retained in the index. Obtain at least one second result according to the index. The normalized result of the license plate frame corresponding to at least part of the candidate image features in the image to be detected is obtained, and then the center coordinates and width and height of each candidate image feature on the original image are restored according to the width and height of the image to be detected; at least one third result is obtained according to the index The character recognition result corresponding to at least some of the candidate character features in the image is obtained, and the last dimension on the dimension of at least some of the candidate character features is subjected to a Softmax operation to obtain the index with the largest probability value in the last dimension. The index with the largest probability value corresponds to the category of the character, and the corresponding character result can be obtained according to the index. If the index exceeds the total number of character categories, it means a blank character or an invalid character, which is not included in the license plate number result. It can be understood that the character result corresponding to the character of the index is used as the character information in the license plate in the license plate recognition result. Thus, through the third result corresponding to the character result, the candidate image feature corresponding to the third result is determined as the target image feature. The first result corresponding to the target image feature and the corresponding second result are respectively used as the type of the license plate in the license plate recognition result and the area where the license plate is located in the image to be detected. Specifically, the type of license plate with a large confidence in the first result corresponding to the target image feature is determined as the type of license plate in the license plate recognition result. It can be considered that the present application determines the license plate recognition result based on each image candidate feature and each character candidate feature with a high accuracy. In addition, when the recognition of the character information in the license plate is relatively accurate, the present application can also better distinguish the type of license plate.

[0083] See also Figure 6 , Figure 6 This is a second flow chart of an embodiment of a license plate recognition method of the present application.

[0084] Step S61: Obtain a training data set.

[0085] Prepare a training data set containing license plate annotation boxes and license plate number annotations. The license plate annotation box is the annotation result of the area where the license plate is located in the sample image. The sample image in the training data set is a large picture containing vehicles and license plates. The label of each sample image includes the category of the license plate in the sample image (single-layer license plate, double-layer license plate), center coordinates (preset position in the license plate), width and height of the license plate, and license plate number. To facilitate the training of the license plate recognition network, the license plate number is uniformly padded to 8 digits. In the case of 7 digits in the original number, insert a blank character "#" at the end.

[0086] See also Figure 7 , Figure 7 It is a schematic diagram of a license plate recognition network in an embodiment of a license plate recognition method of the present application.

[0087] Build a license plate recognition network. The structure diagram of the license plate recognition network is as follows Figure 7 As shown in the figure, it mainly includes: backbone network, feature encoder, feature construction module, multi-layer decoder and multiple preset decoders, each of which includes linear layer Linea and feedforward layer, i.e. feedforward neural network FFN. Please refer to the above content for the functions of each module in the license plate recognition network, which will not be repeated here.

[0088] Step S62: Determine the loss function.

[0089] Determining the loss function may be constructing a loss function and using the data set to train the license plate recognition network. When training the license plate recognition network, the distribution strategy of the sample images in the training data set refers to the Hungarian algorithm used in DETR, and the loss function is calculated for the output of each preset decoding layer of the multi-layer decoder in the license plate recognition network. DETR can be an end-to-end target detection based on Transformer. The loss functions used in training the license plate recognition network include: Figure 3 The linear layer output in the feature construction module shown in Figure 1 uses the Focal Loss function. Figure 7 The first result output by each layer of the multiple preset decoders shown is the license plate category The Focal Loss function is used. Figure 3 The candidate box Bbox output by the second perceptron module in the feature construction module shown in FIG. Figure 7 The second result output by each layer of the multiple preset decoders shown is the license plate frame Use L1 Loss and GIoU Loss supervision, for example Figure 7 The third result output by each layer of the multiple preset decoders shown is the license plate character result Supervised with the standard cross entropy loss function.

[0090] Step S63: Using the training data set and the loss function, the license plate recognition network is trained to obtain a trained license plate recognition network.

[0091] Each sample image in the training data set is used as the input of the license plate recognition network to obtain the license plate prediction result corresponding to each sample image. The loss function is determined by step S62 to determine the difference between the annotation result of each sample image in the training data set and the license plate prediction result corresponding to each sample image, and the license plate recognition network is adjusted to obtain the trained license plate recognition network.

[0092] Step S64: using the image to be detected as the input of the trained license plate recognition network to obtain the license plate recognition result output by the trained license plate recognition network. It can be understood that the execution process of the above step S64 can be to use the trained license plate recognition network to implement the above steps S11 to S13.

[0093] The above scheme obtains at least one initial image feature, and then determines the character offset information about the license plate and the position information of the area of ​​interest in each initial image feature based on each initial image feature. The character offset information can enrich the overall feature information of the license plate. The combination of character offset information and position information is conducive to enriching the position information and semantic information of the character. Further, combining the character offset information and the position information of each area of ​​interest to determine the license plate recognition result of the image to be detected can improve the recognition ability of the license plate number, that is, improve the accuracy of the license plate recognition result, and in this way, the recognition process of characters in various types of license plates can be unified without the need for special processing of different types of license plates.

[0094] See also Figure 8 , Figure 8 It is a structural schematic diagram of an embodiment of the license plate recognition device of the present application. The license plate recognition device 80 includes an acquisition module 81, a first determination module 82 and a second determination module 83. The acquisition module 81 is used to acquire at least one initial image feature, and each initial image feature is extracted from the image feature to be detected collected by the license plate; the first determination module 82 is used to determine the character offset information about the license plate and the position information of the area of ​​interest in each initial image feature based on each initial image feature, and the area of ​​interest is the area where the license plate is located in the image to be detected, and each character offset information is used to represent the position relationship between at least part of the characters in the image to be detected and the preset position in the license plate; the second determination module 83 is used to determine the license plate recognition result of the image to be detected based on each character offset information and the position information of each area of ​​interest, and the license plate recognition result includes at least one of the following: the type of the license plate, the area where the license plate is located in the image to be detected, and the character information in the license plate.

[0095] The above scheme obtains at least one initial image feature, and then determines the character offset information about the license plate and the position information of the area of ​​interest in each initial image feature based on each initial image feature. The character offset information can enrich the overall feature information of the license plate. The combination of character offset information and position information is conducive to enriching the position information and semantic information of the character. Further, combining the character offset information and the position information of each area of ​​interest to determine the license plate recognition result of the image to be detected can improve the recognition ability of the license plate number, that is, improve the accuracy of the license plate recognition result, and in this way, the recognition process of characters in various types of license plates can be unified without the need for special processing of different types of license plates.

[0096] Please refer to the license plate recognition method for the functions performed by each module, which will not be repeated here.

[0097] See also Fig. 9 , Fig. 91 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 90 includes a memory 91 and a processor 92, and the processor 92 is used to execute program instructions stored in the memory 91 to implement the steps in the above license plate recognition method embodiment. In a specific implementation scenario, the electronic device 90 may include but is not limited to: a multi-camera device, a microcomputer, and a server. In addition, the electronic device 90 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0098] Specifically, the processor 92 is used to control itself and the memory 91 to implement the steps in the above-mentioned license plate recognition method embodiment. The processor 92 can also be called a CPU (Central Processing Unit). The processor 92 may be an integrated circuit chip with signal processing capabilities. The processor 92 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 92 can be implemented by an integrated circuit chip.

[0099] The above scheme obtains at least one initial image feature, and then determines the character offset information about the license plate and the position information of the area of ​​interest in each initial image feature based on each initial image feature. The character offset information can enrich the overall feature information of the license plate. The combination of character offset information and position information is conducive to enriching the position information and semantic information of the character. Further, combining the character offset information and the position information of each area of ​​interest to determine the license plate recognition result of the image to be detected can improve the recognition ability of the license plate number, that is, improve the accuracy of the license plate recognition result, and in this way, the recognition process of characters in various types of license plates can be unified without the need for special processing of different types of license plates.

[0100] See also Fig.10 , Fig.10 The computer-readable storage medium 100 stores program instructions 1001, which implement the steps in any of the above license plate recognition method embodiments when executed by a processor.

[0101] The above scheme obtains at least one initial image feature, and then determines the character offset information about the license plate and the position information of the area of ​​interest in each initial image feature based on each initial image feature. The character offset information can enrich the overall feature information of the license plate. The combination of character offset information and position information is conducive to enriching the position information and semantic information of the character. Further, combining the character offset information and the position information of each area of ​​interest to determine the license plate recognition result of the image to be detected can improve the recognition ability of the license plate number, that is, improve the accuracy of the license plate recognition result, and in this way, the recognition process of characters in various types of license plates can be unified without the need for special processing of different types of license plates.

[0102] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0103] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0104] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0105] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A license plate recognition method, characterized in that: The method comprises: Acquire at least one initial image feature, each of which is obtained by extracting features of the image to be detected collected from the license plate; Based on each of the initial image features, determine character offset information about the license plate and position information of a region of interest in each of the initial image features, the region of interest being the region where the license plate is located in the image to be detected, and each of the character offset information is used to represent a positional relationship between at least part of the characters in the image to be detected and a preset position in the license plate; Based on the character offset information and the position information of each region of interest, determining a license plate recognition result of the image to be detected, wherein the license plate recognition result includes at least one of the following: the type of the license plate, the region where the license plate is located in the image to be detected, and the character information in the license plate; The determining of the license plate recognition result of the image to be detected based on the character offset information and the position information of each region of interest includes: For each of the initial image features, using the character offset information and the position information of the region of interest associated with the initial image feature, determining a character feature set from the initial image feature, the character feature set including the character feature of each of the at least some characters; For each of the initial image features, the initial image feature is fused with each of the character features in the corresponding character feature set to obtain a target fused feature; Based on each of the initial image features and each of the target fusion features, a license plate recognition result of the image to be detected is determined.

2. The method according to claim 1, characterized in that The determining of the license plate recognition result of the image to be detected based on each of the initial image features and each of the target fusion features includes: Inputting each of the initial image features and each of the target fusion features into a multi-layer decoder for feature fusion processing to obtain candidate features corresponding to the image to be detected; Perform channel splitting on the candidate features to obtain at least one image candidate feature and at least one character candidate feature; The license plate recognition result is determined based on each of the image candidate features and each of the character candidate features.

3. The method according to claim 2, characterized in that The multi-layer decoder includes a plurality of preset decoding layers in a cascade arrangement, each of which has the same structure but different parameters, and uses the features output by the last preset decoding layer as the features output by the multi-layer decoder, and the features output by the multi-layer decoder are candidate features corresponding to the image to be detected, and each of the preset decoding layers includes a starting attention module, The step of inputting each of the initial image features and each of the target fusion features into a multi-layer decoder for feature fusion processing to obtain candidate features corresponding to the image to be detected includes: Using a first query key-value pair as an input of a starting attention module in a current preset decoding layer, obtaining a first feature output by the starting attention module in the current preset decoding layer, wherein query data, key data, and value data in the first query key-value pair are respectively obtained from each of the target fused features or from features output by a previous preset decoding layer, and the current preset decoding layer is one of the several preset decoding layers; Fusing the first feature output by the starting attention module in the current preset decoding layer with the input of the starting attention module in the current preset decoding layer to obtain a first fused feature; The first fusion feature and the initial image feature are cross-attention processed to obtain the feature output by the current preset decoding layer.

4. The method according to claim 3, characterized in that Each of the preset decoding layers includes a cross attention module, and the cross attention processing of the first fusion feature and the initial image feature to obtain the feature output by the current preset decoding layer includes: Using a second query key-value pair as an input of a cross-attention module in the current preset decoding layer, obtaining a second feature output by the cross-attention module in the current preset decoding layer, wherein query data in the second query key-value pair is obtained by the first fused feature, and key data and value data in the second query key-value pair are respectively obtained by each of the initial image features; Fusing the second feature output by the cross attention module in the current preset decoding layer with the first fused feature to obtain a second fused feature; Based on the second fusion feature, a feature output by the current preset decoding layer is obtained.

5. The method according to claim 1, characterized in that The number of the at least one initial image feature is multiple, and before determining the license plate recognition result of the image to be detected based on each of the initial image features and each of the target fusion features, the method further includes: Determining at least one feature to be constructed based on the confidence ranking corresponding to each of the initial image features, wherein the at least one feature to be constructed is at least part of the initial image features in each of the initial image features, and the confidence of each of the features to be constructed is greater than or equal to a preset confidence; The determining of the license plate recognition result of the image to be detected based on each of the initial image features and each of the target fusion features includes: Based on the target fusion features corresponding to each of the initial image features and each of the features to be constructed, a license plate recognition result of the image to be detected is determined.

6. The method according to any one of claims 1 to 5, characterized in that: The determining, based on each of the initial image features, character offset information about the license plate and position information of an area of ​​interest in each of the initial image features comprises: For each of the initial image features, using the initial image feature as an input of a first perceptron module to obtain the character offset information output by the first perceptron module; For each of the initial image features, the initial image feature is used as an input of a second perceptron module to obtain the position information of the region of interest output by the second perceptron module.

7. The method according to any one of claims 1 to 5, characterized in that The obtaining of at least one initial image feature comprises: Acquire the image to be detected; Performing multi-scale feature extraction on the image to be detected to obtain a multi-scale feature set corresponding to the image to be detected, wherein the multi-scale feature set includes at least two scale features and the scales of the scale features are different; The scale features in the multi-scale feature set are subjected to feature fusion to obtain the at least one initial image feature.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the license plate recognition method as described in any one of claims 1-7.

9. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 7.

Images