Character recognition method and device, license plate recognition method and device, computer equipment, readable storage medium and program product
By calculating the correlation score of the predicted bounding box of the license plate image and using an attention mechanism for feature extraction, the problem of inaccurate recognition caused by inaccurate segmentation in license plate recognition is solved, thereby improving the recognition accuracy and feature representation ability of license plate characters.
Patent Information
- Application Number
- CN202411029357.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-22
AI Technical Summary
The accuracy of license plate recognition in existing technologies is affected by the character segmentation effect, leading to inaccurate recognition.
By acquiring the image to be processed, identifying multiple prediction boxes and calculating relevance scores, selecting the target region, and utilizing channel attention and spatial attention mechanisms for feature extraction and recognition, the accuracy of character recognition is improved.
It improves the accuracy of character recognition, reduces false detections of target regions, enhances feature representation capabilities, and ensures that the position and content of the predicted bounding box conform to the target features.
Smart Images

Figure CN119007178A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium and computer program product for character recognition and license plate recognition. Background Art
[0002] With the advent of the mobile Internet era, intelligent devices have completely penetrated into people's daily life, gradually creating a "smart city". Among them, the smart transportation system occupies the vast majority of people's travel. For example, unattended parking lots calculate parking fees by recognizing license plate numbers and bank check processing automatically recognizes the text information on checks to speed up the check clearing process.
[0003] At present, character recognition is done by automatically identifying characters through processing the vehicle images or video sequences taken by the camera through machine vision, image processing, pattern recognition and other algorithms. For example, the specific position of the characters is first detected through the target, then the characters of the license plate are segmented using the character segmentation algorithm, and finally the license plate number is recognized using the character recognition algorithm. The result of this license plate recognition is completely dependent on the segmentation effect, which leads to the problem of inaccurate license plate recognition. Summary of the invention
[0004] Based on this, it is necessary to provide a character recognition method, device, computer equipment, computer-readable storage medium and computer program product that can improve recognition accuracy in response to the above technical problems.
[0005] In a first aspect, the present application provides a character recognition method, comprising:
[0006] Get the image to be processed;
[0007] The image to be processed is identified to obtain multiple prediction frames; the prediction frames carry geometric features and confidence levels; each prediction frame corresponds to the output feature one by one
[0008] Calculating the appearance features corresponding to each of the prediction boxes according to the output features and the confidence, and calculating the relevance score of each of the prediction boxes relative to each of the other prediction boxes according to the geometric features and the appearance features;
[0009] Obtaining a target area according to the correlation score and the plurality of prediction boxes;
[0010] Feature extraction is performed on the target area to obtain an initial feature map, feature extraction is performed on the initial feature map through a channel attention mechanism and a spatial attention mechanism to obtain a target feature map, and recognition is performed based on the target feature map to obtain a target character.
[0011] In one embodiment, obtaining a target area according to the relevance score and the plurality of prediction boxes includes:
[0012] Obtaining a target confidence of each of the prediction boxes according to the confidence and the relevance score;
[0013] The target area is obtained based on the target confidence and the multiple prediction boxes.
[0014] In one embodiment, the feature is that, according to the geometric features and the appearance features, the correlation of each prediction box with respect to each other prediction box is calculated, and based on each correlation, a correlation score corresponding to each prediction box is obtained, including:
[0015] Mapping each of the appearance features to a first dimensional space, and calculating the distance between each of the appearance features and each of the other appearance features to obtain an appearance weight;
[0016] Mapping each of the geometric features to a second dimensional space, and calculating the distance between each of the geometric features and each of the other geometric features to obtain a geometric weight, wherein the dimension of the second dimensional space is higher than the dimension of the first dimensional space;
[0017] A correlation score corresponding to each of the prediction boxes is obtained according to the appearance weight, the appearance feature, the geometric feature and the geometric weight.
[0018] In one embodiment, according to the above method, obtaining the correlation score corresponding to each of the prediction boxes according to the appearance weight, the appearance feature, the geometric feature and the geometric weight includes:
[0019] Based on the appearance weight and the appearance feature, obtain the relationship weights between each prediction box and other prediction boxes;
[0020] According to the relationship weight and the appearance weight, a relevance score corresponding to each of the prediction boxes is obtained.
[0021] In one embodiment, the calculation is performed based on the output features and the confidence level to obtain the appearance features corresponding to each of the prediction boxes, including:
[0022] Sorting according to the confidence level to obtain ranking features of each of the prediction boxes;
[0023] Mapping the output feature and the ranking feature to the same dimension;
[0024] The output feature and the ranking feature under the same dimension are added together to obtain the appearance feature.
[0025] In one embodiment, extracting features from the initial feature map by using a channel attention mechanism and a spatial attention mechanism to obtain a target feature map includes:
[0026] Performing feature extraction on the initial feature map according to the channel attention weight corresponding to the channel attention mechanism to obtain the first feature map;
[0027] Feature extraction is performed on the first feature map according to the spatial attention weight corresponding to the spatial attention mechanism to obtain the target feature map.
[0028] In one embodiment, the channel attention weight calculation process includes:
[0029] Performing global average pooling and global maximum pooling on each channel of the initial feature map to obtain a first feature vector and a second feature vector;
[0030] The first feature vector and the second feature vector are fused to obtain the channel attention weight.
[0031] In one embodiment, the calculation process of the spatial attention weight includes:
[0032] Performing global average pooling and global maximum pooling on the initial feature map in the spatial dimension respectively to obtain a third eigenvector and a fourth eigenvector;
[0033] The third eigenvector and the fourth eigenvector are fused to obtain the spatial attention weight.
[0034] In one embodiment, the above-mentioned identifying the image to be processed to obtain multiple prediction frames, calculating the correlation score based on each of the prediction frames, and obtaining the target area according to the correlation score and the multiple prediction frames is obtained by a pre-trained detection model; the training process of the detection model includes:
[0035] Obtaining a test sample; the test sample carries a category and a test area identifier;
[0036] Inputting the detection sample into the first initial model for training to obtain a predicted area;
[0037] A first loss function is obtained based on the detection area identifiers corresponding to the prediction area and different categories, and the parameters of the first initial model are adjusted according to the first loss function until the training is completed to obtain the detection model.
[0038] In one embodiment, the feature extraction of the target area is performed to obtain an initial feature map, the initial feature map is processed by a channel attention mechanism and a spatial attention mechanism to obtain a target feature map, and the target character is recognized based on the target feature map by a pre-trained recognition model; the training process of the recognition model includes:
[0039] Acquire a character sample; the character sample carries a character label;
[0040] Inputting the character sample into a second initial model for training to obtain a predicted character;
[0041] A second loss function is obtained based on the predicted character and the character label, and the parameters of the second initial model are adjusted according to the second loss function until the training is completed to obtain the recognition model; wherein the character sample is obtained through preprocessing; the preprocessing includes any one or more of corrosion, expansion, graying, binarization and adding Gaussian noise.
[0042] In a second aspect, the present application provides a charging method based on license plate recognition, comprising:
[0043] Receive a sensing signal, and control a camera to capture a target vehicle according to the sensing signal;
[0044] Acquire a license plate image to be processed, and perform character recognition on the license plate image to be processed according to any one of the methods in the above embodiments to obtain license plate characters;
[0045] Searching for the approach data of the target vehicle corresponding to the license plate characters;
[0046] If the parking fee corresponding to the entry data has been paid, the barrier gate is controlled to be lifted.
[0047] In a third aspect, the present application provides a character recognition device, the device comprising:
[0048] An acquisition module, used for acquiring an image to be processed;
[0049] A detection module, used for identifying the image to be processed to obtain multiple prediction frames; the prediction frames carry geometric features and confidence levels; each prediction frame corresponds to an output feature one by one;
[0050] A correlation calculation module, used to calculate the appearance features corresponding to each of the prediction boxes according to the output features and the confidence, and calculate the correlation score of each of the prediction boxes relative to each of the other prediction boxes according to the geometric features and the appearance features;
[0051] A target area selection module, used for obtaining a target area according to the correlation score and the plurality of prediction boxes;
[0052] The recognition module is used to extract features from the target area to obtain an initial feature map, extract features from the initial feature map through a channel attention mechanism and a spatial attention mechanism to obtain a target feature map, and recognize the target character based on the target feature map.
[0053] In one embodiment, the detection module includes:
[0054] A confidence calculation unit, used to obtain a target confidence of each of the prediction boxes according to the confidence and the correlation score;
[0055] A selection unit is used to obtain the target area based on the target confidence and the multiple prediction boxes.
[0056] In one embodiment, the confidence calculation unit includes:
[0057] an appearance weight subunit, configured to map each of the appearance features to a first dimensional space, and calculate a distance between each of the appearance features and each of the other appearance features to obtain an appearance weight;
[0058] A geometric weight subunit, used for mapping each of the geometric features to a second dimensional space, and calculating the distance between each of the geometric features and each of the other geometric features to obtain a geometric weight, wherein the dimension of the second dimensional space is higher than the dimension of the first dimensional space;
[0059] The score calculation subunit is used to obtain the correlation score corresponding to each of the prediction boxes according to the appearance weight, the appearance feature, the geometric feature and the geometric weight.
[0060] In one embodiment, the score calculation subunit includes:
[0061] An appearance related grandchild unit, used for obtaining the relationship weights between each prediction box and other prediction boxes based on the appearance weight and the appearance feature;
[0062] The grandchild unit is calculated to obtain the relevance score corresponding to each of the prediction boxes according to the relationship weight and the appearance weight.
[0063] In one embodiment, the appearance feature unit includes:
[0064] A sorting subunit, used for sorting according to the confidence level to obtain ranking features of each of the prediction boxes;
[0065] A mapping subunit, used for mapping the output feature and the confidence to the same dimension;
[0066] The feature processing subunit is used to add the output feature and the ranking feature under the same dimension to obtain the appearance feature.
[0067] In one embodiment, the identification module includes:
[0068] A channel weighting unit, configured to weight the initial feature map according to the channel attention weight to obtain the first feature map;
[0069] A spatial weighting unit is used to weight the first feature map according to the channel attention weight to obtain the target feature map.
[0070] In one embodiment, the identification module further includes a channel weight calculation unit; the channel weight calculation unit includes:
[0071] A channel feature subunit, used for performing global average pooling and global maximum pooling on each channel of the initial feature map to obtain a first feature vector and a second feature vector;
[0072] A channel feature fusion unit is used to fuse the first feature vector and the second feature vector to obtain the channel attention weight.
[0073] In one embodiment, the identification module further includes a spatial weight calculation unit; the spatial weight calculation unit includes:
[0074] A spatial feature subunit, used to perform global average pooling and global maximum pooling on the initial feature map in the spatial dimension to obtain a third feature vector and a fourth feature vector;
[0075] The spatial feature fusion unit is used to fuse the third eigenvector and the fourth eigenvector to obtain the spatial attention weight.
[0076] In one embodiment, the apparatus further comprises a first training module, wherein the first training module comprises:
[0077] A test sample acquisition unit, used to acquire a test sample; the test sample carries a category and a test identifier;
[0078] The detection training unit is used to input the detection sample into the first initial model for training, obtain the predicted area, calculate the first difference between the predicted area and the detection identifier corresponding to different categories, and adjust the parameters of the first initial model according to the first difference until the training is completed to obtain the detection model.
[0079] In one embodiment, the above device further includes a second training module; the second training module includes:
[0080] A character sample acquisition unit, used to acquire a character sample; the character sample carries a character label;
[0081] A character training unit is used to input the character sample into a second initial model for training to obtain a predicted character, calculate a second difference between the predicted character and the character table label, and adjust the parameters of the second initial model according to the second difference until the training is completed to obtain the recognition model; wherein the character sample is obtained through preprocessing; the preprocessing includes any one of corrosion, expansion, graying, binarization and adding Gaussian noise.
[0082] In a fourth aspect, the present application provides a toll collection device based on license plate recognition, comprising:
[0083] A signal receiving module is used to receive a sensing signal and control a camera to capture a target vehicle according to the sensing signal;
[0084] A license plate recognition module is used to obtain a license plate image to be processed, and recognize the license plate image to be processed according to any one of the devices to obtain license plate characters;
[0085] A search module, used to search for approach data corresponding to the target vehicle;
[0086] The control module is used to control the barrier gate to be lifted if the parking fee corresponding to the entry data has been paid.
[0087] In a fifth aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.
[0088] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.
[0089] In a seventh aspect, the present application provides a computer program product, including a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.
[0090] The above-mentioned character recognition method, device, computer equipment, computer-readable storage medium and computer program product first identify the image to be processed, obtain multiple prediction frames and calculate the correlation score of each prediction frame, and select the target area based on the correlation score, so as to reduce the false detection of the target area and improve the accuracy of detection. Among them, the correlation score is calculated based on the geometric features and appearance features. The geometric features can ensure that the position and size of the prediction frame are reasonable, and the appearance features can ensure that the content in the prediction frame meets the target features. The combination of the two can improve the accuracy of selecting the target area. In addition, the initial feature map is extracted through the channel attention mechanism and the spatial attention mechanism to obtain the target feature map, and the target character is identified based on the target feature map. In this way, the series connection of the dual attention mechanism can significantly improve the expressiveness of the features, thereby improving the recognition accuracy of the characters. BRIEF DESCRIPTION OF THE DRAWINGS
[0091] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0092] Figure 1 An application environment diagram of a character recognition method in an embodiment;
[0093] Figure 2 A flowchart of the steps of calculating a relevance score in one embodiment;
[0094] Figure 3 is a flow chart of the calculation steps of target confidence in one embodiment;
[0095] Figure 4 A schematic diagram of a flow chart of a charging method based on license plate recognition in another embodiment;
[0096] Figure 5 is a structural block diagram of a character recognition device in one embodiment;
[0097] Figure 6 is a structural block diagram of a toll collection device based on license plate recognition in one embodiment;
[0098] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0099] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0100] In one embodiment, Figure 1 As shown, a character recognition method is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:
[0101] Step S102, obtaining an image to be processed.
[0102] The image to be processed refers to an image that needs to be subsequently subjected to character recognition, and can be any image carrying characters.
[0103] For example, in a license plate recognition scenario, the images to be processed refer to images including vehicles and their license plates, which need to be processed to extract and recognize license plate information. These images to be processed can come from surveillance cameras, traffic enforcement cameras, parking lot entrance cameras, etc.
[0104] For example, in the bank bill recognition scenario, the images to be processed usually refer to images containing bank bills (such as checks, bills of exchange, deposit slips, etc.), which need to be processed to extract and recognize key information on the bills. These images to be processed can be obtained through scanners, mobile phone cameras, automatic teller machines (ATMs), and other devices.
[0105] It should be noted that the two scenarios in this embodiment are only used to illustrate the images to be processed. In other scenarios, the images to be processed may be any images carrying characters.
[0106] Step S104, the image to be processed is recognized to obtain multiple prediction boxes; the prediction boxes carry geometric features and confidence levels; each prediction box corresponds to the output feature one by one.
[0107] Among them, each prediction box usually carries geometric features and confidence. The geometric features include the coordinates of the prediction box, while the confidence indicates the probability that the prediction box contains the target object.
[0108] Optionally, multiple prediction boxes and confidences are based on the last layer output features of the model, so the prediction boxes correspond one to one with the output features. The output features include feature maps of multiple different scales, so there are multiple prediction boxes corresponding to them.
[0109] Among them, the prediction box refers to multiple prediction boxes in the image to be processed detected by object detection algorithms such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), Faster R-CNN (Faster Region-based Convolutional Neural Network), etc. These prediction boxes are areas that include key information.
[0110] The target detection box refers to the detection box obtained from the prediction box according to the correlation score, which is used as the final detection box for character recognition. Optionally, the target detection box can be one or more.
[0111] Optionally, the server identifies multiple prediction boxes from the image to be processed, and then determines one or more of the prediction boxes as the target area. In practical applications, the image detection algorithm may generate multiple overlapping or adjacent prediction boxes, which may contain some key information, but not every prediction box is equally important. By selecting the most relevant target area from multiple prediction boxes, the accuracy of recognition can be improved, ensuring that the extracted information is the most useful and reliable.
[0112] Further, the correlation of each prediction box is calculated, and the target region is obtained from the multiple prediction boxes according to the correlation. Exemplarily, the prediction box with the highest correlation score can be selected as the target region.
[0113] Step S106, calculating the appearance features corresponding to each prediction box according to the output features and the confidence, and calculating the relevance score of each prediction box relative to each other prediction box according to the geometric features and the appearance features.
[0114] Optionally, the output features and confidence can be mapped to a vector space of the same dimension, so that these two different types of information can be directly operated on in the same dimension. Through this mapping, the two types of features can be processed uniformly to better fuse their information content. Through concatenation, the feature vector contains not only the visual information of the image, but also the confidence information of the model. This fusion can enrich the feature expression, allowing the model to consider more information dimensions when making decisions. Confidence can help the model distinguish between high-probability and low-probability prediction boxes. Adding confidence information to the feature vector can improve the model's ability to distinguish when selecting the best prediction box.
[0115] Optionally, the distance between two features can be calculated by inner product and used as the correlation between the two features. For example, assuming that the correlation score of the prediction box m is to be calculated, the correlation with other measurements can be calculated first, and then weighted to obtain the corresponding correlation score.
[0116] Step S108, obtaining a target area according to the correlation score and multiple prediction boxes.
[0117] Optionally, the target confidence of each prediction box is obtained according to the confidence and relevance score; and the target area is obtained based on the target confidence and multiple prediction boxes.
[0118] Furthermore, after obtaining the relevance score, the target confidence of each prediction box is calculated by combining the relevance score and the confidence. For example, the relevance score and the confidence can be multiplied to obtain the target confidence. In this way, the existence probability of the target in the prediction box and the relevance between the prediction box and the target are comprehensively considered, so as to more accurately evaluate the effectiveness of the prediction box.
[0119] Further, the target region is obtained from multiple prediction frames according to the target confidence, and optionally, the prediction frame with the highest target confidence can be selected as the target region. In other embodiments, the prediction frame selected by the target confidence can also be selected as the target region according to actual conditions.
[0120] In this way, the geometric features can ensure that the position and size of the prediction box are reasonable, and the appearance features can ensure that the content in the prediction box meets the target features. Combining the two can improve the accuracy of the prediction box.
[0121] Step S110, extract features from the target area to obtain an initial feature map, extract features from the initial feature map through a channel attention mechanism and a spatial attention mechanism to obtain a target feature map, and recognize the target character based on the target feature map.
[0122] Among them, the initial feature map is the initial feature extracted from the target area. After further feature extraction by the channel attention mechanism and the spatial attention mechanism, the initial feature map is used as the target feature map.
[0123] Among them, the channel attention mechanism proposes important features by weighting each channel of the feature map. Exemplarily, the generated attention weight is applied to each channel of the initial feature map to highlight the features of the important channels.
[0124] The spatial attention mechanism highlights spatially important regions by weighting each position of the feature map. Exemplarily, the generated spatial attention map is applied to each position of the initial feature map to highlight spatially important features.
[0125] Optionally, after extracting features from the target area to obtain an initial feature map, feature extraction is performed on the initial feature map through a channel attention mechanism, and then feature extraction is performed again through spatial attention to obtain a target feature map. In this way, the global and local features of the feature map can be gradually optimized, which can effectively improve the accuracy of character recognition.
[0126] Optionally, after extracting features from the target area to obtain an initial feature map, feature extraction is performed on the initial feature map through a spatial attention mechanism, and then feature extraction is performed through channel attention to obtain a target feature map. In this way, spatial information in the image can be processed first, key spatial areas can be highlighted, and then channel weighting can be performed on the optimized feature map to further enhance global features.
[0127] Optionally, after obtaining the target feature map, the target feature map is flattened into a feature vector so as to be input into the fully connected layer, which maps the feature vector to a predefined character category. Finally, the character category is converted into a probability distribution through the Softmax function (a normalization function), and the category with the largest change is selected as the recognition result, which is the target character.
[0128] In the above embodiment, the image to be processed is first identified to obtain multiple prediction frames and calculate the correlation scores of each prediction frame, and the target area is selected based on the correlation score, which can reduce the false detection of the target area and improve the accuracy of detection. In addition, the initial feature map is extracted through the channel attention mechanism and the spatial attention mechanism to obtain the target feature map, and the target character is identified based on the target feature map. In this way, the series connection of the dual attention mechanism can significantly improve the expressiveness of the features, thereby improving the recognition accuracy of the characters.
[0129] In the above embodiment, the geometric features can ensure that the position and size of the prediction box are reasonable, and the appearance features can ensure that the content in the prediction box meets the target features. Combining the two, the accuracy of the prediction box can be improved.
[0130] Furthermore, in one embodiment, the above-mentioned calculation of the correlation of each prediction box relative to each other prediction box based on the geometric features and appearance features, and obtaining the correlation score corresponding to each prediction box based on the correlations includes: mapping each appearance feature to a first dimensional space, and calculating the distance between each appearance feature and each other appearance feature to obtain an appearance weight; mapping each geometric feature to a second dimensional space, and calculating the distance between each geometric feature and each other geometric feature to obtain a geometric weight, wherein the dimension of the second dimensional space is higher than the dimension of the first dimensional space; obtaining the correlation score corresponding to each prediction box according to the appearance weight, the appearance features, the geometric features and the geometric weight.
[0131] Optionally, the appearance features corresponding to each prediction box can be mapped to the first dimension space, and then the distance of the appearance features corresponding to each prediction box relative to other prediction boxes is calculated by inner product, that is, the correlation between the two features. Finally, all distances are weighted and summed as the appearance weight.
[0132] Optionally, the geometric features corresponding to each prediction box can be mapped to the first dimensional space, and then the relative features of the geometric features corresponding to each prediction box relative to other prediction boxes, that is, the correlation between the two features, are calculated. Finally, all distances are weighted and summed as the geometric weight.
[0133] Finally, according to the appearance weight, appearance features, geometric features and geometric weight, the relevance score corresponding to each prediction box is obtained.
[0134] Since different objects may have similar appearance features but different geometric features, combining these two types of features can reduce the possibility of misidentifying the background or other objects as target objects, thereby more accurately identifying the detection box in the image to be processed.
[0135] Furthermore, in one embodiment, the above-mentioned obtaining of the correlation scores corresponding to the prediction boxes based on the appearance weights, appearance features, geometric features and geometric weights includes: obtaining the relationship weights of each prediction box with other prediction boxes based on the appearance weights and geometric weights; and obtaining the correlation scores corresponding to the prediction boxes based on the relationship weights and appearance weights.
[0136] Exemplary, combined Figure 2 As shown, Figure 2 The following is a flow chart of the steps for calculating the relevance score in one embodiment.
[0137] Combined with formula (1), the appearance weight is calculated:
[0138]
[0139] Among them, W k and W Q Represents a matrix that transforms the appearance features of m and n objects into F A Mapping to low-dimensional space d k , and then use the inner product dot to measure the similarity of the appearance features of the two objects. The m, n objects are the prediction boxes m and n.
[0140] Exemplarily, combined with formula (2), the geometric weight is obtained:
[0141]
[0142] Among them, W G represents the linear parameter, εG The function maps the geometric features of two objects into a high-dimensional space to ensure translation and scale invariance. w, h, x, y are the width, height, and center point of the target, respectively.
[0143]
[0144] After obtaining the appearance weight and geometric weight corresponding to each prediction box and each other prediction box, the relationship weight of each prediction box and each other prediction box is calculated according to the appearance weight and the geometric weight.
[0145] Exemplarily, according to formula (4), the relationship weight is obtained:
[0146]
[0147] Finally, the relationship score can be calculated according to formula (5):
[0148]
[0149] Among them, F R (n) is the entire data set, that is, the relationship features of each prediction box relative to the nth prediction box. Optionally, the relationship features are normalized to obtain the relationship score in this embodiment.
[0150] In this way, by calculating the relationship of each predicted box with respect to other predicted boxes, the context of the scene can be better understood, thereby improving the accuracy of the prediction. Moreover, in scenes containing multiple objects, this method can help the model distinguish and identify different objects, even if they may be similar in appearance.
[0151] In one embodiment, the above calculation is performed based on the output features and the confidence level to obtain the appearance features corresponding to each prediction box, including: sorting according to the confidence level to obtain the ranking features of each prediction box; mapping the output features and the ranking features to the same dimension; adding the output features and the ranking features under the same dimension to obtain the appearance features.
[0152] For example, a rank embed operation is first performed on the classification confidence. Specifically, the position coordinates are assigned eigenvalues [1, n] in descending order according to the classification confidence, that is, the eigenvalue of the coordinate corresponding to the largest classification confidence is 1, and the eigenvalue of the coordinate corresponding to the second largest classification confidence is 2, and a scalar with a sequence length of 1*n is obtained, which is the ranking feature.
[0153] The purpose of performing rank embedding on classification confidence is to sort the confidence of the prediction boxes and give each prediction box a feature value (ranking feature) representing its ranking. This operation can enhance the ability to distinguish between prediction boxes, so that prediction boxes with high confidence receive more attention in subsequent processing.
[0154] Since the scales of ranking features and geometric features are inconsistent, it is necessary to map the geometric features and ranking features to the same dimension. fR Mapped to 128-d dimension, and the geometric features are passed through W f Dimensionality reduction to 128-d dimension. In this embodiment, the ranking features and the output features are mapped to a 128-d dimensional space, and in other embodiments, a 256-d dimensional space may also be used.
[0155] Exemplary, combined Figure 3 As shown, Figure 3 The figure is a flow chart of the calculation steps of target confidence in one embodiment.
[0156] In the above embodiment, the output features and geometric features are combined to calculate the appearance features, which can comprehensively utilize various information, improve the feature expression ability, and enhance the feature differentiation.
[0157] After getting the output feature F n , confidence S n and coordinate position B n After that, the specific operations are as follows:
[0158] 1. First, perform a rank embed operation on the classification confidence. Specifically, the position coordinates are assigned eigenvalues [1, n] in descending order according to the classification confidence, that is, the eigenvalue of the coordinate with the largest classification confidence is 1, and the eigenvalue of the coordinate with the second largest classification confidence is 2, and a scalar with a sequence length of 1*n is obtained.
[0159] 2. Pass the scalar through W fR Mapped to 128-d dimension, and F n The dimension is reduced to 128-d by Wf, and the two are added together as the appearance features of the target relation module.
[0160] 3. The coordinates are used as geometric features and input into the target relationship module together with the appearance features. The target relationship module calculates the relevance score according to the method described in any of the above embodiments.
[0161] 4. Convert the target relation module input into a scalar through Ws and sigmoid normalize it to [0,1]. Multiply the output S1 by the initial S0 to get the final classification confidence. That is, after normalizing the correlation features, get the correlation score, and finally multiply the correlation score by the confidence to get the target confidence.
[0162] In one embodiment, the above-mentioned feature extraction of the initial feature map through the channel attention mechanism and the spatial attention mechanism to obtain the target feature map includes: extracting features from the initial feature map according to the channel attention weights corresponding to the channel attention mechanism to obtain the first feature map; extracting features from the first feature map according to the spatial attention weights corresponding to the spatial attention mechanism to obtain the target feature map.
[0163] In this embodiment, the initial feature map is firstly extracted by the channel attention weight corresponding to the channel attention mechanism to obtain the first feature map, and then the spatial attention weight corresponding to the spatial attention mechanism is used to extract the features of the first feature map to obtain the final target feature map.
[0164] In this way, using the dual attention mechanism in series can improve the recognition accuracy.
[0165] Furthermore, in one embodiment, the above-mentioned channel attention weight calculation process includes: performing global average pooling and global maximum pooling on each channel of the initial feature map respectively to obtain a first feature vector and a second feature vector; and fusing the first feature vector and the second feature vector to obtain a channel attention weight.
[0166] In this embodiment, the channel attention weight is calculated from the channel dimension. The channel dimension refers to the number of channels in the feature map, which indicates the number of features at each position of the feature map. Each channel can be regarded as a separate feature map, corresponding to different feature representations.
[0167] The first vector feature and the second vector feature are respectively features obtained by global average pooling and global maximum pooling of each channel of the initial feature map.
[0168] Among them, global average pooling is a pooling operation used to reduce each channel of the feature map to a single value through average pooling. Since global average pooling is a commonly used technical means in the field, it will not be described in detail here.
[0169] Among them, global maximum pooling is a pooling operation, which is used to reduce each channel of the feature map to a single value through maximum pooling. Since global maximum pooling is a commonly used technical means in the field, it will not be described in detail here.
[0170] In other embodiments, other pooling methods may be used to extract the initial feature map to obtain the first feature vector and the second feature vector.
[0171] For example, assume that the size of the initial feature map is (8, 32, 64, 64), where B = 8 is the batch size (the number of images used each time), C = 32 is the number of channels, H = 64 and W = 64 are the height and width. From the channel dimension, the initial feature map can be regarded as a 64x64 image with 32 channels, each channel representing a different feature.
[0172] Optionally, the first feature vector and the second feature vector are fused. There are many fusion methods, such as concatenating or adding the two vectors, or fusing them through a fully connected layer. A comprehensive feature vector is obtained through the fusion operation.
[0173] Furthermore, the fused feature vector, that is, the comprehensive feature vector, is passed through one or more fully connected layers to generate the attention weights of each channel. These weights are usually normalized by an activation function (such as Sigmoid or Softmax) to ensure that the weights are between 0 and 1, thereby indicating the importance of each channel. In this way, the channel attention weights are obtained.
[0174] Exemplarily, the following is a channel attention calculation process in another embodiment. The specific operation is to first perform global average pooling and maximum pooling of the input features in a space to obtain two 1*1*C channels, then pass the two channels through a fully connected layer to generate a set of attention weights, and adjust the feature map of each channel.
[0175] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) Formula (6)
[0176] Among them, σ is the activation function and MLP is a shared multi-layer perceptron network.
[0177] Furthermore, in one embodiment, the calculation process of the above-mentioned spatial attention weight includes: performing global average pooling and global maximum pooling on the initial feature map in the spatial dimension respectively to obtain a third eigenvector and a fourth eigenvector; and fusing the third eigenvector and the fourth eigenvector to obtain the spatial attention weight.
[0178] In this embodiment, the channel attention weight is calculated from the spatial dimension. The spatial dimension refers to the height and width of the feature map, that is, the spatial size of the feature map. It represents the two-dimensional spatial position of the feature map on each channel.
[0179] The first vector feature and the second vector feature are respectively features obtained by performing global average pooling and global maximum pooling on the spatial dimension of the initial feature map.
[0180] Exemplarily, continuing with the examples in the above embodiments, from the perspective of spatial dimension, the initial feature map may have a spatial dimension of 64x64 in each channel.
[0181] The specific process of obtaining spatial attention can refer to the calculation process of channel attention, which will not be described in detail in this embodiment.
[0182] Exemplarily, the following is a spatial attention calculation process in another embodiment. The specific operation is to maximize and average the feature map in different dimensional spaces, and then use a convolution layer to generate a set of attention weights, apply these weights to the original feature map, and enhance the area of important spatial information.
[0183] M s (F) = σ(f 7*7 ([AvgPool(F);MaxPool(F)])) Formula (7)
[0184] Among them, f 7*7 It is a 7*7 convolution operation.
[0185] In this embodiment, by combining global average pooling and global maximum pooling, both global characteristics and local salient features can be taken into account, thereby generating a more comprehensive channel attention weight.
[0186] In one embodiment, the above-mentioned identification of the image to be processed obtains multiple prediction boxes, calculates the correlation score based on each prediction box, and obtains the target area based on the correlation score and the multiple prediction boxes through a pre-trained detection model; the training process of the detection model includes: obtaining detection samples; the detection samples carry categories and detection area identifiers; inputting the detection samples into the first initial model for training to obtain prediction areas; obtaining a first loss function based on the prediction areas and detection area identifiers corresponding to different categories, and adjusting the parameters of the first initial model according to the first loss function until the training is completed to obtain the detection model.
[0187] Among them, the detection sample refers to the training data used to train the detection model, which can be set according to the specific usage scenario; the detection area mark refers to the area of the object that has been marked on the image.
[0188] Among them, the first initial model can be any model that has not been trained and can perform object detection. For example, the R-CNN (Region-based Convolutional Neural Networks) series, the YOLO (You Only Look Once) series.
[0189] Among them, the prediction region refers to the result of the first initial model's recognition of the detection sample during the training process.
[0190] Exemplarily, if the detection model is used to detect license plates, then the detection samples can be images of different vehicles. The categories are different vehicle models, and the detection region identifiers are the positions of the license plates on different vehicle models.
[0191] Exemplarily, the dataset can be in the following form:
[0192] Category 1: Large civilian vehicle (yellow background with black characters, black border line)
[0193] Category 2: Small civilian vehicle (blue background with white characters, white border line)
[0194] Category 3: Special vehicle for armed police (white background with black characters, red "police" character, black border line)
[0195] Category 4: Vehicles entering and leaving Hong Kong and Macau (black background with white characters, white "Hong" and "Ao", white border line)
[0196] Category 5: Foreign vehicles of embassies and consulates (black background with white characters, red "embassy" and "consulate" characters, white border line)
[0197] Category 6: Training vehicle (yellow background with black characters, black "learning" character, black border line)
[0198] Category 7: Small new energy vehicle (gradually changing white and green background with black characters)
[0199] Category 8: Large new energy vehicle (yellow and green background with black characters)
[0200] Category 9: Vehicle
[0201] Exemplarily, the detection samples are input into the first initial model for training. The first initial model undergoes the detection process described in any of the above embodiments to obtain the prediction region. Then, the first loss function between the prediction region and the detection region identifier corresponding to different categories is calculated, and the parameters of the first initial model are adjusted according to the first loss function until the training is completed to obtain the detection model.
[0202] In this way, category labels are added during training to prevent the same license plate number for different models. For example, military vehicles usually have a different license plate management system from civilian vehicles, and different types of military vehicles may use the same number. For example, some special-purpose vehicles (such as fire trucks, police cars) may have special license plate numbering systems in some countries or regions, allowing different types of vehicles to use the same number but distinguished by different prefixes or colors. For example, temporary license plates (such as temporary mobile permits) can be used on different vehicles for a short period of time, especially for vehicle transportation or temporary test drives.
[0203] In one embodiment, the above-mentioned feature extraction of the target area obtains an initial feature map, the initial feature map is processed by a channel attention mechanism and a spatial attention mechanism to obtain a target feature map, and the target character is recognized based on the target feature map through a pre-trained recognition model; the training process of the recognition model includes: obtaining character samples; the character samples carry character labels; the character samples are input into a second initial model for training to obtain predicted characters; a second loss function is obtained based on the predicted characters and the character labels, and the parameters of the second initial model are adjusted according to the second loss function until the training is completed to obtain a recognition model; wherein, the character samples are obtained through preprocessing; the preprocessing includes any one or more of corrosion, expansion, graying, binarization and adding Gaussian noise.
[0204] The character sample refers to the training data used to train the recognition model. The character sample can be extracted from the image dataset or generated by the generator. Each character sample should correspond to a character label.
[0205] The second initial model can be any model that has not been trained and can perform character recognition. For example, CNN (Convolutional Neural Network), OCR (Optical Character Recognition System) and LPRNet (License Plate Recognition Network). It should be noted that LPRNet is a lightweight license plate recognition model suitable for scenarios where license plates are used.
[0206] The predicted character is only the result obtained by the second initial model in the training process to recognize the character sample, and it is not a real character.
[0207] Exemplarily, the character samples are input into the second initial model for training, and the second initial model obtains predicted characters based on the recognition process described in any of the above embodiments. Then, a second loss function is obtained based on the predicted characters and the character labels, and the parameters of the second initial model are adjusted according to the second loss function until the training is completed to obtain a recognition model.
[0208] Among them, the character samples are obtained through preprocessing, which includes any one or more of corrosion, expansion, grayscale, binarization and adding Gaussian noise to increase the recognition of occluded and worn characters on the license plate, thereby improving the recognition accuracy of the recognition model for occluded and worn license plates.
[0209] Among them, erosion shrinks the boundaries of objects in the image inward and removes pixels on the boundaries.
[0210] Among them, dilation is the inverse operation of erosion, which is used to increase the boundary area of the foreground object in the image. The dilation operation will expand the boundary of the foreground object outward.
[0211] Grayscaling refers to the process of converting a color image into a grayscale image. In a grayscale image, the color of each pixel is represented by a grayscale level, usually ranging from 0 (black) to 255 (white).
[0212] Among them, binarization is the process of converting a grayscale image into a binary image, that is, setting pixels with values greater than a certain threshold to one value (usually 255, indicating white), and setting pixels less than the threshold to another value (usually 0, indicating black).
[0213] Among them, Gaussian noise is a type of noise that obeys Gaussian distribution.
[0214] In the above embodiment, in order to increase the recognition of occluded and worn characters on the license plate, the standardized characters are automatically processed by corrosion, expansion, grayscale and binarization using morphological algorithms, and Gaussian noise is added in an appropriate amount to construct characters with occlusions. The above two types of data are filled with the original character samples to train a recognition model that can accurately recognize the characters.
[0215] In an exemplary embodiment, Figure 4 As shown, a charging method based on license plate recognition is provided, including the following steps 402 to 406. Among them:
[0216] Step S402, receiving a sensing signal, and controlling a camera to capture a target vehicle according to the sensing signal.
[0217] The induction signal refers to the signal sent by the sensor when the induction device detects that the vehicle enters or exits the parking lot. The induction device may be an induction coil, an infrared sensor, a radar sensor, a geomagnetic sensor, etc.
[0218] Exemplarily, when a vehicle passes through an entrance, a sensing device detects the vehicle and sends a sensing signal to a server. After the server receives the sensing signal, it controls a camera to capture a photo of the target vehicle.
[0219] Step S404, obtaining a license plate image to be processed, and performing character recognition on the license plate image to be processed according to the character recognition method in any of the above embodiments to obtain license plate characters.
[0220] After the camera captures the target vehicle, it obtains the license plate image to be processed, and recognizes the license plate image to obtain the license plate characters and the vehicle model. The recognition of the license plate characters can refer to the character recognition method in any of the above embodiments, and will not be repeated here.
[0221] Because the above recognition and character recognition method increases the detection of different models, prevents the wrong charges caused by the same license plate of different models, and also calculates the correlation of each prediction frame relative to other prediction frames, it can increase the detection of different models and prevent the wrong charges caused by the same license plate of different models. In addition, in the character recognition process, due to the spatial attention mechanism and channel attention mechanism, it can prevent misrecognition caused by similar license plate characters.
[0222] Step S406, searching for the target vehicle's approach data corresponding to the license plate characters.
[0223] The entry data refers to the data stored when the vehicle enters the parking lot. Exemplarily, when the vehicle enters the parking lot, the license plate of the target vehicle is identified according to the method described in steps S402 and S404 to obtain the license plate characters and vehicle model, and the identified license plate characters, vehicle model and time of entering the parking lot are stored in the database as the entry data. Optionally, after obtaining the license plate characters and vehicle model, the license plate characters are displayed on the display screen, the barrier for entering the parking lot is raised, and after the vehicle safely enters the parking lot, the barrier is controlled to be lowered.
[0224] When the target vehicle leaves the parking lot, the server recognizes the license plate characters and vehicle model, searches for the corresponding entry data based on the license plate characters, and calculates the parking fee to be paid based on the time of entry into the parking lot.
[0225] Step S408: If the parking fee corresponding to the entry data has been paid, the barrier gate is controlled to be lifted.
[0226] If the server detects that the parking fee corresponding to the entry data has been paid, the gate is controlled to be lifted so that the target vehicle can drive out of the parking lot.
[0227] In the above embodiment, since the character recognition model can accurately recognize the license plate characters, the automation and accuracy of the parking system will be greatly improved.
[0228] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0229] Based on the same inventive concept, the embodiment of the present application also provides a character recognition device for implementing the character recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more character recognition device embodiments provided below can refer to the limitations on the character recognition method above, and will not be repeated here.
[0230] In an exemplary embodiment, Figure 5 As shown, a character recognition device is provided, including: an acquisition module 510, a detection module 520, a correlation calculation module 530, a target area selection module 540 and a recognition module 550, wherein:
[0231] The acquisition module 510 is used to acquire the image to be processed.
[0232] The detection module 520 is used to identify the image to be processed to obtain multiple prediction boxes; the prediction boxes carry geometric features and confidence levels; and each prediction box corresponds to the output feature one by one.
[0233] The correlation calculation module 530 is used to calculate the appearance features corresponding to each prediction box according to the output features and the confidence, and calculate the correlation score of each prediction box relative to each other prediction box according to the geometric features and the appearance features.
[0234] The target region selection module 540 is used to obtain the target region according to the correlation score and the multiple prediction boxes.
[0235] The recognition module 550 is used to extract features from the target area to obtain an initial feature map, extract features from the initial feature map through a channel attention mechanism and a spatial attention mechanism to obtain a target feature map, and recognize the target character based on the target feature map.
[0236] In one embodiment, the detection module includes:
[0237] A confidence calculation unit, used to obtain a target confidence of each of the prediction boxes according to the confidence and the correlation score;
[0238] A selection unit is used to obtain the target area based on the target confidence and the multiple prediction boxes.
[0239] In one embodiment, the confidence calculation unit includes:
[0240] an appearance weight subunit, configured to map each of the appearance features to a first dimensional space, and calculate a distance between each of the appearance features and each of the other appearance features to obtain an appearance weight;
[0241] A geometric weight subunit, used for mapping each of the geometric features to a second dimensional space, and calculating the distance between each of the geometric features and each of the other geometric features to obtain a geometric weight, wherein the dimension of the second dimensional space is higher than the dimension of the first dimensional space;
[0242] The score calculation subunit is used to obtain the correlation score corresponding to each of the prediction boxes according to the appearance weight, the appearance feature, the geometric feature and the geometric weight.
[0243] In one embodiment, the score calculation subunit includes:
[0244] An appearance related grandchild unit, used for obtaining the relationship weights between each prediction box and other prediction boxes based on the appearance weight and the appearance feature;
[0245] The grandchild unit is calculated to obtain the relevance score corresponding to each of the prediction boxes according to the relationship weight and the appearance weight.
[0246] In one embodiment, the appearance feature unit includes:
[0247] A sorting subunit, used for sorting according to the confidence level to obtain ranking features of each of the prediction boxes;
[0248] A mapping subunit, used for mapping the output feature and the confidence to the same dimension;
[0249] The feature processing subunit is used to add the output feature and the ranking feature under the same dimension to obtain the appearance feature.
[0250] In one embodiment, the identification module includes:
[0251] A channel weighting unit, configured to weight the initial feature map according to the channel attention weight to obtain the first feature map;
[0252] A spatial weighting unit is used to weight the first feature map according to the channel attention weight to obtain the target feature map.
[0253] In one embodiment, the identification module further includes a channel weight calculation unit; the channel weight calculation unit includes:
[0254] A channel feature subunit, used for performing global average pooling and global maximum pooling on each channel of the initial feature map to obtain a first feature vector and a second feature vector;
[0255] A channel feature fusion unit is used to fuse the first feature vector and the second feature vector to obtain the channel attention weight.
[0256] In one embodiment, the identification module further includes a spatial weight calculation unit; the spatial weight calculation unit includes:
[0257] A spatial feature subunit, used to perform global average pooling and global maximum pooling on the initial feature map in the spatial dimension to obtain a third feature vector and a fourth feature vector;
[0258] The spatial feature fusion unit is used to fuse the third eigenvector and the fourth eigenvector to obtain the spatial attention weight.
[0259] In one embodiment, the apparatus further comprises a first training module, wherein the first training module comprises:
[0260] A test sample acquisition unit, used to acquire a test sample; the test sample carries a category and a test identifier;
[0261] The detection training unit is used to input the detection sample into the first initial model for training, obtain the predicted area, calculate the first difference between the predicted area and the detection identifier corresponding to different categories, and adjust the parameters of the first initial model according to the first difference until the training is completed to obtain the detection model.
[0262] In one embodiment, the above device further includes a second training module; the second training module includes:
[0263] A character sample acquisition unit, used to acquire a character sample; the character sample carries a character label;
[0264] A character training unit is used to input the character sample into a second initial model for training to obtain a predicted character, calculate a second difference between the predicted character and the character table label, and adjust the parameters of the second initial model according to the second difference until the training is completed to obtain the recognition model; wherein the character sample is obtained through preprocessing; the preprocessing includes any one of corrosion, expansion, graying, binarization and adding Gaussian noise.
[0265] Based on the same inventive concept, the embodiment of the present application also provides a toll collection device based on license plate recognition for implementing the toll collection method based on license plate recognition involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more toll collection device embodiments based on license plate recognition provided below can refer to the limitations of the toll collection method based on license plate recognition above, and will not be repeated here.
[0266] In an exemplary embodiment, Figure 6 As shown, a charging device based on license plate recognition is provided, including: a signal receiving module 610, a license plate recognition module 620, a search module 630, a fare calculation module 640 and a control module 650, wherein:
[0267] The signal receiving module 610 is used to receive the sensing signal and control the camera to capture the target vehicle according to the sensing signal.
[0268] The license plate recognition module 620 is used to obtain the license plate image to be processed, and recognize the license plate image to be processed according to any one of the devices to obtain the license plate characters.
[0269] The search module 630 is used to search for approach data corresponding to the target vehicle from a database according to the license plate characters.
[0270] The control module 640 is used to control the barrier gate to be lifted if the parking fee corresponding to the entry data has been paid.
[0271] Each module in the above-mentioned character recognition device and the toll collection device based on license plate recognition can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.
[0272] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data to be processed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a character recognition and a license plate recognition method are implemented.
[0273] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0274] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.
[0275] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.
[0276] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.
[0277] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0278] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0279] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A character recognition method, characterized in that: The method comprises: Get the image to be processed; Identify the image to be processed to obtain multiple prediction frames; the prediction frames carry geometric features and confidence levels; each prediction frame corresponds to an output feature one by one; Calculating the appearance features corresponding to each of the prediction boxes according to the output features and the confidence, and calculating the relevance score of each of the prediction boxes relative to each of the other prediction boxes according to the geometric features and the appearance features; Obtaining a target area according to the correlation score and the plurality of prediction boxes; Feature extraction is performed on the target area to obtain an initial feature map, feature extraction is performed on the initial feature map through a channel attention mechanism and a spatial attention mechanism to obtain a target feature map, and recognition is performed based on the target feature map to obtain a target character.
2. The method according to claim 1, characterized in that Obtaining a target area according to the relevance score and the plurality of prediction boxes includes: Obtaining a target confidence of each of the prediction boxes according to the confidence and the relevance score; The target area is obtained based on the target confidence and the multiple prediction boxes.
3. The method according to claim 1, characterized in that: The calculating, according to the geometric features and the appearance features, a relevance score of each prediction box relative to each other prediction box comprises: Mapping each of the appearance features to a first dimensional space, and calculating the distance between each of the appearance features and each of the other appearance features to obtain an appearance weight; Mapping each of the geometric features to a second dimensional space, and calculating the distance between each of the geometric features and each of the other geometric features to obtain a geometric weight, wherein the dimension of the second dimensional space is higher than the dimension of the first dimensional space; A correlation score corresponding to each of the prediction boxes is obtained according to the appearance weight, the appearance feature, the geometric feature and the geometric weight.
4. The method according to claim 3, characterized in that The obtaining, according to the appearance weight, the appearance feature, the geometric feature and the geometric weight, a correlation score corresponding to each of the prediction boxes comprises: Based on the appearance weight and the appearance feature, obtaining a relationship weight between each prediction box and each other prediction box; According to the relationship weight and the appearance weight, a relevance score corresponding to each of the prediction boxes is obtained.
5. The method according to claim 2, characterized in that: The calculating according to the output features and the confidence level to obtain the appearance features corresponding to each of the prediction boxes includes: Sorting according to the confidence level to obtain ranking features of each of the prediction boxes; Mapping the output feature and the ranking feature to the same dimension; The output feature and the ranking feature under the same dimension are added together to obtain the appearance feature.
6. The method according to claim 1, characterized in that The initial feature map is subjected to feature extraction through a channel attention mechanism and a spatial attention mechanism to obtain a target feature map, including: Performing feature extraction on the initial feature map according to the channel attention weight corresponding to the channel attention mechanism to obtain the first feature map; Feature extraction is performed on the first feature map according to the spatial attention weight corresponding to the spatial attention mechanism to obtain the target feature map.
7. The method according to claim 6, characterized in that The channel attention weight calculation process includes: Performing global average pooling and global maximum pooling on each channel of the initial feature map to obtain a first feature vector and a second feature vector; The first feature vector and the second feature vector are fused to obtain the channel attention weight.
8. The method according to claim 6, characterized in that The calculation process of the spatial attention weight includes: Performing global average pooling and global maximum pooling on the initial feature map in the spatial dimension respectively to obtain a third eigenvector and a fourth eigenvector; The third eigenvector and the fourth eigenvector are fused to obtain the spatial attention weight.
9. The method according to claim 1, characterized in that: The identifying of the image to be processed to obtain a plurality of prediction frames, calculating a correlation score based on each of the prediction frames, and obtaining a target area according to the correlation score and the plurality of prediction frames is obtained through a pre-trained detection model; The training process of the detection model includes: Obtaining a test sample; the test sample carries a category and a test area identifier; Inputting the detection sample into the first initial model for training to obtain a predicted area; A first loss function is obtained based on the prediction area and the detection area identifiers corresponding to different categories, and the parameters of the first initial model are adjusted according to the first loss function until the training is completed to obtain the detection model.
10. The method according to claim 1, characterized in that The feature extraction of the target area is performed to obtain an initial feature map, the initial feature map is processed by a channel attention mechanism and a spatial attention mechanism to obtain a target feature map, and the target character is recognized based on the target feature map by a pre-trained recognition model; The training process of the recognition model includes: Acquire a character sample; the character sample carries a character label; Inputting the character sample into a second initial model for training to obtain a predicted character; A second loss function is obtained based on the predicted character and the character label, and the parameters of the second initial model are adjusted according to the second loss function until the training is completed to obtain the recognition model; wherein the character sample is obtained through preprocessing; the preprocessing includes any one or more of corrosion, expansion, graying, binarization and adding Gaussian noise.
11. A charging method based on license plate recognition, characterized in that: The method comprises: Receive a sensing signal, and control a camera to capture a target vehicle according to the sensing signal; Acquire a license plate image to be processed, and perform character recognition on the license plate image to be processed according to any one of the methods described in claims 1 to 10 to obtain license plate characters; Searching for the approach data of the target vehicle corresponding to the license plate characters; If the parking fee corresponding to the entry data has been paid, the barrier gate is controlled to be lifted.
12. A character recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed; A detection module, used for identifying the image to be processed to obtain multiple prediction frames; the prediction frames carry geometric features and confidence levels; each prediction frame corresponds to an output feature one by one; A correlation calculation module, used to calculate the appearance features corresponding to each of the prediction boxes according to the output features and the confidence level, and calculate the correlation score of each of the prediction boxes relative to each of the other prediction boxes according to the geometric features and the appearance features; A target area selection module, used for obtaining a target area according to the correlation score and the plurality of prediction boxes; The recognition module is used to extract features from the target area to obtain an initial feature map, extract features from the initial feature map through a channel attention mechanism and a spatial attention mechanism to obtain a target feature map, and recognize the target character based on the target feature map.
13. The device according to claim 12, characterized in that The detection module includes: A confidence calculation unit, used to obtain a target confidence of each of the prediction boxes according to the confidence and the correlation score; A selection unit is used to obtain the target area based on the target confidence and the multiple prediction boxes.
14. The device according to claim 13, characterized in that The confidence calculation unit comprises: an appearance weight subunit, configured to map each of the appearance features to a first dimensional space, and calculate a distance between each of the appearance features and each of the other appearance features to obtain an appearance weight; A geometric weight subunit, used for mapping each of the geometric features to a second dimensional space, and calculating the distance between each of the geometric features and each of the other geometric features to obtain a geometric weight, wherein the dimension of the second dimensional space is higher than the dimension of the first dimensional space; The score calculation subunit is used to obtain the correlation score corresponding to each of the prediction boxes according to the appearance weight, the appearance feature, the geometric feature and the geometric weight.
15. The device according to claim 14, characterized in that The score calculation subunit includes: An appearance related grandchild unit, used for obtaining the relationship weights between each prediction box and other prediction boxes based on the appearance weight and the appearance feature; The grandchild unit is calculated to obtain the relevance score corresponding to each of the prediction boxes according to the relationship weight and the appearance weight.
16. The device according to claim 13, characterized in that The appearance feature unit comprises: A sorting subunit, used for sorting according to the confidence level to obtain ranking features of each of the prediction boxes; A mapping subunit, used for mapping the output feature and the confidence to the same dimension; The feature processing subunit is used to add the output feature and the ranking feature under the same dimension to obtain the appearance feature.
17. The device according to claim 12, characterized in that The identification module comprises: A channel weighting unit, configured to weight the initial feature map according to the channel attention weight to obtain the first feature map; A spatial weighting unit is used to weight the first feature map according to the channel attention weight to obtain the target feature map.
18. The device according to claim 17, characterized in that The identification module further includes a channel weight calculation unit; the channel weight calculation unit includes: A channel feature subunit, used for performing global average pooling and global maximum pooling on each channel of the initial feature map to obtain a first feature vector and a second feature vector; A channel feature fusion unit is used to fuse the first feature vector and the second feature vector to obtain the channel attention weight.
19. The device according to claim 17, characterized in that The recognition module further includes a spatial weight calculation unit; the spatial weight calculation unit includes: A spatial feature subunit, used to perform global average pooling and global maximum pooling on the initial feature map in the spatial dimension to obtain a third feature vector and a fourth feature vector; The spatial feature fusion unit is used to fuse the third eigenvector and the fourth eigenvector to obtain the spatial attention weight.
20. The device according to claim 12, characterized in that The device also includes a first training module, wherein the first training module includes: A test sample acquisition unit, used to acquire a test sample; the test sample carries a category and a test identifier; The detection training unit is used to input the detection sample into the first initial model for training, obtain the predicted area, calculate the first difference between the predicted area and the detection identifier corresponding to different categories, and adjust the parameters of the first initial model according to the first difference until the training is completed to obtain the detection model.
21. The device according to claim 12, characterized in that The device also includes a second training module; the second training module includes: A character sample acquisition unit, used to acquire a character sample; the character sample carries a character label; A character training unit is used to input the character sample into a second initial model for training to obtain a predicted character, calculate a second difference between the predicted character and the character table label, and adjust the parameters of the second initial model according to the second difference until the training is completed to obtain the recognition model; wherein the character sample is obtained through preprocessing; the preprocessing includes any one of corrosion, expansion, graying, binarization and adding Gaussian noise.
22. A toll collection device based on license plate recognition, characterized in that: The device comprises: A signal receiving module is used to receive a sensing signal and control a camera to capture a target vehicle according to the sensing signal; A license plate recognition module, used for acquiring a license plate image to be processed, and recognizing the license plate image to be processed according to any one of the devices in claims 12-20 to obtain license plate characters; A search module, used to search for approach data corresponding to the target vehicle; The control module is used to control the barrier gate to be lifted if the parking fee corresponding to the entry data has been paid.
23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 or 10 are implemented.
24. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 or 11 are implemented.
25. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 or 11 are implemented.
Citation Information
Patent Citations
License plate character recognition method and device, equipment and storage medium
CN115485746A
End-to-end Laos license plate recognition method and device fused with double-layer attention network
CN116597437A
Parking lot charging management system enabling license plate automatic recognition
CN203242037U
