License plate character segmentation method, device and storage medium
By using an improved CenterNet network and a lightweight backbone network, the shortcomings of traditional license plate character segmentation methods in terms of versatility and accuracy are addressed. This enables efficient segmentation of license plates of different styles, improves recognition accuracy, and reduces computational load.
Patent Information
- Application Number
- CN202111449870.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-30
AI Technical Summary
Existing license plate character segmentation methods are insufficient in terms of versatility and segmentation accuracy, especially for license plates of different styles and low quality.
An improved CenterNet network is used for license plate character segmentation. By adding a contour heatmap prediction branch, a loss function is constructed for supervised learning. Combined with a lightweight backbone network such as MobileNetV1-0.25, the prediction accuracy of character center points and contours is improved, adapting to different styles of license plates.
The method improves the versatility and robustness of license plate character segmentation, enhances segmentation accuracy, and reduces computational load, making it applicable to various terminal devices.
Smart Images

Figure CN114120306B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of license plate recognition, and in particular relates to a license plate character segmentation method, device and storage medium. Background Art
[0002] License plate recognition is the core technology of intelligent transportation. The license plate recognition system mainly consists of four parts: license plate positioning, license plate correction, license plate character segmentation and license plate character recognition. Character segmentation refers to the accurate segmentation of each single character in a corrected license plate image. It is a key link in license plate recognition technology. The accuracy of character segmentation directly affects the final license plate recognition rate.
[0003] With the development of license plate recognition technology, character segmentation technology has flourished. Currently, one of the more commonly used character segmentation methods is based on vertical projection. This method obtains the edge position of each character by obtaining the vertical projection curve of the license plate characters based on the peak and trough positions of the curve. The advantages of this method are simple algorithm, fast speed, and better segmentation effect for clear license plates. However, using this segmentation method, different parameters need to be designed for different styles of license plates, and the license plate type needs to be identified in advance before segmentation, which has extremely poor versatility. At the same time, for some low-quality license plates that are stained, adhered, or have inaccurate character positioning, the segmentation effect will be significantly reduced or even fail.
[0004] Therefore, how to provide an effective solution to ensure the versatility and segmentation accuracy of character segmentation has become a difficult problem to be solved urgently in the prior art. Summary of the invention
[0005] The purpose of the present invention is to provide a license plate character segmentation method, device and storage medium to solve the problems of poor versatility and segmentation accuracy in existing license plate segmentation methods.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] In a first aspect, the present invention provides a license plate character segmentation method, comprising:
[0008] Get the license plate image to be recognized;
[0009] Input the license plate image to be identified into the license plate positioning model for positioning detection to obtain the coordinates of the four license plate corner points corresponding to the license plate image to be identified;
[0010] Based on the coordinates of the four license plate corner points corresponding to the license plate image to be identified, the angle of the license plate image to be identified is corrected to obtain a corrected license plate image;
[0011] Performing character segmentation on the corrected license plate image based on an improved CenterNet network to obtain a character region in the corrected license plate image;
[0012] The improved CenterNet network includes a backbone network for feature extraction and a branch prediction network, wherein the branch prediction network includes a contour heat map prediction branch, a character center point prediction branch, a center point offset prediction branch, and a character length and width prediction branch, wherein the contour heat map prediction branch is used to predict the character center point contour, the character center point prediction branch is used to predict the character center point coordinates, the center point offset prediction branch is used to predict the center point offset of the character center point, and the character length and width prediction branch is used to predict the length and width of the character area;
[0013] The improved CenterNet network is obtained by supervised learning using a loss function constructed using the contour error term between the character center point contour predicted by the contour heat map prediction branch and the actual center point contour of the character, the center point error term between the character center point coordinates predicted by the character center point prediction branch and the actual center point coordinates of the character, the center point offset predicted by the center point offset prediction branch, and the length and width error term between the length and width of the character area predicted by the character length and width prediction branch and the actual length and width of the character area as the overall model loss.
[0014] Based on the above disclosed content, the present invention constructs a loss function through the character center point offset predicted by the branch prediction network as the overall loss supervised learning of the improved CenterNet network, and uses the improved CenterNet network to perform character segmentation on the license plate image, that is, the improved CenterNet network can segment the license plate characters with the character center point as the target point, and can segment the characters of license plates of different styles, so that it can be applied to various styles of license plates, and has good versatility and robustness. At the same time, since the contour heat map prediction branch is added to the branch prediction network of the CenterNet network, the center point contour of the license plate character is constrained, so that the network better matches the characteristics of the license plate, thereby further improving the segmentation accuracy of the license plate characters, and then improving the license plate recognition accuracy.
[0015] In a possible design, the character center point contour predicted by the contour heat map prediction branch is:
[0016]
[0017] The contour error term between the character center point contour predicted by the contour heat map prediction branch and the actual center point contour of the character is:
[0018]
[0019] in, Represents the pixel points and line segments in the contour heat map of the corrected license plate image The distance between them, σ is a hyperparameter, and the line segment is the line segment between the center point of the ith character and the center point of the i+1th character, w and h represent the width and height of the contour heat map of the corrected license plate image, respectively. Represents the actual center point outline of the character.
[0020] Through the above design, the present invention can accurately predict the contour error term between the center point contour of the character and the actual center point contour of the character, thereby accurately constraining the center point contour of the license plate character, thereby improving the segmentation accuracy of the license plate character.
[0021] In one possible design, the loss function is:
[0022] L det =L k +λ contour L contour +λ size L size +λ off L off
[0023] Among them, L k represents the center point error term, L contour represents the contour error term, λ contour Represents the weight corresponding to the contour error term, L size represents the length and width error term, λ size Indicates the weight corresponding to the length and width error term, L off represents the center point offset, λ off Indicates the weight corresponding to the center point offset.
[0024] Through the above design, the present invention can set corresponding weights according to different types of error terms to construct the overall loss of the improved CenterNet network. While taking into account different types of error terms, it also takes into account the influence of various error terms on the overall loss, thereby further improving the accuracy of character segmentation.
[0025] In a possible design, the weight corresponding to the contour error term, the weight corresponding to the center point offset, and the weight corresponding to the length and width error term are all 0.1.
[0026] In one possible design, the method further includes:
[0027] Segmenting the character region in the corrected license plate image to obtain a plurality of character regions;
[0028] Character recognition is performed on the multiple character areas, and the character recognition results of the multiple character areas are combined according to the positions of the multiple character areas to obtain the license plate number corresponding to the license plate image to be recognized.
[0029] In one possible design, the backbone network is a MobileNetV1-0.25 network, a MobileNet V2 network, or a MobileNet V3 network.
[0030] Through the above design, the backbone network is set as a lightweight network, which can greatly reduce the amount of model calculation with minimal loss of accuracy, so that the license plate character segmentation method can be widely used in various terminal devices.
[0031] In a possible design, the contour heat map prediction branch includes a first convolution layer, a second convolution layer and a maximum pooling layer, the first convolution layer is used to perform a convolution operation on the first image feature extracted by the backbone network to obtain a second image feature, the second convolution layer is used to perform a secondary convolution operation on the second image feature to obtain a third image feature, the maximum pooling layer is used to perform a maximum pooling operation on the third image feature to obtain the character center point contour, the convolution kernel of the first convolution layer is 3*3, the convolution kernel of the second convolution layer is 1*1, the convolution kernel of the maximum pooling layer is 3*3, the number of input channels of the first convolution layer is 256, the number of output channels of the first convolution layer is 256, the number of input channels of the second convolution layer is 256, and the number of output channels of the second convolution layer is 1.
[0032] Through the above design, by using 1*1 convolution, the number of channels is compressed, so that the amount of calculation can be reduced when the contour error term between the character center point contour predicted by the contour heat map prediction branch and the actual center point contour of the character is reduced, thereby improving the segmentation accuracy of the license plate characters while ensuring the segmentation efficiency of the license plate characters.
[0033] In a second aspect, the present invention provides a license plate character segmentation device, comprising:
[0034] An acquisition unit, used for acquiring a license plate image to be recognized;
[0035] A positioning unit, used for inputting the license plate image to be identified into a license plate positioning model for positioning detection, and obtaining the coordinates of four license plate corner points corresponding to the license plate image to be identified;
[0036] A correction unit, used for correcting the angle of the license plate image to be identified based on the coordinates of the four license plate corner points corresponding to the license plate image to be identified, so as to obtain a corrected license plate image;
[0037] A segmentation unit, used for performing character segmentation on the corrected license plate image based on an improved CenterNet network to obtain a character region in the corrected license plate image;
[0038] The improved CenterNet network includes a backbone network for feature extraction and a branch prediction network, wherein the branch prediction network includes a contour heat map prediction branch, a character center point prediction branch, a center point offset prediction branch, and a character length and width prediction branch, wherein the contour heat map prediction branch is used to predict the character center point contour, the character center point prediction branch is used to predict the character center point coordinates, the center point offset prediction branch is used to predict the center point offset of the character center point, and the character length and width prediction branch is used to predict the length and width of the character area;
[0039] The improved CenterNet network is obtained by supervised learning using a loss function constructed using the contour error term between the character center point contour predicted by the contour heat map prediction branch and the actual center point contour of the character, the center point error term between the character center point coordinates predicted by the character center point prediction branch and the actual center point coordinates of the character, the center point offset predicted by the center point offset prediction branch, and the length and width error term between the length and width of the character area predicted by the character length and width prediction branch and the actual length and width of the character area as the overall model loss.
[0040] In the third aspect, the present invention provides another license plate character segmentation device, comprising: a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute a license plate character segmentation method such as the first aspect or any one of the possible designs in the first aspect.
[0041] In a fourth aspect, the present invention provides a storage medium having instructions stored thereon, which, when executed on a computer, executes a license plate character segmentation method as in the first aspect or any possible design of the first aspect.
[0042] In a fifth aspect, the present invention provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute a license plate character segmentation method as in the first aspect or any one of the possible designs in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A flow chart of the license plate character segmentation method provided by the present invention;
[0044] Figure 2 A schematic diagram of the structure of the improved CenterNet network provided by the present invention;
[0045] Figure 3 A schematic diagram of the outline of the center point of the characters marked on the license plate provided by the present invention;
[0046] Figure 4 A schematic diagram of the structure of the license plate character segmentation device provided by the present invention;
[0047] Figure 5 This is a schematic structural diagram of another license plate character segmentation device provided by the present invention. DETAILED DESCRIPTION
[0048] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be noted that although the description of these embodiments is used to help understand the present invention, it does not constitute a limitation of the present invention. The specific structures and functional details disclosed herein are only used to describe the exemplary embodiments of the present invention. However, the present invention can be embodied in many alternative forms, and it should not be understood that the present invention is limited to the embodiments set forth herein.
[0049] It should be understood that although the terms first, second, etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another unit. For example, a first unit can be referred to as a second unit, and similarly, a second unit can be referred to as a first unit without departing from the scope of the exemplary embodiments of the present invention.
[0050] It should be understood that the term "and / or" that may appear in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this article describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B can represent two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this article generally indicates that the previous and next associated objects are in an "or" relationship.
[0051] Example
[0052] To ensure the universality and segmentation accuracy of character segmentation, an embodiment of the present application provides a license plate character segmentation method, device and storage medium, which have good universality and segmentation accuracy, and good robustness.
[0053] The license plate character segmentation method provided in the embodiment of the present application is applied to a user terminal or a server. The user terminal may be, but is not limited to, a personal computer, a smart phone, a tablet computer, a personal digital assistant (PDA), etc.
[0054] It can be understood that the execution entity does not constitute a limitation on the embodiments of the present application.
[0055] The license plate character segmentation method provided in the embodiment of the present application will be described in detail below.
[0056] like Figure 1 As shown, the first aspect of the embodiment of the present application provides a license plate character segmentation method, which may include the following steps:
[0057] Step S101, obtaining the license plate image to be recognized.
[0058] In the embodiment of the present application, the license plate image to be identified may be a license plate image taken when entering or exiting a parking lot and / or a license plate image taken when entering or exiting a highway toll station, or it may be one or more frames of images extracted from a license plate video, or it may be an image uploaded by a user containing the license plate to be identified, and no specific limitation is made in the embodiment of the present application.
[0059] Step S102: input the license plate image to be identified into the license plate positioning model for positioning detection to obtain the coordinates of the four license plate corner points corresponding to the license plate image to be identified.
[0060] In the embodiment of the present application, a license plate positioning model for identifying the coordinates of the corner points of a license plate is pre-established. The license plate positioning model can be a model trained by taking the license plate image as input and the coordinates of the four corner points corresponding to the license plate image as output. After obtaining the license plate image to be identified, the obtained license plate image to be identified can be used as the input of the license plate positioning model for operation to obtain the coordinates of the four corner points of the license plate corresponding to the license plate image to be identified.
[0061] Among them, the license plate positioning model can be, but is not limited to, a Retinaface detection network, a MobileNet network, etc., and is not specifically limited in the embodiments of the present application.
[0062] Step S103: Based on the coordinates of the four license plate corner points corresponding to the license plate image to be recognized, the angle of the license plate image to be recognized is corrected to obtain a corrected license plate image.
[0063] Specifically, when correcting the license plate image, the license plate can be corrected to a frontal view image using perspective transformation based on the coordinates of the four license plate corner points corresponding to the license plate image to obtain a corrected license plate image.
[0064] Perspective transformation refers to the transformation that utilizes the condition that the perspective center, image point, and target point are collinear, and according to the perspective rotation law, the image receiving surface (perspective surface) is rotated by a certain angle around the trace line (perspective axis), destroying the original projection light beam, while still keeping the projection geometry on the image receiving surface unchanged. Perspective transformation is an existing technology, and the specific principle will not be repeated in the embodiments of this application.
[0065] Step S104, performing character segmentation on the corrected license plate image based on the improved CenterNet network to obtain the character area in the corrected license plate image.
[0066] See also Figure 2 In the embodiment of the present application, an improvement is made on the basis of the original CenterNet network, and a contour heat map prediction branch is added to the branch prediction network of the original CenterNet network, that is, the improved CenterNet network includes a backbone network and a branch prediction network for feature extraction, and the branch prediction network includes a contour heat map prediction branch, a character center point prediction branch, a center point offset prediction branch, and a character length and width prediction branch. The contour heat map prediction branch is used to predict the contour of the character center point, the character center point prediction branch is used to predict the character center point coordinates, the center point offset prediction branch is used to predict the center point offset of the character center point, and the character length and width prediction branch is used to predict the length and width of the character area. The improved CenterNet network is obtained by supervised learning using a loss function constructed with the contour error term between the character center point contour predicted by the contour heat map prediction branch and the actual center point contour of the character, the center point error term between the character center point coordinates predicted by the character center point prediction branch and the actual center point coordinates of the character, the center point offset predicted by the center point offset prediction branch, and the length and width error term between the length and width of the character area predicted by the character length and width prediction branch and the actual length and width of the character area as the overall model loss.
[0067] Specifically, the backbone network can use, but is not limited to, lightweight networks such as MobileNetV1-0.25 network, MobileNet V2 network or MobileNet V3 network. The use of lightweight backbone networks can reduce the amount of computation with minimal loss of accuracy, so as to be deployed on terminals with different computing capabilities. In the embodiment of the present application, the backbone network uses the MobileNetV1-0.25 network.
[0068] Since the aspect ratios of license plates are mainly 3.14 (440mm*140mm), 3.43 (480mm*140mm), and 2.0 (440mm*220mm), in order to avoid excessive deformation of characters due to scaling and affecting accuracy, the resolution of the improved CenterNet network input image is changed from the original 512*512 to 320*120. Its aspect ratio is closer to the actual aspect ratio of the license plate, so it will not affect the accuracy of character segmentation due to excessive deformation caused by scaling.
[0069] The contour heat map prediction branch includes a first convolution layer, a second convolution layer and a maximum pooling layer. The first convolution layer is used to perform a convolution operation on the first image feature extracted by the backbone network to obtain a second image feature. The second convolution layer is used to perform a second convolution operation on the second image feature to obtain a third image feature. The maximum pooling layer is used to perform a maximum pooling operation on the third image feature to obtain the contour of the center point of the character. The convolution kernel of the first convolution layer is 3*3, the convolution kernel of the second convolution layer is 1*1, the convolution kernel of the maximum pooling layer is 3*3, the number of input channels of the first convolution layer is 256, the number of output channels of the first convolution layer is 256, the number of input channels of the second convolution layer is 256, and the number of output channels of the second convolution layer is 1. Among them, the 1*1 convolution does not need to consider the relationship between pixels and surrounding pixels. It can linearly combine pixels on different channels and then perform nonlinear operations to complete the function of dimensionality reduction (that is, while reducing the number of channels, the width and height of the image are not changed, thereby achieving the purpose of reducing the amount of calculation). After the first convolution layer, the second convolution layer and the maximum pooling layer, the output character center point outline is a single-channel image feature with a size of 80*30.
[0070] The character center point prediction branch, the center point offset prediction branch, and the character length and width prediction branch are existing branches in the CenterNet network before improvement, and will not be described in detail in the embodiments of the present application.
[0071] See also Figure 3 , is a schematic diagram of a license plate, from Figure 3 It can be seen that the license plate characters have a characteristic, that is, the centers of the characters are on the same straight line (double-layer license plates are also on 2 lines), which is different from the scattered target positions in other scenes. In other words, the center points of the characters are all on this line. Therefore, in order to improve the accuracy of character segmentation and thus improve the accuracy of character detection, this application constrains the character contours through the newly added contour heat map prediction branch in the branch prediction network to improve the accuracy of character segmentation.
[0072] by Figure 3 As an example of the single-layer license plate shown in the figure, the center point of the license plate characters, from left to right, the two adjacent center points k i and ki+1 The line segment between The pixels between are 1, and the rest are 0, generating a contour heat map of the same size as the license plate image. The contour heat map can be expressed as:
[0073]
[0074] Among them, S contour represents the contour heat map, and p represents S contour The pixel point in, L represents the line segment However, with this representation, the width of the contour is only 1 pixel, 0,1 binary representation, which is not conducive to learning. Therefore, in the actual application of deep learning, Gaussian function is usually used to generate heat map, that is, point p is in the line segment If the distance is within 3σ, the response is generated by Gaussian function, then S contour It is a contour heat map from 0 to 1, and the distance line segment The closer the point is, the larger the value it corresponds to. Therefore, in the embodiment of the present application, the character center point contour predicted by the contour heat map prediction branch can be expressed as:
[0075]
[0076] in, Represents the pixel points and line segments in the contour heat map of the corrected license plate image The distance between them, σ is a hyperparameter, and the line segment is the line segment between the center point of the i-th character and the center point of the i+1-th character.
[0077] The contour error term between the character center point contour predicted by the contour heat map prediction branch and the actual center point contour of the character can be expressed as:
[0078]
[0079] Among them, w and h represent the width and height of the contour heat map of the rectified license plate image, respectively. Represents the actual center point outline of the character.
[0080] After determining the contour error term between the character center point contour and the actual center point contour of the character, the constructed loss function can be expressed as:
[0081] L det =L k +λ contour L contour +λ size L size +λ off L off
[0082] Among them, L k represents the center point error term, L contour represents the contour error term, λ contour Represents the weight corresponding to the contour error term, L size represents the length and width error term, λ size Indicates the weight corresponding to the length and width error term, L off represents the center point offset, λ off Indicates the weight corresponding to the center point offset.
[0083] In the embodiment of the present application, the weight λ corresponding to the contour error term contour , the weight λ corresponding to the center point offset size And the weight λ corresponding to the length and width error term off The value of is 0.1.
[0084] Furthermore, in an embodiment of the present application, after character segmentation is performed on the corrected license plate image based on the improved CenterNet network to obtain the character area in the corrected license plate image, the character area in the corrected license plate image can also be segmented to obtain multiple character areas. Character recognition is then performed on the multiple character areas, and the character recognition results of the multiple character areas are combined according to the positions of the multiple character areas to obtain the license plate number corresponding to the license plate image to be recognized. Since the accuracy of license plate character segmentation can be improved by the improved CenterNet network, the accuracy of license plate recognition can also be improved during license plate recognition.
[0085] In summary, the license plate character segmentation method provided by the present invention constructs a loss function through the character center point offset predicted by the branch prediction network as the overall loss supervised learning of the improved CenterNet network, and uses the improved CenterNet network to perform character segmentation on the license plate image, that is, the improved CenterNet network can segment the license plate characters with the character center point as the target point, and can segment the characters of license plates of different styles, so that it can be applied to license plates of various styles, and has good versatility and robustness. At the same time, since the contour heat map prediction branch is added to the branch prediction network of the CenterNet network, the center point contour of the license plate character is constrained, so that the network is more matched with the license plate characteristics, thereby further improving the segmentation accuracy of the license plate characters, and then improving the license plate recognition accuracy. Secondly, the improved CenterNet network adopts a lightweight MobileNetV1-0.25 network, which can reduce the amount of calculation with minimal loss of accuracy, so that it can be deployed on terminals with different computing powers, so that it can be widely used in different terminal devices. In addition, the image input size of the improved CenterNet network is changed from 320*120 to the original 512*512, avoiding the distortion caused by large-scale scaling of the license plate and affecting the accuracy of character segmentation.
[0086] like Figure 4 As shown, the second aspect of the embodiment of the present application provides a license plate character segmentation device, the license plate character segmentation device comprising:
[0087] An acquisition unit, used for acquiring a license plate image to be recognized;
[0088] A positioning unit, used for inputting the license plate image to be identified into a license plate positioning model for positioning detection, and obtaining the coordinates of four license plate corner points corresponding to the license plate image to be identified;
[0089] A correction unit, used for correcting the angle of the license plate image to be identified based on the coordinates of the four license plate corner points corresponding to the license plate image to be identified, so as to obtain a corrected license plate image;
[0090] A segmentation unit, used for performing character segmentation on the corrected license plate image based on an improved CenterNet network to obtain a character region in the corrected license plate image;
[0091] The improved CenterNet network includes a backbone network for feature extraction and a branch prediction network, wherein the branch prediction network includes a contour heat map prediction branch, a character center point prediction branch, a center point offset prediction branch, and a character length and width prediction branch, wherein the contour heat map prediction branch is used to predict the character center point contour, the character center point prediction branch is used to predict the character center point coordinates, the center point offset prediction branch is used to predict the center point offset of the character center point, and the character length and width prediction branch is used to predict the length and width of the character area;
[0092] The improved CenterNet network is obtained by supervised learning using a loss function constructed using the contour error term between the character center point contour predicted by the contour heat map prediction branch and the actual center point contour of the character, the center point error term between the character center point coordinates predicted by the character center point prediction branch and the actual center point coordinates of the character, the center point offset predicted by the center point offset prediction branch, and the length and width error term between the length and width of the character area predicted by the character length and width prediction branch and the actual length and width of the character area as the overall model loss.
[0093] The working process, working details and technical effects of the hardware device provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.
[0094] like Figure 5 As shown, the third aspect of an embodiment of the present application provides another license plate character segmentation device. Taking the device as a computer main device as an example, the license plate character segmentation device includes: a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the license plate character segmentation method as described in the first aspect of the embodiment.
[0095] For example, the memory may include, but is not limited to, random access memory (RAM), read only memory (ROM), flash memory, first input first output (FIFO) and / or first in last out (FILO), etc. Specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). At the same time, the processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. For example, the processor may not be limited to a microprocessor of the STM32F105 series, a reduced instruction set computer (RSIC) microprocessor, an X86 architecture processor, or a processor with an integrated embedded neural network processing unit (NPU); the transceiver may be, but is not limited to, a wireless fidelity (WIFI) wireless transceiver, a Bluetooth wireless transceiver, a general packet radio service technology (General Packet Radio Service, GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver and / or a 5G transceiver, etc. In addition, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.
[0096] The working process, working details and technical effects of the license plate character segmentation device provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.
[0097] The fourth aspect of this embodiment provides a storage medium storing instructions for the license plate character segmentation method described in the first aspect of the embodiment, that is, the storage medium stores instructions, and when the instructions are run on a computer, the license plate character segmentation method described in the first aspect is executed.
[0098] The storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, a CD, a hard disk, a flash memory, a USB flash drive and / or a memory stick, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
[0099] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.
[0100] The fifth aspect of this embodiment provides a computer program product containing instructions, which, when executed on a computer, enables the computer to execute the license plate character segmentation method as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
[0101] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A license plate character segmentation method, characterized in that: include: Get the license plate image to be recognized; Input the license plate image to be identified into the license plate positioning model for positioning detection to obtain the coordinates of the four license plate corner points corresponding to the license plate image to be identified; Based on the coordinates of the four license plate corner points corresponding to the license plate image to be identified, the angle of the license plate image to be identified is corrected to obtain a corrected license plate image; Performing character segmentation on the corrected license plate image based on an improved CenterNet network to obtain a character region in the corrected license plate image; The improved CenterNet network includes a backbone network for feature extraction and a branch prediction network, wherein the branch prediction network includes a contour heat map prediction branch, a character center point prediction branch, a center point offset prediction branch, and a character length and width prediction branch, wherein the contour heat map prediction branch is used to predict the character center point contour, the character center point prediction branch is used to predict the character center point coordinates, the center point offset prediction branch is used to predict the center point offset of the character center point, and the character length and width prediction branch is used to predict the length and width of the character area; The improved CenterNet network is obtained by supervised learning using a loss function constructed using the contour error term between the character center point contour predicted by the contour heat map prediction branch and the actual center point contour of the character, the center point error term between the character center point coordinates predicted by the character center point prediction branch and the actual center point coordinates of the character, the center point offset predicted by the center point offset prediction branch, and the length and width error term between the length and width of the character area predicted by the character length and width prediction branch and the actual length and width of the character area as the overall model loss.
2. The method according to claim 1, characterized in that The character center point contour predicted by the contour heat map prediction branch is: The contour error term between the character center point contour predicted by the contour heat map prediction branch and the actual center point contour of the character is: in, Represents the pixel points and line segments in the contour heat map of the corrected license plate image The distance between them, σ is a hyperparameter, and the line segment is the line segment between the center point of the i-th character and the center point of the i+1-th character, w and h represent the width and height of the contour heat map of the corrected license plate image, respectively. Represents the actual center point outline of the character.
3. The method according to claim 2, characterized in that The loss function is: L det =L k +λ contour L contour +λ size L size +λ off L off Among them, L k represents the center point error term, L contour represents the contour error term, λ contour Represents the weight corresponding to the contour error term, L size represents the length and width error term, λ size Indicates the weight corresponding to the length and width error term, L off represents the center point offset, λ off Indicates the weight corresponding to the center point offset.
4. The method according to claim 3, characterized in that The weight corresponding to the contour error term, the weight corresponding to the center point offset, and the weight corresponding to the length and width error term are all 0.
1.
5. The method according to claim 1, characterized in that The method further comprises: Segmenting the character region in the corrected license plate image to obtain a plurality of character regions; Character recognition is performed on the multiple character areas, and the character recognition results of the multiple character areas are combined according to the positions of the multiple character areas to obtain the license plate number corresponding to the license plate image to be recognized.
6. The method according to claim 1, characterized in that The backbone network is a MobileNetV1-0.25 network, a MobileNet V2 network or a MobileNet V3 network.
7. The method according to claim 1, characterized in that The contour heat map prediction branch includes a first convolution layer, a second convolution layer and a maximum pooling layer. The first convolution layer is used to perform a convolution operation on the first image feature extracted by the backbone network to obtain a second image feature. The second convolution layer is used to perform a secondary convolution operation on the second image feature to obtain a third image feature. The maximum pooling layer is used to perform a maximum pooling operation on the third image feature to obtain the character center point contour. The convolution kernel of the first convolution layer is 3*3, the convolution kernel of the second convolution layer is 1*1, and the convolution kernel of the maximum pooling layer is 3*3. The number of input channels of the first convolution layer is 256, and the number of output channels of the first convolution layer is 256. The number of input channels of the second convolution layer is 256, and the number of output channels of the second convolution layer is 1.
8. A license plate character segmentation device, characterized in that: include: An acquisition unit, used for acquiring a license plate image to be recognized; A positioning unit, used for inputting the license plate image to be identified into a license plate positioning model for positioning detection, and obtaining the coordinates of four license plate corner points corresponding to the license plate image to be identified; A correction unit, used for correcting the angle of the license plate image to be identified based on the coordinates of the four license plate corner points corresponding to the license plate image to be identified, so as to obtain a corrected license plate image; A segmentation unit, used for performing character segmentation on the corrected license plate image based on an improved CenterNet network to obtain a character region in the corrected license plate image; The improved CenterNet network includes a backbone network for feature extraction and a branch prediction network, wherein the branch prediction network includes a contour heat map prediction branch, a character center point prediction branch, a center point offset prediction branch, and a character length and width prediction branch, wherein the contour heat map prediction branch is used to predict the character center point contour, the character center point prediction branch is used to predict the character center point coordinates, the center point offset prediction branch is used to predict the center point offset of the character center point, and the character length and width prediction branch is used to predict the length and width of the character area; The improved CenterNet network is obtained by supervised learning using a loss function constructed using the contour error term between the character center point contour predicted by the contour heat map prediction branch and the actual center point contour of the character, the center point error term between the character center point coordinates predicted by the character center point prediction branch and the actual center point coordinates of the character, the center point offset predicted by the center point offset prediction branch, and the length and width error term between the length and width of the character area predicted by the character length and width prediction branch and the actual length and width of the character area as the overall model loss.
9. A license plate character segmentation device, characterized in that: include: A memory, a processor and a transceiver which are sequentially connected in communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the license plate character segmentation method as described in any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions are run on a computer, the license plate character segmentation method according to any one of claims 1 to 7 is executed.
