License plate image processing method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a license plate image processing method, device, electronic device and storage medium. The method includes: determining a first license plate image and a second license plate image, the first license plate image is an image obtained by capturing a vehicle to be detected when the fill light is turned on, and the second license plate image is an image obtained by capturing the vehicle to be detected when the fill light is turned off; fusing the first license plate image and the second license plate image to obtain a third license plate image; extracting high-frequency information from the first license plate image, the second license plate image and the third license plate image and fusing them to obtain reference high-frequency information; extracting low-frequency information from the reference license plate image to obtain reference low-frequency information, and fusing the reference high-frequency information and the reference low-frequency information to obtain a fourth license plate image. This solution can realize accurate recognition of license plates and improve the adaptability of license plate recognition in different scenarios.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a license plate image processing method, device, electronic device and storage medium. Background Art

[0002] With the widespread application of intelligent monitoring technology in various fields, especially in the field of traffic management, high-resolution traffic cameras, as an important part of the intelligent transportation system, have been deployed at many intersections and sections. These cameras play an important role in ensuring public safety and maintaining traffic order.

[0003] In practical applications, ambient light conditions have a significant impact on the image acquisition quality of the camera, especially at night or in low light conditions. In order to ensure the acquisition of clear images and accurate identification of vehicles and related information, fill lights are usually used to enhance lighting. This fill light measure greatly improves the overall image quality, especially the image quality inside the car, so that details such as people and objects inside the car can be captured more clearly.

[0004] However, license plates are usually made of reflective materials, which have a high reflective coefficient. Under the illumination of fill light, the license plate will reflect a large amount of light, causing the area where the license plate is located to be easily overexposed, making the text information inside the license plate difficult to clearly recognize, thereby seriously reducing the accuracy of the license plate recognition algorithm. It will not only affect the efficiency and accuracy of traffic violation investigations, but may also cause a series of problems in some key scenarios, such as vehicle tracking and toll management. Summary of the invention

[0005] The present invention provides a license plate image processing method, device, electronic equipment and storage medium, which realize accurate recognition of license plates, not only improve the accuracy of license plate recognition, but also improve the adaptability of license plate recognition in different scenarios.

[0006] In a first aspect, an embodiment of the present invention provides a license plate image processing method, the method comprising:

[0007] Determine a first license plate image and a second license plate image, wherein the first license plate image is a license plate image obtained by capturing an image of the vehicle to be detected when the fill light is turned on, and the second license plate image is a license plate image obtained by capturing the vehicle to be detected when the fill light is turned off;

[0008] Performing license plate image fusion based on the first license plate image and the second license plate image to obtain a third license plate image;

[0009] Extracting high-frequency information from the first license plate image, the second license plate image, and the third license plate image and fusing them to obtain reference high-frequency information;

[0010] Low-frequency information is extracted from the reference license plate image to obtain reference low-frequency information, and a fourth license plate image is obtained based on the fusion of the reference high-frequency information and the reference low-frequency information. The reference license plate image is at least one of the first license plate image, the second license plate image and the third license plate image.

[0011] In a second aspect, an embodiment of the present invention further provides a license plate image processing device, the device comprising:

[0012] A license plate image determination module, used to determine a first license plate image and a second license plate image, wherein the first license plate image is a license plate image obtained by capturing an image of the vehicle to be detected when the fill light is turned on, and the second license plate image is a license plate image obtained by capturing the vehicle to be detected when the fill light is turned off;

[0013] A license plate image fusion module, configured to fuse the first license plate image and the second license plate image to obtain a third license plate image;

[0014] A reference high-frequency information determination module, used to extract high-frequency information from the first license plate image, the second license plate image and the third license plate image and fuse them to obtain reference high-frequency information;

[0015] The fourth license plate image determination module is used to extract low-frequency information of the reference license plate image to obtain reference low-frequency information, and to fuse the reference high-frequency information with the reference low-frequency information to obtain a fourth license plate image, wherein the reference license plate image is at least one of the first license plate image, the second license plate image and the third license plate image.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the license plate image processing method described in any one of the above embodiments.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable medium, wherein the computer-readable medium stores computer instructions, and the computer instructions are used to enable a processor to implement the license plate image processing method described in any one of the above embodiments when executed.

[0021] The technical solution in the embodiment of the present invention is to determine a first license plate image and a second license plate image, wherein the first license plate image is an image obtained by capturing a vehicle to be detected when a fill light is turned on, and the second license plate image is an image obtained by capturing a vehicle to be detected when a fill light is turned off; a third license plate image is obtained by performing license plate image fusion based on the first license plate image and the second license plate image; high-frequency information is extracted from the first license plate image, the second license plate image and the third license plate image and the fusion is performed to obtain reference high-frequency information; low-frequency information is extracted from the reference license plate image to obtain reference low-frequency information, and a fourth license plate image is obtained by fusion based on the reference high-frequency information and the reference low-frequency information, wherein the reference license plate image is at least one of the first license plate image, the second license plate image and the third license plate image. The solution of the present invention not only takes into account the acquisition of valuable and relatively high-resolution image information when the fill light is turned on, but also supplements the complete low-resolution image information corresponding to when the fill light is turned off to address the problem of overexposure that may occur when the fill light is turned on. Although the resolution is lower, the problem of missing details caused by overexposure can be avoided, so that the finally fused license plate image takes into account the advantages of the license plate images captured in the two scenes, thereby facilitating the accurate recognition of the license plate and improving the adaptability of license plate recognition in different scenes.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0024] Figure 1 A flowchart of a license plate image processing method provided by an embodiment of the present invention;

[0025] Figure 2 A schematic flow chart of another license plate image processing method provided by an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of a process of obtaining a sixth license plate image and a seventh license plate image from a fifth license plate image provided by an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of each license plate identifier in a license plate provided by an embodiment of the present invention;

[0028] Figure 5 A schematic diagram of the structure of a license plate image processing device provided by an embodiment of the present invention;

[0029] Figure 6 A schematic structural diagram of an electronic device for implementing a license plate image processing method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0031] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0032] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0033] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0035] Figure 1The flowchart of a license plate image processing method provided by an embodiment of the present invention is a technical solution of the embodiment of the present invention. The technical solution of the embodiment of the present invention is applicable to the situation where the overexposure or poor adaptability of the license plate image taken under the fill light is processed. The method can be executed by a license plate image processing device. The license plate image processing device can be implemented in the form of software and / or hardware, and is generally integrated in any electronic device with network communication function. The electronic device can be a mobile terminal, PC or server, etc. Figure 1 As shown, the license plate image processing method of the embodiment of the present invention may include the following process:

[0036] S101. Determine a first license plate image and a second license plate image, wherein the first license plate image is a license plate image obtained by capturing an image of a vehicle to be detected when a fill light is turned on, and the second license plate image is a license plate image obtained by capturing an image of the vehicle to be detected when the fill light is turned off.

[0037] The license plate image is an image containing the license plate logo. The vehicle to be detected is the vehicle that needs to be recognized by the license plate. The fill light can be used to enhance the lighting and assist the shooting equipment (such as traffic cameras) to capture the image, thereby increasing the image brightness and significantly improving the image quality. The fill light can be turned on or off by the relevant command control.

[0038] The first license plate image is a license plate image obtained by capturing an image of the vehicle to be detected when the fill light is turned on. The second license plate image is a license plate image obtained by capturing the vehicle to be detected from far to near when the fill light is turned off. The first license plate image and the second license plate image can be determined based on the vehicle image captured by the shooting device. The brightness of the first license plate image is greater than the brightness of the second license plate image.

[0039] As an optional but non-limiting implementation, determining the first license plate image and the second license plate image may include the following steps A101-A102:

[0040] Step A101: When a vehicle to be detected entering a target detection area is detected, the fill light associated with the target detection area is controlled to turn on and flash, and when the fill light is turned on and flashes, an image of the vehicle to be detected entering the target detection area is captured and the license plate image is extracted to obtain a first license plate image.

[0041] The target detection area is the area in which the camera can capture images. In real scenes, in order to cope with shooting environments such as night or insufficient light, the target detection area will be associated with a fill light to enhance the lighting when the camera is shooting.

[0042] When it is detected that a vehicle to be detected enters the target detection area, the fill light associated with the target detection area can be controlled to turn on and flash, and at the same time, the image of the vehicle to be detected entering the target detection area is captured when the fill light is turned on and flashes, and then the license plate image can be extracted from the captured image of the vehicle to be detected, thereby obtaining a first license plate image. In general, the quality of the first license plate image is significantly higher, the image brightness is higher, and the font in the license plate area is clear.

[0043] Optionally, the first license plate image may be extracted by first detecting the license plate image in the image of the vehicle to be detected based on a target detection algorithm, and then extracting the license plate image after the license plate image is detected to obtain the first license plate image. For example, the target detection algorithm may be a detection algorithm such as Faster RCNN, YOLO V5, etc.

[0044] Step A102: Continuously photograph the vehicle to be detected in real time while it is traveling toward the target detection area to obtain multiple candidate vehicle images, and extract the license plate image based on the multiple candidate vehicle images to obtain a second license plate image, the difference in shape and size of the corresponding license plates in the first license plate image and the second license plate image is less than a preset difference, and the position difference between the identification characters at the same position in the corresponding license plate identifiers in the first license plate image and the second license plate image is less than a preset difference.

[0045] The images captured when the fill light is turned on and flashing are only images captured in a very short period of time. When the fill light is turned off, the shooting equipment will still continuously and real-time capture the vehicle to be detected.

[0046] Specifically, when the vehicle to be detected is traveling toward the target detection area, the camera will continuously capture multiple candidate vehicle images in real time, and then extract the license plate image based on the multiple candidate vehicle images to obtain the second license plate image. The extraction process of the second license plate image is the same as the extraction process of the first license plate, and will not be described in detail here.

[0047] The preset difference can be set based on actual needs. Preferably, the obtained second license plate image needs to meet the following requirements: not only the difference in shape and size of the corresponding license plates in the first license plate image and the second license plate image is smaller than the preset difference, but also the position difference between the identification characters in the same position in the corresponding license plate identifiers in the first license plate image and the second license plate image is smaller than the preset difference. That is, it is ensured that the second license plate image is as similar to the first license plate image as possible in structure, so as to facilitate the use and processing of subsequent solutions.

[0048] It should be noted that the first license plate image in the embodiment of the present invention is the license plate image corresponding to when the fill light is turned on and flashing. Generally, such a license plate image has a high brightness and the font in the license plate area is clear; however, due to reasons such as the excessively high reflectivity of the physical material of the license plate, the license plate image may be overexposed, and the font in the license plate area may not be clear, or even no font can be identified at all, resulting in a poor quality license plate image; and such overexposure may be overexposure of the entire license plate area, or may be overexposure of part of the license plate area. For the first license plate image corresponding to these special cases, since the image is overexposed and the license plate recognition cannot be successfully performed, it is necessary to use other license plate images that are not overexposed and are related for processing, that is, to use other license plate images corresponding to the same license plate that are taken when the fill light is turned off for processing, to assist in providing effective image information that can be used for license plate recognition. Correspondingly, the second license plate image in the embodiment of the present invention is a license plate image corresponding to the same vehicle to be detected, which is captured in real time after the fill light is turned off. Although such a second license plate image does not have fill light for fill lighting and the image brightness may not be high, there is still useful information about the license plate logo in the second license plate image, which can be used to combine the first license plate image in overexposure and other situations, thereby making the license plate image easier to identify and achieving the effect of improving the accuracy of license plate recognition.

[0049] As an optional but non-limiting implementation, extracting a license plate image based on multiple candidate vehicle images to obtain a second license plate image may include the following steps B101-B102:

[0050] Step B101: Determine a target vehicle image from multiple candidate vehicle images, where the target vehicle image is a first candidate vehicle image selected from multiple candidate vehicle images corresponding to when the fill light has just been adjusted from the on state to the off state, or a second candidate vehicle image selected from multiple candidate vehicle images corresponding to when the vehicle to be detected is at the closest shooting distance to the shooting device.

[0051] Since multiple candidate vehicle images are acquired through continuous real-time shooting, in order to determine the target vehicle image that is most capable of extracting the second vehicle image that meets the requirements from the multiple candidate vehicle images, those candidate images that are closest in shooting time or shooting distance to the image corresponding to the fill light when the fill light is turned on can be preferentially selected to ensure that the second license plate image extracted by the subsequent scheme can be highly similar in structure to the first license plate image, which is convenient for the processing of the subsequent scheme.

[0052] Specifically, a first candidate vehicle image corresponding to the time when the fill light has just been adjusted from the on state to the off state can be selected from multiple candidate vehicle images, or a second candidate vehicle image corresponding to the time when the vehicle to be detected is at the closest shooting distance to the shooting device can be selected from multiple candidate vehicle images, and then the first candidate vehicle image or the second candidate vehicle image can be used as the target vehicle image.

[0053] Step B102: extracting the license plate image of the target vehicle image to obtain a target license plate image, and determining a second license plate image based on the target license plate image.

[0054] After the target vehicle image is determined, the license plate image can be extracted from the target vehicle image to obtain the target license plate image. The process of extracting the target license plate image is the same as the process of extracting the first license plate above, and will not be described in detail here. Then, the second license plate image can be determined based on the target license plate image.

[0055] By adopting the above method, the candidate vehicle image that is closest in shooting time or shooting distance to the image corresponding to when the fill light is turned on can be preferentially selected from multiple candidate vehicle images as the target vehicle image, so as to finally determine the second license plate image based on the target vehicle image. This can ensure that the second license plate image is as highly similar as possible to the first license plate image in structure, which is convenient for processing of subsequent solutions.

[0056] As an optional but non-limiting implementation, determining the second license plate image based on the target license plate image may include the following steps C101-C102:

[0057] Step C101: When the target vehicle image is the second candidate vehicle image, the license plate corner point positions in the first license plate image and the license plate corner point positions in the target license plate image are detected.

[0058] The license plate corner point position is the position corresponding to each corner point of the license plate on the geometric shape. There are generally 4 license plate corner points, including upper left, upper right, lower left and lower right. The license plate corner point position can be determined by first determining the license plate frame in the image, and then determining the position of each license plate corner point based on the corner point detection algorithm.

[0059] Specifically, when the target vehicle image is the second candidate vehicle image, the position of the corner points of the license plate in the target license plate image and the position of the corner points of the license plate in the first license plate image may be detected.

[0060] Step C102: performing an affine transformation on the target license plate image according to the license plate corner point positions in the first license plate image and the license plate corner point positions in the target license plate image to obtain a second license plate image.

[0061] Among them, affine transformation is a two-dimensional image transformation, which changes the geometric structure of the image through operations such as rotation, scaling, translation and tilt, while maintaining the parallelism of parallel lines in the image.

[0062] Since the obtained license plate image is not necessarily a regular front view, the target license plate image may also be tilted. In this regard, the target license plate image can be first affine transformed according to the positions of the license plate corner points in the first license plate image and the license plate corner points in the target license plate image to obtain the second license plate image.

[0063] As an optional but non-limiting implementation, determining the second license plate image based on the target license plate image may also include: when the target vehicle image is the first candidate vehicle image, directly extracting the license plate image based on the first candidate vehicle image to obtain the second license plate image.

[0064] Specifically, when the target vehicle image is the first candidate vehicle image, it means that the first candidate vehicle image at this time is directly the vehicle image corresponding to the screened fill light just after it is adjusted from the on state to the off state. Then, the license plate image can be directly extracted based on the first candidate vehicle image to obtain the second license plate image.

[0065] S102: performing license plate image fusion based on the first license plate image and the second license plate image to obtain a third license plate image.

[0066] Optionally, an image fusion model may be used to fuse the first license plate image with the second license plate image to obtain a third license plate image.

[0067] Since the first license plate image is the license plate image corresponding to when the fill light is turned on and flashing, and the second license plate image is the license plate image corresponding to when the fill light is turned off, the third license plate image obtained by fusing the two images can have richer image information and can retain useful information in both license plate images at the same time.

[0068] S103, extracting high-frequency information from the first license plate image, the second license plate image, and the third license plate image and fusing them to obtain reference high-frequency information.

[0069] The high-frequency information refers to the image information corresponding to the fast-changing areas such as details, edges and textures in the image. For example, the high-frequency information in the license plate image may be the information corresponding to the license plate identification characters or the license plate edge. Optionally, the high-frequency information may be extracted by first using a low-frequency filter to extract the low-frequency information of the image, and then subtracting the low-frequency information from the original image information to obtain the high-frequency information.

[0070] The reference high-frequency information can be obtained by extracting high-frequency information from the first license plate image, the second license plate image and the third license plate image respectively and fusing them. This makes the reference high-frequency information rich and complete in information content, including high-quality image information corresponding to the case where the fill light is turned on, and timely supplementing the corresponding high-frequency information when the fill light is turned off for possible overexposure.

[0071] S104. Extract low-frequency information from the reference license plate image to obtain reference low-frequency information, and fuse the reference high-frequency information with the reference low-frequency information to obtain a fourth license plate image, where the reference license plate image is at least one of the first license plate image, the second license plate image, and the third license plate image.

[0072] Among them, low-frequency information refers to image information corresponding to areas in the image that change slowly or not obviously. For example, the low-frequency information in the image can be information corresponding to a large area of ​​pure color background area. Optionally, the low-frequency information of the image can be extracted by a low-pass filter. For example, a low-pass filter such as a mean filter, a Gaussian filter, and a guided filter can be used.

[0073] For example, taking the mean filter as an example, the parameters of the mean filter can be freely set, and the filter window size in the default parameters is set to 11*11, and the Gaussian variance is set to 5. The following formula can be used to process the image to be processed:

[0074] ;

[0075] Among them, A1(j) is the image to be processed, A2(i) is the low-frequency information corresponding to the image to be processed; W is the mean filter template, Pixel The mean filter window centered at Represents a convolution operation.

[0076] At least one of the first license plate image, the second license plate image and the third license plate image is used as a reference license plate image, and reference low-frequency information is extracted from the reference license plate image to obtain reference low-frequency information, and the fourth license plate image is obtained by fusing the reference high-frequency information with the reference low-frequency information. In this way, the fourth license plate image contains more useful information content, which not only retains the valuable high-resolution image information corresponding to the case where the fill light is turned on and the flash is turned on, but also timely supplements the corresponding complete low-resolution image information corresponding to the case where the fill light is turned off for possible overexposure, thereby facilitating accurate recognition of the license plate.

[0077] The technical solution in the embodiment of the present invention is to determine a first license plate image and a second license plate image, wherein the first license plate image is an image obtained by capturing a vehicle to be detected when a fill light is turned on, and the second license plate image is an image obtained by capturing a vehicle to be detected when a fill light is turned off; a third license plate image is obtained by performing license plate image fusion based on the first license plate image and the second license plate image; high-frequency information is extracted from the first license plate image, the second license plate image and the third license plate image and the fusion is performed to obtain reference high-frequency information; low-frequency information is extracted from the reference license plate image to obtain reference low-frequency information, and a fourth license plate image is obtained by fusion based on the reference high-frequency information and the reference low-frequency information, wherein the reference license plate image is at least one of the first license plate image, the second license plate image and the third license plate image. The solution of the present invention not only takes into account the acquisition of valuable and relatively high-resolution image information when the fill light is turned on, but also supplements the complete low-resolution image information corresponding to when the fill light is turned off to address the problem of overexposure that may occur when the fill light is turned on. Although the resolution is lower, the problem of missing details caused by overexposure can be avoided, so that the finally fused license plate image takes into account the advantages of the license plate images captured in the two scenes, thereby facilitating the accurate recognition of the license plate and improving the adaptability of license plate recognition in different scenes.

[0078] Figure 2 This is a flow chart of another license plate image processing method provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the process of extracting high-frequency information from the first license plate image, the second license plate image and the third license plate image and fusing them to obtain reference high-frequency information in the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the license plate image processing method of the embodiment of the present invention may include the following process:

[0079] S201. Determine a first license plate image and a second license plate image, wherein the first license plate image is a license plate image obtained by capturing an image of a vehicle to be detected when a fill light is turned on, and the second license plate image is a license plate image obtained by capturing an image of the vehicle to be detected when the fill light is turned off.

[0080] S202: Fusing the first license plate image and the second license plate image to obtain a third license plate image.

[0081] As an optional but non-limiting implementation, obtaining a third license plate image by performing license plate image fusion based on the first license plate image and the second license plate image may include the following steps D1-D2:

[0082] Step D1: Input the first license plate image and the second license plate image into the license plate image fusion model.

[0083] Among them, the license plate image fusion model is a neural network model obtained by training based on a preset license plate image data set, which can fuse license plate images captured at different brightnesses. The preset license plate image data set includes multiple license plate image samples, each license plate image sample includes a fifth license plate image, a sixth license plate image and a seventh license plate image. The fifth license plate image is a license plate image collected in an actual scene and has an image quality greater than the preset image quality. The sixth license plate image and the seventh license plate image are both license plate images obtained by performing random blur operations, random color degradation operations and random brightness degradation operations on the fifth license plate image, and the average brightness of the sixth license plate image is lower than the average brightness of the seventh license plate image.

[0084] It should be noted that, regarding the license plate image data set used when training the neural network model, in general, it is difficult to collect the first license plate image, the second license plate image and the third license plate image that meet the requirements from the actual application scene, and it is also difficult to completely construct multiple sets of interrelated image samples based on these license plate images. In view of this, a reverse thinking is adopted in the embodiment of the present invention to construct an image data set that meets the requirements. Specifically, for the construction of one of the license plate image samples, since it is relatively easy to obtain a license plate image with image quality that meets the requirements from the actual scene, then this high-quality license plate image can be subjected to random blurring operation, random color degradation operation and random brightness degradation operation. After such a series of "degradation operations", two associated low-quality license plate images can be obtained, wherein the average brightness of one low-quality license plate image is greater than the average brightness of the other low-quality license plate image. The three images formed in this way can more perfectly simulate the images obtained after the fill light is turned on, the fill light is turned off and the corresponding images are obtained after fusion, thereby constructing a license plate image sample that meets the requirements.

[0085] Among them, the random blur operation refers to blurring the image and randomly increasing the blur of the image to make the image unclear. The random color degradation operation refers to adjusting the color channel data of the image to make the image appear aged or faded. The random brightness degradation operation refers to randomly reducing the brightness of the image to achieve the effect of reducing the brightness of the image.

[0086] The preset image quality can be determined based on actual needs. Generally, license plate images with clear license plate fonts and correct colors are preferred, aiming to select a higher quality license plate image as the fifth license plate image. Only then can the relevant sixth and seventh license plate images be determined through a series of "degradation operations".

[0087] For example, Figure 3 A schematic diagram of a process of obtaining a sixth license plate image and a seventh license plate image from a fifth license plate image is provided in an embodiment of the present invention, such as Figure 3 As shown, the details are as follows:

[0088] First, a random blur operation is performed on the fifth license plate image. Optionally, a Gaussian blur kernel can be used as the blur kernel; for example, based on blur kernels k1 and k2 with random different sigma values, a convolution operation is performed on the fifth license plate image using k1 and k2 respectively, thereby obtaining the first license plate simulation image A11 and the second license plate simulation image B11 after being blurred to different degrees.

[0089] It should be noted that the average brightness values ​​of the first license plate simulation image A11 and the second license plate simulation image B11 must be different, in order to simulate the image when the fill light is turned on through one of them and the image when the fill light is turned on and off through the other.

[0090] Then, random color degradation operations are performed on the first license plate simulation image A11 and the second license plate simulation image B11, respectively, specifically, the color channel values ​​in the images are randomly deflected as a whole. The random color degradation operations on the first license plate simulation image A11 and the second license plate simulation image B11 have the same principle. In the embodiment of the present invention, only one of them is described in detail, and the same principle content is not repeated in detail. Specifically, the random color degradation operation can be performed on the first license plate simulation image A11 based on the following formula:

[0091] ;

[0092] Among them, A12 is the third license plate simulation image obtained after performing a random color degradation operation on the first license plate simulation image A11; p1 is a uniformly sampled random number in the [0,1] interval; a1, a2, a3, and a4 are four different thresholds set in the range of [0,1], and the size relationship between a1, a2, a3, and a4 increases successively; q11, q12, and q13 are different color deflection ratios, and q11, q12, and q13 can be randomly sampled between 0.8 and 1.2, respectively.

[0093] It should be noted that the specific values ​​of a1, a2, a3, and a4 can be freely set. For example, they can be specifically set to a1=0.2, a2=0.4, a3=0.6, and a4=0.8.

[0094] Based on the random color degradation operation of the same principle, a fourth license plate simulation image B12 after color degradation can be obtained based on the second license plate simulation image B11.

[0095] Next, random brightness degradation operations are performed on the third license plate simulation image A12 and the fourth license plate simulation image B12 respectively to adjust the brightness of the images as a whole. Specifically, the random brightness degradation operations can be performed on the third license plate simulation image A12 and the fourth license plate simulation image B12 based on the following formula:

[0096] ;

[0097] ;

[0098] Among them, A13 is the fifth license plate simulation image obtained by performing a random brightness degradation operation on the third license plate simulation image A12; B13 is the sixth license plate simulation image obtained by performing a random brightness degradation operation on the fourth license plate simulation image B12; p2 is a uniformly sampled random number in the [0,1] interval, b1 and b2 are two different thresholds in the [0,1] range; q2 and q3 are different overall brightness change ratios, among which q2 is randomly and uniformly sampled between [1,1.5]. q3 is randomly and uniformly sampled between [0.75,1].

[0099] It should be noted that the specific values ​​of b1 and b2 can be set freely. For example, they can be set to b1=0.5 and b2=0.5.

[0100] After the above three "degradation operations", the image with higher average brightness among the fifth license plate simulation image A13 and the sixth license plate simulation image B13 can be used as the sixth license plate image, and the image with lower average brightness can be used as the seventh license plate image. Then, a license plate sample image can be constructed based on the fifth license plate image, the sixth license plate image and the seventh license plate image. Similarly, other license plate images collected in actual scenes with image quality greater than the preset image quality are selected as the new fifth license plate image, and related degradation operations are performed according to the above principles, so that multiple license plate image samples can be determined, thereby forming a preset license plate image data set.

[0101] It should be noted that the order of the random blur operation, the random color degradation operation and the random brightness degradation operation in the implementation of the present invention is not specifically limited, and the above example only gives one of the three degradation operations. In addition, when selecting different degradation operations, at least one of the random blur operation and the random color degradation operation can be selected, but the random brightness degradation operation needs to be selected to simulate different brightness images in the actual scene when the fill light is turned on and the fill light is turned off.

[0102] After the preset license plate image data set is determined, the preset license plate image data set can be used to train the neural network model. For example, a convolutional neural network model with an architecture such as U-Net or FCN can be used for model training and design.

[0103] For example, taking the use of U-Net convolutional neural network for model training as an example, the model training process can be:

[0104] First, a U-Net neural network model consisting of a convolutional layer, a pooling layer, a deconvolution layer and other structures is constructed. The sixth and seventh license plate images in the preset license plate image dataset are used as model inputs. The predicted license plate image is obtained through the forward propagation mechanism. The pixel difference loss function Loss between the predicted license plate image and the fifth license plate image can be calculated based on the following formula:

[0105] ;

[0106] Among them, G represents the corresponding fifth license plate image; C1 is the predicted license plate image; w1 represents the pixel difference loss weight, and w2 represents the perceptual loss weight; represents the deep feature image obtained after the forward reasoning operation is performed on the fifth license plate image; Represents the deep feature image obtained after performing forward reasoning on the predicted license plate image.

[0107] It should be noted that the values ​​of w1 and w2 can be freely set, for example, w1 can be set to 0.5, and w2 can also be set to 0.5. The forward reasoning operation on the license plate image can be performed by using a pre-trained U-Net network model.

[0108] It should be noted that before the sixth license plate image and the seventh license plate image are used as model input, the sixth license plate image and the seventh license plate image need to be fused in the channel dimension in order to ensure that the width information and height information of the two images are consistent. If the width information and height information of the two images are inconsistent, a bilinear scaling operation can be performed on one of the license plate images to make the width information and height information of the two license plate images consistent after adjustment before continuing the license plate image fusion operation. When adjusting the width information and height information of the image, it is preferred to perform a bilinear scaling operation on the license plate image with a darker average brightness.

[0109] Exemplarily, the sixth license plate image has a width of w, a height of h, and a number of channels of c; the seventh license plate image has a width of w, a height of h, and a number of channels of c. After stacking the two license plate images in the channel dimension, the fused image used as the model input has a width of w, a height of h, and a number of channels of 2*c.

[0110] Then, the model parameters are updated and optimized by the back-propagation algorithm. For example, the model parameters can be updated and optimized using the stochastic gradient descent optimization algorithm (SGD).

[0111] Next, the trained model is evaluated using the validation set to calculate whether indicators such as PSNR (peak signal-to-noise ratio) and SSIM (structural similarity) meet the usage requirements. If the requirements are met, the training of the neural network model is completed and the license plate image fusion model can be obtained.

[0112] Step D2: The license plate identifier corresponding to the first license plate image and the license plate identifier corresponding to the second license plate image are fused through the license plate image fusion model to obtain a third license plate image.

[0113] After obtaining the license plate image fusion model, the first license plate image and the second license plate image can be input into the license plate image fusion model, and the license plate identifier corresponding to the first license plate image and the license plate identifier corresponding to the second license plate image can be fused to obtain a third license plate image.

[0114] By adopting the above scheme, a high-quality license plate image that is relatively easy to collect in an actual scene can be used as the fifth license plate image, and the corresponding sixth and seventh license plate images can be obtained after random blur operation, random color degradation operation and random brightness degradation operation are performed on the fifth license plate image, and then a license plate image sample is determined, and so on, a preset license plate image data set can be obtained. The above scheme solves the difficulty of directly obtaining an image data set that meets the requirements from a real scene to a certain extent, and adopts reverse thinking to construct an image data set that meets the requirements, so that the preset license plate image data set is more accurate and appropriate, which is conducive to improving the accuracy of model training.

[0115] S203 , performing low-pass filtering on the first license plate image, the second license plate image, and the third license plate image, respectively, to obtain high-frequency information included in the first license plate image, the second license plate image, and the third license plate image.

[0116] S204, determining reference weight information of high-frequency information included in each of the first license plate image, the second license plate image and the third license plate image; the reference weight information of the high-frequency information included in the license plate image is determined based on the first weight information and the second weight information of the license plate image, the first weight information is determined based on the confidence of the identification characters at each position in the license plate identifier corresponding to the license plate image, the greater the confidence of the identification characters at each position in the license plate identifier, the greater the fusion weight of the image area occupied by the identification characters at each position in the license plate identifier corresponding to the license plate image indicated by the first weight information when performing high-frequency information fusion, the second weight information is determined based on the detail richness of each pixel in the license plate image, the greater the detail richness of the pixel, the greater the fusion weight of each pixel in the license plate image indicated by the second weight information when performing high-frequency information fusion, the detail richness of the pixel is determined based on the pixel change and the pixel in the neighborhood around the pixel.

[0117] Since the first license plate image is the license plate image corresponding to when the fill light is flashing, the second license plate image is the license plate image corresponding to when the fill light is turned off, and the third license plate image is the license plate image obtained by fusing the first license plate image with the second license plate image. However, due to the existence of problems such as image overexposure, it does not necessarily mean that the clarity of the text in the first license plate image is the best, nor does it necessarily mean that the clarity of the text in the second license plate image is the worst. Therefore, different weight information can be set for the high-frequency information at the same position in these three license plate images, and then the reference high-frequency information can be fused based on the size of the weight information between the three, so that the final reference high-frequency information can have richer and more effective content, thereby facilitating the recognition of the license plate.

[0118] Specifically, the reference weight information of the high-frequency information included in the license plate image is determined based on the first weight information and the second weight information of the license plate image. The reference weight information can be obtained by multiplying the first weight information and the second weight information.

[0119] Among them, the first weight information is determined based on the confidence of the identification characters at each position in the license plate identifier corresponding to the license plate image. Optionally, the identification characters at each position in the license plate identifier corresponding to the license plate image can be identified separately using a license plate recognition algorithm. For example, the license plate recognition algorithm can be a general OCR (Optical Character Recognition) technology. Alternatively, a pre-trained license plate recognition neural network model can also be used to identify the identification characters at each position in the license plate identifier corresponding to the license plate image. After identifying the identification characters at each position, the confidence corresponding to the identification characters at each position can be determined. The greater the confidence of the identification characters at each position in the license plate identifier, the clearer the identifier at this position, and the identification characters at this position in this image can be fused with a greater weight when referring to high-frequency information fusion in the future, that is, the greater the fusion weight of the image area occupied by the identification characters at each position in the license plate identifier corresponding to the license plate image indicated by the first weight information when performing high-frequency information fusion.

[0120] The second weight information is determined based on the richness of details of each pixel in the license plate image. The richer the detail of the pixel, the more effective information there is near the pixel. Therefore, when performing high-frequency information fusion, the greater the fusion weight of each pixel in the license plate image indicated by the second weight information is. The richness of detail of the pixel is determined based on the pixel changes in the pixel and the pixel neighborhood around the pixel.

[0121] As an optional but non-limiting implementation, determining the reference weight information of the high-frequency information respectively included in the first license plate image, the second license plate image, and the third license plate image may include the following steps E1-E2:

[0122] Step E1: The first reference image, the second reference image and the third license plate image are respectively used as the first license plate images to be processed, and license plate recognition is performed on the identification characters at each position in the license plate identifier corresponding to the first license plate image to be processed, so as to obtain the character position and the corresponding confidence of the identification characters at each position in the license plate identifier corresponding to the first license plate image to be processed. The first reference image is a fusion image of the high-frequency information included in the first license plate image and the reference low-frequency information, and the second reference image is a fusion image of the high-frequency information included in the second license plate image and the reference low-frequency information.

[0123] Specifically, when determining the first weight information in the reference weight information, the first reference image, the second reference image, and the third license plate image are respectively used as the first license plate image to be processed. Among them, the first reference image is a fused image obtained by fusing the high-frequency information included in the first license plate image with the reference low-frequency information, and the second reference image is a fused image obtained by fusing the high-frequency information included in the second license plate image with the reference low-frequency information.

[0124] The principle of determining the confidence of each license plate identifier in each first license plate image to be processed is the same. Here, only the example of determining the confidence of the identification characters at each position in the license plate identifier corresponding to one of the first license plate images to be processed is used for illustration. Optionally, OCR (Optical Character Recognition) technology can be used to perform license plate recognition on the identification characters at each position in the license plate identifier corresponding to the first license plate image to be processed, and the character position of the identification characters at each position in the license plate identifier corresponding to the first license plate image to be processed and the confidence corresponding to the identification characters can be obtained.

[0125] Figure 4 It is a schematic diagram of the situation of each license plate identifier in a license plate provided by an embodiment of the present invention. As Figure 4 shown, the license plate identifiers in the license plate are successively "Di", "A", "1", "2", "3", "4", and "5". Among them, the first character "Di" represents the abbreviation of a specific province. S1 represents the position corresponding to the first character "Di", and the confidence of the character identification at this position can be set as SA1; S2 represents the position corresponding to the second character "A", and the confidence of the character identification at this position can be set as SA2; S3 represents the position corresponding to the third character "1", and the confidence of the character identification at this position can be set as SA3, and so on. In this way, the character position of the identification characters at each position in the license plate identifier corresponding to a first license plate image to be processed and the confidence corresponding to the identification characters can be obtained. By the same principle, the relevant information of other first license plate images to be processed can be obtained.

[0126] Step E2: Based on the character position of the identification characters at each position in the license plate identifier corresponding to the first license plate image to be processed and the confidence corresponding to the identification characters, determine the first weight information of the first license plate image to be processed.

[0127] Although the above three first license plate images to be processed (the first reference image, the second reference image, and the third license plate image) contain the same license plate character content, there may be significant differences in the image quality of the characters at the same position in these three first license plate images to be processed. For example, for the first character "地" on the license plate, it may be that the character "地" is unclear in the first reference image (caused by overexposure), slightly clearer in the second reference image, but clearest in the third license plate image. Then, in order to better extract the high-frequency information of the image and perform subsequent fusion, it is necessary to differentiate the confidence levels of the same characters at the same position in different images, so that the subsequent solution can extract and fuse the high-frequency information of the image.

[0128] Specifically, first, set the corresponding initialization weight matrix for each of the first license plate images to be processed, and the weight values of each initialization weight matrix are fixed at 1. For example, the initialization weight matrix of the first reference image is WA1, the initialization weight matrix of the second reference image is WA2, and the initialization weight matrix of the third license plate image is WA3, and the weight values of WA1, WB1, and WC1 are all fixed at 1.

[0129] Then, for the license plate character identifiers at the same position, compare the confidence level relationships of the characters at this position in the three first license plate images to be processed. For the first license plate image corresponding to the highest confidence level, retain the weight value at the character position in the corresponding initialization weight matrix of this first license plate image as 1; for the first license plate image corresponding to a confidence level that is not the highest, assign the weight value at the character position in the corresponding initialization weight matrix of this first license plate image to 0.

[0130] Optionally, in the embodiments of the present invention, the following formula can be used to assign values to each initialization weight matrix:

[0131] ;

[0132] ;

[0133] ;

[0134] Wherein, represents the position where the same license plate identifier character is located; represents the weight value at the position in the initialization weight matrix corresponding to the first reference image, represents the weight value at the position in the initialization weight matrix corresponding to the second reference image, represents the weight value at the The weight value at the position; SA1 represents the confidence of the license plate identification character at the position in the first reference image, SB1 represents the confidence of the license plate identification character at the position in the second reference image, and SC1 represents the confidence of the license plate identification character at the position in the third license plate image. The confidence of the license plate identification character at the position, SB1 represents the confidence of the license plate identification character at the position in the second reference image The confidence of the license plate identification character at the position, SC1 represents the confidence of the license plate identification character at the position in the third license plate image The confidence of the license plate identification character at the position.

[0135] Exemplarily, for the first character "地", if it is determined that SA1 is greater than SB1 and SC1 after comparing the confidence levels, then the weight value at the position of the first character "地" in the WA1 weight matrix is reserved as 1, and the weight values at the position of the first character "地" in the WB1 and WC1 weight matrices are assigned 0. The processing method for each license plate identification character is the same. In the embodiments of the present invention, only the processing of the first character "地" is taken as an example for illustration.

[0136] And so on, the adjustment of each initialized weight matrix can be completed based on the confidence levels of the license plate identification characters at each position in the license plate identification, thereby determining the first weight information of the first license plate image to be processed. The first weight information is the weight matrix obtained after adjusting the initialized weight matrix.

[0137] In the above manner, the first weight information corresponding to the first license plate image, the second license plate image, and the third license plate image is determined. The first weight information can reflect the difference in the confidence levels of the same characters at the same position in different images, reflecting the image quality of the characters at the same position in different images, and thus facilitating the determination of the reference weight for subsequent solutions, as well as guiding the extraction and fusion of high-frequency information of the images.

[0138] As an optional but non-limiting implementation manner, determining the reference weight information of the high-frequency information included in the first license plate image, the second license plate image, and the third license plate image may further include the following steps F1 - F2:

[0139] Step F1: Respectively take the high-frequency information included in the first license plate image, the second license plate image, and the third license plate image as the second license plate image to be processed. For each pixel point to be processed in the second license plate image to be processed, determine the first pixel point quantity and the second pixel point quantity of each pixel point to be processed in the second license plate image to be processed. The first pixel point is the pixel point in the reference neighborhood of the pixel point to be processed, the reference neighborhood is a pixel area with a preset area size centered on the pixel point to be processed, and the second pixel point is the pixel point in the reference neighborhood of the pixel point to be processed used to represent the license plate identifier.

[0140] Specifically, when determining the second weight information in the reference weight information, the high-frequency information included in each of the first license plate image, the second license plate image, and the third license plate image is used as the second license plate image to be processed. Since each second license plate image to be processed has different detail richness, for the same area, some second license plate images to be processed may have more pixels in the area, while some second license plate images to be processed may have relatively fewer pixels. Therefore, the second reference weight information can be further determined based on the richness of the pixels in such an area.

[0141] Specifically, for each pixel to be processed in the second license plate image to be processed, the first pixel number and the second pixel number of each pixel to be processed can be determined. The first pixel is a pixel within a reference neighborhood of the pixel to be processed, and the second pixel is a pixel within the reference neighborhood of the pixel to be processed that is used to characterize the license plate identifier. The reference neighborhood is a pixel area of ​​a preset area size centered on the pixel to be processed.

[0142] Step F2: Determine the second weight information of the second license plate image to be processed according to the first number of pixels and the second number of pixels of each pixel to be processed in the second license plate image to be processed.

[0143] After determining the first number of pixels and the second number of pixels, the following formula may be used to determine the second weight information of the second license plate image to be processed:

[0144] ;

[0145] Among them, (x, y) represents the position of the pixel to be processed; Ω(x, y) represents the reference neighborhood centered on the pixel to be processed; A represents the pixel point of the second license plate image to be processed in the i-th row and j-th column; represents the second weight information corresponding to the pixel point at the (x, y) position of the second license plate image to be processed; N represents the number of pixels in the reference neighborhood, that is, the number of first pixels.

[0146] Optionally, after determining the first weight information and the second weight information, the first weight information and the second weight information may be multiplied to obtain reference weight information. Similarly, reference weight information of high-frequency information included in each of the first license plate image, the second license plate image, and the third license plate image may be obtained.

[0147] Correspondingly, the fourth license plate image can be fused based on the following formula:

[0148] ;

[0149] Among them, D is the pixel in the i-th row and j-th column in the fourth license plate image; A is the pixel in the i-th row and j-th column in the first license plate image; B is the pixel at the i-th row and j-th column in the second license plate image; C is the pixel at the i-th row and j-th column in the third license plate image; E WA is the pixel in the i-th row and j-th column obtained after low-frequency information extraction from the reference license plate image; is the reference weight information corresponding to the pixel in the i-th row and j-th column in the high-frequency information included in the first license plate image, WB is the reference weight information corresponding to the pixel in the i-th row and j-th column in the high-frequency information included in the second license plate image, WC It is the reference weight information corresponding to the pixel in the i-th row and the j-th column in the high-frequency information included in the third license plate image.

[0150] Using the above method, the second weight information corresponding to the first license plate image, the second license plate image and the third license plate image is determined respectively. The second weight information can reflect the richness of pixel information in the same area in different images, thereby facilitating the determination of reference weights for subsequent schemes and guiding the extraction and fusion of high-frequency information of the image.

[0151] S205. According to the reference weight information of the high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image, the high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image is merged to obtain reference high-frequency information.

[0152] S206. Extract low-frequency information from the reference license plate image to obtain reference low-frequency information, and fuse the reference high-frequency information with the reference low-frequency information to obtain a fourth license plate image, where the reference license plate image is at least one of the first license plate image, the second license plate image, and the third license plate image.

[0153] The technical solution in the embodiment of the present invention determines a first license plate image and a second license plate image, wherein the first license plate image is an image obtained by capturing a vehicle to be detected when a fill light is turned on, and the second license plate image is an image obtained by continuously capturing the vehicle to be detected in real time when the fill light is turned off; a third license plate image is obtained by performing license plate image fusion based on the first license plate image and the second license plate image; high-frequency information included in each of the first license plate image, the second license plate image, and the third license plate image is obtained by performing low-pass filtering on the first license plate image, the second license plate image, and the third license plate image; determining the first license plate image The first license plate image, the second license plate image and the third license plate image are each provided with reference weight information of high-frequency information; according to the reference weight information of high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image, the high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image are fused to obtain reference high-frequency information; low-frequency information is extracted from the reference license plate image to obtain reference low-frequency information, and a fourth license plate image is obtained by fusing the reference high-frequency information and the reference low-frequency information, and the reference license plate image is at least one of the first license plate image, the second license plate image and the third license plate image. The scheme of the present invention not only takes into account the valuable high-resolution image information corresponding to the case where the fill light is turned on, but also timely supplements the corresponding complete low-resolution image information corresponding to the case where the fill light is turned off for possible overexposure, so that the license plate image finally obtained by fusion takes into account the advantages of the license plate images captured in two scenes, thereby facilitating accurate recognition of the license plate and improving the adaptability of license plate recognition in different scenes. At the same time, in the process of extracting and fusing the high-frequency information of each image, reference weight information is introduced to comprehensively consider the confidence of the vehicle identification characters at each position in different images and the richness of the pixels in the reference neighborhood. This can better guide the fusion of high-frequency information in subsequent schemes and further enhance the impact of the fusion results on the accuracy of license plate recognition.

[0154] Figure 5 The schematic diagram of the structure of a license plate image processing device provided by an embodiment of the present invention is applicable to image processing of overexposure or poor adaptability of license plate images taken under fill light. The license plate image processing device can be implemented in the form of software and / or hardware, and is generally integrated in any electronic device with network communication function, which can be a mobile terminal, PC or server, etc. Figure 5 As shown, the license plate image processing device of the embodiment of the present invention may include the following:

[0155] The license plate image determination module 501 is used to determine a first license plate image and a second license plate image, wherein the first license plate image is a license plate image obtained by capturing an image of the vehicle to be detected when the fill light is turned on, and the second license plate image is a license plate image obtained by capturing an image of the vehicle to be detected when the fill light is turned off;

[0156] A license plate image fusion module 502, configured to fuse the first license plate image and the second license plate image to obtain a third license plate image;

[0157] A reference high-frequency information determination module 503 is used to extract high-frequency information from the first license plate image, the second license plate image and the third license plate image and fuse them to obtain reference high-frequency information;

[0158] The fourth license plate image determination module 504 is used to extract low-frequency information from the reference license plate image to obtain reference low-frequency information, and to fuse the reference high-frequency information with the reference low-frequency information to obtain a fourth license plate image, wherein the reference license plate image is at least one of the first license plate image, the second license plate image and the third license plate image.

[0159] On the basis of the above embodiment, optionally, determining the first license plate image and the second license plate image includes: when a vehicle to be detected driving into a target detection area is detected, controlling a fill light associated with the target detection area to turn on and flash, and when the fill light is turned on and flashing, capturing an image of the vehicle to be detected entering the target detection area and extracting the license plate image to obtain a first license plate image; continuously and in real time photographing the vehicle to be detected in the process of driving toward the target detection area to obtain a plurality of candidate vehicle images, and extracting the license plate image based on the plurality of candidate vehicle images to obtain a second license plate image, wherein the difference in shape and size of the corresponding license plates in the first license plate image and the second license plate image is less than a preset difference, and the position difference between the identification characters at the same position in the corresponding license plate identifiers in the first license plate image and the second license plate image is less than a preset difference.

[0160] On the basis of the above embodiment, optionally, a second license plate image is obtained based on multiple candidate vehicle images by performing license plate image extraction, including: determining a target vehicle image from multiple candidate vehicle images, the target vehicle image being a first candidate vehicle image corresponding to when the fill light selected from the multiple candidate vehicle images has just been adjusted from the on state to the off state, or being a second candidate vehicle image corresponding to when the vehicle to be detected selected from the multiple candidate vehicle images is at the shortest shooting distance from the shooting device; performing license plate image extraction on the target vehicle image to obtain a target license plate image, and determining a second license plate image based on the target license plate image.

[0161] On the basis of the above embodiment, optionally, determining a second license plate image based on a target license plate image includes: when the target vehicle image is a second candidate vehicle image, detecting the license plate corner point position in the first license plate image and the license plate corner point position in the target license plate image; performing an affine transformation on the target license plate image according to the license plate corner point position in the first license plate image and the license plate corner point position in the target license plate image to obtain a second license plate image.

[0162] Based on the above embodiment, optionally, obtaining a third license plate image by performing license plate image fusion based on the first license plate image and the second license plate image includes: inputting the first license plate image and the second license plate image into a license plate image fusion model; fusing a license plate identifier corresponding to the first license plate image with a license plate identifier corresponding to the second license plate image through the license plate image fusion model to obtain the third license plate image;

[0163] Among them, the license plate image fusion model is a neural network model obtained by training based on a preset license plate image data set, which can fuse license plate images captured at different brightnesses. The preset license plate image data set includes multiple license plate image samples, each of the license plate image samples includes a fifth license plate image, a sixth license plate image and a seventh license plate image. The fifth license plate image is a license plate image collected in an actual scene and has an image quality greater than the preset image quality. The sixth license plate image and the seventh license plate image are both license plate images obtained by performing random blur operations, random color degradation operations and random brightness degradation operations on the fifth license plate image, and the average brightness of the sixth license plate image is lower than the average brightness of the seventh license plate image.

[0164] On the basis of the above embodiment, optionally, extracting high-frequency information from the first license plate image, the second license plate image and the third license plate image and fusing them to obtain reference high-frequency information, includes: performing low-pass filtering on the first license plate image, the second license plate image and the third license plate image respectively to obtain the high-frequency information included in the first license plate image, the second license plate image and the third license plate image; determining reference weight information of the high-frequency information included in the first license plate image, the second license plate image and the third license plate image; the reference weight information of the high-frequency information included in the license plate image is determined based on the first weight information and the second weight information of the license plate image, the first weight information is determined based on the confidence of the identification characters at each position in the license plate identifier corresponding to the license plate image, and the reference weight information of the high-frequency information at each position in the license plate identifier is determined based on the confidence of the identification characters at each position in the license plate identifier. The greater the confidence of the identified character, the greater the fusion weight of the image area occupied by the identification character at each position in the license plate identifier of the license plate image indicated by the first weight information when performing high-frequency information fusion, the second weight information is determined based on the detail richness of each pixel in the license plate image, the greater the detail richness of the pixel, the greater the fusion weight of each pixel in the license plate image indicated by the second weight information when performing high-frequency information fusion, and the detail richness of the pixel is determined based on the pixel change within the pixel and the neighborhood around the pixel; according to the reference weight information of the high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image, the high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image is fused to obtain reference high-frequency information.

[0165] On the basis of the above embodiment, optionally, determining reference weight information of the high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image includes: taking the first reference image, the second reference image and the third license plate image as first license plate images to be processed respectively, and performing license plate recognition on the identification character at each position in the license plate identifier corresponding to the first license plate image to be processed, and obtaining the character position of the identification character at each position in the license plate identifier corresponding to the first license plate image to be processed and the confidence corresponding to the identification character, the first reference image is a fusion image of the high-frequency information included in the first license plate image and the reference low-frequency information, and the second reference image is a fusion image of the high-frequency information included in the second license plate image and the reference low-frequency information; based on the character position of the identification character at each position in the license plate identifier corresponding to the first license plate image to be processed and the confidence corresponding to the identification character, determining the first weight information of the first license plate image to be processed.

[0166] On the basis of the above embodiment, optionally, reference weight information of high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image is determined, including: taking the high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image as the second license plate image to be processed, and for each pixel to be processed in the second license plate image to be processed, determining the first pixel number and the second pixel number of each pixel to be processed in the second license plate image to be processed, the first pixel point is a pixel point within a reference neighborhood of the pixel to be processed, the reference neighborhood is a pixel area of ​​a preset area size centered on the pixel to be processed, and the second pixel point is a pixel point within the reference neighborhood of the pixel to be processed for characterizing the license plate identifier; determining the second weight information of the second license plate image to be processed according to the first pixel number and the second pixel number of each pixel to be processed in the second license plate image to be processed.

[0167] The technical solution in the embodiment of the present invention determines a first license plate image and a second license plate image, the first license plate image is an image obtained by capturing a vehicle to be detected when a fill light is turned on, and the second license plate image is an image obtained by continuously shooting the vehicle to be detected in real time when the fill light is turned off; a third license plate image is obtained by performing license plate image fusion based on the first license plate image and the second license plate image; high-frequency information is extracted from the first license plate image, the second license plate image and the third license plate image and fused to obtain reference high-frequency information; low-frequency information is extracted from the reference license plate image to obtain reference low-frequency information, and a fourth license plate image is obtained by fusion based on the reference high-frequency information and the reference low-frequency information, and the reference license plate image is at least one of the first license plate image, the second license plate image and the third license plate image. By adopting the solution of the present invention, valuable high-resolution image information corresponding to the case where the fill light is turned on is taken into account, and corresponding complete low-resolution image information corresponding to the case where the fill light is turned off is timely supplemented in response to possible overexposure. As a result, the finally fused license plate image takes into account the advantages of the license plate images captured in both scenarios, thereby facilitating accurate recognition of the license plate and improving the adaptability of license plate recognition in different scenarios.

[0168] The license plate image processing device provided in the embodiment of the present invention can execute the license plate image processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the license plate image processing method.

[0169] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0170] Figure 6 A schematic diagram of the structure of an electronic device for implementing a license plate image processing method provided by an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0171] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0172] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0173] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a license plate image processing method.

[0174] In some embodiments, the license plate image processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the license plate image processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the license plate image processing method in any other appropriate manner (e.g., by means of firmware).

[0175] The electronic device provided in the embodiment of the present invention and the license plate image processing method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0176] An embodiment of the present invention provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the license plate image processing method provided in the above embodiment is implemented.

[0177] It should be noted that the computer-readable medium of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above. The above-mentioned computer-readable medium may be contained in the above-mentioned electronic device; or it may exist alone without being assembled into the electronic device.

[0178] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0179] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0180] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0181] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0182] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0183] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0184] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A license plate image processing method, characterized in that: The method comprises: Determine a first license plate image and a second license plate image, wherein the first license plate image is a license plate image obtained by capturing an image of the vehicle to be detected when the fill light is turned on, and the second license plate image is a license plate image obtained by capturing the vehicle to be detected when the fill light is turned off; Performing license plate image fusion based on the first license plate image and the second license plate image to obtain a third license plate image; Extracting high-frequency information from the first license plate image, the second license plate image, and the third license plate image and fusing them to obtain reference high-frequency information; Low-frequency information is extracted from the reference license plate image to obtain reference low-frequency information, and a fourth license plate image is obtained based on the fusion of the reference high-frequency information and the reference low-frequency information. The reference license plate image is at least one of the first license plate image, the second license plate image and the third license plate image.

2. The method according to claim 1, characterized in that Determining a first license plate image and a second license plate image includes: When a vehicle to be detected driving into a target detection area is detected, a fill light associated with the target detection area is controlled to turn on and flash, and when the fill light is turned on and flashes, an image of the vehicle to be detected entering the target detection area is captured and a license plate image is extracted to obtain a first license plate image; Continuously photographing the vehicle to be detected in real time to obtain multiple candidate vehicle images, and extracting the license plate image based on the multiple candidate vehicle images to obtain a second license plate image, the difference in shape and size of the corresponding license plates in the first license plate image and the second license plate image is less than a preset difference, and the position difference between the identification characters at the same position in the corresponding license plate identifiers in the first license plate image and the second license plate image is less than a preset difference.

3. The method according to claim 2, characterized in that Extracting a license plate image based on multiple candidate vehicle images to obtain a second license plate image includes: Determine a target vehicle image from the plurality of candidate vehicle images, the target vehicle image being a first candidate vehicle image corresponding to when the fill light selected from the plurality of candidate vehicle images has just been adjusted from an on state to an off state, or a second candidate vehicle image corresponding to when the vehicle to be detected selected from the plurality of candidate vehicle images is at the shortest shooting distance from the shooting device; A license plate image is extracted from the target vehicle image to obtain a target license plate image, and a second license plate image is determined based on the target license plate image.

4. The method according to claim 1, characterized in that: The method of fusing the first license plate image with the second license plate image to obtain a third license plate image includes: Inputting the first license plate image and the second license plate image into a license plate image fusion model; The license plate identifier corresponding to the first license plate image and the license plate identifier corresponding to the second license plate image are fused by a license plate image fusion model to obtain a third license plate image; Among them, the license plate image fusion model is a neural network model obtained by training based on a preset license plate image data set, which can fuse license plate images captured at different brightnesses. The preset license plate image data set includes multiple license plate image samples, each of the license plate image samples includes a fifth license plate image, a sixth license plate image and a seventh license plate image. The fifth license plate image is a license plate image collected in an actual scene and has an image quality greater than the preset image quality. The sixth license plate image and the seventh license plate image are both license plate images obtained by performing random blur operations, random color degradation operations and random brightness degradation operations on the fifth license plate image, and the average brightness of the sixth license plate image is lower than the average brightness of the seventh license plate image.

5. The method according to claim 1, characterized in that Extracting high-frequency information from the first license plate image, the second license plate image, and the third license plate image and fusing them to obtain reference high-frequency information includes: By performing low-pass filtering on the first license plate image, the second license plate image and the third license plate image respectively, high-frequency information included in the first license plate image, the second license plate image and the third license plate image is obtained; Determine reference weight information of high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image; the reference weight information of the high-frequency information included in the license plate image is determined based on the first weight information and the second weight information of the license plate image, the first weight information is determined based on the confidence of the identification characters at each position in the license plate identifier corresponding to the license plate image, the greater the confidence of the identification characters at each position in the license plate identifier, the greater the fusion weight of the image area occupied by the identification characters at each position in the license plate identifier corresponding to the license plate image indicated by the first weight information when performing high-frequency information fusion, the second weight information is determined based on the detail richness of each pixel point in the license plate image, the greater the detail richness of the pixel point, the greater the fusion weight of each pixel point in the license plate image indicated by the second weight information when performing high-frequency information fusion, the detail richness of the pixel point is determined based on the pixel point and the pixel changes in the neighborhood around the pixel point; According to the reference weight information of the high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image, the high-frequency information respectively included in the first license plate image, the second license plate image and the third license plate image is fused to obtain the reference high-frequency information.

6. The method according to claim 5, characterized in that Determining reference weight information of high-frequency information respectively included in the first license plate image, the second license plate image, and the third license plate image includes: The first reference image, the second reference image and the third license plate image are respectively used as first license plate images to be processed, and license plate recognition is performed on the identification character at each position in the license plate identifier corresponding to the first license plate image to be processed, so as to obtain the character position of the identification character at each position in the license plate identifier corresponding to the first license plate image to be processed and the confidence corresponding to the identification character, wherein the first reference image is a fused image of high-frequency information and reference low-frequency information included in the first license plate image, and the second reference image is a fused image of high-frequency information and reference low-frequency information included in the second license plate image; Based on the character position of the identification character at each position in the license plate identifier corresponding to the first license plate image to be processed and the confidence level corresponding to the identification character, first weight information of the first license plate image to be processed is determined.

7. The method according to claim 5, characterized in that Determining reference weight information of high-frequency information respectively included in the first license plate image, the second license plate image, and the third license plate image includes: The high-frequency information included in each of the first license plate image, the second license plate image, and the third license plate image is respectively used as the second license plate image to be processed, and for each pixel to be processed in the second license plate image to be processed, the first pixel point and the second pixel point number of each pixel to be processed in the second license plate image to be processed are determined, wherein the first pixel point is a pixel point within a reference neighborhood of the pixel point to be processed, the reference neighborhood is a pixel area of ​​a preset area size centered on the pixel point to be processed, and the second pixel point is a pixel point within the reference neighborhood of the pixel point to be processed for characterizing a license plate identifier; The second weight information of the second license plate image to be processed is determined according to the first pixel number and the second pixel number of each pixel to be processed in the second license plate image to be processed.

8. A license plate image processing device, characterized in that: The device comprises: A license plate image determination module, used to determine a first license plate image and a second license plate image, wherein the first license plate image is a license plate image obtained by capturing an image of the vehicle to be detected when the fill light is turned on, and the second license plate image is a license plate image obtained by capturing the vehicle to be detected when the fill light is turned off; A license plate image fusion module, configured to fuse the first license plate image and the second license plate image to obtain a third license plate image; A reference high-frequency information determination module, used to extract high-frequency information from the first license plate image, the second license plate image and the third license plate image and fuse them to obtain reference high-frequency information; The fourth license plate image determination module is used to extract low-frequency information of the reference license plate image to obtain reference low-frequency information, and to fuse the reference high-frequency information with the reference low-frequency information to obtain a fourth license plate image, wherein the reference license plate image is at least one of the first license plate image, the second license plate image and the third license plate image.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the license plate image processing method described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the license plate image processing method described in any one of claims 1-7 when executed by a computer processor.

Images