Image processing method, device, electronic device and storage medium

Abstract

The embodiment of the present application provides an image processing method, device, electronic device and storage medium. The method includes: calculating the average brightness of the first image and the second image respectively; determining the color template image according to the average brightness of the first image and the second image respectively; determining the area between the characters with the largest spacing in the area where the license plate is located in the target image as the candidate logo area; in the candidate logo area, determining the sub-area that matches the brightness and graphic shape of the preset logo area as the new energy logo area; generating a new energy logo mask for representing the new energy logo area; generating a rivet area mask for representing the rivet area; generating a license plate area mask for representing the area where the license plate is located; calculating the mask weight of each pixel in the second image respectively; performing weighted fusion on the color template image and the second image to obtain a first fused image. The integrity of the license plate information in the license plate image can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer image processing, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] In order to facilitate traffic management, relevant personnel of traffic management usually need to obtain the license plate information of each moving vehicle. Usually, relevant personnel often obtain the license plate information by viewing the license plate image, and the license plate image is obtained by photographing the vehicle with a camera deployed near the road.

[0003] However, due to the restrictions of shooting conditions, the license plate in the license plate image may be overexposed, which is particularly obvious for the area where the rivets are located in the license plate (referred to as the rivet area in this article) and the new energy logo in the new energy license plate, resulting in the rivet area and the new energy logo not being clearly presented in the captured license plate image. As a result, relevant personnel are unable to obtain complete license plate information by viewing the license plate image. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and storage medium to improve the integrity of the license plate information in the license plate image, so as to facilitate traffic management by relevant traffic management personnel. The specific technical solution is as follows:

[0005] The present application provides an image processing method, the method comprising:

[0006] Calculate the average brightness of the first image and the second image respectively; wherein the first image is a bright frame image or an infrared frame image, the second image is a dark frame image, and the first image and the second image are obtained by shooting the same license plate at the same time;

[0007] determining a color template image according to an average brightness of the first image and an average brightness of the second image;

[0008] In the area where the license plate is located in the target image, determine the area between the characters with the largest spacing as the candidate mark area, wherein the characters with the largest spacing are adjacent characters with the largest spacing in the horizontal direction in the area where the license plate is located, and the target image is the one with higher image quality between the first image and the second image;

[0009] In the candidate logo area, a sub-area matching the brightness and graphic shape of the preset logo area is determined as the new energy logo area, wherein the preset logo area is the area where the new energy logo is located in the preset new energy license plate image, and the image quality of the preset new energy license plate image is higher than a preset quality threshold;

[0010] generating a new energy mark mask for representing the new energy mark area;

[0011] generating a rivet region mask for representing the rivet region;

[0012] Obtaining the area where the license plate is located in the second image by edge detection, and generating a license plate area mask for representing the area where the license plate is located;

[0013] Calculating the mask weight of each pixel in the second image respectively, wherein the mask weight is obtained by weighted fusion of the rivet area mask, the new energy logo mask and the second image, the weight of each pixel in the rivet area mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, the weight of each pixel in the new energy logo mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, and the weight of each pixel in the second image is negatively correlated with the pixel value of the pixel at the same position in the license plate area mask;

[0014] The color template image and the second image are weightedly fused to obtain a first fused image; wherein the weight of each pixel in the second image is the mask weight, and the weight of each pixel in the color template image is negatively correlated with the mask weight of the pixel at the same position in the second image.

[0015] The present application also provides an image processing device, the device comprising:

[0016] An average brightness calculation module, used to calculate the average brightness of the first image and the second image respectively; wherein the first image is a bright frame image or an infrared frame image, the second image is a dark frame image, and the first image and the second image are obtained by shooting the same license plate at the same time;

[0017] A color template image determination module, used to determine a color template image according to an average brightness of the first image and an average brightness of the second image;

[0018] A candidate sign area determination module is used to determine, within the area where the license plate is located in the target image, an area between characters with the largest spacing as the candidate sign area, wherein the characters with the largest spacing are adjacent characters with the largest spacing in the horizontal direction within the area where the license plate is located, and the target image is the one with higher image quality between the first image and the second image;

[0019] A new energy sign area determination module is used to determine, in the candidate sign area, a sub-area that matches the brightness and graphic shape of the preset sign area as the new energy sign area, wherein the preset sign area is the area where the new energy sign is located in the preset new energy license plate image, and the image quality of the preset new energy license plate image is higher than a preset quality threshold;

[0020] A new energy symbol mask generation module, used to generate a new energy symbol mask for representing the new energy symbol area;

[0021] A rivet area mask generating module, used for generating a rivet area mask for representing a rivet area;

[0022] A license plate area mask generation module, used to obtain the area where the license plate is located in the second image through edge detection, and generate a license plate area mask for representing the area where the license plate is located;

[0023] a mask weight calculation module, used to respectively calculate the mask weight of each pixel in the second image, wherein the mask weight is obtained by weighted fusion of the rivet area mask, the new energy logo mask and the second image, the weight of each pixel in the rivet area mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, the weight of each pixel in the new energy logo mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, and the weight of each pixel in the second image is negatively correlated with the pixel value of the pixel at the same position in the license plate area mask;

[0024] The first fused image obtaining module is used to perform weighted fusion on the color template image and the second image to obtain a first fused image; wherein the weight of each pixel in the second image is the mask weight, and the weight of each pixel in the color template image is negatively correlated with the mask weight of the pixel at the same position in the second image.

[0025] The present application also provides an electronic device, including:

[0026] Memory, used to store computer programs;

[0027] The processor is used to implement any of the above-mentioned image processing methods when executing the program stored in the memory.

[0028] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned image processing methods is implemented.

[0029] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned image processing methods.

[0030] Beneficial effects of the embodiments of the present application:

[0031] An image processing method, device, electronic device and storage medium provided in the embodiments of the present application can utilize the characteristic that license plates of different colors present different brightness in the license plate image, thereby reflecting the degree of reflection of the license plate to light according to the average brightness of the bright frame image or the infrared frame image and the average brightness of the dark frame image, and then derive the brightness information of the real license plate in the image according to the degree of reflection. According to the derived brightness information of the real license plate in the image, the color that the license plate should display in the image can be determined, that is, the color template image that can be used to restore the color that the license plate should display in the image is determined. And for new energy signs: utilizing the characteristic that the new energy sign will increase the spacing between the characters on both sides of the new energy sign, by determining the area between the characters with the largest spacing in the area where the license plate is located in the target image, the area where the new energy sign is approximately located is determined as the alternative sign area. Then, a sub-region matching the brightness and graphic shape of the preset logo region is determined in the candidate annotation region. Since the preset logo region is the region where the new energy logo is located in the preset new energy license plate image, and the brightness and graphic shape of the new energy logo in different license plate images are roughly the same, it can be considered that the sub-region matching the brightness and graphic shape of the preset logo region is also the region where the new energy logo is located, thereby determining a more accurate position of the new energy logo from the candidate annotation region as the new energy logo region, and then generating a new energy logo mask for representing the new energy logo region. And generate a rivet region mask that can represent the rivet region. The license plate area mask, new energy logo mask and rivet area mask that can represent the area where the license plate is located are then integrated to obtain a mask weight that can distinguish the area where the license plate is located from the area where the license plate is not located, and can distinguish the rivet area, new energy logo area and other remaining areas in the license plate. Based on the mask weight, the color template image and the second image are fused, and the color of the remaining other areas in the license plate can be restored by using the color displayed by the license plate in the color template image, and at the same time, the relevant information about the rivet area and the new energy logo area in the second image is retained in the first fused image obtained by fusion, so that the first fused image not only restores the color displayed by the license plate in the image, but also restores the relevant information of the rivet area and the new energy logo area, thereby improving the integrity of the license plate information in the license plate image, which is convenient for relevant traffic management personnel to perform traffic management.

[0032] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0034] Figure 1 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

[0035] Figure 2a A schematic diagram of a new energy license plate provided in an embodiment of the present application;

[0036] Figure 2b A schematic diagram of a preset transition area in a license plate area mask provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of a flow chart of a method for determining the color of a license plate in an image processing method provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of a process for generating a color template image in the image processing method provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of a flow chart of a method for determining candidate marker regions in an image processing method provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of a pixel column provided in an embodiment of the present application;

[0041] Figure 7a A schematic diagram of a flow chart of a method for determining a first pixel column in an image processing method provided in an embodiment of the present application;

[0042] Figure 7b A schematic diagram of a binary region provided in an embodiment of the present application;

[0043] Figure 7c A schematic diagram of a display result after binarization processing of a target image provided in an embodiment of the present application;

[0044] Figure 8 A schematic diagram of a flow chart of a method for determining a new energy sign area in an image processing method provided in an embodiment of the present application;

[0045] Fig. 9 A schematic diagram of a flow chart of a method for generating a new energy logo mask in an image processing method provided in an embodiment of the present application;

[0046] Fig.10a A schematic diagram of a flow chart of a method for generating a rivet area mask in the image processing method provided in an embodiment of the present application;

[0047] Fig.10b A schematic diagram of a flow chart of a method for obtaining a second fused image in the image processing method provided in an embodiment of the present application;

[0048] Fig.11 Another schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

[0049] Fig.12 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0050] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0052] Since there are some overexposed areas in the license plate image, the information in these areas is lost, and thus the relevant personnel cannot obtain the complete license plate information by viewing the license plate image. In the related art, the license plate image can be processed to restore the lost information. The usual processing method is: the area where the license plate is located in the license plate image is fused with the pure color background image designed for the license plate, and the fusion result is used as the restored license plate image. However, this processing method can only enhance the color based on the original color of the license plate in the license plate image, so that the color of the license plate in the restored license plate image is closer to the color of the real license plate, but it cannot restore the new energy logo and rivet area in the license plate, so that the license plate image restored in this processing method lacks the license plate information related to the new energy logo and rivet area, that is, the relevant personnel still cannot obtain the complete license plate information by viewing the restored license plate image, and thus cannot effectively perform traffic management based on the restored license plate image.

[0053] Another processing method is to take the license plate image with a visible light camera and an infrared camera respectively to obtain a visible light license plate image and an infrared license plate image, and use the infrared license plate image to restore the overexposed area in the visible light license plate image to obtain a restored license plate image. However, in general, the area where the new energy logo and the rivet area are located in the infrared image of the license plate will also be overexposed, so this method cannot restore the lost information in the area where the new energy logo and the rivet area are located in the license plate image.

[0054] Based on this, in order to restore the new energy logo and rivet area in the license plate, improve the integrity of the license plate information in the license plate image, and facilitate traffic management by relevant traffic management personnel, this application provides an image processing method, such as Figure 1 As shown, the method includes:

[0055] S101, respectively calculating the average brightness of the first image and the second image.

[0056] The first image is a bright frame image or an infrared frame image, the second image is a dark frame image, and the first image and the second image are obtained by photographing the same license plate at the same time.

[0057] S102: Determine a color template image according to the average brightness of the first image and the average brightness of the second image.

[0058] S103, in the area where the license plate is located in the target image, determine the area between the characters with the largest spacing as the candidate sign area.

[0059] The characters with the largest interval are adjacent characters with the largest interval in the horizontal direction in the area where the license plate is located, and the target image is the one with higher image quality between the first image and the second image.

[0060] S104: In the candidate logo area, a sub-area matching the brightness and graphic shape of the preset logo area is determined as the new energy logo area.

[0061] Among them, the preset logo area is the area where the new energy logo is located in the preset new energy license plate image, and the image quality of the preset new energy license plate image is higher than the preset quality threshold.

[0062] S105: Generate a new energy mark mask for indicating a new energy mark area.

[0063] S106: Generate a rivet area mask for representing the rivet area.

[0064] S107, obtaining the area where the license plate is located in the second image through edge detection, and generating a license plate area mask for representing the area where the license plate is located.

[0065] S108: Calculate the mask weight of each pixel in the second image respectively.

[0066] Among them, the mask weight is obtained by weighted fusion of the rivet area mask, the new energy logo mask and the second image. The weight of each pixel in the rivet area mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, the weight of each pixel in the new energy logo mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, and the weight of each pixel in the second image is negatively correlated with the pixel value of the pixel at the same position in the license plate area mask.

[0067] S109, performing weighted fusion on the color template image and the second image to obtain a first fused image.

[0068] The weight of each pixel in the second image is the mask weight, and the weight of each pixel in the color template image is negatively correlated with the mask weight of the pixel at the same position in the second image.

[0069] By using the embodiments of the present application, the characteristic that license plates of different colors present different brightness in the license plate image can be utilized, so that the degree of reflection of the license plate to light can be reflected according to the average brightness of the bright frame image or the infrared frame image and the average brightness of the dark frame image, and then the brightness information of the real license plate in the image can be derived according to the degree of reflection. According to the derived brightness information of the real license plate in the image, the color that the license plate should display in the image can be determined, that is, the color template image that can be used to restore the color that the license plate should display in the image can be determined. And for the new energy logo: the characteristic that the new energy logo will increase the spacing between the characters on both sides of the new energy logo is utilized, and the area between the characters with the largest spacing in the area where the license plate is located in the target image is determined, and the area where the new energy logo is approximately located is determined as the alternative logo area. Then, a sub-region matching the brightness and graphic shape of the preset logo region is determined in the candidate annotation region. Since the preset logo region is the region where the new energy logo is located in the preset new energy license plate image, and the brightness and graphic shape of the new energy logo in different license plate images are roughly the same, it can be considered that the sub-region matching the brightness and graphic shape of the preset logo region is also the region where the new energy logo is located, thereby determining a more accurate position of the new energy logo from the candidate annotation region as the new energy logo region, and then generating a new energy logo mask for representing the new energy logo region. And generate a rivet region mask that can represent the rivet region. The license plate area mask, new energy logo mask and rivet area mask that can represent the area where the license plate is located are then integrated to obtain a mask weight that can distinguish the area where the license plate is located from the area where the license plate is not located, and can distinguish the rivet area, new energy logo area and other remaining areas in the license plate. Based on the mask weight, the color template image and the second image are fused, and the color of the remaining other areas in the license plate can be restored by using the color displayed by the license plate in the color template image, and at the same time, the relevant information about the rivet area and the new energy logo area in the second image is retained in the first fused image obtained by fusion, so that the first fused image not only restores the color displayed by the license plate in the image, but also restores the relevant information of the rivet area and the new energy logo area, thereby improving the integrity of the license plate information in the license plate image, which is convenient for relevant traffic management personnel to perform traffic management.

[0070] That is to say, the embodiment of the present application can design a color template according to the brightness change of the license plate background color (i.e., determine the color template image through the aforementioned S101-S102), and restore the image features of the key areas by locating the key areas (i.e., the rivet area and the new energy logo area) to generate fusion weights (i.e., mask weights). At the same time, the embodiment of the present application can generate images and restore colors of the new energy logo area and rivet area of ​​the overexposed infrared frame license plate image.

[0071] The above S101-S109 will be described exemplarily below:

[0072] In S101 , the first image and the second image may be images in a YUV (Luminance Chrominance) format or an RGB (Red, Green, Blue) format, and the sizes of the first image and the second image should be consistent.

[0073] In one possible embodiment, the first image is a bright frame image. In another possible embodiment, the first image is an infrared frame image. In yet another possible embodiment, the image qualities of the bright frame image and the infrared frame image can be judged, and the one with higher image quality is used as the first image.

[0074] Furthermore, for the embodiment in which the first image is a bright frame image, an infrared frame image may or may not be acquired. For the embodiment in which the first image is an infrared frame image, a bright frame image may or may not be acquired. For the aforementioned embodiment in which the one with higher image quality is used as the first image, it is necessary to acquire a bright frame image and an infrared frame image respectively. In all these embodiments, it is necessary to acquire a dark frame image.

[0075] For the convenience of description, the following only explains the situation where a bright frame image, an infrared frame image, and a dark frame image need to be acquired separately. For the situation where only a bright frame image and a dark frame image need to be acquired, or only an infrared frame image and a dark frame image need to be acquired, the principle is the same and will not be repeated here.

[0076] In the present application, a visible light camera for shooting visible light images and an infrared camera for shooting infrared light images may be set on the channel, and a white light strobe fill light is configured for the visible light camera, and an infrared strobe fill light is configured for the infrared camera. The same vehicle is photographed simultaneously by a visible light camera and an infrared camera to obtain the bayer data output by the visible light camera (hereinafter referred to as white light Bell data), and the bayer data output by the infrared camera (hereinafter referred to as infrared Bell data) is obtained, and the white light Bell data is input into the ISP (Image Signal Processor). After image processing in the ISP, the original bright frame image and the original dark frame image can be obtained. The red light Bell data is input into the ISP, and after image processing in the ISP, the original infrared frame image can be obtained. Simultaneous shooting in this article refers to shooting within the same time window, and the length of the time window should be less than the preset time threshold.

[0077] The original bright frame image, the original dark frame image and the original infrared frame image can be directly used as the bright frame image, the dark frame image and the infrared frame image, respectively. The original bright frame image, the original dark frame image and the original infrared frame image can also be further processed, and the processing results are used as the bright frame image, the dark frame image and the infrared frame image, respectively.

[0078] Exemplarily, the processing method may be: respectively cut out the area where the license plate is located in the original light frame image, the original dark frame image, and the original infrared frame image to obtain a cropped light frame image, a cropped dark frame image, and a cropped infrared frame image, and respectively adjust the area where the license plate is located in the cropped light frame image, the cropped dark frame image, and the cropped infrared frame image, so that the areas where the license plate is located in the adjusted cropped light frame image, the cropped dark frame image, and the cropped infrared frame image are the same, and the adjusted cropped light frame image, the cropped dark frame image, and the cropped infrared frame image are respectively used as the light frame image, the dark frame image, and the infrared frame image. The following will take this example as an example, that is, take the area where only the license plate is located in the light frame image, the dark frame image, and the infrared frame image as an example to exemplarily illustrate the image processing method provided by the present application.

[0079] Furthermore, it can be understood that, in the case where only bright frame images and dark frame images need to be acquired, an infrared camera may not be deployed.

[0080] In S102, it can be understood that the license plate whose real color is green, yellow, yellow-green, etc. is prone to overexposure, so that the color displayed by the license plate in the license plate image is different from the color that the license plate should be displayed in the image. In order to restore the color that the license plate should be displayed in the image, it is necessary to determine the color that the real license plate should be displayed in the image. Because the license plate colors are different, the degree of reflection of the license plate to light is different, which leads to different brightness of the license plate in the license plate image. Therefore, the degree of reflection of the license plate to light can be reflected according to the average brightness of the first image and the second image, and then the brightness information of the real license plate in the image can be derived according to the reflection degree. According to the derived brightness information of the real license plate in the image, the color that the license plate should be displayed in the image can be determined, and the color that the license plate should be displayed in the image is the color of each pixel in the color template image.

[0081] The size of the color template image should be consistent with the size of the first image and the second image. The following will provide an exemplary description of how to determine the color template image, which will not be repeated here.

[0082] In S103, the image quality detection algorithm is used to determine the one with higher image quality between the first image and the second image as the target image. For example, the effective clarity of the font on the license plate in the first image and the second image can be calculated by a preset font clarity algorithm, and the one with higher clarity is considered to be the image with higher quality and used as the target image.

[0083] See also Figure 2a , Figure 2a This is a schematic diagram of a new energy license plate. Figure 2a In the license plate, "XX" is the text and letters indicating the province and city to which the vehicle belongs, "YY" is the letters in the license plate, and "ZZZZ" is the numbers in the license plate. Figure 2a It can be seen that the area where the characters in the license plate are spaced relatively large in the horizontal direction, that is, the area between the characters "XX" and "YY", is where the new energy logo 21 on the license plate is located. Therefore, if the area between adjacent characters with the largest horizontal spacing in the area where the license plate is located in the target image is determined, that is, the area between the characters with the largest spacing is determined, the area where the new energy logo 21 may be located can be determined as the alternative logo area. The following will provide an exemplary explanation of how to determine the alternative logo area, which will not be repeated here. The license plate images in this application are all obtained with the knowledge and permission of the personnel, and the purpose of obtaining the license plate images in this application is to facilitate the relevant personnel of traffic management to carry out traffic management in order to improve traffic safety.

[0084] In S104, it can be understood that the candidate mark area determined in step S103 is the area between the characters with the largest interval, and Figure 2a As shown, the area between the maximum interval characters (i.e., the area between the second X and the first Y) includes not only the area where the new energy logo is located, but also other areas. Therefore, in order to further accurately determine the area where the new energy logo is located in the target image, the area where the new energy logo is located in the preset new energy license plate image with higher image quality can be used as the preset logo area, and the preset logo area is matched with the sub-area in the candidate logo area in terms of brightness and graphic shape. Then, the sub-area that matches the brightness and graphic shape of the preset logo area can be regarded as the area where the new energy logo is located in the target image, that is, the new energy logo area.

[0085] The preset new energy license plate image with higher image quality herein refers to: a new energy license plate image with image quality higher than a preset quality threshold. The preset quality threshold can be set according to actual needs or past experience, and this application does not impose any restrictions on this.

[0086] If the size ratio between the area where the license plate in the new energy license plate image is located and the area where the license plate in the target image is located is e:f, then multiply the preset logo area size by f / e, and use the product as the matching box size. Select multiple sub-areas with the size of the matching box in the alternative logo area, and calculate the matching degree between the brightness of each sub-area and the brightness of the preset logo area. The sub-area with the largest corresponding matching degree is used as the sub-area that matches the brightness and graphic shape of the preset logo area, that is, as the new energy logo area.

[0087] To facilitate matching, in a possible embodiment, the new energy license plate image can be scaled first so that the area where the license plate is located in the new energy license plate image is the same size as the area where the license plate is located in the target image. In this embodiment, the preset logo area size is used as the matching box size, and then the above-mentioned matching process is performed.

[0088] Specifically, the matching degree between the brightness of the sub-region and the brightness of the preset logo region is determined in the candidate logo region by the following formula (1), including:

[0089]

[0090] Among them, the matching degree between the brightness of the SSIM (x, y) sub-region and the brightness of the preset mark region; x is the sub-region; y is the preset mark region; is the mean value of the pixel values ​​of all pixels in the sub-region; is the mean value of the pixel values ​​of all pixels in the preset mark area; is the covariance of the pixel values ​​of all pixels in the sub-region and the pixel values ​​of all pixels in the preset mark region; is the standard deviation of the pixel values ​​of all pixels in the sub-region; is the standard deviation of the pixel values ​​of all pixels in the preset mark area; c1 and c2 are parameters, and c1=c2=1. The setting of c1 is to avoid the situation where the denominator is equal to 0.

[0091] It is calculated by the following formula (2):

[0092]

[0093] It is calculated by the following formula (3):

[0094]

[0095] In formula (2) and formula (3), x i is the pixel value of the i-th pixel in the sub-region; N1 is the number of pixels in the sub-region.

[0096] It is calculated by the following formula (4):

[0097]

[0098] It is calculated by the following formula (5):

[0099]

[0100] In formula (4) and formula (5), y i is the pixel value of the i-th pixel in the preset mark area; N2 is the number of pixels in the preset mark area. If the size of the preset mark area is used as the size of the matching box, the size of the sub-area selected by the matching box in the candidate mark area is the same as the size of the preset mark area. At this time, in formula (3) and formula (5), the number of pixels in the preset mark area is the same as the number of pixels in the sub-area, that is, N1=N2. In this embodiment, It is calculated by the following formula (6):

[0101]

[0102] In formula (6), x i is the pixel value of the i-th pixel in the sub-region; N1 is the number of pixels in the sub-region, that is, the number of pixels in the preset mark region; y i is the pixel value of the i-th pixel in the preset mark area.

[0103] The method of generating an image or mask in the present application may be: first generate a blank image of a fixed size, and then set the pixel value of each pixel in the blank image to obtain an image or mask. Alternatively, the method of generating an image or mask in the present application may be: first calculate the pixel value of each pixel in the image or mask to be generated, and record the position of each pixel and the corresponding pixel value accordingly, and generate an image or mask according to the position of each pixel and the corresponding pixel value. The method of generating an image or mask will not be described in detail below.

[0104] In S105, the size of the new energy logo mask should be consistent with the size of the first image and the second image. When masking the new energy logo, the pixel values ​​of the pixels in the new energy logo mask located in the new energy logo area can be set to 0, and the pixel values ​​of the pixels in the new energy logo mask that are not located in the new energy logo area can be set to 255; or, the pixel values ​​of the pixels in the new energy logo mask located in the new energy logo area can be set to 255, and the pixel values ​​of the pixels in the new energy logo mask that are not located in the new energy logo area can be set to 0. For the convenience of description below, the image processing method provided in this application is exemplarily described by taking the example of setting the pixel values ​​of the pixels in the new energy logo mask located in the new energy logo area to 255 and setting the pixel values ​​of the pixels in the new energy logo mask that are not located in the new energy logo area to 0.

[0105] In S106, see Figure 2a In the new energy license plate image shown, the area 22 where the four rivets in the license plate are located is the rivet area. The following will provide an exemplary description of how to generate a rivet area mask for representing the rivet area, which will not be repeated here.

[0106] In S107, the four vertices of the license plate in the second image can be obtained by edge detection algorithm, and the boundary of the license plate in the second image can be obtained by connecting the four vertices, and the area surrounded by the boundary of the license plate is the area where the license plate is located. The size of the license plate area mask is consistent with the size of the second image. In the license plate area mask, the pixel values ​​of the pixels located on the boundary of the area where the license plate is located and within the boundary can be set to 255, and the pixel values ​​of the pixels located outside the boundary of the license plate can be set to 0.

[0107] Since the license plate image is restored by fusing the color template image and the second image in the present application, if during the fusion process the pixel values ​​of all pixels in the license plate area adopt the pixel values ​​of the corresponding pixels in the color template image, and the pixel values ​​of all pixels outside the license plate area adopt the pixel values ​​of the corresponding pixels in the second image, then due to the inconsistency in the visual effects of the second image and the color template image, there will be an obvious visual fragmentation near the boundary of the license plate area in the restored license plate image (i.e., the first fused image), affecting the image quality of the restored license plate image. Based on this, in a possible embodiment, a license plate area mask for representing the area where the license plate is located can be generated, and the pixel values ​​of the pixels outside the boundary of the license plate in the license plate area mask are 0, the pixel values ​​of the pixels within the boundary of the license plate and within the preset transition area are positively correlated with the distance between the pixel and the boundary and are not greater than the preset pixel value threshold, and the pixel values ​​of the pixels within the boundary of the license plate and not within the preset transition area are the preset pixel value threshold.

[0108] The preset pixel value threshold can be set according to actual needs or past experience, and this application does not impose any restrictions on this. For example, the preset pixel value threshold can be set to 255, 250, 245, etc.

[0109] For example, assuming that the upper, lower, left, and right boundaries of the license plate are translated by four pixels into the area where the license plate is located to obtain a transition boundary, the area between the transition boundary and the boundary of the license plate is regarded as the transition area, and assuming that the preset pixel value threshold is 255, then along the direction in which the distance between the pixel point and the boundary of the license plate increases, the pixel values ​​of the pixel points located within the boundary of the license plate and within the preset transition area can be set to 63, 127, 191, and 255 respectively. Figure 2b As shown, 23 is the transition boundary, 24 is the boundary of the license plate, and the area between 23 and 24 is the transition area. Figure 2b The small squares shown in the figure are pixels. The pixel value of the pixels filled with a check mark is 63, the pixel value of the pixels filled with a wrong sign "×" is 127, the pixel value of the pixels filled with a triangle symbol is 191, and the pixel value of the pixels filled with a circle symbol is 255.

[0110] In S108, as described above, since the pixel value of the pixel points outside the license plate area in the license plate area mask is 0, the license plate area mask is used as the weight of the new energy logo mask in the weighting process, and the pixel value of the pixel points outside the license plate area in the new energy logo mask can be set to 0. As described above, the pixel points with a pixel value of 0 in the new energy logo mask will be regarded as pixels outside the new energy logo area. Therefore, the area outside the license plate area can be removed from the new energy logo area represented by the new energy logo mask in this way. It can be understood that both the new energy logo area and the rivet area should be located in the license plate area. Therefore, the area outside the license plate area of ​​the new energy logo area represented by the new energy logo mask can be regarded as an inaccurate new energy logo area. That is, this weighting method can make the obtained mask weight more accurately represent the new energy logo area. Similarly, the obtained mask weight can also more accurately represent the rivet area.

[0111] Specifically, the mask weight of each pixel in the second image is calculated by the following formula (7):

[0112]

[0113] Where wt2 is the mask weight of the pixel in the second image; p screws is the pixel value of the pixel at the same position in the rivet area mask; p sign is the pixel value of the pixel at the same position in the new energy logo mask; p mask Y is the pixel value of the pixel at the same position in the license plate area mask;ori is the pixel value of the pixel in the second image under the Y channel.

[0114] In S109, since the weight of each pixel in the rivet area mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, and the weight of each pixel in the new energy logo mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, if a certain pixel is in the rivet area or the new energy logo area, the greater the mask weight of the pixel, the more it is necessary to use the rivet area information or the new energy logo area information contained in the second image at the pixel. If a certain pixel is not in the rivet area or the new energy logo area, the smaller the mask weight of the pixel, the more it is necessary to reflect the color information of the license plate itself at the pixel, that is, the more it is necessary to use the information in the color template image at the pixel. Therefore, when weighted fusion is performed on the color template image and the second image, the weight of each pixel in the second image is the mask weight, and the weight of each pixel in the color template image is negatively correlated with the mask weight of the pixel at the same position in the second image.

[0115] Specifically, the color template image and the second image are weighted fused by the following formulas (8), (9) and (10) to obtain a first fused image:

[0116]

[0117] Where wt2 is the mask weight of the pixel in the second image; Y final is the pixel value of the pixel at the same position in the first fused image under the Y channel; dye is the pixel value of the pixel at the same position in the color template image under the Y channel; ori is the pixel value of the pixel in the second image under the Y channel; U final is the pixel value of the pixel at the same position in the first fused image under the U channel; U dye is the pixel value of the pixel at the same position in the color template image under the U channel; U ori is the pixel value of the pixel in the second image under the U channel; V final is the pixel value of the pixel at the same position in the first fused image under the V channel; V dye is the pixel value of the pixel at the same position in the color template image under the V channel; V ori is the pixel value of the pixel in the second image under the V channel.

[0118] Since the pixel value of a pixel point ranges from [0 to 255], when the pixel value of a pixel point is a binary number, the above formulas (7), (8), (9) and (10) need to be The calculation method is to divide the binary value obtained by 2 8 , that is, divided by 256, so that the pixel value of the pixel point in the calculated first fused image is in the range [0, 255]. They all have this meaning and will not be elaborated in the following text.

[0119] S101-S109 have been described above by way of example. Hereinafter, how to determine the color template image in S102 will be described by way of example. Figure 3 , the aforementioned S102 includes:

[0120] S1021, performing weighted summation on the average brightness of the first image and the average brightness of the second image to obtain a brightness weight as a target brightness weight.

[0121] The weight of the average brightness of the first image is negatively correlated with the average brightness of the second image, and the weight of the average brightness of the second image is positively correlated with the average brightness of the second image.

[0122] Specifically, the target brightness weight is calculated by the following formula (11):

[0123]

[0124] Among them, f luma is the target brightness weight; y_avg nomal is the average brightness of the first image; y_avg drk is the average brightness of the second image.

[0125] S1022, determining a color influencing factor of the license plate according to the target brightness weight and the license plate color.

[0126] As described in S102 above, the degree of reflection of the license plate on light can be reflected according to the average brightness of the first image and the second image, and then the brightness information of the real license plate in the image can be derived according to the degree of reflection. In other words, the color influence factor can be determined according to the average brightness of the first image and the average brightness of the second image, and the color influence factor can reflect the brightness information of the real license plate color in the image.

[0127] Specifically, the color influence factor of the license plate is calculated by the following formulas (12) and (13):

[0128]

[0129] Among them, b_fac is the color influencing factor of the license plate under the B channel; f lumais the target brightness weight; r_fac is the color influencing factor of the license plate under the R channel; a, b, and c are all parameters set according to the color of the license plate.

[0130] S1023, determining a color template image according to the license plate color and the color influencing factor in the first image or the second image.

[0131] As described in S102 above, the size of the color template image should be consistent with the size of the first image and the second image. When generating a color template image, a blank image of the same size as the first image can be generated first, and then the pixel value of each pixel in the blank image is calculated according to the license plate color and the color influence factor in the first image or the second image, so as to obtain the color template image. It is also possible to first calculate the pixel value of each pixel in the color template image according to the license plate color and the color influence factor in the first image or the second image, and record the corresponding relationship between the pixel point and the pixel value, and directly generate the color template image based on the corresponding relationship without generating a blank image, and the pixel value of each pixel in the color template image should be consistent with the recorded corresponding relationship.

[0132] By selecting this embodiment, the characteristic that license plates of different colors present different brightness in the license plate image can be utilized, so as to generate a color influence factor that changes with brightness according to the average brightness of the bright frame image or the infrared frame image and the average brightness of the dark frame image, and reflect the brightness information of the license plate color through the color influence factor, and then determine the color that the license plate should display in the image according to the color of the license plate in the first image or the second image and the color influence factor, that is, determine the color template image that can be used to restore the color that the license plate should display in the image. In addition, the average brightness of the first image and the average brightness of the second image can be weighted and summed to obtain a brightness weight as a target brightness weight, wherein the weight of the average brightness of the first image is negatively correlated with the average brightness of the second image, and the weight of the average brightness of the second image is positively correlated with the average brightness of the second image. According to the target brightness weight and the license plate color, the color influence factor of the license plate can be more accurately determined, so that the color displayed by the license plate in the image can be better restored based on the color influence factor of the license plate, and a more accurate color template image can be obtained, so that the color of the license plate in the first fused image is more accurate, which is more convenient for traffic management personnel to perform traffic management.

[0133] In the above, an exemplary description has been given of how to determine the color template image in the aforementioned S102. It can be understood that the actual color of some license plates may be gradient. Based on this, in order to better reflect the actual gradient color of the license plate, in a possible embodiment, the aforementioned S1023 includes:

[0134] A color template image is determined according to the license plate color and the color influencing factor in the first image or the second image, wherein the color depth of each pixel in the color template image decreases with the distance between the pixel and the lower boundary of the license plate.

[0135] See also Figure 2a The new energy license plate image shown, in which the greater the distance between the pixel point in the area where the license plate is located and the lower boundary of the license plate, the lighter the color depth of the pixel point. Therefore, the color of each pixel point in the color template image is the color displayed by the license plate in the image, and the color depth decreases with the distance between the pixel point and the lower boundary of the license plate.

[0136] Specifically, the pixel value of each pixel in the color template image is calculated by the following formula (14), formula (15), and formula (16):

[0137]

[0138] In formula (14), formula (15), and formula (16), RGB_B final is the pixel value of the pixel in the color template image under the B channel; rgb_max is the grayscale value of the pixel at the same position in the first image or the second image; b_fac is the color influencing factor of the license plate under the B channel; RGB_G final is the pixel value of the pixel in the color template image under the G channel; RGB_R final is the pixel value of the pixel in the color template image under the R channel; r_fac is the color influencing factor of the license plate under the R channel.

[0139] By converting the first image or the second image in RGB format into a grayscale image, the pixel value of the pixel in the grayscale image at the same position as the pixel in the color template image is used as the grayscale value rgb_max of the pixel at the same position in the first image or the second image. wt1 is calculated using the following formula (17):

[0140]

[0141] Among them, CLIP() is a function used to limit the value of wt1 to the range of [0, 256]; d1 is the distance between the pixel point and the lower boundary of the license plate, that is, the vertical coordinate of the pixel point; height is the height of the license plate. In this article, the vertical coordinate of the pixel point refers to: the vertical coordinate of the pixel point in the image coordinate system established with the lower left corner of the image as the origin, the horizontal right direction as the X axis, and the vertical upward direction as the Y axis. The height of the license plate in this article refers to the number of pixels occupied by the license plate in the vertical direction in the image. Similarly, the distance between the pixel point and the lower boundary of the license plate refers to the number of pixels between the pixel point and the pixel point on the lower boundary of the license plate in the vertical direction. Similarly, other distances, heights and other dimensions in this article are measured by the number of pixels.

[0142] By selecting this embodiment, a color template image can be determined based on the license plate color and color influencing factors in the first image or the second image. The pixel value of each pixel in the color template image is calculated in such a way that the depth of the color of each pixel decreases with the distance between the pixel and the lower boundary of the license plate, that is, the color of each pixel is calculated, so that the color template image reflects the color of the license plate displayed in the image as the brightness and height change, better restores the visual effect of the license plate, and obtains more accurate license plate color information, thereby facilitating relevant traffic management personnel to perform traffic management based on more accurate license plate information.

[0143] Combination Figure 1 , Figure 3 As well as the embodiments shown in formula (11) to formula (17), the method of generating a color template image in this application can be referred to Figure 4 As shown, including:

[0144] S401, color image.

[0145] The color image is an RGB image, and S401 is to obtain a second image in RGB format.

[0146] S402, grayscale processing of color image.

[0147] S402 is to convert the second image in RGB format into a grayscale image.

[0148] S403, calculating the average brightness of the bright frame and the dark frame.

[0149] S403 is equivalent to the aforementioned S101, and reference may be made to the relevant description of the aforementioned S101, which will not be repeated here.

[0150] S404, calculating the factor by which the weight changes with brightness.

[0151] The factor by which the weight changes with brightness in S404 is the aforementioned target brightness weight. S404 is equivalent to the aforementioned S1021. Please refer to the relevant description of the aforementioned S1021, which will not be repeated here.

[0152] S405, calculating a B domain color factor according to the brightness change.

[0153] S406, calculating the R domain color factor according to the change in the dark frame brightness.

[0154] The B domain color factor according to the brightness change in S405 is b_fac in the above formula (12), and the R domain color factor according to the brightness change of the dark frame in S406 is r_fac in the above formula (13). S405-S406 are equivalent to the above S1022, and the relevant description of the above S1022 can be referred to, which will not be repeated here.

[0155] S407, pixel height d1 and license plate height.

[0156] S408, calculating the factor by which the weight changes with the height.

[0157] The pixel height d1 in S407 is the ordinate d1 of the pixel point in the aforementioned formula (17), and the license plate height is the height height of the license plate in the aforementioned formula (17). The factor of the weight varying with the height in S408 is wt1 in the formula (17). S407-S408 can refer to the relevant description of the aforementioned formula (17), which will not be repeated here.

[0158] S409, color template generation.

[0159] The color template in S409 is the aforementioned color template image. S409 can refer to the relevant description of the aforementioned formula (14) to formula (17), which will not be repeated here.

[0160] The above has been described in an exemplary manner for generating a color template image. The following will be described in an exemplary manner for determining the candidate marker region in the aforementioned S103.

[0161] In a possible embodiment, the area where all characters in the license plate in the target image are located can be identified by text recognition, and the area between two adjacent characters with the largest interval in the horizontal direction can be used as the candidate mark area.

[0162] In another possible embodiment, see Figure 5 , S103 includes:

[0163] S1031, determining a pixel column that does not contain any characters in the area where the license plate is located in the target image as a first pixel column.

[0164] The area where the characters are located in the target image can be obtained by means of text recognition, and then a pixel column that does not contain the characters can be obtained as the first pixel column.

[0165] S1032 , respectively determine the number of first pixel columns existing between each adjacent second pixel column as the first number corresponding to each adjacent second pixel column.

[0166] The second pixel column is a pixel column other than the first pixel column in the area where the license plate is located in the target image.

[0167] For example, see Figure 6 , assuming that there are three second pixel columns, namely: second pixel column 1, second pixel column 2, second pixel column 3, then adjacent second pixel columns refer to: second pixel column 1 and second pixel column 2, second pixel column 2 and second pixel column 3. There are 3 first pixel columns between second pixel column 1 and second pixel column 2, then the first number corresponding to second pixel column 1 and second pixel column 2 is 3; there are 5 first pixel columns between second pixel column 2 and second pixel column 3, then the first number corresponding to second pixel column 2 and second pixel column 3 is 5.

[0168] S1033, determining the corresponding adjacent second pixel columns with the largest first number, and determining the area between the adjacent second pixel columns as the candidate marker area.

[0169] Based on the example shown in S1032, the first number corresponding to the second pixel column 1 and the second pixel column 2 is 3, and the first number corresponding to the second pixel column 2 and the second pixel column 3 is 5, then, the adjacent second pixel columns with the largest first number are the second pixel column 2 and the second pixel column 3, that is, the area between the second pixel column 2 and the second pixel column 3 is the alternative mark area.

[0170] By selecting this embodiment, a pixel column that does not contain characters in the area where the license plate is located in the target image can be determined as the first pixel column, and the number of first pixel columns between each adjacent second pixel column can be determined as the first number corresponding to each adjacent second pixel column. The second pixel column is the pixel column other than the first pixel column in the area where the license plate is located in the target image, and the adjacent second pixel columns with the largest corresponding first number are determined. The area between the adjacent second pixel columns is determined as the alternative sign area, so as to more accurately determine the alternative sign area, thereby improving the accuracy of the determined new energy sign area and the new energy sign mask, so that the license plate information contained in the first fused image finally obtained by fusion is more accurate and more convenient for traffic management.

[0171] In one possible implementation, see Figure 7a In the aforementioned S1031, the first pixel column may also be determined in the following manner:

[0172] S701, binarization is performed on the area where the license plate is located in the target image to obtain a binarized area.

[0173] The area where the license plate is located in the target image can be binarized by local binarization or global binarization, and the area where the license plate is located after the binarization is used as the binarized area. Figure 7b shown. Figure 7b 71 in is the new energy sign retained after the target image is binarized. It can be understood that Figure 7b This is only a possible binary region diagram, and does not mean that after the target image is binarized, the letters, words, or numbers in the license plate in the target image can be clearly and completely displayed. For example, if the license plate in the target image includes the number 2, then after the target image is binarized, Figure 7c As shown, the horizontal bar below the number 2 may not be displayed.

[0174] S702: For each pixel point in the binary region, calculate a first distance between the pixel point and an upper boundary and / or a lower boundary of the binary region.

[0175] S703: In the binary area, determine a pixel point whose first distance is within a preset distance range as a first pixel point.

[0176] See also Figure 2a In the new energy license plate image shown, the new energy logo is located at about half the height of the license plate in terms of height, and is located at the left half of the license plate in terms of width. Therefore, the area where the new energy logo may be located can be determined in the vertical direction through the method of S702-S703.

[0177] The first distance may refer to the distance between a pixel point in the binary area and the upper boundary of the binary area, or the distance between a pixel point in the binary area and the lower boundary of the binary area, or the distance between a pixel point in the binary area and the upper boundary and the lower boundary of the binary area.

[0178] Since the new energy logo is located at about half the height of the license plate, the preset distance range can be set to [height / 4, 3×height / 4], where height is the height of the license plate. If the first distance of a pixel point is within the preset distance range, it can be considered that the pixel point is located in the area where the new energy logo may exist, and the pixel point can be used as the first pixel point.

[0179] S704 , respectively determine the number of first pixel points whose pixel values ​​are not 0 and are included in each pixel column in the binary area as the second number corresponding to each pixel column.

[0180] In the binary area, when the pixel value of a pixel is 0, it means that the pixel is a pixel constituting a character, and when the pixel value of a pixel is not 0, it means that the pixel is a pixel constituting a character. Determining the number of first pixels with non-zero pixel values ​​included in each pixel column is to determine the number of first pixels that can constitute a character included in each pixel column.

[0181] S705 , determining, in each pixel column, a pixel column whose corresponding second number is less than a preset number threshold as a first pixel column.

[0182] The preset number threshold can be set according to actual needs or past experience, and this application does not impose any restrictions on this. If the second number corresponding to a certain pixel column is not less than the preset number threshold, the pixel column includes more pixels that can form characters, that is, the pixel column contains characters, then the pixel column cannot be used as the first pixel column; if the second number corresponding to a certain pixel column is less than the preset number threshold, the pixel column includes fewer or no pixels that can form characters, that is, the pixel column does not contain characters, then the pixel column can be used as the first pixel column.

[0183] Combination Figure 1 , Figure 5 , Figure 6 as well as Figure 7a In the embodiment shown, the method for determining the new energy mark area in this application can be as follows Figure 8 As shown, see Figure 8 , the methods for determining the new energy sign area include:

[0184] S801, determining the scope of the key area according to the license plate features.

[0185] The key area in S801 is the area where the new energy logo is located.

[0186] S802, narrowing the mark range by reverse inference using a binarization method.

[0187] The aforementioned candidate marker area can be obtained through S802, and S801-S802 are equivalent to the aforementioned S701-S705 and S1031-S1033. Please refer to the relevant descriptions in the aforementioned S701-S705 and S1031-S1033 and no further details will be given here.

[0188] S803, accurately locate the key area by using a template matching method.

[0189] The key area in S803 is the area where the new energy sign is located. S803 is equivalent to the aforementioned S104. Please refer to the relevant description of the aforementioned S104, which will not be repeated here.

[0190] The above has been described in an exemplary manner on how to determine the new energy mark area. The following will be described in an exemplary manner on how to generate the new energy mark mask in the aforementioned S105. Fig. 9 , the aforementioned S105 includes:

[0191] S1051, generating an alternative mask according to the new energy mark area.

[0192] Among them, the pixel value of each pixel point in the alternative mask is calculated in the following way: a pixel area with a pixel point at the same position as the center and a preset length as the side length is determined in the new energy logo area, and the average pixel value of each pixel point in the pixel area is calculated as the pixel value of the pixel point.

[0193] Since the difference between the pixel values ​​of each pixel in the determined new energy logo area may be large, if the new energy logo area is directly used as the new energy logo mask to obtain the first fused image, it may cause obvious color blocks in the first fused image. In order to avoid the above problem as much as possible, it is necessary to blur the edge features of the new energy logo area to reduce the obvious difference between the pixel values ​​of each pixel in the new energy logo area.

[0194] The edge features of the new energy sign area can be blurred by the box filter shown in S1051. Specifically, the edge features of the new energy sign area are blurred by the following formula (18) to generate an alternative mask:

[0195]

[0196] Among them, result(row, col) is the pixel value of the pixel point (row, col) in the alternative mask; n is the number of pixels in the pixel area; patch is the preset length; image(i, j) is the pixel value of the pixel point (i, j) in the pixel area; the pixel area is the area in the new energy logo area with the pixel point (row, col) as the center and the preset length patch as the side length.

[0197] S1052, normalizing the candidate masks to obtain a new energy mark mask.

[0198] Since formula (18) only blurs the edge features of the new energy logo area, but the pixel value after blurring is small, it may cause the weight of the new energy logo area to be small in the subsequent fusion process, so that the new energy logo information cannot be clearly reflected in the fusion result. In order to highlight the new energy logo area, the value of the new energy logo mask is required, that is, to increase the weight of the new energy logo area during fusion. The alternative mask can be processed by normalization to obtain the new energy logo mask. Specifically, the alternative mask is normalized by the following formula (19):

[0199]

[0200] Among them, g' is the pixel value of the pixel in the new energy logo mask; g is the pixel value of the pixel at the same position in the alternative mask; min(g) is the minimum value of the pixel values ​​of all pixels in the alternative mask; max(g) is the maximum value of the pixel values ​​of all pixels in the alternative mask.

[0201] Since the value range of the normalized data is [0, 1] and the value range of the pixel value of the pixel point is [0, 255], the normalized data needs to be multiplied by 255 to obtain the pixel value of the pixel point in the new energy logo mask.

[0202] In the above, an exemplary description is given of how to generate a new energy logo mask. In the following, an exemplary description is given of how to generate a rivet area mask for indicating a rivet area in the aforementioned S106. Fig.10a , the aforementioned S106 includes:

[0203] S1061: Generate a first channel filtered image according to the first image.

[0204] The pixel value of each pixel in the first channel filtered image is calculated in the following way: in the first image, the maximum value of the pixel values ​​of the pixel at the same position under each channel is determined as the pixel value of the pixel.

[0205] S1062: Generate a second channel filtered image according to the second image.

[0206] The pixel value of each pixel in the second channel filtered image is calculated in the following manner: in the second image, the maximum value of the pixel values ​​of the pixel at the same position under each channel is determined as the pixel value of the pixel.

[0207] S1063, performing fuzzy filtering on the second channel filtered image to obtain a blurred image.

[0208] S1064, performing weighted fusion on the blurred image and the second channel filtered image to obtain a second fused image.

[0209] Among them, the weight of each pixel in the blurred image is positively correlated with the distance between the pixel and the horizontal center line of the license plate area, and the weight of each pixel in the second channel filtered image is negatively correlated with the distance between the pixel at the same position in the blurred image and the horizontal center line of the license plate area.

[0210] S1065: Generate a rivet area mask for representing the rivet area.

[0211] Among them, the pixel value of each pixel in the rivet area mask is positively correlated with the distance of the pixel at the same position in the blurred image from the horizontal center line of the license plate area, and is positively correlated with the pixel value of the pixel at the same position in the first channel filtered image, and is negatively correlated with the pixel value of the pixel at the same position in the second fused image.

[0212] By selecting this embodiment, the characteristic that a saturated license plate image can highlight the boundary of the rivet area can be utilized, and the first channel filter image and the second channel filter image with high saturation can be obtained by selecting the maximum value of the pixel value pairs under each channel, and then a fuzzy image is obtained by fuzzy filtering, and the rivet area is located at the upper and lower ends of the license plate. The fuzzy image and the second channel filter image are fused so that the rivet area can be highlighted in the obtained second fused image, and then a rivet area mask for representing the rivet area is generated based on the pixel value of each pixel point in the second fused image, the pixel value of each pixel point in the first channel filter image, and the distance from the pixel point to the horizontal center line of the license plate area. The rivet area mask that can represent the rivet area can be obtained more accurately in the above manner, so that the information of the license plate rivet area in the obtained first fused image is more accurate, which is more convenient for traffic management personnel to perform traffic management.

[0213] The above S1061-S1065 will be described exemplarily below:

[0214] In S1061, exemplarily, assuming that the image coordinate systems of the first image and the first channel filtered image are both established with the upper left corner of the image as the origin, the horizontal right direction as the X axis, and the vertical downward direction as the Y axis, the coordinates of a certain pixel point in the first image are (1, 1), the pixel value of the pixel point under the R channel is 12, the pixel value under the G channel is 56, and the pixel value under the B channel is 123, then the maximum value of the pixel value of the pixel point under each channel is 123, and the pixel value of the pixel point with coordinates (1, 1) in the first channel filtered image is 123. Pixel points at the same position in this application refer to: pixels whose pixel rows and pixel columns are the same. Exemplarily, pixel point 1 is located in the third pixel row from top to bottom in image A, and is located in the fifth pixel column from left to right in image A, and pixel point 2 is located in the third pixel row from top to bottom in image B, and is located in the fifth pixel column from left to right in image B, then pixel point 1 and pixel point 2 are pixels at the same position.

[0215] It can be understood that the first channel filtered image is obtained in the above manner, and the pixel value of each pixel point is set to a larger value as much as possible. Therefore, compared with the first image, the saturation of the first channel filtered image is higher.

[0216] In S1062, illustratively, it is assumed that an image coordinate system is established with the upper left corner of the image as the origin, the horizontal right direction as the X-axis, and the vertical downward direction as the Y-axis. The coordinates of a pixel point in the second image are (4, 5). The pixel value of the pixel point in the R channel is 38, the pixel value in the G channel is 241, and the pixel value in the B channel is 103. The maximum value of the pixel value of the pixel point in each channel is 241, and the pixel value of the pixel point with coordinates (4, 5) in the second channel filtered image is 241.

[0217] It can be understood that the second channel filtered image is obtained in the above manner, and the pixel value of each pixel point is set to a larger value as much as possible. Therefore, compared with the second image, the saturation of the second channel filtered image is higher.

[0218] In S1063, in one possible embodiment, the second channel filtered image may be fuzzy filtered by a Gaussian fuzzy filter with a radius of R to obtain a fuzzy image. R is a value set according to actual needs or past experience. In other possible embodiments, the second channel filtered image may be fuzzy filtered by a fuzzy filter such as a mean filter or a median filter to obtain a fuzzy image.

[0219] By means of fuzzy filtering, the noise and details in the second channel filtered image can be reduced, so that the quality and clarity of the obtained fuzzy image are higher than those of the second channel filtered image.

[0220] In S1064, see Figure 2a In the new energy license plate image shown, the area 22 where the four rivets in the license plate are located is the rivet area. The rivet area 22 is generally located at a position where the width of the upper part of the license plate is 1 / 4 to 3 / 4, and at a position where the width of the lower part of the license plate is 1 / 4 to 3 / 4. That is, the farther a pixel point is from the center line of the license plate, the more likely it is that the pixel point is located in the rivet area.

[0221] As described in S1063, blur filtering can make the quality and clarity of the obtained blurred image higher, but it will also reduce the detail information in the image. Then, in order to highlight the rivet area without losing the detail information in the image, if the distance between the pixel point and the horizontal center line of the license plate area is smaller, the pixel point is less likely to be located in the rivet area, and the pixel point does not need a higher definition of the blurred image to highlight the rivet area, and the detail information in the second channel filtered image needs to be retained, that is, the smaller the weight of the pixel point in the blurred image, the larger the weight of the pixel point in the second channel filtered image.

[0222] If the distance between the pixel point and the horizontal center line of the license plate area is larger, the pixel point is more likely to be located in the rivet area, and the pixel point needs a higher definition of the blurred image to highlight the rivet area, and the detail information in the second channel filtered image does not need to be retained. That is, the larger the weight of the pixel point in the blurred image, the smaller the weight of the pixel point in the second channel filtered image. Therefore, when performing weighted fusion on the blurred image and the second channel filtered image, the weight of each pixel point in the blurred image is positively correlated with the distance of the pixel point from the horizontal center line of the license plate area, and the weight of each pixel point in the second channel filtered image is negatively correlated with the distance of the pixel point at the same position in the blurred image from the horizontal center line of the license plate area, so that the rivet area in the obtained second fused image is clearer and the detail information in the second channel filtered image is retained.

[0223] In S1065, the positive correlation between the pixel value of each pixel in the rivet area mask and the distance between the pixel at the same position in the blurred image and the horizontal center line of the license plate area means that, with other factors unchanged, the pixel value of each pixel in the rivet area mask increases as the distance between the pixel at the same position in the blurred image and the horizontal center line of the license plate area increases. The increase here may refer to monotonic increase or non-monotonic increase.

[0224] The positive correlation between the pixel value of each pixel in the rivet area mask and the pixel value of the pixel at the same position in the first channel filtered image means that, with other factors unchanged, the pixel value of each pixel in the rivet area mask increases as the pixel value of the pixel at the same position in the first channel filtered image increases. The increase here may refer to monotonic increase or non-monotonic increase.

[0225] The negative correlation between the pixel value of each pixel in the rivet area mask and the pixel value of the pixel at the same position in the second fused image means that, with other factors unchanged, the pixel value of each pixel in the rivet area mask decreases as the pixel value of the pixel at the same position in the second fused image increases. The decrease here may refer to a monotonic decrease or a non-monotonic decrease.

[0226] In the above, an exemplary description is given of how to generate a rivet region mask for representing the rivet region in the aforementioned S106. In the following, an exemplary description is given of how to obtain the second fused image in the aforementioned S1064. Fig.10b , the aforementioned S1064 includes:

[0227] S1001, for each pixel in the blurred image, calculate the filtering fusion weight of the pixel according to the second distance between the pixel and the bottom edge of the license plate and the height of the license plate.

[0228] Among them, if the second distance is less than or equal to one-quarter of the height of the license plate, the filtering fusion weight of the pixel point is negatively correlated with the second distance; if the second distance is greater than or equal to three-quarters of the height of the license plate, the filtering fusion weight of the pixel point is positively correlated with the second distance; if the second distance is greater than one-quarter of the height of the license plate and less than three-quarters of the height of the license plate, the filtering fusion weight of the pixel point is 0.

[0229] Specifically, the filter fusion weights of the pixels in the blurred image are calculated using the following formulas (20), (21), and (22):

[0230]

[0231] Among them, fac is the filter fusion weight of the pixel in the blurred image; d2 is the second distance between the pixel in the blurred image and the bottom edge of the license plate; height is the height of the license plate.

[0232] S1002, performing weighted fusion on the blurred image and the second channel filtered image to obtain a second fused image.

[0233] Among them, the weight of each pixel in the blurred image is the filter fusion weight; the weight of each pixel in the second channel filtered image is negatively correlated with the filter fusion weight of the pixel at the same position in the blurred image.

[0234] Referring to formula (20), formula (21) and formula (22), it can be seen that the greater the distance between the pixel point in the blurred image and the horizontal center line of the license plate area, the greater the filter fusion weight of the pixel point; the smaller the distance between the pixel point in the blurred image and the horizontal center line of the license plate area, the smaller the filter fusion weight of the pixel point. In other words, it can be considered that the distance between the pixel point in the blurred image and the horizontal center line of the license plate area is positively correlated with the filter fusion weight of the pixel point.

[0235] Therefore, the weight of each pixel in the blurred image is the filter fusion weight, which is equivalent to the weight of each pixel in the blurred image being positively correlated with the distance between the pixel and the horizontal center line of the license plate area; the weight of each pixel in the second channel filtered image is negatively correlated with the filter fusion weight of the pixel at the same position in the blurred image, which is equivalent to the weight of each pixel in the second channel filtered image being negatively correlated with the distance between the pixel at the same position in the blurred image and the horizontal center line of the license plate area.

[0236] Specifically, the blurred image and the second channel filtered image are weighted fused by the following formula (23) to obtain a second fused image:

[0237] …(twenty three)

[0238] Among them, fac is the filter fusion weight of the pixel in the blurred image; max_chn_dark_blur is the pixel value of the pixel in the blurred image; Max_chn_dark_conbine is the pixel value of the pixel at the same position in the second fused image; max_chn_dark is the pixel value of the pixel at the same position in the second channel filtered image.

[0239] Corresponds to Fig.10b In the illustrated embodiment, if the first image is a bright frame image, the rivet area mask in S1065 is generated by the following formula (24):

[0240]

[0241] Where fac is the filter fusion weight of the pixel in the blurred image; max_chn_norm is the pixel value of the pixel at the same position in the first channel filtered image; Max_chn_dark_conbine is the pixel value of the pixel at the same position in the second fused image; p screws is the pixel value of the pixel at the same position in the rivet area mask; min() is the minimum value function used to convert p screws The value is limited to less than or equal to 256.

[0242] If the first image is an infrared frame image, the rivet area mask in S1065 is generated by the following formula (25):

[0243]

[0244] Where fac is the filter fusion weight of the pixel in the blurred image; max_chn_nir is the pixel value of the pixel at the same position in the first channel filtered image; Max_chn_dark_conbine is the pixel value of the pixel at the same position in the second fused image; p screws is the pixel value of the pixel at the same position in the rivet area mask; min() is the minimum value function used to convert p screws The value is limited to less than or equal to 256.

[0245] An exemplary explanation of how to obtain the second fused image has been given above. In the aforementioned S107, since the confidence of the area where the license plate is located obtained by edge detection may be low, that is, the accuracy of the area where the license plate is located obtained by edge detection is low, it may cause the accuracy of the license plate area mask generated based on the area where the license plate is located obtained by edge detection to be low, thereby making the accuracy of the first fused image obtained according to the license plate area mask low, resulting in low accuracy of the license plate information in the first fused image. Based on this, in order to improve the accuracy of the license plate area mask, and then improve the accuracy of the license plate information in the first fused image, in a possible embodiment, the aforementioned S107 includes: obtaining the area where the license plate is located in the second image by edge detection, and if the confidence of the area where the license plate is located in the second image obtained by edge detection is higher than a preset confidence threshold, then generating a license plate area mask for representing the area where the license plate is located.

[0246] The preset confidence threshold can be set according to actual needs or past experience, and this application does not impose any restrictions on this. Exemplarily, the preset confidence threshold can be set to 0.6, 0.7, 0.8, etc. If the confidence of the area where the license plate is located in the second image obtained by edge detection is higher than the preset confidence threshold, it is considered that the accuracy of the area where the license plate is located in the second image obtained by edge detection is high, and the license plate area mask can be directly generated based on the area where the license plate is located in the second image obtained by edge detection.

[0247] In this embodiment, the image processing method provided by the present application further includes:

[0248] If the confidence of the license plate area in the second image obtained by edge detection is not higher than a preset confidence threshold, the boundary input for the license plate area in the second image is obtained, and a mask for representing the area surrounded by the boundary is generated as a license plate area mask.

[0249] If the confidence of the area where the license plate is located in the second image obtained by edge detection is not higher than the preset confidence threshold, it is considered that the accuracy of the area where the license plate is located in the second image obtained by edge detection is low, and the user can manually calibrate the area where the license plate is located in the second image, and use the boundary of the area where the license plate is located that the user manually calibrates for the second image as the boundary of the area where the license plate is located in the second image, and generate a mask for representing the area surrounded by the boundary as a license plate area mask. The user can also directly input the coordinate information of the four vertices of the license plate in the second image, and use the area surrounded by the lines connecting the four vertices as the area where the license plate is located in the second image, and generate a mask for representing the area where the license plate is located as a license plate area mask. The method of generating the license plate area mask is the same as the method of generating the license plate area mask shown in the aforementioned S107, and the relevant instructions in S107 can be referred to, and will not be repeated here.

[0250] The flowchart of the image processing method provided in the present application can also be as follows Fig.11 As shown, see Fig.11 , the image processing methods include:

[0251] S1101, input a license plate area image.

[0252] The license plate area image in S1101 is the red light Bell data and the white light Bell data in S101. S1101 is equivalent to inputting the red light Bell data and the white light Bell data into the ISP.

[0253] S1102, obtain a bright frame, a dark frame, and an infrared frame.

[0254] The bright frame or infrared frame in S1102 is the aforementioned first image, and the dark frame is the aforementioned second image. S1101-S1102 can refer to the relevant description in the aforementioned S101, which will not be repeated here.

[0255] S1103, color template generation.

[0256] The color template in S1103 is the aforementioned color template image. The method of generating the color template image can refer to the relevant description in the aforementioned S101-S102, which will not be repeated here.

[0257] S1104, positioning of key areas.

[0258] It is understandable that the key areas in the license plate may be different for different types of license plates. For example, for new energy license plates such as new energy green license plates, yellow-green license plates, etc., the key areas are the new energy logo area and rivet area in the license plate; for normal motor vehicle license plates such as yellow license plates and blue license plates, the key areas are the rivet area.

[0259] Therefore, when the license plate type is a new energy license plate, the new energy logo area in the license plate can be determined by the aforementioned S103-S104 method, and the rivet area in the license plate can be determined by the aforementioned S1061-S1064 method, and the aforementioned second fused image can reflect the rivet area in the license plate. When the license plate type is a normal motor vehicle license plate, the rivet area in the license plate can be determined only by the aforementioned S1061-S1064 method.

[0260] S1105, calculating the key area position fusion mask.

[0261] When the license plate type is a new energy license plate, a new energy logo mask for representing the new energy logo area can be generated by the aforementioned S105 method, and a rivet area mask can be generated by the aforementioned S1065 method. When the license plate type is a normal motor vehicle license plate, a rivet area mask can be generated only by the aforementioned S1065 method.

[0262] S1106, performing fusion according to the mask.

[0263] S1106 is equivalent to the aforementioned S108-S109. Please refer to the relevant instructions of the aforementioned S108-S109 and will not be repeated here.

[0264] Corresponding to the aforementioned image processing method, the present application also provides an image processing device, see Fig.12 , the device comprises:

[0265] The average brightness calculation module 1201 is used to calculate the average brightness of the first image and the second image respectively; wherein the first image is a bright frame image or an infrared frame image, the second image is a dark frame image, and the first image and the second image are obtained by shooting the same license plate at the same time;

[0266] A color template image determination module 1202 is used to determine a color template image according to the average brightness of the first image and the average brightness of the second image;

[0267] The candidate sign region determination module 1203 is used to determine the region between the characters with the largest spacing in the region where the license plate is located in the target image as the candidate sign region, wherein the characters with the largest spacing are adjacent characters with the largest spacing in the horizontal direction in the region where the license plate is located, and the target image is the one with higher image quality between the first image and the second image;

[0268] The new energy sign area determination module 1204 is used to determine, in the candidate sign area, a sub-area that matches the brightness and graphic shape of the preset sign area as the new energy sign area, wherein the preset sign area is the area where the new energy sign is located in the preset new energy license plate image, and the image quality of the preset new energy license plate image is higher than a preset quality threshold;

[0269] A new energy symbol mask generation module 1205 is used to generate a new energy symbol mask for representing a new energy symbol area;

[0270] A rivet area mask generating module 1206, used to generate a rivet area mask for representing the rivet area;

[0271] A license plate region mask generating module 1207, used to obtain the region where the license plate is located in the second image by edge detection, and generate a license plate region mask for representing the region where the license plate is located;

[0272] The mask weight calculation module 1208 is used to calculate the mask weight of each pixel in the second image respectively, wherein the mask weight is obtained by weighted fusion of the rivet area mask, the new energy logo mask and the second image, the weight of each pixel in the rivet area mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, the weight of each pixel in the new energy logo mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, and the weight of each pixel in the second image is negatively correlated with the pixel value of the pixel at the same position in the license plate area mask;

[0273] The first fused image obtaining module 1209 is used to perform weighted fusion on the color template image and the second image to obtain a first fused image; wherein the weight of each pixel in the second image is a mask weight, and the weight of each pixel in the color template image is negatively correlated with the mask weight of the pixel at the same position in the second image.

[0274] In a possible embodiment, determining the color template image according to the average brightness of the first image and the average brightness of the second image includes:

[0275] Performing a weighted summation on the average brightness of the first image and the average brightness of the second image to obtain a brightness weight as a target brightness weight, wherein the weight of the average brightness of the first image is negatively correlated with the average brightness of the second image, and the weight of the average brightness of the second image is positively correlated with the average brightness of the second image;

[0276] Determine the color influencing factor of the license plate according to the target brightness weight and the license plate color;

[0277] A color template image is determined according to the license plate color and the color influencing factor in the first image or the second image.

[0278] In a possible embodiment, determining a color template image according to the license plate color and the color influencing factor in the first image or the second image includes:

[0279] A color template image is determined according to the license plate color and the color influencing factor in the first image or the second image, wherein the color depth of each pixel in the color template image decreases with the distance between the pixel and the lower boundary of the license plate.

[0280] In a possible embodiment, the color template image is determined by the following formula, including:

[0281]

[0282] Among them, RGB_B final is the pixel value of the pixel in the color template image under the B channel; rgb_max is the grayscale value of the pixel at the same position in the first image or the second image; b_fac is the color influence factor of the license plate under the B channel; RGB_G final is the pixel value of the pixel in the color template image under the G channel; RGB_R final is the pixel value of the pixel in the color template image under the R channel; r_fac is the color influencing factor of the license plate under the R channel;

[0283] wt1 is calculated by the following formula, including:

[0284]

[0285] Among them, CLIP() is a function used to limit the value of wt1 to the range of [0, 256]; d1 is the distance between the pixel point and the lower boundary of the license plate; height is the height of the license plate.

[0286] In a possible embodiment, in the area where the license plate is located in the target image, determining the area between the characters with the largest spacing as the candidate sign area includes:

[0287] Determine a pixel column that does not contain characters in the area where the license plate is located in the target image as a first pixel column;

[0288] Determine the number of first pixel columns existing between adjacent second pixel columns respectively as the first number corresponding to each adjacent second pixel column, wherein the second pixel columns are pixel columns other than the first pixel columns in the area where the license plate is located in the target image;

[0289] The corresponding adjacent second pixel columns with the largest first number are determined, and the area between the adjacent second pixel columns is determined as the candidate marker area.

[0290] In a possible embodiment, a pixel column that does not contain characters is determined in the area where the license plate is located in the target image as a first pixel column, including:

[0291] Binarize the area where the license plate is located in the target image to obtain a binary area;

[0292] For each pixel point in the binary area, calculating a first distance between the pixel point and an upper boundary and / or a lower boundary of the binary area;

[0293] In the binary area, determine a pixel point whose first distance is within a preset distance range as the first pixel point;

[0294] Determine the number of first pixel points whose pixel values ​​are not 0 contained in each pixel column in the binary area as the second number corresponding to each pixel column;

[0295] In each pixel column, a pixel column whose corresponding second number is smaller than a preset number threshold is determined as a first pixel column.

[0296] In a possible embodiment, generating a new energy mark mask for representing a new energy mark area includes:

[0297] Generate an alternative mask according to the new energy sign area, wherein the pixel value of each pixel point in the alternative mask is calculated in the following manner: determine a pixel area with the pixel point at the same position as the center and a preset length as the side length in the new energy sign area, and calculate the average of the pixel values ​​of each pixel point in the pixel area as the pixel value of the pixel point;

[0298] Normalize the candidate masks to obtain the new energy sign mask;

[0299] Generate a rivet area mask representing the rivet area, including:

[0300] Generate a first channel filtered image according to the first image, wherein the pixel value of each pixel in the first channel filtered image is calculated in the following manner: in the first image, determine the maximum value of the pixel values ​​of the pixel points at the same position under each channel as the pixel value of the pixel point;

[0301] Generate a second channel filtered image according to the second image, wherein the pixel value of each pixel in the second channel filtered image is calculated in the following manner: in the second image, determine the maximum value of the pixel values ​​of the pixel points at the same position under each channel as the pixel value of the pixel point;

[0302] Performing fuzzy filtering on the second channel filtered image to obtain a fuzzy image;

[0303] The blurred image and the second channel filtered image are weightedly fused to obtain a second fused image, wherein the weight of each pixel in the blurred image is positively correlated with the distance between the pixel and the horizontal center line of the license plate area, and the weight of each pixel in the second channel filtered image is negatively correlated with the distance between the pixel at the same position in the blurred image and the horizontal center line of the license plate area;

[0304] A rivet area mask is generated to represent the rivet area; the pixel value of each pixel in the rivet area mask is positively correlated with the distance of the pixel at the same position in the blurred image from the horizontal center line of the license plate area, and is positively correlated with the pixel value of the pixel at the same position in the first channel filtered image, and is negatively correlated with the pixel value of the pixel at the same position in the second fused image.

[0305] In a possible embodiment, performing fuzzy filtering on the second channel filtered image to obtain a fuzzy image includes:

[0306] Performing Gaussian blur filtering on the second channel filtered image to obtain a blurred image;

[0307] The blurred image and the second channel filtered image are weightedly fused to obtain a second fused image, including:

[0308] For each pixel in the blurred image, the filter fusion weight of the pixel is calculated according to the second distance between the pixel and the bottom edge of the license plate and the height of the license plate; if the second distance is less than or equal to one-fourth of the height of the license plate, the filter fusion weight of the pixel is negatively correlated with the second distance; if the second distance is greater than or equal to three-fourths of the height of the license plate, the filter fusion weight of the pixel is positively correlated with the second distance; if the second distance is greater than one-fourth of the height of the license plate and less than three-fourths of the height of the license plate, the filter fusion weight of the pixel is 0;

[0309] The blurred image and the second channel filtered image are weightedly fused to obtain a second fused image, wherein the weight of each pixel in the blurred image is the filtering fusion weight; and the weight of each pixel in the second channel filtered image is negatively correlated with the filtering fusion weight of the pixel at the same position in the blurred image.

[0310] The following formula is used to calculate the filtering fusion weight of each pixel in the blurred image according to the second distance between the pixel and the bottom edge of the license plate and the height of the license plate, including:

[0311]

[0312] Wherein, fac is the filter fusion weight of the pixel in the blurred image; d2 is the second distance between the pixel in the blurred image and the bottom edge of the license plate; height is the height of the license plate;

[0313] The blurred image and the second channel filtered image are weightedly fused by the following formula to obtain a second fused image, including:

[0314]

[0315] Wherein, fac is the filter fusion weight of the pixel in the blurred image; max_chn_dark_blur is the pixel value of the pixel in the blurred image; Max_chn_dark_conbine is the pixel value of the pixel at the same position in the second fused image; max_chn_dark is the pixel value of the pixel at the same position in the second channel filtered image;

[0316] If the first image is a bright frame image, a rivet area mask for representing the rivet area is generated by the following formula, including:

[0317]

[0318] Where fac is the filter fusion weight of the pixel in the blurred image; max_chn_norm is the pixel value of the pixel at the same position in the first channel filtered image; Max_chn_dark_conbine is the pixel value of the pixel at the same position in the second fused image; p screws is the pixel value of the pixel at the same position in the rivet area mask;

[0319] If the first image is an infrared frame image, a rivet area mask for representing the rivet area is generated by the following formula, including:

[0320]

[0321] Where fac is the filter fusion weight of the pixel in the blurred image; max_chn_nir is the pixel value of the pixel at the same position in the first channel filtered image; Max_chn_dark_conbine is the pixel value of the pixel at the same position in the second fused image; p screws is the pixel value of the pixel at the same position in the rivet area mask.

[0322] In a possible embodiment, generating a license plate area mask for indicating an area where a license plate is located includes:

[0323] If the confidence level of the area where the license plate is located in the second image obtained by edge detection is higher than a preset confidence threshold, a license plate area mask for representing the area where the license plate is located is generated;

[0324] The device also includes:

[0325] The license plate boundary acquisition module is used to obtain the boundary input for the area where the license plate is located in the second image if the confidence of the area where the license plate is located in the second image obtained by edge detection is not higher than a preset confidence threshold, and generate a mask for representing the area surrounded by the boundary as a license plate area mask.

[0326] The present application also provides an electronic device, such as Fig.13 As shown, including:

[0327] A memory 131, used for storing computer programs;

[0328] The processor 132 is used to execute the program stored in the memory 131 to implement the following steps:

[0329] Calculate the average brightness of the first image and the second image respectively; wherein the first image is a bright frame image or an infrared frame image, the second image is a dark frame image, and the first image and the second image are obtained by shooting the same license plate at the same time;

[0330] Determining a color template image according to an average brightness of the first image and an average brightness of the second image;

[0331] In the area where the license plate is located in the target image, determine the area between the characters with the largest spacing as the candidate mark area, wherein the characters with the largest spacing are adjacent characters with the largest spacing in the horizontal direction in the area where the license plate is located, and the target image is the one with higher image quality between the first image and the second image;

[0332] In the candidate logo area, a sub-area matching the brightness and graphic shape of the preset logo area is determined as the new energy logo area, wherein the preset logo area is the area where the new energy logo is located in the preset new energy license plate image, and the image quality of the preset new energy license plate image is higher than the preset quality threshold;

[0333] generating a new energy symbol mask for representing a new energy symbol area;

[0334] generating a rivet region mask for representing the rivet region;

[0335] Obtaining the area where the license plate is located in the second image by edge detection, and generating a license plate area mask for representing the area where the license plate is located;

[0336] Calculate the mask weight of each pixel in the second image respectively, wherein the mask weight is obtained by weighted fusion of the rivet area mask, the new energy logo mask and the second image, the weight of each pixel in the rivet area mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, the weight of each pixel in the new energy logo mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, and the weight of each pixel in the second image is negatively correlated with the pixel value of the pixel at the same position in the license plate area mask;

[0337] The color template image and the second image are weightedly fused to obtain a first fused image; wherein the weight of each pixel in the second image is a mask weight, and the weight of each pixel in the color template image is negatively correlated with the mask weight of the pixel at the same position in the second image.

[0338] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 132, the communication interface, and the memory 131 communicate with each other via the communication bus.

[0339] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0340] The communication interface is used for communication between the above electronic device and other devices.

[0341] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0342] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0343] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image processing methods are implemented.

[0344] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any image processing method in the above embodiments.

[0345] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0346] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0347] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, computer-readable storage medium, and computer program product containing instructions, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0348] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Calculate the average brightness of the first image and the second image respectively; wherein the first image is a bright frame image or an infrared frame image, the second image is a dark frame image, and the first image and the second image are obtained by shooting the same license plate at the same time; determining a color template image according to an average brightness of the first image and an average brightness of the second image; In the area where the license plate is located in the target image, determine the area between the characters with the largest spacing as the candidate mark area, wherein the characters with the largest spacing are adjacent characters with the largest spacing in the horizontal direction in the area where the license plate is located, and the target image is the one with higher image quality between the first image and the second image; In the candidate logo area, a sub-area matching the brightness and graphic shape of the preset logo area is determined as the new energy logo area, wherein the preset logo area is the area where the new energy logo is located in the preset new energy license plate image, and the image quality of the preset new energy license plate image is higher than a preset quality threshold; generating a new energy mark mask for representing the new energy mark area; generating a rivet region mask for representing the rivet region; Obtaining the area where the license plate is located in the second image by edge detection, and generating a license plate area mask for representing the area where the license plate is located; Calculating the mask weight of each pixel in the second image respectively, wherein the mask weight is obtained by weighted fusion of the rivet area mask, the new energy logo mask and the second image, the weight of each pixel in the rivet area mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, the weight of each pixel in the new energy logo mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, and the weight of each pixel in the second image is negatively correlated with the pixel value of the pixel at the same position in the license plate area mask; Performing weighted fusion on the color template image and the second image to obtain a first fused image; wherein the weight of each pixel in the second image is the mask weight, and the weight of each pixel in the color template image is negatively correlated with the mask weight of the pixel at the same position in the second image; The step of determining, within the region where the license plate is located in the target image, the region between the characters with the largest spacing as the candidate sign region comprises: Determine a pixel column that does not contain characters in the area where the license plate is located in the target image as a first pixel column; Respectively determine the number of the first pixel columns existing between each adjacent second pixel column as the first number corresponding to each adjacent second pixel column, wherein the second pixel columns are pixel columns other than the first pixel columns in the area where the license plate is located in the target image; The corresponding adjacent second pixel columns with the largest first number are determined, and the area between the adjacent second pixel columns is determined as the candidate marker area.

2. The method according to claim 1, characterized in that The step of determining the color template image according to the average brightness of the first image and the average brightness of the second image includes: Performing a weighted summation on the average brightness of the first image and the average brightness of the second image to obtain a brightness weight as a target brightness weight, wherein the weight of the average brightness of the first image is negatively correlated with the average brightness of the second image, and the weight of the average brightness of the second image is positively correlated with the average brightness of the second image; Determining a color influencing factor of the license plate according to a target brightness weight and a license plate color; A color template image is determined according to the license plate color in the first image or the second image and the color influencing factor.

3. The method according to claim 2, characterized in that The step of determining a color template image according to the license plate color in the first image or the second image and the color influencing factor comprises: A color template image is determined according to the license plate color in the first image or the second image and the color influencing factor, wherein the color depth of each pixel in the color template image decreases with the distance between the pixel and the lower boundary of the license plate.

4. The method according to claim 3, characterized in that The color template image is determined by the following formula, including: ; Among them, RGB_B final is the pixel value of the pixel in the color template image under the B channel; rgb_max is the grayscale value of the pixel at the same position in the first image or the second image; b_fac is the color influence factor of the license plate under the B channel; RGB_G final is the pixel value of the pixel in the color template image under the G channel; RGB_R final is the pixel value of the pixel point in the color template image under the R channel; r_fac is the color influencing factor of the license plate under the R channel; wt1 is calculated by the following formula, including: ; Among them, CLIP() is a function used to limit the value of wt1 to the range of [0, 256]; d1 is the distance between the pixel point and the lower boundary of the license plate; height is the height of the license plate.

5. The method according to claim 1, characterized in that The step of determining a pixel column that does not contain characters in the area where the license plate is located in the target image as a first pixel column includes: Binarize the area where the license plate is located in the target image to obtain a binary area; For each pixel point in the binary area, calculating a first distance between the pixel point and an upper boundary and / or a lower boundary of the binary area; In the binary area, determine a pixel point whose first distance is within a preset distance range as a first pixel point; Respectively determine the number of the first pixel points with non-zero pixel values ​​contained in each pixel column in the binary area as the second number corresponding to each pixel column; In each of the pixel columns, a pixel column corresponding to the second number of pixels that is less than a preset number threshold is determined as a first pixel column.

6. The method according to claim 1, characterized in that The generating of a new energy mark mask for representing the new energy mark area includes: Generate an alternative mask according to the new energy logo area, wherein the pixel value of each pixel point in the alternative mask is calculated in the following manner: determine a pixel area with a pixel point at the same position as the center and a preset length as the side length in the new energy logo area, and calculate the average of the pixel values ​​of each pixel point in the pixel area as the pixel value of the pixel point; Normalizing the candidate mask to obtain the new energy mark mask; The generating a rivet area mask for representing the rivet area includes: Generate a first channel filtered image according to the first image, wherein the pixel value of each pixel in the first channel filtered image is calculated in the following manner: in the first image, determine the maximum value of the pixel values ​​of the pixel points at the same position under each channel as the pixel value of the pixel point; Generate a second channel filtered image according to the second image, wherein the pixel value of each pixel in the second channel filtered image is calculated in the following manner: in the second image, determine the maximum value of the pixel values ​​of the pixel at the same position under each channel as the pixel value of the pixel; Performing fuzzy filtering on the second channel filtered image to obtain a fuzzy image; Performing weighted fusion on the blurred image and the second channel filtered image to obtain a second fused image, wherein the weight of each pixel in the blurred image is positively correlated with the distance between the pixel and the horizontal center line of the license plate area, and the weight of each pixel in the second channel filtered image is negatively correlated with the distance between the pixel at the same position in the blurred image and the horizontal center line of the license plate area; A rivet area mask is generated to represent the rivet area; the pixel value of each pixel in the rivet area mask is positively correlated with the distance of the pixel at the same position in the blurred image from the horizontal center line of the license plate area, and is positively correlated with the pixel value of the pixel at the same position in the first channel filtered image, and is negatively correlated with the pixel value of the pixel at the same position in the second fused image.

7. The method according to claim 6, characterized in that The step of performing fuzzy filtering on the second channel filtered image to obtain a fuzzy image includes: Performing Gaussian blur filtering on the second channel filtered image to obtain a blurred image; The step of performing weighted fusion on the blurred image and the second channel filtered image to obtain a second fused image includes: For each pixel in the blurred image, the filtering fusion weight of the pixel is calculated according to the second distance between the pixel and the bottom edge of the license plate and the height of the license plate; wherein, if the second distance is less than or equal to one-fourth of the height of the license plate, the filtering fusion weight of the pixel is negatively correlated with the second distance; if the second distance is greater than or equal to three-fourths of the height of the license plate, the filtering fusion weight of the pixel is positively correlated with the second distance; if the second distance is greater than one-fourth of the height of the license plate and less than three-fourths of the height of the license plate, the filtering fusion weight of the pixel is 0; Performing weighted fusion on the blurred image and the second channel filtered image to obtain a second fused image, wherein the weight of each pixel in the blurred image is the filtering fusion weight; and the weight of each pixel in the second channel filtered image is negatively correlated with the filtering fusion weight of the pixel at the same position in the blurred image; The filtering fusion weight of each pixel in the blurred image is calculated by the following formula according to the second distance between the pixel and the bottom edge of the license plate and the height of the license plate, including: ; Wherein, fac is the filtering fusion weight of the pixel point in the blurred image; d2 is the second distance between the pixel point in the blurred image and the bottom edge of the license plate; height is the height of the license plate; The blurred image and the second channel filtered image are weightedly fused by the following formula to obtain a second fused image, including: ; Wherein, fac is the filter fusion weight of the pixel in the blurred image; max_chn_dark_blur is the pixel value of the pixel in the blurred image; Max_chn_dark_conbine is the pixel value of the pixel at the same position in the second fused image; max_chn_dark is the pixel value of the pixel at the same position in the second channel filtered image; If the first image is a bright frame image, a rivet area mask for representing the rivet area is generated by the following formula, including: ; Wherein, fac is the filter fusion weight of the pixel in the blurred image; max_chn_norm is the pixel value of the pixel at the same position in the first channel filtered image; Max_chn_dark_conbine is the pixel value of the pixel at the same position in the second fused image; p screws is the pixel value of the pixel point at the same position in the rivet area mask; If the first image is an infrared frame image, a rivet area mask for representing the rivet area is generated by the following formula, including: ; Wherein, fac is the filter fusion weight of the pixel in the blurred image; max_chn_nir is the pixel value of the pixel at the same position in the first channel filtered image; Max_chn_dark_conbine is the pixel value of the pixel at the same position in the second fused image; p screws is the pixel value of the pixel at the same position in the rivet area mask.

8. The method according to claim 1, characterized in that: The generating of a license plate area mask for indicating the area where the license plate is located includes: If the confidence of the area where the license plate is located in the second image obtained by edge detection is higher than a preset confidence threshold, a license plate area mask for representing the area where the license plate is located is generated; The method further comprises: If the confidence of the area where the license plate is located in the second image obtained by edge detection is not higher than a preset confidence threshold, the boundary input for the area where the license plate is located in the second image is obtained, and a mask for representing the area enclosed by the boundary is generated as a license plate area mask.

9. An image processing device, characterized in that: The device comprises: An average brightness calculation module, used to calculate the average brightness of the first image and the second image respectively; wherein the first image is a bright frame image or an infrared frame image, the second image is a dark frame image, and the first image and the second image are obtained by shooting the same license plate at the same time; A color template image determination module, used to determine a color template image according to an average brightness of the first image and an average brightness of the second image; A candidate sign area determination module is used to determine, within the area where the license plate is located in the target image, an area between characters with the largest spacing as the candidate sign area, wherein the characters with the largest spacing are adjacent characters with the largest spacing in the horizontal direction within the area where the license plate is located, and the target image is the one with higher image quality between the first image and the second image; A new energy sign area determination module is used to determine, in the candidate sign area, a sub-area that matches the brightness and graphic shape of the preset sign area as the new energy sign area, wherein the preset sign area is the area where the new energy sign is located in the preset new energy license plate image, and the image quality of the preset new energy license plate image is higher than a preset quality threshold; A new energy symbol mask generation module, used to generate a new energy symbol mask for representing the new energy symbol area; A rivet area mask generating module, used for generating a rivet area mask for representing a rivet area; A license plate area mask generation module, used to obtain the area where the license plate is located in the second image through edge detection, and generate a license plate area mask for representing the area where the license plate is located; a mask weight calculation module, used to respectively calculate the mask weight of each pixel in the second image, wherein the mask weight is obtained by weighted fusion of the rivet area mask, the new energy logo mask and the second image, the weight of each pixel in the rivet area mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, the weight of each pixel in the new energy logo mask is positively correlated with the pixel value of the pixel at the same position in the license plate area mask, and the weight of each pixel in the second image is negatively correlated with the pixel value of the pixel at the same position in the license plate area mask; a first fused image obtaining module, configured to perform weighted fusion on the color template image and the second image to obtain a first fused image; wherein the weight of each pixel in the second image is the mask weight, and the weight of each pixel in the color template image is negatively correlated with the mask weight of the pixel at the same position in the second image; The step of determining, within the region where the license plate is located in the target image, the region between the characters with the largest spacing as the candidate sign region comprises: Determine a pixel column that does not contain characters in the area where the license plate is located in the target image as a first pixel column; Respectively determine the number of the first pixel columns existing between each adjacent second pixel column as the first number corresponding to each adjacent second pixel column, wherein the second pixel columns are pixel columns other than the first pixel columns in the area where the license plate is located in the target image; The corresponding adjacent second pixel columns with the largest first number are determined, and the area between the adjacent second pixel columns is determined as the candidate marker area.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images