An image processing method, apparatus, electronic device, and storage medium

Abstract

This application provides an image processing method, apparatus, electronic device, and storage medium, relating to the field of image processing. By fusing original images of a target vehicle, the clarity of the license plate and window areas in the obtained target fused image is optimized, thereby providing higher accuracy for target vehicle identification. The method includes: acquiring a clear white light image of the vehicle with a clear license plate area using first exposure parameters; acquiring a clear infrared image of the vehicle with a clear window area using infrared strobe illumination; the brightness of the original vehicle infrared image is greater than the brightness of the original vehicle white light image; and fusing the vehicle white light image and the vehicle infrared image to obtain a target fused image with optimized clarity in both the license plate and window areas.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Vehicle recognition includes license plate recognition and driver recognition. As an important component of image recognition in the field of traffic management, vehicle recognition is widely used in parking lot fee management, factory area vehicle access management, and highway toll management.

[0003] Currently, it's common to see trucks on highways with high-intensity lights installed above their license plates. However, when ordinary traffic cameras capture images of these trucks, the license plate area is often overexposed, while the windshield area is often underexposed and unclear. This makes it impossible to effectively identify the license plate and driver. When such vehicles are present, insufficient evidence can hinder effective management. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and storage medium that optimizes the clarity of the license plate area and window area of ​​the obtained target fused image by performing fusion processing on the original image of the target vehicle, thereby providing higher accuracy for identifying the target object.

[0005] In a first aspect, this application provides an image processing method, which includes: acquiring a clear white light image of a vehicle with a license plate area using a first exposure parameter; acquiring a clear infrared image of a vehicle with a window area using infrared strobe illumination; and fusing the white light image of the vehicle with the infrared image of the vehicle to obtain a target fused image with optimized clarity of the license plate area and the window area.

[0006] Understandably, for vehicles with high-intensity lights installed above their license plates, the area around the license plate is already quite bright. Therefore, using the first exposure parameter, a white light image of the vehicle is captured, in which the license plate area is clear. Then, an infrared image of the vehicle is captured using infrared strobe illumination, in which the window area is clear. Finally, the white light image and the infrared image are fused to obtain a target fused image with optimized clarity for both the license plate and window areas. Using this target fused image for object recognition can achieve higher accuracy.

[0007] In some embodiments, the above-mentioned fusion of the vehicle white light image and the vehicle infrared image to obtain a target fused image with a clear license plate area and a clear window area includes: performing a first image processing on the vehicle white light image to obtain a first vehicle white light image with optimized license plate area display effect; performing a second image processing on the vehicle white light image to obtain a second vehicle white light image with optimized vehicle body display effect; and fusing the first vehicle white light image and the second vehicle white light image to obtain a first fused image.

[0008] In some embodiments, the brightness of the second vehicle white light image is greater than the brightness of the first vehicle white light image.

[0009] In some embodiments, the method further includes: fusing the license plate region in the first vehicle white light image with the license plate region in the target fused image to obtain the first vehicle fused image. In some embodiments, the method further includes fusing any two or three of the following: the window region in the vehicle infrared image, the window region in the target fusion image, and the color-optimized window region in the vehicle white light image, to obtain a second vehicle fusion image.

[0010] In some embodiments, the method further includes fusing the first vehicle fusion image and the second vehicle fusion image to obtain a third vehicle fusion image.

[0011] In some embodiments, the above method further includes: performing third image processing on the vehicle white light image to obtain a third vehicle white light image with optimized window area color.

[0012] In some embodiments, the above-mentioned method of using infrared strobe illumination to acquire a clear vehicle infrared image of the window area includes: acquiring an original vehicle infrared image using infrared strobe illumination; and performing a fourth image processing on the original vehicle infrared image to obtain a vehicle infrared image with optimized display effect in the window area.

[0013] In some embodiments, the first image processing includes: interpolation operation, sharpening operation, white balance operation, first color correction operation, and noise reduction operation, but does not include gamma correction operation, dynamic range enhancement operation, or brightness gain operation. The first color correction operation does not include processing the vehicle white light image using color parameters greater than a preset saturation.

[0014] In some embodiments, the second image processing includes: interpolation operation, sharpening operation, white balance operation, first color correction operation, noise reduction operation, gamma correction operation, dynamic range enhancement operation, and brightness gain operation.

[0015] In some embodiments, the third image processing includes: interpolation operation, sharpening operation, white balance operation, second color correction operation, noise reduction operation, gamma correction operation, and dynamic range enhancement operation. The second color correction operation includes processing the vehicle white light image using color parameters greater than a preset saturation, and learning color adjustment parameters for the vehicle white light image using a deep regression network model.

[0016] In some embodiments, the fourth image processing includes: interpolation operation, sharpening operation, noise reduction operation, gamma correction operation, and dynamic range enhancement operation, but does not include white balance operation or color correction operation.

[0017] Secondly, embodiments of this application provide an image processing apparatus, comprising: an image acquisition unit and an image fusion unit. The image acquisition unit may employ a first exposure parameter to acquire a clear white light image of the vehicle's license plate area; the image acquisition unit may also employ infrared strobe illumination to acquire a clear infrared image of the vehicle's window area; the image fusion unit may fuse the vehicle's white light image and the vehicle's infrared image to obtain a target fused image with optimized clarity for both the license plate area and the window area.

[0018] In some embodiments, the image processing apparatus further includes: an image processing unit; the image processing unit can perform a first image processing on the vehicle white light image to obtain a first vehicle white light image with optimized license plate area display effect; the image processing unit can perform a second image processing on the vehicle white light image to obtain a second vehicle white light image with optimized vehicle body display effect; the first vehicle white light image and the second vehicle white light image are fused to obtain a first fused image; and an image fusion unit can fuse the first fused image with a vehicle infrared image to obtain a target fused image.

[0019] In some embodiments, the brightness of the second vehicle white light image is greater than the brightness of the first vehicle white light image.

[0020] In some embodiments, the image fusion unit can fuse the license plate region in the first vehicle white light image with the license plate region in the target fusion image to obtain the first vehicle fusion image.

[0021] In some embodiments, the image fusion unit can fuse any two or three of the following: the window area in the vehicle infrared image, the window area in the target fusion image, and the color-optimized window area in the vehicle white light image, to obtain a second vehicle fusion image.

[0022] In some embodiments, the image fusion unit can fuse the first vehicle fusion image and the second vehicle fusion image to obtain a third vehicle fusion image.

[0023] In some embodiments, the image processing unit may perform third image processing on the vehicle white light image to obtain a third vehicle white light image with optimized window area color.

[0024] In some embodiments, the image acquisition unit may use infrared strobe illumination to acquire the original vehicle infrared image; the image processing unit may perform a fourth image processing on the original vehicle infrared image to obtain a vehicle infrared image with optimized display effect in the window area.

[0025] In some embodiments, the first image processing includes: interpolation operation, sharpening operation, white balance operation, first color correction operation, and noise reduction operation, but does not include gamma correction operation, dynamic range enhancement operation, or brightness gain operation. The first color correction operation does not include processing the vehicle white light image using color parameters greater than a preset saturation.

[0026] In some embodiments, the second image processing includes: interpolation operation, sharpening operation, white balance operation, first color correction operation, noise reduction operation, gamma correction operation, dynamic range enhancement operation, and brightness gain operation.

[0027] In some embodiments, the third image processing includes: interpolation operation, sharpening operation, white balance operation, second color correction operation, noise reduction operation, gamma correction operation, and dynamic range enhancement operation. The second color correction operation includes processing the vehicle white light image using color parameters greater than a preset saturation, and learning color adjustment parameters for the vehicle white light image using a deep regression network model.

[0028] In some embodiments, the fourth image processing includes: interpolation operation, sharpening operation, noise reduction operation, gamma correction operation, and dynamic range enhancement operation, but does not include white balance operation or color correction operation.

[0029] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code including computer instructions; wherein, when the processor executes the computer instructions, it causes an image processing device to perform an image processing method as described in the first aspect and any possible implementation thereof.

[0030] Fourthly, this application provides a computer-readable storage medium comprising: computer software instructions; which, when executed in an image processing apparatus, cause the image processing apparatus to implement the image processing method described in the first aspect and any possible implementation thereof.

[0031] Fifthly, this application provides a computer program product that, when run on an image processing apparatus, causes the image processing apparatus to execute the image processing method described in the first aspect and any possible implementation thereof. The beneficial effects of the second to fifth aspects can be referred to the corresponding description of the first aspect, and will not be repeated here. Attached Figure Description

[0032] Figure 1 A schematic diagram of the structure of an image processing system provided in this application; Figure 2 A schematic diagram of the hardware configuration of a shooting device provided in this application; Figure 3 A schematic diagram of the structure of an image processing device provided in this application; Figure 4 A flowchart illustrating an image processing method provided in this application; Figure 5 A flowchart illustrating yet another image processing method provided in this application; Figure 6 A flowchart illustrating yet another image processing method provided in this application; Figure 7 A flowchart illustrating yet another image processing method provided in this application; Figure 8 A flowchart illustrating yet another image processing method provided in this application; Figure 9 A schematic diagram of an original image provided for this application; Figure 10 Another original image schematic diagram provided for this application; Figure 11 A schematic diagram of a fused image provided in this application; Figure 12 This is a schematic diagram of the hardware structure of an electronic device provided in this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] It should be noted that in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0035] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0036] To facilitate understanding, a brief introduction to the relevant concepts involved in this application will be provided first.

[0037] 1. Shutter speed: One of the automatic exposure parameters of a camera, which controls the length of time the camera is exposed to light. The larger the shutter speed, the longer the exposure time, and the more sufficient the exposure. Under the same conditions, the larger the shutter speed parameter, the brighter the image.

[0038] 2. Gain: One of the automatic exposure parameters of a camera, which controls the amplification of photosensitive pixels. Under the same conditions, the higher the gain, the brighter the image.

[0039] 3. Image signal processing (ISP): This mainly involves performing a series of image processing operations. For example, it can enhance images in Bayer format obtained by image acquisition units such as image sensors to obtain color-coded (YUV) format images, or perform noise reduction algorithms to make the images look clearer.

[0040] 4. Raw image format (RAW): The image format directly captured by the image acquisition unit such as the camera's image sensor.

[0041] 5. Overexposure: An image distortion caused by excessively large aperture during image acquisition, excessively high film sensitivity, excessively long exposure time, or excessively strong ambient light. Specifically, it can manifest as a washed-out image or excessively bright image.

[0042] 6. Wide Dynamic Range (WDR) Images: WDR images are generally acquired in two ways: one is digital wide dynamic range technology, which adjusts the brightness by adjusting the gain of different regions of the image; the other is to acquire multiple frames based on each exposure, with different brightness levels in each exposure, and then merge the multiple frames into one frame based on the brightness of the pixels in the multiple frames.

[0043] Vehicle recognition includes license plate recognition and driver recognition. As an important component of image recognition in the field of traffic management, vehicle recognition is widely used in parking lot fee management, factory area vehicle access management, and highway toll management.

[0044] Currently, it's common to see trucks on highways with high-intensity lights installed above their license plates. However, when ordinary traffic cameras capture images of these trucks, the license plate area is often overexposed, while the windshield area is often underexposed and unclear. This makes it impossible to effectively identify the license plate and driver. When such vehicles commit violations, insufficient evidence may prevent effective enforcement.

[0045] To address this problem, this application provides an image processing method, apparatus, electronic device, and storage medium. For vehicles with high-intensity headlights installed above the license plate, since the light intensity around the license plate is already high, a first exposure parameter is used to acquire a white light image of the target vehicle with lower brightness, resulting in a clear license plate area. Then, infrared strobe illumination is used to acquire an infrared image of the target vehicle with higher brightness, resulting in a clear window area. Finally, the white light image and the infrared image are fused to obtain a target fused image with optimized clarity for both the license plate and window areas. This target fused image can then be used for target vehicle identification, accurately identifying the license plate number in the license plate area and the driver in the window area.

[0046] The image processing method provided in this application can be applied to electronic devices such as cameras and snapshot devices on roads.

[0047] The image processing method provided in this application embodiment can be applied to, for example... Figure 1 The image processing system shown. For example... Figure 1 As shown, the image processing system may include an image processing device 11 and an image capturing device 12.

[0048] The camera device 12 can acquire images of target vehicles in real-world scenarios. Examples include cameras installed at parking lot entrances, road checkpoints, highway toll stations, and electronic cameras mounted above urban roads.

[0049] The image processing device 11 can acquire an image of the target vehicle from the imaging device 12 and process the image of the target vehicle.

[0050] In some embodiments, such as Figure 2As shown, the imaging device 12 includes a first fill light 101, a second fill light 102, a first sensor 103, a second sensor 104, and an image acquisition unit 105. The brightness of the light emitted by the first fill light 101 is less than the brightness of the light emitted by the second fill light 102. The first sensor 103 can transmit the light emitted by the first fill light 101, while the second sensor 104 can transmit the light emitted by the second fill light 102. Therefore, the brightness of the image obtained based on the light from the first fill light 101 is less than the brightness of the image obtained based on the light from the second fill light 102.

[0051] In some embodiments, the light emitted by the first fill light 101 is natural light (white light, wavelength 450-465 nanometers (nm)) with a brightness range of 0 to 4000 lumens (lm), which is the brightness range comfortable for the human eye. When shooting at night, the first fill light 101 can provide supplementary lighting for the target vehicle. For example, when shooting a truck driving in the dark, the first fill light 101 can be used to illuminate the truck, thereby enabling the captured image to achieve a certain level of brightness.

[0052] In some embodiments, the light emitted by the second fill light 102 is infrared light (wavelength 615-650 nm), and the second fill light 102 is an infrared strobe light. Because infrared light is a wavelength invisible to the human eye, using an infrared strobe light to provide supplementary lighting for shooting target vehicles (such as trucks or vans driving in the dark) at night does not limit the brightness of the infrared light. This can improve the brightness of the captured image while avoiding light pollution that may irritate the driver's eyes.

[0053] The first sensor 103 captures the original white light image of the target vehicle through the light emitted by the first supplementary light 101 and outputs the image acquisition unit 105. The second sensor 104 captures the original red light image of the target vehicle through the light emitted by the second supplementary light 102 and outputs the image acquisition unit 105.

[0054] like Figure 3 As shown, the image processing device 11 in this embodiment may include: an image acquisition unit 201, an image processing unit 202, and an image fusion unit 203.

[0055] In some embodiments, the image acquisition unit 201 can acquire images of the target vehicle captured in the actual scene from the shooting device 12.

[0056] In some other embodiments, the image acquisition unit 201 may also read the image of the target vehicle from a database storing images acquired at historical time points. This embodiment does not impose any specific limitations.

[0057] In some embodiments, the image acquisition unit may use a first exposure parameter to acquire a clear white light image of the vehicle in the license plate area; the image acquisition unit may also use infrared strobe illumination to acquire a clear infrared image of the vehicle in the window area; the image fusion unit may fuse the vehicle white light image and the vehicle infrared image to obtain a target fused image with optimized clarity in the license plate area and the window area.

[0058] In some embodiments, the image processing apparatus further includes an image processing unit 202.

[0059] In some embodiments, the image fusion unit 203 can perform a first image processing on the vehicle white light image to obtain a first vehicle white light image with optimized license plate area display effect; the image fusion unit 203 can perform a second image processing on the vehicle white light image to obtain a second vehicle white light image with optimized vehicle body display effect; the first vehicle white light image and the second vehicle white light image are fused to obtain a first fused image; the image fusion unit 203 can also fuse the first fused image with the vehicle infrared image to obtain a target fused image.

[0060] In some embodiments, the brightness of the second vehicle white light image is greater than the brightness of the first vehicle white light image.

[0061] In some embodiments, the image fusion unit 203 can fuse the license plate region in the first vehicle white light image with the license plate region in the target fusion image to obtain the first vehicle fusion image.

[0062] In some embodiments, the image fusion unit 203 can fuse any two or three of the following: the window area in the vehicle infrared image, the window area in the target fusion image, and the color-optimized window area in the vehicle white light image, to obtain a second vehicle fusion image.

[0063] In some embodiments, the image fusion unit 203 can fuse the first vehicle fusion image and the second vehicle fusion image to obtain a third vehicle fusion image.

[0064] In some embodiments, the image fusion unit 203 can perform third image processing on the vehicle white light image to obtain a third vehicle white light image with optimized window area color.

[0065] In some embodiments, the image acquisition unit 201 may use infrared strobe illumination to acquire the original vehicle infrared image; the image fusion unit 203 may perform a fourth image processing on the original vehicle infrared image to obtain a vehicle infrared image with optimized display effect in the window area.

[0066] In addition, in some embodiments, the image acquisition unit 201 in the image processing apparatus 11 may include a shooting unit, which acquires images of the target vehicle in the actual scene. In this case, the image processing method provided in the embodiments of this application can also be applied to the image processing apparatus 11, and the functions that the image processing apparatus 11 can implement and the unit structure it includes can be found in the foregoing embodiments.

[0067] Figure 4 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Exemplarily, the image processing method provided in this embodiment can be applied to... Figure 3 The image processing device shown. (For example) Figure 4 As shown, the image processing method provided in this application embodiment may specifically include the following steps S101 to S103.

[0068] S101. Using the first exposure parameters, acquire a clear white light image of the vehicle in the license plate area.

[0069] For the target vehicle, a clear white light image of the vehicle with the license plate area is captured. The target vehicle can be a vehicle with a strong light installed above the license plate (such as a truck or van). Since the light around the license plate is relatively bright, the license plate area is brighter and clearer. Other areas of the vehicle's white light image (such as the window area and the body area) are less bright and unclear.

[0070] In some embodiments, the first exposure parameter can be a parameter less than a set threshold, such as a low shutter speed parameter and a low gain parameter (e.g., setting the exposure parameters when the above-mentioned shooting unit captures an image), so that the license plate in the obtained image is relatively clear and there is no overexposure problem. The specific parameter value can be set by a technician.

[0071] In some embodiments, when capturing a vehicle white light image, low-brightness supplementary lighting can be applied to the vehicle in the actual shooting environment. The supplementary light is natural light (white light, wavelength 450-465 nanometers (nm)) with a brightness range of 0 to 4000 lumens (lm), which is the brightness range comfortable for the human eye. Because the supplementary light brightness is low, this method avoids overexposure of the license plate area in the vehicle white light image, while providing sufficient brightness to other areas of the vehicle white light image (such as the window area and body area). This allows the driver's face to be seen within the target vehicle through the vehicle white light image.

[0072] S102. Infrared strobe illumination is used to acquire clear infrared images of the vehicle in the window area.

[0073] The brightness of the vehicle's infrared image is greater than the brightness of the vehicle's white light image.

[0074] For the target vehicle, a clear infrared image of the license plate area is captured. The target vehicle can be a vehicle with a strong light installed above the license plate. Due to the use of infrared strobe supplementary lighting, the supplementary lighting brightness is relatively large. Therefore, the brightness of the vehicle infrared image is greater than the brightness of the vehicle white light image. The brightness of the window area and the body area is better, and the window area is also clearer than the vehicle white light image.

[0075] Infrared strobe lighting employs high-brightness infrared light for illumination. Infrared light is a wavelength invisible to the human eye; therefore, using high-brightness infrared light to illuminate vehicles (such as trucks driving in darkness) at night can improve image brightness while avoiding light pollution that could irritate the driver's eyes. For example, infrared strobe lighting can utilize an infrared strobe lamp.

[0076] S103. The vehicle white light image and the vehicle infrared image are fused to obtain a target fused image with optimized clarity in the license plate area and the window area.

[0077] This involves fusing the vehicle's white light image with its infrared image, which is known as white light infrared fusion.

[0078] Since the license plate area in the white light image of the vehicle is clear and the window area in the infrared image of the vehicle is clear, the clarity of the license plate area and the window area in the target fused image obtained after fusion are optimized.

[0079] In some embodiments, for target objects with different brightness levels or large brightness differences in different target areas, a first exposure parameter can be used to acquire a first image, an infrared strobe light can be used to acquire a second image, and then the first image and the second object can be fused to obtain a target object image with a clearer target area after fusion. For example, a target vehicle with light sources installed around the car window or body, or a target person located under a light source (light lamp or sunlight) (the brightness value of the face area is greater than the brightness value of the upper body, and the brightness value of the face area is greater than the brightness value of the lower body), etc. This embodiment does not impose specific limitations.

[0080] Understandably, this embodiment is designed for vehicles with high-intensity lights installed above their license plates. Since the light intensity around the license plate is already high, the first exposure parameter is used to acquire a white light image of the target vehicle with lower brightness, resulting in a clear license plate area. Then, infrared strobe illumination is used to acquire an infrared image of the target vehicle with higher brightness, resulting in a clear window area. Finally, the white light image and the infrared image are fused to obtain a target fused image with optimized clarity for both the license plate and window areas. Using this target fused image for target vehicle identification can achieve higher accuracy.

[0081] In some embodiments, such as Figure 5 As shown, the vehicle white light image and the vehicle infrared image are fused to obtain a target fused image with a clear license plate area and a clear window area. The method may also include steps Sa1 to Sa4.

[0082] Sa1. Perform first image processing on the vehicle white light image to obtain the first vehicle white light image after optimizing the license plate area display effect.

[0083] Since the license plate area in the vehicle white light image is relatively bright, the first image processing includes an image processing step to improve image clarity, but does not include a step to improve brightness. As a result, the brightness of the license plate area in the first vehicle white light image remains normal, and the clarity is optimized and improved.

[0084] In some embodiments, image processing is ISP processing, which improves image parameters such as sharpness, brightness, and color saturation. Typical ISP processing includes interpolation, sharpening, white balance, color correction, noise reduction, gamma correction, and dynamic range enhancement.

[0085] Understandably, the internal processing steps of the ISP will differ depending on the desired effect of the image and the shooting method. For example, if it is necessary to improve the sharpness of bright areas in the image, brightness enhancement operations such as gamma correction and dynamic range enhancement can be omitted; when processing images obtained through infrared illumination, color value adjustments such as white balance and color correction can be omitted.

[0086] In some embodiments, the first image processing may include several of the aforementioned ISP processing methods, specifically including: interpolation operation, sharpening operation, white balance operation, first color correction operation and noise reduction operation, but excluding gamma correction operation, dynamic range enhancement operation and brightness gain operation. The first color correction operation does not include processing the vehicle white light image using color parameters greater than a preset saturation, thereby ensuring that the color of the vehicle white light image remains normal.

[0087] Among these processes, interpolation can use the grayscale values ​​of known neighboring pixels to generate the grayscale values ​​of unknown pixels, resulting in a higher resolution image from the source image. Therefore, after the first image processing of the white light image, the resolution of the first vehicle's white light image is higher. Sharpening can compensate for the image's contours, enhance the edges and areas of grayscale transitions, making the image clearer. Therefore, after the first image processing of the white light image, the clarity of the first vehicle's white light image is higher. White balance can eliminate color cast effects and restore the original colors of the photographed object. Therefore, after the first image processing of the white light image, the color values ​​of the first vehicle's white light image are closer to the true values ​​visible to the human eye. The first color correction operation can eliminate color cast effects by increasing or decreasing its contrast color. Therefore, after the first image processing of the white light image, the color values ​​of the first vehicle's white light image are closer to the true colors visible to the human eye. Noise reduction can reduce noise interference in the image. Therefore, after the first image processing of the white light image, the clarity of the first vehicle's white light image is higher.

[0088] Therefore, after the first image processing, the clarity of the license plate area in the first vehicle white light image is significantly improved compared to the vehicle white light image, and it contains as much detailed information as possible, resulting in a better overall noise reduction effect.

[0089] Sa2. Perform a second image processing on the vehicle white light image to obtain a second vehicle white light image with optimized vehicle body display effect.

[0090] In some embodiments, the brightness of the second vehicle white light image is greater than the brightness of the first vehicle white light image. Since the brightness of the window area in the vehicle white light image is relatively low, the second image processing includes image processing steps to improve image brightness and sharpness, thereby obtaining a second vehicle white light image with a brightness greater than that of the first vehicle white light image.

[0091] In some embodiments, the second image processing may include several of the aforementioned ISP processing methods, specifically including: interpolation, sharpening, white balance, color correction, noise reduction, gamma correction, dynamic range enhancement, and luminance gain. Specifically, gamma correction can be performed to correct the nonlinear photoelectric conversion characteristics of the sensor in electronic devices such as cameras, to ensure that the subsequent display produces an image with normal brightness to the human eye, thus optimizing the brightness of the second vehicle white light image after the second image processing. Dynamic range enhancement can increase the brightness ratio between the brightest and darkest areas of the target vehicle in the shooting scene, thus optimizing the brightness of the second vehicle white light image after the second image processing. Luminance gain is used to increase the brightness of the image, thus optimizing the brightness of the second vehicle white light image after the second image processing.

[0092] Therefore, after the second image processing, the clarity of the window area in the second vehicle white light image is optimized compared to the vehicle white light image, and the brightness of the window area and the vehicle body area is significantly optimized.

[0093] Because the second image processing step adds gamma correction, dynamic range enhancement, and brightness gain operations to increase image brightness compared to the first image processing step, the license plate area may experience overexposure. To resolve this overexposure issue, step Sa3 is required.

[0094] Sa3. The white light image of the first vehicle and the white light image of the second vehicle are fused to obtain a first fused image with clear license plate area and optimized brightness of window area and body area.

[0095] The license plate area of ​​the first fused image is clear, and the brightness of the window area and the body area is optimized.

[0096] The process involves fusing a first vehicle image with lower brightness to a second vehicle image with higher brightness; this is known as wide dynamic range (WDR) fusion. Based on the pixel brightness of the first and second vehicle images, the two images are merged into a single WDR fused image. It's understandable that the clarity and brightness of the license plate area in the first fused image after WDR fusion comes from the first vehicle image, while the brightness of the window area comes from the second vehicle image. Therefore, compared to the original vehicle image, the brightness of the window and vehicle body areas in the first fused image is significantly optimized.

[0097] In some embodiments, an image fusion module with a wide dynamic range fusion chip can be used to perform wide dynamic range fusion processing on the white light image of the first vehicle and the white light image of the second vehicle.

[0098] In other embodiments, a wide dynamic range fusion algorithm can also be used to perform wide dynamic range fusion processing on the white light image of the first vehicle and the white light image of the second vehicle.

[0099] For example, the first vehicle white light image and the second vehicle white light image are fed into a fusion module equipped with a wide dynamic range fusion chip. The fusion algorithm built into the wide dynamic range fusion chip is used to perform wide dynamic range fusion processing on the first vehicle white light image and the second vehicle white light image. Furthermore, the Ispdgain multiple in the second image processing is used as the difference in the exposure ratio of the first vehicle white light image and the second vehicle white light image during the fusion process, thereby obtaining a first fused image with a clear license plate area and optimized brightness in the window area and the vehicle body area.

[0100] Sa4. The first fused image is fused with the vehicle infrared image to obtain a target fused image with optimized clarity in the license plate area and window area, and optimized brightness in the window area and vehicle body area.

[0101] Understandably, since the license plate area of ​​the first fused image is clear and the brightness of the window and body areas is optimized, and the window area of ​​the vehicle infrared image is clear, the target fused image obtained after fusion has the following characteristics: the license plate area is clear, the window area is clear, the brightness of the window area is good, and the brightness of the body area is good.

[0102] In some embodiments, such as Figure 6 As shown, acquiring vehicle infrared images may also include steps Sb1 to Sb2.

[0103] Sb1. Infrared strobe illumination is used to acquire original infrared images of the vehicle.

[0104] Sb2. Perform fourth image processing on the original vehicle infrared image to obtain a vehicle infrared image with optimized display effect in the window area.

[0105] The fourth image processing step includes image processing steps to improve image brightness and clarity.

[0106] In some embodiments, the fourth image processing may include several of the aforementioned ISP processing methods, specifically including: interpolation, sharpening, noise reduction, and gamma correction. After the fourth image processing, the clarity of the window area in the vehicle infrared image is significantly improved compared to the original vehicle infrared image.

[0107] Because infrared light was used for supplemental illumination when acquiring the original vehicle infrared image, the colors in the original vehicle infrared image differ significantly from the true values ​​visible to the human eye. Therefore, white balance and color correction operations are unnecessary in the fourth image processing step. Furthermore, since the original vehicle infrared image is already quite bright, brightness gain operations are not required to avoid overexposure of the overall vehicle image.

[0108] Understandably, the first fused image obtained after wide dynamic range fusion processing has better clarity in the license plate area and better brightness in the window and body areas. The window area of ​​the vehicle infrared image also has better clarity. Therefore, fusing the first fused image with the vehicle infrared image can yield a target fused image with optimized clarity in the license plate and window areas and optimized brightness in the window and body areas. The target fused image contains as much detail as possible and has a better noise reduction effect. Therefore, the vehicle image obtained by this method is easier to identify and has a better visual effect.

[0109] In some embodiments, such as Figure 7 As shown, after obtaining the target fused image, the license plate area and the window area can be further fused. The processing can also include steps S201 to S203.

[0110] S201. Merge the license plate region in the white light image of the first vehicle with the license plate region in the target fusion image to obtain the first vehicle fusion image.

[0111] In some embodiments, the license plate region in the first vehicle white light image is located in the same position as the license plate region in the target fused image. Therefore, the license plate region can be identified by an intelligent region recognition algorithm to obtain the location information of the license plate region. This allows for the image parameter fusion calculation of the image parameters in the license plate region of the first vehicle white light image and the image parameters in the license plate region of the target fused image.

[0112] In some embodiments, the image parameter fusion calculation may be to take the average of the image parameters in the license plate region of the first vehicle white light image and the image parameters in the license plate region of the target fused image, or to add the image parameters in the license plate region of the first vehicle white light image and the image parameters in the license plate region of the target fused image. This embodiment does not impose specific limitations.

[0113] Since the license plate area in the first vehicle white light image has good clarity and the license plate area in the target fused image has good clarity, the clarity of the license plate area in the first vehicle fused image obtained after fusion is further optimized, the license plate area contains as much detailed information as possible, and the image has a better noise reduction effect.

[0114] S202, fuse any two or three of the following: the window area in the vehicle infrared image, the window area in the target fusion image, and the color-optimized window area in the vehicle white light image, to obtain a second vehicle fusion image.

[0115] In some embodiments, to obtain the color-optimized window area in a vehicle white light image, a third image processing can be performed on the vehicle white light image to obtain a third vehicle white light image with color optimization of the window area, thereby obtaining the color-optimized window area.

[0116] In some embodiments, the third image processing may include several of the aforementioned ISP processing methods, specifically including: interpolation operation, sharpening operation, white balance operation, second color correction operation, noise reduction operation, gamma correction operation, and dynamic range enhancement operation. The second color correction operation includes processing the original vehicle white light image using color parameters greater than a preset saturation, thereby obtaining a third vehicle white light image with a higher saturation than the first vehicle white light image.

[0117] In some embodiments, a deep regression network model can be used to learn color adjustment parameters for the original vehicle white light image. Specifically, learning color adjustment parameters for the first original image using a deep regression network model involves generating different color data pairs using varying values ​​of brightness, contrast, and saturation to learn the color adjustment parameters, thereby maintaining the normal color of the target vehicle while reducing color noise.

[0118] Understandably, since vehicle infrared images are acquired through infrared exposure, the window areas in both the infrared image and the target fusion image may appear darker. Therefore, to address this issue, a third-party image processing technique can be applied to the vehicle's white light image to obtain a third vehicle white light image with optimized window area colors, thus enabling the acquisition of the color-optimized window area.

[0119] In some embodiments, the window region in the vehicle infrared image is located in the overall image, and the window region in the third vehicle white light image is located in the overall image. This location is consistent with the location of the window region in the target fused image. Therefore, the location information of the window region can be obtained by using an intelligent region recognition algorithm to identify the location of the window region. This allows for the combination of any two or all three of the image parameters in the window region of the vehicle infrared image, the window region of the third vehicle white light image, and the window region of the target fused image for image parameter fusion calculation.

[0120] In some embodiments, the image parameter fusion calculation may be to take the average of any two or three of the image parameters in the window area of ​​the vehicle infrared image, the image parameters in the window area of ​​the third vehicle white light image, and the image parameters in the window area of ​​the target fused image, or to add any two or three of the image parameters in the window area of ​​the vehicle infrared image, the image parameters in the window area of ​​the third vehicle white light image, and the image parameters in the window area of ​​the target fused image. This embodiment does not impose specific limitations.

[0121] Because the window area in the vehicle infrared image has good clarity, the window area in the target fusion image has good clarity, and the window area with optimized color has been added, the clarity and color of the window area in the second vehicle fusion image obtained after fusion are further optimized. The window area contains as much detail information as possible, the color is more in line with the true color that can be seen by the human eye, and the image has a better noise reduction effect.

[0122] S203. The first vehicle fusion image and the second vehicle fusion image are fused together to obtain the third vehicle fusion image.

[0123] Understandably, fusing the license plate area in the first vehicle's white light image with the license plate area in the target fusion image yields a first vehicle fusion image with a clear license plate area. Fusing the window area in the vehicle's infrared image, the window area in the target fusion image, and the color-optimized window area in the vehicle's white light image yields a second vehicle fusion image with a clear window area. Fusing the first and second vehicle fusion images further yields a third vehicle fusion image with optimized clarity in both the license plate and window areas. The third vehicle fusion image contains as much detail as possible in both the license plate and window areas, exhibits superior noise reduction, and has colors that more closely match the true colors visible to the human eye. Therefore, the accuracy of identifying target vehicles, the precision of license plate number recognition, and the accuracy of driver recognition in the window area can be improved through the third vehicle fusion image.

[0124] The following example, a truck driving in darkness with a high-intensity light installed above its license plate, will be used to further illustrate the method described in the foregoing embodiments of this application. Figure 8 As shown, the method may include the following steps (1) to (10).

[0125] (1) such as Figure 9 As shown, a RAW format vehicle white light image A is obtained by using a low shutter speed, low gain parameters, and a low-brightness white light fill light method.

[0126] (2) For example Figure 10As shown, a raw vehicle infrared image B in RAW format was obtained using normal shutter speed and normal gain parameters, along with infrared strobe illumination.

[0127] (3) Perform first ISP processing on the vehicle white light image A to obtain the first vehicle white light image A1 in YUV format; the ISP processing in this step includes interpolation operation, sharpening operation, white balance operation, first color correction operation and noise reduction operation, wherein the first color correction operation does not include processing the vehicle white light image using color parameters with a saturation greater than the preset value.

[0128] (4) Perform a second ISP process on the vehicle white light image A to obtain a second vehicle white light image A2 in YUV format; the ISP process in this step includes interpolation, sharpening, white balance, first color correction, noise reduction, gamma correction, dynamic range enhancement and brightness gain.

[0129] (5) Send the white light image of the first vehicle and the white light image of the second vehicle into the fusion module with a wide dynamic range fusion chip for wide dynamic range fusion processing to obtain the first fused image C in YUV format.

[0130] (6) Perform fourth ISP processing on the original vehicle red light image B to obtain a vehicle infrared image B1 in YUV format; the ISP processing in this step mainly includes interpolation, sharpening, white balance, color correction and noise reduction.

[0131] (7) Perform white light infrared fusion of the first fused image C and the vehicle infrared image B1 to obtain the target fused image D in YUV format.

[0132] (8) Perform third ISP processing on the vehicle white light image A, and use a deep regression network model to learn color adjustment parameters for the white light image to obtain a third vehicle white light image A3 in YUV format; the ISP processing in this step includes interpolation, sharpening, white balance, second color correction, noise reduction, gamma correction, and dynamic range enhancement. The second color correction includes processing the vehicle white light image using color parameters with a saturation greater than the preset value.

[0133] (9) The license plate area and the window area are identified by the intelligent area recognition algorithm to obtain the license plate area and the window area respectively.

[0134] (10) Perform image parameter fusion calculation by combining the image parameters of the license plate region in the first vehicle white light image A1 with the image parameters of the license plate region in the target fusion image D. Perform image parameter fusion calculation by combining the image parameters of the window region in the vehicle infrared image B1, the image parameters of the window region in the third vehicle white light image A3, and the image parameters of the window region in the target fusion image D to obtain the fused target vehicle image. The target vehicle image is as follows: Figure 11 As shown.

[0135] Understandably, when the target vehicle is a truck with a high-intensity headlight installed above its license plate while driving in the dark, the obtained image of the target vehicle shows good clarity in both the license plate area and the window area, and the colors match the true colors visible to the human eye. The target vehicle image obtained by this method has good noise reduction effect and contains more detailed information. Therefore, by recognizing this target vehicle image, the accuracy of recognizing the license plate number in the license plate area and the accuracy of recognizing the driver in the window area can be improved.

[0136] In the case of implementing the functions of the integrated units described above in hardware, embodiments of this application provide a schematic diagram of the hardware composition of an electronic device, such as... Figure 12 As shown, the electronic device also includes: a processor 301, a communication interface 302, and a bus 304. Optionally, the electronic device may also include a memory 303.

[0137] Processor 301 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0138] Communication interface 302 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0139] The memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0140] As one possible implementation, the memory 303 can exist independently of the processor 301. The memory 303 can be connected to the processor 301 via a bus 304 and is used to store instructions or program code. When the processor 301 calls and executes the instructions or program code stored in the memory 303, it can implement the image processing method provided in the embodiments of this application.

[0141] In another possible implementation, the memory 303 can also be integrated with the processor 301.

[0142] Bus 304 can be an extended industry standard architecture (EISA) bus, etc. Bus 304 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 12 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0143] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the electronic device can be divided into different functional modules to complete all or part of the functions described above.

[0144] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware. The program can be stored in the aforementioned computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be any of the foregoing embodiments or memory. The aforementioned computer-readable storage medium can also be an external storage device of the aforementioned electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the aforementioned electronic device. Further, the aforementioned computer-readable storage medium can include both internal storage units and external storage devices of the aforementioned electronic device. The aforementioned computer-readable storage medium is used to store the aforementioned computer program and other programs and data required by the aforementioned electronic device. The aforementioned computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0145] This application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform any of the image processing methods provided in the above embodiments.

[0146] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0147] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0148] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, include: Using the first exposure parameters, a clear white light image of the vehicle with the license plate area is acquired; wherein, the brightness of the license plate area in the white light image of the vehicle is greater than the brightness of the window area; Infrared strobe illumination is used to capture clear infrared images of the vehicle in the window area; The vehicle white light image is subjected to a first image processing to obtain a first vehicle white light image with optimized license plate area display effect; The vehicle white light image is subjected to a second image processing to obtain a second vehicle white light image with optimized vehicle body display effect; The white light image of the first vehicle and the white light image of the second vehicle are fused to obtain a first fused image; The first fused image and the vehicle infrared image are fused to obtain the target fused image.

2. The method of claim 1, wherein, The brightness of the second vehicle white light image is greater than the brightness of the first vehicle white light image.

3. The method according to claim 1, characterized in that, The method further includes: The license plate region in the first vehicle white light image is fused with the license plate region in the target fused image to obtain the first vehicle fused image.

4. The method according to claim 3, characterized in that, The method further includes: The second vehicle fusion image is obtained by fusing any two or three of the following: the window area in the vehicle infrared image, the window area in the target fusion image, and the color-optimized window area in the vehicle white light image.

5. The method according to claim 4, characterized in that, The method further includes: The first vehicle fusion image and the second vehicle fusion image are fused together to obtain a third vehicle fusion image.

6. The method according to claim 4, characterized in that, The method further includes: The vehicle white light image is subjected to third image processing to obtain a third vehicle white light image with optimized window area color.

7. The method according to any one of claims 1 to 3, characterized in that, The method of using infrared strobe illumination to acquire clear infrared images of the vehicle in the window area includes: The original infrared image of the vehicle is acquired using the aforementioned infrared strobe illumination. The original vehicle infrared image is subjected to a fourth image processing step to obtain a vehicle infrared image with optimized display effect in the window area.

8. The method according to claim 1, characterized in that, The first image processing includes: interpolation operation, sharpening operation, white balance operation, first color correction operation, and noise reduction operation, but does not include gamma correction operation, dynamic range enhancement operation, or brightness gain operation. The first color correction operation does not include processing the vehicle white light image using color parameters with a saturation greater than a preset value.

9. The method according to claim 1, characterized in that, The second image processing includes: interpolation operation, sharpening operation, white balance operation, first color correction operation, noise reduction operation, gamma correction operation, dynamic range enhancement operation, and brightness gain operation.

10. The method according to claim 6, characterized in that, The third image processing includes: interpolation operation, sharpening operation, white balance operation, second color correction operation, noise reduction operation, gamma correction operation, and dynamic range enhancement operation. The second color correction operation includes processing the vehicle white light image using color parameters greater than a preset saturation, and learning color adjustment parameters for the vehicle white light image using a deep regression network model.

11. The method according to claim 7, characterized in that, The fourth image processing includes interpolation, sharpening, noise reduction, gamma correction, and dynamic range enhancement, but excludes white balance and color correction.

12. An image processing apparatus, characterized in that, The device includes: an image acquisition unit and an image fusion unit; The image acquisition unit is used to acquire a clear white light image of the vehicle with the license plate area using a first exposure parameter; wherein, the brightness of the license plate area in the white light image of the vehicle is greater than the brightness of the window area; The image acquisition unit is also used to acquire clear infrared images of the vehicle in the window area using infrared strobe illumination. The image fusion unit is configured to perform a first image processing on the vehicle white light image to obtain a first vehicle white light image with optimized license plate area display effect; perform a second image processing on the vehicle white light image to obtain a second vehicle white light image with optimized vehicle body display effect; fuse the first vehicle white light image and the second vehicle white light image to obtain a first fused image; and fuse the first fused image with the vehicle infrared image to obtain a target fused image.

13. An electronic device, characterized in that, The device includes a processor and a memory, the memory being used to store computer instructions, and the processor being used to retrieve and execute the computer instructions from the memory to implement the image processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes: computer software instructions; when the computer software instructions are executed in an image processing apparatus, the image processing apparatus causes the image processing apparatus to implement the image processing method as described in any one of claims 1 to 11.

Images