Image halo removal method, device, computer equipment and readable storage medium

Abstract

The present application relates to a method, device, computer equipment and computer-readable storage medium for removing halo from an image. The method comprises: obtaining an original halo image, performing multiple scaling processes based on the original halo image to obtain halo images of multiple scales; performing multi-scale feature extraction on the halo images of multiple scales to obtain halo feature maps of multiple scales; performing multi-scale fusion based on the halo feature maps of multiple scales to obtain a halo guide map; based on the halo guide map, removing the halo from the original halo image to obtain a target image. The method can improve the quality of the electronic rearview mirror display image by accurately locating and effectively removing the halo, ensure the driver's clear field of vision, and reduce the risk of night driving. In addition, the entire process does not require the addition of additional physical components, and only relies on software algorithms to achieve high-quality halo removal, which greatly saves manufacturing costs.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, computer equipment, computer-readable storage medium and computer program product for removing image halo. Background Art

[0002] Compared with daytime, driving at night often involves higher risks. One of the important reasons is that when driving at night, both physical and electronic rearview mirrors are easily affected by strong light, which will produce large halos on the mirror / display screen, causing dazzle to the driver and affecting safe driving.

[0003] However, at present, whether it is preventing glare by changing the physical structure of the rearview mirror or preventing glare by image processing algorithms, there are obvious defects. For example, preventing glare by changing the physical structure of the rearview mirror requires adding additional physical components to the rearview mirror, such as filters, light sensors, etc., which increases the manufacturing cost. And adding these physical devices may reduce the brightness of the mirror or screen, which in turn affects the overall visual clarity and the driver's visual experience. Preventing glare through image processing algorithms is often difficult to control accurately, which may lead to over-processing or under-processing, affecting the quality of the final image. And it is unstable when facing halos of different sizes and intensities, and the false detection rate is high. Summary of the invention

[0004] Based on this, it is necessary to provide an image halo removal method, device, computer equipment, computer readable storage medium and computer program product that can improve the halo removal effect and save manufacturing costs in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for removing image halo, comprising:

[0006] Acquire an original halo image, and perform multiple scaling processes based on the original halo image to obtain halo images of multiple scales;

[0007] Performing multi-scale feature extraction on the halo images of the multiple scales to obtain halo feature maps of the multiple scales;

[0008] Perform multi-scale fusion based on the halo feature maps of the multiple scales to obtain a halo guiding map;

[0009] Based on the halo guide map, the halo in the original halo image is removed to obtain a target image.

[0010] In one embodiment, the step of performing multi-scale feature extraction on the halo images of the multiple scales to obtain halo feature maps of the multiple scales includes:

[0011] Convolution processing is performed on the halo images of the multiple scales to obtain convolution features of the multiple scales; for each scale, feature fusion is performed based on the convolution features of the targeted scale and the halo feature map of the previous scale of the targeted scale to obtain the fusion features of the targeted scale; feature extraction is performed on the fusion features of the targeted scale to obtain the halo feature map of the targeted scale.

[0012] In one embodiment, the previous scale is larger than the targeted scale, and for each scale, performing feature fusion based on the convolution feature of the targeted scale and the halo feature map of the previous scale of the targeted scale to obtain the fusion feature of the targeted scale includes:

[0013] For each scale, the halo feature map of the previous scale of the targeted scale is downsampled to obtain a downsampled halo feature map, and the downsampled halo feature map and the convolution feature of the targeted scale have the same scale; the convolution feature of the targeted scale and the downsampled halo feature map are channel-joined to obtain a fusion feature of the targeted scale.

[0014] In one embodiment, performing multi-scale fusion based on the halo feature maps of the multiple scales to obtain a halo guide map includes:

[0015] For each scale, the halo feature map of the targeted scale is upsampled to obtain the upsampled halo feature map; the halo feature map of the previous scale of the targeted scale is determined, and the upsampled halo feature map and the halo feature map of the previous scale are merged to obtain the guiding feature map of the targeted scale; the upsampled halo feature map and the halo feature map of the previous scale have the same scale; the guiding feature map of the last scale among the multiple scales is used as the halo guiding map.

[0016] In one embodiment, removing the halo in the original halo image based on the halo guide map to obtain the target image includes:

[0017] Based on the number of channels of the maximum-scale halo image, the halo guide map is channel-aligned to obtain an aligned halo guide map; based on the pixels of the maximum-scale halo image, the aligned halo guide map is scaled to obtain a scaled halo guide map; the scaled halo guide map is intensity-adjusted to obtain an adjusted guide map; based on the adjusted guide map, the halo in the original halo image is removed to obtain a target image.

[0018] In one embodiment, the step of adjusting the intensity of the scaled halo guidance map to obtain the adjusted guidance map includes:

[0019] The scaled halo guide map is divided into a plurality of sub-region maps; for each sub-region map, an intensity adjustment operator corresponding to the sub-region map is determined; based on the intensity adjustment operator corresponding to the sub-region map, the intensity of the sub-region map is adjusted to obtain an adjusted guide map.

[0020] In a second aspect, the present application also provides an image halo removal device, comprising:

[0021] A scaling module, used for acquiring an original halo image, and performing multiple scaling processes based on the original halo image to obtain halo images of multiple scales;

[0022] An extraction module, used for performing multi-scale feature extraction on the halo images of the multiple scales to obtain halo feature maps of the multiple scales;

[0023] A fusion module, used for performing multi-scale fusion based on the halo feature maps of the multiple scales to obtain a halo guide map;

[0024] The removal module is used to remove the halo in the original halo image based on the halo guide map to obtain a target image.

[0025] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0026] Acquire an original halo image, and perform multiple scaling processes based on the original halo image to obtain halo images of multiple scales;

[0027] Performing multi-scale feature extraction on the halo images of the multiple scales to obtain halo feature maps of the multiple scales;

[0028] Perform multi-scale fusion based on the halo feature maps of the multiple scales to obtain a halo guiding map;

[0029] Based on the halo guide map, the halo in the original halo image is removed to obtain a target image.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0031] Acquire an original halo image, and perform multiple scaling processes based on the original halo image to obtain halo images of multiple scales;

[0032] Performing multi-scale feature extraction on the halo images of the multiple scales to obtain halo feature maps of the multiple scales;

[0033] Perform multi-scale fusion based on the halo feature maps of the multiple scales to obtain a halo guiding map;

[0034] Based on the halo guide map, the halo in the original halo image is removed to obtain a target image.

[0035] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0036] Acquire an original halo image, and perform multiple scaling processes based on the original halo image to obtain halo images of multiple scales;

[0037] Performing multi-scale feature extraction on the halo images of the multiple scales to obtain halo feature maps of the multiple scales;

[0038] Perform multi-scale fusion based on the halo feature maps of the multiple scales to obtain a halo guiding map;

[0039] Based on the halo guide map, the halo in the original halo image is removed to obtain a target image.

[0040] The above-mentioned image halo removal method, device, computer equipment, computer-readable storage medium and computer program product obtain halo images of multiple scales by acquiring the original halo image and performing multiple scaling processes based on the original halo image; this helps the algorithm to be more robust when processing halos of various sizes and intensities, and improves the adaptability to different driving scenes. In addition, multi-scale input can capture more detailed features, reducing missed detection or false detection caused by a single scale.

[0041] Multi-scale feature extraction is performed on the halo images of the multiple scales to obtain halo feature maps of multiple scales; the position and shape of the halo can be more accurately identified, providing a reliable basis for subsequent processing. Multi-scale feature extraction can describe the halo characteristics from different levels, so that the model can better understand the halo phenomenon in complex scenes, thereby improving the removal effect.

[0042] Based on the halo feature maps of the multiple scales, multi-scale fusion is performed to obtain a halo guide map; the advantages of each scale can be combined to obtain a more comprehensive and detailed halo guide map to guide subsequent halo removal operations. It can be understood that the generated halo guide map is not the image after the halo is directly removed, but provides an intermediate representation that allows the degree of halo removal to be adjusted according to actual needs, thereby enhancing the flexibility of the system.

[0043] Based on the halo guide map, the halo in the original halo image is removed to obtain the target image. By accurately locating and effectively removing the halo, the quality of the electronic rearview mirror display is improved, the driver's clear vision is guaranteed, and the risk of night driving is reduced. In addition, the entire process does not require additional physical components such as filters or light sensors, and only relies on software algorithms to achieve high-quality halo removal, which greatly saves manufacturing costs and avoids the brightness loss problem caused by physical devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A diagram showing an application environment of a method for removing image halo in an embodiment;

[0046] Figure 2 is a schematic diagram of a flow chart of a method for removing image halo in one embodiment;

[0047] Figure 3 A schematic diagram of a framework of an image halo removal model in one embodiment;

[0048] Figure 4 is a schematic flow chart of a method for removing image halo in another embodiment;

[0049] Figure 5 is a structural block diagram of an image halo removal device in one embodiment;

[0050] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The image halo removal method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 is used to generate an image halo removal request and send the image halo removal request to the server 104, so that the server 104 removes the halo in the original halo image based on the halo guide map to obtain the target image. Among them, the terminal 102 can be but not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0053] In an exemplary embodiment, Figure 2 As shown, a method for removing image halo is provided, and the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 202 to 208. Among them:

[0054] Step 202 , obtaining an original halo image, performing multiple scaling processes based on the original halo image, and obtaining halo images of multiple scales.

[0055] The original halo image refers to the original image obtained from the electronic rearview mirror or camera, which contains the halo caused by strong light. These halos will seriously affect the driver's vision and cause dazzle.

[0056] The halo images of multiple scales refer to that the original halo image is converted into images of multiple different scales through multiple scaling processes.

[0057] Specifically, first, the original image, i.e., the original halo image, is obtained from the electronic rearview mirror or camera, which contains the halo caused by strong light. These halos will seriously affect the driver's vision and cause glare. Then the original halo image is uniformly scaled to a standard size (e.g., 640x640 size) to ensure the consistency of all input data. Finally, the scaled original halo image is gradually reduced by 2 times, 4 times, 8 times, and 16 times to generate a series of different resolution versions of the same image, i.e., halo images of multiple scales. It can be understood that these halo images of multiple scales will be used in the next multi-scale feature extraction process to help the algorithm more comprehensively understand the existence form and characteristics of the halo.

[0058] In one example, reference Figure 3 ,exist Figure 3 In the data preprocessing module, the input data is the original halo image with a scale of N×W×H. First, the original halo image is scaled by the scaling operator to obtain a halo image with a scale of N×640×640. Then the halo image of N×640×640 is downsampled to obtain a halo image with a scale of N×320×320. The halo image of N×320×320 is further downsampled to obtain a halo image with a scale of N×160×160. The halo image of N×160×610 is further downsampled to obtain a halo image with a scale of N×80×80. The halo image of N×80×80 is further downsampled to obtain a halo image with a scale of N×40×40.

[0059] The halo images of multiple scales obtained through the above steps include: a halo image of a scale of N×640×640, a halo image of a scale of N×320×320, a halo image of a scale of N×160×160, a halo image of a scale of N×80×80, and a halo image of a scale of N×40×40. Optionally, the halo images of multiple scales are channel-aligned to obtain multiple channel-aligned halo images. For example, halo images with a scale of N×640×640, halo images with a scale of N×320×320, halo images with a scale of N×160×160, halo images with a scale of N×80×80, and halo images with a scale of N×40×40 are input to respective convolutional layers for channel alignment, and halo images with a scale of 16×640×640, halo images with a scale of 16×320×320, halo images with a scale of 16×160×160, halo images with a scale of 16×80×80, and halo images with a scale of 16×40×40 are obtained. It can be understood that the above-mentioned channel alignment step can ensure that subsequent processing steps can consistently process various types of image data while retaining or enhancing useful information as much as possible.

[0060] Step 204 , performing multi-scale feature extraction on the halo images of multiple scales to obtain halo feature maps of multiple scales.

[0061] Among them, the halo feature maps of multiple scales refer to a series of feature maps obtained by processing halo images of different resolutions (i.e., different scales). Each feature map is a two-dimensional matrix representing the halo characteristics at a specific scale, which contains information about the location, shape, size, intensity, etc. of the halo. It can be understood that these feature maps are not directly visualized images, but abstract feature expressions extracted by convolutional neural networks (CNN).

[0062] Specifically, feature extraction is performed on the halo image of each scale. Feature extraction is mainly completed through convolution layers. Optionally, the feature extraction process of each scale includes: first, channel splicing the halo image of the current scale with the feature map of the previous scale through channel splicing. Then, the local features in the image are extracted through the convolution layer. Finally, a feature map of a smaller scale is generated through a downsampling operation for multi-scale processing. Through the above steps, halo feature maps of multiple scales are generated. The feature map of each scale contains the feature information of the halo at that scale.

[0063] Step 206: Perform multi-scale fusion based on the halo feature maps of multiple scales to obtain a halo guiding map.

[0064] Among them, the halo guidance map refers to an intermediate representation generated by multi-scale feature extraction and multi-scale fusion steps. It can be understood that the halo guidance map is not the image after the halo is directly removed, but a reference information for guiding the subsequent steps on how to remove the halo from the original halo image. Specifically, the halo guidance map contains information about the halo position, size, intensity and other features, which are extracted and fused from the halo feature maps of different scales. It provides instructions to the data post-processing module on which areas need to be halo removed and how to remove this halo.

[0065] Specifically, firstly, the features of the halo feature map of the smallest scale are fused to obtain the fused feature map of the smallest scale. The fused feature map of the smallest scale is upsampled to obtain the halo guidance map of the smallest scale. Then, the halo feature maps of each subsequent scale are feature fused. The fused feature map of the current scale is channel-joined with the halo guidance map of the previous scale. The joined feature maps are feature fused to obtain the fused feature map of the current scale. The fused feature map of the current scale is upsampled to obtain the halo guidance map of the current scale. Finally, the above steps are repeated until the halo guidance map of the largest scale is generated.

[0066] Step 208: based on the halo guide map, remove the halo in the original halo image to obtain a target image.

[0067] Among them, halo refers to the bright area or light spot on the display screen of the electronic rearview mirror caused by reflection, scattering or diffraction of strong light sources such as car lights and street lights. These light spots will interfere with the driver's vision, cause dazzle, and affect driving safety. In the image, it appears as an abnormally bright area, which may be accompanied by radial lines (starbursts), circular bright spots (glare) or other forms of light bands.

[0068] The target image refers to the final output image after processing and removing the halo phenomenon in the original halo image. This image should restore the true visual effect of the original scene as much as possible, reduce or eliminate the visual interference caused by the halo, so as to ensure that the driver can clearly see the surrounding environment.

[0069] Specifically, a convolutional layer is first used to convert the halo guide map into the same number of channels as the original halo image, so that the halo guide map can be directly applied to the original halo image to achieve pixel-level corresponding processing.

[0070] Then, the channel-aligned halo guide map is scaled to the same size as the original halo image. This usually involves an interpolation algorithm to increase or decrease the number of pixels in the image to fit the target resolution. The purpose is to ensure that the halo guide map can be accurately mapped to every pixel of the original halo image in order to accurately guide the halo removal work.

[0071] Next, an adjustable parameter or interface is provided to allow the user to adjust the degree of halo removal according to the actual situation. This can be achieved through simple mathematical functions (such as linear transformation, gamma correction, etc.), or more complex non-linear transformations. The purpose is to allow the halo removal effect to be fine-tuned according to the user's preferences or the needs of specific application scenarios, so as to obtain a better visual experience.

[0072] Finally, the halo guide map with adjusted intensity is fused with the original halo image to generate the final target image, which should have better clarity and contrast, and effectively reduce or eliminate the interference caused by the halo. This process may be completed by weighted summation, multiplication combination or other forms of image operations, with the purpose of using the information of the halo guide map to suppress or eliminate the halo phenomenon in the original image. Optionally, after completing all the above processing, the result is saved as a new image file, or directly displayed on the screen of the electronic rearview mirror for the driver to view, so as to provide the user with a clean, glare-free field of view and improve driving safety.

[0073] In one of the embodiments, convolution processing is performed on halo images of multiple scales to obtain convolution features of multiple scales; for each scale, feature fusion is performed based on the convolution features of the targeted scale and the halo feature map of the previous scale of the targeted scale to obtain the fused features of the targeted scale; feature extraction is performed on the fused features of the targeted scale to obtain the halo feature map of the targeted scale.

[0074] Among them, the convolution feature refers to the feature map obtained by extracting local features of the halo image through the convolution layer. The fusion feature refers to the feature map obtained by channel splicing the convolution feature map of the current scale with the halo feature map of the previous scale and performing further convolution processing.

[0075] Specifically, feature extraction is performed on the convolution features of the first scale to obtain a halo feature map of the first scale; for each scale among multiple scales except the first scale, feature fusion is performed based on the convolution features of the targeted scale and the halo feature map of the previous scale of the targeted scale to obtain a fused feature of the targeted scale; feature extraction is performed on the fused features of the targeted scale to obtain a halo feature map of the targeted scale.

[0076] Through multi-scale feature extraction, halo information of different scales can be captured. Small-scale features can capture local details, and large-scale features can capture global information. Then, through channel splicing, feature maps of different scales are merged to form a more comprehensive feature representation. Enhance the diversity and richness of features and improve the expressiveness of the model. Through the above steps, the multi-scale feature extraction module can effectively capture and fuse halo features of different scales, thereby improving the accuracy of halo detection and removal.

[0077] In one of the embodiments, for each scale, the halo feature map of the previous scale of the targeted scale is downsampled to obtain a downsampled halo feature map, and the downsampled halo feature map and the convolution feature of the targeted scale have the same scale; the convolution feature of the targeted scale and the downsampled halo feature map are channel-joined to obtain a fusion feature of the targeted scale.

[0078] Specifically, first, the convolution feature map of the current scale and the halo feature map of the previous scale are taken as input, and the halo feature map of the previous scale is downsampled so that it has the same scale as the convolution feature map of the current scale. Optionally, downsampling is usually achieved through pooling (such as maximum pooling or average pooling) or convolution operations with larger steps. Then the convolution feature map of the current scale and the downsampled halo feature map are taken as input, and the convolution feature map of the current scale and the downsampled halo feature map are channel-joined. It can be understood that channel jointing refers to merging the number of channels of two feature maps to form a new feature map. Finally, the feature map after channel jointing is taken as input, and the feature map after channel jointing is further convolved to extract fusion features. Feature fusion is performed through the convolution layer to generate a fusion feature map of the current scale.

[0079] In one example, reference Figure 3 ,exist Figure 3 In the feature extraction module, the halo image with a scale of 16×320×320 is used as the first scale image to be processed. First, the halo image with a scale of 16×640×640, which is the previous scale of the first scale, is subjected to multiple feature extraction through the convolution layer, and the halo image with a scale of 16×640×640 after feature extraction is downsampled to obtain a downsampled halo feature map with a scale of 16×320×320. It should be noted that the downsampled halo feature map with a scale of 16×320×320 and the halo image with a scale of 16×320×320 have the same scale. Then, the downsampled halo feature map with a scale of 16×320×320 and the halo image with a scale of 16×320×320 are channel-joined to obtain a fused feature with a scale of 16×320×320.

[0080] Then the next scale of 16×320×320, i.e., 16×160×160, is used as the image to be processed at the current scale. First, the previous scale of 16×160×160, i.e., the fused features of 16×320×320, is subjected to multiple feature extraction through the convolution layer, and the halo image of 16×320×320 after feature extraction is downsampled to obtain a downsampled halo feature map of 16×160×160. Then, the downsampled halo feature map of 16×160×160 and the halo image of 16×160×160 are channel-joined to obtain a fused feature of 16×160×160.

[0081] Next, the next scale of 16×160×160, that is, 16×80×80, is used as the image to be processed at the current scale, and the above feature extraction, downsampling and channel splicing operations are repeated until the fusion features corresponding to each scale of 16×640×640, 16×320×320, 16×160×160, 16×80×80 and 16×40×40 are obtained.

[0082] Because channel splicing merges feature maps of different scales to form a richer feature representation, the diversity and richness of features are enhanced, allowing the model to better capture the details and global information of the halo. And through downsampling, the size of the feature map is reduced, reducing the computational complexity. The computational efficiency is improved, allowing the model to process images faster in real-time applications.

[0083] In one of the embodiments, for each scale, the halo feature map of the targeted scale is upsampled to obtain an upsampled halo feature map; the halo feature map of the previous scale of the targeted scale is determined, and the upsampled halo feature map and the halo feature map of the previous scale are fused to obtain a guiding feature map of the targeted scale; the upsampled halo feature map has the same scale as the halo feature map of the previous scale; and the guiding feature map of the last scale among multiple scales is used as the halo guiding map.

[0084] Specifically, first, the halo feature map of the current scale is taken as input, and the halo feature map of the current scale is upsampled so that it has the same scale as the halo feature map of the previous scale. Optionally, upsampling can be achieved by methods such as deconvolution (such as transposed convolution) or nearest neighbor interpolation. Then the upsampled halo feature map and the halo feature map of the previous scale are taken as input, and the upsampled halo feature map is fused with the halo feature map of the previous scale to obtain a guided feature map of the targeted scale. Optionally, the fusion method can be a simple addition, multiplication or a more complex convolution operation. Finally, the guided feature map of the last scale is taken as input, and the guided feature map of the last scale is used as the halo guidance map.

[0085] In one example, reference Figure 3 ,exist Figure 3 In the feature fusion module, the image with a scale of 16×40×40 is first processed as the first scale, and the fusion features corresponding to the scale of 16×40×40 are respectively extracted and fused through multiple convolutional layers to obtain the halo feature map corresponding to the scale of 16×40×40 after feature extraction and feature fusion. The halo feature map corresponding to the scale of 16×40×40 is upsampled to obtain the upsampled halo feature map of the scale of 16×80×80.

[0086] Then, the previous scale of the scale 16×40×40 is determined to be 16×80×80. The upsampled halo feature map of the scale 16×80×80 and the halo feature map of the scale 16×80×80 are channel-joined to obtain a guided feature map of the scale 16×80×80. It can be understood that the upsampled halo feature map of the scale 16×80×80 and the halo feature map of the scale 16×80×80 have the same scale.

[0087] Then, the previous scale of 16×40×40 is 16×80×80 as the image to be processed at the current scale. The guided feature map of scale 16×80×80 is subjected to feature fusion through multiple convolutional layers to obtain a guided feature map of scale 16×80×80 after feature fusion. The guided feature map corresponding to the scale of 16×80×80 after feature fusion is upsampled to obtain an upsampled guided feature map of scale 16×160×160. Then, the previous scale of 16×80×80 is determined, that is, the scale of 16×160×160. The upsampled guided feature map of scale 16×160×160 and the halo feature map of scale 16×160×160 are channel-joined to obtain a guided feature map of scale 16×80×80.

[0088] Next, the previous scale of 16×80×80, that is, 16×160×160, is used as the image to be processed at the current scale, and the above-mentioned operations of feature fusion, upsampling, and channel splicing are repeated until the guided feature maps of 16×640×640 are obtained as the halo guide maps.

[0089] Through multi-scale feature fusion, the halo feature information of different scales can be fully utilized. The accuracy of halo detection and removal is improved, because features of different scales can capture different details and global information. Through upsampling, the feature maps of different scales are ensured to be consistent in size. This facilitates subsequent feature fusion and processing, and avoids feature alignment problems caused by inconsistent sizes. Finally, the halo guide map generated by multi-scale feature fusion can more accurately guide the halo removal process. Improve the accuracy and effect of halo removal, reduce erroneous removal and residual halo, and obtain higher quality target images.

[0090] In one of the embodiments, based on the number of channels of the maximum-scale halo image, the channels of the halo guide map are aligned to obtain the aligned halo guide map; based on the pixels of the maximum-scale halo image, the aligned halo guide map is scaled to obtain the scaled halo guide map; the intensity of the scaled halo guide map is adjusted to obtain the adjusted guide map; based on the adjusted guide map, the halo in the original halo image is removed to obtain the target image.

[0091] The number of channels refers to the dimension of an image or feature map at each pixel. For example, an RGB image has 3 channels (red, green, and blue), and a grayscale image has 1 channel. It is understandable that in the process of multi-scale feature extraction and fusion, it is necessary to ensure that the number of channels of feature maps of different scales is consistent in order to perform effective feature fusion and subsequent processing.

[0092] The aligned halo guide map refers to the halo feature map after channel alignment. The purpose is to ensure that the halo guide map is consistent with the halo image of the largest scale in terms of the number of channels for subsequent feature fusion and processing.

[0093] Specifically, the maximum-scale halo image and the halo guide map are first taken as input, and the halo guide map is aligned based on the number of channels of the maximum-scale halo image to obtain the aligned halo guide map. For example, if the maximum-scale halo image has 3 channels (RGB), then ensure that the halo guide map also has 3 channels.

[0094] Then, the aligned halo guide map and the halo image of the largest scale are taken as input, and the aligned halo guide map is scaled based on the pixels of the halo image of the largest scale to obtain the scaled halo guide map. For example, if the size of the halo image of the largest scale is 640x640, the aligned halo guide map is scaled to 640x640.

[0095] Next, the scaled halo guide map is used as input, and the scaled halo guide map is intensity-adjusted to obtain an adjusted guide map. For example, by adjusting the intensity of the pixel value, the intensity distribution of the halo guide map is made more consistent with the intensity distribution of the actual halo.

[0096] Finally, the adjusted guiding map and the original halo image are used as inputs, and based on the adjusted guiding map, the halo in the original halo image is removed to obtain an image after the halo is removed, that is, a target image. For example, the halo area is removed by comparing the adjusted guiding map and the original halo image pixel by pixel.

[0097] Since the halo guide map is consistent with the halo image of the largest scale in terms of the number of channels, the matching degree of the feature map can be improved, and the halo guide map is spatially aligned with the original image, which can improve the accuracy of halo removal. Adjusting the intensity of the halo guide map to make it more consistent with the intensity distribution of the actual halo can improve the effect of halo removal. Based on the adjusted halo guide map, the halo in the original halo image can be accurately removed to obtain a higher quality target image.

[0098] In one of the embodiments, the scaled halo guide map is divided into a plurality of sub-region maps; for each sub-region map, an intensity adjustment operator corresponding to the targeted sub-region map is determined; based on the intensity adjustment operator corresponding to the targeted sub-region map, the intensity of the targeted sub-region map is adjusted to obtain an adjusted guide map.

[0099] The sub-region map refers to dividing the entire scaled halo guide map into multiple smaller, independently processed image regions. It can be understood that each sub-region map is a part of the original halo guide map and usually has a fixed size (e.g., 16x16 pixels or 32x32 pixels).

[0100] Intensity adjustment operators refer to a set of mathematical operations or algorithms used to adjust the halo intensity in a specific sub-region image. Understandably, these operators can dynamically adjust the halo intensity according to the specific characteristics of the sub-region (such as brightness, contrast, etc.), ensuring that the final halo removal effect is more natural and accurate.

[0101] Specifically, the scaled halo guide map is first divided into a plurality of sub-region maps. Each sub-region map is a part of the original halo guide map and has a fixed size (e.g., 16x16 pixels or 32x32 pixels). The division method can be uniform division or non-uniform division based on specific rules (e.g., halo distribution density). Then, for each sub-region map, its characteristics (e.g., brightness, contrast, halo intensity, etc.) are analyzed to determine an intensity adjustment operator suitable for the sub-region map. The selection of the intensity adjustment operator can be automatically selected according to predefined rules or through a training model. Optional methods include but are not limited to linear transformation, gamma correction, histogram equalization or other more complex convolution operations. Next, based on the selected intensity adjustment operator, each sub-region map is intensity adjusted. The adjustment process can be a pixel-by-pixel adjustment to ensure that the halo intensity of each sub-region is properly corrected. Through these adjustments, the halo removal effect of different sub-regions can be made more natural and accurate. Finally, the adjusted guide map is output.

[0102] Since the corresponding intensity adjustment operator is selected or calculated for each sub-region map, adaptive adjustment can be made according to the characteristics of the halo in the region, avoiding the problem of erroneous removal or incomplete removal caused by global unified processing. In addition, the intensity adjustment module supports user adjustment, and users can adjust the halo removal effect according to actual needs, which increases the flexibility of the system and the possibility of personalized settings.

[0103] In an exemplary embodiment, Figure 4 As shown, it includes steps 402 to 408. Among them:

[0104] Step 402, obtaining an original halo image, performing multiple scaling processes based on the original halo image, and obtaining halo images of multiple scales;

[0105] Step 404, performing convolution processing on the halo images of multiple scales to obtain convolution features of multiple scales; for each scale, performing down-sampling processing on the halo feature map of the previous scale of the targeted scale to obtain a down-sampled halo feature map, and the down-sampled halo feature map and the convolution feature of the targeted scale have the same scale; performing channel splicing on the convolution feature of the targeted scale and the down-sampled halo feature map to obtain a fusion feature of the targeted scale; performing feature extraction on the fusion feature of the targeted scale to obtain a halo feature map of the targeted scale;

[0106] Step 406, for each scale, up-sample the halo feature map of the targeted scale to obtain an up-sampled halo feature map; determine the halo feature map of the previous scale of the targeted scale, fuse the up-sampled halo feature map with the halo feature map of the previous scale to obtain a guide feature map of the targeted scale; the up-sampled halo feature map has the same scale as the halo feature map of the previous scale; and use the guide feature map of the last scale among the multiple scales as the halo guide map;

[0107] Step 408, based on the number of channels of the maximum-scale halo image, channels of the halo guide map are aligned to obtain an aligned halo guide map; based on the pixels of the maximum-scale halo image, the aligned halo guide map is scaled to obtain a scaled halo guide map; the scaled halo guide map is divided into a plurality of sub-region maps; for each sub-region map, an intensity adjustment operator corresponding to the sub-region map is determined; based on the intensity adjustment operator corresponding to the sub-region map, intensity adjustment is performed on the sub-region map to obtain an adjusted guide map; based on the adjusted guide map, the halo in the original halo image is removed to obtain a target image.

[0108] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0109] Based on the same inventive concept, the embodiment of the present application also provides an image halo removal device for implementing the image halo removal method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more embodiments of the image halo removal device provided below can refer to the limitations of the image halo removal method above, and will not be repeated here.

[0110] In an exemplary embodiment, Figure 5 As shown, an image halo removal device 500 is provided, comprising: a scaling module 502, an extraction module 504, a fusion module 506 and a removal module 508, wherein:

[0111] A scaling module 502 is used to obtain an original halo image, and perform multiple scaling processes based on the original halo image to obtain halo images of multiple scales;

[0112] An extraction module 504 is used to perform multi-scale feature extraction on halo images of multiple scales to obtain halo feature maps of multiple scales;

[0113] A fusion module 506 is used to perform multi-scale fusion based on the halo feature maps of multiple scales to obtain a halo guide map;

[0114] The removal module 508 is used to remove the halo in the original halo image based on the halo guide map to obtain a target image.

[0115] In one embodiment, the extraction module 504 is used to perform convolution processing on halo images of multiple scales to obtain convolution features of multiple scales; for each scale, perform feature fusion based on the convolution features of the targeted scale and the halo feature map of the previous scale of the targeted scale to obtain the fusion features of the targeted scale; perform feature extraction on the fusion features of the targeted scale to obtain the halo feature map of the targeted scale.

[0116] In one embodiment, the extraction module 504 is used to downsample the halo feature map of the previous scale of the targeted scale for each scale to obtain a downsampled halo feature map, and the downsampled halo feature map and the convolution feature of the targeted scale have the same scale; channel stitching is performed on the convolution feature of the targeted scale and the downsampled halo feature map to obtain a fusion feature of the targeted scale.

[0117] In one embodiment, the fusion module 506 is used to upsample the halo feature map of the targeted scale for each scale to obtain the upsampled halo feature map; determine the halo feature map of the previous scale of the targeted scale, fuse the upsampled halo feature map with the halo feature map of the previous scale to obtain the guiding feature map of the targeted scale; the upsampled halo feature map has the same scale as the halo feature map of the previous scale; and use the guiding feature map of the last scale among multiple scales as the halo guiding map.

[0118] In one embodiment, the removal module 508 is used to perform channel alignment on the halo guide map based on the number of channels of the halo image of the maximum scale to obtain the aligned halo guide map; scale the aligned halo guide map based on the pixels of the halo image of the maximum scale to obtain the scaled halo guide map; adjust the intensity of the scaled halo guide map to obtain the adjusted guide map; and remove the halo in the original halo image based on the adjusted guide map to obtain the target image.

[0119] In one embodiment, the removal module 508 is used to divide the scaled halo guide map into multiple sub-region maps; for each sub-region map, determine the intensity adjustment operator corresponding to the sub-region map; based on the intensity adjustment operator corresponding to the sub-region map, adjust the intensity of the sub-region map to obtain the adjusted guide map.

[0120] Each module in the above-mentioned image halo removal device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0121] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to image halo removal. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for removing image halo is implemented.

[0122] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0124] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0125] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0129] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for removing image halo, characterized in that: The method comprises: Acquire an original halo image, and perform multiple scaling processes based on the original halo image to obtain halo images of multiple scales; Performing convolution processing on the halo images of the multiple scales to obtain convolution features of the multiple scales; for each scale, performing downsampling processing on the halo feature map of the previous scale of the targeted scale to obtain a downsampled halo feature map, wherein the downsampled halo feature map and the convolution feature of the targeted scale have the same scale; performing channel splicing on the convolution feature of the targeted scale and the downsampled halo feature map to obtain a fusion feature of the targeted scale; performing feature extraction on the fusion feature of the targeted scale to obtain a halo feature map of the targeted scale; For each scale, upsampling the halo feature map of the targeted scale is performed to obtain the upsampled halo feature map; determining the halo feature map of the previous scale of the targeted scale, fusing the upsampled halo feature map with the halo feature map of the previous scale to obtain the guide feature map of the targeted scale; the upsampled halo feature map has the same scale as the halo feature map of the previous scale; and using the guide feature map of the last scale among the multiple scales as the halo guide map; Based on the halo guide map, the halo in the original halo image is removed to obtain a target image.

2. The method according to claim 1, characterized in that The step of removing the halo in the original halo image based on the halo guide map to obtain a target image includes: Based on the number of channels of the halo image of the largest scale, performing channel alignment on the halo guide map to obtain an aligned halo guide map; Based on the pixels of the maximum-scale halo image, scaling the aligned halo guidance map to obtain a scaled halo guidance map; Adjusting the intensity of the scaled halo guide map to obtain an adjusted guide map; Based on the adjusted guiding image, the halo in the original halo image is removed to obtain a target image.

3. The method according to claim 2, characterized in that The step of adjusting the intensity of the scaled halo guide map to obtain an adjusted guide map includes: Dividing the scaled halo guide map into a plurality of sub-region maps; For each sub-region map, determine an intensity adjustment operator corresponding to the sub-region map; Based on the intensity adjustment operator corresponding to the targeted sub-region map, the intensity of the targeted sub-region map is adjusted to obtain an adjusted guiding map.

4. The method according to claim 1, characterized in that: The original halo image is an original image obtained from an electronic rearview mirror or a camera.

5. An image halo removal device, characterized in that: The device comprises: A scaling module, used for acquiring an original halo image, and performing multiple scaling processes based on the original halo image to obtain halo images of multiple scales; An extraction module is used to perform convolution processing on the halo images of the multiple scales to obtain convolution features of the multiple scales; for each scale, downsample the halo feature map of the previous scale of the targeted scale to obtain a downsampled halo feature map, and the downsampled halo feature map and the convolution feature of the targeted scale have the same scale; perform channel splicing on the convolution feature of the targeted scale and the downsampled halo feature map to obtain a fusion feature of the targeted scale; perform feature extraction on the fusion feature of the targeted scale to obtain a halo feature map of the targeted scale; A fusion module is used to upsample the halo feature map of the targeted scale for each scale to obtain the upsampled halo feature map; determine the halo feature map of the previous scale of the targeted scale, fuse the upsampled halo feature map with the halo feature map of the previous scale to obtain the guide feature map of the targeted scale; the upsampled halo feature map has the same scale as the halo feature map of the previous scale; and use the guide feature map of the last scale among the multiple scales as the halo guide map; The removal module is used to remove the halo in the original halo image based on the halo guide map to obtain a target image.

6. The device according to claim 5, characterized in that The removal module is used to perform channel alignment on the halo guide map based on the number of channels of the halo image of the maximum scale to obtain an aligned halo guide map; perform scaling processing on the aligned halo guide map based on the pixels of the halo image of the maximum scale to obtain a scaled halo guide map; and perform intensity adjustment on the scaled halo guide map to obtain an adjusted guide map; Based on the adjusted guiding image, the halo in the original halo image is removed to obtain a target image.

7. The device according to claim 6, characterized in that The removal module is used to divide the scaled halo guide map into multiple sub-region maps; for each sub-region map, determine the intensity adjustment operator corresponding to the sub-region map; based on the intensity adjustment operator corresponding to the sub-region map, adjust the intensity of the sub-region map to obtain the adjusted guide map.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images