A method for removing ghosting and multi-exposure image fusion in dynamic scenes

Abstract

The present invention discloses a method for removing ghosting multi-exposure image fusion in dynamic scenes. Firstly, a suitable exposure image in an exposure sequence is selected as a reference image. Then, a relative exposure weight is calculated based on the gradient direction information of the pixel. Abnormal dynamic pixels and ghosting pixels are eliminated through dynamic pixel correction. Finally, a six-layer image Laplacian pyramid is used for fusion to obtain a ghost-free fused image. The present invention can effectively cope with dynamic fusion requirements in various situations, and avoid the ghosting problem that may be caused by moving objects during the exposure fusion process, thereby achieving efficient multiple exposure fusion in dynamic scenes.

Description

Technical Field

[0001] The present invention relates to technical fields such as image fusion technology, multi-exposure fusion technology, dynamic scene image processing, and more particularly to a method for fusion of multi-exposure images for ghost removal in dynamic scenes. Background Art

[0002] In recent years, drone outdoor line inspection and defect detection need to be based on images taken by drones. Therefore, whether as many details as possible can be restored from drone images is very important for the results of intelligent inspection and detection. Since drones work in outdoor natural environments, the dynamic range of outdoor real natural scenes can span about nine orders of magnitude. Therefore, under a certain exposure setting, a single low dynamic range image taken by an ordinary digital camera cannot show the full dynamic range of a natural scene, and the details of low or high brightness areas will be lost in scenes with strong brightness changes. In order to improve the accuracy of drone intelligent inspection and defect detection and capture as many outdoor scene details as possible, the same scene can be exposed multiple times using different exposure parameters to capture detailed information in various brightness ranges, and then multiple images with different exposure parameters that retain details of different brightness areas are fused. This is multi-exposure fusion technology.

[0003] When a drone takes multiple exposures of the same scene, it may sometimes record dynamic scenes because the drone is hovering in the air. Direct fusion of dynamic scene multi-exposure images will cause ghosting. Therefore, when studying the fusion of dynamic scene multi-exposure images, it is necessary to focus on the ghost removal problem.

[0004] At present, direct multiple exposure image fusion algorithms are mainly divided into two categories: fusion methods based on transform domain (referred to as transform domain method) and fusion methods based on spatial domain (referred to as spatial domain method). Transform domain method means that the image sequence is first transformed to a certain transform domain for fusion, and then restored to the original image sequence. Spatial domain method means that useful information of each part is directly extracted in the image domain for fusion.

[0005] In dynamic scenes, the objects described by each image in the input sequence are often different due to the presence of moving objects in the scene. If the absolute exposure weight is used for fusion directly, ghosting will occur in the fused image. The reason for ghosting is that the moving objects have unpredictable colors and brightness. If the ordinary weighted fusion method is used, the pixels of these moving objects will also occupy a certain weight, thus affecting the final fusion result. The ghosting problem has become a serious limitation of HDR display technology.

[0006] At present, the algorithms for ghost removal can be roughly divided into two categories. One is the method without reference image, which aims to completely eliminate all moving objects. Since this type of method requires many input images to eliminate ghosts, the most popular one is the method based on reference image. This method selects an input image as a reference image and retains the moving objects contained in the reference image. However, the existing ghost removal fusion technology uses complex algorithms, and the final fusion effect is not ideal and still has defects. Summary of the invention

[0007] In view of the problems in the above background technology, a method for removing ghosting multi-exposure image fusion in dynamic scenes is provided. By selecting a reference image, calculating the relative exposure weight for the gradient direction information of the exposure image and the reference image, and then performing dynamic pixel correction on the weight distribution map, removing abnormal dynamic pixels and ghosting pixels, and smoothing the image by the Laplace image pyramid, it can effectively cope with the dynamic fusion requirements in various situations, and avoid the ghosting trouble caused by moving objects during the exposure fusion process, thereby achieving efficient multiple exposure fusion in dynamic scenes.

[0008] A method for removing ghosting and multi-exposure images from a dynamic scene according to the present invention comprises:

[0009] S1 obtains multiple exposure images in dynamic scenes;

[0010] S2 inputs the exposure images into an exposure sequence in order of increasing exposure levels;

[0011] S3 selects an exposure image in the input exposure sequence as a reference image;

[0012] S4 calculates the relative exposure weight according to the gradient direction information of the pixels at the same position in the reference image and other images in the input exposure sequence to obtain a weight distribution map, and performs threshold processing to eliminate static areas with inappropriate exposure;

[0013] S5 performs dynamic pixel correction on the weight distribution map after the threshold processing, selects the dynamic details in the reference image and retains the static details in the entire input exposure sequence;

[0014] S6: removing abnormal dynamic pixels in the dynamic details that do not satisfy the proportional relationship between pixel brightness and exposure, and ghost pixels whose pixel brightness changes suddenly in two consecutive exposure images, to obtain a filtered weight distribution map;

[0015] S7 uses a multi-resolution fusion tool to smooth the filtered weight distribution map.

[0016] Aiming at the problem of ghost elimination when fusing multiple exposure image sequences of dynamic scenes, the present invention proposes an efficient multiple exposure fusion algorithm. Firstly, a suitable exposure image in the exposure sequence is selected as the reference image. Then, the relative exposure weight is calculated based on the gradient direction information of the pixel. The abnormal dynamic pixels and ghost pixels are eliminated through dynamic pixel correction. Finally, a six-layer image Laplacian pyramid is used for fusion to obtain a ghost-free fused image.

[0017] The exposure refers to the exposure degree of the image. The exposure degree may include overexposure, normal exposure and underexposure. The exposure is also called the exposure value, which represents all the camera aperture and shutter combinations that can give the same exposure.

[0018] The exposure image refers to an automatically exposed image of a camera device under normal lighting and scene conditions in the prior art. In order to select an image from multiple images as a reference to facilitate synthesis of the multiple images, the normally exposed image can be used as a reference image, and other images such as overexposed images and underexposed images can be used as non-reference images.

[0019] The input exposure sequence refers to a series of images obtained by arranging a plurality of exposure images in order of increasing exposure levels.

[0020] The pixel is the smallest unit in an image represented by a digital sequence. The image has continuous shades of light and dark. If you magnify the image several times, you will find that these continuous tones are actually composed of many small squares of similar colors. These small squares are the smallest units that make up the image: pixels. This smallest graphic unit is usually displayed as a single dyed dot on the screen. The higher the pixel, the richer the color palette it has, and the more it can express the realism of the color.

[0021] The relative exposure weight refers to the weight of the current image relative to the reference image.

[0022] The threshold processing is a method for image segmentation in image processing. Common threshold segmentation methods include simple threshold, adaptive threshold, etc. Image threshold segmentation is a widely used segmentation technology. It uses the difference in grayscale characteristics between the target area to be extracted and its background in the image, regards the image as a combination of two types of areas (target area and background area) with different grayscale levels, and selects a more reasonable threshold to determine whether each pixel in the image belongs to the target area or the background area, thereby generating a corresponding binary image.

[0023] The static area refers to an area where pixel information in a plurality of consecutive exposure images in the entire input exposure sequence is not much different.

[0024] The weight distribution map can reflect the texture complexity weight of each pixel in the image. The edge of the image is generally achieved by performing gradient calculation on the image.

[0025] The multi-resolution fusion is a common image processing method. The Laplacian pyramid structure proposed in the present invention is widely used in multi-scale image fusion.

[0026] The smoothing process means that after the image is processed by threshold value, the originally continuous weight distribution will change into a discontinuous weight block, so it needs to be smoothed by a multi-resolution fusion tool.

[0027] Specifically, the selection of the reference image includes the following steps:

[0028] Manually select the image with the most appropriate exposure in the input exposure sequence and the image with exposure close to the image but higher brightness as candidate images;

[0029] Calculating the relative exposure weights of the two candidate images to obtain a weight distribution map;

[0030] Calculating the appropriate exposure rate and average grayscale difference of the two candidate images;

[0031]

[0032]

[0033]

[0034] Among them I r is the grayscale image of the direct fusion result, For I r The average value of R r is the binary image of the direct fusion result, n is the number of pixels of the input image, is the average gray value of the entire exposure sequence, R wep For moderate exposure, I ad is the average grayscale difference;

[0035] If the average value of the appropriate exposure rates of the two candidate images is less than 0.9, the candidate image with the smaller average grayscale difference among the two candidate images is selected as the reference image; otherwise, the candidate image with the larger appropriate exposure rate among the two candidate images is selected as the reference image.

[0036] Furthermore, the step of calculating the relative exposure weight according to the gradient direction information of pixels at the same position in the reference image and other images in the input exposure sequence to obtain a weight distribution map, and performing threshold processing to eliminate static areas with inappropriate exposure includes:

[0037] Use a 5×5 Gaussian kernel gradient template to obtain the gradient information of the image:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] Among them I i (x, y) is the normalized grayscale value of the pixel at (x, y) in the i-th exposure image in the input exposure sequence, θ i (x, y) is the gradient direction of the pixel at (x, y) in the i-th exposure image in the input exposure sequence, Gx i (x,y) and Gy i (x, y) are the horizontal and vertical gradients of the i-th exposure image in the input exposure sequence, respectively. x and g y Then represent the Gaussian kernel gradient template in the horizontal and vertical directions respectively;

[0044] Calculate the gradient direction θ of the reference image r , then the gradient angle between the pixel at (x, y) in the i-th image and the pixel at the same position in the reference image is calculated by the following formula:

[0045]

[0046] Among them, (2l+1)×(2l+1) is the mean filter template of the same size, and is the enhancement d i (x, y) is robust, improving the ability to resist interference factors such as noise, and l = 4;

[0047] According to the gradient direction angle, the relative exposure weight can be obtained by the following formula

[0048]

[0049] Where σ is the standard deviation, and σ=0.2 is taken.

[0050] Furthermore, the step of dynamic pixel correction includes:

[0051] Based on the RGB color channel, the average brightness value of the image is used instead of the exposure value to participate in the calculation:

[0052]

[0053] in is the average gray value, I(x,y) is the gray value of any pixel in the exposure image;

[0054] As long as the pixel does not satisfy the proportional relationship between brightness and exposure in any RGB channel, the pixel is judged as an abnormal dynamic pixel and is removed;

[0055] If the brightness of a certain pixel in two consecutive exposure images in the input exposure sequence changes suddenly, the pixel is determined to be a ghost pixel and is eliminated.

[0056] Furthermore, the ghost pixels are determined by power law curve control:

[0057]

[0058] Among them, CVDT is the minimum grayscale threshold used to remove ghost pixels. is the average color value of a RGB channel of the input exposure image, is the average color value of the reference image in the same RGB channel; c = 1, γ = 0.5;

[0059] If the color difference between a pixel in the reference image and the pixel at the same position in other images exceeds the corresponding CVDT value, the pixel in the reference image is determined to be a reference object pixel and obtains all weights, while all other pixel weights at this position are cleared.

[0060] Furthermore, the step of the multi-resolution fusion tool smoothing the filtered weight distribution map includes:

[0061] Use a 5×5 Gaussian low-pass filter template to perform blur filtering on each exposure image in the input exposure sequence;

[0062] The Gaussian low-pass filter template of size 5×5 is:

[0063]

[0064] Downsample the exposure image by 1 / 2;

[0065] Obtain a Gaussian image pyramid whose size is halved successively;

[0066] Upsampling the images in the Gaussian image pyramid respectively, subtracting the upsampled image of the lth layer of the pyramid from the image of the l-1th layer to obtain a detail image as the l-1th layer of the image Laplacian pyramid;

[0067] The fusion process algorithm is:

[0068]

[0069] Where N is the number of exposure images in the input exposure sequence, L{R} L is the lth layer of the fusion result Laplacian pyramid, is the lth layer in the Gaussian pyramid of the normalized weight distribution map of the kth exposure image in the input exposure sequence, is the lth level of the Laplacian pyramid of the kth image in the input exposure sequence;

[0070] Calculate the exposure images in the input exposure sequence respectively and The same layers of the two pyramids of the same exposure image are multiplied correspondingly, and then the products of the N layers of the entire N-exposure sequence are accumulated to obtain the l-th layer of the Laplacian pyramid of the fusion result. The fusion result is obtained by restoring L{R} according to the inverse process of Laplacian pyramid construction.

[0071] Furthermore, during the fusion process, the detail image represented by the Laplacian pyramid is added to the original image with a coefficient of 0.25; in order to achieve a balance between information preservation and artifact elimination, a 6-layer Laplacian pyramid is used.

[0072] Furthermore, when determining ghost pixels through power-law curve control, at most CVDT values ​​between the reference image and four exposure images with exposures closest to the reference image are calculated.

[0073] Furthermore, the present invention also provides a readable storage medium having a control program stored thereon, characterized in that: when the control program is executed by a processor, the method for removing ghosting and multi-exposure image fusion in a dynamic scene as described in any one of the above items is implemented.

[0074] Furthermore, the present invention also provides a computer control system, including a storage device, a processor, and a control program stored in the storage device and executable by the processor, characterized in that when the processor executes the control program, the method for removing ghosting multi-exposure images in a dynamic scene as described in any one of the above items is implemented.

[0075] In order to more clearly understand the present invention, the specific implementation of the present invention will be described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a flow chart of a method for fusion of multiple exposure images for removing ghosting in a dynamic scene according to an embodiment of the present invention;

[0077] Figure 2a , Figure 2b , Figure 2c , Figure 2d A comparison diagram of the results of the dynamic scene image fusion method of an embodiment of the present invention in an outdoor scene with the prior art;

[0078] Figure 3a , Figure 3b , Figure 3c This is a comparison chart of the results of the dynamic scene image fusion method according to an embodiment of the present invention in an indoor scene with the prior art. DETAILED DESCRIPTION

[0079] See also Figure 1 , which is a flow chart of a method for removing ghosting and multi-exposure image fusion in a dynamic scene according to an embodiment of the present invention.

[0080] There are two main problems with the existing ghost removal during multi-exposure fusion of dynamic scenes: first, when the camera shakes during the shooting process, multiple input images will be misaligned, causing the generated HDR image to appear blurred due to ghosting. This problem can be solved by pre-processing with various image registration or alignment methods. Second, when there are obviously moving objects in the captured scene, the objects will be displaced between the exposure images of the dynamic scene. Whether it is synthesizing the HDR image and then further tone mapping it for display, or directly fusing multiple exposures, obvious ghost traces will be produced in the generated image. In this regard, the present application studies the method of multi-exposure image fusion for ghost removal in dynamic scenes.

[0081] The steps of a method for fusion of multiple exposure images for ghost removal in a dynamic scene of this embodiment are as follows:

[0082] S1 obtains multiple exposure images in dynamic scenes;

[0083] S2 inputs the exposure images into an exposure sequence in order of increasing exposure levels;

[0084] S3 selects an exposure image in the input exposure sequence as a reference image;

[0085] S4 calculates the relative exposure weight according to the gradient direction information of the pixels at the same position in the reference image and other images in the input exposure sequence to obtain a weight distribution map, and performs threshold processing to eliminate static areas with inappropriate exposure;

[0086] S5 performs dynamic pixel correction on the weight distribution map after the threshold processing, selects the dynamic details in the reference image and retains the static details in the entire input exposure sequence;

[0087] S6: removing abnormal dynamic pixels in the dynamic details that do not satisfy the proportional relationship between pixel brightness and exposure, and ghost pixels whose pixel brightness changes suddenly in two consecutive exposure images, to obtain a filtered weight distribution map;

[0088] S7 uses a multi-resolution fusion tool to smooth the filtered weight distribution map.

[0089] The present invention mainly solves two major technical problems: how to effectively respond to dynamic fusion requirements according to different scenes to provide a good visual experience; and how to prevent the exposure fusion of dynamic scenes from being bothered by ghosting that may be caused by moving objects in various situations.

[0090] The present invention proposes an efficient multiple exposure fusion algorithm. Firstly, the relative exposure weight is calculated based on the gradient direction information of the pixel. Then, two indicators are defined: moderate exposure rate and average grayscale difference. The algorithm automatically selects a suitable reference image and eliminates the unwanted objects in the exposure image based on the reference image. Finally, a six-layer image Laplacian pyramid is used for fusion to obtain a ghost-free fused image.

[0091] Image edge processing is generally achieved by performing gradient operations on the image. Since different objects have different texture detail characteristics, they have different gradient information. If two pixels describe different object contents, their gradient directions will also be significantly different. Based on this relationship, the degree of similarity and difference of pixel contents can be judged by the angle between the gradient directions.

[0092] In the fusion of multiple exposure images, the user can manually select the most suitable reference image. The embodiment of the present invention provides an automatic reference image selection algorithm. First, the image with the most appropriate exposure in the input exposure sequence and the image with an exposure close to the image but higher brightness are selected as candidate images; since the exposure images are input into the exposure sequence in the order of increasing exposure in the embodiment of the present invention, the middle and the last exposure images in the sequence may be preferred; the weight distribution diagrams of the two reference images are obtained respectively. Then, the two indicators proposed by the present invention: appropriate exposure rate and average grayscale difference are used to select a more suitable reference image from the two candidate reference images.

[0093] Calculating the relative exposure weights of the two candidate images to obtain a weight distribution map; calculating the appropriate exposure rate and average grayscale difference of the two candidate images;

[0094]

[0095]

[0096]

[0097] Among them I r is the grayscale image of the direct fusion result, For I r The average value of R r is the binary image of the direct fusion result, n is the number of pixels of the input image, is the average gray value of the entire exposure sequence, R wepFor moderate exposure, I ad is the average grayscale difference;

[0098] Direct fusion refers to fusion directly according to weight distribution without going through the Laplace pyramid;

[0099] The binary image, that is, each pixel on the image has only two possible values ​​or grayscale states, is often represented by black and white, B&W, or monochrome images.

[0100] The grayscale image refers to an image with only one sampled color per pixel. Such images are usually displayed as grayscale from the darkest black to the brightest white. In theory, this sample can display different shades of any color, or even different colors at different brightnesses. Grayscale images are different from black and white images. In the field of computer graphics, black and white images only have two colors, black and white; however, grayscale images have many levels of color depth between black and white. Grayscale images are often obtained by measuring the brightness of each pixel in a single electromagnetic spectrum such as visible light. Grayscale images for display are usually saved with a nonlinear scale of 8 bits per sampled pixel, so that there can be 256 levels of grayscale (if 16 bits are used, there are 65536 levels).

[0101] The moderate exposure rate R wep By counting the ratio of the number of pixels in the appropriate exposure area in the direct fusion result to the total number of pixels in the image, the detail retention of the image can be reflected to a certain extent. The appropriate exposure rate is a ratio, a value between 0 and 1. If the appropriate exposure rate is very high, close to 1, it means that there are basically no non-appropriately exposed objects in the result image, and the information is concentrated in the appropriate exposure area; on the contrary, if the fusion result has a relatively low appropriate exposure rate, such as less than 0.5 or close to 0, it means that there are non-appropriately exposed objects in the image that cannot be ignored.

[0102] The average grayscale difference I ad Represents the difference in average grayscale between the direct fusion result and the exposure sequence. The smaller this value is, the closer the two are, which means that various brightness information (underexposure, moderate exposure, and overexposure) of the entire exposure sequence are retained to a certain extent.

[0103] If the average value of the appropriate exposure rates of the two candidate reference images is less than 0.9, it means that some reference objects are outside the appropriate exposure area. At this time, the weight distribution map of the candidate reference image with a smaller average grayscale difference is selected. This value indicates that the exposure of the resulting image is closer to the average exposure of the entire sequence, and can better display the information of the reference objects outside the appropriate exposure area. Otherwise, the weight distribution map of the candidate reference image with a larger appropriate exposure rate is selected to retain more details and texture information of the appropriate exposure of the reference objects. At this time, there is no need to consider the relatively unimportant average grayscale difference in this case.

[0104] After selecting the reference image, the gradient direction of the reference image is calculated according to the following formula: In this embodiment, a gradient template with a 5×5 Gaussian kernel is used to obtain the gradient information of the image:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] Among them I i (x, y) is the normalized grayscale value of the pixel at (x, y) in the i-th exposure image in the input exposure sequence. The normalized grayscale value is the data that can be directly obtained from the image, that is, the pixel at this location is normalized to the range of [0, 1]; θ i (x, y) is the gradient direction of the pixel at (x, y) in the i-th exposure image in the input exposure sequence, Gx i (x,y) and Gy i (x, y) are the horizontal and vertical gradients of the i-th exposure image in the input exposure sequence, respectively. x and g y They represent the Gaussian kernel gradient templates in the horizontal and vertical directions respectively; in the calculation process, in order to avoid calculating θ i (x, y) when the denominator is 0, this embodiment in Gy i Add a small constant 10 after (x,y) -25 .

[0111] Calculate the gradient direction θ of the reference image r , then the gradient angle between the pixel at (x, y) in the i-th image and the pixel at the same position in the reference image is calculated by the following formula:

[0112]

[0113] Where (2l+1)×(2l+1) is a mean filter template of the same size. In this embodiment, to enhance d i (x, y) is robust, improving the ability to resist interference factors such as noise, and l = 4;

[0114] According to the gradient direction angle, the relative exposure weight can be obtained by the following formula

[0115]

[0116] Here, σ is the standard deviation, and in this embodiment, σ=0.2.

[0117] The relative exposure weight is calculated based on the gradient direction information of the pixels at the same position in the reference image and other images in the input exposure sequence to obtain a weight distribution map, and threshold processing is performed to eliminate static areas with inappropriate exposure. Although some static areas with inappropriate exposure can be correctly eliminated, some moving objects will also be confused. Therefore, the weight distribution map after threshold processing must be corrected. In the embodiment of the present invention, the correction algorithm for dynamic pixels corrects the areas affected by threshold errors in two steps. The dynamic exposure sequence fusion result based on the correction algorithm can successfully distinguish static areas from dynamic areas, thereby selecting only dynamic details in the reference image while retaining static detail information in the entire exposure sequence. This correction algorithm is based on RGB channels, rather than grayscale or brightness images.

[0118] There is a relationship in the exposure sequence: the brightness of a pixel at a certain position in the sequence is proportional to the exposure of the input image, that is, the higher the exposure of the image, the greater the brightness value of the pixel at a certain position, otherwise it can be regarded that the content of the pixel has changed. In order to facilitate analysis and calculation, the embodiment of the present invention uses the image average brightness value, which is simpler to calculate, instead of the exposure to participate in the calculation:

[0119]

[0120] in is the average gray value, and I(x,y) is the gray value of any pixel in the exposure image; both are based on RGB channels.

[0121] As long as the pixel does not satisfy the proportional relationship between brightness and exposure in any RGB channel, the pixel is judged as an abnormal dynamic pixel and is eliminated; the weight distribution map after screening based on the proportional relationship is smoothed by a 19×19 mean filter template to enhance the anti-noise performance.

[0122] In a general multiple exposure sequence, the change in pixel brightness describing the same object in the moderately exposed area will not be very drastic relative to the change in exposure, and will only fluctuate within a small range centered on the image exposure. If the color brightness of a pixel in the next exposure image changes significantly, the pixel can be determined as a ghost pixel and removed. The algorithm of the present invention uses a power law curve to control the minimum grayscale threshold for determining a ghost pixel:

[0123]

[0124] Among them, CVDT is the minimum grayscale threshold used to remove ghost pixels. is the average color value of a RGB channel of the input exposure image, is the average color value of the reference image in the same RGB channel; in this embodiment, c=1, γ=0.5;

[0125] If the color difference between a pixel in the reference image and the pixel at the same position in other images exceeds the corresponding CVDT value, the pixel in the reference image is determined to be a reference object pixel and obtains all weights, while all other pixel weights at this position are cleared.

[0126] Referring to the shape of the power law curve, if the exposure difference between the two images is too large, the calculated CVDT minimum grayscale threshold will be very close to the exposure difference, making it difficult to distinguish between ideal pixels and ghost pixels. Therefore, the embodiment of the present invention at most calculates the CVDT value between the reference image and the other four input images with the closest exposure to it, and other input images are not considered.

[0127] Threshold processing will cause the originally continuous weight distribution to change into discontinuous weight blocks, so it is necessary to use a multi-resolution fusion tool to smooth it. The embodiment of the present invention uses a standard image Laplacian pyramid to participate in the fusion process. The Laplacian pyramid can be used to seamlessly fuse images, and the smoothing process steps include:

[0128] Use a 5×5 Gaussian low-pass filter template to perform blur filtering on each exposure image in the input exposure sequence;

[0129] The Gaussian low-pass filter template of size 5×5 is:

[0130]

[0131] Then downsample the exposure image by 1 / 2;

[0132] Repeat the blur filtering and downsampling process until a Gaussian image pyramid with a size that is halved is obtained;

[0133] The images in the Gaussian image pyramid are upsampled respectively, and the upsampled image of the lth layer of the pyramid is subtracted from the image of the l-1th layer to obtain a detail image as the l-1th layer of the image Laplacian pyramid; for the same image, the top images of the Gaussian image pyramid and the Laplacian pyramid are the same. According to the normalized weight distribution map controlled by the relative exposure weight, a Gaussian pyramid of each weighted image can be constructed; the Laplacian pyramid is calculated from each input image.

[0134] The fusion process algorithm is:

[0135]

[0136] Where N is the number of exposure images in the input exposure sequence, L{R} L is the lth layer of the fusion result Laplacian pyramid, is the lth layer in the Gaussian pyramid of the normalized weight distribution map of the kth exposure image in the input exposure sequence, is the lth level of the Laplacian pyramid of the kth image in the input exposure sequence;

[0137] Calculate the exposure images in the input exposure sequence respectively and The same layers of the two pyramids of the same exposure image are multiplied correspondingly, and then the products of the N layers of the entire N-exposure sequence are accumulated to obtain the l-th layer of the Laplacian pyramid of the fusion result. The final fusion result can be obtained by restoring L{R} according to the inverse process of Laplacian pyramid construction.

[0138] To improve the detail enhancement effect, during the fusion process, the detail image represented by the Laplacian pyramid is added to the original image at a coefficient of 0.25; if the coefficient is too low, the enhancement effect will be weak; if the coefficient is too high, the image will become more distorted and lose its natural feel.

[0139] By changing the number of pyramid layers, it can be found that too many pyramid layers may lose some information, while too few pyramid layers may cause obvious halo artifacts due to incomplete smoothing. Through comparative experiments, the embodiment of the present invention adopts a 6-layer Laplacian pyramid to achieve a relative balance between information retention and artifact elimination.

[0140] Please refer to FIG. 2 , which is a comparison diagram of the results of the dynamic scene image fusion method according to an embodiment of the present invention in an outdoor scene with those of the prior art;

[0141] Figure 2a It is a multi-scale image enhancement technology in the prior art, which improves the information retention of bright and dark areas by using a smoothed weight distribution Gaussian pyramid. Figure 2bIt is a fusion algorithm in the prior art. It proposes two indicators, visibility and consistency, based on gradient amplitude and direction, respectively, to determine the weight distribution and eliminate moving objects that are not desired to appear in the results. Figure 2c This is a new object detection algorithm in the prior art, which eliminates ghost images by correcting the weight distribution map. Figure 2d This is the fusion result of the embodiment of the present invention.

[0142] in Figure 2b There is an obvious ghosting phenomenon. This is because the dynamic scene is located in a public place with a large flow of people and is taken at different times. As a result, moving objects, i.e. passers-by, appear in different positions and shapes in different input images. Some of these passers-by block the background information that should be properly exposed, while others are located in the non-properly exposed area. Figure 2b In the method, the cumulative difference of the gradient direction of some areas in different input images is almost the same, and it is impossible to accurately separate them through the Gaussian curve, which eventually leads to the failure of the method, the weight distribution is relatively even, and the ghost phenomenon is introduced. The other three results, due to the advantage of occupying the reference image, pre-select the reference object detail information displayed in the result image, thereby eliminating most of the possibility of ghost interference. It can be seen that when the ideal pixel only exists occasionally in the entire dynamic exposure sequence, the result of the present invention is fully capable of coping with the interference of randomly appearing moving objects, while minimizing the impact on detail retention.

[0143] Please refer to FIG. 3 , which is a comparison diagram of the results of the dynamic scene image fusion method according to an embodiment of the present invention in an indoor scene with the prior art.

[0144] Figure 3a It is a fusion algorithm in the prior art. It proposes two indicators, visibility and consistency, based on gradient amplitude and direction, respectively, to determine the weight distribution and eliminate moving objects that are not desired to appear in the results. Figure 3b This is a new object detection algorithm in the prior art, which eliminates ghost images by correcting the weight distribution map. Figure 3c This is the fusion result of the embodiment of the present invention.

[0145] Visible in Figure 3a The results show severe ghosting caused by moving objects, such as the baby's face, the toy dog's ears, and the shadow of the doll's hand on the sofa, which have moved slightly in the sequence and are already blurred, as well as the rugby ball, which is mostly in the underexposed area. The static parts still retain good details, clearly showing the texture details of the teddy bear's hat and legs, as well as the side information of the sofa that only exists in the brightest input image. Figure 3bAlthough some ghost images were removed, such as the shadow of the doll's hand and the bright front of the football, there were still obvious ghost images on the toy dog's ears and the dark part of the football. In addition, the details of the teddy bear's hat and legs, as well as the side of the sofa, were not well preserved. Figure 3a The results of the present invention are better than those of the static details. Figure 3a It is equivalent to eliminating most of the ghost images at the same time, thus meeting the prerequisite and requirement for fusing the dynamic sequence of overlapping moving objects.

[0146] An embodiment of the present invention designs a method for removing ghosting multi-exposure image fusion in dynamic scenes, which solves two key technical problems: how to effectively respond to dynamic fusion requirements according to different scenes to provide a good visual experience; and how to make the exposure fusion of dynamic scenes free from the ghosting troubles that may be caused by moving objects in various situations, regardless of whether the original information of the moving objects is in the moderate exposure area.

[0147] The core of the present invention lies in the algorithm of relative exposure weight, how to determine abnormal dynamic pixels and ghost pixels and remove them, and how to automatically select reference images through appropriate exposure rate and average grayscale difference. Compared with the prior art, the present invention provides an efficient multiple exposure fusion algorithm, firstly, a suitable exposure image in the exposure sequence is selected as the reference image, then the relative exposure weight is calculated based on the gradient direction information of the pixel, abnormal dynamic pixels and ghost pixels are removed through dynamic pixel correction, and finally a six-layer image Laplacian pyramid is used for fusion to obtain a ghost-free fusion image. A relative balance is achieved between information retention and artifact elimination, and the threshold processing of the dynamic area and static area of ​​the exposure image is more accurate and reliable than the prior art. The concept of appropriate exposure rate and average grayscale difference is introduced to select the reference image, which is more scientific and efficient than the method of manually selecting the reference image in the prior art; in terms of multi-resolution fusion tool selection, the mature Laplacian pyramid fusion technology is adopted, and the six-layer Laplacian pyramid is preferably selected through control experiments, which improves the detail enhancement effect and avoids image distortion.

[0148] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications to the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims and equivalent technologies of the present invention, the present invention also includes these changes and modifications.

Claims

1. A method for removing ghosting and multi-exposure image fusion in dynamic scenes, comprising: Acquire multiple exposure images in dynamic scenes; Inputting the exposure images into an exposure sequence in order of increasing exposure levels; Selecting an exposure image in the input exposure sequence as a reference image; Calculating relative exposure weights according to gradient direction information of pixels at the same position in the reference image and other images in the input exposure sequence to obtain a weight distribution map, and performing threshold processing to eliminate static areas with inappropriate exposure; Performing dynamic pixel correction on the weight distribution map after threshold processing, selecting dynamic details in the reference image and retaining static details in the entire input exposure sequence; Abnormal dynamic pixels that do not satisfy the proportional relationship between pixel brightness and exposure in the dynamic details, as well as ghost pixels whose pixel brightness changes suddenly in two consecutive exposure images, are eliminated to obtain a filtered weight distribution map; The filtered weight distribution map is smoothed using a multi-resolution fusion tool.

2. The method for removing ghosting and multi-exposure images in dynamic scenes according to claim 1, characterized in that: The selection of the reference image comprises the following steps: Manually select the image with the most appropriate exposure in the input exposure sequence and an image with exposure close to the image but higher brightness as two candidate images; Calculating the relative exposure weights of the two candidate images to obtain a weight distribution map; Calculating the appropriate exposure rate and average grayscale difference of the two candidate images; where i r is the grayscale image of the direct fusion result, For I r The average value, R r is the binary image of the direct fusion result, n is the number of pixels of the input image, is the average gray value of the entire exposure sequence, R wep For moderate exposure, I ad is the average grayscale difference; If the average value of the appropriate exposure rates of the two candidate images is less than 0.9, the candidate image with the smaller average grayscale difference among the two candidate images is selected as the reference image; otherwise, the candidate image with the larger appropriate exposure rate among the two candidate images is selected as the reference image.

3. The method for removing ghosting and multi-exposure images in dynamic scenes according to claim 1, characterized in that: The step of calculating the relative exposure weight according to the gradient direction information of pixels at the same position in the reference image and other images in the input exposure sequence to obtain a weight distribution map, and performing threshold processing to eliminate static areas with inappropriate exposure includes: Use a 5×5 Gaussian kernel gradient template to obtain the gradient information of the image: Among them I i (x, y) is the normalized grayscale value of the pixel at (x, y) in the i-th exposure image in the input exposure sequence, θ i (x, y) is the gradient direction of the pixel at (x, y) in the i-th exposure image in the input exposure sequence, Gx i (x,y) and Gy i (x, y) are the horizontal and vertical gradients of the i-th exposure image in the input exposure sequence, respectively. x and g y Then represent the Gaussian kernel gradient template in the horizontal and vertical directions respectively; Calculate the gradient direction θ of the reference image r , then the gradient angle between the pixel at (x, y) in the i-th image and the pixel at the same position in the reference image is calculated by the following formula: Among them, (2l+1)×(2l+1) is the mean filter template of the same size, and is the enhancement d i (x, y) is robust, improving the ability to resist interference factors, taking l = 4, and the interference factors include noise; According to the gradient direction angle, the relative exposure weight can be obtained by the following formula Where σ is the standard deviation, and σ=0.2 is taken.

4. The method for removing ghosting and multi-exposure images in dynamic scenes according to claim 1, characterized in that: The step of dynamic pixel correction comprises: Based on the RGB color channel, the average brightness value of the image is used instead of the exposure to participate in the calculation. As long as the pixel does not satisfy the proportional relationship between brightness and exposure in any RGB channel, the pixel is determined to be an abnormal dynamic pixel and is removed; If the brightness of a certain pixel in two consecutive exposure images in the input exposure sequence changes suddenly, the pixel is determined to be a ghost pixel and is eliminated.

5. The method for removing ghosting and multi-exposure images in dynamic scenes according to claim 1, characterized in that: The ghost pixels are determined by the power law curve control: Among them, CVDT is the minimum grayscale threshold used to remove ghost pixels. is the average color value of a RGB channel of the input exposure image, is the average color value of the reference image in the same RGB channel; c = 1, γ = 0.5; If the color difference between a pixel in the reference image and the pixel at the same position in other images exceeds the corresponding CVDT value, the pixel in the reference image is determined to be a reference object pixel and obtains all weights, while all other pixel weights at this position are cleared.

6. The method for removing ghosting and multi-exposure images in dynamic scenes according to claim 1, characterized in that: The step of smoothing the filtered weight distribution map by the multi-resolution fusion tool comprises: Use a 5×5 Gaussian low-pass filter template to perform blur filtering on each exposure image in the input exposure sequence; The Gaussian low-pass filter template of size 5×5 is: Downsample the exposure image by 1 / 2; Obtain a Gaussian image pyramid whose size is halved successively; Upsampling the images in the Gaussian image pyramid respectively, subtracting the upsampled image of the lth layer of the pyramid from the image of the l-1th layer to obtain a detail image as the l-1th layer of the image Laplacian pyramid; The fusion process algorithm is: Where N is the number of exposure images in the input exposure sequence, L{R} l is the lth layer of the fusion result Laplacian pyramid, is the lth layer in the Gaussian pyramid of the normalized weight distribution map of the kth exposure image in the input exposure sequence, is the lth level of the Laplacian pyramid of the kth image in the input exposure sequence; Calculate the exposure images in the input exposure sequence respectively and The same layers of the two pyramids of the same exposure image are multiplied correspondingly, and then the products of the N layers of the entire N-exposure sequence are accumulated to obtain the l-th layer of the Laplacian pyramid of the fusion result. The fusion result is obtained by restoring L{R} according to the inverse process of Laplacian pyramid construction.

7. The method for removing ghosting and multi-exposure images in dynamic scenes according to claim 6, characterized in that: During the fusion process, the detail image represented by the Laplacian pyramid is added to the original image with a coefficient of 0.25; in order to achieve a balance between information retention and artifact elimination, a 6-layer Laplacian pyramid is used.

8. The method for removing ghosting and multi-exposure images in dynamic scenes according to claim 5, characterized in that: When determining ghost pixels through power-law curve control, at most CVDT values ​​between a reference image and four exposure images with exposures closest to the reference image are calculated.

9. A readable storage medium having a control program stored thereon, characterized in that: When the control program is executed by the processor, the method for removing ghosting and multi-exposure images in a dynamic scene as described in any one of claims 1 to 8 is implemented.

10. A computer control system, comprising a memory, a processor, and a control program stored in the memory and executable by the processor, characterized in that: When the processor executes the control program, the method for removing ghosting and multi-exposure images from a dynamic scene as described in any one of claims 1 to 8 is implemented.

Images