Ghost detection method, image fusion method and corresponding device

Abstract

The present application relates to the field of image processing technology, and provides a ghost detection method, an image fusion method and a corresponding device. The ghost detection method includes: obtaining multiple frames of first images collected at a first exposure compensation value, and at least one frame of second image collected at other exposure compensation values ​​different from the first exposure compensation value; calculating the first ghost area and the movement speed and movement direction of the moving object in the first ghost area according to the multiple frames of the first image, and the first ghost area is the area containing the moving object in the multiple frames of the first image; adjusting the area of ​​the first ghost area according to the movement speed and movement direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and one of the first frames, and obtaining the second ghost area corresponding to the frame of the second image. The above method effectively avoids the interference of the highlight area in the image on the ghost detection, and improves the accuracy of ghost detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a ghost detection method, an image fusion method and a corresponding device. Background Art

[0002] At present, users have higher and higher requirements for the quality of mobile phone photography, especially night scene shooting, which has attracted the attention and love of many users as a highlight function. At present, in order to achieve image noise reduction, multiple frames of images with different exposure compensation values ​​are usually collected and fused when shooting night scenes. However, when there are moving objects (such as pedestrians and vehicles) in the picture, ghost images are likely to appear at the position of the moving objects in the fused image, which seriously affects the photo quality and user experience.

[0003] For example, in Figure 1 In the case of a night shot, because the passers-by in the picture are walking, three consecutive human figures (arrows point to the positions) appear in the taken photo (which has been denoised after multi-frame fusion), which is the so-called ghost image.

[0004] In order to effectively eliminate ghosting, the area containing moving objects in the multi-frame images before fusion, i.e., the ghost area, must be effectively detected. However, for images collected under different exposure compensation values, there are significant differences in their highlight areas. When using existing methods for detection, it is easy to misjudge the changing highlight areas as ghost areas, which may cause the quality of the fused image to deteriorate. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a ghost detection method, an image fusion method and a corresponding device to improve the above-mentioned technical problems.

[0006] To achieve the above objectives, this application provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a ghost detection method, including: acquiring multiple frames of first images and at least one frame of second image; wherein the first image is an image captured at a first exposure compensation value, and the second image is an image captured at other exposure compensation values ​​different from the first exposure compensation value; calculating a first ghost area and a movement speed and movement direction of a moving object in the first ghost area based on the multiple frames of first images; wherein the first ghost area is an area containing moving objects in the multiple frames of first images; adjusting the area of ​​the first ghost area based on the movement speed and movement direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and one of the first images, to obtain a second ghost area corresponding to the frame of the second image.

[0008] In the above method, the first ghost area is calculated using multiple frames of the first image. Since the multiple frames of the first image are all collected under the first exposure compensation value, their highlight areas are basically the same, so the detection of the first ghost area will basically not be disturbed by the highlight area, and its detection accuracy is relatively high. The second ghost area is not calculated using the first image and the second image, but is estimated based on the motion trend (referring to the motion speed and direction) of the moving object in the first ghost area. Therefore, although the exposure compensation value corresponding to the second image is different from the first exposure value, and its highlight area is significantly different from the first image, it will not affect the calculation result of the second ghost area. Therefore, the detection of the second ghost area will not be disturbed by the highlight area, and its detection accuracy is relatively high.

[0009] Furthermore, since the detection results of the first ghost area and the second ghost area have high accuracy, if multiple frames of the first image and at least one frame of the second image are fused based on the detected ghost area, a high-quality fusion result can also be obtained, which is conducive to improving the user experience in night scene shooting, etc. However, it should be understood that the detected ghost area is not limited to being used for image fusion.

[0010] In an implementation of the first aspect, calculating the first ghost area based on the multiple frames of first images includes: selecting a reference frame from the multiple frames of first images; and calculating the first ghost area based on the pixel value difference between a non-reference frame and a reference frame in the multiple frames of first images.

[0011] Taking the reference frame as the reference system, if an object moves between the reference frame and the non-reference frame, the position of the object in the reference frame and the non-reference frame must be different. Therefore, by calculating the pixel value difference between the non-reference frame and the reference frame, the area containing the moving object in the non-reference frame and the reference frame, i.e., the first ghost area, can be determined.

[0012] In an implementation of the first aspect, the first ghost area is calculated based on the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image, including: calculating the initial ghost area based on the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image; dividing the initial ghost area into at least one first ghost area that is not connected to each other.

[0013] There may be multiple moving objects in the first image. These moving objects may be located at different positions in the picture, and each may have different movement rules, which makes it difficult to uniformly calculate the movement speed and direction later. Therefore, an initial ghost area can be first calculated based on the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image, and then the initial ghost area can be divided into at least one first ghost area that is not connected to each other (each first ghost area is considered to contain a different moving object). In the subsequent steps, each first ghost area is used to predict the second ghost area, and the result may be more accurate.

[0014] In an implementation of the first aspect, the initial ghost area is calculated based on the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image, including: calculating the pixel value difference between each non-reference frame and the reference frame in the multiple frames of the first image at each pixel position to obtain multiple initial motion masks; superimposing the multiple initial motion masks to obtain a first motion mask; determining whether the pixel value at each pixel position in the first motion mask is greater than a first threshold, and determining a set of pixel positions whose pixel values ​​are greater than the first threshold as the initial ghost area.

[0015] According to the above calculation process, the pixel value at each pixel position in the first motion mask can be understood as: the superposition of the motion intensity of all non-reference frames relative to the reference frame at the pixel position. The larger the pixel value, the greater the motion intensity, and the higher the possibility that the pixel position belongs to the ghost area. Therefore, by setting the first threshold, the initial ghost area can be determined.

[0016] In an implementation of the first aspect, the method of judging whether the pixel value at each pixel position in the first motion mask is greater than a first threshold value, and determining the set of pixel positions whose pixel values ​​are greater than the first threshold value as the initial ghost area, includes: judging whether the pixel value at each pixel position in the first motion mask is greater than a first threshold value, and setting the pixel value at the corresponding pixel position in the second motion mask whose pixel value is greater than the first threshold value to a first value; wherein the pixel value in the second motion mask takes the first value or the second value, the set of pixel positions whose pixel value takes the first value is the initial ghost area, and the set of pixel positions whose pixel value takes the second value is the non-initial ghost area; the method of dividing the initial ghost area into at least one first ghost area that is not connected to each other includes: dividing the initial ghost area into at least one first ghost area that is not connected to each other by performing connectivity detection on pixels in the second motion mask that are located in the initial ghost area.

[0017] The second motion mask is a binary image. The second motion mask is used to record the position of the first ghost area. Firstly, it is convenient. Secondly, it is also convenient to apply some existing algorithms on the binary image, such as denoising algorithm, connectivity detection algorithm, expansion algorithm, etc., which will be mentioned later.

[0018] In an implementation manner of the first aspect, selecting a reference frame from the multiple frames of first images includes: selecting a first image with the highest definition among the multiple frames of first images as the reference frame.

[0019] If it is necessary to fuse multiple frames of first images, many solutions will use a reference frame as a fusion benchmark, and select the frame with the highest clarity among the multiple frames of first images as a reference frame, which is beneficial to improving the quality of the fused image. However, it should be pointed out that the method provided in the first aspect is only a ghost detection method. As for whether to perform multi-frame fusion based on the detected ghost area, it is optional.

[0020] In an implementation of the first aspect, the movement speed and movement direction of the moving object in the first ghost area are calculated based on the multiple frames of first images, including: calculating the movement speed and movement direction of the moving object in the first ghost area based on two frames of first images in the multiple frames of first images.

[0021] In many scenarios, the acquisition time intervals of multiple frames of the first image are relatively close (for example, only tens of milliseconds). During this period of time, the movement of objects in the picture can be approximately considered to be linear. Therefore, even if only two frames are taken, the estimation of the movement trend of the moving objects in the first ghost area is relatively accurate.

[0022] In an implementation of the first aspect, the calculation of the movement speed and movement direction of the moving object in the first ghost area based on two first images in the multiple first images includes: calculating the movement speed and movement direction of the moving object in the first ghost area based on an adjacent first image and a first image that is farthest apart from the adjacent first image in acquisition time; wherein the adjacent first image is the first image in the multiple first images that is closest to the second image in acquisition time interval.

[0023] Calculating the speed and direction of the moving object in the first ghost area using two first image frames that are acquired far apart in time may be more accurate, because the calculation result reflects the accumulated movement trend of the moving object in the first image over a longer period of time.

[0024] In an implementation of the first aspect, the movement speed and movement direction of the moving object in the first ghost area are calculated based on the multiple frames of first images, including: according to each non-adjacent first image and adjacent first image in the multiple frames of first images, a group of movement speeds and movement directions of the moving object in the first ghost area are calculated, and at least one group of movement speeds and movement directions are obtained in total; wherein the adjacent first image is the first image in the multiple frames of first images that is closest to the second image in terms of acquisition time interval; and the mean of the at least one group of movement speeds and movement directions is calculated to obtain the movement speed and movement direction of the moving object in the first ghost area.

[0025] Since the above implementation method uses the information in each frame of the first image to calculate the moving speed and moving direction of the moving object in the first ghost area, the calculation result has a high accuracy.

[0026] In an implementation of the first aspect, the movement speed and movement direction of the moving object in the first ghost area are calculated based on the multiple frames of the first image, including: calculating the movement speed and movement direction at each pixel position in the first ghost area using the optical flow method based on the multiple frames of the first image; calculating the mean of the movement speed and movement direction at each pixel position in the first ghost area to obtain the movement speed and movement direction of the moving object in the first ghost area.

[0027] The movement speed and direction at each pixel position can be expressed as a vector (the magnitude of the vector is the movement speed, and the direction of the vector is the movement direction), namely, the optical flow. The average of the vectors at each pixel position in the first ghost area is obtained, and the obtained mean vector can represent the movement speed and direction of the moving object in the first ghost area.

[0028] In an implementation of the first aspect, the area of ​​the first ghost area is adjusted according to the movement speed and direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and one frame of the first image to obtain the second ghost area corresponding to the frame of the second image, including: adjusting the area of ​​the first ghost area according to the movement speed and direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and an adjacent first image to obtain the second ghost area corresponding to the frame of the second image; wherein the adjacent first image is the first image among the multiple frames of first images that is closest to the second image in terms of acquisition time interval.

[0029] Although an acquisition time interval can be calculated between any frame of the first image and the second image, according to the definition of adjacent first images, the acquisition time interval between adjacent first images and the second image is the shortest. Therefore, adjusting the area of ​​the first ghost area according to this time interval is also relatively accurate, and the area of ​​the first ghost area will not be adjusted too large.

[0030] In an implementation of the first aspect, after obtaining the second ghost area corresponding to any frame of the second image, the method further includes: performing dilation processing on the second ghost area corresponding to the frame of the second image.

[0031] Since estimating the second ghost area purely by using the movement trend of the moving object in the first ghost area is inevitably inaccurate, expanding the area of ​​the second ghost area is beneficial to including the moving object in the second image.

[0032] In an implementation manner of the first aspect, the first exposure compensation value is a positive number, and the other exposure compensation values ​​are all non-positive numbers.

[0033] If the first image and the second image are collected at night and are to be used for fusion denoising after the ghost area is detected, they can be collected according to the above-mentioned setting method of the exposure compensation value, which is conducive to improving the quality of the fused image. The reason is that in the fused image, the pixel values ​​in most areas are mainly derived from images with positive exposure compensation values, because these images have higher brightness and richer details in most areas, and the pixel values ​​in a small number of highlight areas are mainly derived from images with non-positive exposure compensation values, because these images have lower brightness and are not easy to be overexposed in the highlight area.

[0034] In an implementation manner of the first aspect, each frame of the second image is captured under a different exposure compensation value.

[0035] If the first image and the second image are collected at night and are used for fusion denoising after the ghost area is detected, the second image is relatively less important than the first image in the fusion process because the highlight area in the fused image usually does not occupy too large an area and does not have too many details. Therefore, in order to reduce the amount of calculation as much as possible while ensuring the quality of the fused image, the following can be done:

[0036] At the first exposure compensation value (positive number), multiple frames of the first image are collected for fusion and denoising, ensuring that most areas in the fused image have high image quality; at each other exposure compensation value (non-positive number), only one frame of the second image is collected, which is mainly used to generate pixels in the highlight area. If necessary, the denoising intensity can be increased for the fusion of the second image, so that a good fusion effect can be obtained when only a small number of images are used for fusion.

[0037] In second aspect, an embodiment of the present application provides an image fusion method, including: obtaining a first ghost area corresponding to multiple frames of first images and a second ghost area corresponding to each second image in at least one frame of second images using the ghost detection method provided by the first aspect or any possible implementation method of the first aspect; using the first ghost area to fuse the multiple frames of first images to obtain a first fused image; using the second ghost area corresponding to each frame of second image to fuse the at least one frame of second image with the first fused image to obtain a final result image.

[0038] Since the detection results of the first ghost area and the second ghost area have high accuracy, a high-quality fusion result can be obtained by fusing multiple frames of the first image and at least one frame of the second image based on the detected ghost area.

[0039] In an implementation of the second aspect, the use of the first ghost area to fuse the multiple frames of first images to obtain a first fused image includes: using the pixel value at each pixel position in the reference frame of the multiple frames of first images located in the first ghost area as the pixel value of the first fused image at the corresponding pixel position; fusing the pixel value at each pixel position in the multiple frames of first images located in non-first ghost areas as the pixel value of the first fused image at the corresponding pixel position.

[0040] In the above implementation, the pixel values ​​in the reference frame are retained within the first ghost area, and multi-frame fusion is not performed, thereby avoiding the generation of ghosts in the first fused image; multi-frame fusion is performed outside the first ghost area to reduce noise in the picture.

[0041] In an implementation of the second aspect, the first exposure compensation value is a positive number, the other exposure compensation values ​​are all non-positive numbers, and the using the second ghost area corresponding to each frame of the second image to fuse the at least one frame of the second image with the first fused image to obtain a final result image includes: taking the second image with the largest difference between the corresponding exposure compensation value and the first exposure compensation value as the initial second fused image, and in the order of gradually decreasing difference between the corresponding exposure compensation value and the first exposure compensation value, sequentially using the second ghost area corresponding to each frame of the second image to fuse the current second fused image with the next frame of the image to obtain a new second fused image; wherein, if the frame of the second image is not the last frame of the second image, the next frame of the image is the next frame of the second image, and if the frame of the second image is the last frame of the second image, the next frame of the image is the first fused image, and the second fused image obtained after the fusion of the first fused images is the final result image.

[0042] The second image frames are fused in sequence in the order of gradually decreasing differences between the corresponding exposure compensation values ​​and the first exposure compensation values, so that the span of the exposure compensation values ​​in each fusion is not too large, which is beneficial to improving the quality of the fused image.

[0043] In an implementation of the second aspect, a second ghost area corresponding to a frame of a second image is used to fuse a current second fused image with a next frame of image to obtain a new second fused image, including: calculating the pixel value difference between the current second fused image and the next frame of image at each pixel position in the second ghost area corresponding to the frame of the second image, and determining a set of pixel positions whose pixel value difference is greater than a second threshold as a third ghost area corresponding to the frame of the second image; and using the third ghost area corresponding to the frame of the second image to fuse the current second fused image with the next frame of image to obtain a new second fused image.

[0044] Since the second ghost area is obtained by adjusting the area of ​​the first ghost area, in order to include the moving object in the second image as much as possible, the area adjustment can be extended, so the area of ​​the second ghost area will be relatively large, which may include the background area (or the area where the non-moving object is located). If the strategy of not fusing in the second ghost area is adopted during multi-frame fusion, the background area therein cannot be fused and denoised (the area where the moving object is located is not fused and denoised in order to avoid ghosting), which has an adverse effect on the quality of the fused image.

[0045] In the above implementation, the motion intensity at each pixel position is evaluated by calculating the pixel value difference of the two frames of images to be fused in the second ghost area (similar to the calculation of the initial motion mask in the previous article), and then some pixel positions in the second ghost area where no motion actually occurs or the motion is not obvious (the pixel value difference is not greater than the second threshold) are excluded, so that the third ghost area with a smaller area can cover the moving object more accurately, which is beneficial to improve the quality of the fused image.

[0046] In an implementation of the second aspect, the third ghost area corresponding to the second image of the frame is used to fuse the current second fused image with the next frame image to obtain a new second fused image, including: taking the pixel value at each pixel position in the next frame image located in the third ghost area as the pixel value of the new second fused image at the corresponding pixel position; and fusing the current second fused image with the pixel value at each pixel position in the next frame image located in a non-third ghost area, and taking the pixel values ​​at the corresponding pixel positions of the new second fused image.

[0047] In the above implementation, the pixel values ​​in the next frame image are retained within the third ghost area, and multi-frame fusion is not performed, thereby avoiding the generation of ghosts in the new second fused image; multi-frame fusion is performed outside the third ghost area to reduce noise in the picture.

[0048] In an implementation of the second aspect, before calculating the pixel value difference between the current second fused image and the next frame image at each pixel position within the second ghost area corresponding to the second frame image, the method also includes: adjusting the brightness of the current second fused image to match the brightness of the next frame image.

[0049] The second image to be fused and the current result image are usually not considered as images captured under the same exposure compensation value, so the pixel values ​​of the two are quite different in overall brightness. Directly calculating the pixel value difference is likely to result in a large calculated difference, which cannot optimize the second ghost area. In the above implementation, the brightness of the second image to be fused is first adjusted to match (equal or close to) the brightness of the current result image, and then the pixel value difference between it and the current result image at each pixel position in the second ghost area corresponding to the second image of the frame is calculated, which can improve this problem.

[0050] In an implementation of the second aspect, adjusting the brightness of the current second fused image to match the brightness of the next frame image includes: using a histogram matching algorithm to adjust the brightness of the current second fused image to match the brightness of the next frame image.

[0051] In an implementation of the second aspect, the second ghost area corresponding to each frame of the second image is used to fuse the at least one frame of the second image with the first fused image to obtain a final result image, including: taking the first fused image as the initial current result image, and sequentially using the second ghost area corresponding to each frame of the second image to fuse the frame of the second image with the current result image to obtain a new current result image; wherein the current result image obtained after the last frame of the second image is fused is the final result image.

[0052] The above fusion method is simple to operate, but if the span between the exposure compensation value corresponding to the second image to be fused and the first exposure compensation value is too large, it may affect the quality of the fused image. Therefore, this solution is more recommended when the span between the exposure compensation value corresponding to the second image and the first exposure compensation value is not large.

[0053] In an implementation of the second aspect, a second ghost area corresponding to a frame of a second image is used to fuse the frame of the second image with a current result image to obtain a new current result image, including: calculating the pixel value difference between the frame of the second image and the current result image at each pixel position in the second ghost area corresponding to the frame of the second image, and determining a set of pixel positions whose pixel value difference is greater than a third threshold as a fourth ghost area corresponding to the frame of the second image; and using the fourth ghost area corresponding to the frame of the second image to fuse the frame of the second image with the current result image to obtain a new current result image.

[0054] In the above implementation, the motion intensity at each pixel position is evaluated by calculating the pixel value difference between the two frames of images to be fused in the second ghost area, and then some pixel positions in the second ghost area where no motion actually occurs or the motion is not obvious are excluded, so that the obtained fourth ghost area with a smaller area can cover the moving object more accurately, which is beneficial to improving the quality of the fused image.

[0055] In an implementation of the second aspect, the fourth ghost area corresponding to the second image of the frame is used to fuse the second image of the frame with the current result image to obtain a new current result image, including: taking the pixel value at each pixel position in the current result image located in the fourth ghost area as the pixel value of the new current result image at the corresponding pixel position; and fusing the current result image with the pixel value at each pixel position in the second image of the frame located in a non-fourth ghost area, and taking the pixel values ​​at the corresponding pixel positions as the new current result image.

[0056] In the above implementation, the pixel values ​​in the current result image are retained within the fourth ghost area, and multi-frame fusion is not performed, thereby avoiding the generation of ghosts in the new current result image; multi-frame fusion is performed outside the fourth ghost area to reduce noise in the picture.

[0057] In a third aspect, an embodiment of the present application provides a ghost detection device, comprising: an image acquisition module, used to acquire multiple frames of first images and at least one frame of second image; wherein the first image is an image captured at a first exposure compensation value, and the second image is an image captured at other exposure compensation values ​​different from the first exposure compensation value; a first ghost detection module, used to calculate a first ghost area and a movement speed and movement direction of a moving object in the first ghost area based on the multiple frames of first images; wherein the first ghost area is an area containing moving objects in the multiple frames of first images; a second ghost detection module, used to adjust the area of ​​the first ghost area according to the movement speed and movement direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and one of the first frames, to obtain a second ghost area corresponding to the frame of the second image.

[0058] In a fourth aspect, an embodiment of the present application provides an image fusion device, comprising: a third ghost detection module, used to obtain a first ghost area corresponding to multiple frames of first images and a second ghost area corresponding to each second image in at least one frame of second images using the ghost detection method provided by the first aspect or any possible implementation method of the first aspect; a first fusion module, used to fuse the multiple frames of first images using the first ghost area to obtain a first fused image; and a second fusion module, used to fuse the at least one frame of second image with the first fused image using the second ghost area corresponding to each frame of second image to obtain a final result image.

[0059] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the method provided by the first aspect, the second aspect, or any possible implementation of these two aspects is executed.

[0060] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory stores computer program instructions, and when the computer program instructions are read and run by the processor, the method provided by the first aspect, the second aspect, or any possible implementation of these two aspects is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 shows the ghosting that occurs in the fused image;

[0063] Figure 2 A ghost detection method provided by an embodiment of the present application is shown;

[0064] Figure 3 A calculation process of the first ghost area is shown;

[0065] Figure 4 An example of fusion of the first image and the second image is shown;

[0066] Figure 5 An image fusion method provided by an embodiment of the present application is shown;

[0067] Figure 6The structure of a ghost detection device provided by an embodiment of the present application is shown;

[0068] Figure 7 The structure of an image fusion device provided in an embodiment of the present application is shown;

[0069] Figure 8 The structure of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0071] The terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0072] The terms "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and should not be understood as indicating or implying relative importance, nor should they be understood as requiring or implying any such actual relationship or order between these entities or operations.

[0073] Figure 2 A ghost detection method provided by an embodiment of the present application is shown, and the method can be but is not limited to Figure 8 The electronic device 500 shown performs, referring to Figure 2 , the method comprising:

[0074] Step S110: Acquire multiple frames of first images and at least one frame of second image.

[0075] Exposure compensation is a method of exposure control: by intentionally changing the exposure parameters (such as aperture and shutter) automatically calculated by the camera, the camera's exposure changes, making the captured photos brighter or darker.

[0076] The exposure compensation value can be regarded as a quantitative representation of the exposure compensation strength. The exposure compensation value can be in an interval centered on 0, such as [-2,2], [-3,3], etc. When the exposure compensation value is 0, it means that the exposure automatically determined by the camera is used for shooting. When the exposure compensation value is a positive number, it means that the exposure is increased (relative to the exposure compensation value of 0). When the exposure compensation value is a negative number, it means that the exposure is reduced (relative to the exposure compensation value of 0). The larger the absolute value of the exposure compensation value, the greater the degree of increase or decrease in exposure. In some existing devices, the exposure compensation value is taken in intervals. For example, in [-2,2], with an interval of 1, EV-2, EV-1, EV0, EV+1, and EV+2 can be obtained. The difference between adjacent exposure compensation values ​​is 1EV, which means that the exposure is doubled. For example, EV+1 doubles the exposure relative to EV0. Of course, 1 / 2 and 1 / 3 are also used as the intervals between adjacent exposure compensation values.

[0077] In step S110, multiple frames of the first image are collected at a first exposure compensation value, and at least one frame of the second image is collected at another exposure compensation value different from the first exposure compensation value. Note that if there are multiple frames of the second image, it is not limited here that the exposure compensation values ​​corresponding to each frame of the second image are different. For example, an image combination that meets the requirements in step S110 is: 5 frames of the first image are collected at EV0, 5 frames of the second image are collected at EV-1, and 5 frames of the second image are collected at EV-2.

[0078] When capturing the first image and the second image, the capture can be performed in a gradual order of exposure compensation values, which is convenient for many existing devices. For example, first capture 5 frames of images at EV0, then capture 5 frames of images at EV-1, and finally capture 5 frames of images at EV-2, and the exposure compensation value decreases in the order of 0, -1, and -2; or first capture 5 frames of images at EV-2, then capture 5 frames of images at EV-1, and finally capture 5 frames of images at EV0, and the exposure compensation value increases in the order of -2, -1, and 0.

[0079] The inventor has found through long-term research that for night scene shooting, when multiple frames of images collected at different exposure compensation values ​​are used for fusion denoising, in the fused image, the pixel values ​​of most areas are mainly derived from images with positive exposure compensation values, and only the pixel values ​​of a small number of highlight areas are mainly derived from images with non-positive exposure compensation values. The reason is understandable. At night, the light is dim, and increasing the exposure can retain the details of the picture in most areas, but for the highlight area, reducing the exposure can avoid overexposure.

[0080] Therefore, if the first image and the second image are collected at night (or other dark environments, which are considered to be night for simplicity), and are used for fusion denoising after the ghost area is detected, the first image can be collected under the condition of a positive exposure compensation value, and the second image can be collected under the condition of a non-positive exposure compensation value, so that the fused image will retain richer details. For example, a combination of images that meets the requirements is: 5 frames of the first image are collected at EV+1, 5 frames of the second image are collected at EV0, and 5 frames of the second image are collected at EV-1.

[0081] The inventor further found that in the fused image, the highlight area usually does not occupy too large an area and does not have too many details. Therefore, relatively speaking, the second image is less important than the first image in the process of multi-frame fusion denoising. Therefore, in some implementations, if the first image captures N frames (N>1) at the first exposure compensation value, the second image only needs to capture less than N frames at each other exposure compensation value, which can reduce the amount of calculation during fusion and will not have a significant impact on the quality of the fused image. In particular, in order to reduce the amount of calculation as much as possible, the second image only needs to capture 1 frame at each other exposure compensation value. For example, a combination of images that meets the requirements is: 5 frames of the first image are captured at EV+1, 1 frame of the second image is captured at EV0, and 1 frame of the second image is captured at EV-1. In addition, if you want to further improve the fusion effect at this time, considering that the number of frames of the second image captured at each exposure compensation value is small, you can also increase the denoising intensity when fusion highlights.

[0082] Step S120: Calculate the first ghost area and the moving speed and moving direction of the moving object in the first ghost area according to the multiple frames of first images.

[0083] Step S120 is divided into two parts, one part is to calculate the first ghost area, and the other part is to calculate the moving speed and moving direction of the moving object in the first ghost area. The calculation of the first ghost area is first introduced below:

[0084] The first ghost area may refer to an area containing a moving object in the multiple first image frames. In some implementations, a reference frame may be first selected from the multiple first image frames, and then the first ghost area may be calculated based on the pixel value difference between the non-reference frame and the reference frame in the multiple first image frames.

[0085] For example, a frame (e.g., the first frame) may be fixedly selected as the reference frame, a frame may be randomly selected as the reference frame, and so on. If it is considered that when multiple frames of the first image are fused in the subsequent steps, many schemes will also use the reference frame as the fusion benchmark, then the frame with the highest definition may be selected as the reference frame to improve the quality of the fused image. However, it should be noted that Figure 2The method shown is only a ghost detection method, and whether to perform multi-frame fusion according to the detected ghost area is optional. To evaluate the clarity of the image, a method based on a gradient operator (eg, Sobel operator) may be used but is not limited thereto.

[0086] The first image in the multiple first images except the reference frame is called a non-reference frame. With the reference frame as the reference system, if an object moves between the reference frame and the non-reference frame, the position of the object in the reference frame and the non-reference frame must be different. Therefore, by calculating the pixel value difference between the non-reference frame and the reference frame, the area containing the moving object in the non-reference frame and the reference frame, i.e., the first ghost area, can be determined.

[0087] Furthermore, considering the situation where there are multiple moving objects, these moving objects may be located at different positions in the first image, and each may have different movement rules, which makes it inconvenient to uniformly calculate the movement speed and direction subsequently. Therefore, in some implementations, an initial ghost area can be first calculated based on the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image, and then the initial ghost area can be divided into at least one first ghost area that is not connected to each other. It can be considered that different first ghost areas contain different moving objects. In the subsequent steps, each first ghost area is used to predict the second ghost area, and the result may be more accurate.

[0088] A possible calculation process of the first ghost area is given below, and reference is made to Figure 3 To explain:

[0089] Step A: Calculate the pixel value difference between each non-reference frame and the reference frame at each corresponding pixel position to obtain multiple initial motion masks.

[0090] exist Figure 3 3 frames of first images are shown in FIG. 1 , where the second frame is a reference frame. For simplicity, each frame of the first image includes only 9 pixels, and the pixel value of each pixel is shown in the figure. The pixel values ​​of the first frame of the first image and the second frame of the first image at each corresponding pixel position are subtracted and the absolute value is taken to obtain an initial motion mask image, which is recorded as initial motion mask image 1. Similarly, the pixel values ​​of the third frame of the first image and the second frame of the first image at each corresponding pixel position are subtracted and the absolute value is taken to obtain another initial motion mask image, which is recorded as initial motion mask image 2.

[0091] The pixel value at each pixel position in an initial motion mask image can be understood as the motion intensity of a non-reference frame relative to the reference frame at the pixel position. The larger the pixel value, the greater the motion intensity, and the higher the possibility that the pixel position belongs to the ghost area.

[0092] Step B: superimpose multiple initial motion masks to obtain a first motion mask.

[0093] Reference Figure 3 , the first motion mask can be obtained by directly adding the pixel values ​​at the corresponding pixel positions of the initial motion mask image 1 and the initial motion mask image 2. Of course, this is only one superposition method, and superposition methods such as weighted summation and averaging of the pixel values ​​at the corresponding pixel positions of each initial motion mask are not excluded.

[0094] The pixel value at each pixel position in the first motion mask can be understood as: the superposition of the motion intensity of all non-reference frames relative to the reference frame at the pixel position. The larger the pixel value, the greater the motion intensity, and the higher the possibility that the pixel position belongs to the ghost area.

[0095] Step C: determining whether the pixel value at each pixel position in the first motion mask is greater than a first threshold, and determining a set of pixel positions whose pixel values ​​are greater than the first threshold as an initial ghost area.

[0096] exist Figure 3 In the example, the first threshold T1 is set to 30. Thus, according to the first threshold, the set consisting of the three pixel positions (0, 1), (1, 0), and (1, 1) can be determined as the initial ghost area.

[0097] The position of the initial ghost area can be recorded by each pixel position it contains, by the pixel position located at its edge, or by the second motion mask, etc. The second motion mask is a binary image, in which each pixel value can only take the first value or the second value. If a certain pixel value takes the first value, it indicates that the pixel position where the pixel value is located belongs to the initial ghost area. If a certain pixel value takes the second value, it indicates that the pixel position where the pixel value is located belongs to the non-initial ghost area. For example, the first value can be 1, and the second value can be 0.

[0098] When executing step C, if it is determined that the pixel value at each pixel position in the first motion mask is greater than the first threshold, the pixel value at the corresponding pixel position of the pixel position in the second motion mask can be set to the first value, otherwise the pixel value at the corresponding pixel position of the pixel position in the second motion mask can be set to the second value. After traversing all pixel positions in the first motion mask, the second motion mask can be obtained.

[0099] Reference Figure 3 In the obtained second motion mask, the pixel values ​​at the three pixel positions (0,1), (1,0), and (1,1) are set to 1, so that these three pixel positions constitute the initial ghost area.

[0100] In some implementations, the obtained second motion mask may be smoothed and filtered to eliminate some noise points therein, so as to prevent these noise points from being misjudged as the first ghost area in step D.

[0101] Step D: Divide the initial ghost area into at least one first ghost area that is not connected to each other.

[0102] Since the position of each pixel constituting the initial ghost area has been known in step C, it is easy to determine the connectivity of the initial ghost area, and then the first ghost area can be divided.

[0103] Furthermore, if the second motion mask is used to represent the ghost area, especially when the first value is 1 and the second value is 0, some existing connectivity detection algorithms are directly used to detect the pixels in the initial ghost area in the second motion mask, so that the initial ghost area can be divided into at least one first ghost area that is not connected to each other. Figure 3 In the example, since the three pixel positions in the initial ghost area are connected, only one first ghost area is obtained after executing step D.

[0104] In some implementations, after executing step A, a ghost area can be directly obtained based on each initial motion mask (by comparing the pixel value in the initial motion mask with a threshold), and then the ghost areas are unioned to obtain the initial ghost area.

[0105] In addition, in some simpler implementations, the above initial ghost area may be directly used as the first ghost area without further division.

[0106] In addition to calculating the first ghost area using the pixel value difference between the non-reference frame and the reference frame, there are other solutions, such as performing moving target detection in multiple frames of the first image (for example, a method based on a neural network or background modeling can be used), and then taking the union of the regions where the moving targets are detected in each frame as the first ghost area. Although moving target detection can capture moving objects more accurately, the complexity of these algorithms will also be higher.

[0107] The following describes the calculation of the moving speed and moving direction of the moving object in the first ghost area:

[0108] In some implementations, the optical flow method can be used to first calculate the motion speed and motion direction at each pixel position in the first ghost area, and then the average of the motion speed and motion direction at each pixel position in the first ghost area is calculated, and finally the motion speed and motion direction of the moving object in the first ghost area are obtained. It should be noted here that, according to the definition, the first ghost area contains the moving object in multiple frames of the first image. Ideally, the first ghost area just covers the moving object, but in practice, the first ghost area may also cover content other than the moving object, but it is not easy to strictly distinguish these two parts in the first ghost area. Therefore, when calculating the above-mentioned motion speed and motion direction, it can be considered that the detection result of the first ghost area is ideal, and each pixel position in the first ghost area (rather than only the pixel position belonging to the moving object) will participate in the calculation, and the calculated result is the motion speed and motion direction of the moving object in the first ghost area.

[0109] The speed and direction of each pixel position can be expressed as a vector (the magnitude of the vector is the speed, and the direction of the vector is the direction), i.e., optical flow. The vectors at each pixel position in the first ghost area are averaged, and the average vector obtained can represent the speed and direction of the moving object in the first ghost area. The optical flow methods that can be used include the DIS algorithm, the FlowNet algorithm, etc. These algorithms are relatively mature and have high accuracy in estimating optical flow.

[0110] In other implementations, considering that the first ghost area is essentially formed by the movement of objects in multiple frames of first images, the moving target tracking method can also be used to determine the moving speed and direction of the moving object in the first ghost area.

[0111] It can be understood that no matter which of the above methods is used, calculating the movement speed and movement direction of the moving object in the first ghost area requires at least two different first images, and the two first images are referenced to each other to determine the movement situation therein.

[0112] Therefore, in a simpler implementation, the motion speed and motion direction of the moving object in the first ghost area can be calculated based on two first image frames selected from the multiple first image frames. In many scenarios (for example, when taking photos), the acquisition time intervals of the multiple first image frames are relatively close, and the motion of the object in the picture during this time period can be approximately considered to be linear, so even if only two of the frames are taken, the estimation of the motion trend (referring to the motion speed and motion direction) of the moving object in the first ghost area is relatively accurate.

[0113] For example, two frames of first images may be randomly selected, adjacent first images and the first image that is farthest apart from the adjacent first images in acquisition time may be selected, and so on.

[0114] Among them, the adjacent first image refers to the first image in the multiple first image frames that is closest to the second image (can be any second image frame) in terms of acquisition time interval. For practical considerations, multiple images under each exposure compensation value are generally acquired continuously, so there are only two sequences of image acquisition: first acquire the first image and then the second image, for example, first acquire 5 first images in sequence at EV+1, then acquire 1 second image at EV0, and finally acquire 1 second image at EV-1; and first acquire the second image and then the first image, for example, first acquire 1 second image at EV-1, then acquire 1 second image at EV0, and finally acquire 5 first images in sequence at EV+1.

[0115] According to the definition of the adjacent first image, for the former, the last frame first image acquired at EV+1 is the adjacent first image, and the first image farthest apart from it in acquisition time is the first frame first image acquired at EV+1; for the latter, the first frame first image acquired at EV+1 is the adjacent first image, and the first image farthest apart from it in acquisition time is the last frame first image acquired at EV+1.

[0116] It is not difficult to see that in the above two image acquisition methods, the adjacent first image and the first image farthest apart from it in acquisition time must be the earliest acquired first image and the latest acquired first image (or vice versa), and using two frames of first images far apart in acquisition time to calculate the movement speed and direction of the moving object in the first ghost area may be more accurate, because the calculation result reflects the accumulated movement trend of the moving object in the first image over a longer period of time.

[0117] Taking the optical flow method as an example, the adjacent first image and the first image with the longest acquisition time interval are used as algorithm inputs. The algorithm will output the optical flow field between the two input images. The optical flow in the first ghost area in the optical flow field is averaged to obtain the movement speed and direction of the moving object in the first ghost area.

[0118] In a more accurate implementation, each frame of the first image can be used to calculate the speed and direction of movement of the moving object in the first ghost area. For example, a set of speeds and directions of movement of the moving object in the first ghost area can be calculated based on each non-adjacent first image and adjacent first image in multiple frames of first images, and at least one set of speeds and directions of movement can be obtained. This step is essentially to select two frames of first images each time to calculate a set of speeds and directions of movement of the moving object in the first ghost area, one of which is fixed to the adjacent first image, and the other is selected from other first images. The calculation method has been described above. Then, the mean of at least one set of speeds and directions of movement is calculated, and finally the speed and direction of movement of the moving object in the first ghost area are obtained. Since the information in multiple frames of first images is used, this implementation method may be more accurate in estimating the movement trend of the moving object in the first ghost area, but its calculation amount is also large. Optionally, the adjacent first image in this implementation method can also be replaced by another frame of the first image.

[0119] Step S130: adjusting the area of ​​the first ghost area according to the moving speed and direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and one frame of the first image, to obtain the second ghost area corresponding to the frame of the second image.

[0120] The area adjustment principle of the first ghost area can be: at least the first ghost area before adjustment should be covered, and the moving objects in the second ghost area should be covered as much as possible. The so-called "one of the first frames" in step S130 is not limited to which frame of the multiple first frames it is. Here, the adjacent first image is selected as an example.

[0121] Reference Figure 4 In this scenario, two frames of the first image are first acquired at EV+1, and then one frame of the second image is acquired at EV0 (assuming that the acquisition is performed at a constant speed with a period of T). The moving objects in the image are represented by white circles. After executing step S120, the first ghost area (represented by the shadow) can be obtained. Figure 4 It can be seen that the first ghost area roughly covers the moving object in the two first images. In addition, it is also obtained that the moving direction of the moving object in the first ghost area is rightward (consistent with the moving direction of the moving object), and the moving speed is d / T, where d represents the diameter of the circle.

[0122] According to the definition of the neighboring first image, Figure 4 The adjacent first image in is the first image of the second frame, and the acquisition time interval between the first image and the second image is T. Therefore, the first ghost area is extended to the right by a distance of T×(d / T)=d to obtain the second ghost area, which not only covers the first ghost area, but also covers the moving object in the second image.

[0123] If you first capture 1 frame of the second image at EV0, and then capture 2 frames of the first image at EV+1, and the other conditions remain unchanged, the moving direction of the moving object in the first ghost area is to the left, and the moving speed is d / T. At this time, you need to expand the first ghost area to the right by a distance of T×(d / T)=d to obtain the second ghost area. In summary, if you capture the first image first and then the second image, you should expand it in the same direction as the moving direction of the moving object in the first ghost area; if you capture the second image first and then the first image, you should expand it in the opposite direction to the moving direction of the moving object in the first ghost area.

[0124] Figure 4 There is only one frame of the second image in the image. If there are multiple frames of the second image, the corresponding second ghost area can be obtained according to the above method. If the position of the first ghost area is recorded using a binary mask (second motion mask), the position of the second ghost area can also be recorded using a binary mask. In some implementations, after obtaining the second ghost area corresponding to a frame of the second image, it can also be dilated to appropriately expand its area. Because it is inevitable that there will be inaccuracies when estimating the second ghost area purely based on the motion trend of the moving object in the first ghost area, expanding the area of ​​the second ghost area is beneficial to include the moving object in the second image. If a binary mask is used to represent the second ghost area, the existing region dilation algorithm can be directly executed on the binary mask.

[0125] Although an acquisition time interval can be calculated between any frame of the first image and the second image, and therefore the second ghost area can be estimated, it is more accurate to select the second ghost area calculated from the adjacent first image. The reasons are analyzed as follows:

[0126] According to the definition of the adjacent first image, the adjacent first image and the second image (any frame of the second image) are closest in acquisition time interval. According to the definition of the first ghost area, the first ghost area should also include the position of the moving object in the adjacent first image. Therefore, the expansion of the area of ​​the first ghost area can be regarded as an expansion based on this position, that is, the time starting point of the area expansion behavior should be the moment of acquiring the adjacent first image.

[0127] Therefore, the adjacent first image is selected for calculating the acquisition time interval in step S130, so that the starting point of the time interval coincides with the time starting point of the area expansion behavior (both are the moments when the adjacent first image is acquired). In this way, the calculated second ghost area has a smaller area and a higher accuracy.

[0128] In contrast, if a non-adjacent first image is selected to calculate the acquisition time interval in step S130, the starting point of the time interval is inconsistent with the time starting point of the area expansion behavior, so the calculated second ghost area is larger (so the calculated time interval must be larger than the time interval calculated when the adjacent first image is used) and the accuracy is lower. Of course, the calculated second ghost area is larger, which does not necessarily produce a negative effect, because the estimation of the second ghost area purely based on the movement trend of the moving object in the first ghost area is inevitably inaccurate. A second ghost area with a larger area is conducive to including the moving object in the second image (similar to the reason for the expansion of the second ghost area above).

[0129] In the ghost detection method provided in the embodiment of the present application, the first ghost area and the second ghost area are detected respectively. Among them, the first ghost area is calculated using multiple frames of the first image. Since the multiple frames of the first image are all collected under the first exposure compensation value, their highlight areas are basically the same, so the detection of the first ghost area will basically not be disturbed by the highlight area, and its detection accuracy is relatively high. The second ghost area is not calculated using the first image and the second image, but is estimated based on the movement trend of the moving object in the first ghost area. Therefore, although the exposure compensation value corresponding to the second image is different from the first exposure value, its highlight area is also significantly different from the first image, but it will not affect the calculation result of the second ghost area. Therefore, the detection of the second ghost area will not be disturbed by the highlight area, and its detection accuracy is relatively high.

[0130] Furthermore, since the detection results of the first ghost area and the second ghost area have high accuracy, if multiple frames of the first image and at least one frame of the second image are fused based on the detected ghost area, a high-quality fusion result can also be obtained, for example, which is beneficial to improving the user experience in night scene shooting and other aspects.

[0131] It should be understood that the detected ghost area is not limited to being used for image fusion, for example, it can also be used for output display, used as a training sample, etc.

[0132] Figure 5 An image fusion method provided by an embodiment of the present application is shown, and the method can be but is not limited to Figure 8 The electronic device 500 shown performs, referring to Figure 5 , the method comprising:

[0133] Step S210: using the ghost detection method provided in the embodiment of the present application to obtain a first ghost area corresponding to multiple frames of first images and a second ghost area corresponding to each second image in at least one frame of second images.

[0134] The content of step S210 has been introduced in the previous text and will not be repeated here.

[0135] Step S220: using the first ghost region to fuse multiple frames of first images; wherein the fused image is called a first fused image.

[0136] In some implementations, the fusion can be performed as follows: the pixel value at each pixel position in the reference frame of the multiple frames of the first image located in the first ghost area is used as the pixel value at the corresponding pixel position of the first fused image; the pixel value at each pixel position in the non-first ghost area in the multiple frames of the first image is fused and used as the pixel value at the corresponding pixel position of the first fused image.

[0137] Simply put, the pixel values ​​in the reference frame are retained in the first ghost area, and multi-frame fusion is not performed, so as to avoid ghosting in the first fused image; multi-frame fusion is performed outside the first ghost area to reduce the noise in the picture. Multi-frame fusion under the same exposure compensation value can adopt existing methods such as weighted summation, which will not be explained in detail here. In addition, it should be pointed out that in the ghost detection stage, not every method will select a reference frame from multiple frames of the first image. If the reference frame has not been selected before, it can be selected during multi-frame fusion. The method of selecting the reference frame has been introduced in the previous article and will not be repeated.

[0138] Continue to refer to Figure 4 , the two frames of first images collected at EV+1 are fused using the first ghost area. In the first ghost area, the pixel values ​​in the first frame of the first image (reference frame) are directly used. Therefore, in the first fused image, the position of the circle is the same as that in the first frame of the first image, and no ghost appears at the position corresponding to the circle in the second frame of the first image.

[0139] It is understandable that the fusion of multiple frames of first images is not limited to the above method. For example, in the first ghost area, the pixel values ​​in the reference frame may not be directly retained, but multiple frames may be fused. However, the pixel values ​​in the reference frame will have a higher weight.

[0140] It should be noted that step S220 does not need to wait until step S210 is fully executed before being executed, and can also be executed after the first ghost area is calculated in step S210.

[0141] Step S230: using the second ghost region corresponding to each frame of the second image to fuse at least one frame of the second image with the first fused image; wherein the fused image is referred to as a final result image.

[0142] There are many ways to integrate, only two of them are listed below:

[0143] Method (1)

[0144] The first fused image is used as the initial current result image, and then iteration begins: in each iteration, the second ghost region corresponding to a frame of the second image is used to fuse the frame of the second image with the current result image to obtain a new current result image. The iteration continues until the last frame of the second image is fused with the first fused image, and the current result image obtained at this time is also the final result image. Among them, if the second ghost region is dilated, the dilated second ghost region can be used during fusion.

[0145] As for the fusion order of at least one frame of the second image, it is not limited, for example, it can be in the order of gradually increasing differences between the corresponding exposure compensation values ​​and the first exposure compensation values. For example, 5 frames of the first image are collected at EV+1, 1 frame of the second image is collected at EV0, and 1 frame of the second image is collected at EV-1. After the 5 frames of the first image are fused into the first fused image, the second image collected at EV0 can be fused with the first fused image (the initial current result image) to obtain the current result image, and then the second image collected at EV-1 can be fused with the current result image to obtain the final result image.

[0146] If multiple frames of second images are captured at a certain exposure compensation value, the multiple frames of second images can be fused in the order of increasing acquisition time intervals between each frame of the second image and the adjacent first image. For example, 5 frames of first images are captured sequentially at EV+1, and then 2 frames of second images are captured sequentially at EV0. After the 5 frames of first images are fused into the first fused image, the second image captured first at EV0 can be fused with the first fused image to obtain the current result image, and then the second image captured later at EV0 can be fused with the current result image to obtain the final result image.

[0147] In one round of iteration, the second ghost region corresponding to a frame of the second image is used to fuse the frame of the second image with the current result image to obtain a new current result image. The following method can be used:

[0148] The pixel value at each pixel position in the second ghost area of ​​the current result image is used as the pixel value at the corresponding pixel position of the new current result image; the pixel value of each pixel position in the second image frame and the pixel value at each pixel position in the non-second ghost area of ​​the current result image are fused and used as the pixel value at the corresponding pixel position of the new current result image. This process is similar to the fusion of the first image and will not be explained in detail. The multi-frame fusion under different exposure compensation values ​​can adopt existing methods such as HDR fusion.

[0149] Continue to refer to Figure 4Although the second ghost region can be used to fuse the second image and the first fused image (the arrow indicating fusion is not shown in the figure), so that there is no ghost in the final result image, the second ghost region is expanded from the first ghost region, and the area of ​​the first ghost region is relatively large (because it contains moving objects in multiple frames of the first image), so the area of ​​the second ghost region is also relatively large, in addition to covering the two circles in the second image and the first fused image, it also covers the background area between the two circles. Therefore, if the pixel values ​​in the first fused image are directly retained in the second ghost region during fusion, the pixel values ​​in the first fused image will also be retained in the background area between the two circles, and multi-frame fusion denoising is not performed, thereby affecting the quality of the final result image.

[0150] Therefore, in some implementations, the second ghost region corresponding to the second image of the frame can be optimized by calculating the pixel value difference between the second image to be fused and the current result image (the first fused image also belongs to the current result image), excluding some areas where no moving objects exist (areas with small pixel value difference), and then image fusion is performed. Calculating the pixel value difference between the second image to be fused and the current result image is similar to calculating the initial motion mask in the previous text. Since the pixel value difference can represent the motion intensity at the corresponding pixel position, this optimization strategy for the second ghost region is reasonable. The specific approach is:

[0151] First, the pixel value difference between the second image to be fused and the current result image at each pixel position in the second ghost area corresponding to the second image of the frame is calculated, and the set of pixel positions whose pixel value difference is greater than the third threshold is determined as the fourth ghost area corresponding to the second image of the frame, that is, the optimized second ghost area. The principle of determining the fourth ghost area by combining the pixel value difference is similar to the principle of determining the first ghost area by the pixel value difference in the previous text. Then, the fourth ghost area corresponding to the second image of the frame is used to fuse the second image of the frame with the current result image to obtain a new current result image. The specific fusion method can be:

[0152] The pixel value at each pixel position in the fourth ghost region of the current result image is used as the pixel value at the corresponding pixel position of the new current result image; the pixel value at each pixel position in the non-fourth ghost region of the second image of the frame is fused to form the pixel value at the corresponding pixel position of the new current result image. This process is similar to the fusion of the first image and will not be explained in detail.

[0153] Reference Figure 4The fourth ghost area corresponding to the second image calculated according to the above method only contains two shaded circles, which only covers the two white circles in the second image and the first fused image, but does not cover the background area between the two white circles. Therefore, when the second image and the first fused image are fused, multi-frame fusion noise reduction will be performed in the background area, so that the quality of the final result image is improved.

[0154] Furthermore, in many cases, the second image to be fused and the current result image cannot be regarded as images captured under the same exposure compensation value, so the pixel values ​​of the two have a large difference in overall brightness. Directly calculating the pixel value difference is likely to result in a large calculated difference, which cannot optimize the second ghost area. Figure 4 In the figure, the first fused image (the initial current result image) is a fusion of multiple frames of the first image captured at EV+1, so its corresponding exposure compensation value can also be regarded as EV+1, that is, it differs by 1EV from the exposure compensation value (EV0) corresponding to the second image, that is, the exposure difference is doubled. Obviously, the overall brightness of the pixel values ​​of the two is significantly different.

[0155] To solve this problem, in some implementations, the brightness of the second image to be fused can be first adjusted to match the brightness of the current result image, and then the pixel value difference between the second image to be fused and the current result image at each pixel position in the second ghost region corresponding to the second image of the frame is calculated. The match here can be understood as being close, but not necessarily strictly equal.

[0156] In some implementations, the brightness of the second image to be fused can be adjusted to match the brightness of the current result image using a histogram matching algorithm. In other implementations, brightness matching can also be achieved by adjusting the brightness in blocks. For example, the second image to be fused and the current result image are divided into 32 blocks (or other numbers of blocks) in the same way, and then the mean pixel values ​​of the two images in each block are calculated respectively. Then, the mean pixel value of the current result image in each block is divided by the mean pixel value of the second image to be fused in the corresponding block to calculate 32 transformation coefficients. Finally, the pixel values ​​of the second image to be fused in each block are multiplied by the transformation coefficient corresponding to the block, and the overall brightness of the block can be adjusted to match the overall brightness of the corresponding block in the current result image. Of course, there are many other implementations, which are not listed here.

[0157] Method (2)

[0158] The second image with the largest difference between the corresponding exposure compensation value and the first exposure compensation value is taken as the initial second fused image, and then the iteration begins: in each round of iteration, the second ghost area corresponding to a frame of the second image is taken according to the fusion order, and the second ghost area is used to fuse the current second fused image with the next frame of the second image to obtain a new second fused image. Iteration continues until the second ghost area corresponding to the last frame of the second image also participates in the fusion (or the first fused image also participates in the fusion), and the second fused image obtained at this time is the final result image.

[0159] Among them, the fusion order refers to the order of the second images during fusion, specifically the order in which the difference between the exposure compensation value corresponding to the second image and the first exposure compensation value gradually decreases. In particular, if multiple frames of second images are captured under a certain exposure compensation value, then these multiple frames of second images can be fused in the order in which the capture time interval between each frame of the second image and the adjacent first image gradually decreases.

[0160] If the second image frame is not the last second image frame in the fusion sequence, the next image frame of the second image frame is the next second image frame; if the second image frame is the last second image frame in the fusion sequence, the next image frame of the second image frame is the first fused image.

[0161] If the second ghost area is dilated, the dilated second ghost area can be used during fusion.

[0162] For example, 5 frames of first images are collected at EV+1, 1 frame of second image is collected at EV0, 1 frame of second image is collected at EV-1, and 1 frame of second image is collected at EV-2. After the 5 frames of first images are fused into the first fused image, the second image collected at EV-2 can be fused with the second image collected at EV-1, and then the obtained second fused image can be fused with the second image collected at EV0, and then the obtained second fused image can be fused with the first fused image to obtain the final result image.

[0163] In one round of iteration, the second ghost region corresponding to a frame of the second image is used to fuse the current second fused image with the next frame of the image to obtain a new second fused image. The following method can be used:

[0164] The pixel value at each pixel position in the second ghost region in the next frame image is used as the pixel value at the corresponding pixel position of the new second fused image; the pixel value at each pixel position in the current second fused image and the pixel value at each pixel position in the next frame image not in the second ghost region are fused and used as the pixel value at the corresponding pixel position of the new second fused image. This process is similar to the fusion of the first image and will not be explained in detail.

[0165] In some implementations, for a frame of the second image, the pixel value difference between the current second fused image and the next frame of the second image can be calculated to optimize the second ghost area corresponding to the frame of the second image, exclude some areas where no moving objects exist, and then perform image fusion. The reasons for this processing have been analyzed above. The specific method can be:

[0166] First, the pixel value difference between the current second fused image and the next frame image at each pixel position in the second ghost area corresponding to the second frame image is calculated, and the set of pixel positions whose pixel value difference is greater than the second threshold is determined as the third ghost area corresponding to the second frame image, that is, the optimized second ghost area. Then, the third ghost area corresponding to the second frame image is used to fuse the current second fused image with the next frame image to obtain a new second fused image. The specific fusion method can be:

[0167] The pixel value at each pixel position in the third ghost region in the next frame image is used as the pixel value at the corresponding pixel position of the new second fused image; the pixel value at each pixel position in the current second fused image and the pixel value at each pixel position in the next frame image not in the third ghost region are fused and used as the pixel value at the corresponding pixel position of the new second fused image. This process is similar to the fusion of the first image and will not be explained in detail.

[0168] Furthermore, in many cases, for a frame of the second image, the current second fused image and the next frame of the second image cannot be regarded as images captured under the same exposure compensation value, so that the pixel values ​​of the two are quite different in overall brightness. Directly calculating the difference in pixel values ​​is likely to result in a large calculated difference, which cannot optimize the second ghost area. To solve this problem, in some implementations, the brightness of the current second fused image can be first adjusted to match the brightness of the next frame, and then the pixel value difference between the current second fused image and the next frame at each pixel position in the second ghost area corresponding to the second image is calculated. The match here can be understood as being close, but not necessarily strictly equal.

[0169] In some implementations, the brightness of the current second fused image may be adjusted to match the brightness of the next frame image using a histogram matching algorithm, but other methods may also be used.

[0170] Let's compare method (1) and method (2): method (1) is simple to operate, but it may involve fusing images with a large exposure compensation value span (for example, the first image is collected at EV+1, the second image is collected at EV-2, and the exposure compensation value difference between the two is 3EV). Such image fusion effect is poor, and it is also difficult to optimize the ghost area; method (2) fuses each frame of the second image in the order of gradually decreasing difference between the corresponding exposure compensation value and the first exposure compensation value, so that the span of the exposure compensation value of the two frames of images fused each time is not too large, which is conducive to improving the quality of the fused image. Note that according to the above content, although the overall brightness of the two frames of images can be forcibly brought closer by brightness adjustment, this method will change the pixel value of the original image after all, and when the brightness difference between the two frames of images is too large, some brightness adjustment methods are not applicable.

[0171] Of course, if the span between the exposure compensation value corresponding to each frame of the second image and the first exposure compensation value is not large, the final result image obtained by adopting the method (1) also has good quality.

[0172] Figure 6 FIG. 3 shows a functional module diagram of a ghost detection device 300 provided in an embodiment of the present application. Figure 6 , the ghost detection device 300 comprises:

[0173] The image acquisition module 310 is used to acquire multiple frames of first images and at least one frame of second image; wherein the first image is an image acquired at a first exposure compensation value, and the second image is an image acquired at another exposure compensation value different from the first exposure compensation value;

[0174] A first ghost detection module 320, configured to calculate a first ghost area and a moving speed and a moving direction of a moving object in the first ghost area according to the plurality of first image frames; wherein the first ghost area is an area including the moving object in the plurality of first image frames;

[0175] The second ghost detection module 330 is used to adjust the area of ​​the first ghost area according to the movement speed and movement direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and one frame of the first image, so as to obtain the second ghost area corresponding to the frame of the second image.

[0176] In one implementation of the ghost detection device 300, the first ghost detection module 320 calculates the first ghost area based on the multiple frames of first images, including: selecting a reference frame from the multiple frames of first images; calculating the first ghost area based on the pixel value difference between the non-reference frame and the reference frame in the multiple frames of first images.

[0177] In one implementation of the ghost detection device 300, the first ghost detection module 320 calculates the first ghost area based on the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image, including: calculating the initial ghost area based on the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image; dividing the initial ghost area into at least one first ghost area that is not connected to each other.

[0178] In one implementation of the ghost detection device 300, the first ghost detection module 320 calculates an initial ghost area based on the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image, including: calculating the pixel value difference between each non-reference frame and the reference frame in the multiple frames of the first image at each pixel position to obtain multiple initial motion masks; superimposing the multiple initial motion masks to obtain a first motion mask; determining whether the pixel value at each pixel position in the first motion mask is greater than a first threshold, and determining a set of pixel positions whose pixel values ​​are greater than the first threshold as the initial ghost area.

[0179] In one implementation of the ghost detection device 300, the first ghost detection module 320 determines whether the pixel value at each pixel position in the first motion mask is greater than a first threshold, and determines the set of pixel positions whose pixel values ​​are greater than the first threshold as the initial ghost area, including: determining whether the pixel value at each pixel position in the first motion mask is greater than the first threshold, and setting the pixel value at the corresponding pixel position in the second motion mask whose pixel value is greater than the first threshold to a first value; wherein the pixel value in the second motion mask takes the first value or the second value, the set of pixel positions whose pixel value takes the first value is the initial ghost area, and the set of pixel positions whose pixel value takes the second value is the non-initial ghost area; the first ghost detection module 320 divides the initial ghost area into at least one first ghost area that is not connected to each other, including: dividing the initial ghost area into at least one first ghost area that is not connected to each other by performing connectivity detection on pixels in the second motion mask that are located in the initial ghost area.

[0180] In an implementation of the ghost detection device 300 , the first ghost detection module 320 selects a reference frame from the multiple frames of first images, including: selecting a first image with the highest definition among the multiple frames of first images as a reference frame.

[0181] In one implementation of the ghost detection device 300, the first ghost detection module 320 calculates the movement speed and movement direction of the moving object in the first ghost area based on the multiple frames of first images, including: calculating the movement speed and movement direction of the moving object in the first ghost area based on two frames of first images in the multiple frames of first images.

[0182] In one implementation of the ghost detection device 300, the first ghost detection module 320 calculates the movement speed and direction of the moving object in the first ghost area based on two first images in the multiple first images, including: calculating the movement speed and direction of the moving object in the first ghost area based on an adjacent first image and a first image that is farthest apart from the adjacent first image in acquisition time; wherein the adjacent first image is the first image in the multiple first images that is closest to the second image in acquisition time interval.

[0183] In one implementation of the ghost detection device 300, the first ghost detection module 320 calculates the movement speed and movement direction of the moving object in the first ghost area based on the multiple frames of first images, including: calculating a group of movement speeds and movement directions of the moving object in the first ghost area according to each non-adjacent first image and adjacent first image in the multiple frames of first images, and obtaining at least one group of movement speeds and movement directions in total; wherein the adjacent first image is the first image in the multiple frames of first images that is closest to the second image in terms of the acquisition time interval; and calculating the mean of the at least one group of movement speeds and movement directions to obtain the movement speed and movement direction of the moving object in the first ghost area.

[0184] In one implementation of the ghost detection device 300, the first ghost detection module 320 calculates the movement speed and movement direction of the moving object in the first ghost area based on the multiple frames of the first image, including: calculating the movement speed and movement direction at each pixel position in the first ghost area using the optical flow method based on the multiple frames of the first image; calculating the mean of the movement speed and movement direction at each pixel position in the first ghost area to obtain the movement speed and movement direction of the moving object in the first ghost area.

[0185] In one implementation of the ghost detection device 300, the second ghost detection module 330 adjusts the area of ​​the first ghost area according to the movement speed and direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and one of the frames of the first image, to obtain the second ghost area corresponding to the frame of the second image, including: adjusting the area of ​​the first ghost area according to the movement speed and direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and an adjacent first image, to obtain the second ghost area corresponding to the frame of the second image; wherein the adjacent first image is the first image among the multiple frames of the first image that is closest to the second image in terms of the acquisition time interval.

[0186] In an implementation of the ghost detection device 300, the second ghost detection module 330 is further used to: after obtaining the second ghost area corresponding to any frame of the second image, perform dilation processing on the second ghost area corresponding to the frame of the second image.

[0187] In an implementation of the ghost detection device 300, the first exposure compensation value is a positive number, and the other exposure compensation values ​​are all non-positive numbers.

[0188] In an implementation of the ghost detection device 300 , each frame of the second image is collected under a different exposure compensation value.

[0189] The ghost detection device 300 provided in the embodiment of the present application, its implementation principle and the technical effects produced have been introduced in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the method embodiment.

[0190] Figure 7 FIG. 4 shows a functional module diagram of an image fusion device 400 provided in an embodiment of the present application. Figure 7 , the image fusion device 400 includes:

[0191] A third ghost detection module 410 is used to obtain a first ghost area corresponding to a plurality of frames of first images and a second ghost area corresponding to each second image in at least one frame of second images by using the ghost detection method provided in the embodiment of the present application;

[0192] A first fusion module 420, configured to fuse the multiple frames of first images using the first ghost region to obtain a first fused image;

[0193] The second fusion module 430 is configured to fuse the at least one frame of the second image with the first fused image by using the second ghost region corresponding to each frame of the second image to obtain a final result image.

[0194] In one implementation of the image fusion device 400, the first fusion module 420 uses the first ghost area to fuse the multiple frames of first images to obtain a first fused image, including: using the pixel value at each pixel position in the reference frame of the multiple frames of first images located in the first ghost area as the pixel value of the first fused image at the corresponding pixel position; fusing the pixel value at each pixel position in the multiple frames of first images located in a non-first ghost area as the pixel value of the first fused image at the corresponding pixel position.

[0195] In one implementation of the image fusion device 400, the first exposure compensation value is a positive number, and the other exposure compensation values ​​are all non-positive numbers. The second fusion module 430 uses the second ghost area corresponding to each frame of the second image to fuse the at least one frame of the second image with the first fused image to obtain a final result image, including: taking the second image with the largest difference between the corresponding exposure compensation value and the first exposure compensation value as the initial second fused image, and in the order of gradually decreasing difference between the corresponding exposure compensation value and the first exposure compensation value, sequentially using the second ghost area corresponding to each frame of the second image to fuse the current second fused image with the next frame of the image to obtain a new second fused image; wherein, if the frame of the second image is not the last frame of the second image, the next frame of the image is the next frame of the second image, and if the frame of the second image is the last frame of the second image, the next frame of the image is the first fused image, and the second fused image obtained after the fusion of the first fused images is the final result image.

[0196] In one implementation of the image fusion device 400, the second fusion module 430 uses the second ghost area corresponding to a frame of the second image to fuse the current second fused image with the next frame of the image to obtain a new second fused image, including: calculating the pixel value difference between the current second fused image and the next frame of the image at each pixel position in the second ghost area corresponding to the frame of the second image, and determining the set of pixel positions whose pixel value difference is greater than the second threshold as the third ghost area corresponding to the frame of the second image; using the third ghost area corresponding to the frame of the second image to fuse the current second fused image with the next frame of the image to obtain a new second fused image.

[0197] In one implementation of the image fusion device 400, the second fusion module 430 uses the third ghost area corresponding to the second image of the frame to fuse the current second fused image with the next frame image to obtain a new second fused image, including: using the pixel value at each pixel position in the next frame image located in the third ghost area as the pixel value of the new second fused image at the corresponding pixel position; fusing the current second fused image with the pixel value at each pixel position in the next frame image located in a non-third ghost area, and using them as the pixel value of the new second fused image at the corresponding pixel position.

[0198] In one implementation of the image fusion device 400, the second fusion module 430 is also used to: before calculating the pixel value difference between the current second fused image and the next frame image at each pixel position within the second ghost area corresponding to the second frame image, the method also includes: adjusting the brightness of the current second fused image to match the brightness of the next frame image.

[0199] In one implementation of the image fusion device 400, the second fusion module 430 adjusts the brightness of the current second fused image to match the brightness of the next frame image, including: using a histogram matching algorithm to adjust the brightness of the current second fused image to match the brightness of the next frame image.

[0200] In one implementation of the image fusion device 400, the second fusion module 430 uses the second ghost area corresponding to each frame of the second image to fuse the at least one frame of the second image with the first fused image to obtain a final result image, including: taking the first fused image as the initial current result image, and sequentially using the second ghost area corresponding to each frame of the second image to fuse the frame of the second image with the current result image to obtain a new current result image; wherein the current result image obtained after the last frame of the second image is fused is the final result image.

[0201] In one implementation of the image fusion device 400, the second fusion module 430 uses the second ghost area corresponding to a frame of the second image to fuse the frame of the second image with the current result image to obtain a new current result image, including: calculating the pixel value difference between the frame of the second image and the current result image at each pixel position in the second ghost area corresponding to the frame of the second image, and determining the set of pixel positions whose pixel value difference is greater than a third threshold as the fourth ghost area corresponding to the frame of the second image; using the fourth ghost area corresponding to the frame of the second image to fuse the frame of the second image with the current result image to obtain a new current result image.

[0202] In one implementation of the image fusion device 400, the second fusion module 430 uses the fourth ghost area corresponding to the second image of the frame to fuse the second image of the frame with the current result image to obtain a new current result image, including: using the pixel value at each pixel position in the current result image located in the fourth ghost area as the pixel value of the new current result image at the corresponding pixel position; fusing the current result image with the pixel value at each pixel position in the second image of the frame located in a non-fourth ghost area, and using them as the pixel value of the new current result image at the corresponding pixel position.

[0203] The image fusion device 400 provided in the embodiment of the present application, its implementation principle and the technical effects produced have been introduced in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the method embodiment.

[0204] Figure 8 A possible structure of an electronic device 500 provided in an embodiment of the present application is shown. Figure 8The electronic device 500 includes: a processor 510, a memory 520 and a communication interface 530. These components are interconnected and communicate with each other through a communication bus 540 and / or other forms of connection mechanisms (not shown).

[0205] The memory 520 includes one or more (only one is shown in the figure), which may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The processor 510 and other possible components may access the memory 520 and read and / or write data therein.

[0206] The processor 510 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 510 can be a general-purpose processor, including a central processing unit (CPU), a micro control unit (MCU), a network processor (NP) or other conventional processors; it can also be a special-purpose processor, including a neural network processor (NPU), a graphics processor (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. In addition, when there are multiple processors 510, some of them can be general-purpose processors and the other part can be special-purpose processors.

[0207] The communication interface 530 includes one or more (only one is shown in the figure), which can be used to communicate directly or indirectly with other devices to exchange data. The communication interface 530 can include an interface for wired and / or wireless communication.

[0208] One or more computer program instructions may be stored in the memory 520 , and the processor 510 may read and execute these computer program instructions to implement the ghost detection method and / or image fusion method provided in the embodiments of the present application.

[0209] Understandably, Figure 8 The structure shown is for illustration only. The electronic device 500 may also include Figure 8 More or fewer components as shown, or with Figure 8 Different configurations are shown. Figure 8 The components shown in the figure can be implemented by hardware, software or a combination thereof. The electronic device 500 may be a physical device, such as a PC, a laptop, a tablet computer, a mobile phone, a server, an embedded device, etc., or a virtual device, such as a virtual machine, a virtualized container, etc. Moreover, the electronic device 500 is not limited to a single device, but may also be a combination of multiple devices or a cluster consisting of a large number of devices.

[0210] The present application also provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are read and executed by a computer processor, the ghost detection method and / or image fusion method provided by the present application are executed. For example, the computer-readable storage medium can be implemented as Figure 8 The memory 520 in the electronic device 500.

[0211] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A ghost detection method, characterized in that: include: Acquire multiple frames of first images and at least one frame of second image; wherein the first image is an image captured at a first exposure compensation value, and the second image is an image captured at another exposure compensation value different from the first exposure compensation value; Calculate a first ghost area and a moving speed and a moving direction of a moving object in the first ghost area according to the multiple frames of the first image; wherein the first ghost area is an area containing the moving object in the multiple frames of the first image; The area of ​​the first ghost area is expanded according to the moving speed and direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and one frame of the first image to obtain the second ghost area corresponding to the frame of the second image.

2. The ghost detection method according to claim 1, characterized in that: The calculating the first ghost area according to the multiple frames of first images includes: Selecting a reference frame from the plurality of first image frames; The first ghost area is calculated according to the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image.

3. The ghost detection method according to claim 2, characterized in that: The calculating the first ghost area according to the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image includes: Calculating an initial ghost area according to a pixel value difference between a non-reference frame and a reference frame in the plurality of first image frames; The initial ghost area is divided into at least one first ghost area which is not connected to each other.

4. The ghost detection method according to claim 3, characterized in that: The calculating the initial ghost area according to the pixel value difference between the non-reference frame and the reference frame in the multiple frames of the first image comprises: Calculating the pixel value difference between each non-reference frame and the reference frame at each pixel position in the multiple frames of the first image to obtain multiple initial motion masks; superimposing the multiple initial motion masks to obtain a first motion mask; It is determined whether a pixel value at each pixel position in the first motion mask is greater than a first threshold, and a set of pixel positions whose pixel values ​​are greater than the first threshold is determined as the initial ghost area.

5. The ghost detection method according to any one of claims 1 to 4, characterized in that: Calculating the moving speed and moving direction of the moving object in the first ghost area according to the multiple frames of first images includes: According to two frames of first images among the multiple frames of first images, a moving speed and a moving direction of a moving object in the first ghost area are calculated.

6. The ghost detection method according to claim 5, characterized in that: The calculating the moving speed and moving direction of the moving object in the first ghost area according to two frames of the first image in the plurality of frames of the first image comprises: The movement speed and direction of the moving object in the first ghost area are calculated based on the adjacent first image and the first image that is farthest apart from the adjacent first image in acquisition time; wherein the adjacent first image is the first image among the multiple frames of first images that is closest to the second image in acquisition time interval.

7. The ghost detection method according to any one of claims 1 to 4, characterized in that: Calculating the moving speed and moving direction of the moving object in the first ghost area according to the multiple frames of first images includes: According to each non-adjacent first image and adjacent first image in the multiple frames of first images, a set of motion speeds and motion directions of the moving object in the first ghost area are calculated to obtain at least one set of motion speeds and motion directions; wherein the adjacent first image is the first image in the multiple frames of first images that is closest to the second image in terms of acquisition time interval; The mean of the at least one set of motion speed and motion direction is calculated to obtain the motion speed and motion direction of the moving object in the first ghost area.

8. The ghost detection method according to any one of claims 1 to 4, characterized in that: Calculating the moving speed and moving direction of the moving object in the first ghost area according to the multiple frames of first images includes: Calculate the motion speed and motion direction of each pixel position in the first ghost area by using an optical flow method according to the multiple frames of first images; The average of the motion speed and the motion direction at each pixel position in the first ghost area is calculated to obtain the motion speed and the motion direction of the moving object in the first ghost area.

9. The ghost detection method according to claim 1, characterized in that: The method of expanding the area of ​​the first ghost area according to the moving speed and moving direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and one frame of the first image to obtain the second ghost area corresponding to the frame of the second image includes: The area of ​​the first ghost area is expanded according to the movement speed and direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and the adjacent first image to obtain the second ghost area corresponding to the frame of the second image; wherein the adjacent first image is the first image among the multiple frames of the first image that is closest to the second image in terms of the acquisition time interval.

10. The ghost detection method according to claim 1, characterized in that: After obtaining the second ghost area corresponding to any frame of the second image, the method further includes: The second ghost area corresponding to the second image of the frame is expanded.

11. The ghost detection method according to claim 1, characterized in that: The first exposure compensation value is a positive number, and the other exposure compensation values ​​are all non-positive numbers.

12. The ghost detection method according to claim 11, characterized in that: Each frame of the second image is collected under a different exposure compensation value.

13. An image fusion method, characterized in that: include: Using the ghost detection method described in any one of claims 1 to 12, obtaining a first ghost area corresponding to a plurality of frames of first images and a second ghost area corresponding to each second image in at least one frame of second images; fusing the plurality of first image frames using the first ghost region to obtain a first fused image; The at least one frame of the second image is fused with the first fused image using the second ghost region corresponding to each frame of the second image to obtain a final result image.

14. The image fusion method according to claim 13, characterized in that: The step of fusing the multiple frames of first images using the first ghost area to obtain a first fused image includes: Using the pixel value at each pixel position in the first ghost area in the reference frame in the multiple frames of first images as the pixel value at the corresponding pixel position of the first fused image; The pixel value at each pixel position in the non-first ghost region in the multiple frames of the first image is fused to serve as the pixel value at the corresponding pixel position of the first fused image.

15. The image fusion method according to claim 13, characterized in that: The first exposure compensation value is a positive number, the other exposure compensation values ​​are all non-positive numbers, and the method of using the second ghost area corresponding to each frame of the second image to fuse the at least one frame of the second image with the first fused image to obtain a final result image includes: taking the second image whose corresponding exposure compensation value has the largest difference with the first exposure compensation value as the initial second fused image, and fusing the current second fused image with the next frame of image by using the second ghost region corresponding to each frame of the second image in the order in which the difference between the corresponding exposure compensation value and the first exposure compensation value gradually decreases, to obtain a new second fused image; Among them, if the second image of this frame is not the last second image, the next frame image is the next second image; if the second image of this frame is the last second image, the next frame image is the first fused image, and the second fused image obtained after the fusion of the first fused image is the final result image.

16. The image fusion method according to claim 15, characterized in that: Using a second ghost region corresponding to a frame of the second image to fuse the current second fused image with the next frame of the image to obtain a new second fused image, including: Calculate the pixel value difference between the current second fused image and the next frame image at each pixel position in the second ghost region corresponding to the second image of the frame, and determine a set of pixel positions whose pixel value difference is greater than a second threshold as a third ghost region corresponding to the second image of the frame; The current second fused image is fused with the next frame image using the third ghost region corresponding to the frame of the second image to obtain a new second fused image.

17. The image fusion method according to claim 13, characterized in that: The step of fusing the at least one second image frame with the first fused image using the second ghost region corresponding to each second image frame to obtain a final result image includes: The first fused image is used as the initial current result image, and the second ghost area corresponding to each frame of the second image is used to fuse the second image of that frame with the current result image in turn to obtain a new current result image; wherein the current result image obtained after the fusion of the last frame of the second image is the final result image.

18. The image fusion method according to claim 17, characterized in that: Using a second ghost region corresponding to a frame of the second image to fuse the frame of the second image with the current result image to obtain a new current result image, including: Calculate the pixel value difference between the second image of the frame and the current result image at each pixel position in the second ghost area corresponding to the second image of the frame, and determine a set of pixel positions whose pixel value difference is greater than a third threshold as a fourth ghost area corresponding to the second image of the frame; The fourth ghost area corresponding to the second image of the frame is used to fuse the second image of the frame with the current result image to obtain a new current result image.

19. A ghost detection device, characterized in that: include: An image acquisition module, used to acquire multiple frames of first images and at least one frame of second image; wherein the first image is an image acquired at a first exposure compensation value, and the second image is an image acquired at another exposure compensation value different from the first exposure compensation value; A first ghost detection module, configured to calculate a first ghost area and a moving speed and a moving direction of a moving object in the first ghost area according to the plurality of first image frames; wherein the first ghost area is an area including the moving object in the plurality of first image frames; The second ghost detection module is used to expand the area of ​​the first ghost area according to the moving speed and direction of the moving object in the first ghost area and the acquisition time interval between each frame of the second image and one frame of the first image, so as to obtain the second ghost area corresponding to the frame of the second image.

20. An image fusion device, characterized in that: include: A third ghost detection module, configured to obtain a first ghost area corresponding to the plurality of first image frames and a second ghost area corresponding to each second image in at least one second image frame by using the ghost detection method described in any one of claims 1 to 12; A first fusion module, configured to fuse the multiple frames of first images using the first ghost region to obtain a first fused image; The second fusion module is used to fuse the at least one frame of the second image with the first fused image by using the second ghost area corresponding to each frame of the second image to obtain a final result image.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 18 is executed.

22. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer program instructions, and when the computer program instructions are read and executed by the processor, the method according to any one of claims 1 to 18 is executed.

Images