Image Fusion Method and Device

Abstract

This application relates to the field of image processing, and particularly to an image fusion method and apparatus. An image fusion method, the method comprising: obtaining illumination scene information of a first image sequence; determining, according to the illumination scene information, whether the first image sequence is captured in a normal illumination scene; if the first image sequence is captured in an abnormal illumination scene, obtaining foreground offset information of image frames in the first image sequence; determining, according to the foreground offset information, an alignment manner of the image frames in the first image sequence; aligning the image frames in the first image sequence according to the alignment manner; and fusing the aligned image frames into a super-resolution image. In the embodiments of the present invention, when there are few image features, the image is partitioned, and a part containing more feature information is selected for alignment to improve the alignment accuracy of the image, enhance the super-resolution reconstruction effect, and optimize the visual effect of the reconstructed subject.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular, to an image fusion method and apparatus.

Background Art

[0002] Image super-resolution reconstruction is a technology that converts low-resolution images into high-resolution images and is widely used in image processing systems. For multi-frame based image super-resolution reconstruction, generally, multiple frames of images need to be aligned first, and then the aligned multiple frames of images are fused into a target image. The accuracy of image alignment directly affects the detail effect of the fused target image.

[0003] Since there is less effective information available in some extreme exposure or low-light scenarios, it is impossible to distinguish the motion states of foreground objects in the image well, which is likely to result in extracting incorrect feature information during the image alignment process. If only the traditional feature method is used to perform full-image alignment using the feature information of the entire image, the accuracy of image alignment will be reduced due to the influence of some incorrect features, and further lead to a deviation in the super-resolution reconstruction effect.

Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an image fusion method and apparatus. By dividing an image into a foreground region and a background region, and selecting the region containing more feature information for image alignment and fusion, the problem of low image alignment accuracy and deviation in super-resolution reconstruction effect in special scenarios is solved.

[0005] In a first aspect, an embodiment of the present invention provides an image fusion method, the method including:

[0006] Obtain the illumination scene information of the first image sequence;

[0007] Determine whether the first image sequence is taken in a normal illumination scene according to the illumination scene information;

[0008] If the first image sequence is taken in an abnormal illumination scene, obtain the foreground offset information of the image frames in the first image sequence;

[0009] Determine the alignment method of the image frames in the first image sequence according to the foreground offset information;

[0010] Align the image frames in the first image sequence according to the alignment method;

[0011] Fuse the aligned image frames into a super-resolution image.

[0012] Optionally, the obtaining the foreground offset information of the image frames in the first image sequence includes:

[0013] Determine the reference frame and non-reference frame in the first image sequence;

[0014] Obtain the foreground image of the reference frame and the foreground image of the non-reference frame;

[0015] Align the foreground image of the non-reference frame with the foreground image of the reference frame by template matching method to obtain the foreground offset information of the non-reference frame.

[0016] Optionally, the obtaining the foreground image of the reference frame and the foreground image of the non-reference frame includes:

[0017] Perform semantic segmentation processing on the reference frame to obtain a binary mask image;

[0018] Compare the reference frame with the mask image to obtain the foreground image of the reference frame;

[0019] Compare the non-reference frame with the mask image to obtain the foreground image of the non-reference frame.

[0020] Optionally, the aligning the foreground image of the non-reference frame with the foreground image of the reference frame by template matching method to obtain the foreground offset information of the non-reference frame includes:

[0021] Perform erosion processing on the foreground image of the reference frame and the foreground image of the non-reference frame to obtain the foreground image connectivity domain of the reference frame and the foreground image connectivity domain of the non-reference frame;

[0022] Align the foreground image connectivity domains of the reference frame and the non-reference frame at corresponding positions by template matching method to obtain the offset values at each position of the foreground image connectivity domain;

[0023] Add up the offset values at each position of the foreground image connectivity domain to obtain the foreground offset information of the non-reference frame.

[0024] Optionally, the determining the alignment method of the image frames in the first image sequence according to the foreground offset information includes:

[0025] If the foreground offset information of the non-reference frame is less than or equal to a preset first offset threshold, align the foreground image of the non-reference frame with the foreground image of the reference frame by feature method;

[0026] If the foreground offset information of the non-reference frame is greater than the preset first offset threshold, re-determine the alignment method of the non-reference frame and the reference frame through the background offset information of the non-reference frame.

[0027] Optionally, before re-determining the alignment method of the non-reference frame and the reference frame through the background offset information of the non-reference frame, the method further includes:

[0028] Obtain the background image of the reference frame and the background image of the non-reference frame;

[0029] Reduce the background image of the reference frame and the background image of the non-reference frame proportionally according to a preset ratio to obtain the first background image of the reference frame and the first background image of the non-reference frame;

[0030] Align the first background image of the reference frame and the first background image of the non-reference frame by template matching to obtain the first background offset information of the non-reference frame;

[0031] Align the background image of the reference frame and the background image of the non-reference frame by feature method to obtain the second background offset information of the non-reference frame.

[0032] Optionally, the re-determining the alignment manner between the non-reference frame and the reference frame by the background offset information of the non-reference frame includes:

[0033] If the difference between the second background offset information and the first background offset information of the non-reference frame is less than or equal to a preset second offset value threshold, align the background image of the non-reference frame and the background image of the reference frame by feature method;

[0034] If the difference between the second background offset information and the first background offset information of the non-reference frame is greater than a preset second offset value threshold, align the non-reference frame and the reference frame by template matching.

[0035] In a second aspect, an embodiment of the present invention provides an image fusion device, including:

[0036] A first acquisition module, which acquires the illumination scene information of the first image sequence;

[0037] A first determination module, which determines whether the first image sequence is captured in a normal illumination scene according to the illumination scene information;

[0038] A second acquisition module, if the first image sequence is captured in an abnormal illumination scene, acquires the foreground offset information of the image frames in the first image sequence;

[0039] A second determination module, which determines the alignment manner of the image frames in the first image sequence according to the foreground offset information;

[0040] An alignment module, which aligns the image frames in the first image sequence according to the alignment manner;

[0041] A fusion module, which fuses the aligned image frames into a super-resolution image.

[0042] In a third aspect, an embodiment of the present invention provides an image fusion device, including:

[0043] At least one processor; and

[0044] At least one memory communicatively connected to the processor, wherein:

[0045] The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of the first aspect by invoking the program instructions.

[0046] Optionally, a computer-readable storage medium includes a stored program, wherein when the program runs, it controls the device where the computer-readable storage medium is located to execute the method according to any one of the first aspect.

[0047] Through the above solutions, the problems of poor image alignment accuracy and poor super-resolution reconstruction effect when image features are missing due to some special scenarios are solved.

BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0049] Figure 1 It is a flowchart of an image fusion method provided by an embodiment of the present invention;

[0050] Figure 2 It is a flowchart of another image fusion method provided by an embodiment of the present invention;

[0051] Figure 3 It is a schematic structural diagram of an image fusion device provided by an embodiment of the present invention;

[0052] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

DETAILED DESCRIPTION

[0053] In order to better understand the technical solutions of the present invention, the embodiments of the present invention will be described in detail below with reference to the drawings.

[0054] It should be clear that the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0055] In order to achieve image super-resolution reconstruction, it is necessary to pre-align multiple frames of images. Image alignment is an important method in the field of image processing. For two images in a set of image data, by finding a spatial transformation to map one image to the other, the points corresponding to the same position in the two images are made to correspond one by one. The purpose is to quantitatively compare the offset information of the scenes or objects in two frames of images, or to fuse multiple frames of images of the same object obtained under different conditions.

[0056] Traditional image alignment methods include template matching method and feature method. The template matching method requires cropping out a local template image from one frame of image and searching for a sub-image similar to the local template image in another image. It is generally used to study the positional relationship between local features and another image when the specific position of local features is clear. The template matching method specifically includes algorithms such as gray-scale image correlation matching.

[0057] The feature method requires extracting all the features of one frame of image, then generating descriptors, and matching the features of two frames of images according to the similarity of the descriptors. It is generally used for full-image alignment of two frames of images with more features before image fusion. The feature method specifically includes algorithms such as Scale-invariant feature transform (SIFT), Speeded Up Robust Features (SURF), Oriented Fastand Rotated Brief (ORB), etc.

[0058] In the process of image super-resolution reconstruction, the same features in multiple frames of images are usually extracted by the feature method for alignment and fusion. However, whether using the template matching method or the feature method for image alignment, both require the two images to contain sufficient features to improve the alignment accuracy.

[0059] In overexposed and low-light scenes, alignment often fails because insufficient effective features cannot be extracted. When there is foreground interference in the scene, it is easy for the features in the image to change due to the movement of the foreground object, which may lead to errors in feature-based alignment and poor super-resolution reconstruction results.

[0060] In the embodiments of the present invention, in an abnormal illumination scene with foreground interference, through scene semantic segmentation, the image frame is divided into foreground and background, and motion offset analysis is respectively performed on the foreground and background. The part with smaller offset, that is, the part containing more identical features, is selected for alignment, so as to improve the alignment accuracy of the image, enhance the super-resolution reconstruction effect, and optimize the visual effect of the reconstructed subject.

[0061] The embodiments of the present invention provide an image fusion method. AsFigure 1 As shown in Figure 1 , the processing steps of this method include:

[0062] 101. Obtain the illumination scene information of the first image sequence.

[0063] Specifically, receive the first image sequence captured by the shooting unit, read the illumination data when shooting the first image sequence through the light sensor, and determine the illumination scene information of the first image sequence through the illumination data.

[0064] The illumination scene information is usually divided into: dark light scene, normal scene, and overexposed scene.

[0065] In some embodiments, the dark light scene can be a night scene without lights; the normal scene can be a scene with moderate illumination intensity such as an office or a shopping mall; the overexposed scene can be a scene such as a sunny day with sufficient sunlight facing away from the sun or facing away from the light source indoors.

[0066] Among them, the first image sequence is composed of multiple image frames obtained by the shooting unit through the zero-delay shooting function in a short time, usually including 5 to 10 image frames. Since each image frame in the first image sequence is captured in an extremely short time, the illumination scene information of the entire first image sequence can be determined by reading the illumination scene information of any one image frame.

[0067] 102. Determine whether the first image sequence is captured in a normal illumination scene according to the illumination scene information.

[0068] Specifically, when the illumination scene information of the first image sequence is a dark light scene or an overexposed scene, it is determined that the first image sequence is captured in an abnormal illumination scene.

[0069] Among them, the illumination scene information such as dark light, normal, and overexposed is determined by the first illumination threshold and the second illumination threshold calculated in advance through a training model. By calculating a large number of images, image illumination data, and the number of features included in the images, the corresponding relationship between the number of features included in the images and the illumination data is obtained, and the critical points where the number of features included in the images is significantly reduced are determined as the two illumination thresholds, that is, once the illumination is lower than the first illumination threshold or higher than the first threshold, the features in the image frame will be significantly reduced.

[0070] When the illumination scene information of the first image sequence is between the first illumination threshold and the second illumination threshold, it is determined that the first image sequence is captured in a normal illumination scene; when the illumination scene of the first image sequence is less than the first illumination threshold or greater than the second illumination threshold, it is determined that the first image sequence is in a dark light scene or an overexposed scene, that is, an abnormal illumination scene is captured.

[0071] If the first image sequence is captured in a normal lighting scene, the full images of each image frame in the first image sequence are directly aligned by the feature method; if the first image sequence is captured in an abnormal lighting scene, the subsequent steps are continued.

[0072] 103. If the first image sequence is captured in an abnormal lighting scene, obtain the foreground offset information of the image frames in the first image sequence.

[0073] Specifically, determine a reference frame in the first sequence of images, and determine the remaining image frames as non-reference frames. Align each non-reference frame with the reference frame in turn through the template matching method to obtain the foreground offset information of each non-reference frame.

[0074] Among them, determine the image frame with the highest clarity as the reference frame through the clarity detection algorithm. Any clarity detection algorithm can be used in the process of determining the reference frame, such as the method of determining the reference frame according to the maximum value information of the gradient statistics of the image.

[0075] In the embodiment of the present invention, any non-reference frame in the first image sequence is taken as an example for illustration.

[0076] When determining the foreground offset information of the non-reference frame, it is necessary to first perform semantic segmentation processing on the reference frame to obtain a binary mask image with the same size as the first sequence of images, and represent the background by 0 and the foreground by 1. Comparing the mask image with the reference frame can obtain the foreground image of the reference frame; comparing the mask image with the non-reference frame image can obtain the foreground image of the non-reference frame.

[0077] Erode the foreground images of the reference frame and the non-reference frame respectively to obtain the foreground image connected domain of the reference frame and the foreground image connected domain of the non-reference frame. Align the foreground image connected domains of the reference frame and the non-reference frame at the corresponding positions through the template matching method to obtain the offset values at each position of the foreground image connected domain. And add up the offset values at each position of the foreground image connected domain to obtain the foreground offset information of the non-reference frame.

[0078] Among them, the foreground offset information of the non-reference frame is used to characterize whether there is a large-scale offset change in the foreground image of the non-reference frame compared with the foreground image of the reference frame. Thus, it can be determined whether the non-reference frame still has sufficient features to be aligned with the reference frame with high precision through the foreground offset information.

[0079] 104. Determine the alignment method of the image frames in the first image sequence according to the foreground offset information.

[0080] Specifically, if the foreground offset information of a non-reference frame is less than or equal to a preset first offset threshold, the foreground image of the non-reference frame is aligned with the foreground image of the reference frame by a feature method; if the foreground offset information of the non-reference frame is greater than the preset first offset threshold, the alignment method between the non-reference frame and the reference frame is re-determined by the background offset information of the non-reference frame.

[0081] Among them, the fact that the foreground offset information of the non-reference frame is less than or equal to the preset first offset threshold indicates that the foreground image of the non-reference frame has not changed significantly compared with the foreground image of the reference frame, and there are still enough features in the foreground to achieve high-precision alignment. Therefore, the foreground images of the reference frame and the non-reference frame are aligned by the feature method.

[0082] The fact that the foreground offset information of the non-reference frame is greater than the preset first offset threshold indicates that the foreground image of the non-reference frame has changed significantly compared with the foreground image of the reference frame, and there are not enough features to achieve high-precision alignment. If the foreground images of the reference frame and the non-reference frame are still aligned by the feature method, it is likely to cause alignment errors due to incorrect feature extraction. Therefore, it is necessary to re-determine whether alignment can be based on the background image through the background offset information of the non-reference frame.

[0083] 105. Align the image frames in the first image sequence according to the alignment method.

[0084] Specifically, according to the alignment methods determined for each non-reference frame, each non-reference frame is aligned with the reference frame respectively.

[0085] 106. Fuse the aligned image frames into a super-resolution image.

[0086] In the embodiments of the present invention, in the case of fewer image features, the relatively important foreground image in the image is divided out in the image frame through scene semantic segmentation, and motion analysis is performed on the foreground image. When the motion offset of the foreground object is small and relatively more features are retained, only the foreground image with more retained features is used to align the image frames, so as to improve the alignment accuracy of the images, enhance the super-resolution reconstruction effect, and optimize the visual effect of the reconstructed subject.

[0087] In some embodiments, the foreground offset information of the image frame exceeds the predetermined first offset value threshold, indicating that the foreground object in the foreground image has moved or changed significantly, and the extractable features are reduced. If the alignment is still based on the foreground image of the image frame, it will cause incorrect feature extraction, resulting in a decrease in the image alignment accuracy and a poor image fusion effect. At this time, it should be re-determined whether there are still enough features retained in the background image of the image frame, and the image alignment is performed according to the background image.

[0088] See Figure 2, another image fusion method provided by the embodiments of the present invention, is used to align through a background image after a large - scale offset of a foreground object in a first image sequence results in feature loss. As Figure 2 shown, the processing steps of this method include:

[0089] 201. Obtain the illumination scene information of the first image sequence.

[0090] 202. Determine whether the first image sequence is captured in a normal illumination scene according to the illumination scene information.

[0091] 203. If the first image sequence is captured in an abnormal illumination scene, obtain the foreground offset information of the image frames in the first image sequence.

[0092] 204. Determine the alignment method of the image frames in the first image sequence according to the foreground offset information.

[0093] 205. Obtain the background offset information of the image frames in the first image sequence.

[0094] Specifically, invert the mask image, where 0 represents the foreground and 1 represents the background. Compare the mask image with the reference frame and non - reference frames to obtain the background image of the reference frame and the background image of the non - reference frames.

[0095] Reduce the background image of the reference frame and the background image of the non - reference frame proportionally by a preset ratio to obtain the first background image of the reference frame and the first background image of the non - reference frame, and align the first background image of the reference frame and the first background image of the non - reference frame through the template matching method to obtain the first background offset information of the non - reference frame. The preset ratio is usually set according to the size of the image frame.

[0096] Align the background image of the reference frame and the background image of the non - reference frame through the feature method to obtain the second background offset information of the non - reference frame.

[0097] Among them, after the overall size of the image is reduced, the specific content in the image will also be reduced correspondingly. Some motion changes in the large - image state will also be reduced as the image size is reduced, so as to achieve the effect of removing the motion changes in the image. The first background offset information obtained after aligning the first background image of the reference frame and the first background image of the non - reference frame can be regarded as the offset information after removing the motion changes. Taking the first background information as the benchmark, determine whether there is a large - scale offset change in the background image of the non - reference frame compared with the background image of the reference frame through the second background information. Thus, it can be determined whether the non - reference frame still has sufficient features to be aligned with the reference frame with high precision through the background offset information.

[0098] 206. Re - determine the alignment method of the image frames in the first image sequence according to the background offset information.

[0099] Specifically, if the difference between the second background offset information of the non-reference frame and the first background offset information is less than or equal to a preset second offset value threshold, the background images of the non-reference frame and the reference frame are aligned by the feature method.

[0100] If the difference between the second background offset information of the non-reference frame and the first background offset information is greater than the preset second offset value threshold, the non-reference frame and the reference frame are aligned by the template matching method.

[0101] Among them, the difference between the second background offset information and the first background offset information being less than or equal to the preset second offset threshold indicates that the actual offset information between the background images of the non-reference frame and the reference frame is not much different from the offset information after eliminating the motion offset change, that is, the background image of the non-reference frame has not changed significantly compared with the background image of the reference frame, and there are still enough features in the background to achieve high-precision alignment. Therefore, the background images of the reference frame and the non-reference frame are aligned by the feature method.

[0102] The difference between the second background offset information and the first background offset information being greater than the preset second offset threshold indicates that the actual offset information between the background images of the non-reference frame and the reference frame is quite different from the offset information after eliminating the motion offset change, that is, the background image of the non-reference frame has changed significantly compared with the background image of the reference frame, and there are not enough features in the background to achieve high-precision alignment. If the background images of the reference frame and the non-reference frame are still aligned by the feature method, alignment errors are likely to occur due to incorrect feature extraction. Since it has been previously determined through the foreground offset information of the non-reference frame that the non-reference frame cannot be aligned through the foreground image, the alignment between the reference frame and the non-reference frame can only be performed by the template matching method.

[0103] 207. Align the image frames in the first image sequence according to the alignment method.

[0104] 208. Fuse the aligned image frames into a super-resolution image.

[0105] In the embodiment of the present invention, in the case of fewer image features, the image frames are divided into foreground and background through scene semantic segmentation, and different methods of motion offset analysis are used for the foreground and background. When the foreground contains insufficient features, the alignment accuracy of the image can be improved by selecting the background containing more feature information for alignment, the super-resolution reconstruction effect can be improved, and the visual effect of the reconstructed subject can be optimized.

[0106] Corresponding to the above image fusion method, the embodiment of the present invention also provides an image fusion device. Refer to Figure 3 , which is a schematic structural diagram of an image fusion device provided by the embodiment of the present invention. AsFigure 3 As shown, the device may include: a first acquisition module 301, a first determination module 302, a second acquisition module 303, a second determination module 304, an alignment module 305, and a fusion module 306.

[0107] The first acquisition module 301 acquires the illumination scene information of the first image sequence;

[0108] The first determination module 302 determines whether the first image sequence is captured in a normal illumination scene according to the illumination scene information;

[0109] If the first image sequence is captured in an abnormal illumination scene, the second acquisition module 303 acquires the foreground offset information of the image frames in the first image sequence;

[0110] The second determination module 304 determines the alignment mode of the image frames in the first image sequence according to the foreground offset information;

[0111] The alignment module 305 aligns the image frames in the first image sequence according to the alignment mode;

[0112] The fusion module 306 fuses the aligned image frames into a super-resolution image.

[0113] Figure 3 The image fusion device provided in the illustrated embodiment can be used to execute the technical solutions of the method embodiments shown in this specification. The implementation principle and technical effects can be further referred to the relevant descriptions in the method embodiments.

[0114] Figure 4 It is a schematic structural diagram of an embodiment of an electronic device in this specification. The electronic device can be the host device or the terminal device in the embodiments of the present invention. As Figure 4 shown, the above electronic device may include at least one processor; and at least one memory communicatively connected to the above processing unit, wherein: the memory stores program instructions executable by the processing unit, and the above processor can execute the image fusion method provided in this embodiment by calling the above program instructions.

[0115] Among them, the above electronic device can be a device capable of having an intelligent conversation with the user. The specific form of the above electronic device is not limited in the embodiments of this specification. It can be understood that the electronic device here is the machine mentioned in the method embodiments.

[0116] Figure 4 shows a block diagram of an exemplary electronic device suitable for implementing the embodiments of this specification. Figure 4 The shown electronic device is only an example and should not bring any limitation to the functions and usage scope of the embodiments of this specification.

[0117] As Figure 4 shown, the electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors 410, a communication interface 420, a memory 430, and a communication bus 440 connecting different system components (including the memory 430, the communication interface 420, and the processor 410).

[0118] The communication bus 440 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnection (PCI) bus.

[0119] The electronic device typically includes a variety of computer system-readable media. These media can be any available media accessible by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0120] The memory 430 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 430 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present specification.

[0121] A program / utility with a set (at least one) of program modules may be stored in the memory 430. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment. The program modules generally perform the functions and / or methods described in the embodiments of the present specification.

[0122] The processor 410 executes various functional applications and data processing by running programs stored in the memory 430, such as implementing the image fusion method provided in the embodiments shown in this specification.

[0123] An embodiment of this specification provides a non-transitory computer-readable storage medium that stores computer instructions, and the computer instructions cause the computer to execute the image fusion method provided in the embodiments shown in this specification.

[0124] The above non-transitory computer-readable storage medium may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (hereinafter referred to as: ROM), an erasable programmable read-only memory (hereinafter referred to as: EPROM) or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0125] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal may take various forms, including - but not limited to - electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0126] The program code contained on the computer-readable medium may be transmitted by any appropriate medium, including - but not limited to - wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0127] Computer program code for performing the operations of this specification may be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0128] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0129] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this specification, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0130] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of this specification includes additional implementations where the functions may be performed not in the order shown or discussed, including in a substantially simultaneous manner or in the reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of this specification pertain.

[0131] Depending on the context, as used herein, the word "if" can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detected (stated condition or event)" or "in response to detecting (stated condition or event)".

[0132] It should be noted that the terminals involved in the embodiments of this specification may include, but are not limited to, personal computers (Personal Computer; hereinafter referred to as: PC), personal digital assistants (Personal Digital Assistant; hereinafter referred to as: PDA), wireless handheld devices, tablet computers (Tablet Computer), mobile phones, MP3 players, MP4 players, etc.

[0133] In the embodiments provided in this specification, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0134] In addition, in each embodiment of this specification, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0135] The above-mentioned integrated units implemented in the form of software functional units can be stored in a computer-readable storage medium. The above-mentioned software functional units stored in a storage medium include several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (Processor) to execute some steps of the methods described in the embodiments of this specification.

[0136] The above are only the preferred embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included within the scope of protection of this specification.

Claims

1. An image fusion method, characterized in that, the method includes: Obtain the illumination scene information of the first image sequence; Determine whether the first image sequence is taken in a normal illumination scene according to the illumination scene information; If the first image sequence is taken in an abnormal illumination scene, obtain the foreground offset information of the image frames in the first image sequence; Determine the alignment method of the image frames in the first image sequence according to the foreground offset information; Align the image frames in the first image sequence according to the alignment method; Fuse the aligned image frames into a super-resolution image; The determining the alignment method of the image frames in the first image sequence according to the foreground offset information includes: If the foreground offset information of a non-reference frame is less than or equal to a preset first offset threshold, align the foreground image of the non-reference frame with the foreground image of the reference frame by a feature method; If the foreground offset information of a non-reference frame is greater than the preset first offset threshold, re-determine the alignment method of the non-reference frame and the reference frame according to the background offset information of the non-reference frame.

2. The method according to claim 1, characterized in that, the obtaining the foreground offset information of the image frames in the first image sequence includes: Determine the reference frame and non-reference frames in the first image sequence; Obtain the foreground image of the reference frame and the foreground image of the non-reference frame; Align the foreground image of the non-reference frame with the foreground image of the reference frame by a template matching method to obtain the foreground offset information of the non-reference frame.

3. The method according to claim 2, characterized in that, the obtaining the foreground image of the reference frame and the foreground image of the non-reference frame includes: Perform semantic segmentation processing on the reference frame to obtain a binary mask image; Compare the reference frame with the mask image to obtain the foreground image of the reference frame; Compare the non-reference frame with the mask image to obtain the foreground image of the non-reference frame.

4. The method according to claim 2, characterized in that, the aligning the foreground image of the non-reference frame with the foreground image of the reference frame by a template matching method to obtain the foreground offset information of the non-reference frame includes: Perform erosion processing on the foreground image of the reference frame and the foreground image of the non-reference frame to obtain the foreground image connectivity domain of the reference frame and the foreground image connectivity domain of the non-reference frame; Align the foreground image connectivity domains of the reference frame and the non-reference frame at corresponding positions by a template matching method to obtain the offset values at each position of the foreground image connectivity domain; Add up the offset values at each position of the foreground image connectivity domain to obtain the foreground offset information of the non-reference frame.

5. The method according to claim 1, characterized in that, before re-determining the alignment method of the non-reference frame and the reference frame according to the background offset information of the non-reference frame, the method further includes: Obtain the background image of the reference frame and the background image of the non-reference frame; Reduce the background image of the reference frame and the background image of the non-reference frame proportionally to the first background image of the reference frame and the first background image of the non-reference frame according to a preset ratio; Align the first background image of the reference frame and the first background image of the non-reference frame through template matching to obtain the first background offset information of the non-reference frame; Align the background image of the reference frame and the background image of the non-reference frame through feature method to obtain the second background offset information of the non-reference frame.

6. The method according to claim 5, wherein, the re-determining the alignment manner between the non-reference frame and the reference frame through the background offset information of the non-reference frame includes: if the difference between the second background offset information of the non-reference frame and the first background offset information is less than or equal to a preset second offset value threshold, align the background image of the non-reference frame and the background image of the reference frame through feature method; if the difference between the second background offset information of the non-reference frame and the first background offset information is greater than the preset second offset value threshold, align the non-reference frame and the reference frame through template matching method.

7. An image fusion device, wherein, comprising: a first acquisition module, which acquires the illumination scene information of the first image sequence; a first determination module, which determines whether the first image sequence is captured in a normal illumination scene according to the illumination scene information; a second acquisition module, if the first image sequence is captured in an abnormal illumination scene, acquires the foreground offset information of the image frames in the first image sequence; a second determination module, which determines the alignment manner of the image frames in the first image sequence according to the foreground offset information; an alignment module, which aligns the image frames in the first image sequence according to the alignment manner; a fusion module, which fuses the aligned image frames into a super-resolution image; the determining the alignment manner of the image frames in the first image sequence according to the foreground offset information includes: if the foreground offset information of the non-reference frame is less than or equal to a preset first offset threshold, align the foreground image of the non-reference frame and the foreground image of the reference frame through feature method; if the foreground offset information of the non-reference frame is greater than the preset first offset threshold, re-determine the alignment manner between the non-reference frame and the reference frame through the background offset information of the non-reference frame.

8. An image fusion device, wherein, comprising: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 6 by invoking the program instructions.

9. A computer-readable storage medium, wherein, the computer-readable storage medium includes a stored program, wherein when the program runs, it controls the device where the computer-readable storage medium is located to execute the method according to any one of claims 1 to 6.

Images