Image fusion method, image fusion device and storage medium

Abstract

The present disclosure relates to an image fusion method, an image fusion device and a storage medium. The image fusion method includes: determining multiple frames of images to be fused; determining a reference image in the multiple frames of images, and determining the motion state of a non-reference image in the multiple frames of images relative to the reference image, wherein the non-reference image is an image in the multiple frames of images that is different from the reference image; in response to the non-reference image including a first image, replacing the first image with the reference image, and performing image fusion based on the replaced reference image, wherein the first image is a non-reference image whose motion state is a non-static state. The present disclosure can improve the ghosting phenomenon caused by the fusion of non-static images.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image fusion method, an image fusion device and a storage medium. Background Art

[0002] With the development of science and technology, image deblurring, as a commonly used image processing method, has been used by people in various aspects of daily life to solve the problem of imaging blur caused by factors such as local motion in the image.

[0003] In related technologies, image parameters can be optimized by multi-frame fusion through algorithm processing, and multiple pixels can be used to complete the task that originally required one pixel to achieve the purpose of offsetting image blur and optimizing image quality. However, these methods have the problem of ghosting caused by image fusion. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image fusion method, an image fusion device and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image fusion method, comprising:

[0006] Determine a plurality of image frames to be fused; determine a reference image in the plurality of image frames, and determine a motion state of a non-reference image in the plurality of image frames relative to the reference image, wherein the non-reference image is an image in the plurality of image frames that is different from the reference image, and in response to the non-reference image including a first image, replace the first image with the reference image, and perform image fusion based on the replaced reference image, wherein the first image is a non-reference image whose motion state is not a stationary state.

[0007] In one implementation, the image fusion method further includes: in response to the non-reference image including a second image, performing image fusion based on the second image, wherein the second image is a non-reference image whose motion state is a static state.

[0008] In one implementation, performing image fusion based on the replaced reference image includes:

[0009] A first weight is determined, where the first weight is a weight corresponding to the fusion of the first image; and based on the first weight, the replaced reference image is fused.

[0010] In one implementation manner, determining the motion state of the non-reference image in the multiple frames of images relative to the reference image includes: dividing the reference image and the non-reference image in the multiple frames of images into blocks based on the same block size; determining, for each image block in the non-reference image, the motion state of the image block at the same position relative to the reference image; the first image is the first image block in the non-reference image whose motion state is non-stationary; replacing the first image with the reference image, and performing image fusion based on the replaced reference image, including: replacing the first image block with a first reference image block, and performing image fusion based on the replaced first reference image block, the first reference image block being an image block at the same position as the first image block.

[0011] In one implementation, the determining of the multiple frames of images to be fused includes: determining the multiple frames of images having filtering strengths within a preset filtering strength range as the multiple frames of images to be fused.

[0012] According to a second aspect of an embodiment of the present disclosure, there is provided an image fusion device, including:

[0013] An acquisition unit is used to determine a plurality of frames of images to be fused; a fusion unit is used to determine a reference image in the plurality of frames of images, and determine a motion state of a non-reference image in the plurality of frames of images relative to the reference image, wherein the non-reference image is an image in the plurality of frames of images different from the reference image, and in response to the non-reference image including a first image, the first image is replaced with the reference image, and image fusion is performed based on the replaced reference image, wherein the first image is a non-reference image whose motion state is not a stationary state.

[0014] In one implementation, the fusion unit is further configured to: in response to the non-reference image including a second image, perform image fusion based on the second image, where the second image is a non-reference image whose motion state is a static state.

[0015] In one implementation, the fusion unit performs image fusion on the replaced reference image in the following manner: determining a first weight, where the first weight is a weight corresponding to fusion of the first image; and fusing the replaced reference image based on the first weight.

[0016] In one implementation, the fusion unit determines the motion state of the non-reference image in the multiple frames of images relative to the reference image in the following manner:

[0017] The method comprises the steps of: dividing the benchmark image and the non-benchmark images in the multiple frames into blocks based on the same block size; determining the motion state of the image block at the same position relative to the benchmark image for each image block in the non-benchmark image; the first image is the first image block in the non-benchmark image whose motion state is not a stationary state; replacing the first image with the benchmark image, and performing image fusion based on the replaced benchmark image, including: replacing the first image block with a first benchmark image block, and performing image fusion based on the replaced first benchmark image block, wherein the first benchmark image block is an image block at the same position as the first image block.

[0018] In one implementation, the acquisition unit determines the multiple frames of images to be fused in the following manner: multiple frames of images with filtering strengths within a preset filtering strength range are determined as the multiple frames of images to be fused.

[0019] According to a third aspect of an embodiment of the present disclosure, there is provided an image fusion device, including:

[0020] A processor; a memory for storing processor executable instructions; wherein the processor is configured to: execute the image fusion method described in the first aspect or any one of the embodiments of the first aspect.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to execute the image fusion method described in the first aspect or any one of the embodiments of the first aspect.

[0022] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects: determining a reference image from multiple frames of images, and determining the motion state of a non-reference image relative to the reference image. In the process of fusing multiple frames of images, replacing a non-reference image whose motion state is a non-stationary state with a reference image can improve the problem of ghosting generated by fusing non-stationary state images.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] Figure 1 The figure is a flowchart of image fusion according to an exemplary embodiment.

[0026] Figure 2The figure is a flow chart of an image fusion method according to an exemplary embodiment.

[0027] Figure 3 The present invention is a flowchart showing a method of determining a motion state of a non-reference image relative to a reference image in multiple frames of images according to an exemplary embodiment.

[0028] Figure 4 The figure is a schematic diagram of image block division of multiple frame image blocks according to an exemplary embodiment of the present disclosure.

[0029] Figure 5 The figure is a block diagram of an image fusion device according to an exemplary embodiment.

[0030] Figure 6 It is a block diagram of a device for image fusion according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] The image fusion method provided by the embodiments of the present disclosure can be applied to image processing scenarios of multi-frame noise reduction (MFNR). For example, the embodiments of the present disclosure can be applied to image processing devices (such as terminals) including image acquisition devices such as cameras to improve the image signal-to-noise ratio through multi-frame noise reduction, so that the acquired images have richer details and less noise. For example, it can be a scenario where the camera in the terminal takes pictures or records videos.

[0033] In the related art, image fusion is performed by frame - by - frame superposition to obtain an optimized image. For example, by fusing the pixel points at the same positions of multiple frames of images, multiple pixel points jointly complete the task that originally only one pixel point could complete, and thereby obtain the clearest pixel point at each position, and finally restore the repaired image. Multi - frame noise reduction (mfnr) is to perform motion detection on the obtained original image file, and perform image fusion according to the motion detection result to reduce noise and improve the signal - to - noise ratio. For example, in one solution, the frame with the highest clarity is selected from the obtained multiple frames of images as the reference image (also called the anchor frame). And the motion state of the other frames except the reference frame in the multiple frames of images is detected. Among them, the motion state detection process is equivalent to a filtering process. According to the filter strength, the image can be divided into a local motion image part, a full - motion image part, and a part that cannot be completely determined as a local motion image part. For example, for the image part where 0 < filter strength < 63, it can be understood as a part that cannot be completely determined as a local motion image part.

[0034] In the related art, the processing method for noise reduction of the local motion image part in multiple frames of images is based on the motion detection result, and image fusion is performed according to the degree of local motion. For the part that can be completely determined as a local motion image part, no image fusion is performed, so no ghosting will occur. However, for the part that cannot be completely determined as a local motion image part, the anchor frame is used as the clearest reference image for image fusion with other frame images. This kind of image fusion method may produce ghosting, and the effect of image fusion between the anchor frame as the clearest frame and other frames is worse than using the anchor to replace these frames for image fusion.

[0035] In view of this, the embodiments of the present disclosure provide an image fusion method. In this image fusion method, when performing image fusion, for the part that cannot be completely determined as a local motion image part, the reference image is used to replace the non - stationary frame image in the part that cannot be completely determined as a local motion image part, so as to improve the ghosting problem generated by the non - stationary frame image in the process of image fusion of multiple frames of images.

[0036] Figure 1 is a flowchart of an image fusion shown according to an exemplary embodiment. As Figure 1 shown, the image fusion method provided by the embodiments of the present disclosure can be applied to a terminal, and includes the following steps.

[0037] In step S11, multiple frames of images to be subjected to image fusion are determined.

[0038] The multiple frames of images to be fused in the embodiments of the present disclosure may be, for example, continuous frame images captured of a moving object by setting an exposure time, or may be multiple frames of images that have been captured and can be fused.

[0039] Among them, in an implementation scheme of an embodiment of the present disclosure, the multi-frame images required for image fusion can be determined based on actual needs. Among them, on the one hand, the multi-frame images required for image fusion can be multi-frame images that need to be synthesized based on the photography needs. On the other hand, the multi-frame images required for image fusion can be multi-frame images of a video that needs to be fused based on the video playback needs. Among them, the image after fusion needs to have an image gain value. Therefore, in the embodiment of the present disclosure, the number of multi-frame images under different gain values ​​can be reasonably determined according to the gain value.

[0040] In the disclosed embodiment, the automatic exposure (Automatic Exposure, AE) parameters of the image acquisition device for image capture can be reasonably set to capture multiple frames of images for synthesizing the desired picture. For example, the exposure time, shutter speed, camera sensitivity and other parameters can be adjusted through the automatic exposure setting in the camera to obtain multiple frames of images that meet the gain value. The image acquisition device with the above-mentioned automatic exposure parameters is controlled to take pictures after the focus is completed and the AE convergence is completed, wherein the scene of the picture is a scene with a moving object. For example, an original image file (RAW image) containing meta information such as brightness value and exposure value can be captured.

[0041] The obtained n images may be any images that can be fused. It is understandable that the obtained n images may be images captured based on a continuous time window or images captured based on a non-continuous time window, and the meta information contained in the original image file may also be any brightness value and / or exposure value within the capture range.

[0042] It can be further understood that the multiple frames of images to be subjected to image fusion in the embodiments of the present disclosure can be understood as images that require image fusion processing.

[0043] In one implementation, the disclosed embodiment may determine multiple frames of images with filtering strengths within a preset filtering strength range as multiple frames of images to be fused, for example, images that cannot be completely determined as local motion image parts in conventional techniques.

[0044] In one example, in the embodiments of the present disclosure, based on the mfnr algorithm, the purpose of screening the motion state of photos can also be achieved by adjusting the filtering intensity. For example, the algorithm determines that an image with a filtering intensity less than 0 is a completely moving image, an image with a filtering intensity in the range of 0 to 63 is a non-static image, and an image with a filtering intensity greater than 63 is a completely static image. And based on the images that are completely determined to be locally moving, image fusion is not performed, and image fusion is performed on the images that cannot be completely determined to be locally moving. Among them, in the embodiments of the present disclosure, multiple frames of images with a filtering intensity within a preset filtering intensity range are determined as multiple frames of images to be subjected to image fusion.

[0045] In step S12, a reference image is determined among multiple frames of images, and the motion state of non-reference images in the multiple frames of images relative to the reference image is determined.

[0046] In the embodiments of the present disclosure, when determining a reference image among multiple frames of images, it can be selected based on the sharpness of the image. For example, the image with the highest sharpness among multiple frames of images is selected as the reference image. Among them, sharpness, sometimes also called "clarity", is an index reflecting the clarity of the image plane and the sharpness of the image edge.

[0047] In the embodiments of the present disclosure, after determining the reference image, the motion state of non-reference images in the multiple frames of images to be subjected to image fusion relative to the reference image can be determined. Among them, the non-reference image can be understood as other frame images in the multiple frames of images to be subjected to image fusion except the reference image. Or the non-reference image can also be understood as an image different from the reference image in the multiple frames of images to be subjected to image fusion.

[0048] In the embodiments of the present disclosure, when determining the motion state of a non-reference image, the motion state of the non-reference image in the multiple frames of images to be subjected to image fusion can be divided into a static state and a non-static state. For example, in the embodiments of the present disclosure, the motion state of an image with 0 < filter strength < 63 can be set to a non-static state. The motion state of an image with filter strength ≥ 63 is set to a static state.

[0049] In the embodiments of the present disclosure, for the sake of convenience of description, a non-reference image with a non-static motion state is called a first image, and a non-reference image with a static motion state is called a second image.

[0050] In other words, the motion states included in the non-reference images in the multiple frames of images to be subjected to image fusion include a static state and / or a non-static state. The non-reference image is an image different from the reference image in the multiple frames of images, and includes a first image (a non-reference image with a non-static motion state) and a second image (a non-reference image with a static motion state).

[0051] In step S13 , in response to the non-reference image including the first image, the first image is replaced with the reference image, and image fusion is performed based on the replaced reference image.

[0052] The image fusion method provided by the embodiment of the present disclosure performs image fusion after replacing the first image in multiple frames of images to be fused with a reference image, thereby improving the ghosting problem caused by non-static frame images in the multiple frames during the image fusion process.

[0053] In the image fusion method provided by the embodiment of the present disclosure, in response to the non-reference image including the second image, the second image is directly fused.

[0054] Figure 2 FIG. 1 is a flow chart of an image fusion method according to an exemplary embodiment. Figure 2 As shown, the implementation process of step S21, step S22 and step S23 in the image fusion method provided in the embodiment of the present disclosure is the same as Figure 1 The execution method shown is similar and will not be repeated here.

[0055] In step S24, in response to the non-reference image including the second image, image fusion is performed based on the second image.

[0056] Among them, since the second image is a non-reference image whose motion state is a static state, when image fusion is performed, the non-reference image in a static state will not produce a ghost problem.

[0057] The following disclosed embodiments illustrate the image fusion method involved in the above embodiments in combination with practical applications.

[0058] In one implementation, when performing image fusion on images in multiple frames, a normalized weight value may be assigned to each frame in the multiple frames, and when the images are fused, a weight corresponding to the image is assigned to the fused image. In one example, the multiple frames include n frames, and the weights corresponding to the n frames are N1, N2...Nn, respectively. Among them, the sum of N1, N2...Nn is 1. When performing image fusion, the first frame is assigned a weight value of N1, and the second frame is assigned a weight value of N2 for image fusion. The frame obtained by fusing the first frame with the second frame is then fused with the third frame assigned a weight value of N3. ...and so on, until the fusion of the nth frame is completed.

[0059] In the image fusion method provided by the embodiment of the present disclosure, when the first image is replaced with the reference image and image fusion is performed based on the replaced reference image, the original weight value of the first image can be kept unchanged for image fusion between frames.

[0060] In one example, in the embodiment of the present disclosure, the weight value corresponding to the first image is referred to as the first weight. In the embodiment of the present disclosure, the weight corresponding to the image fusion after the base image replaces the first image is set to the first weight, and the replaced base image is assigned the first weight for image fusion. It can be understood that the replacement of the image does not change the original weight of the frame image at the corresponding position. In one example, it is assumed that the base image is set to A, the weight value corresponding to the base image is set to a, the first image corresponding to a frame in the multi-frame image is set to B, and the weight value corresponding to the first image B is set to b. When image fusion is performed at the position of the first image after being replaced by the base image, the image value A and the weight value b are brought into the algorithm and participate in the image fusion process of the algorithm. Alternatively, it can also be understood that each pixel in the image value A is assigned a weight value of b.

[0061] The image fusion method provided by the embodiment of the present disclosure uses a reference image to replace the first image, participates in image fusion, and keeps the corresponding weights unchanged, which can improve the interference components in the image fusion process and improve the clarity of the restored image.

[0062] In one example, in the embodiment of the present disclosure, the weight value corresponding to the second image is referred to as the second weight. In the embodiment of the present disclosure, the weight corresponding to the second image can be determined as the second weight, and the second image is assigned the second weight for image fusion.

[0063] Among them, in one implementation of an embodiment of the present disclosure, image fusion can be performed on the image to be processed. For example, the multiple frames of images in the embodiment of the present disclosure can be any number of images. In one example, it is assumed that there is a reference image, a first image, and a second image. Based on the inter-frame superposition algorithm detection, the reference image is determined, and the reference frame image is set to a, the weight value corresponding to the reference frame image is set to A, the first image is set to b, the weight value corresponding to the first image is set to B, the second image is set to c, and the weight value corresponding to the first image of this frame is set to C. When performing image fusion, three frames of images can be assigned corresponding weights for image fusion, for example: image fusion is performed using a weight balance of A*a, B*a, and C*c.

[0064] The image fusion method provided by the embodiment of the present disclosure not only retains the result of multi-frame superposition optimization image in the MFR algorithm, but also obtains an image with better optimization effect by screening and replacing non-static images, and improves the problem of ghosting caused by image fusion.

[0065] In the process of image fusion in the embodiment of the present disclosure, each frame of image may be divided into image blocks, and image fusion may be performed based on the image blocks to improve the image fusion processing efficiency.

[0066] When performing image fusion based on image blocks, the embodiment of the present disclosure can determine the motion state between the image blocks, and perform image fusion based on the motion state of the image blocks.

[0067] Figure 3 FIG. 1 is a flow chart showing a method for determining a motion state of a non-reference image relative to a reference image in multiple frames of images according to an exemplary embodiment. Figure 3 As shown, determining the motion state of the non-reference image includes the following steps:

[0068] In step S121 , the reference image and non-reference images in the multiple frames of images are divided into blocks based on the same block size.

[0069] In the embodiment of the present disclosure, the image block of the desired size can be obtained by setting the length and width of the intercepted image block. In one embodiment of the present disclosure, the size and number of image blocks can be set to any value. When the number of blocks is 1, it is equivalent to that the image has no blocks. The smaller the image block, the finer the fusion of pixels in the processed image block can be, and the better the effect achieved.

[0070] In one example, in an embodiment of the present disclosure, the reference image and non-reference images in the multiple-frame image are divided into blocks based on the same block size.

[0071] Figure 4 The figure is a schematic diagram of image block division of multiple frame image blocks according to an exemplary embodiment of the present disclosure.

[0072] See also Figure 4 As shown, in the embodiment of the present disclosure, each frame of multiple frames of images can be segmented into image blocks using the same image block size. For example, Figure 4 In the embodiment, each of the first frame, the second frame, ... the Nth frame is divided into m*n image blocks. The image blocks included in each frame have the same size at the same position.

[0073] It can be understood that the divided image blocks in the embodiment of the present disclosure may be image blocks of any shape, any number and / or any size, and the embodiment of the present disclosure is not limited to this.

[0074] In step S122, for each image block in the non-reference image, the motion state of the image block at the same position relative to the reference image is determined respectively.

[0075] The first image is a first image block in the non-reference image whose motion state is not a stationary state.

[0076] In one implementation of the disclosed embodiment, the same motion state label may be assigned to all pixels in each image block based on the motion state of each image block.

[0077] It is understandable that the motion state of the image involved in the embodiments of the present disclosure may be the motion state of multiple image blocks including image segmentation. The motion states of different image blocks in the same image may be the same or different.

[0078] In step S123, the first image block is replaced with a first reference image block, and image fusion is performed based on the replaced first reference image block, where the first reference image block is an image block at the same position as the first image block.

[0079] In the embodiment of the present disclosure, when image fusion is performed, image fusion is performed on image blocks at the same position. Figure 4 As shown, assuming that the image block at position A1 in the first frame is the reference image block, when image fusion is performed, image fusion is performed on the image block at position A1 in each of the first frame, the second frame, ... the Nth frame. In an example, assuming that the image block at position A1 in the second frame is the first image block, the first image block at position A1 in the second frame can be replaced with the reference image block at position A1 in the first frame, and then, according to the weight value of the first image block at position A1 in the second frame, the reference image block at position A1 in the second frame is image fused with the reference image block at position A1 in the first frame.

[0080] The image fusion method provided by the embodiment of the present disclosure determines a reference image in multiple frames of images, and determines the motion state of a non-reference image relative to the reference image. In the process of multi-frame image fusion, the non-reference image whose motion state is a non-static state is replaced by the reference image, which can improve the ghosting problem caused by fusing non-static images.

[0081] Based on the same concept, the embodiment of the present disclosure also provides an image fusion device.

[0082] It is understandable that in order to realize the above functions, the image fusion device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0083] Figure 5 is a block diagram of an image fusion device according to an exemplary embodiment. Figure 5The image fusion device 100 is applied to an image fusion device, such as a mobile terminal including a camera. The image fusion device 100 includes an acquisition unit 101 and a fusion unit 102 .

[0084] The acquisition unit 101 is used to determine a plurality of frames of images to be fused. The fusion unit 102 is used to determine a reference image in the plurality of frames of images, and determine a motion state of a non-reference image in the plurality of frames of images relative to the reference image, wherein the non-reference image is an image in the plurality of frames of images that is different from the reference image, and in response to the non-reference image including a first image, replace the first image with the reference image, and perform image fusion based on the replaced reference image, wherein the first image is a non-reference image whose motion state is not a stationary state.

[0085] In one implementation, the fusion unit 102 is further configured to: in response to the non-reference image including a second image, perform image fusion based on the second image, where the second image is a non-reference image whose motion state is a static state.

[0086] In one implementation, the fusion unit 102 performs image fusion on the replaced reference image in the following manner: determining a first weight, where the first weight is a weight corresponding to the fused first image, and fusing the replaced reference image based on the first weight.

[0087] In one embodiment, the fusion unit 102 determines the motion state of the non-reference image in the multi-frame image relative to the reference image in the following manner: based on the same block size, the reference image and the non-reference image in the multi-frame image are divided into blocks. For each image block in the non-reference image, the motion state of the image block at the same position relative to the reference image is determined respectively. The first image is the first image block in the non-reference image whose motion state is not stationary. The first image is replaced with the reference image, and image fusion is performed based on the replaced reference image, including: replacing the first image block with the first reference image block, and performing image fusion based on the replaced first reference image block, the first reference image block is an image block at the same position as the first image block.

[0088] In one implementation, the acquisition unit 101 determines the multiple frames of images to be fused in the following manner: The multiple frames of images with filtering strengths within a preset filtering strength range are determined as the multiple frames of images to be fused.

[0089] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0090] Figure 62 is a block diagram of a device for image fusion according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0091] Reference Figure 6 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0092] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0093] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0094] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0095] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0096] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0097] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0098] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0099] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0100] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0101] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0102] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0103] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0104] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0105] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0106] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0107] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image fusion method, characterized in that: The image fusion method comprises: Determine a plurality of frames of images to be fused; Determine a reference image in the multiple frames of images, and determine a motion state of a non-reference image in the multiple frames of images relative to the reference image, wherein the non-reference image is an image in the multiple frames of images that is different from the reference image; In response to the non-reference image including a first image, replacing the first image with the reference image, and performing image fusion based on the replaced reference image, wherein the first image is a non-reference image whose motion state is not a stationary state; The step of determining the multiple frames of images to be fused includes: Determining whether the image is a local motion image based on the filtering strength of the image; In response to determining that the image is an image with local motion, determining that the image is a multi-frame image without graphic fusion; In response to the image being unable to be determined as an image with local motion, the image is determined to be a multi-frame image to be subjected to image fusion.

2. The image fusion method according to claim 1, characterized in that: The image fusion method further comprises: In response to the non-reference image including a second image, image fusion is performed based on the second image, where the second image is a non-reference image whose motion state is a static state.

3. The image fusion method according to claim 1 or 2, characterized in that: The image fusion based on the replaced reference image includes: Determine a first weight, where the first weight is a weight corresponding to fusing the first image; Based on the first weight, the replaced reference image is fused.

4. The image fusion method according to claim 1, characterized in that: The determining of the motion state of the non-reference image in the multiple frames of images relative to the reference image includes: Based on the same block size, the reference image and the non-reference images in the multiple frames of images are divided into blocks; For each image block in the non-reference image, determining a motion state of the image block at the same position relative to the reference image; The first image is a first image block in the non-reference image whose motion state is not a stationary state; Replacing the first image with the reference image and performing image fusion based on the replaced reference image includes: The first image block is replaced with a first reference image block, and image fusion is performed based on the replaced first reference image block, where the first reference image block is an image block at the same position as the first image block.

5. The image fusion method according to claim 1, characterized in that: The step of determining the multiple frames of images to be fused includes: The multiple frames of images whose filtering strengths are within a preset filtering strength range are determined as the multiple frames of images to be subjected to image fusion.

6. An image fusion device, characterized in that: The image fusion device comprises: An acquisition unit, used for determining a plurality of frames of images to be fused; a fusion unit, configured to determine a reference image in the multiple image frames, and determine a motion state of a non-reference image in the multiple image frames relative to the reference image, wherein the non-reference image is an image in the multiple image frames that is different from the reference image, and in response to the non-reference image including a first image, replace the first image with the reference image, and perform image fusion based on the replaced reference image, wherein the first image is a non-reference image whose motion state is not a stationary state; The acquisition unit determines the multiple frames of images to be fused in the following manner: Determining whether the image is a local motion image based on the filtering strength of the image; In the case where the image is determined to be an image with local motion, determining the image to be a multi-frame image without graphic fusion; When the image cannot be determined as an image with local motion, the image is determined to be a multi-frame image to be subjected to image fusion.

7. The image fusion device according to claim 6, characterized in that: The fusion unit is also used for: In response to the non-reference image including a second image, image fusion is performed based on the second image, where the second image is a non-reference image whose motion state is a static state.

8. The image fusion device according to claim 6 or 7, characterized in that: The fusion unit performs image fusion on the replaced reference image in the following manner: Determine a first weight, where the first weight is a weight corresponding to fusing the first image; Based on the first weight, the replaced reference image is fused.

9. The image fusion device according to claim 6, characterized in that: The fusion unit determines the motion state of the non-reference image in the multiple frames of images relative to the reference image in the following manner: Based on the same block size, the reference image and the non-reference images in the multiple frames of images are divided into blocks; For each image block in the non-reference image, determining a motion state of the image block at the same position relative to the reference image; The first image is a first image block in the non-reference image whose motion state is not a stationary state; Replacing the first image with the reference image and performing image fusion based on the replaced reference image includes: The first image block is replaced with a first reference image block, and image fusion is performed based on the replaced first reference image block, where the first reference image block is an image block at the same position as the first image block.

10. The image fusion device according to claim 6, characterized in that: The acquisition unit determines the multiple frames of images to be fused in the following manner: The multiple frames of images whose filtering strengths are within a preset filtering strength range are determined as the multiple frames of images to be subjected to image fusion.

11. An image fusion device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the image fusion method described in any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to execute the image fusion method according to any one of claims 1 to 5.

Images