A method and apparatus for multi-frame image fusion

Abstract

Embodiments of the present application disclose a method and device for multi-frame image fusion. The method can include: obtaining N frames of first images, each of the N frames of first images being an M-channel image, the M-channel image corresponding to M different types of features, M and N being positive integers greater than 0; determining M pseudo images corresponding to the M different types of features based on the N frames of first images, and determining a second image based on the M pseudo images; wherein each of the M pseudo images is an N-channel image, the N channel being a channel composed of values of a target channel extracted from the N frames of first images, the target channel being one of the M channels, and different pseudo images corresponding to different target channels. The embodiments of the present application can improve the effect of the multi-frame image fusion scheme, and improve the stability of the quality of the output image after fusion.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method and apparatus for multi-frame image fusion. Background Technology

[0002] Image processing technology is a highly developed field encompassing various methods for analyzing, processing, and interpreting image data through algorithms and computer technology. This field combines computer science, mathematics, and engineering to improve and optimize the quality and information content of images. Multi-frame image fusion, as a branch of image processing, can synthesize multiple images from different sources into a single image, providing richer and more accurate information in the final fused image than the individual images.

[0003] When users take photos in low-light or high dynamic range lighting conditions, the limited light intake and sensor size often result in images with low signal-to-noise ratios and low dynamic range in a single shot. Multi-frame image fusion enhancement methods can be used to combine multiple frames into a single high-quality image with a wide dynamic range. However, the image quality of most current multi-frame image fusion methods is highly dependent on the quality of the reference frame. If the reference frame is of poor quality, such as exhibiting jitter or blurring, instability may occur, leading to a significant reduction in the final quality of the fused multi-frame image. Therefore, a multi-frame image fusion scheme is needed that can improve the fusion effect and address the issue of unstable fused image quality. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of this application is to provide a method and apparatus for multi-frame image fusion, which can improve the effect of multi-frame image fusion scheme and improve the stability of fused image quality.

[0005] In a first aspect, embodiments of this application provide a method for multi-frame image fusion, which may include: acquiring N first images, each of the N first images being an M-channel image, the M-channel image including M different types of features; determining M pseudo-images corresponding to the M different types of features based on the N first images, and determining a second image based on the M pseudo-images; wherein each of the M pseudo-images is an N-channel image, the N channels being channels formed by extracting the values ​​of a target channel from the N first images respectively, the target channel being one of the M channels, and different pseudo-images corresponding to different target channels.

[0006] Existing multi-frame image fusion methods are all designed based on a reference frame. That is, for a multi-frame input image, one frame is selected as the reference frame, and this reference frame is used as the main information source in the multi-frame image fusion process. This makes the fusion effect highly dependent on the quality of the reference frame. For example, when there are quality problems such as jitter or blurring in the reference frame, the fusion effect will be poor. Therefore, the quality of the fused image generated by a multi-frame image fusion scheme based on a reference frame is unstable. To address this technical problem, the embodiments of this application can determine corresponding pseudo-images based on different channel types, and perform image fusion processing based on these pseudo-images. This eliminates the dependence of the multi-frame image fusion scheme on the reference frame, realizes a reference-free image fusion technique, and ensures the stability of the fused image quality. Specifically, in this embodiment, N frames of first images are first acquired. These N frames can be a series of images with the same subject or the same shooting scene (e.g., images obtained by continuous shooting in the same scene). For each frame of the first images, a corresponding pseudo image is determined based on the type of the image channel (e.g., the values ​​of the three different types of channels, RGB, are extracted from each frame of the first image, and the values ​​of the same type of channel are combined into an N-channel pseudo image, forming a total of 3 pseudo images). Image fusion processing is then performed based on the pseudo images (e.g., by manipulating the pseudo images, channels that are beneficial to improving the final fusion quality are given greater weights, and the image features are further manipulated based on the weight information). Finally, the output image after image fusion is determined, which improves the effect of the multi-frame image fusion scheme. In summary, the embodiments of this application can achieve multi-frame image fusion without a reference frame (for example, determining a pseudo image based on the channel type in the multi-frame sample images, operating on the pseudo image, and then fusing the high-quality features in the N frame sample images) to generate a high-quality fused image, thereby avoiding the quality of the fused image from deteriorating as the quality of the reference frame image deteriorates, and improving the stability of the quality of the fused output image.

[0007] In one possible implementation, determining the second image based on the M pseudo-images includes: determining M first feature maps corresponding to the M pseudo-images, where each of the M pseudo-images corresponds to one first feature map; determining M second feature maps corresponding to the M first feature maps based on an attention mechanism, where each of the M first feature maps corresponds to one second feature map; and performing image fusion processing on the M second feature maps to obtain the second image. In this embodiment, after determining the corresponding feature maps based on the pseudo-images, the weight of each feature channel in the feature map can be determined based on a channel attention mechanism, and image fusion processing can be performed on the feature maps to obtain the output image. Specifically, in this embodiment, M first feature maps corresponding to each pseudo-image can be determined first (e.g., feature maps are determined by extracting features from pseudo-images through convolution). Then, an attention mechanism is used to assign greater weights to feature channels that are beneficial to improving the final fusion quality, and M weighted second feature maps are generated. Finally, image fusion processing is performed on the M second feature maps to determine the output image (e.g., a fused feature map can be determined by concatenating the M second feature maps along their channels, and this fused feature map is input into a neural network with gradually decreasing output channels composed of a series of convolutions, and finally, the enhanced fused output image is determined). In summary, compared to current multi-frame image fusion schemes where the quality of the output image is affected by the quality of the reference frame, this embodiment can determine the better-quality features in N frames based on the M pseudo-images (e.g., weighting the feature maps generated based on the M pseudo-images, and performing subsequent multi-frame image fusion based on the channel weights) to generate a high-quality output image, greatly improving the effect of the multi-frame image fusion scheme.

[0008] In one possible implementation, determining the M first feature maps corresponding to the M pseudo-images includes: performing a first feature extraction on the M pseudo-images through a first convolution operation to determine the corresponding M first feature maps, where each of the M first feature maps includes Q first feature channels, and Q is an integer greater than 0. In this embodiment, features can be extracted from the M pseudo-images to determine the M first feature maps, enabling efficient fusion of subsequent multi-frame images. For example, features can be extracted from the pseudo-images through convolution operations to more accurately capture image information and features, thereby providing a richer and more detailed data foundation for subsequent multi-frame image fusion processing and analysis.

[0009] In one possible implementation, determining the M second feature maps corresponding to the M first feature maps based on the attention mechanism includes: for each of the M first feature maps, determining the weight of each of the Q first feature channels in each of the first feature maps; weighting the Q first feature channels based on the weights of the Q first feature channels in each of the M first feature maps to determine the Q second feature channels corresponding to the Q first feature channels in each of the first feature maps, where each of the Q first feature channels corresponds to one second feature channel; and determining the M second feature maps corresponding to the M first feature maps, where each second feature map includes the Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map. In this embodiment, a channel attention mechanism can be used to select channels contributed by different frames in the pseudo-image, assigning greater weight to channels that are beneficial to improving the final fusion quality. Specifically, in this embodiment, the weights of Q first feature channels in each first feature map can be determined through a channel attention mechanism, and these feature channels are weighted to generate Q second feature channels corresponding to the Q first feature channels, thereby forming M second feature maps. Subsequent multi-frame image fusion methods can generate high-quality fused images based on the feature channel weight information obtained in this embodiment. In summary, by assigning greater weights to channels that are beneficial to improving the final fusion quality, this embodiment not only improves the accuracy of feature extraction but also optimizes the effect of multi-frame image fusion. It can more effectively process and analyze information in multi-frame images, thereby achieving higher-quality image fusion in various application scenarios (such as photos taken under low light or high dynamic range lighting conditions).

[0010] In one possible implementation, the image fusion processing of the M second feature maps to obtain the second image includes: fusing the M second feature maps to determine a third feature map, the third feature map including P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; and performing a second convolution operation on the third feature map to obtain the second image. In this embodiment, by performing image fusion processing on the M second feature maps to obtain the final fused output image, efficient fusion and optimization of image features are achieved. Specifically, the third feature map can be determined by fusing the M second feature maps (e.g., by stitching the M second feature maps together on the channels), wherein the third feature map contains P second feature channels (i.e., feature channels in all the second feature maps). Subsequently, a second convolution operation is performed on the third feature map to generate the final second image (e.g., by feeding the third feature map into a neural network with gradually decreasing output channels composed of a series of convolutions, ultimately obtaining a fused and enhanced output image). In summary, this application provides a feature map fusion method that can generate a fused feature map with richer feature information by concatenating M second feature maps on channels. The fused feature map is then subjected to convolution operations (for example, the third feature map is fed into a neural network with gradually decreasing output channels composed of a series of convolutions) to obtain high-quality feature information. Finally, a fused and enhanced output image is generated, which improves the effect and stability of the multi-frame image fusion scheme.

[0011] In one possible implementation, the model is applied to an electronic device, which includes a multi-frame image fusion model. The step of determining M pseudo-images corresponding to M different types of features based on the N first frames and determining a second image based on the M pseudo-images includes: inputting the N first frames into the multi-frame image fusion model and outputting the second image. In this embodiment, N sample images can be input into the multi-frame image fusion model, pseudo-images can be determined based on the channel types in the multi-frame sample images, and operations can be performed on the pseudo-images to determine the features with better quality in the N first frames. Finally, image fusion is performed to determine the fused output image. Further, the multi-frame image fusion model can be a pre-trained machine learning algorithm model (e.g., a convolutional neural network model). Determining the fused output image based on this model avoids the image quality of the fused multi-frame image deteriorating due to instability in the quality of the reference frame images, thereby improving the stability of the fused output image quality.

[0012] In one possible implementation, the multi-frame image fusion model includes a channel recombination sub-model, an attention sub-model, and a multi-layer convolutional sub-model. The step of inputting the N first frames of images into the multi-frame image fusion model and outputting the second image includes: inputting the N first frames of images into the channel recombination sub-model to output M pseudo-images; performing first feature extraction on the M pseudo-images to determine corresponding M first feature maps, each of the M first feature maps including Q first feature channels; inputting the M first feature maps into the attention sub-model to output M second feature maps corresponding to the M first feature maps, each of the second feature maps including Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map; performing image fusion processing on the M second feature maps to obtain the second image, and outputting the second image. In this embodiment, by using the multi-frame image fusion model (including the channel recombination sub-model, attention sub-model, and multi-layer convolutional model), N first frames of images can be effectively processed and fused to generate a high-quality second image. Specifically, firstly, the N frames of the first image can be converted into M pseudo-images using a channel reorganization sub-model, and features are extracted from these M pseudo-images to determine the corresponding M first feature maps, where each feature map contains Q first feature channels. Subsequently, these feature maps can be further optimized using an attention sub-model to generate the corresponding M second feature maps (which can be weighted feature maps). Finally, image fusion processing (e.g., fusing the M second feature maps and inputting them into a convolutional network with gradually decreasing output channels) is used to generate the second image (i.e., the image after fusion). In summary, this embodiment of the application achieves multi-frame image fusion by integrating multiple sub-models (e.g., channel reorganization, attention mechanism, and multi-layer convolution sub-models) in the multi-frame image fusion model, which not only improves the effect of the multi-frame image fusion scheme but also ensures the stability of the quality of the fused output image.

[0013] In one possible implementation, the step of performing image fusion processing on the M second feature maps to obtain the second image and outputting the second image includes: fusing the M second feature maps to determine a third feature map, the third feature map including P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; inputting the P second feature channels of the third feature map into the multilayer convolutional sub-model, and outputting the second image, wherein the second image is the M-channel image, and the second image is obtained based on R second feature channels from the P second feature channels, the R second feature channels being feature channels determined after one or more second feature extractions from the P second feature channels. In this embodiment, the third feature map can be determined by concatenating the M second feature maps along their channels, and features in the third feature map can be further extracted based on the weights of the feature channels to obtain the fused output image. Specifically, in this embodiment, a third feature map is first determined by fusing M second feature maps, where the third feature map contains P second feature channels (i.e., all second feature channels in the P second feature maps). Further, these second feature channels can be input into a multi-layer convolutional sub-model (for example, this multi-layer convolutional sub-model can be a neural network with gradually decreasing output channels composed of a series of convolutions; this neural network performs one or more feature extractions on the third feature map, highlighting important features and weakening unimportant information, so that the feature map focuses more on the feature channel information with higher weights during fusion), generating a second image after image fusion. In summary, in this embodiment, by concatenating the weighted feature maps along their channels and feeding them into a neural network with gradually decreasing output channels composed of multiple convolutions, a fused and enhanced output image with M channels is finally generated. This achieves the fusion of high-quality features from N frames of input images, thereby improving the quality of the image after fusion of multiple frames.

[0014] In one possible implementation, the multi-frame image fusion model is trained by inputting N third-frame images as sample data and using the fused fourth image as a label, or it is trained based on historical sample documents. In this embodiment, the multi-frame image fusion model can be trained on a server by inputting pre-prepared N third-frame images as sample data and using a preset fused and enhanced fourth image as a label. Alternatively, it can be trained based on historical sample documents, which may include the fused and enhanced fourth image and N third-frame images. By analyzing this historical data, the multi-frame image fusion model can learn and train from past events to improve the effectiveness of the multi-frame image fusion scheme. In summary, this embodiment can train the multi-frame image fusion model by inputting N third-frame images as sample data or by analyzing historical events. This allows the model to effectively identify and distinguish between high-quality and low-quality training results, enhancing the adaptability and accuracy of the multi-frame image fusion model when processing real-world data, obtaining higher-quality fused images, and further improving the effectiveness of the multi-frame image fusion scheme based on this model.

[0015] In one possible implementation, the N third frames are determined by data degradation and / or sequential adjustment of preset sample images. Existing multi-frame image fusion schemes can obtain N training images by data degradation of a single sample image, where the reference frame image is aligned with the FOV of the preset sample image. That is, one frame is selected from the multi-frame training images as the reference frame, and this reference frame is used as the primary information source in the multi-frame image fusion process. This results in the fusion effect being highly dependent on the quality of the reference frame. For example, when quality issues such as jitter or blurring occur in the reference frame, the fusion effect will be poor. Therefore, the quality of the fused image generated by a multi-frame image fusion scheme based on the reference frame is unstable. To address this technical problem, in this embodiment, the order of N training images can be randomly shuffled before being input into a multi-frame image fusion model (e.g., a convolutional neural network model). This prevents the model from assuming the position of the image aligned with the FOV of the sample image (i.e., it cannot determine the reference frame), thereby training the model to select high-quality frame channels (e.g., generating pseudo-images based on N images, manipulating the pseudo-images, determining the weighted feature map corresponding to the pseudo-images through an attention mechanism, and finally inputting the weighted feature map into a multi-layer convolutional sub-model with gradually decreasing output channels composed of a series of convolutions). In subsequent multi-frame image fusion applications, high-quality images are generated based on this multi-frame image fusion model.

[0016] In one possible implementation, the M-channel image is an RGB channel type image. In this embodiment, the M-channel image in the multi-frame image fusion scheme can be an image of RGB channels. Since RGB channel images have wide applications and high universality in color representation, using this type of image can ensure the integrity and accuracy of color information during the fusion process, and helps to preserve and enhance color details in the image, thereby improving the visual quality of the final fused image.

[0017] Secondly, embodiments of this application provide a multi-frame image fusion apparatus, which may include:

[0018] The first image acquisition unit acquires N frames of first images. Each frame of the N frames of first images is an M-channel image. The M-channel image corresponds to M different types of features, where M and N are both positive integers greater than 0.

[0019] The second image determination unit determines M pseudo images corresponding to the M different types of features based on the N frames of the first image, and determines the second image based on the M pseudo images;

[0020] Each of the M pseudo-images is an N-channel image. The N channels are channels formed by combining the values ​​of the target channel extracted from the N frames of the first image. The target channel is one of the M channels, and the target channels corresponding to different pseudo-images are different.

[0021] Existing multi-frame image fusion methods are all designed based on a reference frame. That is, for a multi-frame input image, one frame is selected as the reference frame, and this reference frame is used as the main information source in the multi-frame image fusion process. This makes the fusion effect highly dependent on the quality of the reference frame. For example, when there are quality problems such as jitter or blurring in the reference frame, the fusion effect will be poor. Therefore, the quality of the fused image generated by a multi-frame image fusion scheme based on a reference frame is unstable. To address this technical problem, the embodiments of this application can determine corresponding pseudo-images based on different channel types, and perform image fusion processing based on these pseudo-images. This eliminates the dependence of the multi-frame image fusion scheme on the reference frame, realizes a reference-free image fusion technique, and ensures the stability of the fused image quality. Specifically, in this embodiment, N frames of first images are first acquired. These N frames can be a series of images with the same subject or the same shooting scene (e.g., images obtained by continuous shooting in the same scene). For each frame of the first images, a corresponding pseudo image is determined based on the type of the image channel (e.g., the values ​​of the three different types of channels, RGB, are extracted from each frame of the first image, and the values ​​of the same type of channel are combined into an N-channel pseudo image, forming a total of 3 pseudo images). Image fusion processing is then performed based on the pseudo images (e.g., by manipulating the pseudo images, channels that are beneficial to improving the final fusion quality are given greater weights, and the image features are further manipulated based on the weight information). Finally, the output image after image fusion is determined, which improves the effect of the multi-frame image fusion scheme. In summary, the embodiments of this application can achieve multi-frame image fusion without a reference frame (for example, determining a pseudo image based on the channel type in the multi-frame sample images, operating on the pseudo image, and then fusing the high-quality features in the N frame sample images) to generate a high-quality fused image, thereby avoiding the quality of the fused image from deteriorating as the quality of the reference frame image deteriorates, and improving the stability of the quality of the fused output image.

[0022] In one possible implementation, the second image determining unit is specifically used for:

[0023] M first feature maps corresponding to the M pseudo images are determined, wherein each of the M pseudo images corresponds to one first feature map;

[0024] Based on the attention mechanism, M second feature maps are determined corresponding to the M first feature maps, and each of the M first feature maps corresponds to one second feature map;

[0025] The second image is obtained by performing image fusion processing on the M second feature maps.

[0026] In one possible implementation, the second image determining unit is specifically used for:

[0027] The first feature is extracted from the M pseudo-images by the first convolution operation to determine the corresponding M first feature maps. Each of the M first feature maps includes Q first feature channels, where Q is an integer greater than 0.

[0028] In one possible implementation, the second image determining unit is specifically used for:

[0029] For each of the M first feature maps, determine the weight of each of the Q first feature channels in each first feature map;

[0030] Based on the weights of the Q first feature channels of each first feature map, the Q first feature channels are weighted to determine the Q second feature channels corresponding to the Q first feature channels of each first feature map, and each of the Q first feature channels corresponds to one second feature channel;

[0031] The M first feature maps are determined to correspond to the M second feature maps, and each second feature map includes Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map.

[0032] In one possible implementation, the second image determining unit is specifically used for:

[0033] The M second feature maps are fused to determine a third feature map, which includes P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps.

[0034] The second convolution operation is performed on the third feature map to obtain the second image.

[0035] In one possible implementation, it is applied to an electronic device, which includes a multi-frame image fusion model;

[0036] The second image determination unit is specifically used for:

[0037] The first N frames of images are input into the multi-frame image fusion model, and the second image is output.

[0038] In one possible implementation, the multi-frame image fusion model includes a channel recombination sub-model, an attention sub-model, and a multi-layer convolutional model; the second image determination unit is specifically used for:

[0039] The N frames of the first image are input into the channel recombination sub-model, and the M pseudo images are output.

[0040] First feature extraction is performed on the M pseudo images to determine the corresponding M first feature maps, each of the M first feature maps including Q first feature channels;

[0041] The M first feature maps are input into the attention sub-model, and the M second feature maps corresponding to the M first feature maps are output. Each second feature map includes Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map.

[0042] The M second feature maps are subjected to image fusion processing to obtain the second image, and the second image is output.

[0043] In one possible implementation, the second image determining unit is specifically used for:

[0044] The M second feature maps are fused to determine a third feature map, which includes P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps.

[0045] The P second feature channels of the third feature map are input into the multilayer convolutional sub-model to output the second image, wherein the second image is the M-channel image, and the second image is obtained by converting R second feature channels from the P second feature channels. The R second feature channels are feature channels determined after the P second feature channels have undergone one or more second feature extractions.

[0046] In one possible implementation, the multi-frame image fusion model is trained by inputting N frames of third images as sample data and using the fused fourth image as a label, or it is trained based on historical sample documents.

[0047] In one possible implementation, the N-frame third image is determined by data degradation and / or sequential adjustment of a preset sample image.

[0048] In one possible implementation, the M-channel image is an RGB channel type image.

[0049] Thirdly, embodiments of this application provide a computer storage medium for storing computer software instructions used in an apparatus for multi-frame image fusion provided in the second aspect above, which includes a program designed to execute the aspects described above.

[0050] Fourthly, embodiments of this application provide a computer program including instructions that, when executed by a computer, enable the computer to perform the processes executed in the multi-frame image fusion apparatus described in the second aspect. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0052] Figure 1 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0053] Figure 2 This is a schematic diagram of the software structure of the electronic device provided in the embodiments of this application.

[0054] Figure 3A This is a schematic flowchart of a multi-frame image fusion method provided in an embodiment of this application.

[0055] Figure 3B This is an example diagram of an N-frame first image provided in the embodiments of this application.

[0056] Figure 3C This is a schematic diagram of a pseudo-image provided in an embodiment of this application.

[0057] Figure 4A This is a schematic diagram of another multi-frame image fusion method provided in the embodiments of this application.

[0058] Figure 4B This is a schematic diagram of a first feature map provided in an embodiment of this application.

[0059] Figure 4C This is a schematic diagram of a second feature map provided in an embodiment of this application.

[0060] Figure 4D This is a schematic diagram of a third feature map provided in an embodiment of this application.

[0061] Figure 4E This is a schematic diagram of an image fusion processing method provided in an embodiment of this application.

[0062] Figure 4F This is a schematic diagram of an input image provided in an embodiment of this application.

[0063] Figure 4G This is a schematic diagram of an output image provided in an embodiment of this application.

[0064] Figure 5A This is a schematic diagram of a multiple convolution operation provided in an embodiment of this application.

[0065] Figure 5B This is a photographic illustration provided in an embodiment of this application.

[0066] Figure 5C This is a schematic diagram of a determined output image provided in an embodiment of this application.

[0067] Figure 6 This is a schematic diagram of a multi-frame image fusion model training module provided in an embodiment of this application. Detailed Implementation

[0068] The embodiments of this application will now be described with reference to the accompanying drawings.

[0069] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0070] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0071] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. For example, a component may communicate via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0072] First, some of the terms used in this application will be explained to facilitate understanding by those skilled in the art.

[0073] (1) Signal-to-noise ratio (SNR) refers to the ratio between the strength of the useful signal and the strength of the background noise. This concept is widely used in electronics, communication engineering, data processing, and other fields to evaluate the clarity and reliability of information during signal transmission. A high SNR means that the useful information portion of the signal is much larger than the noise portion, resulting in a clearer signal and less error. Conversely, a low SNR means a high noise level, which may affect the effective interpretation of the signal. SNR is usually calculated using a logarithmic scale and expressed in decibels (dB). In practical applications, improving the SNR is a key way to improve system performance, especially in communication systems, audio processing, and medical imaging.

[0074] (2) Reference Frame: A coordinate system used to describe and analyze the motion and position of an object. It is the foundation for understanding and calculating the motion of an object because the motion of an object is described relative to a specific reference point or reference frame. The motion state of the same object may differ in different reference frames.

[0075] (3) Artifacts refer to structures or features in an image that do not actually exist. They are not part of the original scene but are generated during image acquisition, processing, or transmission. Artifacts can be caused by a variety of reasons, including limitations of the equipment, data processing errors, and signal transmission interference.

[0076] (4) Convolutional Neural Network (CNN) is a type of neural network in deep learning specifically designed for processing data with a grid structure (such as images). It consists of one or more convolutional layers that extract features by performing a sliding window operation on the input data through filters. CNNs are widely used in image and video recognition, image classification, medical image analysis, and other fields.

[0077] (5) Channel Attention is a concept in CNNs that refers to the degree of importance the network places on different channels (such as the red, green, and blue channels in an RGB image) when processing images. It is an application of the attention mechanism in CNNs, aiming to enhance the network's response to important features of specific channels while suppressing unimportant features. This mechanism helps the network learn features in images more effectively, thereby improving the model's performance and accuracy.

[0078] (6) Deep Learning Network refers to a highly complex and hierarchical machine learning model built based on artificial neural networks. It is a key technology in fields such as artificial intelligence, computer vision, and natural language processing, and is applied to various scenarios such as image recognition, speech processing, and predictive analytics. The construction process of a deep learning network typically includes steps such as multi-layer neuron configuration, weight training, feature learning, and model optimization, aiming to solve complex pattern recognition and data analysis problems by mimicking the way the human brain processes information.

[0079] (7) A feature map is the output of a layer in a convolutional neural network (CNN). When an image is input into a CNN, each convolutional layer processes the image (or the feature map of the previous layer) using its filters (or kernels), generating new feature maps. These feature maps are essentially representations of the image after being processed by specific filters.

[0080] (8) In sample images, FOV (Field of View) refers to the physical space or scene range that an image can cover or observe. In photography and videography, FOV usually refers to the field of view that a lens can capture, which depends on the lens's focal length and the size of the camera's sensor. A wide-angle lens has a larger FOV and can capture a wider scene; while a telephoto lens has a smaller FOV and is usually used to photograph distant objects.

[0081] (9) RAW image: refers to the original image acquired by the image sensor without any processing. RAW image retains the original data of each pixel obtained from the sensor, including brightness, color and other relevant information.

[0082] (10) Channel value: refers to the numerical value of the color channels in a digital image. In color images, three basic color channels, red (R), green (G) and blue (B), are usually used to represent the color of each pixel. Each channel has a numerical value that represents the color intensity or brightness of that channel.

[0083] First, the specific technical problem to be solved in this application is analyzed and proposed. In the prior art, techniques for multi-frame image fusion include the following solutions:

[0084] Solution: A method for multi-frame image fusion based on reference frames, which may include the following steps 1 and 2.

[0085] Step 1: Obtain training data by performing noise addition and displacement operations on a sample image (Ground Truth).

[0086] Step 2: Select one frame from the N training images and align it with the FOV of the sample image as a reference frame.

[0087] The above scheme is currently mainly applicable when the reference frame quality is relatively good, but it also has the following drawbacks:

[0088] Disadvantage 1: Over-reliance on reference frame quality. Current multi-frame image fusion methods heavily depend on the quality of the reference frame. That is, from multiple input images, only one frame is selected as the reference frame, and this reference frame is used as the primary source of information during the fusion process. This makes the fusion effect highly dependent on the quality of the reference frame. If the reference frame has problems, such as blurriness, jitter, or improper exposure, the quality of the fused image will be greatly reduced. Due to this excessive reliance on the quality of the reference frame during multi-frame image fusion, processing low-quality reference frames may lead to distortion and blurred details in the final fused image, thus affecting the stability and visual effect of the final image.

[0089] Disadvantage 2: Poor anti-interference capability. The reliance on reference frames in multi-frame image fusion methods limits their application in complex or changing environments. For example, in situations with unstable lighting, fast movement, or unsatisfactory shooting conditions, reference frames are easily interfered with, leading to errors during the fusion process. This method exhibits significant performance degradation when handling dynamic scenes or unstable environments, making it difficult to adapt to varying shooting conditions, thus limiting its versatility and reliability in practical applications.

[0090] Disadvantage 3: Insufficient flexibility and adaptability. Current multi-frame image fusion methods rely too heavily on fixed reference frames, which limits their applicability to different scenarios and conditions. When ideal reference frames are unavailable or difficult to obtain, the application scope of this method is limited. Furthermore, when the environment changes or objects in the image move, the reference frame may no longer be valid, requiring the selection or generation of new reference frames, increasing processing complexity and time costs, and reducing the method's flexibility and adaptability.

[0091] To address the issue that current multi-frame image fusion schemes fail to meet actual business needs and to achieve the goal of satisfying user and actual business requirements, considering the shortcomings of existing technologies, the technical problem that this application aims to solve includes one or more of the following three aspects:

[0092] 1. Reduce dependence on reference frames (Disadvantage 1). Current multi-frame image fusion methods rely heavily on the quality of reference frames, resulting in a significant drop in fusion quality when the reference frame is of poor quality. Therefore, it is necessary to develop a new multi-frame image fusion method that can maintain good fusion results even when the reference frame quality is not ideal, or to develop entirely new algorithms that no longer rely on a single reference frame or do not rely on reference frames at all, but can comprehensively utilize information from multiple frames to improve the quality and detail preservation of the final fused image.

[0093] 2. Improve anti-interference capability (Disadvantage 2). In dynamic or complex environments, reference frames are easily affected by interference, leading to instability in the fusion process. To address this issue, this application requires a multi-frame image fusion technique that can effectively resist environmental interference. This may include enhancing the algorithm's adaptability to factors such as changes in lighting and motion blur, ensuring stable fusion results even under less than ideal shooting conditions. Furthermore, this technique should be able to automatically identify and correct interference in the reference frame, such as automatically adjusting exposure or stabilizing camera shake, thereby guaranteeing the quality of the fused image.

[0094] 3. Increase flexibility and adaptability (Disadvantage 3). Current multi-frame image fusion methods have limited applicability in different scenarios. Therefore, it is necessary to develop a more flexible and adaptable multi-frame image fusion method that can work effectively even without an ideal reference frame or even without a reference frame at all. This method should be able to automatically select or generate the optimal reference frame in various environmental conditions and scenarios, or completely abandon the traditional reference frame-dependent framework and instead adopt a more dynamic and adaptive fusion strategy to ensure optimal fusion results under various conditions.

[0095] In summary, existing multi-frame image fusion methods cannot meet the practical business requirements of multi-frame image fusion. Therefore, the multi-frame image fusion method provided in this application is used to solve the above-mentioned technical problems.

[0096] To facilitate understanding of the embodiments of this application, the exemplary electronic devices provided in the embodiments of this application will be described below.

[0097] Please see Figure 1 , Figure 1This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 100 is a smart terminal device, which can be of various types, and this embodiment of the application does not limit its specific type. For example, the terminal device can be a mobile phone, and can also include tablet computers, desktop computers, desktop computers with touch-sensitive surfaces or touch panels, laptop computers, handheld computers, smart screens, wearable devices (such as smartwatches, smart bracelets, etc.), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, in-vehicle systems, smart headphones, game consoles, and can also be Internet of Things (IoT) devices or smart home devices such as smart water heaters, smart lights, smart air conditioners, etc.

[0098] Please see Figure 1 The following is combined with Figure 1 A detailed description of each component of the electronic device 100 is provided below:

[0099] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0100] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0101] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors to optimize the multi-frame image fusion process.

[0102] The controller can serve as the central nervous system and command center of the electronic device 100. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions. By generating operation control signals based on the instruction opcode and timing signals, the controller coordinates the various processing units to control the fetching and execution of instructions, ensuring the efficiency and accuracy of the image fusion process. The GPU can be used to accelerate image processing and fusion algorithms, while the ISP specifically processes image signals to improve the quality of the fused image. The DSP and NPU can be used to execute multi-frame image fusion models to perform convolutional operations such as feature extraction.

[0103] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are being used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of image fusion.

[0104] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0105] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0106] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0107] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0108] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0109] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0110] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0111] The modem processor may include a modulator and a demodulator. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0112] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0113] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0114] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0115] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0116] Electronic device 100 achieves efficient multi-frame image fusion through its ISP, camera 193, video codec, GPU, display screen 194, and application processor.

[0117] The ISP is used to process image data captured by camera 193 during this process. For example, when performing image fusion, the ISP can optimize parameters such as exposure and color temperature of multiple frames to ensure the quality of the fused image. At the same time, the ISP can also perform algorithmic optimization on image noise and brightness to improve the overall quality of the fused image.

[0118] Camera 193 is used to capture multiple frames of still images or videos. This image data is converted into electrical signals by a photosensitive element (such as a CCD or CMOS phototransistor) and then transmitted to the ISP for preliminary processing. The processed digital image signal is output to the DSP for further processing, such as conversion into standard RGB, YUV, or other image signal formats. In multi-frame image fusion applications, electronic device 100 may include multiple cameras 193 to capture images from different angles and conditions, enhancing the fusion effect.

[0119] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0120] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0121] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0122] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0123] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0124] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0125] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0126] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0127] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0128] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0129] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0130] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0131] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100.

[0132] The 180C barometric pressure sensor is used to measure barometric pressure.

[0133] The magnetic sensor 180D includes a Hall sensor.

[0134] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0135] Distance sensor 180F is used to measure distance.

[0136] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode.

[0137] The 180L ambient light sensor is used to detect ambient light intensity.

[0138] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0139] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature processing strategy.

[0140] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0141] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0142] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0143] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to different touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0144] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0145] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0146] The software system of electronic device 100 can adopt a layered architecture. Figure 2 This is a schematic diagram of the software structure of the electronic device 100 provided in the embodiments of this application.

[0147] The layered architecture divides the system into several layers, each with a clear role and division of labor to effectively support the implementation of multi-frame image fusion schemes. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer, driver layer, and hardware layer.

[0148] The application layer may include a series of application packages. In this embodiment, the application package may include a camera, a gallery, etc.

[0149] The application framework layer provides APIs and a programming framework for the application layer. In this embodiment, the application framework layer includes an improved camera access interface specifically designed to support the capture and processing of multi-frame images. For example, this interface can support continuous shooting mode and ensure accurate synchronization and transmission of image data.

[0150] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0151] The camera hardware abstraction layer can provide virtual hardware for camera device 1, camera device 2, or more camera devices. The camera algorithm library may include runtime code and data that implement the shooting methods provided in the embodiments of this application.

[0152] The driver layer is the layer between hardware and software. It includes drivers for various hardware components, such as camera drivers, digital signal processor drivers, and image processor drivers.

[0153] The hardware layer includes sensors, image signal processors (ISPs), digital signal processors (DSPs), and image processors. In multi-frame image fusion applications, there can be multiple sensors to support the capture of images from different angles or at different times, while the ISP, DSP, and image processor work together to process and optimize this image data.

[0154] The camera device driver is used to drive the image sensor of the camera to acquire images and to drive the image signal processor to preprocess the images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0155] The following describes, in conjunction with the above software structure, the software and hardware workflows of the embodiments of this application when taking pictures using the electronic device 100.

[0156] In response to a user's action of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface in the application framework layer to launch the camera application. This then sends a command to the camera device (camera device and / or other camera devices) in the camera hardware abstraction layer to start the camera. The camera hardware abstraction layer forwards this command to the camera device driver in the kernel layer. The camera device driver can then activate the corresponding camera's image sensor and acquire image light signals through the image sensor. One camera device in the camera hardware abstraction layer corresponds to one image sensor in the hardware layer.

[0157] Then, the camera's image sensor can transmit the acquired image light signal to the image signal processor for preprocessing to obtain the image electrical signal (that is, the original image, such as a RAW image), and transmit the original image to the camera hardware abstraction layer through the camera device driver.

[0158] The camera hardware abstraction layer can send the original image to the camera algorithm library. The camera algorithm library stores program code that implements the multi-frame image fusion method provided in the embodiments of this application. Based on the digital signal processor and image processor, the camera algorithm library executes the above code, enabling the electronic device 100 to perform some or all of the steps of the multi-frame image fusion method provided in the embodiments of this application.

[0159] The camera algorithm library can send processed images (such as RGB images) to the camera hardware abstraction layer. The camera hardware abstraction layer can then display the images. Simultaneously, the camera algorithm library can perform various image processing tasks, such as noise reduction, color correction, and contrast adjustment, to improve image quality or perform specific computer vision tasks.

[0160] Optionally, through the interface provided by the camera hardware abstraction layer, the original image or the image processed by the camera algorithm library can be stored in a specific storage unit. Furthermore, in response to the user's operation of opening the gallery application, such as clicking the gallery application icon and then clicking to view the image, the gallery application calls the corresponding interface to access the image data in the storage unit, and then calls the image decoding library to decode the stored image data into an image that can be displayed on the screen for display in the application.

[0161] It should be noted that, in the process of taking pictures using the electronic device 100 in this application embodiment, the original image obtained and the image processed by the camera algorithm library can be a static image obtained based on the camera's image sensor or a dynamic video frame. This application embodiment does not limit this.

[0162] based on Figures 1-2 The provided system architecture, combined with the multi-frame image fusion method provided in the embodiments of this application, specifically analyzes and solves the technical problems raised in this application.

[0163] See also Figure 3A , Figure 3A This is a schematic flowchart of a multi-frame image fusion method provided in an embodiment of this application. This method can be applied to the above-mentioned... Figures 1-2 In the system architecture described above, the one in which Figure 1 The electronic device 100 in the middle can be used to support and perform Figure 3A The method flow shown consists of steps S300-S301. This method may include the following steps S300-S301.

[0164] Step S300: Obtain the first image of N frames.

[0165] Specifically, each of the N first images is an M-channel image, and the M-channel image includes M different types of features. The initial step involves acquiring multiple frames of images. The system can acquire a series of consecutive image frames (i.e., first images). These multiple first images can be a series of images with the same subject or the same shooting scene (e.g., images captured consecutively in the same scene). Each first image is a multi-channel image (e.g., RGB channels), where each channel captures different feature information in the image. In practical applications, especially under low-light conditions or high dynamic range environments, a single-frame image may be affected by insufficient light or sensor limitations, resulting in a low signal-to-noise ratio and low dynamic range. The system can improve image quality through multi-frame image fusion technology. For example, suppose a user acquires N first images by clicking the continuous shooting function on their phone. These images can be a continuously captured image sequence, with each frame capturing different lighting details of the same scene. Each frame is an image with RGB channels. (See also...) Figure 3B , Figure 3B This is an example diagram of an N-frame first image provided in the embodiments of this application, such as... Figure 3B As shown, Figure 3B It can include N images of 256*256 pixels. The information of each pixel includes RGB channels. The value of each type of channel in the RGB channels of each frame can be regarded as a whole. That is, each frame image includes three channels: R, G, and B. The RGB channels of each frame image can be processed separately in the subsequent image fusion processing scheme, and finally a multi-frame fused image is obtained.

[0166] In one possible implementation, the M-channel image is an RGB channel type image. In this embodiment, the M-channel image in the multi-frame image fusion scheme can be an image of RGB channels. Since RGB channel images have wide applications and high universality in color representation, using this type of image can ensure the integrity and accuracy of color information during the fusion process, and helps to preserve and enhance color details in the image, thereby improving the visual quality of the final fused image.

[0167] Step S301: Based on the N frames of the first image, determine M pseudo images corresponding to the M different types of features, and determine the second image based on the M pseudo images.

[0168] Specifically, each of the M pseudo-images is an N-channel image. The N channels are formed by combining the values ​​of the target channels extracted from the N frames of the first image. The target channel is one of the M channels, and different pseudo-images correspond to different target channels. The system first identifies the acquired N frames of the first image and extracts M different types of features. Each feature can correspond to a channel (e.g., RGB channel). Based on the features in the N frames of the image, the system determines M pseudo-images, each of which is an N-channel image. The system can construct the N channels by extracting the values ​​of specific target channels from the N frames of the first image. Each pseudo-image corresponds to a different target channel. For example, assuming the M-channel image is an RGB image and the N frames of the first image are 10 frames, the corresponding M pseudo-images are pseudo-images of the R, G, and B channel types, respectively. (See [link to documentation]). Figure 3C , Figure 3C This is a schematic diagram of a pseudo-image provided in the embodiments of this application. Figure 3C The upper half can include 10 frames of first images, and the lower half can include 3 pseudo-images. The system can determine a pseudo-image by extracting the R, G, and B channel values ​​of each of the 10 first images and combining the values ​​of channels of the same type. Each pseudo-image is a 10-channel image, including channels representing the same type of feature from each frame. For example... Figure 3C As shown, the pseudo-image corresponding to the R channel type can include the values ​​of all R channels in the aforementioned 10 frames of the first image; similarly, the pseudo-images corresponding to the G channel type and the B channel type can also include the values ​​of all G channels and B channels in the aforementioned 10 frames of the first image, respectively. Finally, the system can perform image fusion processing on the aforementioned pseudo-images based on an image fusion algorithm. For example, it can determine the channel with the best quality from each pseudo-image and fuse it into the final output image. This eliminates the need for alignment with the reference frame image during image fusion, avoiding the influence of the quality of the reference frame image on the quality of the fused image. Thus, it ensures the effectiveness of the image fusion scheme and generates a high-quality fused image without relying on the reference frame.

[0169] Optionally, in the above method steps S300-S301, the specific implementation of determining M pseudo-images corresponding to the M different types of features based on the N frames of the first image, and determining the second image based on the M pseudo-images, can be found in [reference needed]. Figure 4A , Figure 4A This is a schematic diagram of another multi-frame image fusion method provided in the embodiments of this application; as shown Figure 4A As shown: It may also include the following steps S400-S402:

[0170] Step S400: Determine the M first feature maps corresponding to the M pseudo images.

[0171] Specifically, each of the M pseudo-images corresponds to a first feature map. In the process of multi-frame image fusion, visual features need to be extracted from the input M pseudo-images (for example, the pseudo-images can be used as input images into a convolutional neural network (CNN) to extract the features of each image) to generate the corresponding M first feature maps.

[0172] In one possible implementation, determining the M first feature maps corresponding to the M pseudo-images includes: performing a first feature extraction on the M pseudo-images through a first convolution operation to determine the corresponding M first feature maps, where each of the M first feature maps includes Q first feature channels, and Q is an integer greater than 0. Specifically, the system first analyzes the M pseudo-images, extracting visual features from the images through a first convolution operation, and generating corresponding M first feature maps, each of which includes Q feature channels. For example, see [link to example]. Figure 4B , Figure 4B This is a schematic diagram of a first feature map provided in an embodiment of this application. Assuming the M-channel image is an RGB image and the N-frame first image consists of 10 frames, the corresponding M pseudo-images are pseudo-images of three channel types: R, G, and B. Each pseudo-image is processed by a convolutional neural network (CNN) containing multiple convolutional kernels to detect and extract various features from the image. After convolution, each pseudo-image is converted into three corresponding first feature maps. These feature maps contain 256 feature channels (the specific number of feature channels depends on the convolutional kernels; the number of feature channels can also be other numbers, and this embodiment does not impose specific limitations). Each feature channel represents different types of feature information in the image, such as edges and textures. The first feature map corresponding to the R-channel pseudo-image can also include the R-channel color information from the 10 frames of the first image, and the first feature maps corresponding to the G-channel and B-channel pseudo-images can also include their corresponding feature information.

[0173] Step S401: Determine the M second feature maps corresponding to the M first feature maps based on the attention mechanism.

[0174] Specifically, each of the M first feature maps corresponds to a second feature map. The system further processes these first feature maps using an attention mechanism-based method. The attention mechanism is a technique that mimics human visual focus on specific regions. The system can identify and enhance features in the first feature maps based on this attention (e.g., determining the weights of feature channels in the feature maps based on a channel attention mechanism), and generate M second feature maps (e.g., these second feature maps can be feature maps after weighted processing of feature channels), with each first feature map corresponding to a second feature map. In the above process, the system can identify and emphasize specific visual elements based on the attention mechanism. For example, in a multi-frame image fusion scheme, the system can select channels contributing from different frames based on the attention mechanism, assigning greater weight to channels that are beneficial to improving the final fusion quality, thereby improving the effect of subsequent image fusion.

[0175] In one possible implementation, determining the M second feature maps corresponding to the M first feature maps based on an attention mechanism includes: for each of the M first feature maps, determining the weight of each of the Q first feature channels in each of the first feature maps; weighting the Q first feature channels based on their weights to determine the Q second feature channels corresponding to the Q first feature channels in each of the M first feature maps, where each of the Q first feature channels corresponds to one second feature channel; and determining the M second feature maps corresponding to the M first feature maps, where each second feature map includes the Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map. In this embodiment, the M second feature maps corresponding to the M first feature maps can be determined using an attention mechanism. Specifically, in this embodiment, the system first processes M first feature maps, which can be corresponding first feature maps generated based on M pseudo-images. Each first feature map includes Q first feature channels. The system can determine the weight of each feature channel through a channel attention mechanism (e.g., these weights can be used to reflect the contribution of the feature channel to the improvement of image fusion quality), and perform weighted processing on these first feature channels (e.g., by multiplying each first feature channel by its corresponding weight) to determine the Q second feature channels corresponding to the Q first feature channels in each first feature map. Each second feature channel contains feature information adjusted by weights, more accurately reflecting the contribution of each feature in the image to the image fusion quality. For example, see [link to relevant documentation]. Figure 4C , Figure 4C This is a schematic diagram of a second feature map provided in an embodiment of this application. Figure 4CThe image fusion process can include three second feature maps corresponding to three first feature maps. Assuming that each first feature map has 256 first feature channels, the weights corresponding to each first feature channel can be determined based on a channel attention mechanism. This channel attention mechanism can be a module in a convolutional neural network, which includes fully connected layers, activation functions, and normalization steps. It is used to calculate the importance score of each feature channel and calculate a weight value for each feature channel based on the importance score. Furthermore, the values ​​in the first feature channels can be multiplied by their corresponding weight values ​​to determine 256 second feature channels corresponding to the 256 first feature channels (for example, in the second feature channels, the features of feature channels that are beneficial to improving the final fusion quality will be enhanced, while the features of feature channels with lower importance will be weakened). Based on the above 256 second feature channels, a weighted feature map is determined. This weighted feature map can better reflect the feature information of image fusion, which not only improves the accuracy of feature extraction but also optimizes the effect of multi-frame image fusion. It can more effectively process and analyze the information in multi-frame images, thereby achieving higher quality image fusion in various application scenarios (such as photos taken under low light or high dynamic range lighting conditions).

[0176] Step S402: Perform image fusion processing on the M second feature maps to obtain the second image.

[0177] Specifically, in this embodiment, the weighted second feature maps determined in step S401 can be fused to generate the final second image. For example, image fusion of the second feature maps combines the information from these feature maps to generate a comprehensive image containing all feature information. This comprehensive image is then processed to generate the final fused image. For instance, M second feature maps can be concatenated across channels to determine a fused feature map. This fused feature map is then input into a neural network consisting of a series of convolutions with progressively fewer output channels, ultimately determining the fused and enhanced output image.

[0178] In one possible implementation, the image fusion processing of the M second feature maps to obtain the second image includes: fusing the M second feature maps to determine a third feature map, the third feature map including P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; and performing a second convolution operation on the third feature map to obtain the second image. Specifically, the system can determine the third feature map by fusing the M second feature maps (e.g., performing channel-wise concatenation on the M second feature maps), wherein the third feature map contains P second feature channels (i.e., feature channels in all the second feature maps), and then performing a second convolution operation on the third feature map to generate the final second image (e.g., feeding the third feature map into a neural network with gradually decreasing output channels composed of a series of convolutions to finally obtain a fused and enhanced output image). For example, see [link to relevant documentation]. Figure 4D , Figure 4D This is a schematic diagram of a third feature map provided in an embodiment of this application, such as... Figure 4D As shown, Figure 4D The image can include three second feature maps and a fused third feature map. Assuming each second feature map is a weighted feature map, a preliminary fused image feature map (i.e., the third feature map) can be obtained by concatenating the three second feature maps along their channels. This third feature map can include all the second feature channels from the three second feature maps. Furthermore, a convolution operation can be performed on the third feature map to generate the fused and enhanced output image. (See [link to documentation]). Figure 4E , Figure 4E This is a schematic diagram of an image fusion processing method provided in an embodiment of this application, such as... Figure 4E As shown, the third feature map can be input into a neural network consisting of a series of convolutions with progressively fewer output channels. Assuming the input channels in this neural network are the feature channels in the third feature map, the number of input channels gradually decreases with each convolutional layer, eventually reducing to a feature map with 256 output channels (the number of channels reduced in each layer and the final number of output channels can also be other numbers; this application does not impose specific limitations). The final result is a 3-channel (i.e., RGB type channels) output image that has undergone any enhancement (the output image can have pixels of 256*256, or other sizes; this application does not impose specific limitations). Furthermore, the above embodiments describe the data parameter level; the following will demonstrate an example of the image display level, which can be found in [reference needed]. Figure 4F , Figure 4F This is a schematic diagram of an input image provided in an embodiment of this application. In this embodiment, the input image can be obtained by... Figure 4FThe multiple input images are subjected to image fusion processing (see the embodiments described in steps S400-S402 for details, which will not be repeated here). Figure 4F These are sample images taken when the scene has a high dynamic range. Figure 4F The images in the image cannot fully reproduce the information in the scene. In this case, a multi-frame image fusion enhancement method can be used to improve the image quality. Figure 4F The images are fused to improve the image quality, resulting in a 3-channel enhanced output image. (See attached image.) Figure 4G , Figure 4G This is a schematic diagram of an output image provided in an embodiment of this application. Figure 4G The output image is enhanced through fusion. This application not only improves the accuracy of feature extraction but also optimizes the effect of multi-frame image fusion, enabling more effective processing and analysis of information in multiple frames. This allows for higher-quality image fusion in various application scenarios (such as photos taken under low-light or high dynamic range lighting conditions).

[0179] Optionally, in the above method steps S300-S301, the specific implementation of determining M pseudo-images corresponding to the M different types of features based on the N frames of the first image, and determining the second image based on the M pseudo-images, can also be implemented based on a machine algorithm model.

[0180] Specifically, determining M pseudo-images corresponding to the M different types of features based on the N first frames of images, and determining the second image based on the M pseudo-images, includes: inputting the N first frames of images into the multi-frame image fusion model and outputting the second image. In this embodiment, N sample images can be input into the multi-frame image fusion model, pseudo-images can be determined based on the channel types in the multi-frame sample images, and operations can be performed on the pseudo-images to determine the frame with better channel quality in the N first frames. Finally, image fusion is performed to determine the fused output image. Furthermore, the multi-frame image fusion model can be a pre-trained machine learning algorithm model (e.g., a convolutional neural network model). Determining the fused output image based on this multi-frame image fusion model avoids the image quality of the fused multi-frame images deteriorating due to instability in the quality of the reference frame images, ensuring the quality of image fusion and improving the stability of the multi-frame image fusion scheme.

[0181] In one possible implementation, the multi-frame image fusion model includes a channel recombination sub-model, an attention sub-model, and a multi-layer convolutional sub-model. The step of inputting the N first frames of images into the multi-frame image fusion model and outputting the second image includes: inputting the N first frames of images into the channel recombination sub-model to output M pseudo-images; performing first feature extraction on the M pseudo-images to determine corresponding M first feature maps, each of the M first feature maps including Q first feature channels; inputting the M first feature maps into the attention sub-model to output M second feature maps corresponding to the M first feature maps, each of the second feature maps including Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map; performing image fusion processing on the M second feature maps to obtain the second image, and outputting the second image. In this embodiment, by using the multi-frame image fusion model (including the channel recombination sub-model, attention sub-model, and multi-layer convolutional model), N first frames of images can be effectively processed and fused to generate a high-quality second image. Specifically, firstly, the N frames of the first image are converted into M pseudo-images using a channel reorganization sub-model. Then, feature extraction is performed on these M pseudo-images to determine the corresponding M first feature maps, each containing Q first feature channels. Subsequently, these feature maps are further optimized using an attention sub-model to generate the corresponding M second feature maps (which can be weighted feature maps). Finally, image fusion processing (e.g., fusing the M second feature maps and inputting them into a convolutional network with progressively fewer output channels) is used to generate the second image (i.e., the image after fusion). In summary, this embodiment of the application achieves multi-frame image fusion by integrating multiple sub-models (e.g., channel reorganization, attention mechanism, and multi-layer convolutional sub-models) in the multi-frame image fusion model, which not only improves the effect of the multi-frame image fusion scheme but also ensures the stability of the quality of the fused output image.

[0182] In one possible implementation, the step of performing image fusion processing on the M second feature maps to obtain the second image and outputting the second image includes: fusing the M second feature maps to determine a third feature map, the third feature map including P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; inputting the P second feature channels of the third feature map into the multilayer convolutional sub-model, and outputting the second image, wherein the second image is the M-channel image, and the second image is obtained based on R second feature channels from the P second feature channels, the R second feature channels being feature channels determined after one or more second feature extractions from the P second feature channels. In this embodiment, the third feature map can be determined by concatenating the channels of the M second feature maps, and features in the third feature map can be further extracted based on the weights of the feature channels to obtain the fused output image. Specifically, in this embodiment, a third feature map is first determined by fusing M second feature maps, where the third feature map contains P second feature channels (i.e., feature channels in all the second feature maps). Then, these second feature channels are input into a multi-layer convolutional sub-model (for example, this multi-layer convolutional sub-model can be a neural network with gradually decreasing output channels composed of a series of convolutions, through which the third feature map is extracted once or multiple times, highlighting features that improve image fusion quality, making the feature map more focused on the feature channel information with higher weights during fusion). Finally, the fused second image is output. In summary, in this embodiment, by concatenating three feature maps weighted based on a channel attention mechanism and feeding them into a neural network with gradually decreasing output channels composed of multiple convolutions, an M-channel fused and enhanced output image is generated. This achieves the selection of high-quality features from N frames of input images for fusion, thereby improving the quality of the image after multi-frame image fusion.

[0183] In one possible implementation, the fused feature maps can be input into a neural network consisting of a series of convolutions with progressively fewer output channels to generate a fused output image. Specifically, the system first extracts features from M pseudo-images to determine the corresponding M first feature maps, as shown in [reference needed]. Figure 5A , Figure 5A This is a schematic diagram of a multiple convolution operation provided in an embodiment of this application, such as... Figure 5AThe convolutional neural network shown can contain multiple convolutional layers, activation functions, and pooling layers to capture visual information from images. Each pseudo-image first passes through a 3x3 convolutional layer, which uses a 3x3 filter to extract spatial features from the image, generating M corresponding first feature maps. Each first feature map can include C first feature channels. These first feature channels can be visual features such as edges, corners, textures, and colors, or feature representations of a part of the image or a specific object. Further, the system can weight the first feature maps using an attention sub-model (for example, the attention sub-model can be a convolutional neural network model based on an attention mechanism, which can assign larger weights to channels that are beneficial to improving the final fusion quality) to generate M weighted second feature maps. Finally, the system can perform image fusion processing on the M second feature maps (for example, the M second feature maps can be concatenated into a third feature map, which is then input into a neural network with gradually decreasing output channels composed of a series of convolutions) to finally obtain the fused second image. For example, see [link to relevant documentation]. Figure 5B , Figure 5B This is a photographic illustration provided in an embodiment of this application. Figure 5B This may include a terminal device 200, which allows users to continuously capture images of the current scene by clicking the capture button 201 to obtain multiple input images. (See also...) Figure 5C , Figure 5C This is a schematic diagram illustrating the determination of an output image provided in an embodiment of this application. Figure 5C The input image may include a first input image 30-A, a second input image 30-B, a third input image 30-C, and an output image 30-D. Assuming that the embodiment of this application performs image fusion processing on three frames of input images, the above three frames of input images can be input into a multi-frame image fusion model. Through pseudo-image conversion, determining the channel weights based on the channel attention mechanism, and inputting the fused feature map into a neural network composed of a series of convolutions with gradually decreasing output channels, the output image 30-D is finally determined. The output image 30-D can be a 3-channel (e.g., RGB channel) fused and enhanced output image.

[0184] Based on the above Figures 3A-5C The embodiments described herein provide a training module for the multi-frame image fusion model applied to the above embodiments. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of a multi-frame image fusion model training module provided in an embodiment of this application. Figure 6 It may include a multi-frame image fusion model training module 600. Wherein:

[0185] Multi-frame image fusion model training module 600: used to perform multi-frame image fusion processing on sample images without reference frames to improve image quality. Specifically, the multi-frame image fusion model training module 600 may include a data degradation unit 2001, a channel recombination unit 2002, a channel weight determination unit 2003, a feature map fusion unit 2004, an output image determination unit 2005, and a training unit 2006; after acquiring sample images, the system can perform data degradation processing on the sample images based on the data degradation unit 2001 (e.g., noise reduction, resolution reduction, etc., to simulate data quality changes that may occur under different image capture conditions) to obtain N frames of training images, and input the N frames of training images into the model for training after randomly shuffling the frame order; further, the system can recombine the above multi-frame training images through the channel recombination unit 2002, determine M pseudo images based on the channel type in each frame of training images (e.g., for RGB images, pseudo images corresponding to the R, G, and B channel types can be determined), and generate initial feature maps corresponding to the M pseudo images; further... Furthermore, the system can also determine the weights of the feature channels corresponding to each frame of training image in the pseudo image based on the channel weight determination unit 2003 (for example, it can be determined by a convolutional network with channel attention), and generate the corresponding M weighted feature maps; further, the system can also perform channel-wise concatenation on the above M weighted feature maps based on the feature map fusion unit 2004 to obtain a feature map after preliminary fusion, and input the fused feature map into a neural network with gradually decreasing output channels composed of a series of convolutions, and obtain a 3-channel fused and enhanced output image in the output image determination unit 2005. Finally, the system performs loss calculation and backpropagation training on the output image based on the training unit 2006. Through step-by-step iteration, the system continuously optimizes the multi-frame image fusion model, so that the loss between the fused image generated by the model and the reference image is minimized, and finally a fused image with high image quality is generated. In summary, the embodiments of this application can shuffle the order of multi-frame training images before inputting them into the model, so that the subsequent multi-frame training image fusion cannot be aligned with the FOV of the sample image, thereby eliminating the influence of the reference frame on the fusion effect of multi-frame training images, and selecting the channel of high-quality frames based on the subsequent processing unit, thus improving the stability of the image fusion scheme.

[0186] In one possible implementation, the multi-frame image fusion model is trained by inputting N third-frame images as sample data and using the fused fourth image as the label, or it is trained based on historical sample documents. In this embodiment, the multi-frame image fusion model can be trained on a server by inputting pre-prepared N third-frame images as sample data and using a preset fused and enhanced fourth image as the label. Alternatively, it can be trained based on historical sample documents, which may include the fused and enhanced fourth image and N third-frame images. By analyzing this historical data, the multi-frame image fusion model can learn and train from past events to improve the effectiveness of the multi-frame image fusion scheme. For example, the system can perform loss calculation and backpropagation training on the fused output image. Specifically, the system first quantifies the prediction error of the final fused image using a predetermined loss function. This loss function can be mean squared error (MSE), cross-entropy loss, or other loss functions suitable for a specific task, used to measure the specific numerical value of the difference between the predicted image and the target image. Further, the system can use the loss value calculated by the loss to guide the update process of the network weights, i.e., the backpropagation process, to reduce the prediction error. The backpropagation process can be implemented based on the backpropagation algorithm, in which the parameters of each convolutional layer, activation function, and other network elements can be adjusted. Furthermore, during the overall training process, by receiving and processing N frames of input image data, and through gradual iteration, the system continuously optimizes the network parameters to minimize the loss between the fused image and the reference image, ultimately generating a fused image with high accuracy. In summary, the embodiments of this application can train a multi-frame image fusion model by inputting N frames of third images as sample data or analyzing historical events, thereby enabling the model to effectively identify and distinguish between high-quality and low-quality training results, enhancing the adaptability and accuracy of the multi-frame image fusion model when processing real data, obtaining higher-quality fused images, and further improving the effect of the multi-frame image fusion scheme based on the multi-frame image fusion model.

[0187] In one possible implementation, the N third frames are determined by data degradation and / or sequential adjustment of preset sample images. Existing multi-frame image fusion schemes can obtain N training images by data degradation of a single sample image, where the reference frame image is aligned with the FOV of the preset sample image. That is, one frame is selected from the multi-frame training images as the reference frame, and this reference frame is used as the primary information source in the multi-frame image fusion process. This results in the fusion effect being highly dependent on the quality of the reference frame. For example, when quality issues such as jitter or blurring occur in the reference frame, the fusion effect will be poor. Therefore, the quality of the fused image generated by a multi-frame image fusion scheme based on the reference frame is unstable. To address this technical problem, in this embodiment, the order of N training images can be randomly shuffled before being input into a multi-frame image fusion model (e.g., a convolutional neural network model). This prevents the model from assuming the position of the image aligned with the FOV of the sample image (i.e., achieving no reference frame), thereby training the model to select high-quality frame channels (e.g., generating pseudo-images based on N images, manipulating the pseudo-images, determining the weighted feature map corresponding to the pseudo-images through an attention mechanism, and finally inputting the weighted feature map into a multi-layer convolutional sub-model with gradually decreasing output channels composed of a series of convolutions). In subsequent multi-frame image fusion applications, high-quality images are generated based on this multi-frame image fusion model.

[0188] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0189] This application also provides a terminal device, which may include a memory and a processor. The memory may be used to store computer programs; the processor may be used to invoke the computer programs in the memory, so that the terminal device executes the methods executed on the terminal device side in any of the above embodiments.

[0190] This application also provides a terminal device, which may include a memory and a processor. The memory may be used to store computer programs; the processor may be used to invoke the computer programs in the memory, so that the terminal device executes the methods executed on the terminal device side in any of the above embodiments.

[0191] This application also provides a chip system, which includes at least one processor for implementing the functions involved on the terminal device side in any of the above embodiments.

[0192] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0193] The chip system can consist of chips or include chips and other discrete components.

[0194] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0195] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0196] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0197] This application also provides a computer program product, which includes a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the method executed on the terminal device side in any of the above embodiments.

[0198] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run, it causes the computer to perform the method executed on the terminal device side in any of the above embodiments.

[0199] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0200] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0201] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0202] In summary, the above description is merely an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.

Claims

1. A method of multi-frame image fusion, characterized in that, The method is applied to an electronic device, and the method comprises: obtaining N continuous first images, each of the N first images being an M-channel image, the M-channel image corresponding to M different types of features, M and N being positive integers greater than 0; sending the N first images to a camera algorithm library in a camera hardware abstraction layer; the camera algorithm library stores a multi-frame image fusion model, the multi-frame image fusion model comprising a channel reorganization sub-model, a convolutional neural network sub-model based on an attention mechanism, and a multi-layer convolution sub-model; extracting features from the N first images by using the channel reorganization sub-model, and outputting M pseudo images; each of the M pseudo images is an N-channel image, the N channel being a channel formed by combining values of a target channel in the N first images, the target channel being one of the M channels, and different pseudo images corresponding to different target channels; performing first feature extraction on the M pseudo images to determine M first feature maps, each of the M first feature maps comprising Q first feature channels; Q is an integer greater than 0; determining weights corresponding to each of the first feature channels in the M first feature maps by using the convolutional neural network sub-model based on the attention mechanism, and performing weighted processing on each of the first feature channels according to the weights, to output M second feature maps, each of the second feature maps comprising Q second feature channels corresponding to the first feature channels; performing channel splicing on the M second feature maps to determine a third feature map, the third feature map comprising P second feature channels, P being a product of M and Q; extracting features from the third feature map by using the multi-layer convolution sub-model, and outputting a second image after image fusion, the second image being the M-channel image, the number of channels of the second image being less than the number of channels of the third feature map.

2. The method of claim 1, wherein, The first feature extraction on the M pseudo images to determine the M first feature maps comprises: performing first feature extraction on the M pseudo images by using a first convolution operation to determine the M first feature maps.

3. The method according to claim 1 or 2, characterized in that, The multi-frame image fusion model is obtained by inputting N third images as sample data and a fourth image after fusion as a label for training, or is obtained based on historical sample documents.

4. The method of claim 3, wherein, The N third images are determined after data degradation and / or sequential adjustment of a preset sample image.

5. The method according to any one of claims 1-4, characterized in that, The M-channel image is an RGB channel type image.

6. An apparatus for multi-frame image fusion, the apparatus comprising: The device comprises: a first image acquisition unit configured to obtain N continuous first images, each of the N first images being an M-channel image, the M-channel image corresponding to M different types of features, M and N being positive integers greater than 0; a second image determination unit configured to: The N frames of first images are sent to a camera algorithm library in a camera hardware abstraction layer; the camera algorithm library stores a multi-frame image fusion model, and the multi-frame image fusion model includes a channel reorganization sub-model, a convolutional neural network sub-model based on an attention mechanism, and a multi-layer convolution sub-model; Features of the N frames of first images are extracted through the channel reorganization sub-model, and M pseudo images are output; each of the M pseudo images is an N-channel image, and the N channels are channels composed of values of target channels extracted from the N frames of first images, and the target channels are different in different pseudo images. First feature maps corresponding to the M pseudo images are determined through first feature extraction on the M pseudo images, and each of the M first feature maps includes Q first feature channels; Q is an integer greater than 0. Weights corresponding to each of the first feature channels in the M first feature maps are determined through the convolutional neural network sub-model based on the attention mechanism, each of the first feature channels is weighted processed according to the weights, and M second feature maps are output, and each of the second feature maps includes Q second feature channels corresponding to the first feature channels. A third feature map is determined by channel splicing the M second feature maps, and the third feature map includes P second feature channels, and P is a product of M and Q. A second image fused from images is output by feature extraction on the third feature map through the multi-layer convolution sub-model, and the second image is the M-channel image, and a channel number of the second image is less than a channel number of the third feature map.

7. The apparatus of claim 6, wherein, The second image determination unit is specifically configured to: First feature maps corresponding to the M pseudo images are determined through first feature extraction on the M pseudo images by a first convolution operation, and each of the M first feature maps includes Q first feature channels, and Q is an integer greater than 0.

8. The apparatus of claim 6 or 7, wherein, The multi-frame image fusion model is obtained by training through inputting N frames of third images as sample data and a fourth image fused from images as a label, or is obtained based on historical sample documents.

9. The apparatus of claim 8, wherein, The N frames of third images are determined after data degradation and / or sequential adjustment on preset sample images.

10. The apparatus of any one of claims 6-9, wherein, The M-channel image is an RGB channel type image.

11. A computer storage medium, characterized in that The computer storage medium stores a computer program, and the computer program is executed by a processor to implement the method in any one of claims 1-5.

12. A computer program product, characterised in that, The computer program product includes instructions, and when the instructions are executed by a computer, the computer executes the method in any one of claims 1-5.

Images