Multi-frame image fusion method and device

Abstract

The embodiment of the invention discloses a multi-frame image fusion method and device, and the method comprises the steps: obtaining N frames of first images, each frame of first image in the N frames of first images being an M-channel image, the M-channel image corresponding to M different types of features, and both M and N being positive integers greater than 0; determining M pseudo images corresponding to the M different types of features based on the N frames of first images, and determining a second image based on the M pseudo images; wherein each pseudo image in the M pseudo images is an N-channel image, the N channels are channels formed by respectively extracting values of target channels in the N frames of first images, the target channels are one of the M channels, and different pseudo images correspond to different target channels. By adopting the embodiment of the invention, the effect of a multi-frame image fusion scheme can be improved, and the stability of the quality of the output image after fusion is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for multi-frame image fusion. Background Art

[0002] The field of image processing technology is a highly developed field that covers various methods of analyzing, processing, and interpreting image data through algorithms and computer technology. This field combines computer science, mathematics, and engineering to improve and optimize the quality and information content of images. Multi-frame image fusion, as a branch of image processing, can synthesize multiple images from different sources into a single image to provide richer and more accurate information in the final fused image than the individual images.

[0003] When users take photos under low-light conditions or lighting conditions with a high dynamic range, due to the limitations of the amount of light entering and the photosensitive device, a single shot will usually result in an image with a relatively low signal-to-noise ratio and dynamic range. At this time, a multi-frame image fusion enhancement method can be used to fuse multiple multi-frame images to obtain a high-quality image with a wide dynamic range. However, the image quality obtained by most current multi-frame image fusion methods is related to the quality of the reference frame. If the quality of the reference frame is poor, for example, there are problems such as jitter and blur in the reference frame, poor stability may occur, resulting in a significant reduction in the quality of the final multi-frame image fusion. Therefore, it is necessary to provide a multi-frame image fusion solution that can improve the effect of image fusion and solve the problem of unstable fused image quality. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present application is to provide a method and device for multi-frame image fusion, which can improve the effect of the multi-frame image fusion solution and improve the stability of the fused image quality.

[0005] In a first aspect, an embodiment of the present application provides a method for multi-frame image fusion, which may include: acquiring N frames of first images, each of the N frames of first images being an M-channel image, and the M-channel image including M different types of features; determining M pseudo-images corresponding to the M different types of features based on the N frames of first images, and determining a second image based on the M pseudo-images; wherein each of the M pseudo-images is an N-channel image, and the N channels are channels formed by respectively extracting the values ​​of target channels in the N frames of first images, the target channel is one of the M channels, and different pseudo-images correspond to different target channels.

[0006] Existing multi-frame image fusion methods are all designed based on reference frames, that is, for multiple frames of input images, one frame is selected as the reference frame in the multi-frame image, and the reference frame is used as the main information source in the multi-frame image fusion process, resulting in the fusion effect being very dependent on the quality of the reference frame. For example, when quality problems such as jitter and blur appear in the reference frame, the fusion effect will be poor. Therefore, the quality of the fused image generated by the multi-frame image fusion scheme based on the reference frame is unstable. In response to this technical problem, in the embodiment of the present application, the corresponding pseudo-image can be determined based on different channel types, and image fusion processing can be implemented based on the pseudo-image to eliminate the dependence of the multi-frame image fusion scheme on the reference frame, realize the image fusion technology without reference frame, and ensure the stability of the quality of the fused image. Specifically, in an embodiment of the present application, N frames of first images are first obtained, and the N frames of first images can be a series of images with the same shooting object or the same shooting scene (for example, images obtained by continuous shooting in the same scene), and a corresponding pseudo image is determined for the type of image channel of each frame in the N frames of first images (for example, the values ​​of three different types of channels, RGB, are extracted from each frame of the first image, and the values ​​of the channels of the same type are combined into an N-channel pseudo image, constituting a total of 3 pseudo images), and image fusion processing is performed based on the above pseudo images (for example, the pseudo images can be operated to give a larger weight to the channels that are conducive to improving the final fusion quality, and the image features are further operated based on the weight information), and finally the output image after image fusion is determined, thereby improving the effect of the multi-frame image fusion scheme. In summary, in the embodiments of the present application, multi-frame image fusion can be achieved without a reference frame (for example, a pseudo image is determined based on the type of channels in the multi-frame sample images, and the pseudo image is operated to fuse the higher quality features in the N-frame sample images) to generate a high-quality fused image, thereby avoiding the quality of the fused image from deteriorating as the quality of the reference frame image deteriorates, and improving the stability of the quality of the fused output image.

[0007] In a possible implementation, determining the second image based on the M pseudo images includes: determining M first feature maps corresponding to the M pseudo images, each of the M pseudo images corresponding to a first feature map; determining M second feature maps corresponding to the M first feature maps based on an attention mechanism, each of the M first feature maps corresponding to a second feature map; performing image fusion processing on the M second feature maps to obtain the second image. In an embodiment of the present application, after determining the corresponding feature map based on the pseudo image, the weight of each feature channel in the feature map can be determined based on the channel attention mechanism, and the feature map can be subjected to image fusion processing to obtain the output image. Specifically, in the embodiment of the present application, the M first feature maps corresponding to each pseudo image can be first determined (for example, the feature map is determined by extracting the features of the pseudo image through convolution), and then the feature channels that are conducive to improving the final fusion quality are given a larger weight through the attention mechanism, and M second feature maps after weighted processing are generated, and the M second feature maps are subjected to image fusion processing to determine the output image (for example, a fused feature map can be determined by splicing the M second feature maps on the channel, and the fused feature map is input into a neural network composed of a series of convolutions with gradually reduced output channels, and finally the fused and enhanced output image is determined). In summary, compared with the current multi-frame image fusion scheme, the quality of the output image will be affected by the quality of the reference frame. In the embodiment of the present application, the better quality features in the N frames of images can be determined based on the M pseudo images (for example, the feature maps generated based on the M pseudo images are weighted, and the subsequent multi-frame image fusion is performed based on the weights of the channels) to generate a high-quality output image, which greatly improves the effect of the multi-frame image fusion scheme.

[0008] In a possible implementation, the determining the M first feature maps corresponding to the M pseudo images includes: performing a first feature extraction on the M pseudo images through a first convolution operation to determine the corresponding M first feature maps, wherein each of the M first feature maps includes Q first feature channels, where Q is an integer greater than 0. In the embodiment of the present application, features can be extracted from the M pseudo images to determine the M first feature maps, so as to achieve efficient fusion of subsequent multi-frame images. For example, features can be extracted from the pseudo images through a convolution operation to more accurately capture the information and features of the image, thereby providing a richer and more detailed data basis for subsequent multi-frame image fusion processing and analysis.

[0009] In a possible implementation, the determination of the M second feature maps corresponding to the M first feature maps based on the attention mechanism includes: for each first feature map in the M first feature maps, determining the weight of each first feature channel in the Q first feature channels of each first feature map; weighting the Q first feature channels based on the weights of the Q first feature channels of each first feature map, determining the Q second feature channels corresponding to the Q first feature channels of each first feature map, each first feature channel in the Q first feature channels corresponding to a second feature channel; determining the M second feature maps corresponding to the M first feature maps, each second feature map including the Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map. In the embodiment of the present application, the channels contributed by different frames in the pseudo image can be selected through the channel attention mechanism, and the channels that are conducive to improving the final fusion quality can be given a larger weight. Specifically, in the embodiment of the present application, the weights of the Q first feature channels in each first feature map can be determined through the channel attention mechanism, and these feature channels are weighted to generate Q second feature channels corresponding to the Q first feature channels, thereby forming M second feature maps. The subsequent multi-frame image fusion method can generate a high-quality fused image based on the weight information of the feature channels obtained in the embodiment of the present application. In summary, in the embodiment of the present application, by giving a larger weight to the channel that is conducive to improving the final fusion quality, not only the accuracy of feature extraction is improved, but also the effect of multi-frame image fusion is optimized, and the information in the multi-frame images can be processed and analyzed more effectively, thereby achieving higher quality image fusion in various application scenarios (such as photos taken under low light conditions or lighting conditions with a high dynamic range, etc.).

[0010] In a possible implementation, the image fusion processing of the M second feature maps to obtain the second image includes: fusing the M second feature maps to determine the third feature map, the third feature map includes P second feature channels, P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; performing a second convolution operation on the third feature map to obtain the second image. In an embodiment of the present application, by performing image fusion processing on the M second feature maps to obtain the final fused output image, efficient fusion and optimization of image features are achieved. Specifically, the third feature map can be determined by fusing the M second feature maps (for example, splicing the M second feature maps on the channel), wherein the third feature map contains P second feature channels (i.e., the feature channels in all the second feature maps), and then the third feature map is subjected to a second convolution operation to generate the final second image (for example, the third feature map is sent to a neural network composed of a series of convolutions with gradually reduced output channels, and finally a fused and enhanced output image is obtained). In summary, a method for feature map fusion is provided in an embodiment of the present application, which can generate a fused feature map with richer feature information by splicing M second feature maps on channels, and perform a convolution operation on the fused feature map (for example, sending the third feature map to a neural network with a gradually decreasing output channel composed of a series of convolutions) to obtain high-quality feature information, and finally generate a fused and enhanced output image, thereby improving the effect and stability of the multi-frame image fusion scheme.

[0011] In a possible implementation, it is applied to an electronic device, and the electronic device includes a multi-frame image fusion model; the determining of the M pseudo images corresponding to the M different types of features based on the N frames of the first image, and the determining of the second image based on the M pseudo images, includes: inputting the N frames of the first image into the multi-frame image fusion model, and outputting the second image. In an embodiment of the present application, by inputting N frames of sample images into the multi-frame image fusion model, the pseudo image can be determined based on the type of channel in the multi-frame sample image, and the pseudo image can be operated to determine the features with better quality in the N frames of the first image, and finally image fusion is performed to determine the fused output image. Furthermore, the multi-frame image fusion model can be a pre-trained machine learning algorithm model (such as a convolutional neural network model), and the fused output image is determined based on the multi-frame image fusion model, thereby avoiding the image quality after the fusion of the multi-frame images from deteriorating due to the instability of the quality of the reference frame image, so as to improve the stability of the quality of the fused output image.

[0012] In a possible implementation, the multi-frame image fusion model includes a channel reorganization submodel, an attention submodel and a multi-layer convolution submodel; the inputting the N frames of the first image into the multi-frame image fusion model and outputting the second image includes: inputting the N frames of the first image into the channel reorganization submodel and outputting the M pseudo images; performing first feature extraction on the M pseudo images to determine the corresponding M first feature maps, each of the M first feature maps including Q first feature channels; inputting the M first feature maps into the attention submodel and outputting the M second feature maps corresponding to the M first feature maps, each of the second feature maps including Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map; performing image fusion processing on the M second feature maps to obtain the second image, and outputting the second image. In the embodiment of the present application, by using a multi-frame image fusion model (including a channel reorganization submodel, an attention submodel and a multi-layer convolution submodel), N frames of the first image can be effectively processed and fused to generate a high-quality second image. Specifically, first, the N frames of first images can be converted into M pseudo images through the channel reorganization sub-model, and the features of the M pseudo images are extracted to determine the corresponding M first feature maps, each of which contains Q first feature channels; then, the attention sub-model can be used to further optimize these feature maps to generate corresponding M second feature maps (the second feature maps can be weighted feature maps); finally, through image fusion processing (for example, the M second feature maps are fused and input into a convolutional network with gradually reduced output channels) to generate a second image (that is, the image after image fusion). In summary, in the embodiments of the present application, multi-frame image fusion is achieved by integrating multiple sub-models in the multi-frame image fusion model (such as channel reorganization, attention mechanism, and multi-layer convolution sub-models), which not only improves the effect of the multi-frame image fusion scheme, but also ensures the stability of the quality of the fused output image.

[0013] In a possible implementation, the image fusion processing of the M second feature maps to obtain the second image and output the second image includes: fusing the M second feature maps to determine a third feature map, the third feature map includes P second feature channels, P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; inputting the P second feature channels of the third feature map into the multi-layer convolution sub-model, and outputting the second image, wherein the second image is the M-channel image, the second image is obtained by converting R second feature channels in the P second feature channels, and the R second feature channels are feature channels determined after one or more second feature extractions of the P second feature channels. In an embodiment of the present application, the third feature map can be determined by splicing the M second feature maps on the channels, and the features in the third feature map can be further extracted based on the weights of the feature channels to obtain a fused output image. Specifically, the embodiment of the present application first determines the third feature map by fusing M second feature maps, wherein the third feature map contains P second feature channels (i.e., all second feature channels in the P second feature maps); further, these second feature channels can be input into a multi-layer convolution sub-model (for example, the multi-layer convolution sub-model can be a neural network with gradually decreasing output channels composed of a series of convolutions, and the third feature map is subjected to one or more feature extractions through the neural network to highlight important features and weaken unimportant information, so that the feature map is more focused on feature channel information with higher weights during fusion), to generate a second image after image fusion. In summary, in the embodiment of the present application, the weighted feature maps can be spliced ​​on the channels and sent to a neural network with gradually decreasing output channels composed of multiple layers of convolutions for processing, and finally an M-channel fused and enhanced output image is generated, which realizes the selection of high-quality features from N frames of input images for fusion, so as to improve the quality of the image after fusion of multiple frames of images.

[0014] In a possible implementation, the multi-frame image fusion model is obtained by inputting N frames of third images as sample data and training the fused fourth image as a label, or training based on historical sample documents. In an embodiment of the present application, the multi-frame image fusion model can be trained in the server by inputting N frames of third images prepared in advance as sample data and using the preset fused and enhanced fourth image as a label, or the multi-frame image fusion model can be trained based on historical sample documents, which may include the fused and enhanced fourth image and N frames of third images. By analyzing these historical data, the multi-frame image fusion model can learn and train from past events to improve the effect of the multi-frame image fusion scheme. In summary, the embodiment of the present application can train the multi-frame image fusion model by inputting N frames of third images as sample data or historical event analysis, so that the model can effectively identify and distinguish high-quality and low-quality training results, enhance the adaptability and accuracy of the multi-frame image fusion model when processing actual data, so as to obtain a higher quality fused image, and further improve the effect of the multi-frame image fusion scheme based on the multi-frame image fusion model.

[0015] In a possible implementation, the N frames of third images are determined after data degradation and / or sequence adjustment of a preset sample image. In the existing multi-frame image fusion scheme, N frames of training images can be obtained by data degradation of a sample image, wherein the reference frame image is aligned with the preset sample image FOV, that is, for the multi-frame training images, one frame is selected as the reference frame in the multi-frame image, and the reference frame is used as the main source of information in the multi-frame image fusion process, resulting in the fusion effect being highly dependent on the quality of the reference frame. For example, when quality problems such as jitter and blur appear in the reference frame, the fusion effect will be poor. Therefore, the quality of the fused image generated by the multi-frame image fusion scheme based on the reference frame is unstable. To address this technical problem, in an embodiment of the present application, the order of N frames of training images can be randomly shuffled before they are input into a multi-frame image fusion model (such as a convolutional neural network model), so that the model cannot assume the position of the image aligned with the sample image FOV (that is, the reference frame cannot be determined), thereby training the model to obtain the ability to select high-quality frame channels (for example, a pseudo image can be generated based on N frames of images, and the pseudo image can be operated on, and the weighted feature map corresponding to the pseudo image can be determined through the attention mechanism, and finally the weighted feature map can be input into a multi-layer convolution sub-model with a gradually decreasing output channel composed of a series of convolutions), and high-quality images can be generated based on the multi-frame image fusion model in the subsequent application of the multi-frame image fusion solution.

[0016] In a possible implementation, the M-channel image is an RGB channel type image. In the embodiment of the present application, the M-channel image in the multi-frame image fusion solution can be an image of three channel types, RGB. Since RGB channel images have a wide range of applications and high universality in color expression, the use of this type of image can ensure the integrity and accuracy of color information during the fusion process, and help retain and enhance the color details in the image, thereby improving the visual quality of the final fused image.

[0017] In a second aspect, an embodiment of the present application provides a device for fusing multiple frames of images, which may include:

[0018] A first image acquisition unit is configured to acquire N frames of first images, wherein each of the N frames of first images is an M-channel image, and the M-channel images correspond to M different types of features, and both M and N are positive integers greater than 0;

[0019] a second image determining unit, which determines M pseudo images corresponding to the M different types of features based on the N frames of first images, and determines a second image based on the M pseudo images;

[0020] Among them, each of the M pseudo images is an N-channel image, the N channels are channels composed of the values ​​of the target channels in the first images of the N frames respectively extracted, the target channel is one of the M channels, and different pseudo images correspond to different target channels.

[0021] Existing multi-frame image fusion methods are all designed based on reference frames, that is, for multiple frames of input images, one frame is selected as the reference frame in the multi-frame image, and the reference frame is used as the main information source in the multi-frame image fusion process, resulting in the fusion effect being very dependent on the quality of the reference frame. For example, when quality problems such as jitter and blur appear in the reference frame, the fusion effect will be poor. Therefore, the quality of the fused image generated by the multi-frame image fusion scheme based on the reference frame is unstable. In response to this technical problem, in the embodiment of the present application, the corresponding pseudo-image can be determined based on different channel types, and image fusion processing can be implemented based on the pseudo-image to eliminate the dependence of the multi-frame image fusion scheme on the reference frame, realize the image fusion technology without reference frame, and ensure the stability of the quality of the fused image. Specifically, in an embodiment of the present application, N frames of first images are first obtained, and the N frames of first images can be a series of images with the same shooting object or the same shooting scene (for example, images obtained by continuous shooting in the same scene), and a corresponding pseudo image is determined for the type of image channel of each frame in the N frames of first images (for example, the values ​​of three different types of channels, RGB, are extracted from each frame of the first image, and the values ​​of the channels of the same type are combined into an N-channel pseudo image, constituting a total of 3 pseudo images), and image fusion processing is performed based on the above pseudo images (for example, the pseudo images can be operated to give a larger weight to the channels that are conducive to improving the final fusion quality, and the image features are further operated based on the weight information), and finally the output image after image fusion is determined, thereby improving the effect of the multi-frame image fusion scheme. In summary, in the embodiments of the present application, multi-frame image fusion can be achieved without a reference frame (for example, a pseudo image is determined based on the type of channels in the multi-frame sample images, and the pseudo image is operated to fuse the higher quality features in the N-frame sample images) to generate a high-quality fused image, thereby avoiding the quality of the fused image from deteriorating as the quality of the reference frame image deteriorates, and improving the stability of the quality of the fused output image.

[0022] In a possible implementation manner, the second image determining unit is specifically configured to:

[0023] Determine M first feature maps corresponding to the M pseudo images, each of the M pseudo images corresponds to a first feature map;

[0024] Determine, based on an attention mechanism, M second feature maps corresponding to the M first feature maps, where each of the M first feature maps corresponds to a second feature map;

[0025] Perform image fusion processing on the M second feature maps to obtain the second image.

[0026] In a possible implementation manner, the second image determining unit is specifically configured to:

[0027] A first feature extraction is performed on the M pseudo images through a first convolution operation to determine the corresponding M first feature maps, each of the M first feature maps includes Q first feature channels, where Q is an integer greater than 0.

[0028] In a possible implementation manner, the second image determining unit is specifically configured to:

[0029] For each of the M first feature maps, determining a weight of each of the Q first feature channels of each of the first feature maps;

[0030] Performing weighted processing on the Q first feature channels based on the weights of the Q first feature channels of each of the first feature maps, determining Q second feature channels corresponding to the Q first feature channels of each of the first feature maps, each first feature channel in the Q first feature channels corresponding to one second feature channel;

[0031] The M second feature maps corresponding to the M first feature maps are determined, each of the second feature maps including Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map.

[0032] In a possible implementation manner, the second image determining unit is specifically configured to:

[0033] The M second feature maps are fused to determine a third feature map, where the third feature map includes P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps;

[0034] Perform a second convolution operation on the third feature map to obtain the second image.

[0035] In a possible implementation, the method is applied to an electronic device, wherein the electronic device includes a multi-frame image fusion model;

[0036] The second image determination unit is specifically configured to:

[0037] The N frames of first images are input into the multi-frame image fusion model, and the second image is output.

[0038] In a possible implementation, the multi-frame image fusion model includes a channel reorganization sub-model, an attention sub-model and a multi-layer convolution sub-model; the second image determination unit is specifically used to:

[0039] Input the N frames of first images into the channel reorganization sub-model, and output the M pseudo images;

[0040] Performing first feature extraction on the M pseudo images to determine corresponding M first feature maps, each of the M first feature maps including Q first feature channels;

[0041] Input the M first feature maps into the attention sub-model, and output M second feature maps corresponding to the M first feature maps, each of the second feature maps including Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map;

[0042] Perform image fusion processing on the M second feature maps to obtain the second image, and output the second image.

[0043] In a possible implementation manner, the second image determining unit is specifically configured to:

[0044] The M second feature maps are fused to determine a third feature map, where the third feature map includes P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps;

[0045] The P second feature channels of the third feature map are input into the multi-layer convolution sub-model, and the second image is output, wherein the second image is the M-channel image, the second image is obtained by converting R second feature channels out of the P second feature channels, and the R second feature channels are feature channels determined after one or more second feature extractions of the P second feature channels.

[0046] In a possible implementation, the multi-frame image fusion model is obtained by inputting N frames of third images as sample data and using the fused fourth image as a label for training, or is obtained by training based on historical sample documents.

[0047] In a possible implementation manner, the N frames of third images are determined after data degradation and / or sequence adjustment are performed on preset sample images.

[0048] In a possible implementation manner, the M-channel image is an RGB channel type image.

[0049] In a third aspect, an embodiment of the present application provides a computer storage medium for storing computer software instructions used in a multi-frame image fusion device provided in the second aspect above, which includes a program designed for executing the above aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer program, which includes instructions. When the computer program is executed by a computer, the computer can execute the process executed in the device for multi-frame image fusion in the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0052] Figure 1 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0053] Figure 2 It is a schematic diagram of the software structure of the electronic device provided in the embodiment of the present application.

[0054] Figure 3A It is a flowchart of a method for multi-frame image fusion provided in an embodiment of the present application.

[0055] Figure 3B This is an example diagram of N first images provided in an embodiment of the present application.

[0056] Figure 3C A schematic diagram of a pseudo image provided in an embodiment of the present application.

[0057] Figure 4A A schematic diagram of another method for multi-frame image fusion provided in an embodiment of the present application.

[0058] Figure 4B This is a schematic diagram of a first characteristic diagram provided in an embodiment of the present application.

[0059] Figure 4C This is a schematic diagram of a second characteristic diagram provided in an embodiment of the present application.

[0060] Figure 4D This is a schematic diagram of a third characteristic diagram provided in an embodiment of the present application.

[0061] Figure 4E A schematic diagram of image fusion processing provided in an embodiment of the present application.

[0062] Figure 4F A schematic diagram of an input image provided in an embodiment of the present application.

[0063] Figure 4G A schematic diagram of an output image provided in an embodiment of the present application.

[0064] Figure 5A A schematic diagram of multiple convolution operations provided in an embodiment of the present application.

[0065] Figure 5B This is a schematic diagram of a photographing method provided in an embodiment of the present application.

[0066] Figure 5C A schematic diagram of determining an output image provided in an embodiment of the present application.

[0067] Figure 6 It is a schematic diagram of a multi-frame image fusion model training module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0069] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0070] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0071] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. For example, a component may communicate through a local and / or remote process according to a signal having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0072] First, some terms in this application are explained to facilitate understanding by those skilled in the art.

[0073] (1) Signal-to-Noise Ratio (SNR) refers to the ratio between the strength of the useful signal and the strength of the background noise. This concept is widely used in electronics, communication engineering, data processing and other fields to evaluate the clarity and reliability of information during signal transmission. A high signal-to-noise ratio means that the portion of useful information in the signal is much larger than the noise portion, resulting in a clearer signal and smaller errors. Conversely, a low signal-to-noise ratio means a high noise level that may affect the effective interpretation of the signal. The calculation of the signal-to-noise ratio is usually expressed in decibels (dB) using a logarithmic scale. In practical applications, improving the signal-to-noise ratio is a key way to improve system performance, especially in communication systems, audio processing, and medical imaging.

[0074] (2) Reference Frame: A coordinate system used to describe and analyze the motion and position of an object. It is the basis for understanding and calculating the motion of an object, because the motion of an object is described relative to a specific reference point or reference frame. The motion state of the same object may be different in different reference frames.

[0075] (3) Artifacts refer to structures or features in an image that do not exist in reality. They are not part of the original scene but are generated during the image acquisition, processing, or transmission process. Artifacts may be caused by a variety of reasons, including equipment limitations, data processing errors, signal transmission interference, etc.

[0076] (4) Convolutional Neural Network (CNN) is a type of neural network in deep learning that is specifically designed to process grid-structured data (such as images). It consists of one or more convolutional layers that extract features by sliding windows over the input data through filters. CNN is widely used in image and video recognition, image classification, medical image analysis, and other fields.

[0077] (5) Channel Attention is a concept in CNN, which refers to the degree to which the network pays attention to different channels (such as the red, green, and blue channels in RGB images) when processing images. It is an application of the attention mechanism in CNN, which aims to enhance the network's response to important features of a specific channel while suppressing unimportant features. This mechanism can help the network learn features in the image more effectively, thereby improving the performance and accuracy of the model.

[0078] (6) Deep Learning Network refers to a highly complex and hierarchical machine learning model built on artificial neural networks. It is a key technology in the fields of artificial intelligence, computer vision, and natural language processing, and is used in a variety of scenarios such as image recognition, speech processing, and predictive analysis. The construction process of a deep learning network usually includes steps such as multi-layer neuron configuration, weight training, feature learning, and model optimization, aiming to solve complex pattern recognition and data analysis problems by imitating the way the human brain processes information.

[0079] (7) Feature Map refers to the output result of a certain layer in a convolutional neural network (CNN). When an image is input into a CNN, each convolutional layer uses its filter (or convolution kernel) to process the image (or the feature map of the previous layer) to generate a new feature map. These feature maps are essentially representations of the image after being processed by a specific filter.

[0080] (8) The FOV (Field of View) in the sample image refers to the range of physical space or scene that can be covered or observed by the image. In photography and videography, FOV usually refers to the range of viewing angles that a lens can capture, which depends on the focal length of the lens and the size of the camera sensor. A wide-angle lens has a larger FOV and can capture a wider scene; a telephoto lens has a smaller FOV and is usually used to capture distant objects.

[0081] (9) RAW image: refers to the original image acquired by the image sensor without any processing. The RAW image retains the original data of each pixel obtained from the sensor, including brightness, color and other related information.

[0082] (10) Channel value: refers to the numerical value of the color channel in a digital image. In color images, three basic color channels, red (R), green (G), and blue (B), are usually used to represent the color of each pixel. Each channel has a numerical value that represents the color intensity or brightness of the channel.

[0083] First, the technical problems to be solved by this application are analyzed and proposed. In the prior art, the technology of multi-frame image fusion includes the following solutions:

[0084] Solution: A method for performing multi-frame image fusion based on a reference frame may specifically include the following steps 1 and 2.

[0085] Step 1: A sample image (ground truth) is degraded by adding noise and displacement to obtain training data.

[0086] Step 2: Select one frame from the N training images and align it with the sample image FOV as a reference frame.

[0087] The above solution is currently mainly applicable to the case where the reference frame quality is good, but it also has the following disadvantages:

[0088] Disadvantage 1: Over-reliance on the quality of the reference frame. The current multi-frame image fusion method relies heavily on the quality of the reference frame, that is, one of the multiple input frames is selected as the reference frame in the multi-frame image, and the reference frame is used as the main source of information in the multi-frame image fusion process, resulting in the fusion effect being very dependent on the quality of the reference frame. If there are problems with the reference frame, such as blur, jitter, or improper exposure, the quality of the fused image will be greatly reduced. Due to the excessive reliance on the quality of the reference frame in the multi-frame image fusion process, when processing low-quality reference frames, the final fused image may be distorted and the details may be blurred, thus affecting the stability and visual effect of the final image.

[0089] Disadvantage 2: Poor anti-interference ability. In the multi-frame image fusion method, the reliance on the reference frame limits its application in complex or changing environments. For example, in the case of unstable light, fast movement speed or unsatisfactory shooting conditions, the reference frame is easily disturbed, resulting in errors in the fusion process. The performance of this method is significantly reduced when dealing with dynamic scenes or unstable environments, and it is difficult to adapt to changing shooting conditions, thus limiting its extensiveness and reliability in practical applications.

[0090] Disadvantage 3: Lack of flexibility and adaptability. Current multi-frame image fusion methods rely too much on fixed reference frames, which limits their applicability in different scenarios and conditions. In the absence of an ideal reference frame or when the reference frame is not easy to obtain, the scope of application of this method is limited. In addition, when the environment changes or the objects in the image move, the reference frame may no longer be valid, which requires reselecting or generating the reference frame, increasing the complexity and time cost of the processing and reducing the flexibility and adaptability of the method.

[0091] In order to solve the problem that the current multi-frame image fusion solution does not meet the actual business needs and achieve the goal of meeting the needs of users and actual business, taking into account the shortcomings of the existing technology, the technical problems actually to be solved by this application include one or more of the following three aspects:

[0092] 1. Reduce the dependence on the reference frame (disadvantage 1). The current multi-frame image fusion method relies heavily on the quality of the reference frame, which results in a significant decrease in the fusion effect when the reference frame quality is poor. Therefore, it is necessary to develop a new multi-frame image fusion method that should be able to maintain a good fusion effect when the reference frame quality is not ideal, or to develop a completely new algorithm that no longer relies on a single reference frame or does not rely on a reference frame, but can comprehensively utilize the information of multiple frames to improve the quality of the final fused image and retain details.

[0093] 2. Improve anti-interference ability (disadvantage 2). In dynamic or complex environments, the reference frame is easily interfered with, resulting in an unstable fusion process. To solve this problem, it is necessary to provide a multi-frame image fusion technology that can effectively resist environmental interference in the embodiments of the present application. This may include enhancing the algorithm's adaptability to factors such as light changes and motion blur, ensuring that a stable fusion effect can be obtained even under undesirable shooting conditions. In addition, this technology should be able to automatically identify and correct interference in the reference frame, such as automatically adjusting exposure or stabilizing jitter, thereby ensuring the quality of the fused image.

[0094] 3. Increase flexibility and adaptability (disadvantage 3). The current multi-frame image fusion method has limited applicability in different scenarios. To this end, it is necessary to develop a more flexible and adaptable multi-frame image fusion method that can work effectively without an ideal reference frame or even without a reference frame. It can automatically select or generate the optimal reference frame in a variety of different environmental conditions and scenarios, or completely abandon the traditional reference frame-dependent framework and adopt a more dynamic and adaptive fusion strategy to ensure the best fusion effect under various conditions.

[0095] In summary, the existing multi-frame image fusion method cannot meet the actual business needs of multi-frame image fusion. Therefore, the multi-frame image fusion method provided in this application is used to solve the above technical problems.

[0096] To facilitate understanding of the embodiments of the present application, an exemplary electronic device provided in the embodiments of the present application is first introduced below.

[0097] See also Figure 1 , Figure 11 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 100 is a smart terminal device, which can be of various types. The embodiment of the present application does not limit its specific type. For example, the terminal device can be a mobile phone, and can also include a tablet computer, a desktop computer, a desktop computer with a touch-sensitive surface or a touch panel, a laptop computer, a handheld computer, a notebook computer, a smart screen, a wearable device (such as a smart watch, a smart bracelet, etc.), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a car machine, a smart headset, a game console, and can also be an Internet of Things (IOT) device or a smart home device such as a smart water heater, a smart lamp, a smart air conditioner, etc.

[0098] See also Figure 1 , the following combination Figure 1 The components of the electronic device 100 are described in detail:

[0099] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0100] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0101] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors to optimize the multi-frame image fusion process.

[0102] Among them, the controller can be the nerve center and command center of the electronic device 100. The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions. According to the instruction operation code and the timing signal, the operation control signal is generated to coordinate each processing unit to complete the control of fetching and executing instructions, ensuring the efficiency and accuracy of the image fusion process. The GPU can be used to accelerate image processing and fusion algorithms, while the ISP specializes in processing image signals to improve the quality of the fused image. The DSP and NPU can be used to execute a multi-frame image fusion model to achieve convolution operations such as feature extraction.

[0103] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of image fusion.

[0104] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0105] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0106] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0107] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0108] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0109] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0110] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0111] The modem processor may include a modulator and a demodulator. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and may be arranged in the same device as the mobile communication module 150 or other functional modules.

[0112] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0113] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0114] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0115] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0116] The electronic device 100 realizes efficient multi-frame image fusion through its ISP, camera 193, video codec, GPU, display screen 194, application processor and other components.

[0117] In this process, the ISP is used to process the image data captured by the camera 193. For example, when performing image fusion, the ISP can optimize the parameters such as exposure and color temperature of multiple frames of images to ensure the quality of the fused image. At the same time, the ISP can also perform algorithm optimization on the noise and brightness of the image to improve the overall quality of the fused image.

[0118] The camera 193 is used to capture multiple frames of static images or videos. These image data are converted into electrical signals through photosensitive elements (such as CCD or CMOS phototransistors), and then passed to the ISP for preliminary processing. The processed digital image signal is output to the DSP for further processing, such as conversion into image signals in standard RGB, YUV and other formats. In the application of multi-frame image fusion, the electronic device 100 may include multiple cameras 193 to capture images from different angles and conditions to enhance the fusion effect.

[0119] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0120] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0121] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0122] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0123] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0124] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0125] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0126] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0127] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0128] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0129] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0130] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0131] The gyro sensor 180B may be used to determine the motion posture of the electronic device 100 .

[0132] The air pressure sensor 180C is used to measure air pressure.

[0133] The magnetic sensor 180D includes a Hall sensor.

[0134] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0135] The distance sensor 180F is used to measure the distance.

[0136] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.

[0137] The ambient light sensor 180L is used to sense the brightness of ambient light.

[0138] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0139] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy.

[0140] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0141] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0142] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0143] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0144] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0145] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0146] The software system of the electronic device 100 may adopt a layered architecture. Figure 2 It is a schematic diagram of the software structure of the electronic device 100 provided in an embodiment of the present application.

[0147] The layered architecture divides the system into several layers, each with clear roles and division of labor to effectively support the implementation of the multi-frame image fusion solution. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the hardware abstraction layer, the driver layer, and the hardware layer.

[0148] The application layer may include a series of application packages. In the embodiment of the present application, the application package may include a camera, a gallery, etc.

[0149] The application framework layer provides an API and programming framework for the application layer. In this embodiment, the application framework layer includes an improved camera access interface specifically used to support the capture and processing of multiple frames of images. For example, the interface can support continuous shooting mode and ensure accurate synchronization and transmission of image data.

[0150] The hardware abstraction layer is an interface layer between the application framework layer and the driver layer, and provides a virtual hardware platform for the operating system. In the embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0151] The camera hardware abstraction layer may provide virtual hardware for camera device 1, camera device 2 or more camera devices. The camera algorithm library may include operating codes and data for implementing the shooting method provided in the embodiment of the present application.

[0152] The driver layer is a layer between hardware and software. The driver layer includes drivers for various hardware. The driver layer may include camera device drivers, digital signal processor drivers, and image processor drivers.

[0153] The hardware layer includes sensors, image signal processors (ISP), digital signal processors (DSP), image processors, etc. In multi-frame image fusion applications, there can be multiple sensors to support capturing images from different angles or at different time points, while the ISP, DSP, and image processor work together to process and optimize the image data.

[0154] The camera device driver is used to drive the image sensor of the camera to collect images and drive the image signal processor to pre-process the images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0155] In combination with the above software structure, the following exemplarily illustrates the software and hardware workflow of the embodiment of the present application when taking photos by using the electronic device 100.

[0156] In response to the user's operation of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera by calling the camera device (camera device and / or other camera devices) in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the kernel layer. The camera device driver can start the image sensor of the corresponding camera and collect the image light signal through the image sensor. A camera device in the camera hardware abstraction layer corresponds to an image sensor in the hardware layer.

[0157] Then, the camera's image sensor can transmit the collected image light signal to the image signal processor for preprocessing to obtain an image electrical signal (that is, the original image, such as a RAW image), and transmit the above original image to the camera hardware abstraction layer through the camera device driver.

[0158] The camera hardware abstraction layer can send the original image to the camera algorithm library. The camera algorithm library stores the program code for implementing the method for multi-frame image fusion provided in the embodiment of the present application. Based on the digital signal processor and the image processor, the camera algorithm library executes the above code, so that the electronic device 100 can execute some or all steps of the method for multi-frame image fusion provided in the embodiment of the present application.

[0159] The camera algorithm library can send the processed image (such as RGB image) to the camera hardware abstraction layer. Then, the camera hardware abstraction layer can send it for display. At the same time, the camera algorithm library can also perform various image processing tasks, such as noise reduction, color correction, contrast adjustment, etc., to improve image quality or perform specific computer vision tasks.

[0160] Optionally, the above-mentioned original image or the image processed by the camera algorithm library can be stored in a specific storage unit through the interface provided by the camera hardware abstraction layer. Furthermore, in response to the user's operation of opening the gallery application, such as clicking the gallery application icon and then clicking to view the image, the gallery application calls the corresponding interface to access the image data in the storage unit, and then calls the image decoding library to decode the stored image data into an image that can be displayed on the display screen, so as to be displayed in the application.

[0161] It should be noted that in the embodiment of the present application, when taking photos using the electronic device 100, the original image obtained and the image processed by the camera algorithm library can be a static image obtained based on the image sensor of the camera or a dynamic video frame, and the embodiment of the present application does not limit this.

[0162] based on Figure 1-Figure 2 The provided system architecture, combined with the multi-frame image fusion method provided in the embodiments of the present application, specifically analyzes and solves the technical problems raised in the present application.

[0163] See also Figure 3A , Figure 3A This is a schematic diagram of a multi-frame image fusion method provided in an embodiment of the present application. The method can be applied to the above Figure 1-Figure 2 In the system architecture described in Figure 1 The electronic device 100 may be used to support and execute Figure 3A The method flow shown in FIG. 1 includes steps S300 to S301. The method may include the following steps S300 to S301.

[0164] Step S300: Acquire N frames of first images.

[0165] Specifically, each of the N frames of first images is an M-channel image, and the M-channel image includes M different types of features; first, multiple frames of images are collected, and the system can obtain a series of continuous image frames (i.e., first images), and the multiple frames of first images can be a series of images with the same shooting object or the same shooting scene (for example, images obtained by continuous shooting in the same scene); wherein each frame of the first image is a multi-channel image (for example, RGB channels), wherein each channel captures different feature information in the image. In practical applications, especially in low-light conditions or high dynamic range environments, a single-frame image may be affected by insufficient light or sensor limitations, resulting in a low signal-to-noise ratio and dynamic range of the image. The system can improve the image quality through multi-frame image fusion technology. For example, assuming that the user obtains N frames of first images by clicking the continuous shooting function of the mobile phone, these images can be a sequence of continuously captured images, each frame captures different lighting details of the same scene, wherein each frame of the image is an image with RGB channels, see Figure 3B , Figure 3B This is an example of an N-frame first image provided in an embodiment of the present application, such as Figure 3B As shown, Figure 3B It can include N 256*256 pixel images, and the information of each pixel includes RGB channels. The value of each type of channel in the RGB channels in each frame of the picture can be regarded as a whole, that is, each frame of the image includes R, G, and B three channel information. In the subsequent image fusion processing scheme, the RGB channels of each frame of the image can be processed separately, and finally a multi-frame fused image is obtained.

[0166] In a possible implementation, the M-channel image is an RGB channel type image. In the embodiment of the present application, the M-channel image in the multi-frame image fusion solution can be an image of three channel types, RGB. Since RGB channel images have a wide range of applications and high universality in color expression, the use of this type of image can ensure the integrity and accuracy of color information during the fusion process, and help retain and enhance the color details in the image, thereby improving the visual quality of the final fused image.

[0167] Step S301: determining M pseudo images corresponding to the M different types of features based on the N frames of first images, and determining a second image based on the M pseudo images.

[0168] Specifically, each of the M pseudo images is an N-channel image, and the N channels are channels formed by respectively extracting the values ​​of the target channels in the N-frame first images, and the target channel is one of the M channels, and different pseudo images correspond to different target channels. The system first identifies the acquired N-frame first images and extracts M different types of features, each of which may correspond to a channel (e.g., RGB channel), and determines M pseudo images based on the features in the above N-frame images, and each pseudo image is an N-channel image. The system can construct N channels by extracting the values ​​of specific target channels in the N-frame first images, and each pseudo image corresponds to a different target channel. Exemplarily, assuming that the M-channel image is an RGB image, and the N-frame first images are 10-frame first images, then the corresponding M pseudo images are pseudo images of three channel types, namely, R, G, and B, respectively. See Figure 3C , Figure 3C is a schematic diagram of a pseudo image provided in an embodiment of the present application, Figure 3C The upper half may include 10 frames of first images, and the lower half may include 3 pseudo images. The system may determine a pseudo image by respectively extracting the values ​​of the R channel, the G channel, and the B channel of each of the 10 frames of first images, and combining the values ​​of the same type of channels, thereby determining the 3 pseudo images, wherein each pseudo image is a 10-channel image, respectively including channels of the same type of features in each frame of the image; for example, Figure 3C As shown, the pseudo image corresponding to the R channel type may include all R channel values ​​in the above 10 frames of the first image; and similarly, the pseudo image corresponding to the G channel type and the pseudo image corresponding to the B channel type may also include all G channel and B channel values ​​in the above 10 frames of the first image, respectively. Finally, the system may perform image fusion processing on the above pseudo images based on the image fusion algorithm. For example, the channel with the best quality may be determined from each pseudo image and fused into the final output image, so that the image does not need to be aligned with the reference frame image during fusion, thereby avoiding the influence of the quality of the reference frame image on the quality of the fused image, thereby ensuring the effect of the image fusion scheme without relying on the reference frame and generating a high-quality fused image.

[0169] Optionally, in the above method steps S300 to S301, in step S301, M pseudo images corresponding to the M different types of features are determined based on the N frames of the first image, and the second image is determined based on the M pseudo images. For specific implementation, see Figure 4A , Figure 4A Schematic diagram of another method for multi-frame image fusion provided in an embodiment of the present application; Figure 4A As shown: the following steps S400-S402 may also be included:

[0170] Step S400: Determine M first feature maps corresponding to the M pseudo images.

[0171] Specifically, each of the M pseudo images corresponds to a first feature map. In the process of multi-frame image fusion, it is first necessary to extract visual features from the M pseudo images input (for example, the pseudo images can be input as input images into a convolutional neural network (CNN) to extract features of each image), and generate corresponding M first feature maps.

[0172] In a possible implementation, the determining of the M first feature maps corresponding to the M pseudo images includes: performing a first feature extraction on the M pseudo images through a first convolution operation to determine the corresponding M first feature maps, each of the M first feature maps including Q first feature channels, where Q is an integer greater than 0. Specifically, the system first analyzes the M pseudo images, and can extract visual features in the images through a first convolution operation, and generate corresponding M first feature maps, each of which includes Q feature channels. For example, see Figure 4B , Figure 4B A schematic diagram of a first feature map provided in an embodiment of the present application, assuming that the M channel image is an RGB image, and the N frames of the first image are 10 frames of the first image, then the corresponding M pseudo images are pseudo images of three channel types of R, G, and B, and each pseudo image passes through a convolutional neural network (CNN) containing multiple convolution kernels, which is used to detect and extract various features in the image. After the convolution operation, each pseudo image is converted into three corresponding first feature maps, which contain 256 feature channels (the number of the feature channels depends on the convolution kernel, and the number of feature channels can also be other numbers, which is not specifically limited in the embodiment of the present application), each feature channel represents different types of feature information in the image, such as edge, texture and other information, and the first feature map corresponding to the pseudo image of the R channel type can also include the R channel color information in the 10 frames of the first image, and the first feature map corresponding to the pseudo image of the G channel type and the pseudo image of the B channel type can also include the corresponding feature information.

[0173] Step S401: Determine the M second feature maps corresponding to the M first feature maps based on the attention mechanism.

[0174] Specifically, each of the M first feature maps corresponds to a second feature map. The system further processes these first feature maps using a method based on an attention mechanism. The attention mechanism is a technology that imitates human vision to focus on specific areas. The system can identify and strengthen the features in the first feature map based on the attention (for example, determine the weights of the feature channels in the feature map based on the channel attention mechanism), and generate M second feature maps (for example, the second feature map can be a feature map after weighted processing of the feature channels), and each first feature map corresponds to a second feature map. In the above process, the system can identify and emphasize specific visual elements based on the attention mechanism. For example, in a multi-frame image fusion scheme, channels contributing to different frames can be selected based on the attention mechanism, and channels that are conducive to improving the final fusion quality can be given a larger weight, thereby improving the effect of subsequent image fusion.

[0175] In a possible implementation, the determining the M second feature maps corresponding to the M first feature maps based on the attention mechanism includes: for each first feature map in the M first feature maps, determining the weight of each first feature channel in the Q first feature channels of each first feature map; performing weighted processing on the Q first feature channels based on the weights of the Q first feature channels of each first feature map, determining the Q second feature channels corresponding to the Q first feature channels of each first feature map, each first feature channel in the Q first feature channels corresponding to a second feature channel; determining the M second feature maps corresponding to the M first feature maps, each second feature map including the Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map. In the embodiment of the present application, the M second feature maps corresponding to the M first feature maps can be determined by the attention mechanism. Specifically, in an embodiment of the present application, the system first processes M first feature maps, which may be corresponding first feature maps generated based on M pseudo images, and each first feature map includes Q first feature channels. The system may determine the weight of each feature channel through a channel attention mechanism (for example, these weights may be used to reflect the contribution of the feature channel to the improvement of image fusion quality), and perform weighted processing on these first feature channels (for example, this may be achieved by multiplying each first feature channel by its corresponding weight) to determine Q second feature channels corresponding to the Q first feature channels in each first feature map, wherein each second feature channel contains feature information adjusted by weight, which more accurately reflects the contribution of each feature in the image to the image fusion quality. For example, see Figure 4C , Figure 4C is a schematic diagram of a second characteristic diagram provided in an embodiment of the present application, Figure 4CThe 3 second feature maps corresponding to the 3 first feature maps may be included. Assuming that the number of first feature channels in each first feature map is 256, the weight corresponding to each first feature channel may be determined based on a channel attention mechanism. The channel attention mechanism may be a module in a convolutional neural network, which includes a fully connected layer, an activation function, and a normalization step, for calculating the importance score of each feature channel, and calculating a weight value for each feature channel based on the importance score; further, the value in the first feature channel may be multiplied by its corresponding weight value to determine 256 second feature channels corresponding to the 256 first feature channels (for example, in the second feature channel, the features of the feature channels that are conducive to improving the final fusion quality will be enhanced, while the feature channels with lower importance will be weakened), and a weighted feature map is determined based on the above 256 second feature channels. The weighted feature map can better reflect the feature information of image fusion, which not only improves the accuracy of feature extraction, but also optimizes the effect of multi-frame image fusion, and can more effectively process and analyze the information in the multi-frame images, thereby achieving higher quality image fusion in various application scenarios (such as photos taken under low light conditions or light conditions with a high dynamic range).

[0176] Step S402: Perform image fusion processing on the M second feature maps to obtain the second image.

[0177] Specifically, in an embodiment of the present application, the weighted second feature maps determined in the above step S401 may be fused to generate a final second image. Exemplarily, image fusion of the second feature maps may combine the information in these feature maps to generate a comprehensive image containing all feature information, and the comprehensive image may be processed to generate the final fused image. For example, M second feature maps may be spliced ​​on channels to determine a fused feature map, and the fused feature map may be input into a neural network composed of a series of convolutions with gradually decreasing output channels, and finally a fused and enhanced output image may be determined.

[0178] In one possible implementation, the image fusion processing of the M second feature maps to obtain the second image includes: fusing the M second feature maps to determine a third feature map, the third feature map includes P second feature channels, P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; performing a second convolution operation on the third feature map to obtain the second image. Specifically, the system can determine the third feature map by fusing the M second feature maps (for example, performing channel stitching on the M second feature maps), wherein the third feature map contains P second feature channels (i.e., the feature channels in all the second feature maps), and then performing a second convolution operation on the third feature map to generate the final second image (for example, the third feature map is sent to a neural network composed of a series of convolutions with gradually decreasing output channels, and finally a fused and enhanced output image is obtained). For example, see Figure 4D , Figure 4D A third characteristic diagram provided in an embodiment of the present application is shown in FIG. Figure 4D As shown, Figure 4D It may include three second feature maps and a fused third feature map. Assuming that each second feature map is a weighted feature map, a preliminarily fused image feature map (i.e., a third feature map) can be obtained by splicing the three second feature maps on the channels. The third feature map may include all the second feature channels in the above three second feature maps. Furthermore, the above third feature map may be convolved to generate a fused and enhanced output image. See Figure 4E , Figure 4E A schematic diagram of image fusion processing provided in an embodiment of the present application is shown in FIG. Figure 4E As shown, the third feature map can be input into a neural network composed of a series of convolutions and whose output channels are gradually reduced. Assuming that the input channels in the neural network are the feature channels in the third feature map, the input channels will gradually decrease after each layer of convolution, and finally be reduced to a feature map with 256 output channels (the number of channels reduced in each layer and the final number of output channels can also be other numbers, which are not specifically limited in the embodiments of the present application), and finally a 3-channel (i.e., RGB type channel) output image that has been enhanced in any way is obtained (the pixels of the output image can be 256*256 or other sizes, which are not specifically limited in the embodiments of the present application); further, what is described in the above embodiments is the data parameter level, and the following will be demonstrated with an example of the image display level, which can be seen in Figure 4F , Figure 4F is a schematic diagram of an input image provided in an embodiment of the present application. In the embodiment of the present application, Figure 4FThe multiple frames of input images in the image fusion process are processed (for details, please refer to the embodiment described in step S400 to step S402, which will not be described in detail here), Figure 4F These are sample images taken when the dynamic range of the scene is high. Figure 4F The image in the scene cannot completely restore the information in the scene. In this case, the multi-frame image fusion enhancement method can be used to Figure 4F The images in the image are fused to improve the image effect, and finally a 3-channel fused and enhanced output image is obtained. Figure 4G , Figure 4G is a schematic diagram of an output image provided in an embodiment of the present application, Figure 4G The embodiment of the present application not only improves the accuracy of feature extraction, but also optimizes the effect of multi-frame image fusion, and can more effectively process and analyze the information in multi-frame images, thereby achieving higher quality image fusion in various application scenarios (such as photos taken under low light or lighting conditions with a high dynamic range).

[0179] Optionally, in the above method steps S300-S301, the specific implementation of determining M pseudo images corresponding to the M different types of features based on the N frames of first images in step S301, and determining the second image based on the M pseudo images can also be implemented based on a machine algorithm model.

[0180] Specifically, the method of determining the M pseudo images corresponding to the M different types of features based on the N frames of the first image, and determining the second image based on the M pseudo images, includes: inputting the N frames of the first image into the multi-frame image fusion model, and outputting the second image. In an embodiment of the present application, the pseudo images can be determined based on the types of channels in the multi-frame sample images by inputting N frames of sample images into the multi-frame image fusion model, and the pseudo images can be operated to determine the frame with better channel quality in the N frames of the first image, and finally image fusion is performed to determine the fused output image. Furthermore, the multi-frame image fusion model can be a pre-trained machine learning algorithm model (such as a convolutional neural network model), and the fused output image is determined based on the multi-frame image fusion model, thereby avoiding the image quality after the fusion of the multi-frame images from deteriorating due to the instability of the quality of the reference frame image, ensuring the quality of the image fusion, and improving the stability of the multi-frame image fusion solution.

[0181] In a possible implementation, the multi-frame image fusion model includes a channel reorganization submodel, an attention submodel and a multi-layer convolution submodel; the inputting the N frames of the first image into the multi-frame image fusion model and outputting the second image includes: inputting the N frames of the first image into the channel reorganization submodel and outputting the M pseudo images; performing first feature extraction on the M pseudo images to determine the corresponding M first feature maps, each of the M first feature maps including Q first feature channels; inputting the M first feature maps into the attention submodel and outputting the M second feature maps corresponding to the M first feature maps, each of the second feature maps including Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map; performing image fusion processing on the M second feature maps to obtain the second image, and outputting the second image. In the embodiment of the present application, by using the multi-frame image fusion model (including the channel reorganization submodel, the attention submodel and the multi-layer convolution submodel), the N frames of the first image can be effectively processed and fused to generate a high-quality second image. Specifically, first, the N frames of first images can be converted into M pseudo images through the channel reorganization sub-model, and then the features of the M pseudo images are extracted to determine the corresponding M first feature maps, each of which contains Q first feature channels; then, the attention sub-model is used to further optimize these feature maps to generate corresponding M second feature maps (the second feature maps can be weighted feature maps); finally, through image fusion processing (for example, the M second feature maps are fused and input into a convolutional network with gradually reduced output channels) to generate a second image (that is, the image after image fusion). In summary, in the embodiments of the present application, multi-frame image fusion is achieved by integrating multiple sub-models in the multi-frame image fusion model (such as channel reorganization, attention mechanism, and multi-layer convolution sub-models), which not only improves the effect of the multi-frame image fusion scheme, but also ensures the stability of the quality of the fused output image.

[0182] In a possible implementation, the image fusion processing of the M second feature maps to obtain the second image and output the second image includes: fusing the M second feature maps to determine a third feature map, the third feature map includes P second feature channels, P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; inputting the P second feature channels of the third feature map into the multi-layer convolution sub-model, and outputting the second image, wherein the second image is the M-channel image, the second image is obtained by converting R second feature channels in the P second feature channels, and the R second feature channels are feature channels determined after one or more second feature extractions of the P second feature channels. In an embodiment of the present application, the third feature map can be determined by splicing the channels of the M second feature maps, and the features in the third feature map can be further extracted based on the weights of the feature channels to obtain a fused output image. Specifically, the embodiment of the present application first determines the third feature map by fusing M second feature maps, wherein the third feature map contains P second feature channels (i.e., the feature channels in all second feature maps); then, these second feature channels are input into a multi-layer convolution sub-model (for example, the multi-layer convolution sub-model can be a neural network with gradually decreasing output channels composed of a series of convolutions, and the third feature map is subjected to one or more feature extractions through the neural network, highlighting the features that improve the image fusion quality, so that the feature map is more focused on the feature channel information with higher weights during fusion), and finally outputs the second image after image fusion. In summary, in the embodiment of the present application, three feature maps weighted by the channel attention mechanism are spliced ​​on the channels, and sent to a neural network with gradually decreasing output channels composed of multiple layers of convolutions for processing, and finally an M-channel fused and enhanced output image is generated, which realizes the selection of high-quality features from N frames of input images for fusion, so as to improve the quality of the image after fusion of multiple frames of images.

[0183] In one possible implementation, the fused feature map can be input into a neural network composed of a series of convolutions with gradually decreasing output channels to generate a fused output image. Specifically, the system first extracts features from the M pseudo images to determine the corresponding M first feature maps, see Figure 5A , Figure 5A A schematic diagram of multiple convolution operations provided in an embodiment of the present application is shown in FIG. Figure 5AThe convolutional neural network shown in the figure may include multiple convolutional layers, activation functions, and pooling layers to capture visual information from the image. Each pseudo image first passes through a 3x3 convolutional layer, which uses a filter of size 3x3 to extract spatial features from the image, thereby generating corresponding M first feature maps, each of which may include C first feature channels. These first feature channels may be visual features such as edges, corners, textures, colors, etc., or feature representations of a part of the image or a specific object; further, the system may perform weighted processing on the above-mentioned first feature maps through an attention sub-model (for example, the attention sub-model may be a convolutional neural network model based on an attention mechanism, and the attention sub-model may be used to assign a larger weight to the channels that are conducive to improving the final fusion quality) to generate corresponding M weighted second feature maps; finally, the system may perform image fusion processing on the M second feature maps (for example, the M second feature maps may be spliced ​​by channels, fused into a third feature map, and the third feature map may be input into a neural network composed of a series of convolutions with gradually decreasing output channels), and finally obtain a fused second image. For example, see Figure 5B , Figure 5B This is a photographing schematic diagram provided in an embodiment of the present application. Figure 5B The terminal device 200 may be included, and the user can continuously shoot the current scene by clicking the shooting button 201 to obtain multiple frames of input images. Figure 5C , Figure 5C is a schematic diagram of determining an output image provided in an embodiment of the present application, Figure 5C It may include a first input image 30-A, a second input image 30-B, a third input image 30-C and an output image 30-D. Assuming that in the embodiment of the present application, image fusion processing is performed on 3 frames of input images, the above 3 frames of input images can be input into a multi-frame image fusion model, and pseudo-image conversion, channel weights based on a channel attention mechanism are determined, and the fused feature map is input into a neural network composed of a series of convolutions and the output channels are gradually reduced. Finally, the output image 30-D is determined, and the output image 30-D can be a 3-channel (for example, RGB channel) output image that has been fused and enhanced.

[0184] Based on the above Figure 3A-Figure 5C The embodiment described in the present application provides a training module for the multi-frame image fusion model applied in the above embodiment, see Figure 6 , Figure 6 is a schematic diagram of a multi-frame image fusion model training module provided in an embodiment of the present application, Figure 6 The multi-frame image fusion model training module 600 may be included. Among them:

[0185] Multi-frame image fusion model training module 600: used to perform multi-frame image fusion processing on sample images without reference frames to improve image quality. Specifically, the multi-frame image fusion model training module 600 may include a data degradation unit 2001, a channel reorganization unit 2002, a channel weight determination unit 2003, a feature map fusion unit 2004, an output image determination unit 2005 and a training unit 2006; after acquiring the sample image, the system may perform data degradation processing on the sample image based on the data degradation unit 2001 (for example, noise reduction, resolution reduction, etc., to simulate the data quality changes that may occur under different image capture conditions) to obtain N frames of training images, and randomly shuffle the frame order of the N frames of training images and input them into the model for training; further, the system may reorganize the above-mentioned multi-frame training images through the channel reorganization unit 2002, determine M pseudo images based on the type of channels in each frame of training image (for example, for RGB images, pseudo images corresponding to the three channel types of R, G, and B can be determined), and generate initial feature maps corresponding to the M pseudo images; further In the step, the system can also determine the weight of the feature channel corresponding to each frame of the training image in the pseudo image based on the channel weight determination unit 2003 (for example, it can be determined by a convolutional network with channel attention), and generate corresponding M weighted feature maps; further, the system can also splice the above-mentioned M weighted feature maps on the channel based on the feature map fusion unit 2004 to obtain a feature map after preliminary fusion, and input the fused feature map into a neural network with gradually reduced output channels composed of a series of convolutions, and obtain a 3-channel fused and enhanced output image in the output image determination unit 2005, and finally perform loss calculation and back propagation training on the output image based on the training unit 2006. Through step-by-step iteration, the system continuously optimizes the multi-frame image fusion model to minimize the loss between the fused image generated by the model and the reference image, and finally generates a fused image with higher image quality. In summary, the embodiment of the present application can shuffle the order of multiple frames of training images before inputting them into the model, so that the subsequent multiple frames of training images cannot be aligned with the sample image FOV during fusion, thereby eliminating the influence of the reference frame on the fusion effect of the multiple frames of training images, and selecting channels of high-quality frames based on subsequent processing units, thereby improving the stability of the image fusion solution.

[0186] In a possible implementation, the multi-frame image fusion model is obtained by inputting N frames of third images as sample data and training the fused fourth image as a label, or is obtained by training based on historical sample documents. In an embodiment of the present application, the multi-frame image fusion model can be trained in a server by inputting N pre-prepared third images as sample data and using a preset fused and enhanced fourth image as a label, or, the multi-frame image fusion model can be trained based on historical sample documents, which may include the fused and enhanced fourth image and N frames of third images. By analyzing these historical data, the multi-frame image fusion model can learn and train from past events to improve the effect of the multi-frame image fusion solution. Exemplarily, the system can perform Loss loss calculation and back propagation training on the fused output image; specifically, the system can first quantify the prediction error of the final fused image through a predetermined loss function. The loss function can be mean square error (MSE), cross entropy loss or other loss functions suitable for specific tasks, which are used to measure the specific numerical value of the difference between the predicted image and the target image; further, the system can guide the update process of the network weights, that is, the back propagation process, through the loss value obtained by Loss calculation to reduce the prediction error; wherein, the back propagation process can be implemented based on the back propagation algorithm, in which the parameters of each convolutional layer, activation function and other network elements can be adjusted in the back propagation algorithm structure; further, in the overall training process, by receiving N frames of input image data and processing them, the system can continuously optimize the network parameters through step-by-step iterations to minimize the loss between the fused image and the reference image, and finally generate a fused image with higher accuracy. In summary, the embodiment of the present application can train the multi-frame image fusion model by inputting N frames of third images as sample data or historical event analysis, so that the model can effectively identify and distinguish high-quality and low-quality training results, enhance the adaptability and accuracy of the multi-frame image fusion model when processing actual data, so as to obtain a higher quality fused image, and further improve the effect of the multi-frame image fusion solution based on the multi-frame image fusion model.

[0187] In a possible implementation, the N frames of third images are determined after data degradation and / or sequence adjustment of a preset sample image. In the existing multi-frame image fusion scheme, N frames of training images can be obtained by data degradation of a sample image, wherein the reference frame image is aligned with the preset sample image FOV, that is, for the multi-frame training images, one frame is selected as the reference frame in the multi-frame image, and the reference frame is used as the main source of information in the multi-frame image fusion process, resulting in the fusion effect being highly dependent on the quality of the reference frame. For example, when quality problems such as jitter and blur appear in the reference frame, the fusion effect will be poor. Therefore, the quality of the fused image generated by the multi-frame image fusion scheme based on the reference frame is unstable. To address this technical problem, in an embodiment of the present application, the order of N frames of training images can be randomly shuffled before they are input into a multi-frame image fusion model (such as a convolutional neural network model), so that the model cannot assume the position of the image aligned with the sample image FOV (i.e., achieving no reference frame), thereby training the ability to select high-quality frame channels (for example, a pseudo image can be generated based on N frames of images, and the pseudo image can be operated on, and the weighted feature map corresponding to the pseudo image can be determined through the attention mechanism, and finally the weighted feature map can be input into a multi-layer convolution sub-model with a gradually decreasing output channel composed of a series of convolutions), and high-quality images can be generated based on the multi-frame image fusion model in the subsequent application of the multi-frame image fusion solution.

[0188] It should be understood that each step in the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0189] The present application also provides a terminal device, which may include: a memory and a processor. The memory may be used to store a computer program; the processor may be used to call the computer program in the memory, so that the terminal device executes the method executed by the terminal device side in any of the above embodiments.

[0190] The present application also provides a terminal device, which may include: a memory and a processor. The memory may be used to store a computer program; the processor may be used to call the computer program in the memory, so that the terminal device executes the method executed by the terminal device side in any of the above embodiments.

[0191] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the terminal device side in any of the above embodiments.

[0192] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0193] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0194] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0195] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0196] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0197] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method executed by the terminal device side in any of the above embodiments.

[0198] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the terminal device side in any of the above embodiments.

[0199] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0200] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0201] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0202] In short, the above is only an embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of this application shall be included in the protection scope of this application.

Claims

1. A method for multi-frame image fusion, characterized in that: The method comprises: Acquire N frames of first images, each of the N frames of first images is an M-channel image, the M-channel images correspond to M different types of features, and both M and N are positive integers greater than 0; Determine M pseudo images corresponding to the M different types of features based on the N frames of first images, and determine a second image based on the M pseudo images; Among them, each of the M pseudo images is an N-channel image, the N channels are channels composed of the values ​​of the target channels in the first images of the N frames respectively extracted, the target channel is one of the M channels, and different pseudo images correspond to different target channels.

2. The method according to claim 1, characterized in that The determining the second image based on the M pseudo images comprises: Determine M first feature maps corresponding to the M pseudo images, each of the M pseudo images corresponds to a first feature map; Determine, based on an attention mechanism, M second feature maps corresponding to the M first feature maps, where each of the M first feature maps corresponds to a second feature map; Perform image fusion processing on the M second feature maps to obtain the second image.

3. The method according to claim 2, characterized in that The determining of the M first feature maps corresponding to the M pseudo images comprises: A first feature extraction is performed on the M pseudo images through a first convolution operation to determine the corresponding M first feature maps, each of the M first feature maps includes Q first feature channels, where Q is an integer greater than 0.

4. The method according to claim 3, characterized in that The determining, based on the attention mechanism, the M second feature maps corresponding to the M first feature maps comprises: For each of the M first feature maps, determining a weight of each of the Q first feature channels of each of the first feature maps; Performing weighted processing on the Q first feature channels based on the weights of the Q first feature channels of each of the first feature maps, determining Q second feature channels corresponding to the Q first feature channels of each of the first feature maps, each first feature channel in the Q first feature channels corresponding to one second feature channel; The M second feature maps corresponding to the M first feature maps are determined, each of the second feature maps including Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map.

5. The method according to any one of claims 2 to 4, characterized in that: The performing image fusion processing on the M second feature maps to obtain the second image includes: The M second feature maps are fused to determine a third feature map, where the third feature map includes P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; Perform a second convolution operation on the third feature map to obtain the second image.

6. The method according to any one of claims 1 to 5, characterized in that Applied to an electronic device, the electronic device includes a multi-frame image fusion model; the method of determining M pseudo images corresponding to the M different types of features based on the N frames of first images, and determining the second image based on the M pseudo images, includes: The N frames of first images are input into the multi-frame image fusion model, and the second image is output.

7. The method according to claim 6, characterized in that The multi-frame image fusion model includes a channel reorganization sub-model, an attention sub-model and a multi-layer convolution sub-model; the inputting the N frames of first images into the multi-frame image fusion model and outputting the second image includes: Input the N frames of first images into the channel reorganization sub-model, and output the M pseudo images; Performing first feature extraction on the M pseudo images to determine corresponding M first feature maps, each of the M first feature maps including Q first feature channels; Input the M first feature maps into the attention sub-model, and output M second feature maps corresponding to the M first feature maps, each of the second feature maps including Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map; Perform image fusion processing on the M second feature maps to obtain the second image, and output the second image.

8. The method according to claim 7, characterized in that The performing image fusion processing on the M second feature maps to obtain the second image, and outputting the second image, comprises: The M second feature maps are fused to determine a third feature map, where the third feature map includes P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; The P second feature channels of the third feature map are input into the multi-layer convolution sub-model, and the second image is output, wherein the second image is the M-channel image, and the second image is obtained by converting R second feature channels out of the P second feature channels, and the R second feature channels are feature channels determined after one or more second feature extractions of the P second feature channels.

9. The method according to claim 7 or 8, characterized in that: The multi-frame image fusion model is obtained by inputting N frames of third images as sample data and using the fused fourth image as a label for training, or is obtained by training based on historical sample documents.

10. The method according to claim 9, characterized in that The N frames of third images are determined after data degradation and / or sequence adjustment are performed on preset sample images.

11. The method according to any one of claims 1 to 10, characterized in that The M channel image is an RGB channel type image.

12. A device for fusion of multiple frames of images, characterized in that: The device comprises: A first image acquisition unit is configured to acquire N frames of first images, wherein each of the N frames of first images is an M-channel image, and the M-channel images correspond to M different types of features, and both M and N are positive integers greater than 0; a second image determining unit, which determines M pseudo images corresponding to the M different types of features based on the N frames of first images, and determines a second image based on the M pseudo images; Among them, each of the M pseudo images is an N-channel image, the N channels are channels composed of the values ​​of the target channels in the first images of the N frames respectively extracted, the target channel is one of the M channels, and different pseudo images correspond to different target channels.

13. The device according to claim 12, characterized in that The second image determination unit is specifically configured to: Determine M first feature maps corresponding to the M pseudo images, each of the M pseudo images corresponds to a first feature map; Determine, based on an attention mechanism, M second feature maps corresponding to the M first feature maps, where each of the M first feature maps corresponds to a second feature map; Perform image fusion processing on the M second feature maps to obtain the second image.

14. The device according to claim 13, characterized in that The second image determination unit is specifically configured to: A first feature extraction is performed on the M pseudo images through a first convolution operation to determine the corresponding M first feature maps, each of the M first feature maps includes Q first feature channels, where Q is an integer greater than 0.

15. The device according to claim 14, characterized in that The second image determination unit is specifically configured to: For each of the M first feature maps, determining a weight of each of the Q first feature channels of each of the first feature maps; Performing weighted processing on the Q first feature channels based on the weights of the Q first feature channels of each of the first feature maps, determining Q second feature channels corresponding to the Q first feature channels of each of the first feature maps, each first feature channel in the Q first feature channels corresponding to one second feature channel; The M second feature maps corresponding to the M first feature maps are determined, each of the second feature maps including Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map.

16. The device according to any one of claims 13 to 15, characterized in that The second image determination unit is specifically configured to: The M second feature maps are fused to determine a third feature map, where the third feature map includes P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; Perform a second convolution operation on the third feature map to obtain the second image.

17. The device according to any one of claims 12 to 16, characterized in that Applied to an electronic device, the electronic device includes a multi-frame image fusion model; the second image determination unit is specifically used to: The N frames of first images are input into the multi-frame image fusion model, and the second image is output.

18. The device according to claim 17, characterized in that The multi-frame image fusion model includes a channel reorganization sub-model, an attention sub-model and a multi-layer convolution sub-model; the second image determination unit is specifically used to: Input the N frames of first images into the channel reorganization sub-model, and output the M pseudo images; Performing first feature extraction on the M pseudo images to determine corresponding M first feature maps, each of the M first feature maps including Q first feature channels; Input the M first feature maps into the attention sub-model, and output M second feature maps corresponding to the M first feature maps, each of the second feature maps including Q second feature channels corresponding to the Q first feature channels of the corresponding first feature map; Perform image fusion processing on the M second feature maps to obtain the second image, and output the second image.

19. The device according to claim 18, characterized in that The second image determination unit is specifically configured to: The M second feature maps are fused to determine a third feature map, where the third feature map includes P second feature channels, where P is the product of M and Q, and the P second feature channels include all the second feature channels in the M second feature maps; The P second feature channels of the third feature map are input into the multi-layer convolution sub-model, and the second image is output, wherein the second image is the M-channel image, and the second image is obtained by converting R second feature channels out of the P second feature channels, and the R second feature channels are feature channels determined after one or more second feature extractions of the P second feature channels.

20. The device according to claim 18 or 19, characterized in that The multi-frame image fusion model is obtained by inputting N frames of third images as sample data and using the fused fourth image as a label for training, or is obtained by training based on historical sample documents.

21. The device according to claim 20, characterized in that The N frames of third images are determined after data degradation and / or sequence adjustment are performed on preset sample images.

22. The device according to any one of claims 12 to 21, characterized in that The M channel image is an RGB channel type image.

23. A computer storage medium, characterized in that The computer storage medium stores a computer program, which implements the method according to any one of claims 1 to 11 when executed by a processor.

24. A computer program, characterized in that The computer program comprises instructions, and when the computer program is executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 11.

Images