Method and Electronic Device for Multi-Frame Image Fusion

Abstract

The present application provides a method for multi-frame image fusion and an electronic device. The electronic device takes every n frames of continuously acquired multi-frame images as one round to generate one frame of fused image. Specifically, the electronic device sequentially acquires the first to the nth frames of images. When acquiring the first frame of image, it calculates the corresponding first weight. By analogy, it continues to acquire the kth frame of image and calculates the kth weight corresponding to the kth frame of image (k takes integers from 2 to n in sequence, n≥2). The (k-1)th intermediate image and the kth frame of image are weighted and summed using the normalized (k-1)th weight and the kth weight to obtain the kth intermediate image. Among them, when k = 2, the (k-1)th intermediate image is the first frame of image. When k = n, this kth intermediate image is the fused image of this round, and then the electronic device also outputs this fused image. Among them, every time a new intermediate image is calculated, it replaces the previous intermediate image and is cached to the buffer.

Description

Technical Field

[0001] This application relates to the field of terminals, and in particular, to a method for multi-frame image fusion and an electronic device. Background Art

[0002] Multi-frame image fusion technology requires hardware in an electronic device to support buffering multiple frames of images. Specifically, taking an electronic device that uses an Image Signal Processor (ISP) to perform multi-frame image fusion as an example, an ISP with a relatively large storage space needs to be pre-designed. This storage space can be used for the ISP to cache the latest multiple frames of images in real time, providing data support for subsequent fusion.

[0003] Based on this, how to optimize the method of multi-frame image fusion to ensure the real-time performance of image fusion while reducing the cost of hardware storage. Summary of the Invention

[0004] This application provides a method for multi-frame image fusion and an electronic device. The electronic device takes every n frames of images continuously acquired as one round to generate one frame of fused image. That is, whenever a new frame of image is acquired, an image fusion is performed on the new image and the previous intermediate result to obtain a new intermediate result. Until the last frame of the round is acquired, an image fusion is performed on the last frame of image and the previous intermediate result, and the last intermediate result obtained is the fused image finally obtained in this round. Such an iterative multi-frame image fusion method only needs to cache the most recent intermediate result, reducing the occupation of cache space. And it also distributes the computational amount of the electronic device when each new frame of image is acquired, rather than concentrating on when the last frame of image is acquired. Therefore, it can also reasonably use the computing resources of the electronic device.

[0005] In a first aspect, this application provides a method for multi-frame image fusion. The method includes: sequentially acquiring the first frame of image to the nth frame of image; when the first frame of image is acquired, determining the first weight of the first frame of image; when the kth frame of image is acquired, where k sequentially takes integer values from 2 to n, determining the kth weight of the kth frame of image, normalizing the kth weight and the (k - 1)th weight, using the normalized kth weight and the normalized (k - 1)th weight to perform weighted summation on the kth frame of image and the (k - 1)th intermediate image to obtain the kth intermediate image, clearing the (k - 1)th intermediate image cached in the buffer buffer, and caching the kth intermediate image in the buffer; when k = 2, the (k - 1)th intermediate image is the first frame of image. Outputting the nth intermediate image, and the nth intermediate image is an HDR image.

[0006] Implementing the method provided in the first aspect, first, since the computational load of fusing multiple frames of images is distributed when each frame of image arrives, rather than performing a large amount of operations when the last frame of image arrives as in traditional multi-frame image fusion, and the computational loads of weight calculation, weight normalization, and weighted summation for each frame of image are all fixed, it is convenient for hardware to achieve stable real-time pipelining and avoid instantaneous high computations by the hardware. Secondly, specifically, since each newly generated intermediate image clears the previous intermediate image and stores the new intermediate image, only the storage space for at least one frame of image needs to be provided to cache the latest intermediate image during each round of image fusion.

[0007] Combined with the method described in the first aspect, the method includes: sequentially obtaining the (n + 1)-th frame of image to the 2n-th frame of image; when the (n + 1)-th frame of image is obtained, determining the (n + 1)-th weight of the (n + 1)-th frame of image; when the (n + k)-th frame of image is obtained, where k sequentially takes integer values from n + 2 to 2n, determining the (n + k)-th weight of the (n + k)-th frame of image, normalizing the (n + k)-th weight and the (n + k - 1)-th weight, performing weighted summation on the (n + k)-th frame of image and the (n + k - 1)-th intermediate image using the normalized (n + k)-th weight and the normalized (n + k - 1)-th weight to obtain the (n + k)-th intermediate image, clearing the (n + k - 1)-th intermediate image cached in the buffer, and caching the (n + k)-th intermediate image in the buffer; when k = 2, the (n + k - 1)-th intermediate image is the (n + 1)-th frame of image. Outputting the 2n-th intermediate image, the 2n-th intermediate image being an HDR image, and the n-th intermediate image and the 2n-th intermediate image being two consecutive frames in the output video.

[0008] In this way, in the scenario of continuously inputting an HDR video, by using the method for iteratively fusing multiple frames of images into an HDR image provided in this application, an HDR stream can be output in real time.

[0009] Combined with the method described in the first aspect, before obtaining the first frame of image, the method includes: receiving an operation of selecting the high dynamic range (HDR) mode.

[0010] Combined with the method described in the first aspect, the brightness range of the n-th intermediate image is greater than the brightness range of each frame of image from the first frame of image to the n-th frame of image.

[0011] In this way, the brightness range of the image can be enhanced through image fusion to output high-quality images.

[0012] The method described in the first aspect is applied to an electronic device, which includes a cache and the buffer. The method further includes: when the first frame of image is obtained, caching the first frame of image in the cache or the buffer; when the k-th frame of image is obtained, clearing the (k-1)-th frame of image in the cache and caching the k-th frame of image in the cache, or when the k-th frame of image is obtained, clearing the (k-1)-th frame of image in the buffer and caching the k-th frame of image in the buffer.

[0013] In this way, when a new frame of image is obtained each time, the new frame of image can be stored for intermediate state fusion processing, and then the previous frame of image is cleared when the next new frame of image arrives, so that useless images can be cleared in time and space can be saved.

[0014] The method described in the first aspect is applied to an electronic device, which includes a processor, a cache and the buffer. If the processing rate of the processor is higher than the first value, when the first frame of image is obtained, caching the first frame of image in the cache, and when the k-th frame of image is obtained, clearing the (k-1)-th frame of image in the cache and caching the k-th frame of image in the cache; if the processing rate of the processor is lower than the first value, when the first frame of image is obtained, caching the first frame of image in the buffer, and when the k-th frame of image is obtained, clearing the (k-1)-th frame of image in the buffer and caching the k-th frame of image in the buffer.

[0015] In this way, if the processor has excellent performance and its data processing rate is fast enough, reading data from the buffer for processing may not be able to exert the performance of the processor. Therefore, each newly obtained frame of image can be cached in the cache, because the cache can support a higher read / write rate, which can not only reduce the occupancy of the buffer but also improve the rate of the fused image.

[0016] The method described in the first aspect is applied to an electronic device, which includes a cache and the buffer. The method further includes: when determining the first weight of the first frame of image, caching the first weight in the cache or the buffer; when determining the k-th weight of the k-th frame of image, caching the k-th weight in the cache or the buffer, and after performing weighted summation on the k-th frame of image and the (k-1)-th intermediate image using the normalized k-th weight and the normalized (k-1)-th weight, clearing the (k-1)-th weight in the cache.

[0017] In this way, each time a new weight of a new frame of image is determined, the new weight can be stored for the fusion processing of the intermediate state, and then the weight of the previous frame of image can be cleared when determining the new weight of the next frame of new image, so that useless weights can be cleared in time and space can be saved.

[0018] Combined with the method described in the first aspect, this method is applied to an electronic device, which includes a processor, a cache, and the buffer. If the processing rate of the processor is higher than the first value, when determining the first weight of the first frame of image, the first weight is cached in the cache, and when determining the k-th weight of the k-th frame of image, the k-th weight is cached in the cache; if the processing rate of the processor is lower than the first value, when determining the first weight of the first frame of image, the first weight is cached in the buffer, and when determining the k-th weight of the k-th frame of image, the k-th weight is cached in the buffer. After weighted summation of the k-th frame of image and the k-1 intermediate image using the normalized k-th weight and the normalized k-1 weight, the k-1 weight in the buffer is cleared.

[0019] In this way, if the processor performance is excellent and its data processing rate is fast enough, reading data from the buffer for processing may not be able to exert the performance of the processor. Therefore, the weight of each newly determined frame of image can be cached in the cache because the cache can support a higher read / write rate, which can not only reduce the occupancy of the buffer but also improve the rate of the fused image.

[0020] Combined with the method described in the first aspect, to determine the first weight corresponding to the j-th frame of image, where j takes integer values from 1 to n in sequence, specifically includes: obtaining the j-th weight according to one or more of the contrast, saturation, and brightness of the j-th frame of image.

[0021] In this way, the proportion of this frame of image in the fused image can be measured based on one or more types of information in the image, improving the quality of the fused image.

[0022] Combined with the method described in the first aspect, n is any integer greater than or equal to 2. When the value of n is larger, the more storage space can be saved, and the more space for optimizing the distribution of the calculation amount can be obtained.

[0023] In a second aspect, the present application provides an electronic device, including a cache, a buffer, one or more processors, and a computer program stored on the memory, and the processor executes the computer program to perform the method described in any one of the foregoing first aspects.

[0024] In a third aspect, the present application provides a chip system, which includes a cache, a buffer, a processor, and a computer program stored in a memory. The processor executes the computer program to implement the method described in any one of the foregoing first aspects.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method described in any one of the foregoing first aspects.

[0026] In a fifth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method described in any one of the foregoing first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 FIG. is a schematic diagram of an application scenario for generating an HDR image based on multi-frame image fusion provided by an embodiment of the present application;

[0028] Figure 2 FIG. is a schematic diagram of a traditional multi-frame image fusion method provided by an embodiment of the present application;

[0029] Figure 3 FIG. is a schematic diagram of an iterative multi-frame image fusion method provided by an embodiment of the present application;

[0030] Figure 4 FIG. is a schematic diagram of the hardware architecture of an electronic device provided by an embodiment of the present application;

[0031] Figure 5 FIG. is a schematic diagram of the hardware architecture of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The technical solutions in the embodiments of the present application will be clearly and elaborately described below with reference to the drawings. In the present application, referring to "embodiment" means that a specific feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described in the present application may be combined with other embodiments.

[0033] In the following embodiments of this application, the term "user interface (UI)" is a media interface for interaction and information exchange between an application or an operating system and a user. It realizes the conversion between the internal form of information and the form that can be received by the user. The user interface is source code written in specific computer languages such as Java and Extensible Markup Language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that can be recognized by the user. The common manifestation form of the user interface is the graphical user interface (GUI), which refers to the user interface related to computer operations displayed in a graphical manner. It can be visual interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and Widgets displayed on the display screen of an electronic device.

[0034] Currently, the multi-frame image fusion method is applied in more and more scenarios. In this application, only the scenario of high-dynamic range (HDR) image generation is taken as an example to introduce the multi-frame image fusion method. For the introduction of application scenarios, please refer to the following description of Figure 1 .

[0035] Figure 1 An application scenario of generating HDR images based on multi-frame image fusion is exemplarily shown.

[0036] As Figure 1 shown, first, an electronic device can continuously capture multiple frames of images with a camera to obtain a raw image stream, which includes but is not limited to the first raw image, the second raw image, the nth raw image, and so on. Then, the graphics processing unit (GPU) in the electronic device can, based on the obtained raw image stream, fuse the latest n frames of raw images obtained each time into one frame of HDR image in real time, so as to output an HDR image stream in real time, which includes but is not limited to the first HDR image and the second HDR image.

[0037] Among them, the shooting parameters corresponding to multiple frames of images included in the original image stream are different. For example, when the camera enables functions such as automatic exposure, the exposure parameters used for shooting each frame of image are different, resulting in possible differences in contrast, saturation, brightness, etc. in each frame of image. In addition, the name of the original image is only an example. "Original" can refer to the unprocessed images captured by the camera in the electronic device, or the images obtained after preprocessing on the basis of the unprocessed images. In this application, "original" mainly refers to the images before the fusion process, and it does not limit whether other processes except image fusion are performed.

[0038] Among them, one or more pieces of information such as the contrast, saturation, and brightness of the HDR image are richer compared to one or more pieces of information such as the contrast, saturation, and brightness of each frame of the original image used to fuse and generate this frame of HDR image. This is because a frame of HDR image is obtained by fusing n frames of original images, and one or more pieces of information such as the contrast, saturation, and brightness in each of the n frames of original images all contribute to this frame of HDR image. In addition, the name of the HDR image is only an example. "HDR" usually specifically refers to the dynamic range of brightness. In this application, in addition to the expanded brightness range, the images generated after image fusion may also have other information such as richer saturation and contrast information. Therefore, this application can also refer to the images generated after image fusion as high-definition images, etc.

[0039] Figure 1 Only one application scenario of multi-frame image fusion is exemplarily shown. In addition, the multi-frame image fusion method provided in this application can also be applied in other scenarios. When the application scenarios are different, in the multi-frame image fusion method, the factors affecting the weight of each frame of image may be different, and the weighted calculation of multi-frame images may be different, but they all adopt an iterative calculation method rather than a traditional calculation method. For the introduction of the weighted fusion calculation method, please specifically refer to the following description of Figures 2 to 3 of.

[0040] Generally, traditional multi-frame image fusion methods include the following stages: First, continuously receive multiple frames of images, input each newly received image into a buffer, and calculate the weight corresponding to each frame of image when a new frame of image is received each time. Also input the weight corresponding to each newly received frame of image into the buffer. Second, take n frames of images as a cycle, where n≥2. Whenever a new round of the nth frame of image is received and the weight corresponding to the nth frame of image is calculated, then normalize the weights corresponding to the n frames of images in this round, and perform weighted summation based on the n frames of images in this round and the normalized weights corresponding to the n frames of images respectively. The result of the weighted summation is a frame of fused image. Finally, after performing image fusion on the n frames of images in this round, the n frames of images and the weights corresponding to the n frames of images cached in the buffer can also be cleared to ensure that the released space in the buffer can cache the complete data required for the next round of image fusion. And so on, continue to receive multiple frames of images, and perform weighted summation on the n frames of images in the next round to obtain the next frame of fused image.

[0041] Next, in combination with Figure 2 , taking n = 5 as an example, an exemplary traditional multi-frame image fusion method will be shown.

[0042] As Figure 2As shown, when the GPU in the electronic device receives the first frame of image, it will input it into the buffer for caching. The storage items in the buffer include the first frame of image (which can be denoted as I0). And when the GPU in the electronic device receives the first frame of image, the arithmetic items executed include calculating the weight of the first frame of image, and then the weight of the first frame of image is also input into the buffer for caching. The storage items in the buffer are increased by the weight of the first frame of image (which can be denoted as W0). When the GPU in the electronic device receives the second frame of image, it will input it into the buffer for caching. The storage items in the buffer are increased by the second frame of image (which can be denoted as I1). And when the GPU in the electronic device receives the second frame of image, the arithmetic items executed include calculating the weight of the second frame of image, and then the weight of the second frame of image is also input into the buffer for caching. The storage items in the buffer are increased by the weight of the second frame of image (which can be denoted as W1). And so on. When the GPU in the electronic device receives the third frame of image (denoted as I2), the fourth frame of image (denoted as I3), and the fifth frame of image (denoted as I4), the same arithmetic operations and storage operations as those of the first frame of image and the second frame of image mentioned above will be executed, so that five frames of images (denoted as I0 - I4) and the weights corresponding to the five frames of images (denoted as W0 - W4) are cached in the buffer of the electronic device. In addition, until the 5th frame of image is received, the weights of the 5 frames can be normalized. Based on the normalized weights of the 5 frames and the 5 frames of images, the fusion processing (i.e., weighted operation) can be carried out to output a frame of fused image. That is, after calculating W4, it is also necessary to normalize W0 - W4, and then use the normalized W0 - W4 to perform weighted summation on I0 - I4, so as to output the fused image (denoted as O4). And so on. The method for the subsequent GPU of the electronic device to receive and cache the next five frames of images (denoted as I5 - I9), calculate and cache the weights corresponding to the next five frames of images (denoted as W5 - W9), normalize W5 - W9, and perform weighted summation on I5 - I9 based on the normalized W5 - W9 to output the fused image (denoted as O9) is the same as the calculation operations and caching operations corresponding to the current five frames of images introduced above, and will not be elaborated here one by one.

[0043] Optionally, the GPU can calculate the weight of each frame of image immediately upon receiving it. After calculating the weight, it inputs the frame of image and the weight into the buffer for caching. Alternatively, when the GPU receives each frame of image, it can first input the frame of image into the buffer for caching, then calculate its weight, and finally input the weight into the buffer for caching. The calculation method of the image weight can refer to the description in the following text and will not be elaborated here. Optionally, in addition to using the GPU to perform image fusion, other types of processors can also be used in the electronic device, and the buffer can be the memory integrated in the GPU or the memory independent of the GPU. The embodiments of the present application do not limit this.

[0044] Figure 2 Taking the generation of one frame of fused image from five frames of images with n = 5 as an example, and only taking the execution of two rounds of image fusion (i.e., continuously outputting two frames of fused images) as an example, in addition, the traditional image fusion method can also be that n is other values, or only supports the output of one frame of fused image, or supports the output of multiple frames of fused images, i.e., the fused image stream output. Figure 2 The content shown should not constitute a limitation to the present application.

[0045] By analyzing Figure 2 It can be seen from the traditional image fusion method shown that during the process of the GPU in the electronic device executing each round of image fusion, before receiving the last frame of image in each round, the computing workload of the GPU is relatively small. For example, the computing workload of the GPU is distributed when each frame of image arrives before the last frame of image, and only the weight corresponding to each frame of image is calculated. Most of the computing workload of the GPU is concentrated on receiving the last frame of image in each round. For example, it is necessary to calculate the weight of the last frame of image, calculate the normalization result of the weights of all images, calculate the weighted sum of all images, etc. This will result in a longer output delay for each round of image fusion. In the scenario of real-time output of the fused image stream, its real-time performance is poor, affecting the user experience. And before the GPU calculates the weighted sum of all images, it is necessary to store all the images and weights in this round in the buffer, which will increase the design cost of the hardware storage.

[0046] With the high requirements of users for fused images, the GPU in the electronic device needs to generate one frame of fused image based on more frames of original images to ensure the quality of the fused image. That is to say, in order to improve the quality of the fused image, it is necessary to increase the value of n. As the value of n increases (for example, n ≥ 3), the buffer needs to cache more data, and thus has a greater demand for the cache space of the buffer. And as the value of n increases, after receiving the nth frame of image in each round, the GPU needs to calculate more data, which further affects the real-time performance of multi-frame image fusion to a greater extent.

[0047] To solve the foregoing technical problems, the present application provides a method for multi-frame image fusion and an electronic device. The electronic device uses an iterative calculation method to generate a fused image by taking every n frames of continuously acquired multi-frame images as one round, and then generates m fused images in m consecutive rounds (m≥1). Among them, the process of generating a fused image in any one round includes: the electronic device sequentially acquires the first to the nth frames of images. When the first frame of image is acquired, the corresponding first weight is calculated. By analogy, continue to acquire the kth frame of image and calculate the kth weight corresponding to the kth frame of image (k takes integers from 2 to n in sequence, n≥2). The (k-1)th intermediate image and the kth frame of image are weighted and summed using the normalized (k-1)th weight and the kth weight to obtain the kth intermediate image. Among them, when k = 2, the (k-1)th intermediate image is the first frame of image. When k = n, this kth intermediate image is the fused image of this round, and the electronic device also outputs this fused image. Among them, each time a new intermediate image is calculated, it replaces the previous intermediate image and is cached in the buffer.

[0048] Among them, n can be any integer greater than or equal to 2. When n is equal to 2, the iterative image fusion method provided by the present application is the same as the traditional image fusion method. Therefore, when n is any integer greater than or equal to 3, the iterative image fusion method provided by the present application can have the following beneficial effects compared with the traditional image fusion method.

[0049] (1) Improve the real-time performance of multi-frame image fusion. Specifically, since the computational amount of multi-frame image fusion is dispersed when each frame of image arrives, rather than performing a large number of operations when the last frame of image arrives as in traditional multi-frame image fusion, and the computational amounts of weight calculation, weight normalization, and weighted summation for each frame of image are fixed, it is convenient for the hardware to perform stable and real-time pipelining to avoid the hardware from instantaneously executing high operations.

[0050] (2) Reduce the hardware storage cost. Specifically, since each time a new intermediate image is generated, the previous intermediate image is cleared and the new intermediate image is stored. In this way, during each round of image fusion, at least the storage space for one frame of image is provided to cache the latest intermediate image. If the hardware computing power of the electronic device is insufficient, another frame of image storage space can also be provided to cache the latest acquired image, rather than at least providing the storage space for caching n frames of images and n weights in traditional multi-frame image fusion, where each weight is actually a frame of weight image, that is, a frame of weight image contains the weight values corresponding to each pixel point.

[0051] (3) When using a deep learning network model for multi-frame image fusion, the iterative image fusion method can also be adopted, which can also save the buffer space for the original images. Except for the different specific operation methods, the form of real-time pipelining and the utilization of the buffer space are basically the same.

[0052] Next, in combination with Figure 3 , taking n = 5 as an example, an iterative multi-frame image fusion method will be exemplarily shown.

[0053] As Figure 3 shown, when the GPU in the electronic device receives the first frame of image (which can be denoted as I0), the operation items to be executed include calculating the first weight of the first frame of image (which can be denoted as W0). Since this first frame of image is used as the first intermediate image (which can be denoted as O0) for subsequent intermediate-state image fusion, the first intermediate image is also input into the buffer for caching, and the storage items in the buffer include the first intermediate image.

[0054] Continuing to refer to Figure 3 , when the GPU in the electronic device receives the second frame of image (which can be denoted as I1), the operation items to be executed include calculating the second weight of the second frame of image (which can be denoted as W1), calculating the normalization result of the first weight of the first frame of image and the second weight of the second frame of image, and performing weighted summation on the first intermediate image and the second frame of image using the normalized first weight and second weight to obtain the second intermediate image (which can be denoted as O1). Since this second intermediate image is used for subsequent intermediate-state image fusion, the first intermediate image in the buffer is also cleared, and the second intermediate image is input into the buffer for caching, and the storage items in the buffer include the second intermediate image. Optionally, when the GPU in the electronic device receives the second frame of image, it can be temporarily input into the buffer for caching, or directly operated on, specifically depending on the performance of the GPU. When the performance of the GPU is high, its computing power is strong, and preferably, it can quickly read and write data from the cache rather than from the buffer. Therefore, there is no need to input the second frame of image into the buffer for caching.

[0055] Continuing to refer to Figure 3, and so on. When the GPU in the electronic device receives the third frame of image (denoted as I2), the fourth frame of image (denoted as I3), and the fifth frame of image (denoted as I4), it will perform the same arithmetic operations and storage operations as those for the aforementioned first frame of image and second frame of image. This enables the electronic device to calculate its weight based on the currently newly received image each time, normalize the new weight and the weight of the previous frame of image, perform weighted summation on the previous intermediate image and the current new image using the normalized new weight and the weight of the previous frame of image to obtain a new intermediate image, and store the new intermediate image. Optionally, when the performance of the electronic device is low, in addition to storing the current new intermediate image in the buffer, the next frame of new image will also be stored in the buffer after the next frame of new image arrives and before the new intermediate image is generated. When the performance of the GPU is high, its computing power is strong, and preferably, it can quickly read and write data from the cache instead of slowly reading and writing data from the buffer. Therefore, there is no need to input the second frame of image into the buffer for caching. Optionally, when the electronic device receives the last frame of image, i.e., the fifth image, if the fifth intermediate image (which can be denoted as O4) is calculated, this fifth intermediate image can be output as the finally generated fused image without being stored in the buffer.

[0056] Continue to refer to Figure 3 , after the electronic device receives five frames of images in the previous round (denoted as I0 - I4) and performs one round of image fusion to output one frame of fused image (denoted as O4), if the electronic device continues to receive five frames of images in the subsequent round (denoted as I5 - I9), it will perform similar calculation and storage operations as those for the five frames of images in the previous round, so as to output the fused image (denoted as O9). The method will not be elaborated here one by one.

[0057] Figure 3 Only taking the case where n = 5, i.e., five frames of images generate one frame of fused image as an example, in addition, the iterative image fusion method can also be applied to other values of n. Figure 3 Also only taking the execution of two rounds of image fusion (i.e., continuously outputting two frames of fused images) as an example for illustration, in addition, the iterative image fusion method can also be applied to the output scenario that only supports the output of one frame of fused image, or the output scenario that supports the output of multiple frames of fused images, i.e., the output scenario of a fused image stream. Figure 3 The content shown should not constitute a limitation to this application.

[0058] Regardless of what n equals, when using the iterative multi-frame image fusion method provided by this application, the following calculation formula is adopted.

[0059] Formula for normalizing the weight: , .

[0060] Among them, when only the first frame of image (denoted as I0) arrives, then . When the second frame of image (denoted as I1) arrives, then .

[0061] Formula for the fused image: , .

[0062] Among them, when only the first frame of image (I0) arrives, then , that is, the input image also serves as the first intermediate image generated by fusion .

[0063] Among them, when the last frame, that is, the nth frame of image (In) arrives, then which is the final fused image to be output.

[0064] In the above formula, represents the frame order of the original image to be input, represents the pixel position in the image, represents the number of original images required as input to generate one frame of fused image, represents the th pixel information of the sequentially input original image, represents the th pixel information of the sequentially output intermediate image.

[0065] By analyzing the iterative image fusion method shown in Figure 3 : During the process of each round of image fusion executed by the GPU of the electronic device, a large amount of operations will not be concentrated when a certain frame of image arrives. Instead, when each frame of image in each round is received, the amount of operations of the GPU is almost the same. For example, when other frames of images except the first frame arrive, the weight of the new frame of image, the normalization result with the weight of the previous frame of image, and the new intermediate image calculated using the normalization result will be calculated. This can facilitate the stable and real-time pipelining implementation of the hardware, improve the real-time performance of multi-frame image fusion, and avoid increasing the time delay of image fusion due to the instantaneous high operation of the hardware. Moreover, during the process of each round of image fusion executed by a high-performance GPU, only a buffer sufficient to store one frame of image is required. During the process of each round of image fusion executed by a low-performance GPU, only a buffer sufficient to store two frames of image is required, instead of storing all images and weights in each round in the buffer as in the traditional method, thus greatly reducing the design cost of hardware storage.

[0066] Next, an exemplary method for calculating the weight of an image involved in image fusion will be shown.

[0067] The basic principle of image fusion is to combine the information of multiple frames of images taken under different shooting parameters (such as different exposure parameters) to generate a high-quality image containing the key information and features in the multiple frames of images. Among these multiple frames of images, the amount of key information contained in each original image to be fused is different. Therefore, weights are needed to measure the amount of key information contained in each original image to be fused. That is to say, the weight of an image refers to the relative contribution degree of the original image before fusion in the finally fused image during the process of fusing based on the original images to generate a fused image.

[0068] In an optional manner, the calculation method of the weight can be calculated according to any one or more of the following information: one or more information such as the contrast (Contrast), saturation (Saturation), and well-exposedness (Well-exposedness) of the image. For the time being, only the example of jointly calculating the weight by combining the three pieces of information will be introduced later. In other optional manners, the calculation of the weight can also be based on other information required, and the calculation method of the weight should not constitute a limitation to this application.

[0069] Contrast describes the degree of difference between different gray levels in an image. High contrast means that there is a large difference in gray levels in the image (i.e., large gradient information), which usually makes the image look more distinct and clear. Low contrast means that the difference in gray levels is small (i.e., small gradient information), which usually results in the image looking relatively blurred or lacking in layering. Taking an image with a 256-bit depth as an example, the gray values of the pixel points in the overexposed area of the image are almost all 255, and the gray values of the pixel points in the underexposed area of the image are almost all 0. The gray values in the overexposed area and the underexposed area are very smooth, and the gradient information is almost 0. These areas lack detailed information and therefore need to be discarded, that is, the set of weight values corresponding to these areas in the weight map is 0. Therefore, the Laplace operator can be used to extract the gradient information of the image to represent the contrast of the image. The smaller the gradient information at a pixel point, the smaller the proportion of the pixel value at that point in image fusion. On the contrary, the larger the gradient information at a pixel point, the larger the proportion of the pixel value at that point in image fusion.

[0070] Among them, the calculation formula for the contrast of the image is: . Among them, represents the frame order of the original image to be input, represents the pixel position in the image, represents the number of original images required to generate a frame of fused image, represents the pixel information of the th sequentially input original image, The pixel information of the grayscale image corresponding to the original images input in sequence represents the Laplacian operator represents the contrast information of the nth image output in sequence represents the Laplacian operator

[0071] Saturation describes the purity or vividness of the colors in an image. The higher the saturation, the more vivid the colors look; the lower the saturation, the closer the colors are to gray or white. When taking an image, high exposure can cause the colors in the captured image to be unsaturated. By calculating the standard deviation of the R, G, and B channels of each pixel point in the image, the saturation can be obtained.

[0072] Among them, the calculation formula for the contrast of the image is: , . Among them, represents the frame sequence of the original image to be input represents the pixel position in the image represents the number of original images required as input to generate one frame of the fused image represents the pixel information of the R channel of the mth original image input in sequence represents the pixel information of the G channel of the mth original image input in sequence represents the pixel information of the B channel of the mth original image input in sequence represents the average value of the R, G, and B channels of the mth original image input in sequence represents the saturation information of the nth image output in sequence

[0073] Brightness describes the light and dark levels in an image. Taking an image with a 256-bit depth as an example, in the image, areas where the pixel values are close to 255 have overexposure problems, areas where the pixel values are close to 0 have underexposure problems, and areas where the pixel values are close to 128 have good exposure. Therefore, the weight at the pixel points where the pixel values are close to 128 can be set higher, and the pixel values at the pixel points where the pixel values are far from 128, that is, close to 255 or 0, can be set lower.

[0074] Optionally, the calculation formula for the brightness of the image can be measured using a Gaussian curve. Specifically, for example, the formula is: .

[0075] Among them, represents the frame sequence of the original image to be input represents the pixel position in the image represents the number of original images required as input to generate one frame of the fused image, represents the pixel information of the R channel of the th sequentially input original image, represents the pixel information of the G channel of the th sequentially input original image, represents the pixel information of the B channel of the th sequentially input original image, represents the luminance information of the th sequentially output image. The significance of 0.5 is that in image processing, pixel values are usually normalized to the range [0, 1]. Therefore, the distance of the pixel value from 0.5, which is the same as the distance from 128, can reflect the luminance level of the pixel. If the pixel value is close to 0.5, it indicates that the luminance of the pixel is moderate. If the luminance value is far from 0.5, it means the pixel is too dark or too bright. The significance of σ equal to 0.2 is that in the Gaussian curve, σ determines the width of the curve. The larger σ is, the wider the curve, indicating a larger influence range of the pixel value; the smaller σ is, the narrower the curve, indicating a smaller influence range of the pixel value. In this example, σ equal to 0.2 means the Gaussian curve is relatively narrow and has a greater impact only on pixels with luminance values close to 0.5, while having less impact on pixels far from 0.5. This can more precisely evaluate the exposure of each pixel. In addition, since a color image usually consists of multiple color channels (such as the RGB three channels), each channel contains pixel values. Therefore, when calculating the luminance of the image, multiple color channels are also considered. The Gaussian curve is applied to each color channel separately to comprehensively consider the luminance information of all color channels, and then the results are multiplied to obtain a measurement criterion: multiplying the Gaussian curve results of each color channel can obtain a comprehensive measurement criterion. This criterion reflects the exposure of the image in each color channel. If the exposure of a certain color channel is poor (i.e., the Gaussian curve result is small), then the exposure of the entire image will also be affected.

[0076] In summary, using the method based on the distance from 0.5 of luminance with a Gaussian curve where σ is equal to 0.2, considering multiple color channels and applying the Gaussian curve separately to each channel and multiplying the results to obtain a measurement criterion can comprehensively consider the luminance information of the image in each color channel and precisely evaluate the exposure of each pixel.

[0077] As mentioned above, according to the three aspects of contrast, saturation, and luminance, the proportion of each pixel in multiple frames of images during fusion can be obtained. The formula for obtaining the weight by comprehensively considering contrast, saturation, and luminance is: , in addition, in order to avoid the problem of too high or too low pixel values in multi-frame image fusion, it is also necessary to normalize the weights for normalization.

[0078] Next, the types of electronic devices and the software and hardware architectures involved in the multi-frame image fusion method provided by this application will be introduced in detail.

[0079] The electronic device can be a portable terminal device equipped with iOS®, Android®, Microsoft® or other operating systems, such as mobile phones, tablets, desktop computers, laptop computers, handheld computers, notebooks, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices and / or smart city devices, and so on.

[0080] Figure 4 Shows a schematic diagram of the hardware architecture of the electronic device 100.

[0081] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a sensor module 180, a camera 193, a display screen 194, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a touch sensor 180K, etc.

[0082] It can be understood that the structure schematically shown in the embodiments of this application does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0083] The processor 110 may include one or more processing units. For example, the processor 110 may include a graphics processing unit (GPU), an image signal processor (ISP), an application processor (AP), a modem processor, a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0084] In the embodiments of the present application, the GPU included in the processor 110 is hardware specifically used to perform image-related calculations, mainly used to accelerate the image processing process. For example, the GPU is mainly used to perform image fusion, render the fused image and send it to the display screen for display. For the specific implementation method of the GPU to perform image fusion, reference can be made to the previous introduction of Figure 3 , which will not be elaborated here.

[0085] In the embodiments of the present application, a memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system. For example, the cache memory may store the weights of the images mentioned above and the latest received images, etc., which can not only facilitate the CPU and GPU to quickly process images, but also reduce the design cost of the buffer.

[0086] The USB interface 130 is an interface that conforms to the USB standard specification. Specifically, it may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used for data transmission between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0087] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniled, a microLed, a micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0088] In the embodiments of the present application, the electronic device 100 can jointly implement the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, and is connected to the display screen 194, the CPU, the AP, etc. The GPU can receive the image stream continuously transmitted by the CPU or the AP, perform fusion processing on multiple frames of images, and then output a frame of fused image or a fused image stream to the display screen.

[0089] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, etc. format. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0090] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera's photosensitive element. The optical signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise and brightness of the image through algorithms. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be disposed in the camera 193.

[0091] In the embodiments of the present application, the electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc. Specifically, the image stream continuously captured by the camera 193 can be sent to the application processor after preprocessing by the ISP, and then sent to the GPU for image fusion. The finally generated fused image stream is then sent to the display screen 194 for display.

[0092] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 110, and can be used to store the operating system or executable programs (such as machine instructions) of other running programs, and can also be used to store data of users and application programs, etc. The non-volatile memory can also store executable programs and store data of users and application programs, etc., and can be pre-loaded into the random access memory for the processor 110 to directly read and write.

[0093] In the embodiments of the present application, the buffer is located in the RAM. The buffer is used to cache the intermediate images during the image fusion process. Optionally, when the performance of the CPU and GPU is low (for example, the data processing rate is lower than the first value), the buffer is also used to cache the input original images and / or weights. When the performance of the CPU and GPU is high (for example, the data processing rate is higher than the first value), the Cache between the CPU and the RAM is used to cache the input original images and / or weights.

[0094] Among them, Cache is a small-capacity but extremely fast memory between the central processing unit and the main memory. It is mainly used to solve the speed difference problem between the CPU and the main memory, thereby improving the overall performance of the system. The design and management of Cache are crucial for improving the overall performance of the computer system. In a computer system, Cache is usually composed of static random access memory (SRAM), which is faster but more expensive than dynamic random access memory (DRAM). Therefore, the capacity of Cache is relatively small, but its speed is very fast and it can quickly respond to the data requests of the CPU.

[0095] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to achieve the data storage function. For example, files such as music and videos are saved in the external non-volatile memory.

[0096] The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can include at least two parallel plates with conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations with the same touch position but different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0097] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as the "touch control screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from that of the display screen 194.

[0098] In the embodiments of the present application, the electronic device can receive an operation for taking an image or an operation for selecting the HDR mode through the pressure sensor 180A or the touch sensor 180K, etc. The embodiments of the present application do not limit this and do not elaborate in detail.

[0099] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of the present application, taking the Android® system with a layered architecture as an example, the software structure of the electronic device 100 is exemplarily described.

[0100] Figure 5 It is a schematic diagram of the software architecture of the electronic device 100 in the embodiments of the present application.

[0101] As Figure 5 shown, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android® system is divided into five layers, from top to bottom are the application layer, the application framework layer, the hardware abstraction layer, the kernel layer, and the hardware layer.

[0102] The application layer can include a series of application packages. For example, the application package can include applications such as the camera.

[0103] Among them, the camera application includes but is not limited to: a shooting module, and the shooting module includes an HDR mode, a portrait mode, or others, etc. The shooting module is used to execute the photographing and video recording functions of the camera application, including outputting a preview interface in the shooting mode and taking a photo or recording a video after the shooting control is pressed, etc. When the user selects the HDR mode, the electronic device executes the iterative multi-frame image fusion method introduced above to output an HDR photo or an HDR video. In addition, the method for generating an HDR image by multi-frame image fusion provided in the present application can be applied not only in the camera application but also in other image processing applications. The embodiments of the present application do not limit this.

[0104] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer can include a camera service, a view system, and so on.

[0105] The camera service can also be referred to as a camera access interface and a camera device, and is used to provide application programming interfaces and programming frameworks for camera applications.

[0106] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0107] The hardware abstraction layer is an interface layer located between the application framework layer and the kernel layer, and provides a virtual hardware platform for the operating system.

[0108] In the embodiments of the present application, the hardware abstraction layer can include a camera hardware abstraction layer and a camera algorithm library.

[0109] Among them, the camera hardware abstraction layer (Camera HAL) can provide virtual hardware for one or more camera devices, such as virtual hardware for camera device 1 and camera device 2. Camera HAL can obtain image data from underlying cameras and image signal processors, and transmit it to the camera algorithm library. Also, Camera HAL can obtain information from the camera algorithm library. Camera HAL is mainly used to control the corresponding sensors to collect images.

[0110] The camera algorithm library can include one or more algorithm modules, such as an algorithm module for performing image fusion based on multiple frames of images captured by a camera, so as to implement HDR mode, etc. Among them, the algorithm module for image fusion is the module that implements the foregoing Figure 3 shown method.

[0111] The kernel layer is the layer between hardware and software. The kernel layer includes drivers for various hardware. In some embodiments, the kernel layer can include a camera device driver, a digital signal processor driver, and an image processor driver, etc. Among them, the camera device driver is used to drive the sensor of the camera to collect images and drive the image signal processor to preprocess the images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images. The kernel layer is the layer between hardware and software.

[0112] The hardware layer includes sensors such as camera 1, camera 2, TOF, multispectral sensor, image signal processor, digital signal processor, and image processor, etc.

[0113] The following exemplarily describes the working processes of the software and hardware of the electronic device 100 in combination with the capture and photographing scenario.

[0114] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including information such as touch coordinates and the timestamp of the touch operation). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation and the control corresponding to the click operation being the control of the camera application icon as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture a static image or video through the camera 193.

[0115] It should be understood that the steps in the above method embodiments can be completed by the integrated logic circuit of the hardware in the processor or the instructions in software form. The method steps disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware processor, or executed and completed by the combination of the hardware and software modules in the processor.

[0116] The present application also provides an electronic device, which may include a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the method executed by the electronic device in any of the above embodiments.

[0117] The present application also provides a chip system, including a processing circuit and an interface circuit. The interface circuit is used to receive computer instructions and transmit them to the processing circuit, and the processing circuit is used to run the computer instructions to implement the method executed by the electronic device in any of the above embodiments.

[0118] The present application also provides a chip system, which includes at least one processor for implementing the method executed by the electronic device in any of the above embodiments. In a possible design, the chip system further includes a memory, and the memory is used to store program instructions and data, and the memory is located inside or outside the processor.

[0119] The chip system can be composed of chips or can include chips and other discrete devices.

[0120] Optionally, there may be one or more processors in the chip system. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor that implements its functions by reading software code stored in a memory.

[0121] Optionally, there may also be one or more memories in the chip system. The memory may be integrated with the processor or may be separately provided from the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transitory processor, such as a read-only memory (ROM). It may be integrated with the processor on the same chip or may be separately provided on different chips. The embodiments of the present application do not specifically limit the type of the memory and the setting manner of the memory and the processor.

[0122] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processing circuit (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0123] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method executed by the electronic device in any one of the above embodiments is implemented.

[0124] The present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method executed by the electronic device in any one of the above embodiments is implemented.

[0125] The various embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0126] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, fiber optic, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state disk (SSD)), etc.

[0127] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware with a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The foregoing storage medium includes: various media that can store program codes such as ROM or random access memory RAM, magnetic disks, or optical discs.

[0128] In the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B can represent A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality of" means two or more than two.

[0129] The terms "first" and "second" are only used for descriptive purposes and cannot be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more than two.

[0130] In summary, the above description is only an embodiment of the technical solution of this application and is not intended to limit the protection scope of this application. Any modifications, equivalent replacements, improvements, etc. made according to the disclosure of this application shall be included within the protection scope of this application.

Claims

1. A method for multi-frame image fusion, characterized in that: The method comprises: Sequentially acquire the first frame image to the nth frame image, where n≥3; When the first frame of image is acquired, determining a first weight of the first frame of image; When the k-th frame image is acquired, k takes integer values ​​from 2 to n in sequence, determines the k-th weight of the k-th frame image, normalizes the k-th weight and the k-1-th weight, uses the normalized k-th weight and the normalized k-1-th weight to perform weighted summation on the k-th frame image and the k-1-th intermediate image to obtain the k-th intermediate image, clears the k-1-th intermediate image, the k-1-th weight, and the k-th frame image cached in the buffer, and caches the k-th intermediate image in the buffer; when k=2, the k-1-th intermediate image is the 1st frame image; An nth intermediate image is output, where the nth intermediate image is a high dynamic range HDR image.

2. The method according to claim 1, characterized in that The method comprises: Sequentially acquire the n+1th frame image to the 2nth frame image; When the n+1th frame image is acquired, determining the n+1th weight of the n+1th frame image; When the n+kth frame image is acquired, k takes integer values ​​from n+2 to 2n in sequence, determines the n+kth weight of the n+kth frame image, normalizes the n+kth weight and the n+k-1th weight, uses the normalized n+kth weight and the normalized n+k-1th weight to perform weighted summation on the n+kth frame image and the n+k-1th intermediate image to obtain the n+kth intermediate image, clears the n+k-1th intermediate image cached in the buffer, and caches the n+kth intermediate image in the buffer; when k=2, the n+k-1th intermediate image is the n+1th frame image; A 2nth intermediate image is output, where the 2nth intermediate image is an HDR image, and the nth intermediate image and the 2nth intermediate image are two consecutive frames of images in the output video.

3. The method according to claim 1 or 2, characterized in that: Before acquiring the first frame of image, the method includes: An operation of selecting a high dynamic range (HDR) mode is received.

4. The method according to claim 1, characterized in that The brightness range of the nth intermediate image is greater than the brightness range of each frame image from the first frame image to the nth frame image.

5. The method according to claim 1, characterized in that The method is applied to an electronic device, the electronic device comprising a cache memory and the buffer, and the method further comprises: When the first frame of image is acquired, caching the first frame of image in the cache or the buffer; When the k-th frame image is obtained, the k-1-th frame image in the cache is cleared and the k-th frame image is cached in the cache; or, when the k-th frame image is obtained, the k-1-th frame image in the buffer is cleared and the k-th frame image is cached in the buffer.

6. The method according to claim 5, characterized in that The method is applied to an electronic device, wherein the electronic device comprises a processor, a cache and the buffer. If the processing rate of the processor is higher than the first value, when the first frame image is acquired, the first frame image is cached in the cache, and when the kth frame image is acquired, the k-1th frame image in the cache is cleared, and the kth frame image is cached in the cache; If the processing rate of the processor is lower than the first value, when the first frame image is acquired, the first frame image is cached in the buffer, and when the kth frame image is acquired, the k-1th frame image in the buffer is cleared and the kth frame image is cached in the buffer.

7. The method according to claim 1, characterized in that The method is applied to an electronic device, the electronic device includes a cache and the buffer, and the method further includes: When determining the first weight of the first frame image, caching the first weight in the cache or the buffer; When determining the kth weight of the kth frame image, the kth weight is cached in the cache or the buffer, and after weighted summing the kth frame image and the k-1th intermediate image using the normalized kth weight and the normalized k-1th weight, the k-1th weight in the cache is cleared.

8. The method according to claim 7, characterized in that The method is applied to an electronic device, wherein the electronic device comprises a processor, a cache and the buffer. If the processing rate of the processor is higher than the first value, when determining the first weight of the first frame image, the first weight is cached in the cache, and when determining the kth weight of the kth frame image, the kth weight is cached in the cache; If the processing rate of the processor is lower than the first value, then when determining the first weight of the first frame image, the first weight is cached in the buffer, and when determining the kth weight of the kth frame image, the kth weight is cached in the buffer, and after weighted summing the kth frame image and the k-1th intermediate image using the normalized kth weight and the normalized k-1th weight, the k-1th weight in the buffer is cleared.

9. The method according to claim 1, characterized in that: Determine the jth weight corresponding to the jth frame image, where j is an integer value from 1 to n, specifically including: The jth weight is obtained according to one or more of the contrast, saturation and brightness of the jth frame image.

10. The method according to claim 1, characterized in that The n is any integer greater than or equal to 3.

11. An electronic device, characterized in that: The method comprises one or more memories, one or more processors, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 10.

12. A chip system, characterized in that: The chip system includes a cache, a buffer, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images