Multi-frame image fusion method and electronic equipment

Abstract

The invention provides a multi-frame image fusion method and electronic equipment. And the electronic equipment generates a frame of fusion image by taking every n frames of images in the plurality of frames of images which are continuously acquired as a round. Specifically, the electronic equipment sequentially obtains a first frame image to an nth frame image, and when the first frame image is obtained, a first weight corresponding to the first frame image is calculated; the kth frame of image is continuously obtained and the kth weight (k is an integer from 2 to n in sequence, n is greater than or equal to 2) corresponding to the kth frame of image is calculated, the normalized k-1 weight and kth weight are adopted to perform weighted summation on the k-1 intermediate image and the kth frame of image to obtain a kth intermediate image, when k is equal to 2, the k-1 intermediate image is the first frame of image, and when k is equal to n, the kth intermediate image is the second frame of image. The k-th intermediate image is the fusion image of the current round, and the electronic equipment further outputs the fusion image. Wherein every time a new intermediate image is calculated, the new intermediate image replaces the previous intermediate image to be cached into the buffer.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to a method and electronic device for multi-frame image fusion. Background Art

[0002] Multi-frame image fusion technology requires hardware support in electronic devices to buffer multi-frame images. Specifically, taking the example of electronic devices using image signal processors (ISPs) to perform multi-frame image fusion, it is necessary to pre-design an ISP with a large storage space. This storage space can be used by the ISP to cache the latest multi-frame images in real time to provide data support for subsequent fusion.

[0003] Based on this, how to optimize the method of multi-frame image fusion to ensure the real-time performance of image fusion and reduce the cost of hardware storage. Summary of the invention

[0004] The present application provides a method and electronic device for multi-frame image fusion. The electronic device uses every n frames of images in the continuously acquired multi-frame images as a round to generate a frame of fused image. That is, every time a new frame of image is acquired, the new image is fused with the intermediate result of the previous frame to obtain a new intermediate result, until the last frame of image in a round is acquired, the last frame of image and the intermediate result of the previous frame are fused, and the intermediate result of the last frame obtained is the final fused image obtained in this round. Such an iterative multi-frame image fusion method only needs to cache the intermediate result of the most recent frame, which reduces the occupancy of the cache space, and also disperses the calculation amount of the electronic device each time a new frame of image is acquired, rather than concentrating it on the acquisition of the last frame of image. Therefore, the computing resources of the electronic device can also be reasonably used.

[0005] In a first aspect, the present application provides a method for fusion of multiple frames of images, the method comprising: sequentially acquiring the 1st frame image to the nth frame image; when the 1st frame image is acquired, determining the 1st weight of the 1st frame image; when the kth frame image is acquired, k sequentially takes integer values ​​from 2 to n, determining the kth weight of the kth frame image, normalizing the kth weight and the k-1th weight, using the normalized kth weight and the normalized k-1th weight to perform weighted summation on the kth frame image and the k-1th intermediate image to obtain the kth intermediate image, clearing the k-1th intermediate image cached in a buffer, and caching the kth intermediate image in the buffer; when k=2, the k-1th intermediate image is the 1st frame image. Outputting the nth intermediate image, the nth intermediate image is an HDR image.

[0006] In implementing the method provided in the first aspect, first, since the amount of calculation for multi-frame image fusion is dispersed when each frame of image arrives, rather than performing a large number of operations when the last frame of image arrives as in traditional multi-frame image fusion, and weight calculation is performed for each frame of image, the amount of calculation for weight normalization and weighted summation is fixed, which facilitates the hardware to perform stable and real-time pipeline implementation to avoid hardware instantaneous execution of high operations. Secondly, specifically, since each new intermediate image frame is generated, the previous intermediate image frame will be cleared and the new intermediate image will be stored, so that in each round of image fusion process, at least one frame of image storage space is provided to cache the latest intermediate image.

[0007] In combination with the method described in the first aspect, the method includes: sequentially acquiring the n+1th frame image to the 2nth frame image; when the n+1th frame image is acquired, determining the n+1th weight of the n+1th frame image; when the n+kth frame image is acquired, k sequentially takes integer values ​​from n+2 to 2n to determine the n+kth weight of the n+kth frame image, normalizing the n+kth weight and the n+k-1th weight, using the normalized n+kth weight and the normalized n+k-1th weight to perform weighted summation on the n+kth frame image and the n+k-1th intermediate image to obtain the n+kth intermediate image, clearing the n+k-1th intermediate image cached in the buffer, and caching the n+kth intermediate image in the buffer; when k=2, the n+k-1th intermediate image is the n+1th frame image. The 2nth intermediate image is output, and the 2nth intermediate image is an HDR image. The nth intermediate image and the 2nth intermediate image are two consecutive frames of images in the output video.

[0008] In this way, in a scenario where HDR video is continuously input, the method for iteratively fusing HDR images with multiple frames provided in the present application can output an HDR stream in real time.

[0009] In combination with the method described in the first aspect, before acquiring the first frame of image, the method includes: receiving an operation of selecting a high dynamic range HDR mode.

[0010] In combination with the method described in the first aspect, the brightness range of the nth intermediate image is greater than the brightness range of each frame image from the first frame image to the nth frame image.

[0011] In this way, the brightness range of the image can be improved through image fusion, and high-quality images can be output.

[0012] In combination with the method described in the first aspect, the method is applied to an electronic device, which includes a cache and the buffer, and the method also includes: when the first frame image is acquired, caching the first frame image in the cache or the buffer; when the kth frame image is acquired, clearing the k-1th frame image in the cache and caching the kth frame image in the cache, or, when the kth frame image is acquired, clearing the k-1th frame image in the buffer and caching the kth frame image in the buffer.

[0013] In this way, each time a new frame of image is acquired, the new frame of image can be stored for intermediate fusion processing, and then the previous frame of image is cleared when the next new frame of image arrives, so that useless images can be cleared in time and space can be saved.

[0014] In combination with the method described in the first aspect, the method is applied to an electronic device, which includes a processor, a cache and the buffer. If the processing rate of the processor is higher than a first value, then when the first frame image is acquired, the first frame image is cached in the cache, and when the kth frame image is acquired, the k-1th frame image in the cache is cleared and the kth frame image is cached in the cache; if the processing rate of the processor is lower than the first value, then when the first frame image is acquired, the first frame image is cached in the buffer, and when the kth frame image is acquired, the k-1th frame image in the buffer is cleared and the kth frame image is cached in the buffer.

[0015] In this way, if the processor performance is better and the data processing rate is fast enough, reading data from the buffer for processing may not be able to fully utilize the processor's performance. Therefore, each newly acquired frame of image can be cached in the cache, because the cache can support higher read and write rates, which can reduce the buffer occupancy and increase the rate of image fusion.

[0016] In combination with the method described in the first aspect, the method is applied to an electronic device, which includes a cache and the buffer, and the method also includes: when determining the first weight of the first frame image, caching the first weight in the cache or the buffer; when determining the kth weight of the kth frame image, caching the kth weight in the cache or the buffer, and after using the normalized kth weight and the normalized k-1th weight to perform weighted summation on the kth frame image and the k-1th intermediate image, clearing the k-1th weight in the cache.

[0017] In this way, each time a new weight for a new frame of image is determined, the new weight can be stored for intermediate state fusion processing, and then the weight of the previous frame of image can be cleared when the new weight for the next frame of image is determined. This way, useless weights can be cleared in time and space can be saved.

[0018] In combination with the method described in the first aspect, the method is applied to an electronic device, which includes a processor, a cache and the buffer. If the processing rate of the processor is higher than the first value, then when determining the first weight of the first frame image, the first weight is cached in the cache, and when determining the kth weight of the kth frame image, the kth weight is cached in the cache; if the processing rate of the processor is lower than the first value, then when determining the first weight of the first frame image, the first weight is cached in the buffer, and when determining the kth weight of the kth frame image, the kth weight is cached in the buffer, and after weighted summing the kth frame image and the k-1th intermediate image using the normalized kth weight and the normalized k-1th weight, the k-1th weight in the buffer is cleared.

[0019] In this way, if the processor performance is better and the data processing rate is fast enough, reading data from the buffer for processing may not be able to fully utilize the processor's performance. Therefore, the weight of each newly determined frame of image can be cached in the cache, because the cache can support higher read and write rates, which can reduce the buffer occupancy and increase the rate of image fusion.

[0020] In combination with the method described in the first aspect, determine the first weight corresponding to the jth frame image, where j takes integer values ​​from 1 to n in sequence, specifically including: obtaining the jth weight based on one or more of the contrast, saturation and brightness of the jth frame image.

[0021] In this way, the proportion of the frame image in the fused image can be measured based on one or more information in the image, thereby improving the quality of the fused image.

[0022] In combination with the method described in the first aspect, n is any integer value greater than or equal to 2. When the value of n is larger, more storage space can be saved, and more allocation space of the amount of calculation that can be optimized is obtained.

[0023] In a second aspect, the present application provides an electronic device, comprising a cache and a buffer, one or more processors, and a computer program stored on the memory, wherein the processor executes the computer program to perform the method as described in any one of the aforementioned first aspects.

[0024] In a third aspect, the present application provides a chip system comprising a cache, a buffer, a processor, and a computer program stored in a memory, wherein the processor executes the computer program to implement a method as described in any one of the first aspects above.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0026] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method as described in any one of the first aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of an application scenario for generating an HDR image based on multi-frame image fusion provided in an embodiment of the present application; Figure 2 A schematic diagram of a traditional multi-frame image fusion method provided in an embodiment of the present application; Figure 3 A schematic diagram of an iterative multi-frame image fusion method provided in an embodiment of the present application; Figure 4 A schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present application; Figure 5 A schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. Mentioning "embodiment" in the present application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application may be combined with other embodiments.

[0029] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form that the user can receive. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0030] At present, more and more scenes are applying multi-frame image fusion methods. This application temporarily only takes the scene of high dynamic range (HDR) image generation as an example to introduce the multi-frame image fusion method. For the introduction of application scenarios, please refer to the following Figure 1 Description.

[0031] Figure 1 An application scenario of generating an HDR image based on multi-frame image fusion is exemplified.

[0032] like Figure 1 As shown, first, the electronic device can use a camera to continuously shoot multiple frames of images to obtain an original image stream, and the original image stream includes but is not limited to a first original image, a second original image, an nth original image, etc. Then, the graphics processing unit (GPU) in the electronic device can fuse the n frames of original images newly acquired each time into one frame of HDR image in real time based on the acquired original image stream, thereby outputting an HDR image stream in real time, and the HDR image stream includes but is not limited to the first HDR image and the second HDR image.

[0033] Among them, the multiple frames of images included in the original image stream correspond to different shooting parameters. For example, when the camera turns on functions such as automatic exposure, the exposure parameters used when shooting each frame of the image are different, resulting in different contrast, saturation, brightness, etc. in each frame of the image. In addition, the name of the original image is only an example. "Original" can refer to the unprocessed image captured by the camera in the electronic device, or it can be an image obtained through preprocessing based on the unprocessed image. In this application, "original" mainly refers to the image before the fusion process is performed, and it does not limit whether it has undergone other processing except image fusion processing.

[0034] Among them, one or more information such as contrast, saturation and brightness of the HDR image is richer than one or more information such as contrast, saturation and brightness of each frame of the original image used to fuse and generate the HDR image. This is because one frame of HDR image is obtained by fusing n frames of original images, and one or more information such as contrast, saturation and brightness in each of the n frames of original images have made a certain contribution to the HDR image. In addition, the name of the HDR image is only an example. "HDR" usually refers specifically to the dynamic range of brightness. In this application, in addition to the expanded brightness range, the image generated after image fusion may also have other information such as saturation contrast. Therefore, this application can also refer to the image generated after image fusion as a high-definition image, etc.

[0035] Figure 1 This is only an example of an application scenario of multi-frame image fusion. In addition, the multi-frame image fusion method provided in this application can also be applied in other scenarios. When the application scenario is different, the factors affecting the weight of each frame image in the multi-frame image fusion method may be different, and the weighted calculation of the multi-frame images may be different, but they all use an iterative calculation method rather than a traditional calculation method. For a detailed introduction to the weighted fusion calculation method, please refer to the following Figures 2 to 3 Description.

[0036] Generally, the traditional multi-frame image fusion method includes the following stages: first, continuously receive multiple frames of images, input each newly received image into a buffer, calculate the weight corresponding to each newly received frame of image, and input the weight corresponding to each newly received frame of image into the buffer. Secondly, take n frames of image as a cycle, where n≥2, and each time a new round of n-th frame of image is received and the weight corresponding to the n-th frame of image is calculated, the weights corresponding to the n-frame images of this round are normalized, and the weighted sum is performed based on the normalized weights corresponding to the n-frame images of this round and the n-frame images, and the result of the weighted sum is a frame of fusion image. Finally, after the image fusion of the n-frame images of this round, the weights corresponding to the n-frame images of this round and the n-frame images cached in the buffer can also be cleared to ensure that the space released by the buffer can cache the complete data required for the next round of image fusion. By analogy, continue to receive multiple frames of image, and perform weighted summation on the n-frame images of the next round to obtain the next frame of fusion image.

[0037] Next, combine Figure 2 , taking n=5 as an example, a traditional multi-frame image fusion method is illustrated.

[0038] like Figure 2As shown, when the GPU in the electronic device receives the first frame image, it will input it into the buffer for caching, and the storage items in the buffer include the first frame image (which can be recorded as I0), and when the GPU in the electronic device receives the first frame image, the operation items executed include calculating the weight of the first frame image, and then the weight of the first frame image is also input into the buffer for caching, and the storage items in the buffer are increased by the weight of the first frame image (which can be recorded as W0). When the GPU in the electronic device receives the second frame image, it will input it into the buffer for caching, and the storage items in the buffer are increased by the second frame image (which can be recorded as I1), and when the GPU in the electronic device receives the second frame image, the operation items executed include calculating the weight of the second frame image, and then the weight of the second frame image is also input into the buffer for caching, and the storage items in the buffer are increased by the weight of the second frame image (which can be recorded as W1). By analogy, when the GPU in the electronic device receives the third frame image (denoted as I2), the fourth frame image (denoted as I3) and the fifth frame image (denoted as I4), it will perform the same calculation operation and storage operation as the first frame image and the second frame image, so that the buffer of the electronic device caches five frames of images (denoted as I0-I4) and the weights corresponding to the five frames of images (denoted as W0-W4). In addition, it is not until the fifth frame image is received that the five frames of weights can be normalized, and based on the normalized five frames of weights and five frames of images, fusion processing (i.e., weighted operation) can be performed to output a frame of fused image, that is, after calculating W4, W0-W4 needs to be normalized, and then the normalized W0-W4 is used to perform weighted summation on I0-I4, so as to output a fused image (denoted as O4). By analogy, the GPU of the subsequent electronic device receives and caches the next round of five frames of images (denoted as I5-I9), calculates and caches the weights corresponding to the next round of five frames of images (denoted as W5-W9), normalizes W5-W9, and performs weighted summation of I5-I9 based on the normalized W5-W9 to output a fused image (denoted as O9). The method is the same as the calculation operations and cache operations corresponding to the execution of the current round of five frames of images described above, and will not be elaborated here.

[0039] Optionally, the GPU can calculate the weight of each frame of image immediately upon receiving it, and after calculating its weight, input the frame of image and the weight into the buffer for caching together, or the GPU can first input the frame of image into the buffer for caching when receiving each frame of image, and then calculate its weight, and finally input the weight into the buffer for caching. Among them, the calculation method of image weights can refer to the description below, which will not be repeated here. Optionally, in addition to using GPU to perform image fusion in electronic device processing, other types of processors can also be used, and the buffer can be a memory integrated in the GPU or a memory independent of the GPU, which is not limited to the embodiments of the present application.

[0040] Figure 2 Only n=5, i.e., generating one fused image from five frames of images, is used as an example, and only two rounds of image fusion (i.e., continuously outputting two fused images) are used as an example. In addition, the traditional image fusion method can also be n equal to other values, or only support the output of one fused image, or support the output of multiple fused images, i.e., fused image streams. Figure 2 The content shown should not be construed as limiting the present application.

[0041] By analyzing Figure 2 It can be seen from the traditional image fusion method shown that: in the process of the GPU of the electronic device performing each round of image fusion, before receiving the last frame of the image in each round, the GPU's computational load is relatively small. For example, the GPU's computational load is dispersed when each frame of the image before the last frame of the image arrives, and only the weight corresponding to each frame of the image is calculated. Most of the GPU's computational load is concentrated on receiving the last frame of the image in each round. For example, it is necessary to calculate the weight of the last frame of the image, calculate the normalized result of the weights of all images, and calculate the weighted sum of all images, etc. This will cause a long output delay for each round of image fusion. In the scenario of real-time output of the fused image stream, its real-time performance is poor, affecting the user experience. In addition, before the GPU performs the weighted sum of all images, all images and weights of this round need to be stored in the buffer, which will increase the design cost of hardware storage.

[0042] As users have higher requirements for fused images, the GPU in electronic devices needs to generate a fused image based on more frames of original images to ensure the quality of the fused image. In other words, in order to improve the quality of the fused image, the value of n needs to be increased. As the value of n increases (for example, n ≥ 3), the buffer needs to cache more data, which in turn requires a greater buffer space. In addition, as the value of n increases, after receiving the nth frame of each round, the GPU needs to calculate more data, which in turn affects the real-time performance of multi-frame image fusion to a greater extent.

[0043] In order to solve the above technical problems, the present application provides a method and electronic device for multi-frame image fusion. The electronic device adopts an iterative calculation method, takes each n-frame image in the continuously acquired multi-frame image as a round to generate a frame of fused image, and then generates m-frame fused images (m≥1) in m consecutive rounds. Among them, the process of generating a frame of fused image in any round includes: the electronic device sequentially acquires the 1st to n-th frame images, and when the first frame image is acquired, calculates the corresponding first weight. By analogy, continue to acquire the k-th frame image and calculate the k-th weight corresponding to the k-th frame image (k is an integer from 2 to n in turn, n≥2), and use the normalized k-1th weight and k-th weight to perform weighted summation on the k-1th intermediate image and the k-th frame image to obtain the k-th intermediate image, wherein when k=2, the k-1th intermediate image is the first frame image, and when k=n, the k-th intermediate image is the fused image of this round, and the electronic device also outputs the fused image. Among them, each new intermediate image calculated replaces the previous intermediate image to be cached in the buffer.

[0044] Wherein, n can be any integer greater than or equal to 2. When n is equal to 2, the iterative image fusion method provided by the present application is the same as the traditional image fusion method. Therefore, when n is any integer greater than or equal to 3, the iterative image fusion method provided by the present application can have the following beneficial effects compared with the traditional image fusion method.

[0045] (1) Improve the real-time performance of multi-frame image fusion. Specifically, the calculation of multi-frame image fusion is dispersed when each frame arrives, instead of performing a large number of operations when the last frame arrives as in traditional multi-frame image fusion. In addition, the weight calculation, weight normalization and weighted summation are all fixed for each frame, which facilitates the hardware to perform stable and real-time pipeline implementation and avoids the hardware from performing high operations instantaneously.

[0046] (2) Reduce hardware storage costs. Specifically, each time a new intermediate image is generated, the previous intermediate image is cleared and the new intermediate image is stored. In this way, in each round of image fusion, at least one frame of image storage space is provided to cache the latest intermediate image. If the hardware computing power of the electronic device is insufficient, another frame of image storage space can be provided to cache the latest acquired image, instead of providing at least storage space for caching n frames of images and n weights in traditional multi-frame image fusion, where each weight is actually a frame of weight image, that is, a frame of weight image contains the weight value corresponding to each pixel.

[0047] (3) When using a deep learning network model to fuse multiple frames of images, an iterative image fusion method can also be used, which can also save the buffer space for the original image. Except for the specific calculation method, the form of real-time pipeline and the use of buffer space are basically the same.

[0048] Next, combine Figure 3 , taking n=5 as an example, an iterative multi-frame image fusion method is illustrated.

[0049] like Figure 3 As shown, when the GPU in the electronic device receives the first frame image (which can be denoted as I0), the executed operation items include calculating the first weight of the first frame image (which can be denoted as W0). Since the first frame image is used as the first intermediate image (which can be denoted as O0) for subsequent intermediate image fusion, the first intermediate image is also input into the buffer for caching, and the storage items in the buffer include the first intermediate image.

[0050] Continue to refer Figure 3 , when the GPU in the electronic device receives the second frame image (which can be recorded as I1), the operation items to be executed include calculating the second weight of the second frame image (which can be recorded as W1), calculating the normalized result of the first weight of the first frame image and the second weight of the second frame image, and using the normalized first weight and second weight to perform weighted summation on the first intermediate image and the second frame image to obtain the second intermediate image (which can be recorded as O1). Since the second intermediate image is used for subsequent intermediate image fusion, the first intermediate image in the buffer is also cleared, and the second intermediate image is input into the buffer for caching. The storage items in the buffer include the second intermediate image. Optionally, when the GPU in the electronic device receives the second frame image, it can be temporarily input into the buffer for caching, or it can be directly operated on, depending on the performance of the GPU. When the performance of the GPU is high, its computing power is strong, and it is preferred that data can be quickly read and written from the cache memory (cache) instead of reading and writing data from the buffer, so there is no need to input the second frame image into the buffer for caching.

[0051] Continue to refer Figure 3, and so on, when the GPU in the electronic device receives the third frame image (denoted as I2), the fourth frame image (denoted as I3) and the fifth frame image (denoted as I4), it will perform the same calculation operation and storage operation as the aforementioned first frame image and second frame image, so that the electronic device calculates its weight based on the current newly received image each time, and normalizes the new weight and the weight of the previous frame image, and uses the normalized new weight and the weight of the previous frame image to perform weighted summation on the previous frame intermediate image and the current new image to obtain a new intermediate image, and stores the new intermediate image. Optionally, when the performance of the electronic device is low, in addition to storing the current new intermediate image in the buffer, the next frame new image will be stored in the buffer after the next frame new image arrives and before the new intermediate image is generated; when the performance of the GPU is high, its computing power is strong, and it is preferred to quickly read and write data from the cache memory (cache) instead of slowly reading and writing data from the buffer, so there is no need to input the second frame image into the buffer for caching. Optionally, after the electronic device receives the last frame of image, i.e., the fifth image, if a fifth intermediate image (which may be denoted as O4) is calculated, the fifth intermediate image may be output as the final fused image without being stored in a buffer.

[0052] Continue to refer Figure 3 After the electronic device receives the previous round of five frames of images (denoted as I0-I4) and performs a round of image fusion to output a fused image (denoted as O4), if the subsequent electronic device continues to receive the next round of five frames of images (denoted as I5-I9), it performs calculation and storage operations similar to the previous round of five frames of images, thereby outputting a fused image (denoted as O9), which will not be described in detail here.

[0053] Figure 3 Only an example is used to illustrate that n=5, that is, five frames of images generate one frame of fused image. In addition, the iterative image fusion method can also be applied to other n values. Figure 3 Only two rounds of image fusion (i.e., continuous output of two fused images) are used as an example. In addition, the iterative image fusion method can also be applied to output scenarios that only support one fused image frame, or output scenarios that support multiple fused images, i.e., output scenarios that fused image streams. Figure 3 The content shown should not be construed as limiting the present application.

[0054] Regardless of how much n is equal to, the iterative multi-frame image fusion method provided in this application uses the calculation formula shown below.

[0055] The formula for normalized weights is: , .

[0056] Among them, when only the first frame image (denoted as I0) arrives, then When the second frame image (denoted as I1) arrives, .

[0057] The formula for fusing images is: , .

[0058] Among them, when only the first frame image (I0) arrives, then , that is, the input image It also serves as the first intermediate image generated by fusion. .

[0059] Among them, when the last frame, that is, the nth frame image (In), arrives, That is the final fused image to be output.

[0060] In the above formula, Represents the frame order of the original image to be input, represents the pixel position in the image, Represents the number of original input images required to generate a fused image. Representative The pixel information of the original image input sequentially, Representative The pixel information of the intermediate images output sequentially.

[0061] By analyzing Figure 3 It can be seen from the iterative image fusion method shown that: in the process of executing each round of image fusion by the GPU of the electronic device, a large amount of calculations will not be concentrated when a certain frame of image arrives, but when receiving each frame of image in each round, the amount of calculation of the GPU is almost the same. For example, when other frame images except the first frame arrive, the weight of the new frame image will be calculated, and the normalized result of the weight of the previous frame image, and the new intermediate image calculated using the normalized result will be calculated. This can facilitate the stable and real-time pipeline implementation of the hardware, improve the real-time performance of multi-frame image fusion, and avoid increasing the delay of image fusion due to the instantaneous execution of high calculations by the hardware. In addition, in the process of executing each round of image fusion using a high-performance GPU, only a buffer sufficient to store one frame of image is required, and in the process of executing each round of image fusion using a low-performance GPU, only a buffer sufficient to store two frames of image is required. All images and weights of each round are no longer stored in the buffer as in the traditional method, thereby greatly reducing the design cost of hardware storage.

[0062] Next, a method for calculating the weights of images involved in image fusion is exemplified.

[0063] The basic principle of image fusion is to combine the information of multiple frames of images taken with different shooting parameters (such as different exposure parameters) to generate a high-quality image containing key information and features in multiple frames of images. In the multiple frames of images, the amount of key information contained in each frame of the original image to be fused is different, so weights are needed to measure the amount of key information contained in each frame of the original image to be fused. In other words, the weight of the image refers to the process of adjusting the relative contribution of the original image before fusion in the final fused image during the process of fusion based on the original image to generate a fused image.

[0064] In an optional manner, the weight calculation method can be calculated based on any one or more of the following information: image contrast, image saturation, and image brightness. The following text will only temporarily introduce the weight calculation by combining the three information as an example. In other optional methods, the weight calculation can also be based on other information used as needed, and the weight calculation method should not constitute a limitation to this application.

[0065] Contrast describes the degree of difference between different grayscales in an image. High contrast means that there is a large difference in grayscale in the image (i.e., the gradient information is large), which usually makes the image look more vivid and clear, while low contrast means that the difference in grayscale is small (i.e., the gradient information is small), which usually makes the image look blurry or lack of layering. Taking a 256-bit deep image as an example, the grayscale values ​​of the pixels contained in the overexposed area of ​​the image are almost all 255, and the grayscale values ​​of the pixels contained in the underexposed area of ​​the image are almost all 0. The grayscale values ​​in the overexposed and underexposed areas are very smooth, and the gradient information is almost 0. These areas lack detail information and therefore need to be discarded, that is, the weight value set corresponding to these areas in the weight map is 0. Therefore, the Laplace operator can be used to extract the gradient information of the image to characterize the contrast of the image. The smaller the gradient information, the smaller the proportion of the pixel value at the pixel point when the image is fused. On the contrary, the larger the gradient information, the larger the proportion of the pixel value at the pixel point when the image is fused.

[0066] The calculation formula of image contrast is: .in, Represents the frame order of the original image to be input, represents the pixel position in the image, Represents the number of original input images required to generate a fused image. Representative The pixel information of the original image input sequentially, Representative The pixel information of the grayscale image corresponding to the original images input sequentially, represents the Laplace operator, Representative The contrast information of the images output sequentially, stands for Laplace operator.

[0067] Saturation describes the purity or vividness of the colors in an image. The higher the saturation, the more vivid the colors appear, and the lower the saturation, the closer the colors appear to gray or white. When capturing an image, high exposure results in unsaturated colors in the captured image. Saturation is calculated by calculating the standard deviation of the R, G, and B channels for each pixel in the image.

[0068] The calculation formula of image contrast is: , .in, Represents the frame order of the original image to be input, represents the pixel position in the image, Represents the number of original input images required to generate a fused image. Representative The pixel information of the R channel of the original image input sequentially, Representative The pixel information of the G channel of the original image input sequentially, Representative The pixel information of the B channel of the original image input sequentially, Representative The mean of the R, G, and B channels of the original images input sequentially, Representative The saturation information of the images output sequentially.

[0069] Brightness describes the degree of light and darkness in an image. Taking an image with a 256-bit depth as an example, areas with pixel values ​​close to 255 in the image are overexposed, areas with pixel values ​​close to 0 in the image are underexposed, and areas with pixel values ​​close to 128 are well exposed. Therefore, the weights of pixels with pixel values ​​close to 128 can be set higher, and the pixel values ​​of pixels far from 128, that is, close to 255 or 0, can be set lower.

[0070] Optionally, the calculation formula of the image brightness can be measured using a Gaussian curve, and the specific formula is, for example: .

[0071] in, Represents the frame order of the original image to be input, represents the pixel position in the image, Represents the number of original input images required to generate a fused image. Representative The pixel information of the R channel of the original image input sequentially, Representative The pixel information of the G channel of the original image input sequentially, Representative The pixel information of the B channel of the original image input sequentially, Representative The meaning of 0.5 is that in image processing, pixel values ​​are usually normalized to the range of [0,1]. Therefore, the distance of the pixel value from 0.5, that is, the distance of the pixel value from 128, can reflect the brightness level of the pixel. If the pixel value is close to 0.5, it means that the brightness of the pixel is moderate. If the brightness value is far from 0.5, it means that the pixel is too dark or too bright. The meaning of σ equal to 0.2 is that in the Gaussian curve, σ determines the width of the curve. The larger σ is, the wider the curve is, indicating that the influence range of the pixel value is larger; the smaller σ is, the narrower the curve is, indicating that the influence range of the pixel value is smaller. In this example, σ is equal to 0.2, which means that the Gaussian curve is relatively narrow, and only has a greater impact on pixels with brightness values ​​close to 0.5, and has a smaller impact on pixels far from 0.5. This can more accurately evaluate the exposure of each pixel. In addition, since color images are usually composed of multiple color channels (such as RGB channels), each of which contains pixel values, multiple color channels are also taken into account when calculating the image brightness. Gaussian curves are applied to each color channel separately to comprehensively consider the brightness information of all color channels, and then the results are multiplied to obtain a measure: Multiplying the Gaussian curve results of each color channel can obtain a comprehensive measure This standard reflects the exposure of the image in each color channel. If the exposure of a color channel is poor (that is, the Gaussian curve result is small), the exposure of the entire image will also be affected.

[0072] In summary, using brightness-based The distance from 0.5 uses a Gaussian curve with σ equal to 0.2. Consider applying Gaussian curves to multiple color channels and multiplying the results to obtain the metric. The method is used to calculate the exposure of the image, which can comprehensively consider the brightness information of the image in each color channel and accurately evaluate the exposure of each pixel.

[0073] According to the above, the weight of each pixel in the multi-frame image can be obtained based on the three aspects of contrast, saturation and brightness. The formula for obtaining the weight by combining contrast, saturation and brightness is: In addition, in order to avoid the problem of too high or too low pixel values ​​in multi-frame image fusion, the weights Normalize.

[0074] Next, the types of electronic devices and software and hardware architectures involved in the multi-frame image fusion method provided in this application are introduced in detail.

[0075] The electronic device may be a portable terminal device running iOS®, Android®, Microsoft® or other operating systems, such as a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device and / or a smart city device, etc.

[0076] Figure 4 A schematic diagram of the hardware architecture of the electronic device 100 is shown.

[0077] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a sensor module 180, a camera 193, a display screen 194, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180K, etc.

[0078] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0079] The processor 110 may include one or more processing units, for example, the processor 110 may include a graphics processing unit (GPU), an image signal processor (ISP), an application processor (AP), a modem processor, a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0080] In the embodiment of the present application, the GPU included in the processor 110 is hardware specifically used to perform image-related calculations, mainly used to accelerate the image processing process. For example, the GPU is mainly used to perform image fusion, render the fused image and send it to the display screen for display. For the specific implementation method of GPU performing image fusion, please refer to the above description of Figure 3 The introduction of will not be elaborated here.

[0081] In an embodiment of the present application, a memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory (Cache). The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system. For example, the cache memory (Cache) may store the weights of the images mentioned above and the latest received images, etc., which can facilitate the CPU and GPU to quickly process images and reduce the design cost of the buffer.

[0082] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0083] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microled, micro-oled, quantum dot light emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0084] In the embodiment of the present application, the electronic device 100 can realize the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connecting the display screen 194 and the CPU, AP, etc. The GPU can receive the image stream continuously transmitted by the CPU or AP, and after fusing multiple frames of images, output a frame of fused image or fused image stream to the display screen.

[0085] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0086] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0087] In the embodiment of the present application, the electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor, etc. Specifically, the image stream continuously shot by the camera 193 can be sent to the application processor after being pre-processed by the ISP, and then sent to the GPU for image fusion, and the finally generated fused image stream is then sent to the display screen 194 for display.

[0088] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The non-volatile memory can also store executable programs and store user and application data, and can be loaded into the random access memory in advance for direct reading and writing by the processor 110.

[0089] In an embodiment of the present application, the buffer is located in the RAM, and the buffer is used to cache intermediate images in the image fusion process. Optionally, when the CPU and GPU performance are low (for example, the rate of processing data is lower than the first value), the buffer is also used to cache the input original image and / or weight. When the CPU and GPU performance are high (for example, the rate of processing data is higher than the first value), the Cache between the CPU and RAM is used to cache the input original image and / or weight.

[0090] Among them, cache is a small-capacity but extremely fast memory between the central processing unit and the main memory. It is mainly used to solve the speed difference problem between the CPU and the main memory, thereby improving the overall performance of the system. The design and management of cache is crucial to improving the overall performance of the computer system. In a computer system, cache is usually composed of static random access memory (SRAM), which is faster than dynamic random access memory (DRAM), but also more expensive. Therefore, the capacity of cache is relatively small, but its speed is very fast and can quickly respond to the data request of the CPU.

[0091] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0092] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0093] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0094] In an embodiment of the present application, the electronic device may receive an operation for capturing an image, or receive an operation for selecting an HDR mode, etc., through a pressure sensor 180A or a touch sensor 180K. The embodiment of the present application does not limit this and does not elaborate on this.

[0095] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android® system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0096] Figure 5 Schematic diagram of the software architecture of the electronic device 100 according to an embodiment of the present application.

[0097] like Figure 5 As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android® system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the hardware abstraction layer, the kernel layer, and the hardware layer.

[0098] The application layer can include a series of application packages. For example, an application package can include applications such as a camera.

[0099] Among them, the camera application includes but is not limited to: a shooting module, which includes an HDR mode, a portrait mode or others. The shooting module is used to execute the photo and video recording functions of the camera application, including outputting a preview interface in the shooting mode and taking photos or recording videos after the shooting control is pressed. When the user selects the HDR mode, the electronic device executes the iterative multi-frame image fusion method described above to output HDR photos or HDR videos. In addition, the method for generating HDR images by multi-frame image fusion provided in the present application can be used in camera applications as well as in other image processing applications, and the embodiments of the present application are not limited to this.

[0100] The application framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a camera service, a view system, and so on.

[0101] The camera service may also be referred to as a camera access interface and a camera device, which is used to provide an application programming interface and a programming framework for camera applications.

[0102] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0103] The hardware abstraction layer is an interface layer located between the application framework layer and the kernel layer, providing a virtual hardware platform for the operating system.

[0104] In the embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0105] The Camera Hardware Abstraction Layer (Camera HAL) can provide virtual hardware for one or more camera devices, such as virtual hardware for Camera Device 1 and Camera Device 2. Camera HAL can obtain image data from the underlying camera and image signal processor and transmit it to the camera algorithm library. In addition, Camera HAL can also obtain information from the camera algorithm library. Camera HAL is mainly used to control the corresponding sensor to collect images.

[0106] The camera algorithm library may include one or more algorithm modules, for example, an algorithm module for performing image fusion based on multiple frames of images captured by a camera, thereby realizing an HDR mode, etc., wherein the algorithm module for image fusion realizes the aforementioned Figure 3 Module for the method shown.

[0107] The kernel layer is a layer between hardware and software. The kernel layer includes drivers for various hardware. In some embodiments, the kernel layer may include a camera device driver, a digital signal processor driver, and an image processor driver. Among them, the camera device driver is used to drive the sensor of the camera to collect images and drive the image signal processor to pre-process the image. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images. The kernel layer is a layer between hardware and software.

[0108] The hardware layer includes sensors such as camera 1, camera 2, TOF, multi-spectral sensor, image signal processor, digital signal processor and image processor, etc.

[0109] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0110] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0111] It should be understood that each step in the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0112] The present application also provides an electronic device, which may include a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the method executed by the electronic device in any one of the above embodiments.

[0113] The present application also provides a chip system, including a processing circuit and an interface circuit, the interface circuit is used to receive computer instructions and transmit them to the processing circuit, and the processing circuit is used to run the computer instructions to implement a method executed by an electronic device as in any of the above embodiments.

[0114] The present application also provides a chip system, which includes at least one processor, for implementing the method executed by the electronic device in any of the above embodiments. In a possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0115] A chip system can be composed of chips or include chips and other discrete devices.

[0116] Optionally, there may be one or more processors in the chip system. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0117] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0118] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0119] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method executed by the electronic device in any one of the above embodiments is implemented.

[0120] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the method executed by the electronic device in any of the above embodiments.

[0121] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0122] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0123] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0124] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0125] The terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0126] In short, the above is only an embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of this application shall be included in the protection scope of this application.

Claims

1. A method for multi-frame image fusion, characterized in that: The method comprises: Sequentially acquire the first frame image to the nth frame image; When the first frame of image is acquired, determining a first weight of the first frame of image; When the k-th frame image is acquired, k takes integer values ​​from 2 to n in sequence, determines the k-th weight of the k-th frame image, normalizes the k-th weight and the k-1-th weight, uses the normalized k-th weight and the normalized k-1-th weight to perform weighted summation on the k-th frame image and the k-1-th intermediate image to obtain the k-th intermediate image, clears the k-1-th intermediate image cached in the buffer, and caches the k-th intermediate image in the buffer; when k=2, the k-1-th intermediate image is the 1st frame image; An nth intermediate image is output, where the nth intermediate image is a high dynamic range HDR image.

2. The method according to claim 1, characterized in that: The method comprises: Sequentially acquire the n+1th frame image to the 2nth frame image; When the n+1th frame image is acquired, determining the n+1th weight of the n+1th frame image; When the n+kth frame image is acquired, k takes integer values ​​from n+2 to 2n in sequence, determines the n+kth weight of the n+kth frame image, normalizes the n+kth weight and the n+k-1th weight, uses the normalized n+kth weight and the normalized n+k-1th weight to perform weighted summation on the n+kth frame image and the n+k-1th intermediate image to obtain the n+kth intermediate image, clears the n+k-1th intermediate image cached in the buffer, and caches the n+kth intermediate image in the buffer; when k=2, the n+k-1th intermediate image is the n+1th frame image; A 2nth intermediate image is output, where the 2nth intermediate image is an HDR image, and the nth intermediate image and the 2nth intermediate image are two consecutive frames of images in the output video.

3. The method according to claim 1 or 2, characterized in that: Before acquiring the first frame of image, the method includes: An operation of selecting a high dynamic range (HDR) mode is received.

4. The method according to claim 1, characterized in that The brightness range of the nth intermediate image is greater than the brightness range of each frame image from the first frame image to the nth frame image.

5. The method according to claim 1, characterized in that: The method is applied to an electronic device, the electronic device comprising a cache memory and the buffer, and the method further comprises: When the first frame of image is acquired, caching the first frame of image in the cache or the buffer; When the k-th frame image is obtained, the k-1-th frame image in the cache is cleared and the k-th frame image is cached in the cache; or, when the k-th frame image is obtained, the k-1-th frame image in the buffer is cleared and the k-th frame image is cached in the buffer.

6. The method according to claim 5, characterized in that The method is applied to an electronic device, wherein the electronic device comprises a processor, a cache and the buffer. If the processing rate of the processor is higher than the first value, when the first frame image is acquired, the first frame image is cached in the cache, and when the kth frame image is acquired, the k-1th frame image in the cache is cleared, and the kth frame image is cached in the cache; If the processing rate of the processor is lower than the first value, when the first frame image is acquired, the first frame image is cached in the buffer, and when the kth frame image is acquired, the k-1th frame image in the buffer is cleared and the kth frame image is cached in the buffer.

7. The method according to claim 1, characterized in that The method is applied to an electronic device, the electronic device includes a cache and the buffer, and the method further includes: When determining the first weight of the first frame image, caching the first weight in the cache or the buffer; When determining the kth weight of the kth frame image, the kth weight is cached in the cache or the buffer, and after weighted summing the kth frame image and the k-1th intermediate image using the normalized kth weight and the normalized k-1th weight, the k-1th weight in the cache is cleared.

8. The method according to claim 7, characterized in that The method is applied to an electronic device, wherein the electronic device comprises a processor, a cache and the buffer. If the processing rate of the processor is higher than the first value, when determining the first weight of the first frame image, the first weight is cached in the cache, and when determining the kth weight of the kth frame image, the kth weight is cached in the cache; If the processing rate of the processor is lower than the first value, then when determining the first weight of the first frame image, the first weight is cached in the buffer, and when determining the kth weight of the kth frame image, the kth weight is cached in the buffer, and after weighted summing the kth frame image and the k-1th intermediate image using the normalized kth weight and the normalized k-1th weight, the k-1th weight in the buffer is cleared.

9. The method according to claim 1, characterized in that: Determine the jth weight corresponding to the jth frame image, where j is an integer value from 1 to n, specifically including: The jth weight is obtained according to one or more of the contrast, saturation and brightness of the jth frame image.

10. The method according to claim 1, characterized in that The n is any integer greater than or equal to 2.

11. An electronic device, characterized in that: The method comprises one or more memories, one or more processors, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 10.

12. A chip system, characterized in that: The chip system includes a cache, a buffer, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images