Image processing method and device, equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, equipment, a storage medium and a program product. According to one embodiment of the invention, the method comprises the steps: carrying out the similarity calculation of a plurality of frames of to-be-processed images obtained at present; in response to a plurality of images of which the similarity is greater than or equal to a similarity threshold in the plurality of frames of to-be-processed images, fusing the plurality of images to obtain a fused image; and generating shot images corresponding to the plurality of images based on the fused image, wherein the number of the shot images is smaller than that of the plurality of images. The shooting speed can be increased, the number of repeated shot images can be reduced, the storage space of the equipment is saved, the situation that a user manually deletes repeated images is avoided, and then the user experience can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of terminal technology, and in particular to an image processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of terminal technology, more and more electronic devices (such as mobile phones) support burst shooting. Users can take photos anytime and anywhere using electronic devices, such as capturing scenes with moving objects, like pets playing or sports competitions, thus meeting users' needs to preserve precious memories.

[0003] In related technologies, when a user uses the burst mode to capture a scene, they will find that they have taken many images with almost the same content. However, users usually only want to keep a few relatively clear images. Taking other redundant images will obviously reduce the shooting speed, increase the power consumption of the device, waste the storage space of the device, and require the user to delete them manually, which will greatly affect the user experience. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, the present disclosure provides an image processing method, apparatus, device, storage medium, and program product to solve the defects in the related technologies.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, the method comprising:

[0006] Calculate the similarity of the currently acquired multiple frames of images to be processed;

[0007] In response to the existence of several images with a similarity greater than or equal to a similarity threshold in the multiple frames of images to be processed, the several images are fused to obtain a fused image;

[0008] Based on the fused image, a captured image corresponding to the plurality of images is generated, wherein the number of captured images is less than the number of the plurality of images.

[0009] In some embodiments, the method further includes:

[0010] In response to receiving the raw image output from the image sensor, the raw image is processed based on the real-time pipeline to obtain a real-time pipeline image;

[0011] In response to receiving an image acquisition command, the real-time pipeline image is cached in an image cache queue;

[0012] The similarity calculation of the currently acquired multiple frames of images to be processed includes:

[0013] Multiple frames of the real-time pipeline images are extracted from the image cache queue and used as the multiple frames to be processed for similarity calculation.

[0014] In some embodiments, prior to fusing the plurality of images, the method further includes:

[0015] The images are then subjected to noise reduction processing.

[0016] In some embodiments, fusing the plurality of images to obtain a fused image includes:

[0017] Select one frame image from the plurality of images that meets the image quality requirements, and use it as the reference frame image;

[0018] Determine the inter-frame differences between the reference frame image and other frame images in the plurality of images;

[0019] The fusion weights of other frame images to which it belongs are determined based on the inter-frame differences, and the fusion weights of the same frame image are negatively correlated with the inter-frame differences;

[0020] The fused image is obtained by performing a weighted sum operation on the plurality of images based on the fusion weights.

[0021] In some embodiments, selecting a frame image from the plurality of images that meets the image quality requirements includes:

[0022] Select one frame from the plurality of images whose contrast and / or signal-to-noise ratio meet the corresponding conditions.

[0023] In some embodiments, generating a captured image corresponding to the plurality of images based on the fused image includes:

[0024] The fused image is processed based on the offline pipeline to obtain an offline pipeline image;

[0025] The offline pipeline image is encoded using an image encoder to obtain the captured image.

[0026] In some embodiments, the method further includes:

[0027] In response to the absence of several images with a similarity greater than or equal to the similarity threshold among the multiple frames to be processed, corresponding captured images are generated based on the multiple frames to be processed.

[0028] According to a second aspect of the present disclosure, an image processing apparatus is provided, the apparatus comprising:

[0029] The similarity calculation module is used to calculate the similarity of the currently acquired multiple frames of images to be processed.

[0030] An image fusion module is used to fuse several images in the multiple frames of images to be processed, where the similarity is greater than or equal to a similarity threshold, to obtain a fused image.

[0031] An image generation module is used to generate captured images corresponding to the plurality of images based on the fused image, wherein the number of captured images is less than the number of the plurality of images.

[0032] In some embodiments, the apparatus further includes an image caching module;

[0033] The image caching module includes:

[0034] A real-time processing unit is used to process the raw image output by the image sensor in response to receiving the raw image to obtain a real-time pipeline image based on the real-time pipeline.

[0035] An image caching unit is used to cache the real-time pipeline image into an image caching queue in response to receiving an image acquisition instruction;

[0036] The similarity calculation module is also used to extract multiple frames of the real-time pipeline images from the image cache queue and use them as the multiple frames to be processed for similarity calculation.

[0037] In some embodiments, the apparatus further includes:

[0038] The noise reduction module is used to perform noise reduction processing on the several images respectively.

[0039] In some embodiments, the image fusion module includes:

[0040] An image selection unit is used to select a frame image from the plurality of images that meets the image quality requirements, and use it as a reference frame image;

[0041] A difference determination unit is used to determine the inter-frame differences between the reference frame image and other frame images (excluding the reference frame image) in the plurality of images.

[0042] The weight determination unit is used to determine the fusion weight of other frame images to which it belongs based on the inter-frame differences. The fusion weight of the same frame image is negatively correlated with the inter-frame differences.

[0043] An image fusion unit is used to perform a weighted sum operation on the plurality of images based on the fusion weights to obtain the fused image.

[0044] In some embodiments, the image selection unit is further configured to select a frame image from the plurality of images whose contrast and / or signal-to-noise ratio meet the corresponding conditions.

[0045] In some embodiments, the image generation module includes:

[0046] An offline processing unit is used to process the fused image based on the offline pipeline to obtain an offline pipeline image;

[0047] An image encoding unit is used to encode the offline pipeline image based on an image encoder to obtain the captured image.

[0048] In some embodiments, the apparatus further includes:

[0049] Other generation modules are used to generate corresponding captured images based on the multiple frames of images to be processed, in response to the absence of several images in the multiple frames of images to be processed having a similarity greater than or equal to the similarity threshold.

[0050] According to a third aspect of the present disclosure, an electronic device is provided, the device comprising:

[0051] Processor and memory used to store computer programs;

[0052] The processor is configured to implement the image processing method described above when executing the computer program.

[0053] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the image processing method described in any of the preceding claims.

[0054] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method described in any of the preceding claims.

[0055] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0056] This disclosure calculates the similarity of multiple frames of images to be processed and, in response to the existence of several images in the multiple frames of images to be processed with a similarity greater than or equal to a similarity threshold, fuses the several images to obtain a fused image. Then, based on the fused image, it generates a captured image corresponding to the several images. The number of captured images is less than the number of several images. Because several images with high similarity in the multiple frames of images to be processed are fused, the number of subsequently generated captured images is less than the number of several images. This not only speeds up the shooting process but also reduces the number of duplicate captured images, saves device storage space, and avoids users manually deleting duplicate images, thereby improving the user experience.

[0057] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0058] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0059] Figure 1A This is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure;

[0060] Figure 1B This is a schematic diagram illustrating the effect of an image processing method according to an exemplary embodiment of the present disclosure;

[0061] Figure 2 This is a flowchart illustrating an image processing method according to another exemplary embodiment of the present disclosure;

[0062] Figure 3 This is a flowchart illustrating how to fuse the plurality of images according to an exemplary embodiment of the present disclosure;

[0063] Figure 4A This is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to generate a captured image corresponding to the plurality of images based on the fused image;

[0064] Figure 4B This is a logical schematic diagram of an image processing method according to an exemplary embodiment of the present disclosure;

[0065] Figure 5 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure;

[0066] Figure 6 This is a block diagram illustrating yet another image processing apparatus according to an exemplary embodiment of the present disclosure;

[0067] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0068] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0069] With the development of terminal technology, more and more electronic devices (such as mobile phones) support fast continuous shooting. At present, the competition between different device manufacturers has become fierce. Each manufacturer is constantly iterating and upgrading its technology and competing in terms of supporting fast continuous shooting. Usually, they are increasing the speed of taking a photo or increasing the number of photos taken while ensuring high image quality.

[0070] In implementing the embodiments of this disclosure, the inventors discovered that when a user captures a scene using the burst shooting function, they find that their photo album contains a large number of almost identical images, while the user typically only wants to keep a few relatively clear images. Since image capture involves image acquisition and image processing, capturing a large number of almost identical images obviously reduces shooting speed, increases device power consumption, wastes device storage space, and requires manual deletion by the user, significantly impacting the user experience.

[0071] In view of the above, this disclosure provides the following image processing methods, apparatus, devices, storage media, and program products to solve the aforementioned drawbacks in the related art.

[0072] Figure 1A This is a flowchart illustrating an image processing method according to an exemplary embodiment; the method of this embodiment can be executed by an image processing device, which can be configured in an electronic device (such as a mobile phone, tablet computer, camera, camcorder, etc.). Figure 1A As shown, the method includes the following steps S101-S103:

[0073] In step S101, similarity calculation is performed on the currently acquired multiple frames of images to be processed.

[0074] In this embodiment, after acquiring multiple frames of images to be processed, similarity calculation can be performed on the currently acquired multiple frames of images to be processed.

[0075] The aforementioned multi-frame images to be processed can be multiple original images acquired by the image sensor of an electronic device, or multiple images obtained by preliminary processing of the acquired original images.

[0076] It is worth noting that the above-described method for preliminary processing of the acquired multiple frames of raw images can be arbitrarily set according to the needs of the scenario, and this embodiment does not limit it. In other embodiments, the method for preliminary processing of the acquired multiple frames of raw images may also be as follows. Figure 2 The embodiments shown will not be described in detail here.

[0077] The similarity algorithm used for the above similarity calculation can be selected from relevant technologies as needed, such as an algorithm for calculating image similarity based on mean squared error (MSE), etc. This embodiment does not limit this.

[0078] In step S102, in response to the existence of several images in the multiple frames of images to be processed with a similarity greater than or equal to a similarity threshold, the several images are fused to obtain a fused image.

[0079] In this embodiment, after performing similarity calculation on the currently acquired multiple frames of images to be processed, it is possible to detect whether there are several images in the multiple frames of images to be processed with a similarity greater than or equal to the similarity threshold. If so, these several images can be fused to obtain a fused image.

[0080] For example, when several frames of images with a similarity higher than the similarity threshold are detected in multiple frames to be processed, these frames can be regarded as images with almost identical content taken in a burst. Therefore, these frames can be fused to obtain a fused image, thereby reducing the effect of duplicate images.

[0081] It is worth noting that the above similarity threshold can be set based on actual business needs, such as setting it to 95%, etc. This embodiment does not limit this.

[0082] The above-mentioned method of fusing the images can be selected from relevant technologies according to the needs of the scenario, and this embodiment does not limit it.

[0083] In other embodiments, to further improve the quality of the fused image, the aforementioned images can be denoised separately to obtain denoised images, which can then be fused to obtain a fused image. It is understood that when shooting moving objects in burst mode, the captured multi-frame images may be blurry. By denoising the multi-frame images separately, noise can be effectively removed, the signal-to-noise ratio improved, and thus the quality of the fused image obtained from subsequent image fusion can be enhanced.

[0084] The method for fusing these images described above can be found in the following... Figure 3 The embodiments shown will not be described in detail here.

[0085] In step S103, a captured image corresponding to the plurality of images is generated based on the fused image.

[0086] In this embodiment, after fusing the plurality of images to obtain a fused image, a captured image corresponding to the plurality of images can be generated based on the fused image, wherein the number of captured images is less than the number of the plurality of images.

[0087] In some embodiments, after obtaining the fused image, the fused image can be input into a preset post-processing flow to generate a captured image corresponding to the plurality of images. This embodiment does not limit the form of the post-processing flow.

[0088] In other embodiments, the method of generating captured images corresponding to the plurality of images based on the fused images may also be described below. Figure 4A The embodiments shown will not be described in detail here.

[0089] Alternatively, the electronic device can also generate corresponding captured images based on the multiple frames of images to be processed, provided that none of the other images in the multiple frames have a similarity greater than or equal to the similarity threshold. For example, Figure 1B This is a schematic diagram illustrating the effect of an image processing method according to an exemplary embodiment of the present disclosure. For example... Figure 1B As shown, if multiple frames of images to be processed include image 1, image 2, ..., image n, and after similarity calculation, it is found that images 1 to n are several images with a similarity greater than or equal to the similarity threshold, then images 1 to n can be fused to obtain a fused image, which is then processed through a post-processing process to obtain the captured image corresponding to images 1 to n (that is, after images 1 to n are processed by the image processing method of this embodiment, a captured image can be obtained); conversely, if the similarity between images 1 to n is less than the similarity threshold, these images can be directly processed through a post-processing process to obtain the captured images corresponding to images 1 to n respectively (that is, each frame of images to be processed corresponds to a captured image, and a total of n captured images can be obtained).

[0090] It is worth noting that when there are a number of images (e.g., L frames, where L is less than M) in a set of multiple images to be processed (e.g., M frames) with a similarity greater than or equal to the similarity threshold, the L frames can be fused and then post-processed. The remaining M frames are not fused but only post-processed.

[0091] As described above, the method of this embodiment calculates the similarity of multiple frames of images to be processed, and in response to the existence of several images in the multiple frames of images to be processed with a similarity greater than or equal to a similarity threshold, fuses the several images to obtain a fused image, and then generates a captured image corresponding to the several images based on the fused image. Since several images with high similarity in the multiple frames of images to be processed are fused, the number of subsequently generated captured images is less than the number of the several images. This not only speeds up the shooting process but also reduces the number of duplicate captured images, saves device storage space, and avoids users manually deleting duplicate images, thereby improving the user experience.

[0092] Figure 2 This is a flowchart illustrating an image processing method according to another exemplary embodiment of the present disclosure; the method of this embodiment can be executed by an image processing device, which can be configured in an electronic device (such as a mobile phone, tablet computer, camera, camcorder, etc.). Figure 2 As shown, the method includes the following steps S201-S205:

[0093] In step S201, in response to receiving the raw image output by the image sensor, the raw image is processed based on the real-time pipeline to obtain a real-time pipeline image.

[0094] In this embodiment, the image sensor can output the acquired raw image, and then the electronic device, in response to receiving the raw image output by the image sensor, processes the raw image based on the real-time pipeline to obtain a real-time pipeline image.

[0095] For example, an image sensor can output the acquired raw image to the real-time pipeline of an ISP (Image Signal Processor) for processing.

[0096] In related technologies, a real-time pipeline can refer to a series of algorithms and techniques used in digital cameras, smartphones, or other electronic devices with image capture capabilities to process and optimize raw image data received from image sensors in real time. The design goal of this real-time pipeline is to transform raw image data (i.e., unprocessed image data) into high-quality images usable on displays or storage media in a short time with superior performance.

[0097] In step S202, in response to receiving an image acquisition instruction, the real-time pipeline image is cached in an image cache queue.

[0098] In this embodiment, after the original image is processed based on the real-time pipeline to obtain the real-time pipeline image, if the electronic device receives an image acquisition instruction (e.g., an instruction issued by clicking a camera button), the real-time pipeline image can be cached in the image cache queue.

[0099] Understandably, when a user presses the shutter button, they expect the electronic device to respond immediately and capture an image. However, since image processing and storage on permanent storage media (such as an SD card or internal storage) can take some time, using a cache ensures that the user can continue shooting the next frame even while the previous frame is being processed and stored.

[0100] In step S203, multiple frames of the real-time pipeline images are extracted from the image cache queue and used as multiple frames of images to be processed for similarity calculation.

[0101] In this embodiment, after the real-time pipeline images are cached in the image cache queue, if the number of images in the image cache queue is detected to reach a preset number threshold (e.g., N, where N is a positive integer), then N frames of the real-time pipeline images can be extracted from the image cache queue and used as multiple frames of images to be processed for similarity calculation.

[0102] It is worth noting that caching allows electronic devices to continue capturing new images while processing and storing images in the background. This is important for burst mode, which enables users to take multiple photos in succession without waiting for each photo to be fully processed and stored, thus improving the efficiency of similarity calculation.

[0103] In step S204, in response to the existence of several images with a similarity greater than or equal to a similarity threshold in the multiple frames of images to be processed, the several images are fused to obtain a fused image;

[0104] In step S205, a captured image corresponding to the plurality of images is generated based on the fused image, wherein the number of captured images is less than the number of the plurality of images.

[0105] The explanations and descriptions of steps S204-S205 can be found above. Figure 1A Steps S102-S103 in the illustrated embodiment will not be described in detail here.

[0106] As described above, this embodiment processes the original image based on the real-time pipeline in response to receiving the original image output from the image sensor to obtain a real-time pipeline image. In response to receiving an image acquisition command, the real-time pipeline image is cached in an image cache queue, and multiple frames of the real-time pipeline image are extracted from the image cache queue and used as multiple frames to be processed for similarity calculation. This can improve the image quality and the efficiency of similarity calculation, thereby improving the quality and efficiency of subsequent image fusion and the generation of captured images.

[0107] Figure 3 This is a flowchart illustrating how to fuse the plurality of images according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to fuse the plurality of images as an example for illustrative explanation.

[0108] like Figure 3 As shown, the fusion of the plurality of images described in step S102 or S204 above may include the following steps S301-S304:

[0109] In step S301, a frame image that meets the image quality requirements is selected from the plurality of images and used as a reference frame image.

[0110] In this embodiment, when fusing the plurality of images, a frame image whose image quality meets the requirements can be selected from the plurality of images and used as a reference frame image.

[0111] The method for selecting a frame that meets the image quality requirements can be found in relevant technical documents, and this embodiment does not limit this method.

[0112] For example, when selecting a frame image that meets the image quality requirements from the plurality of images, a frame image whose contrast and / or signal-to-noise ratio meet the corresponding conditions can be selected from the plurality of images. That is to say, meeting the image quality requirements in this embodiment may include meeting the corresponding conditions for contrast and / or signal-to-noise ratio.

[0113] It is understood that the aforementioned reference frame image can be understood as the "anchor frame" among the several images, and the anchor frame among the several images is usually a high-quality frame among the several images.

[0114] In step S302, the inter-frame differences between the reference frame image and other frame images (excluding the reference frame image) are determined.

[0115] In this embodiment, after selecting the reference frame image among the plurality of images, the inter-frame differences between the reference frame image and other frame images among the plurality of images can be determined.

[0116] For example, metrics from related technologies (such as pixel difference, structural similarity index SSIM, etc.) can be used to compare the differences between other frame images and reference frame images.

[0117] In step S303, the fusion weights of the other frame images to which it belongs are determined based on the inter-frame differences.

[0118] In this embodiment, after determining the inter-frame differences between the reference frame image and other frames (excluding the reference frame image), the fusion weights of the other frames can be determined based on these inter-frame differences. The fusion weight of the same frame image is negatively correlated with the inter-frame differences; for example, the smaller the difference, the higher the weight, and vice versa. This fully utilizes the advantages of each frame image to reflect richer and more comprehensive information in the subsequently obtained fused image, while reducing the impact of noise on image quality.

[0119] In step S304, the plurality of images are weighted and summed based on the fusion weights to obtain the fused image.

[0120] In this embodiment, after determining the fusion weights of other frame images to which it belongs based on the inter-frame differences, the fusion weights can be used to perform a weighted sum operation on the several images to obtain the fused image.

[0121] As described above, this embodiment selects a frame image whose image quality meets the requirements from the plurality of images and uses it as a reference frame image. Then, it determines the inter-frame differences between the other frames in the plurality of images (excluding the reference frame image) and the reference frame image, and determines the fusion weights of the other frames based on the inter-frame differences. Then, it performs a weighted sum operation on the plurality of images based on the fusion weights to obtain the fused image. By using the inter-frame differences between each frame image and the reference frame image with better quality to assign weights to each frame image, the advantages of each frame image can be fully utilized, thereby reflecting richer and more comprehensive information in the fused image and reducing the impact of noise on image quality, thus improving the quality of the fused image.

[0122] Figure 4A This is a flowchart illustrating how to generate a captured image corresponding to the plurality of images based on the fused image according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to generate a captured image corresponding to the plurality of images based on the fused image as an example for illustrative explanation.

[0123] like Figure 4A As shown, the step S103 or S205 above, which involves generating a captured image corresponding to the plurality of images based on the fused image, may include the following steps S401-S402:

[0124] In step S401, the fused image is processed based on the offline pipeline to obtain an offline pipeline image.

[0125] In this embodiment, after the several images are fused to obtain a fused image, the fused image can be processed based on the offline pipeline to obtain an offline pipeline image.

[0126] It is understood that the offline pipeline in related technologies can be used to further process the data output from the real-time pipeline and transmit the zoom level data of the camera application corresponding to the real-time pipeline to the application framework layer. In this embodiment, the fused image is processed based on the offline pipeline. Since the image can be processed in an environment without a network connection, the security of the image data can be ensured, and the processing speed of the fused image can be improved.

[0127] In step S402, the offline pipeline image is encoded based on an image encoder to obtain the captured image.

[0128] In this embodiment, after processing the fused image based on the offline pipeline to obtain the offline pipeline image, the offline pipeline image can be encoded based on the image encoder to obtain the captured image.

[0129] It is worth noting that the type of image encoder mentioned above can be set based on scenario requirements, such as setting it to a JPEG encoder, etc., and this embodiment does not limit this. Taking a JPEG encoder as an example, after encoding the offline pipeline image, a JPEG captured image can be obtained. It is understood that JPEG encoding has advantages such as high compression ratio, controllable image quality, wide compatibility, and high color fidelity, thus ensuring the quality of the captured image after encoding, thereby improving the user's shooting experience.

[0130] As described above, this embodiment processes the fused image based on the offline pipeline to obtain an offline pipeline image, and encodes the offline pipeline image based on an image encoder to obtain the captured image. This enables efficient and secure processing of the fused image using the offline pipeline, and the image can be processed into the required format (e.g., JPEG format). In other words, it enables the generation of a captured image corresponding to the several images based on the fused image with high quality, thereby improving the user's shooting experience.

[0131] For example, Figure 4B This is a logical schematic diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. For example... Figure 4B As shown, the image sensor outputs raw image data to the ISP's real-time pipeline for processing. If a capture command is received, the image data is cached in a buffer queue. When the number of images in the buffer queue reaches a preset threshold (e.g., N, where N is a positive integer), N frames of the real-time pipeline images are extracted from the buffer queue and used as multiple frames to be processed for similarity calculation. Furthermore, if several images with a similarity greater than or equal to the similarity threshold are detected among these multiple frames, noise reduction is performed on each of these images individually, followed by fusion of the denoised images (the specific fusion method can be found above). Figure 3 The embodiment shown (not described in detail here) obtains a fused image, which is then processed by an offline pipeline and an image encoder to obtain a captured image corresponding to the plurality of images; however, when there are no plurality of images in the plurality of images to be processed with a similarity greater than or equal to the similarity threshold, the plurality of images to be processed can be directly processed by an offline pipeline and an image encoder to obtain captured images corresponding to the plurality of images to be processed respectively.

[0132] As described above, this embodiment can improve the signal-to-noise ratio of subsequent captured images by performing noise reduction processing on the images, thereby improving the image quality. Furthermore, by fusing multiple similar image frames into one image, it can save the device's power consumption and storage space, save users the trouble of manually deleting duplicate images, improve the device's shooting speed, and break through the limitation on the number of images captured in continuous shooting mode, thereby improving the user experience.

[0133] Figure 5 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure; the apparatus of this embodiment can be configured in an electronic device (such as a mobile phone, tablet computer, camera, camcorder, etc.). Figure 5As shown, the device may include: a similarity calculation module 110, an image fusion module 120, and an image generation module 130, wherein:

[0134] The similarity calculation module 110 is used to calculate the similarity of the currently acquired multiple frames of images to be processed;

[0135] The image fusion module 120 is used to fuse the images in response to the existence of several images with a similarity greater than or equal to a similarity threshold in the multiple frames of images to be processed, thereby obtaining a fused image.

[0136] The image generation module 130 is used to generate captured images corresponding to the plurality of images based on the fused image, wherein the number of captured images is less than the number of the plurality of images.

[0137] As described above, the device in this embodiment calculates the similarity of multiple frames of images to be processed, and in response to the existence of several images in the multiple frames of images to be processed with a similarity greater than or equal to a similarity threshold, fuses the several images to obtain a fused image, and then generates a captured image corresponding to the several images based on the fused image. The number of captured images is less than the number of several images. Since several images with high similarity in the multiple frames of images to be processed are fused, the number of subsequently generated captured images is less than the number of several images. This not only speeds up the shooting process but also reduces the number of duplicate captured images, saves device storage space, and avoids users manually deleting duplicate images, thereby improving the user experience.

[0138] Figure 6 This is a block diagram illustrating another image processing apparatus according to an exemplary embodiment of the present disclosure; the apparatus of this embodiment can be configured in an electronic device (such as a mobile phone, tablet computer, camera, camcorder, etc.). The similarity calculation module 210, image fusion module 220, and image generation module 230 are as described above. Figure 5 The similarity calculation module 110, image fusion module 120 and image generation module 130 in the illustrated embodiment have the same function, and will not be described in detail here.

[0139] The above-mentioned device may also include an image buffer module 240;

[0140] Image caching module 240 may include:

[0141] Real-time processing unit 241 is configured to process the raw image based on the real-time pipeline in response to receiving the raw image output by the image sensor, thereby obtaining a real-time pipeline image.

[0142] Image caching unit 242 is used to cache the real-time pipeline image into an image caching queue in response to receiving an image acquisition instruction;

[0143] Furthermore, the similarity calculation module 210 can also be used to extract multiple frames of the real-time pipeline images from the image cache queue and use them as the multiple frames to be processed for similarity calculation.

[0144] In some embodiments, the above-described apparatus may further include:

[0145] The noise reduction processing module 250 is used to perform noise reduction processing on the plurality of images respectively.

[0146] In some embodiments, the image fusion module 220 described above may include:

[0147] The image selection unit 221 is used to select a frame image from the plurality of images that meets the image quality requirements, and use it as a reference frame image;

[0148] The difference determination unit 222 is used to determine the inter-frame difference between the reference frame image and other frame images in the plurality of images, excluding the reference frame image.

[0149] The weight determination unit 223 is used to determine the fusion weight of other frame images to which it belongs based on the inter-frame differences. The fusion weight of the same frame image is negatively correlated with the inter-frame differences.

[0150] The image fusion unit 224 is used to perform a weighted sum operation on the plurality of images based on the fusion weights to obtain the fused image.

[0151] In some embodiments, the image selection unit 221 described above can also be used to select a frame of image from the plurality of images whose contrast and / or signal-to-noise ratio meet the corresponding conditions.

[0152] In some embodiments, the image generation module 230 described above may include:

[0153] Offline processing unit 231 is used to process the fused image based on the offline pipeline to obtain an offline pipeline image;

[0154] Image encoding unit 232 is used to encode the offline pipeline image based on the image encoder to obtain the captured image.

[0155] In some embodiments, the above-described apparatus may further include:

[0156] Other generation modules 260 are used to generate corresponding captured images based on the multiple frames of images to be processed, in response to the absence of several images in the multiple frames of images to be processed having a similarity greater than or equal to the similarity threshold.

[0157] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0158] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0159] Reference Figure 7 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0160] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the image processing method described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0161] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0162] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900.

[0163] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display panel and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0164] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0165] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0166] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 can detect the on / off state of device 900, the relative positioning of components such as the display panel and keypad of device 900, changes in the position of device 900 or a component of device 900, the presence or absence of user contact with device 900, the orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0167] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0168] In an exemplary embodiment, device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image processing method described above.

[0169] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of the device 900 to complete the image processing method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0170] In an exemplary embodiment, a computer program product including instructions is also provided, which can be executed by the processor 920 of the device 900 to perform the image processing method described above.

[0171] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the foregoing claims.

[0172] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Calculate the similarity of the currently acquired multiple frames of images to be processed; In response to the existence of several images with a similarity greater than or equal to a similarity threshold in the multiple frames of images to be processed, the several images are fused to obtain a fused image; Based on the fused image, a captured image corresponding to the plurality of images is generated, wherein the number of captured images is less than the number of the plurality of images.

2. The method according to claim 1, characterized in that, The method further includes: In response to receiving the raw image output from the image sensor, the raw image is processed based on the real-time pipeline to obtain a real-time pipeline image; In response to receiving an image acquisition command, the real-time pipeline image is cached in an image cache queue; The similarity calculation of the currently acquired multiple frames of images to be processed includes: Multiple frames of the real-time pipeline images are extracted from the image cache queue and used as the multiple frames to be processed for similarity calculation.

3. The method according to claim 1, characterized in that, Before fusing the plurality of images, the method further includes: The images are then subjected to noise reduction processing.

4. The method according to claim 1 or 3, characterized in that, The process of fusing the plurality of images to obtain a fused image includes: Select one frame image from the plurality of images that meets the image quality requirements, and use it as the reference frame image; Determine the inter-frame differences between the reference frame image and other frame images in the plurality of images; The fusion weights of other frame images to which it belongs are determined based on the inter-frame differences, and the fusion weights of the same frame image are negatively correlated with the inter-frame differences; The fused image is obtained by performing a weighted sum operation on the plurality of images based on the fusion weights.

5. The method according to claim 4, characterized in that, Selecting a frame image from the plurality of images that meets the image quality requirements includes: Select one frame from the plurality of images whose contrast and / or signal-to-noise ratio meet the corresponding conditions.

6. The method according to claim 1, characterized in that, The step of generating a captured image corresponding to the plurality of images based on the fused image includes: The fused image is processed based on the offline pipeline to obtain an offline pipeline image; The offline pipeline image is encoded using an image encoder to obtain the captured image.

7. The method according to claim 1, characterized in that, The method further includes: In response to the absence of several images with a similarity greater than or equal to the similarity threshold among the multiple frames to be processed, corresponding captured images are generated based on the multiple frames to be processed.

8. An image processing apparatus, characterized in that, The device includes: The similarity calculation module is used to calculate the similarity of the currently acquired multiple frames of images to be processed. An image fusion module is used to fuse several images in the multiple frames of images to be processed, where the similarity is greater than or equal to a similarity threshold, to obtain a fused image. An image generation module is used to generate captured images corresponding to the plurality of images based on the fused image, wherein the number of captured images is less than the number of the plurality of images.

9. An electronic device, characterized in that, The device includes: Processor and memory used to store computer programs; The processor is configured to implement the image processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the image processing method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method according to any one of claims 1 to 7.

Images